A while ago, I put together a test sample of 15,569 CPAN metadata files from a minicpan repository. I'm happy to report that with the latest release today, CPAN::Meta can handle almost 99% of metafiles found "in the wild". Here is the breakdown:

• no errors: 11411 (73.29%)
• fixable errors: 3994 (25.65%)
• parser errors: 124 (0.80%)
• missing 'name' or 'version' fields: 32 (0.21%)
• remaining errors (not fixable): 8 (0.05%)

The "unfixable" errors range from missing mandatory fields deep within the structure that can't be assumed from context to invalid module names or else just strange data that somehow slipped into a metafile.

The "torture" part of the test was partly a test of my own patience, since I was tweaking CPAN::Meta and re-running the test against the 15,000 distributions over and over again. One quick-win worth mentioning was Parallel::Iterator, which let me take advantage of all four cores of my CPU. As expected, it cut the runtime by nearly three-quarters (11.5 seconds vs 38.3 seconds). If anyone is interested in seeing how I used it, my quick-and-dirty CPAN::Meta torture program is available online.

Dist::Zilla is modular, not monolithic

Part of the confusion about dzil is that it can do a lot of different things and not everyone likes everything it can do. At its core, dzil is just a CPAN distribution release manager:

• Collect distribution files into a distribution directory
• Create a tarball of the directory

This is no big deal -- it duplicates the functionality of ExtUtils::MakeMaker, Module::Build and Module::Install. The difference is that it doesn't require running Makefile.PL or Build.PL to work.

What does this do for us? How does this help a Perl programmer release stuff to CPAN? (Hi, TIOBE!)

With dzil, a developer can customize the release process outside Makefile.PL or Build.PL, which means that end-users never need to know about it. They don't need prerequisites just to run the PL file, they don't need things tucked away in inc/, they don't need to write Makefile snippets, and so on.

Instead, developers can extend dzil with plugins that stay on their development machine. Almost everything in dzil is a plugin -- there is no distinction between "core" behavior and "add-ons". Unlike Module::Build, which is easy to subclass once but makes it hard to "mix-in" subclass functionality, dzil is built from the ground up to coordinate the behavior of multiple plugins. This lets a developer mix and match useful functions whenever desired.

For example, it's easy to add a plugin to upload things to CPAN automatically. Once added, here's what dzil does for me:

• Collect distribution files into a distribution directory
• Create a tarball of the directory
• Upload the tarball to CPAN

Replacing existing release scripts with Dist::Zilla

Before dzil, I already had some personal scripts that I wrote years ago to manage my uploads. They worked, but were hard to extend and customize to do different things for different distributions. I used them to do some useful sanity checks on my distributions before uploading.

But dzil can provide the same functionality through yet more plugins, which lets dzil manage my distributions like this:

• Collect distribution files into a distribution directory
• Make sure everything is checked into git
• Make sure distribution tests pass
• Tag the release with git
• Push the tag to my git repo
• Create a tarball of the directory
• Upload the tarball to CPAN

If I find that someone has already written other useful pre-release plugins, I can just drop them in:

• Make sure additional tests in the xt/ directory pass
• Make sure I've written something in the Changes file

One of the knocks on dzil is that the files in your repository aren't the same as the final release but note that in the examples above no files have been modified before release.

dzil can do that -- and I'll show how and why I use it that way -- but that's a choice that every developer can make. Even without it, I can benefit from all the release management plugins that people write -- something that I never could do with my homegrown release scripts. Cool!

Letting Dist::Zilla automate busywork

If you're ready to take the next step with dzil, you can have it produce boilerplate files in your distribution -- things that ought to be there, but that you don't really need in the repository or that should be generated on the fly. This can include simple text files like LICENSE, but when you realize that you no longer need your Makefile.PL or Build.PL file to build your distribution, you can let dzil fill in the blanks of a generic template for those, too! You can even let it take a guess at your prerequisites, eliminating one more thing to remember to update before release.

Now my release process in dzil might look like this:

• Collect distribution files into a distribution directory
• Examine the code to find prerequisites (including minimum versions)
• Generate a Makefile.PL (or Build.PL)
• Generate a MANIFEST
• Generate a META.yml and/or META.json files
• Generate a LICENSE file
• Generate a README file from the main module's Pod
• Generate some generic *.t files
• Make sure everything is checked into git
• Make sure distribution tests pass
• Make sure additional tests in the xt/ directory pass
• Make sure I've written something in the Changes file
• Tag the release with git
• Create a tarball of the directory
• Upload the tarball to CPAN

And I still haven't modified any files -- I'm only adding files in the distribution directory so I don't have to generate them in the first place or maintain them by hand anymore.

But wait! Without a Makefile.PL or a Build.PL, how do you test the distribution?

You can do it one of two ways:

$ prove -l t
$ dzil test

The first will just run your tests using the files in lib/ under prove (add the -v flag if you want, or specify a particular file to test). The second will go through all the steps of building the distribution directory and generating files and then will run Makefile.PL (or Build.PL) and "make test" ("Build test") within the generated distribution.

If you're willing to take the next step and let dzil modify your files as it puts them into the distribution directory, you can start leaving out boilerplate from your files and have dzil add it automatically. It can also figure out easily computed things -- like the next version number to use -- and add those too.

Here are some of the things I let it do for me:

• Calculate the next version and write $VERSION into my files
• Find an abstract and write a NAME section in the Pod for every module
• Write the $VERSION into the Pod for every module
• Add AUTHORS and COPYRIGHT sections to the Pod for every module
• Add a formal copyright and license statement to every file
• Translate the WikiDoc Pod dialect I prefer into regular Pod
• Add the $VERSION and timestamp to my Changes file

That's a lot of busy-work I no longer have to think about. I can just focus on code and tests and leave all the scaffolding to dzil. As the number of distributions I maintain grows, that's a huge time savings and it enforces rigor and consistency to my releases. I will never again forget to timestamp a Changes file, or rather, I can now always forget and just let dzil do it for me.

It does mean that sometimes I'll get a bug report or a patch against the release and I have to do a little extra work to translate it back to a fix in the repository when line numbers don't quite match. This is a cost that I'm willing to bear in exchange for the productivity boost I get the rest of the time.

Whether to use Dist::Zilla and how you can get started

Hopefully, I've shown how dzil is useful and how you can choose to use as much or as little as you'd like. Nevertheless, here are some reasons why you might not want to use it, or at least not yet:

• There is limited support (so far) for customizing the build process, supporting dynamic (e.g. OS-specific) prerequisites and building XS modules. This is getting better as authors write new plugins -- see Dist::Zilla::Plugin::MakeMaker::Awesome as an example.
• There are lots of plugins to keep track of and configuring them manually in each distribution can feel a bit tedious. Some authors have been writing PluginBundles to encapsulate their default configuration. Will every author need one of these or will some sort of global configuration be possible? These are questions still being worked out.
• Collaboration is a bit harder. Ideally, any collaborators will install the same set of dzil plugins that you have and can work with dzil the same way you do, but that's still extra work that you're asking of contributors, particularly if they aren't already fans of dzil.
• dzil is still evolving rapidly and over the last few months, it's not been uncommon to have some upgrade of dzil break a bunch of plugins. Hopefully this will be less common as the API stabilizes and the pace of core dzil development slows. Because of the pace of development, documentation has also lagged a bit.

If you do want to give it a try, my advice on getting up the learning curve is to start with a very simple dzil dist.ini file that you maintain by hand. Don't use dzil PluginBundles until you're more familiar with the action of each plugin. The dzil tutorial is a great reference for getting started. See the section on converting an existing distribution for some good starter examples.

Alternatively, if you want to see an example dist.ini that does all the things I mention above in this article, see Dist::Zilla::PluginBundle::DAGOLDEN. In the documentation for the PluginBundle, I show the equivalent dist.ini. You can comment out parts you don't understand and then selectively re-enable them to see what they do.

See Walking through a real dist.ini for another good example.

Important: While you're playing around learning dzil, replace the "UploadToCPAN" plugin with "FakeRelease" in your dist.ini. This will simulate a release without actually uploading to CPAN, which could save you some potential embarrassment.

Articles by other people

• Dist::Zilla -- part 1
• My new love/hate relationship with Dist::Zilla
• Dist::Zilla Pros and Cons

Unfortunately, some CPAN metadata consumers aren't quite ready yet for version 2 of the Meta Spec, as this example shows:

Hopefully, search.cpan.org and other sites will be updated soon. One of the nice advantages of the version 2 Meta spec is that it can differentiate between a repository access URL and a repository web browser URL. So search.cpan.org will be able to provide a hyperlink to a repo browser instead of (or in addition to) the URL that you would give to your version control client.

(In unrelated news, I am still voting for mst to give a talk called "Patches Welcome" as his forfeit for losing the Iron Man competition.)

It is with great pleasure that we present you with the release of the CPAN Meta Spec version 2.

This document specifies the format for the next version of CPAN distribution metadata files -- the META.yml and now META.json files found in most distributions on CPAN.

Version 2 is the result of a call for proposals last year that resulted in 34 proposals and over 300 discussion emails. The final proposal was reviewed by a working group consisting of the maintainers of most of the major Perl toolchain modules, administrators of the major CPAN indexing sites and representatives from several major linux distributions that package CPAN libraries.

The new release includes not only the actual specification but new support libraries for the toolchain to work with META files in a more consistent way.

As support is added to tools like Module::Build, Module::Install or Dist::Zilla over the coming months, CPAN authors will see several new options for providing additional information about their distributions.

Some notable changes to the specification include:

• use of JSON as the preferred serialization format (i.e. META.json)
• prereqs are more granular, allowing every "action" (e.g. 'build', 'test') to have every type of relationship (e.g. 'requires', 'recommends')
• adoption of Debian-inspired semantics for 'recommends' and 'suggests' for optional prereqs
• supports listing multiple licenses
• clearer rules for adding custom fields
• an indicator of release stability
• improved options for pointing to related resources, particularly code repositories

The specification version is also changing to an integer, so it can be the "major" number on CPAN::Meta and the decimals can reflect bug fixes or other non-specification changes to the library that implements any particular spec version.

Over the next month or so, we'll be blogging about some of the details of the change.

Thank you to Ken Williams for creating the initial 'META.yml' spec and maintaining it through version 1.4 and thank you to everyone who contributed ideas, patches, discussion and critique in the development of the version 2 spec.

Sincerely,
Ricardo Signes and David Golden

The new repository is available publicly on github.com:

• Browse the repository
• Clone it: git://github.com/dagolden/module-build.git

I have done my best to clean up the merge/branch/tag history, but have not bothered to clean up empty commits left over from the original cvs2svn conversion. If anyone sees any glaring errors please let me know.

I hope this migration makes it easy for people to contribute to Module-Build. I was pleasantly surprised to find that within 12 hours of it being publishing on github, it's already being watched by several people.

At the very least, it will make working on feature branches much easier to manage, which will make it easier to experiment with new ideas without affecting the main line of development.

If anyone would like to start applying patches from the Module::Build bug queue, or creating patches for other open tickets, that would be huge help.

I will be "closing out" the old Subversion repository with a pointer to the new location shortly.

There is a branch called "cplusplus" in the EU::CB repository on github. (See the branches dropdown.)

Work is loosely coordinated on the #toolchain IRC channel on irc.perl.org. Ask for user "Sno|Laptop" if you would like to contribute in any way.

Here's the story: I'm a recent convert to Dist::Zilla. I really like it -- if I had more time I'd write a longer post about how easy it makes your life as a developer.

The problem is that it can make it too easy. I was writing a Dist::Zilla plugin to check "xt" tests before release. Unfortunately, the @Classic plugin that I added to a test includes the UploadToCPAN plugin -- which, if you have a .pause file with your credentials, will helpfully upload modules straight to CPAN.

You can probably see where this is going. Yes, I ran my tests and promptly uploaded a test distribution called "Foo-1.23" to CPAN. Thankfully, the "Foo" namespace is already claimed, so it's an unauthorized distribution and doesn't get indexed. I've already scheduled it for deletion, so hopefully, by the time you read this, it will just be a bad memory.

But, my mistake aside, Dist::Zilla rocks. If you develop for CPAN, you should really give it a try.

Just be very careful when you type 'dzil release'.

To help out anyone else that needs to do this (and to help my future self), I have over time created maintenance branches for every major release of perl from 5.6.1 to 5.8.8 in my perl github repo. You may need some extra fiddling to make some things build or keep them from building, but it should get you a lot closer. For example, on perl 5.6.1, you need to disable DB_File with a -Ui_db to Configure.

Second, you may need to get the toolchain working on the old perl to deal with dependencies. I have a couple pieces of advice there:

• Don't run perl via a symlink. If you have multiple perl interpreters in your PATH, your Makefile might wind up targeting the first perl in the PATH instead of the one you used to run Makefile.PL
• Install ExtUtils::MakeMaker by hand as the very first thing you do. This will definitely help your bootstrapping.

That's my wisdom gained from a couple hours lost tonight. I hope it saves someone else some time.

It's a remarkable achievement, not only technically, but in the reaction it has sparked. As one of the (sometimes reluctant) maintainers of CPAN and CPANPLUS, I've realized that I both love and hate cpanminus.

• I love that Miyagawa has done so much with so little and in such a short span of time
• I hate that fanboy-types flocked to it and trashed the older clients without noting cpanminus' limitations
• I love that Perl toolchain maintainers have rallied around Miyagawa and contributed their wisdom to make cpanminus better instead of rejecting it
• I hate that one of Perl's great strengths (CPAN) has legacy clients that are so unwieldy, hated and difficult to maintain

Miyagawa graciously acknowledges standing on the shoulders of giants. Still, I can't shake the nagging thought that cpanminus should never have been necessary in the first place.

What I've come to realize is that cpanminus is another example of the power of not being all things to all people. Miyagawa doesn't promise that it works for all of CPAN or that it works everywhere that Perl does. He doesn't have to. Making it work for 99% of CPAN for 99% of people is more than good enough.

I've been co-maintaining various parts of the Perl toolchain for a while now. It's a frustrating challenge needing to make thing work everywhere, for everything, and trying as hard as possible not to break backwards compatibility. Plus, I don't even get to use CPAN to make life easier. I don't get to use handy tools like Moose or DateTime or Regexp::Common or SQLite or anything in the Config::* namespace or even basic tools like Archive::Zip. Nearly everything is done by hand.

Things have to work with just core Perl on a diverse set of platforms and with an incredibly limited set of assumptions. For example, the Perl core still doesn't come with an HTTP client, so CPAN has to rely on FTP or command line programs to bootstrap LWP. (This is something I personally plan to tackle during the Perl 5.13 development series later this year.)

I think this is an ongoing challenge for core Perl development in general. It's a lot of work to be all things to all people and I keep wondering whether making things simpler and better for 99% of people would be a better choice. (Anyone else for use strict by default? I hope that finally comes to pass in Perl 5.14.) chromatic writes about this topic often in his Modern Perl blog and I usually tend to agree with the points he makes. (October February 2009 had a particularly good series of posts.)

In the meantime, I look at cpanminus with greed and envy. Miyagawa++