Paying respect to Module::Build

Earlier this week, I proposed that Module::Build be deprecated from the Perl 5 core. After some discussion, this proposal has been accepted by the Pumpking.

I want to take a moment to discuss what this means, why I suggested it, and what I think Module::Build trail blazed for Perl 5.

Deprecation means a warning in v5.19 and removal in v5.22

Soon, in Perl 5 version 19 (the current development series), the Module::Build included in the core library path will issue a warning when used. If Module::Build is then installed from CPAN, it will be installed into the "sitelib" path and the deprecation warning will stop.

Sometime in Perl 5 version 21 (starting late Q2 2014), Module::Build will most likely no longer ship with the core Perl and will need to be installed from CPAN.

Fortunately, CPAN clients already recognize a 'configure requires' directive in CPAN distribution meta files (META.yml or META.json) and can bootstrap Module::Build before Build.PL runs.

So... the biggest impact on end-users will be that when Perl 5 version 20 is released in Q2 2014, people will have to install Module::Build from CPAN to squelch the deprecation warning.

Update: Miyagawa reminded me on IRC that the major CPAN clients will install from CPAN automatically if they see a deprecated module in a prereq list, so actually, most people won't even notice. Shiny!

Module::Build was good, but not good enough, and it plateaued

Before Module::Build, we only had ExtUtils::MakeMaker and it sucked. Michael Schwern, its long-time maintainer, even wrote a presentation called MakeMaker is DOOMED! that encouraged people to switch to Module::Build. In hindsight, this was premature.

For all the many problems that Module::Build fixed, it introduced some of its own, built up its own technical debt, and suffered a crisis of maintenance.

I tried to roughly estimate the amount of effort going into Module::Build during four phases of its life. I used lines of Changes file as my metric (though I think "git diff --stats" would be fairly similar):

  • 2001-2007: Ken Williams author and maintainer → 2397 lines of Changes
  • 2007-2009: Ken and Eric Wilhelm tag-team → 310 lines of Changes
  • 2009-2011: David Golden maintainer → 1033 lines of Changes
  • 2011-now: Leon Timmermans maintainer → 95 lines of Changes

After my announcement that I was stepping down as Module::Build maintainer, no one volunteered for seven months until Leon kindly offered to be a "caretaker" and shepherd some patches and releases -- partly as a side effect of his work on a Module::Build replacement called Module::Build::Tiny, which itself was a serious spin off of a half-joke of my own called Acme::Module::Build::Tiny.

Module::Build innovated things now taken for granted

The best thing that Module::Build did was define a de facto specification for using a Build.PL to drive a perl-based (rather than Makefile-based) install program. That work has been formalized into a Build.PL Spec, so other Perl-based builders can be developed.

Module::Build also introduced the META.yml file that evolved into the CPAN::Meta::Spec that is in widespread use today. The META.yml file also helped solve a tricky bootstrapping problem: by specifying configure_requires dependencies within the META file, CPAN clients could install whatever modules were necessary to run Build.PL.

With the release of Perl v5.10, both CPAN and CPANPLUS supported configure_requires, meaning that the groundwork for future Build.PL-based alternatives was already in place!

Module::Build also introduced the install_base parameter as a way to specify a custom install location. It was much easier to understand than PREFIX from ExtUtils::MakeMaker, and was subsequently adopted by ExtUtils::MakeMaker as INSTALL_BASE. This is a critical part of the magic behind tools like local::lib and Carton.

The other crucial innovation was that — for the first time — customizing the build, test and install process could be done by writing only Perl code rather than writing Perl code to spit out Makefile fragments. It made building complex modules much easier — particularly Alien modules like Alien::wxWidgets. That then made projects like Padre possible.

Module::Build also spawned a counter-reaction in the form of Module::Install, which tried to make the easy Perl customization possible, while shielding users from pure ExtUtils::MakeMaker and avoiding the bootstrap problems of Module::Build by bundling itself in inc/. Module::Install then triggered a counter-reaction in the form of Dist::Zilla, which then led to Dist::Milla and Minilla.

Module::Build was the trail blazer for the tools that came after.

Module::Build made its own unique mistakes

People have complained that Module::Build was bloated. In lines of code, it's actually comparable to ExtUtils::MakeMaker. The bigger problem is that it puts 4,200 of its 5,800 lines of code in just one file: Module::Build::Base. (ExtUtils::MakeMaker split similar functionality across three mega files.)

More than just size, Module::Build is complex. The Build.PL file runs configuration and serializes the results into some files, which the Build file uses to reconstruct the original Module::Build object. Arguments can modify properties at any stage. And since Build.PL might really be a subclass, there's a lot of meta object stuff going on just to manage the configuration before ever getting around to the real business of building and installing modules.

It also suffered from feature creep. Instead of just being an install tool, it became a swiss-army-knife author's tool, with release-time features never needed by end-users, but which forced end-users to upgrade Module::Build just to run Build.PL without error. It added new concepts, like "optional features", which were poorly specified and have never achieved much traction.

One of the big, valid complaints is that it never incorporated a proper dependency system. Actions (build, test, etc.) could depend on each other, but there was nothing like Makefile's ability to detect that since file "A" changed, then action "B" had to run.

My personal pet peeve — possibly one of the big reasons I got discouraged doing maintenance — was that it also included Module::Build::Compat, which was used to generate a Makefile.PL from the Build.PL. While this seemed like a benefit to ease transition, it meant that Module::Build needs to maintain feature-compatibility — and in many cases bug-compatibility — with ExtUtils::MakeMaker effectively forever.

Module::Build promised easy subclassing and this was mostly true. But re-use and sharing was nearly impossible. If you had a subclass to do one thing and I had a subclass that did something else and you wanted to combine them, you pretty much had to copy and paste code. Contrast that with Dist::Zilla's incredible plugin ecosystem — where just about anything you want to do has been written up into a plugin that you can just drop in.

Module::Build will live on as a CPAN distribution

Module::Build never became the uncontested successor to ExtUtils::MakeMaker. It's not used as part of the Perl 5 core build process. It originally went in at least in part to ease adoption, but now all CPAN clients can bootstrap it on demand.

It's not a bad module, but it has no reason to live in the Perl 5 core any more.
Removing it means one less thing for the already-stretched Perl 5 porters to maintain, update and support.

Module::Build helped us through a critical transition away from purely Makefile based installers. It will continue to live on CPAN and will continue to support the thousands of distributions that rely on it. If a motivated maintainer came along, it might even start to innovate again, or pay down its technical debt.

I give it — and its creator, Ken Williams — my respect for what it accomplished, even while I bid it farewell from the core.

This entry was posted in p5p, perl programming, toolchain and tagged , , , . Bookmark the permalink. Both comments and trackbacks are currently closed.

27 Comments

  1. Jay
    Posted May 29, 2013 at 12:05 pm | Permalink

    So when do we deprecate EUMM?

    • Posted May 29, 2013 at 1:06 pm | Permalink

      Not until a replacement is written, as it is used to build modules in core.

      • Posted May 30, 2013 at 10:37 pm | Permalink

        "Not until a replacement is written"?? You seriously think a replacement has not been written? This entire conversation is about its replacement. Perhaps switching over to it would be work, but a replacement has been written, and it was deemed unwanted.

        • Posted May 31, 2013 at 1:43 pm | Permalink

          > Perhaps switching over to it would be work, but a replacement has been written, and it was deemed unwanted.

          Perhaps it would be useful to examine the reasons why it was determined to not be a good solution, then?

          • Posted May 31, 2013 at 1:59 pm | Permalink

            No one volunteered to do the work to convert the core build process to use Module::Build instead of ExtUtils::MakeMaker. As a result, no one ever found (and then addressed) the challenges in doing so.

            The lack of a real dependency tree concerns me -- I wouldn't want lots of things unnecessarily rebuilt/recompiled on every "make".

            MakeMaker also has some specialized routines for building when PERL_CORE is set that I don't believe Module::Build ever replicated. (E.g. things don't get built to blib.)

            It's not impossible, but it's work. And no one ever stepped up, just like no one stepped up to continue developing Module::Build in any serious way after I stepped down.

            • Ken Williams
              Posted July 18, 2013 at 11:56 pm | Permalink

              I'm doing some M::B work today and I saw this comment entry for the first time. A couple of things I wanted to respond to.

              > No one volunteered to do the work to convert the core build process to use Module::Build instead of ExtUtils::MakeMaker. As a result, no one ever found (and then addressed) the challenges in doing so.

              I'm sure you're right. But my sense is that most of the work would be just taking *out* the assumption the core makes that everything is built using EU::MM. That work will need to be done no matter what replaces EU::MM.

              > The lack of a real dependency tree concerns me -- I wouldn't want lots of things unnecessarily rebuilt/recompiled on every "make".

              Have you ever seen that happen in M::B? If so, can you post a bug report?

              > MakeMaker also has some specialized routines for building when PERL_CORE is set that I don't believe Module::Build ever replicated. (E.g. things don't get built to blib.)

              Can you do `git grep PERL_CORE` in the M::B tree and post a bug report about what you think is missing? In particular, 'blib' is a settable property in M::B.

              • Posted July 19, 2013 at 10:51 am | Permalink

                Regarding dependency tree:

                (1) Additional C support files are always compiled

                (2) There's no way to specify a dependency on a .h file so that changes to the .h trigger recompilation

                Essentially, a Makefile is declarative. Module::Build is imperative. It tries to figure out dependencies, but its model is incomplete, which is why I said "not impossible, but requires work".

                • Posted July 19, 2013 at 11:58 am | Permalink

                  > (1) Additional C support files are always compiled

                  No, in compile_c() it checks the object file against the code file like so:


                  return $obj_file if $self->up_to_date($file, $obj_file);

                  I write complicated Makefiles quite a lot in my day job, not to manage code builds but to manage data flow dependencies for large experiments. I like the model. But 90%-ish of CPAN, and the perl core, doesn't really need much dependency management at all, and/or doesn't contain any C code. I don't think this is much of a reason for M::B not getting traction in core.

  2. Posted May 29, 2013 at 6:43 pm | Permalink

    "Miyagawa reminded me on IRC that the major CPAN clients will install from CPAN automatically if they see a deprecated module in a prereq list," Unless, of course, they didn't put the module in the prereq list, because they figured there was no point because it was in core.

    There's a bunch of things that give me a Bad Feeling about this stuff. The perl culture is starting to look flaky to me: bold new directions are announced and then abandoned a couple of years later. We're told how horrible EUMM is and now the anointed successor is being downplayed, and if there's a recommendation for yet-another-successor I've missed it (Module::Build::Tiny, I presume, but no one's quite willing to say so yet).

    • Posted May 29, 2013 at 7:49 pm | Permalink

      > Unless, of course, they didn't put the module in the prereq list, because they figured there was no point because it was in core.

      I don't see how that can be viewed as anything but an author error. It should not be presumed that something will be in core forever (and indeed something that is in core now wasn't always in core - e.g. 'parent' was added in 5.10.1, so it would equally be a mistake to leave it out of the prereqs list, or 5.8.x installs would break).

      Also, this is something the wondeful cpantesters network would catch, likely in the first dev (5.19.x, 5.21.x etc) release where the module was removed from core. Almost all errors caused by changes to core are caught before the corresponding stable version is released.

      > The perl culture is starting to look flaky to me: bold new directions are announced and then abandoned a couple of years later.

      One man's flakiness is another man's agility, I suppose; and you may be mistaking one person's declaration with an official announcement from the p5porters.

      We haven't figured out what the canonical successor should be. It might be MBT (or something MBT develops into), or it may be something new that doesn't exist yet. But I'm excited to find out!

      • Jay
        Posted May 30, 2013 at 3:23 am | Permalink

        > I don't see how that can be viewed as anything but an author error. It should not be presumed that something will be in core forever (and indeed something that is in core now wasn't always in core - e.g. 'parent' was added in 5.10.1, so it would equally be a mistake to leave it out of the prereqs list, or 5.8.x installs would break).

        Oh yeah, introduce breaking change, modules break (oh its author error). Also it was until recently a quite reasonable expectation for core to be append only, with even Switch hanging around until 5.14 (5.13.1)

        > Also, this is something the wondeful cpantesters network would catch, likely in the first dev (5.19.x, 5.21.x etc) release where the module was removed from core. Almost all errors caused by changes to core are caught before the corresponding stable version is released.

        Yeah I definitely saw that with modules fixing the now removed bare qx() syntax, and problems with hash randomization changes, before 5.18 was released (but, really, the opposite of that).

        • Posted May 30, 2013 at 6:42 am | Permalink

          Also it was until recently a quite reasonable expectation for core to be append only

          The deprecation of Switch was announce in Perl 5.10.1 in 2009. More modules were announced in Perl 5.12.0 in 2010. I don't think of that as "until recently".

          Yeah I definitely saw that with modules fixing the now removed bare qx() syntax, and problems with hash randomization changes

          CPAN Testers can find the breakage, but it's still up to authors to do the work to fix it. For hash randomization, there was an extensive p5p effort to file bug reports for all modules that failed tests.

          Don't blame p5p for lazy authors.

          • Posted May 31, 2013 at 12:28 am | Permalink

            Don't blame p5p for lazy authors.

            It's not helpful to invoke laziness here. Are p5p-ers "lazy" because they don't want to maintain as many core modules? The argument for removal basically boils down to "hey, I bet we could remove this thing without breaking too much stuff now", right? I don't fault them for it, and I'm not saying it's wrong, but it is definitely lazy.

            Authors do have a reasonable expectation that if they upload something to CPAN that works now, it will continue to work in the future. Perl has historically been very good about that, better than most things. So I don't fault authors for that either.

            By contrast, the qw() and hash randomization issues were really things that needed to be fixed, and the resultant breakage was regrettable but unavoidable. So the context is quite different.

            • Posted May 31, 2013 at 1:26 pm | Permalink

              My "lazy" comment was in regards to authors who get a bug report from p5p that their module is going to break on the next stable release of perl and then do nothing about it.

    • Posted May 29, 2013 at 8:33 pm | Permalink

      Joseph's concern is largely unfounded. Even if Module::Build isn't in the configure_requires prerequisite list, CPAN.pm (and possibly other clients, I'm not sure) will add it automatically if a Build.PL exists and no Makefile.PL exists. (Once added automatically, the logic about installing deprecated modules from CPAN kicks in.)

      Further, since mid-2009 Module::Build automatically adds a configure_requires prerequisite unless the author explicitly disables it.

      Any Module::Build-based module shipped in the last 4 years will have a configure_requires. Anything since 2007 might have configure_requires anyway if the author cared about making it work on Perl v5.6 or v5.8 (with an upgraded CPAN client that honored configure_requires). And anything that doesn't have it at all will be assumed to need it anyway.

      As for bold new directions announced and abandoned -- the Module::Build effort started in 2001. It's 12 years later! That's a reasonable amount of time to judge adoption and future potential.

    • Leon Timmermans
      Posted May 30, 2013 at 8:58 am | Permalink

      > We're told how horrible EUMM is and now the anointed successor is being downplayed, and if there's a recommendation for yet-another-successor I've missed it (Module::Build::Tiny, I presume, but no one's quite willing to say so yet).

      MBT is not the successor. It's the proof that writing a successor doesn't have to be all that complicated. There can be many successors, because writing one no longer requires black magic.

  3. Leon Timmermans
    Posted May 30, 2013 at 8:48 am | Permalink

    As current release manager of Module::Build, I can agree with just about everything David said here.

    • Joel Berger
      Posted May 30, 2013 at 11:50 am | Permalink

      Having worked with MB extensively, I agree too. It was great, now lets build something greater! Thanks to configure_requires and the Build.PL spec we can, with no need for blessing from p5p (meaning, being placed into the core).

  4. Posted May 31, 2013 at 12:52 am | Permalink

    Thanks, David, for the kind words about the project.

    I'm of a split mind about de-coring. On the one hand, I do think it's going to cause a shitload of innocent people a shitload of grief. I have a sense, perhaps unfounded, that configure_requires, while helpful, isn't the panacea it's been made out to be in this/p5p discussion.

    On the other hand, coring has indeed been a barrier, both logistical & psychological, to M::B's development, and as @miyagawa pointed out, there's a lot more freedom when not in core.

    Anyway, one way or another, M::B does still have a lot of life left in it, IMO. So despite "paying respect" in the title of this post, I think it's too early for a funeral.

  5. Posted May 31, 2013 at 1:35 pm | Permalink

    I do think it's going to cause a shitload of innocent people a shitload of grief. I have a sense, perhaps unfounded, that configure_requires, while helpful, isn't the panacea it's been made out to be in this/p5p discussion.

    I keep hearing FUD, but no one has demonstrated an actual problem.

    I have actually gone back to 5.8.9 -- without Module::Build -- installed the latest CPAN.pm, and then tried to install a module that uses Build.PL without configure_requires. I actually found a bug in CPAN.pm and then fixed it, which will be in 5.20 long before when it's relevant for 5.22.

    Instead of speculating, I'm the one actually going and seeing what happens.

    So, please, either go find evidence of problems — so we can fix it — or stop encouraging FUD.

  6. Posted July 19, 2013 at 12:18 am | Permalink

    I forgot to mention - Randy Sims did a ton of work on M::B too, but he's not reported on https://github.com/Perl-Toolchain-Gang/Module-Build/graphs/contributors , probably because he doesn't have a GitHub account as far as I can tell.

    • Posted July 19, 2013 at 10:36 am | Permalink

      Stupid Github. Nothing I can do about that, unfortunately.

      • Posted July 19, 2013 at 11:52 am | Permalink

        Right, just wanted to make sure his contributions weren't overlooked for people looking over this post.

  7. Steffen Winkler
    Posted July 20, 2013 at 1:20 pm | Permalink

    Module::Build is able to create ppm and ppd files. Is there an alternative to create them with a newer build tool?

    • Posted July 20, 2013 at 3:39 pm | Permalink

      Not that I'm aware of.

    • Leon Timmermans
      Posted July 21, 2013 at 9:29 am | Permalink

      Such a tool shouldn't be hard to create (you could easily extract the logic from Module::Build), but it really shouldn't be part of the build tool. Just as is currently already done for debs and rpms.

      So well volunteered ;-)

    • Posted July 25, 2013 at 4:57 am | Permalink

      In theory, PPM::Make should be able to create them. I never used M::B for this.

© 2009-2014 David Golden All Rights Reserved