How to add 'provides' metadata via Makefile.PL

My last post about PAUSE permission problems suggested to manually add a 'provides' field to your metadata files if PAUSE can't determine what packages are in your distribution. I realized that people might not know how to do that, so this is a quick tutorial.

One reason PAUSE might not be able to find your package names is if you generate your .pm files for some reason. I'm going to use a super-simplified example distribution to show what to do.

Consider a distribution for a hypothetical "Acme::Provides" with these four files:

  • Makefile.PL — our distribution build tool
  • Provides.pm.PL — our module generator
  • t/00-load.t — a test that the built module can be loaded
  • MANIFEST — a listing of these four files

The Provides.pm.PL generates Acme/Provides.pm directly into the blib directory when make runs, so there is no .pm file hanging out in lib for PAUSE to examine.

When we run the Makefile.PL we get a Makefile that will generate the .pm file we need. We can see that it does so by running make and make test:

$ perl Makefile.PL
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Acme::Provides
Writing MYMETA.yml and MYMETA.json

$ make
"/Users/david/.plenv/versions/20.2t/bin/perl5.20.2" "-Iblib/arch" "-Iblib/lib" Provides.pm.PL blib/lib/Acme/Provides.pm

$ make test
PERL_DL_NONLAZY=1 "/Users/david/.plenv/versions/20.2t/bin/perl5.20.2" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-load.t .. ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( 0.02 usr  0.01 sys +  0.01 cusr  0.00 csys =  0.04 CPU)
Result: PASS

We can build the distribution directory with make distdir:

$ make distdir
rm -rf Acme-Provides-0.01
"/Users/david/.plenv/versions/20.2t/bin/perl5.20.2" "-MExtUtils::Manifest=manicopy,maniread" \
                -e "manicopy(maniread(),'Acme-Provides-0.01', 'best');"
mkdir Acme-Provides-0.01
mkdir Acme-Provides-0.01/t
Generating META.yml
Generating META.json

Here is the generated Acme-Provides-0.01/META.json file (omitting 'no_index' and 'prereqs' fields for brevity):

{
   "abstract" : "Demonstration of adding provides metadata",
   "author" : [
      "David Golden <dagolden@cpan.org>"
   ],
   "dynamic_config" : 1,
   "generated_by" : "ExtUtils::MakeMaker version 7.04, CPAN::Meta::Converter version 2.150001",
   "license" : [
      "artistic_1"
   ],
   "meta-spec" : {
      "url" : "http://search.cpan.org/perldoc?CPAN::Meta::Spec",
      "version" : "2"
   },
   "name" : "Acme-Provides",
   "release_status" : "stable",
   "version" : "0.01"
}

That doesn't tell PAUSE about our generated module, so if we uploaded this distribution, it wouldn't get indexed because we wouldn't be claiming the package name "Acme::Provides" to match the distribution tarball name "Acme-Provides-0.01.tar.gz".

However, we can use the META_ADD directive in the the Makefile.PL to add that information ourselves:

    META_ADD => {
        provides => {
            'Acme::Provides' => {
                file => 'Provides.pm.PL',
                version => '0.01',
            },
        },
    },

Check out the revised file here: Makefile.PL.

Now, if we re-run Makefile.PL and regenerate the distribution directory with make distdir, we can see our 'provides' data added to the generated Acme-Provides-0.01/META.json file (again omitting 'no_index' and 'prereqs' fields for brevity):

{
   "abstract" : "Demonstration of adding provides metadata",
   "author" : [
      "David Golden <dagolden@cpan.org>"
   ],
   "dynamic_config" : 1,
   "generated_by" : "ExtUtils::MakeMaker version 7.04, CPAN::Meta::Converter version 2.150001",
   "license" : [
      "artistic_1"
   ],
   "meta-spec" : {
      "url" : "http://search.cpan.org/perldoc?CPAN::Meta::Spec",
      "version" : "2"
   },
   "name" : "Acme-Provides",
   "provides" : {
      "Acme::Provides" : {
         "file" : "Provides.pm.PL",
         "version" : "0.01"
      }
   },
   "release_status" : "stable",
   "version" : "0.01"
}

Now, if we upload this distribution, PAUSE will see us claiming "Acme::Provides" and all should be well.

Posted in cpan | Tagged , , , | Comments closed

What to do if PAUSE tells you this distribution name can only be used by users with permission for X, which you do not have

Over the last year, a handful of CPAN authors have been bitten by PAUSE complaining that they don't have permissions for a distribution name they've uploaded.

What's going on? (short explanation)

PAUSE used to have a gaping security hole; it's now closed. As a result, when an author uploads a distribution with a name like Foo-Bar-Baz-1.23.tar.gz, the author must have primary or co-maintainer permissions on the package name matching the distribution (Foo::Bar::Baz, in this case) or else the distribution will not be indexed. It's still on CPAN, but won't be added to the index that allows people to easily install it.

How to fix it

If you are uploading Foo-Bar-Baz-1.23.tar.gz, make sure you have a "lib/Foo/Bar/Baz.pm" file containing a "package Foo::Bar::Baz" statement.

If you use any sort of clever syntax mangler like Moops that doesn't use "package" statements, be sure your generated META.json or META.yml file includes a "provides" field claiming the package name matching the distribution name. If you don't understand what that means or how to make it happen, you shouldn't be using Moops or anything like it until you do. (Update: I posted an example in ""How to add 'provides' metadata via Makefile.PL".)

What's going on? (long explanation)

Many CPAN ecosystem tools (like rt.cpan.org) treat a distribution (i.e. tarball) name as a significant entity for permissions, etc. But historically, nothing required distribution names to have anything to do with the modules they contained. This led to an interesting security hole: by uploading a distribution matching an existing distribution on CPAN, but with entirely new, unrelated modules, PAUSE would index the modules and associate them with the distribution. The author of said distribution would then be treated as a fully-authorized administrator over the shared distribution name.

Example: Let's say I wanted to hijack the Moose RT queue. I could have uploaded Moose-666.tar.gz containing lib/Not/Really/Moose.pm with "package Not::Really::Moose" and a $VERSION of 666. That would create an index entry like this:

Not::Really::Moose        666      DAGOLDEN/Moose-666.tar.gz

'lo and behold, because I had an indexed distribution "DAGOLDEN/Moose-666.tar.gz", I would become an administrator of the Moose RT queue. And MetaCPAN would think that "666" was the latest release of Moose.

To fix this, PAUSE now ties distribution names to the package namespace permissions system. While I can still upload Moose-666.tar.gz, because I don't have permissions over the "Moose" package name, my bogus distribution would not be indexed. Without being indexed, the ecosystem doesn't use it to give me any permissions.

A small handful of distributions were grandfathered (e.g. libwww-perl) and don't have to follow this rule, but all new distributions do.

Unfortunately, PAUSE's upload reporting has some bugs and other distribution problems can wind up incorrectly reported as a permissions problem. These are actually pretty rare. Still, I hope to work with Andreas at the QA hackathon to fix the upload reporting.

But if you get this error message, it's 90% or more likely that you've got one of these problems:

  1. You don't have a module "Foo::Bar::Baz" in a distribution called Foo-Bar-Baz-$VERSION.tar.gz; fix it by adding that module
  2. You think you have a "Foo::Bar::Baz" module, but PAUSE can't find it or understand your package declaration; fix your package declaration or use a 'provides' field in META.json to be explicit
  3. You have a "Foo::Bar::Baz" module, PAUSE can find it, but for some weird, historical reason someone *else* actually owns that namespace and you never noticed before

If you've ruled out #1 and #2 yourself, please feel free to contact modules@perl.org for help, but be patient, as it make take a while for an admin to see your email and investigate.

I hope this explanation helps anyone mystified by this error message.

Posted in cpan, perl programming, toolchain | Tagged , , , | Comments closed

Thoughts on getting Perl 6 for Christmas

As rumored, at FOSDEM this year, Larry Wall announced that Perl 6 will be available this Christmas[1]. I followed the blog coverage (e.g. "Fosdem 2015: It's Christmas!"), listened to Miyagawa's podcast with Larry and was inspired to read the (unfortunately quite dated) guide "Perl 5 to Perl 6".

My last blog post about Perl 6 was in 2013 – "Is Perl 6 pointless, hopeless or just not done?" (which was really about Perl 5) – and it followed my 2010 post, "Thoughts on Perl 6 hype and backlash".

Now that it's 2015 and Christmas is coming, I decided to reflect again on Perl 6.

Stuff I'm excited about

The gradual typing should be a big win for optimization and types of computation that are unavoidably slow in Perl 5. If Perl 6 can overtake Perl 5 in speed at release (or soon after), that will be a strong reason for Perl 5 people to learn Perl 6.

Subroutine signatures and a proper, built-in OO system will be a relief. Even with Perl 5 gaining experimental signature support, it's a crude patch over a gap, whereas Perl 6 has it from the start.

Lazy lists and evaluation look like a potential big win, since avoiding work is the best optimization.

The Unicode model looks to be one of the sanest and most well-integrated of any programming language I've seen.

The model for rationals (instead of floats) looks really groundbreaking, particularly if it turns out to be fast. Saving people from decimal math bugs with floating point representation seems like a win.

The "awesome" error reporting looks at least fun and likely will really help people learning the language.

Stuff that concerns me

The language is BIG. One of the appeals of Go to me is how small the language is. Perl 6 strikes me as the opposite philosophy. Look at the list of operators for an example. It will make it harder to master and puts a bigger context switching cost going between languages. I think it will also make idiomatic Perl 6 code harder to read for newbies and outsiders.

The documentation also seems to have significant gaps. E.g. at the time wrote this, the Input/Output docs page is completely empty. Plus, there have been so many changes to the project (and there are different backends with different feature support) that I'm never quite sure if any docs, example or presentation I see is actually still relevant.

One of my bigger areas of concern is the toolchain/ecosystem. I know that Perl 6 plans to allow loading different releases of a module at the same time, but this appear unimplemented still. The distribution of Perl 6 modules seems git repository based, with an extremely primitive way of specifying dependencies (lists of module names, without even version numbers).

While I've spoken about lessons of the toolchain to Perl 6 people at Perl QA hackathons in the past (and will again this year), I can't figure out if this part of Perl 6 results from benign neglect or an active decision to discard decades of lessons learned from the Perl 5 CPAN.

The concurrency model seems sensible (see "Composable Concurrency in Perl 6"), but that paper dates from 2013 and I don't know how much is still relevant. Certainly, I don't hear a lot about concurrency and I don't know if that's because it's a given and boring or because it's not actually going to turn out to be a big deal or because it's not done. Seeing "Synchronization" partially done and other concurrency ideas (like parallelized hyper operators and junction etc.) not yet implemented on the feature matrix is disconcerting.

Conclusions

It's still not clear to me what the "killer feature" is. If Perl 6 can be a much faster Perl 5 (by absolute performance or better concurrency), then I think it will be familiar enough that we'll see adoption by Perl 5 developers willing to trade the learning-curve for higher performance.

But I don't see anything that would drive adoption by other communities or new developers. The sheer size and complexity will be daunting. By contrast, Go – a small, highly-opinionated language – has become wildly popular in certain niches. It solves several serious developer pain points that appeal to both static and dynamic language enthusiasts.

I think the Perl 6 team needs to find a few key examples of features of the language that really shine compared to other languages. Not "clever features" – features that get back to the mantra of making hard things possible in areas that programmers encounter every day.

It also needs much better documentation. Someone needs to start the equivalent of "Modern Perl", with the goal of having a compact, tight reference for Perl 6 and aim to have it ready at least in draft form by Christmas.

I've added the Perl 6 weekly summary to my newsreader and look forward to seeing what happens during the rest of the year.

[1] If you haven't heard the joke, Larry was fond of saying that Perl 6 would be out by Christmas, but not saying which Christmas. Now he's actually said which Christmas.

Posted in perl programming, perl6, Uncategorized | Tagged , , | Comments closed

Sometimes, it really IS a bug in Perl

Sometimes, you find a bug so bizarre that you start to think it must be a bug in Perl itself. Nine times out of ten or more, it's not, and some more determined digging will turn up whatever you did wrong.

It's such a common tendency that I usually bias the other way – the moment I start to think Perl might have a bug, I dismiss it and look harder for the bug in my code.

But sometimes, it really IS a bug in Perl.

Here's what happened to me last week:

  • I tried installing a module from CPAN and had a test failure in the boolean.pm module dependency.
  • The error messages were coming from the latest release candidate of Test::More, which I had installed to help flush out issues
  • Downgrading to the stable Test::More made the problem go away
  • Investigating what broke in the release candidate got strange; something in an eval seemed to be having an impossible result

Graham Knop did some incredibly diligent digging and was able to produce a simplified reproduction independent of Test::More. Here's what was necessary:

  • Perl 5.20.0 or 5.20.1 with threads
  • Load boolean.pm using the '-truth' option
  • Construct an anonymous function with a closure
  • Watch literal strings ("Hello world") in the closure DISAPPEAR and become undefined

I must explain that what the '-truth' option does is incredibly unwise – perhaps even "evil". When loaded with '-truth', boolean.pm unlocks the read-only status of Perl's internal variables representing the values for true and false, replaces them with boolean.pm objects, and relocks the variable. That means that 1 == 1 returns a boolean.pm object rather than the usual "1" for truth.

Here's an example program that demonstrates the bug.

use strict;
use warnings;
use boolean -truth;

my $name = 'welp';

my $not_broken = sub {
  print "what\n";
};

my $broken = sub {
  print "what\n";
  my $str = $name;
};

print "calling not_broken:\n";
$not_broken->();
print "calling broken:\n";
$broken->();

On Perl 5.18.2-threaded and Perl 5.20.0-not-threaded the output is:

calling not_broken:
what
calling broken:
what

But on Perl 5.20.0-threaded and 5.20.1-threaded, the output is:

calling not_broken:
what
calling broken:
Use of uninitialized value0 in print at boolean-broken.pl line 12.

Note that line 12 is print "what\n" and value0 is the first argument to print. The literal string has become undefined!

So, yes, this is actually a bug in Perl. It's triggered by a very dubious feature of boolean.pm, but it doesn't occur in 5.18.2 with threads or in any 5.20 without threads, so this is very specific to 5.20.x with threads. That's a bug.

Fortunately, there's some good news:

  • Perl 5.21.5 already disallowed unlocking internal read-only variables (which broke boolean.pm tests, too)
  • boolean.pm has deprecated the '-truth' option in light of this bug and the change in 5.21.5
  • Father Chrysostomos jury-rigged the latest bleadperl to allow unlocking the read only true/false variables and the problem did not occur under a threaded build, so the the bug appears fixed even if the internal unlocking were still allowed

The moral of the story is that while you shouldn't jump to the conclusion that some strange behavior is a bug in Perl, it's worth keeping in mind that sometimes, it really is.

Posted in p5p, perl programming | Tagged , , | Comments closed

Moving CPAN RT tickets to Github, now improved

When I wrote about migrating RT tickets to Github a couple years ago, it was a quick-and-dirty job. This week, I finally cleaned it up and added some features I'd been missing:

  • Automatically closing the migrated RT ticket with a hyperlink to the new Github ticket
  • Migrating "stalled" RT tickets, as well as "new" and "open" tickets
  • Including all comments, not just the opening one, along with creator email and creation time stamp
  • Including hyperlinks to any file attachments
  • Including a list of all "requestors" email addresses (people following the ticket)
  • Improving distribution name detection for people who don't use Dist::Zilla
  • Adding a "dry-run" mode
  • Adding a "single-ticket" mode

Here's a draft of what a newly migrated ticket might look like (from a test run, not an actual migration, but it shows an attachment):

Migrated ticket example

Or, here's an actual ticket migrated over for IO::CaptureOutput: Bad joining of STDERR + STDOUT in windows [rt.cpan.org #55164].

I've created a github repository for this at zzz-rt-to-github to make it easy for people to grab the script or contribute back. The README file there has instructions.

Enjoy!

Posted in cpan, perl programming | Tagged , , , , , | Comments closed

© 2009-2015 David Golden All Rights Reserved