dagolden | Whatever comes to mind

Updating minicpan alphas more frequently

By david | Published: July 1, 2009

At the Perl QA hackathon, Andreas Koenig, the maintainer of PAUSE, demonstrated a way to decrease the lag between new distribution uploads to PAUSE and syndication to CPAN mirrors. (See File::Rsync::Mirror::Recent on CPAN for details). Running minicpan against a one of the new, fast updating mirrors results a very fresh minicpan, but, until now, only for regular releases, not for alpha development releases.

Last year, I wrote CPAN::Mini::Devel to include development releases in a minicpan, but it depends on an obscure, large, infrequently updated index.¹ Now, with smaller, fast-updating index files available for rsync, I released CPAN::Mini::Devel::Recent to take advantage of them for minicpan as well.

This is all still experimental, but if you do Perl programming and want bleeding-edge development releases within minutes of their upload to PAUSE, then find a fast mirror, try out CPAN::Mini::Devel::Recent and let me know how it works for you. After installation, you can configure it to be used automatically by adding “class: CPAN::Mini::Devel::Recent” to your .minicpanrc file.

indices/find-ls.gz [↩]

Posted in cpan, perl | Leave a comment

Metabase: Opinions from anyone about anything

By david | Published: June 25, 2009

Explaining Metabase

At YAPC::NA, Ricardo and I finally released some early alpha Metabase modules to CPAN. Metabase is a project we started in 2008 at the Oslo QA Hackathon to provide a new transport and storage infrastructure for CPAN Testers and other CPAN distribution metadata. We’ve been working out of github until now, but think it’s time to encourage others in the Perl programming community to take a look.

One of the challenges we’ve had is just explaining Metabase to people. Here’s what I’ve come up with: Metabase is a database framework and web API to store and search opinions from anyone about anything. In the terminology of Metabase, Users store Facts about Resources. Facts are really opinions or claims; there is no inherent truth in them. Resources are logical identifiers, generally in the form of a URI, since the things we’re most interested in describing are online.

Since CPAN Testers is our first use case, I’ll use that to illustrate the concept. Currently, CPAN Testers email test reports about distributions on CPAN. In the Metabase world, each CPAN tester is a User. The Resource is a CPAN distribution file – a ‘distfile’ URI that describes a distribution tarball on CPAN such as cpan://distfile/DAGOLDEN/Capture-Tiny-0.06.tar.gz. The Fact is the test report. Today that’s just the text of the email message, but in the future it will be structured data. ¹

It’s easy to see how this can be generalized to an entire ecosystem of facts and opinions about CPAN distributions, such as CPANTS, ratings, annotations, tags, etc. Today, every CPAN website has its own database and its own API so mashups and remixes require building an interface to each. Metabase offers the possibility of a unified way for all these tools to store, share and cross-reference information about CPAN.

Design considerations

Metabase has some similarities to other, general ontology frameworks, but we decided to focus our efforts around some pragmatic tradeoffs using CPAN Testers to guide the design. We think these decisions generalize well to other uses, but we’re trying to follow a YAGNI strategy and only implement what CPAN Testers will need before worrying about more general cases.

(1) High fact volume

There are currently about 18,000 “latest” distributions on CPAN, and the entire history of CPAN is a bit over 100,000 distributions. By contrast, there are over 4 million CPAN Testers reports already and the total is growing by about 250,000 per month. Reports are usually consumed as a batch process to create summary statistics on the CPAN Testers website and relatively few reports are ever reviewed in detail. Therefore, the Metabase design to date prioritizes Fact submission over search and retrieval.

(2) Minimalist Perl for client libraries

Many CPAN Testers want to run smoke tests using a Perl installation that has as few extra modules installed as possible. Therefore, any Metabase components used by clients such as Metabase::Fact or Metabase::Client::Simple need minimal non-core dependencies. On the other hand, anything that will run on the server side can take full advantage of Modern Perl tools like Moose, DBIC and Catalyst.

(3) No barriers to contribution

Today, new CPAN Testers don’t have to register or get authorized to start contributing, they just start sending emails to a mailing list that collects reports. For Metabase, users generate a user profile fact and submit it as a credential with their facts. Metabase manages user identities within the Metabase itself – adding new users automatically – allowing new contributors to join without any external pre-authorization hurdles that might discourage participation.²

(4) Open to extension and evolution

We want to make CPAN Testers tools available for corporations or perl5-porters or anyone with customized testing needs. We want people to come up with new facts and new resources. That means defining interfaces well and leaving implementation details open to change. Metabase allows lots of choices about things like underlying data storage technologies or permissible fact types.

Metabase specifies data storage capabilities, but the actual database storage is pluggable, from flat files to relational databases to cloud services. Metabase defines how Fact classes can provide validation and custom index data on top of a very simple data model. Moreover, Metabase is a fairly dumb repository; any intelligent analysis must be done by third parties extracting and transforming data.

Where do we go from here?

I’ve successfully submitted test reports to a local Metabase, but for Metabase to become what we need for CPAN Testers (and more), there are still quite a number of things to do:

Improve documentation
Test and refine Metabase components
Design and implement search capabilities
Establish one or more ‘development server’ Metabases for testing robustness and throughput
Migrate the 4 million CPAN Testers reports from NNTP to Metabase
Migrate CPAN Testers downstream analytics to use Metabase as a source
Have some CPAN Testers pilot submitting reports to Metabase
Develop and release new CPAN Testers clients that can submit structured Metabase reports

If you are interested in following or participating in Metabase, there is a (still low-volume) mailing list you can subscribe to and several of the key developers can be found on #toolchain on irc.perl.org.

Technically, a CPAN Testers report will be a collection of more granular facts like prerequisites, test output, environment and so on. [↩]
Verification or authorization is optional [↩]

Posted in cpan, cpan-testers, metabase, perl | 2 Comments

Module::Build bug triage help requested

By david | Published: June 19, 2009

In an attempt to fix up Perl’s Module::Build in preparation for a new 0.34 release, I’ve been hacking away at the bug queue, but there are so many stale bug reports that it’s hard to figure out what really needs work.

Here’s how you can help:

Go to the Module Build RT queue
Click on the “Severity” header to sort by severity
Pick ONE bug of Normal severity

If you don’t see one you like, pick one randomly. Find a bug number with a final digit that matches the last digit of your birth year. Or roll a d10. Whatever. Just pick ONE.

Figure out if the bug still exists based on the code in the repository trunk
Update the ticket with your findings

In particular, if you think the severity can be lowered, please indicate that. I would generally say that “Normal” bugs would be functional, whereas documentation gaps could be marked down to “Unimportant” and new feature requests should be “Wishlist”. If a ticket is waiting for feedback from the reporter or waiting for some other module to be fixed, please indicate it needs to be marked “stalled”.

If you’re feeling particularly inspired, write a patch. Or even just write a test to confirm the bug. Anything helps.

Thanks very much!

Posted in cpan, perl | Leave a comment

Team Maintenance for the Win

By david | Published: June 15, 2009

I released Module::Build 0.33_01 this weekend. I had co-maintainer rights so I could help manage the RT queue, but had never before released a dev version of Module::Build, so I didn’t exactly know what was involved before I started. Here’s how it went:

I read the release documentation in the devtools/ directory of the repository
I updated the documentation slightly based on an email conversation I had with Eric Wilhelm, who wrote the release tool
I installed the release tool from Eric’s repository
I ran the release tool to build the tarball
I uploaded the tarball to PAUSE
I updated the documentation again to reflect a trivial error and workaround
I ran the “bump version” tool to prepare the repo for new commits

The elapsed time was maybe an hour. This far exceeded my expectations. It was downright easy.

To be clear in case anyone missed it: I became the release manager for a CPAN distribution I co-maintain for the first time, without hassle, in about an hour.

Making a CPAN project ready for team maintenance is not hard. It’s just takes applying the three great virtues of a programmer (laziness, impatience and hubris) to the project.

Document your release process in the repository
Automate your release process with easily available tools
When things change, document the change

Posted in perl | 1 Comment

Stop with the Perl golf already!

By david | Published: June 8, 2009

It’s annoying to read critics complaining that “Perl looks like line noise” since that’s definitely not my experience… at least, most of the time.

There are some places where I think Perl actually does look a bit “noisy”:

sigils ($@%&), but only when most variables have single-letter names ($i,$j,$k, etc.)
punctuation-only special variables ($/, $], etc.), but I find these are used infrequently in most “real” code I write or see
explicit dereferencing (@{$foo->{bar}})
regular expressions, but this is hardly Perl’s fault

But when I think about where the reputation for line noise comes from, the thing that jumps out at me is the Perl (sub)culture that turned code golf and obfuscation into a source of pride — a demonstration of high art and coding wizardry.

Think about it. IBM has a technical paper on JAPH’s and describes them as “a mainstay of the Perl culture”. Obfuscated perl shows up on tee-shirts. Perl Monks has an entire section on Obfuscated Perl. The sigils themselves even became the logo on the official shirt for YAPC::NA 2008 as a visual pun on the Chicago city flag.

So, of course, people are going to equate Perl with line noise.

And who am I to be throwing stones? I’ve done it too.

I recently gave an answer on StackOverflow to the question, “What’s the shortest perl program you can write for the number guessing game?” That question and the subsequent answers were the genesis of this article.

I had the idea to show how the guessing game could be played recursively to golf by avoiding variables to hold guesses. Just for the sake of argument, if I were writing it out for someone to understand it, here’s how I might do it:

use strict;
use warnings;
my $target = 1 + int( rand(10) );
my @message = ('', "too much\n", "too few\n");

sub guess {
  print "Gimme your guess 1-10:\n";
  my $guess = <STDIN>;
  my $comparision = $guess <=> $target;
  if ( $comparison ) {
    print $message[$comparison];
    return guess();
  }
  else {
    return $target;
  }
}

my $answer = guess();
print "Bingo! It was $answer\n";

And here’s the short version I came up with:

($t,@m)=(1+int rand 10,'',"too much\n","too few\n");sub g{print
"Gimme your guess 1-10:\n";syswrite(STDOUT,$m[<><=>$t])?g():$t;}
print"Bingo! It was @{[g()]}\n"

That’s not even that bad as obfuscation goes, but I looked to the two really nasty bits ( “<><=>$t” and “@{[g()]}”) and though to myself, “Damn! No wonder people complain about line noise!”

It reminds me of a quote about discrmination by the US chief justice. Adapted to this context, it means the way to stop people thinking that Perl looks like line noise is to stop Perl from looking like line noise.

That’s my new pledge. No more Perl golf and no more JAPHs for me. The Perl I’ll take pride in is Perl that people can read.

Posted in perl | 17 Comments

CPAN Testers supports Perl programming

By david | Published: May 31, 2009

Some recent articles point out some methodology flaws in the TIOBE index of programming languages and suggest the Perl community act to promote Perl:

I just want to point out a couple ways that the CPAN Testers project is doing its part to support Perl programming.

First, the new CPAN Testers website design template includes the phrase “CPAN Testers, supporting the Perl programming language since 1999″ — which means it’s now on 50k+ pages, one for every CPAN distribution/version ever tested.

Second, I just released Test::Reporter 1.54, which adds the magic “perl programming” phrase to every new CPAN Testers report submitted. If only we had done that on day 1, there would now be over 3.8 million searchable examples on various NNTP mirrors.

Is this silly? Time will tell…

Posted in cpan-testers, perl | Leave a comment

Perl version number puzzles

By david | Published: May 26, 2009

I’m working on a longer article about version numbers in Perl, why there are now so many ways to screw them up, and some recommendations. In the meantime, I realized that some of my examples for that article might be a fun puzzle for others with some curiosity and time to kill.

Format

Each puzzle consists of the output of several Perl one-liners using different versions of Perl. They are run in a directory with a single file, Foo.pm, like this:

package Foo;
use strict;
use warnings;
our $VERSION = 1.23;
1;

In each puzzle, line 4 (the $VERSION line) is different and the puzzle is to figure out the $VERSION line given only the sample output.

Puzzle 1

$ perl5.10.0 -e 'use Foo 0.001001'

$ perl5.8.9 -e 'use Foo 0.001001'
Foo version 0.001001 required--this is only version 0.001001 at -e line 1.
BEGIN failed--compilation aborted at -e line 1.

$ perl5.8.9 -Mversion -e 'use Foo 0.001001'

Puzzle 2

N.B. This puzzle intentionally requests a higher version number than is in Foo.

$ perl5.10.0 -e 'use Foo v0.1_1'
Foo version v0.1_1 required--this is only version v0.0_1 at -e line 1.
BEGIN failed--compilation aborted at -e line 1.

$ perl5.8.0 -e 'use Foo v0.1_1'
Foo version v0.1.1 required--this is only version v0.0.1  at -e line 1.
BEGIN failed--compilation aborted at -e line 1.

$ perl5.6.2 -e 'use Foo v0.1_1'
Foo version 0.011 required--this is only version 0  at -e line 1.
BEGIN failed--compilation aborted at -e line 1.

Puzzle 3

$ perl5.10.0 -e 'use Foo 0.000001'
Foo version 0.000001 required--this is only version v0.0_1 at -e line 1.
BEGIN failed--compilation aborted at -e line 1.

$ perl5.10.0 -e 'use Foo v0.0_1'

$ perl5.8.9 -e 'use Foo v0.0_1'

$ perl5.8.0 -e 'use Foo v0.0_1'
Foo version v0.0.1 required--this is only version v0.0_1  at -e line 1.
BEGIN failed--compilation aborted at -e line 1.

$ perl5.6.2 -e 'use Foo v0.0_1'
Foo version 0.001 required--this is only version v0.0_1  at -e line 1.
BEGIN failed--compilation aborted at -e line 1.

Puzzle 4

$ perl5.10.0 -e 'use Foo v0.1.1'

$ perl5.8.9 -e 'use Foo v0.1.1'

$ perl5.10.0 -e 'use Foo v0.2.1'
Foo version v0.2.1 required--this is only version v0.1.1 at -e line 1.
BEGIN failed--compilation aborted at -e line 1.

$ perl5.8.9 -e 'use Foo v0.2.1'
Foo version 0.002001 required--this is only version 0.001001 at -e line 1.
BEGIN failed--compilation aborted at -e line 1.

Posted in perl | Leave a comment

Time for second edition Perl Best Practices?

By david | Published: May 21, 2009

I just noticed that Perl Best Practices is copyright 2005. When it came out, I thought it showed a professional side to Perl programming, a side that is more than just line noise and slapped-together glue scripts.

I remember someone’s review more or less said that it was something that you could show your boss or show your team and use to cut through the TIMTOWTDI excuse for people’s bad code.

I didn’t agree with everything, and I wasn’t alone. There are critiques scattered across various reviews, blogs and posts. For example, here were a couple that I dug up:

But by provoking a debate about just what was or wasn’t a best practice, Damian helped jump-start broader attempts to advocate a more professional approach to Perl programming. (c.f. Perl::Critic, Modern Perl Books, Enlightened Perl, or Catalyzed.org)

Four years is a long time in the Perl world. Even if the PBP Table of Contents is about right, many of the ‘best practices’ themselves are probably woefully out of date. A new version would probably include features in Perl5 Version 10 like given/when, smart matching and named captures. Moose is a much better ‘best practice’ for object-oriented Perl than Class::Std.¹. Perl::Critic should be in there. And I’m sure there are plenty more.

Why do we need a second edition? Because anything new has to compete for oxygen and credibility against this one book with a great title written in 2005. And only a book — not an article on the web — can have the same “smack it down on the desk to end the argument” impact that PBP had when it came out.

Perl Best Practices have changed. It’s time for the second edition.

O’Reilly, are you listening?

And inside-out objects are better done with Object::InsideOut or Class::InsideOut anyway [↩]

Posted in perl | 11 Comments

May 2009 Perl Seminar NY talks online

By david | Published: May 21, 2009

I’m coming up on my Iron Man deadline, so this short post will have to do while I work on two or three drafts of longer articles.

On Tuesday, I gave two talks at Perl Seminar NY — which, for TIOBE’s sake, is about Perl programming. This was the last Perl Seminar for the 2008-2009 season and meetings will resume in the fall.

The first talk was about the Perl QA hackathon in Birmingham, which I blogged about earlier. The talk focuses less on my own work and a bit more on describing the major projects at the hackathon, some of my thoughts about how to get the most out of attending a hackathon and some thoughts for organizers on throwing a good hackathon. My slides can be downloaded as a pdf: Quality Hacking in Birmingham.

My second talk was about how to capture output in Perl. It introduces a new module I wrote, Capture::Tiny, that replaces a dozen or more different tools to capture output (many of which are limited or broken). I plan to blog more about that, but in the meantime, slides are here: How Not to Capture Output in Perl

Posted in perl | 2 Comments

Kitty Hawk moment for CPAN Testers 2.0

By david | Published: May 13, 2009

It flies!

Not far, not well, and it isn’t ready for passengers, but over the weekend I was able to get CPAN Testers 2.0 off the ground for a few brief moments.

When I wrote about the Perl QA hackathon, I mentioned that I had been able to take saved CPAN Testers reports and inject them into a locally running Metabase. To (ab)use my Wright Brothers analogy, if that was ‘gliding’ then this last weekend was the first test of ‘powered flight’ as I got CPAN.pm and a modified CPAN::Reporter to successfully send test reports into the local Metabase in the normal course of installing new modules on my development computer.

User profiles

One big change from the earlier attempt is the use of a persistent user profile. Previously, Test::Reporter::Transport::Metabase generated a dummy Metabase profile on the fly from the tester’s email address. Now, it loads a persistent user profile (a Metabase::User::Profile object from the Metabase-Fact distribution). For my experiment, this was specified in the transport option in the CPAN::Reporter config file:¹

transport = Metabase uri http://localhost:3000/ profile
/home/david/.cpanreporter/profile.json

I also made it much easier to generate profiles, by adding a small profile-generating program to the Metabase-Fact distribution:

$ metabase-profile -o /home/david/.cpanreporter/profile.json
Enter full name: David
Enter email address: david@example.com
Enter password/secret: afasdfasdfa
Writing profile to '/home/david/.cpanreporter/profile.json'

What’s next?

There is still no public Metabase for CPAN Testers, so at this point it remains an interesting experimental flight to prove the concept. However, I plan to clean up the code a bit and start releasing development versions to CPAN so that the existence of a Metabase-ready CPAN Testers client ratchets up the pressure to get the server side moving forward.

In the meantime, I’m starting to take some of the lessons learned and apply them to work on CPAN::Testers::Client, which will be the unified CPAN Testers 2.0 client to replace CPAN::Reporter and CPANPLUS::Internals::Report.

Related git repositories

All of the code for the test flight is unreleased. However, it is available in several git repositories for those curious enough or foolhardy enough to attempt to repeat this feat.²

The ‘transport’ option needs to be on one line, but my WP theme seems to have a problem wrapping long lines, darn it! [↩]
In metabase-web/example/cpan-testers, run the quickstart.pl program to start up a local metabase in /tmp. Then install Test::Reporter::Transport::Metabase (and prerequisites) and the dev version of CPAN::Reporter. Generate and configure a profile as described above, and then good luck! [↩]