Setting up a Perl Development Environment with plenv

This is a slightly modified copy of something I posted internally at work about setting up a development environment using plenv, which keeps your working Perl isolated from your system Perl. Many expert Perl developers already have some variation of this, but I'm posting it as a public service for people who've wanted this but never got around to it.

(I used to use perlbrew, but switched to plenv and haven't looked back).

Step 0: Install compilation tools

Make sure you have a command-line compilation environment ready:

  • Mac: install Xcode or Xcode command line tools
  • Debian-based: sudo apt-get install build-essential
  • Fedora-based: sudo yum install make gcc

Step 1: Install plenv

(This is excerpted almost verbatim from the documentation for plenv)

This will get you going with the latest version of plenv:

1.1 Check out plenv into ~/.plenv.

$ git clone git:// ~/.plenv

1.2 Add ~/.plenv/bin to your $PATH for access to the plenv command-line utility.

$ echo 'export PATH="$HOME/.plenv/bin:$PATH"' >> ~/.bash_profile

Ubuntu note: Modify your ~/.profile instead of ~/.bash_profile.
Zsh note: Modify your ~/.zshrc file instead of ~/.bash_profile.

1.3 Add plenv init to your shell to enable shims and autocompletion.

$ echo 'eval "$(plenv init -)"' >> ~/.bash_profile

Same as in previous step, use ~/.profile on Ubuntu, ~/.zshrc for Zsh.

1.4 Restart your shell as a login shell so the path changes take effect. You can now begin using plenv.

$ exec $SHELL -l

1.5 Install perl-build, which provides a "plenv install" command that simplifies the process of installing new Perl versions.

$ git clone git:// ~/.plenv/plugins/perl-build/

1.6 Build a version of Perl. The following line builds in parallel, enables threads and disables man pages (to save space).

$ plenv install -j 9 -D usethreads -D man1dir=none -D man3dir=none 5.20.1 

1.7 Rebuild the shim executables. You should do this any time you install a new Perl executable (for example, when installing a new Perl version, or when installing a cpanm that provides a command).

$ plenv rehash

You can get help using plenv with "plenv help".

Step 2: Enable the Perl you just built

You can set your Perl in two ways:

  • local Perl – within a directory (and subdirectories); keeps things isolated but might surprise you if you're working out of multiple directories
  • global Perl – always the default Perl in your PATH; consistent everywhere, but could break code that assumes a different version of Perl or different installed dependencies

To set the Perl locally, change to the directory you want and run this command (to set the version to 5.20.1, as built, above):

$ plenv local 5.20.1

To set the Perl globally, run this command:

$ plenv global 5.20.1

Verify you have the Perl you intended:

$ perl -v

Step 3: Install the "cpanm" CPAN client

The "cpanm" command line tool is a popular, zero-config CPAN client. Plenv can install it for you into the currently running Perl:

$ plenv install-cpanm

Step 4: Install CPAN modules you need

For example, to install Moose, use this command:

$ cpanm Moose

If all dependencies build OK, Moose will be installed. If errors occur, a log file location will be shown for further analysis.

You can install multiple CPAN modules by listing them on the command line:

$ cpanm DBI DBD::SQLite JSON::MaybeXS

You can get help with "cpanm --help".

Posted in perl programming, toolchain | Tagged , , , | Comments closed

Flaming people versus flaming code

I think the Perl community in general (perhaps like many open source communities) suffers from two interrelated flaws:

(1) We think flaming someone's code is an acceptable or useful form of technical critique
(2) We react to flames against our code as personal attacks

When both occur, we have problems.

Statements like "this code is stupid" or "this code is insane" aren't constructive because they don't contain any useful information about what was seen as a problem and because most people reasonably take pride in their work and don't like hearing it disparaged.

Better: "This code is insane because it changes the public API" or "This code is stupid because it adds too many dependencies".

Even better: "I don't like this code because it changes the public API" or "I wouldn't use this code because it adds too many dependencies"

It's possible this generalizes into the "you/I" forms of argument.

E.g. "You're insane" vs "I'm angry" vs "When you add extra dependencies to a module I depend on, I get angry because I try to keep a minimum dependency tree and it means I have to do work to find an alternative to something I used to be able to rely on."

The next time someone's code bothers you, step back from your initial inclination to flame the code, and find a more constructive way to convey the specifics of what bothers you.

Posted in perl programming | Tagged , , | Comments closed

Back on the iron blogging horse

When the Perl Iron Man challenge was going, I blogged regularly. At a certain point, I stepped it down, and then, since last May, stopped.

It's time to get back on that horse.

I'm going to be aiming to meet the original Iron Man goals: 4 posts every 32 days, with no more than 10 days between posts.

And for January, I'm going to play catch up with a couple quick articles to follow.

If you've fallen off in your blogging, I invite you to join me.

Posted in Uncategorized | Comments closed

The next Test::More might break fragile test modules

I recently submitted a patch for Test::More that improves subtest readability, but might break fragile test modules. As part of the pull-request discussion, I promised to blog about the change to warn people, so here it is.

When Test::More added subtests (usable as of 0.96), the name of the subtest only appeared at the end of the output, like this:

ok 1 - regular test
    ok 1 - first test
    ok 2 - second test
    ok 3 - third test
ok 2 - my first subtest

That's a bit confusing, because it looks like the indented tests belong to the "ok - regular test", not the "ok 2 - my first subtest".

In Test::More 0.99, a leading comment was added to clarify:

ok 1 - regular test
    # Subtest: my first subtest
    ok 1 - first test
    ok 2 - second test
    ok 3 - third test
ok 2 - my first subtest

That's better, but still hard to pick out in a large amount of test output.

My pull request outdents the comment like this:

ok 1 - regular test
# Subtest: my first subtest
    ok 1 - first test
    ok 2 - second test
    ok 3 - third test
ok 2 - my first subtest

I think this is much easier to read because it brackets the subtest information at the same indentation level. The subtest details are clearly between the opening comment and the matching "ok".

This is likely to be merged and released soon.

Since comments don't have any meaning in TAP, it shouldn't change how tests are parsed. But CPAN test modules might — in their own tests — be looking for specific output and start breaking. Such tests are inherently fragile, and should be changed to use a TAP parser instead of parsing literal text, but in the meantime, if your favorite test class fails its tests, this Test::More change might be the reason.

Posted in perl programming | Tagged , , , , | Comments closed

Perl QA Hackathon 2014 Report

tl;dr → At the 2014 Perl QA hackathon in Lyon, I worked on PAUSE, Module::Metadata, version number semantics, Test::Harness,, CPAN::Reporter, Dist::Zilla and more.

Why do I love the QA hackathon?

As I mentioned in my TPF grant application, the QA hackathon allows me to work in a concentrated way for several days on parts of the Perl/CPAN toolchain and testing infrastructure that are "too big" for ad hoc development during the year. It also gives me an opportunity for face-to-face collaboration with other toolchain/quality hackers, which means getting answers, insights and making decisions much faster than happens over email, IRC, or ticket trackers.

The QA hackathon is like the best conference "hallway track" mashed up with a coding marathon with some of the most incredibly talented hackers in the Perl community.

Hacking at the QAH

What was different? What was the same? What worked? What didn't?

The big difference this year is that the organizers wisely shrunk the size of the event back to the 30-ish number that had been typical for most of the early years of the hackathon. At that number of attendees, everyone can pretty much know what everyone else is working on and basic logistics take up much less time.

There were several new faces, including Karen Etheridge, Graham ("one-p") Knop, and Neil Bowers. And there were a lot of familiar faces, including many that I only see once a year at the hackathon.

In addition to the smaller group, I really appreciated how much the organizers optimized for productive time. Breakfast and lunch were provided at the venue, and there was only a single organized dinner out and another organized dinner at the hotel. That meant less time in transit for food and more time to get stuff done.

Sadly, as in some previous years, the network was flaky, both at the hotel and the conference venue, which is always a distraction and occasional barrier to getting work done.

Giving thanks

Before I give the day-by-day recap, I want remind readers that each year, the QA hackathon happens because of the dedicated volunteer work of the organizers and the financial support of sponsors.

I offer many thanks to Philippe Bruhat and Laurent Boivin for putting together an excellent hackathon, to for providing our venue, and to Wendy van Dijk for helping each day with critical logistics: ensuring we did not lack for food or drink! I also particularly want to thank the The Perl Foundation for the travel grant that allowed me attend.

All our sponsors deserve great thanks! These companies are putting their money where their mouth is to make the Perl ecosystem better for everyone:, SPLIO, Grant Street Group, DYN, Campus Explorer, EVOZON, elasticsearch, Eligo, Mongueurs de Perl, WenZPerl for the Perl6 Community, PROCURA and Made In Love.

If you think the QA hackathon work is valuable for the Perl community, please consider making a late contribution to the hackathon fund to support the 2015 QA Hackathon.

Thank you, also, to my fellow participants. You're the reason I keep going back.

Day-by-day Recap

I'm going to give a pretty detailed, stream-of-consciousness replay, because I think it will give readers some insight into the frenetic way the QA hackathon tends to work. It's rare (at least for me) to be working on just one project for very long. Frequently I jumped back and forth between discussions with people and actually coding.

Day 0

Ricardo Signes and I flew together and arrived in Lyon in the afternoon with the usual red-eye flight exhaustion. We met up with Karen and Barbie at the airport for the ride to the hotel. We met up with most of the rest of the hackers that night for drinks and dinner and started to swap ideas about what we might be working on.

Barbie, Ricardo and Tux

Day 1

On Thursday, after introductions and the "stand-up" where we each talked about our plans, I pulled together a bunch of people to talk about PAUSE issues and tasks. One of the big topics was how to implement some of the decisions taken in the Lancaster Consensus the previous year. We also talked about how to get stricter about case-sensitivity, to avoid the "ElasticSearch renaming" problem.

Another related topic was separating assigning permissions from indexing modules, so that someone could get permissions on the namespace of a module while still releasing non-indexed developer versions of it.

Later that day, I summarized all the discussion into a PAUSE distribution permissions and indexing rules document. The big change is that — per Lancaster Consensus — your distribution "name" (the first part of the tarball filename) will need to match a Perl package that you have upload permissions for.

Ricardo and others then went off to implement various parts of this and solve the problem of existing distributions that don't match a package and I'll let him/others cover that in other blog posts. (Update: see Ricardo's blog post)

Next, I talked to Karen and Graham about improving the security of Module::Metadata (which has to evaluate code to determine $VERSION) so that perhaps it could eventually be used by PAUSE. They took some prototype work I already had and started running with it, checking with Christian Walde about how to handle sub-process issues portably to Windows.

At other points during the day, I sent Tim Bunce some ideas for how role-based testing might help DBI testing and helped Jens Rehsack with a warnocked takeover request for some of Adam Kennedy's modules.

I also took a moment to get everyone's attention so that I could hand out a special award (only partly in jest) to Peter Rabbitson for his efforts keeping backwards compatibility for Perl 5.6. I called it the Wandering Albatross Award and gave him a stuffed albatross. It was also Peter's birthday, so I got us to sing to him. He took it all well and the albatross kept him company for the rest of the hackathon.

Peter and the albatross

With all these discussions going on, it wasn't until after dinner that I got to any of my own coding, but I managed to do a couple cool things before bedtime:

  • Revived a patch for Test::Harness to let authors define rules for parallel testing via a file in their distributions (more in Day 2)
  • Sent a pull request with a configuration option to automatically switch on PERL_MM_USE_DEFAULT for prompt-free installation.

Day 2

On Friday, I finished my work on a test rules file for Test::Harness and fired off a pull request for it. The problem was that some people are setting parallel testing by default to speed up module installation. This usually works, but some distributions have tests that won't work in parallel. Fixing a test suite like that is a lot of work, but instead, they'll be able to add a testrules.yml file that specifies their tests don't work in parallel and need to be run in series. The rules code for this was always in TAP::Harness, but there was no way for authors to control it short of customizing ExtUtils::MakeMaker or Module::Build. Now there will be.

Jérôme Quelin stopped by to discuss some cpan -O bugs finding outdated modules. There were two cases in which apparently identical versions between the locally-installed module and CPAN were being reported as out of date. One of those turned out to be a bug in the decimal precision of the report and I sent Jerome to file a ticket on App::Cpan. The other turned out to be a bug in how compared "undef" and "0" and I send Andreas a pull request to fix it.

I then worked on my own pull request backlog for a bit to get things shipped:

  • Test::API patches for fewer dependencies and a class-api test
  • Term::Title fixes for non-interactive testing
  • CPAN::Reporter::Smoker patch to skip dev distributions

All that work got me to dinner time and I went with a small group (Ricardo, Peter, Karen, Graham, and Leon) to have a dinner discussion about various issues and challenges relating to version objects and the toolchain. We decided to momentarily set aside the current state of affairs with respect to version objects and just talk about the different ways that versions are represented in Perl (decimal and tuple forms). We sketched out our ideal semantics and transformations and I wrote it up after we got back to the hotel. (Leon would later do some prototyping on Saturday and Sunday).

We'll continue the discussion virtually over the next several months and see if it leads to a concrete proposal for how to rationalize version number semantics in perl 5.21.

Day 3

My plan for Saturday was to dive deep into Earlier this year, Andreas discovered a major regression in the "force" pragma for the CPAN shell stemming from my work in Lancaster last year. As an emergency fix, he reverted a dozen or so commits. My goal was to try to recover the reversions, while squashing all the bugs.

Unfortunately, shit happens. The CPAN Testers Metabase (which collects test report submissions) chose that morning to stop working. I discovered that the EC2 instance it was on had gotten wedged, and in the kind of way that EC2's usual "stop" command wouldn't even work. So I spent the rest of the morning in EC2 hackery to get Metabase back up and the CPAN Testers reports flowing again.

It was ironic that Metabase died when I was at the hackathon, and yet the hackathon gave me all the round tuits I needed to get it fixed.

With the Metabase repaired, I got back to I covered my bases: first I sent Andreas a pull request to revert something else that needed reverting if the reversions were going to stand, then I branched off before the reversions and figured out how to fix the 'force' pragma bug directly.

Along the way, I stopped to fix CPAN::Reporter's prereqs reporting when used with a that supports recommends/suggests prerequisites.

And throughout the day, I continued the version numbering semantics discussions from the previous evening to test, clarify and refine our understanding.

In the evening, I spent some hours trying and failing to replicate another bug that Andreas had described.

Day 4

Sunday morning started out with a snag — we were unable to get into the office and had to work from the hotel. I took advantage of the time to get Andreas to walk me in great detail through the bug I couldn't replicate. Thank goodness he keeps copious notes! By the time we got to the hackathon venue, I had a good hypothesis for what I needed to do to replicate it.

With only one afternoon to go, I tried to avoid further discussions and focus on code:

  • I implemented metadata fragment conversion in CPAN::Meta::Converter, the lack of which was blocking Leon's CPAN::Meta::Merge pull request. The I fixed up CPAN::Meta::Merge to use the new feature the way I implemented it. Along the way, I roped Karen into writing some additional tests for fragment conversion and generally sanity checking my code. CPAN::Meta::Merge will make it much easier for distribution packagers to safely and sanely create META files from a mix of detected and provided metadata.
  • I cleaned up CPAN::Reporter's repository, made its tests more efficient and shipped it
  • When working on CPAN::Reporter, I wished there was an easy way to run "dzil test" with parallel testing, so I implemented that and sent Ricardo a pull request. (He made it even more general and shipped a new Dist::Zilla within the hour.)

All this got me wondering why dzil build ran so slowly on CPAN::Reporter and someone suggested I try running PERL5OPT=-d:NYTProf dzil build and looking at the flame graph of the result:

CPAN::Reporter flame graph

(note: this image is a reconstruction *after* optimization, but it gives you the idea)

Ricardo and a bunch of people looked over my shoulder as we analyzed it and we realized how terribly slow the PerlMinimum plugin was. I also realized my own InsertCopyright plugin wasn't using the new PPI caching mechanism. So I swapped out PerlMinimum for PerlMinimumFast and patched InsertCopyright. That cut my build time by about 40%.

We also realized that the biggest subroutines were PPI "find" ones, and that inspired Ricardo to think about ways of indexing the PPI DOM for more efficient queries.

I then reviewed the Module::Metadata work that Karen and Graham had been working on and gave it a thumbs up but for some minor comments, and looked at some other pull requests that had been flying around.

We stopped for clean up, group pictures, and headed back to the hotel for dinner.

After dinner, I finally got a chance to finish my work and had a clean branch that avoided reverting recommends/suggests support, fixed the force pragma bug, avoided the other bug that Andreas showed me, and cherry-picked half a dozen commits that had come in after the reversion (including three of my own from the hackathon).

Day 5

Monday was a travel day. While on the plane, I worked on cleaning up all my work to be less confusing when I sent it to Andreas.

I also worked on a way of reporting deep dependencies during automated testing, so that Andreas' analysis service can more easily detect when test failures are due to a deep dependency. It's not done, but I hope to make it a standard part of CPAN testing sometime "soon" (i.e. before next year's hackathon).

And that was it. I got home, took a shower, talked to my wife, and fell into bed exhausted.

The 2014 Perl QA hackathon was over.

Posted in perl programming | Tagged , , , , , , , , | Comments closed

© 2009-2016 David Golden All Rights Reserved