Parallel map with Parallel::Iterator

By dagolden | Published: August 3, 2010

Do you have a multi-core processor? Would you like an easy way for Perl to use all your cores to run an iterative task in parallel? Then you should check out Parallel::Iterator.

Parallel::Iterator provides a very simple API to execute a "map" function in parallel across multiple processes. It handles all the forking and inter-process communication for you, so all you have to do is focus on the task at hand. This is useful for a CPU-intensive task that you'd like to spread across multiple cores or for tasks with relatively long I/O delays.

Here is a very simple example, adapted from the documentation. Suppose you need to fetch a lot of web-pages from URL's provided on STDIN. You just need to write the "worker" subroutine to fetch a page and Parallel::Iterator does the rest.

use strict;
use warnings;
use LWP::UserAgent;
use Parallel::Iterator qw/iterate_as_array/;

# this worker fetches a page or returns undef
my $ua = LWP::UserAgent->new( env_proxy => 1 );
my $worker = sub {
  my $url = shift;
  my $resp = $ua->get($url);
  return undef unless $resp->is_success;
  return $resp->decoded_content;
};

# this gets a list of pages and fetches them in parallel
my @urls = split "\n", do { local $/; <STDIN> };
my @pages = iterate_as_array( $worker, @urls );

# now do stuff with the results
...

With just that little code, the URL's to fetch are split across 10 sub-processes, the page fetching happens in parallel and the results are collected into the @pages array.

Parallel::Iterator has a number of additional features for customizing the number of subprocesses, or iterating the results (instead of getting back an array) and for batching the input to subprocesses.

It may not be the right tool all of the time, but if you need to parallelize a task quickly, Parallel::Iterator should be in your tool-box.

Posted in perl-programming | Tagged , | 3 Comments

perlprogramming.org looking for a nice home

By dagolden | Published: July 27, 2010

I picked up this domain name a while ago since it was available and people were saying how Perl needs better visibility. (I was probably inspired by Tim's post about TIOBE). Currently, I just redirect to perl.org. The domain is coming up for renewal and I'll gladly sign it over to someone who can make a good case for what they plan to do with it to benefit the perl community.

Posted in perl-programming | Tagged , | Leave a comment

My OSCON talks are online

By dagolden | Published: July 25, 2010

I've posted my OSCON talks (one regular talk and one lightning talk) in my Talks page. But for those wanting direct links, here they are:

  • Free QA! -- a non-technical talk about the history and social architecture choices of the CPAN Testers project
  • Perl 5, Version 13 -- a lightning talk summarizing notable changes in the Perl 5.13 development series
Posted in cpan-testers, perl-programming | Tagged , | Leave a comment

Perl 5.13.3 is released

By dagolden | Published: July 20, 2010

[Reposted from my announcement to the the perl5-porters mailing list]

Look at Crowley, doing 110 mph on the M40 heading towards
Oxfordshire. Even the most resolutely casual observer would
notice a number of strange things about him. The clenched teeth,
for example, or the dull red glow coming from behind his
sunglasses. And the car. The car was a definite hint.

Crowley had started the journey in his Bentley, and he was
dammned if he wasn't going to finish it in the Bentley as well.
Not that even the kind of car buff who owns his own pair of
motoring goggles would have been able to tell it was a vintage
Bentley. Not any more. They wouldn't have been able to tell
that it was a Bentley. They would only offer fifty-fifty that it
had ever even been a car.

There was no paint left on it, for a start. It might still have
been black, where it wasn't a rusty, smudged reddish-brown, but
this was a dull charcoal black. It traveled in its own ball of
flame, like a space capsule making a particularly difficult
re-entry.

There was a thin skin of crusted, melted rubber left around the
metal wheel rims, but seeing that the wheel rims were still
somhow riding an inch above the road surface this didn't seem to
make an awful lot of difference to the suspension.

It should have fallen apart miles back.

-- Neil Gaiman and Terry Pratchett, "Good Omens"

It gives me great pleasure to announce the release of Perl 5.13.3.

This is the fourth DEVELOPMENT release in the 5.13.x series leading to a stable release of Perl 5.14.0. You can find a list of high-profile changes in this release in the file "perl5133delta.pod" inside the distribution.

You can (or will shortly be able to) download the 5.13.3 release from:

http://search.cpan.org/~dagolden/perl-5.13.3/

The release's SHA1 signatures are:

This release corresponds to commit 414abf8 in Perl's git repository. It is tagged as 'v5.13.3'.

We welcome your feedback on this release.

If Perl 5.13.3 works well for you, please use the 'perlthanks' tool included with this distribution to tell the all-volunteer development team how much you appreciate their work.

If you discover issues with Perl 5.13.3, please use the 'perlbug' tool included in this distribution to report them.

If you write software in Perl, it is particularly important that you test your software against development releases. While we strive to maintain source compatibility with prior stable versions of Perl wherever possible, it is always possible that a well-intentioned change can have unexpected consequences. If you spot a change in a development version which breaks your code, it's much more likely that we will be able to fix it before the next stable release. If you only test your code against stable releases of Perl, it may not be possible to undo a backwards-incompatible change which breaks your code.

Perl 5.13.3 represents approximately one month of development since Perl 5.13.2, and contains 12,184 lines of changes across 575 files from 104 authors and committers.

Notable changes in this release:

  • \o{...} has been added as a string escape for octals.
  • \N{} and charnames::vianame now know about the abbreviated character names listed by Unicode, such as NBSP, SHY, etc.
  • Most dual-life module have been synchronized with the latest production release on CPAN.
  • There is a new internal function PL_blockhook_register for XS code to hook into Perl's lexical scope mechanism

There is one major known issue:

  • Bug fixes involving CvGV reference counting break Sub::Name (currently version 0.04). A patch has been sent upstream to the maintainer.

Thank you to the following for contributing to this release:

Abhijit Menon-Sen, Abigail, Alex Davies, Alex Vandiver, Alexandr Ciornii, Andreas J. Koenig, Andrew Rodland, Andy Dougherty, Aristotle Pagaltzis, Arkturuz, Ben Morrow, Bo Borgerson, Bo Lindbergh, Brad Gilbert, Bram, Brian Phillips, Chas. Owens, Chip Salzenberg, Chris Williams, Craig A. Berry, Curtis Jewell, Dan Dascalescu, Daniel Frederick Crisman, Dave Rolsky, David Caldwell, David E. Wheeler, David Golden, David Leadbeater, David Mitchell, Dennis Kaarsemaker, Eric Brine, Father Chrysostomos, Florian Ragwitz, Frank Wiegand, Gene Sullivan, George Greer, Gerard Goossen, Gisle Aas, Goro Fuji, Graham Barr, H.Merijn Brand, Harmen, Hugo van der Sanden, James E Keenan, James Mastros, Jan Dubois, Jerry D. Hedden, Jesse Vincent, Jim Cromie, John Peacock, Jos Boumans, Josh ben Jore, Karl Williamson, Kevin Ryde, Leon Brocard, Lubomir Rintel, Maik Hentsche, Marcus Holland-Moritz, Matt Johnson, Matt S Trout, Max Maischein, Michael Breen, Michael G Schwern, Moritz Lenz, Nga Tang Chan, Nicholas Clark, Nick Cleaton, Nick Johnston, Niko Tyni, Offer Kaye, Paul Marquess, Philip Hazel, Philippe Bruhat, Rafael Garcia-Suarez, Rainer Tammer, Reini Urban, Ricardo Signes, Richard Soderberg, Robin Barker, Ruslan Zakirov, Salvador Fandino, Salvador Ortiz Garcia, Shlomi Fish, Sinan Unur, Sisyphus, Slaven Rezic, Steffen Mueller, Stepan Kasal, Steve Hay, Steve Peters, Sullivan Beck, Tim Bunce, Todd Rinaldo, Tom Christiansen, Tom Hukins, Tony Cook, Vincent Pit, Yuval Kogman, Yves Orton, Zefram, brian d foy, chromatic, kmx, Ævar Arnfjörð Bjarmason

Many of the changes included in this version originated in the CPAN modules included in Perl's core. We're grateful to the entire CPAN community for helping Perl to flourish.

Development versions of Perl are released monthly on or about the 20th of the month by a monthly "release manager". You can expect following upcoming releases:

  • August 20 - Florian Ragwitz
  • September 20 - Steve Hay
  • October 20 - Tatsuhiko Miyagawa
  • November 20 - Chris Williams
Posted in perl-programming | Tagged , | Leave a comment

How to join the CPAN Testers 2.0 Public Beta

By dagolden | Published: July 13, 2010

In case you missed the recent launch announcement, CPAN Testers 2.0 is live and the email mailing list used to submit CT 1.0 reports will be shut down around August 31, 2010. I consider this the start of the "public beta" period for CT 2.0. Most of the beta testers so far have been experienced testers, but I'm sure we'll see lots of new questions and issues to fix as more people start switching over to CT 2.0 before the deadline.

If you are already a CPAN Tester, a "profile file" has been pre-generated for you and you should submit a request that it be emailed to you. This pre-generated profile will identify your new CT 2.0 reports in a way that will make it easier to link them to the CT 1.0 old reports as those are converted.

If you are new to CPAN Testing, you will need to generate a profile file. Install Metabase::Fact and then run the metabase-profile program from the command line. It will prompt you to provide some information and output a profile file. The first time you use this profile file to submit a report, your profile will be automatically registered on the CT 2.0 server.

While the CT 2.0 server has launched, the clients that submit reports are, for now, the same ones used for CT 1.0 -- either CPAN::Reporter or CPANPLUS (or one of the smoke frameworks written for them like minismokebox or CPAN::Reporter::Smoker). The transport framework originally intended for email gets hijacked to send reports via http to CT 2.0 instead.

To send reports to CT 2.0, you will need to install the Test::Reporter::Transport::Metabase module and then configure your client to use that transport instead of the default one. Instructions are available on the CPAN Testers Wiki. While the docs are a bit sparse as I write this, I expect they will continue to evolve and improve as more people contribute to them.

If you're really stuck, please check the FAQ or ask for help on the CPAN Testers Discuss mailing list. If you are a CPAN Tester, I highly recommend joining the list to keep up on the latest issues and solutions in the migration to CT 2.0. If you do ask for help, please be sure to include the version number of all the relevant testing modules you're using along with your perl -V output.

Another good place to go for quick questions is the #cpantesters-discuss channel on irc.perl.org.

In other news, I'll be speaking about CPAN Testers at OSCON in my Free QA! talk. If you or someone you know will be there, please check it out or recommend it.

Posted in cpan-testers, metabase, perl-programming | Tagged , | 1 Comment

© 2009-2010 David Golden All Rights Reserved