Why HTTP::Tiny?

I've recently been collaborating with Christian Hansen on HTTP::Tiny, a minimalist, HTTP/1.1 client library for Perl. For basic web client tasks like grabbing a single page or mirroring a file, it does the job in a fraction of the code that would be needed to install LWP::UserAgent. Because it has no non-core dependencies, it is ideal for what I need to get CPAN.pm to bootstrap itself with pure Perl.

Here's a quick look at how you would use it for the two common tasks I mentioned:

    use HTTP::Tiny;
    my $http = HTTP::Tiny->new;
    my $response;

    # get a single page
    $response = $http->get('http://example.com/');
    die "Failed!\n" unless $response->{success};
    print $response->{content};

    # mirror a file
    $response = $http->mirror('http://example.com/file.tar.gz', 'file.tar.gz');
    die "Failed!\n" unless $response->{success};
    print "Unchanged!\n" if $response->{status} eq '304';

The example above is almost the same as you'd get using LWP::UserAgent with one big exception. HTTP::Tiny returns the response as a hash reference rather than as an object. Just like LWP::UserAgent, the mirror method will send an If-Modified-Since header for an existing file to skip downloading if the file is unchanged. HTTP::Tiny doesn't (yet) handle query parameters -- you have to prepare those yourself, but for simple downloads, you don't generally need those anyway.

Where did HTTP::Tiny come from? When I was working on getting CPAN.pm to support a pure-Perl HTTP bootstrap, I started with HTTP::Lite. When I discussed it on #p5p, Christian Hansen pointed out a number of serious shortcomings and decided that it would be easier for him to write a new, lightweight HTTP/1.1 client from scratch rather than try to redo the plumbing in HTTP::Lite.

The result is HTTP::Tiny and I've been collaborating with Christian to get it ready for use by CPAN.pm and ready for the Perl core. Unlike HTTP::Lite, HTTP::Tiny is a conditionally conforming HTTP/1.1 client. It supports both redirection and mirroring, which HTTP::Lite does not, both of which are important features for a CPAN client.

As a "Tiny" module, HTTP::Tiny achieves its HTTP/1.1 conformance in a just a fraction the code required for LWP::UserAgent and its non-core dependents. Don't get me wrong -- LWP::UserAgent is a great piece of software. But sometimes -- like for the Perl core or for a fatpacked application -- something much smaller and simpler will do just as well.

Let's see what a difference there is. CPANdeps shows us all of LWP::UserAgent's dependencies and highlights the ones that are non-core (as of Perl 5.10.1):

I used David A. Wheeler's SLOCCOUNT to count lines of code in each of the distributions containing these modules as well as for HTTP::Lite and HTTP::Tiny (based on the "soon-to-be" 0.007 release). In all cases, I excluded files in t/ and examples directories. (Lines of Perl includes any programs distributed with the distribution.)

Here are the results:

Distribution        .pm files Lines (Perl) Lines (C)  Total Lines
-----------------   --------- ------------ ----------  -----------
libwww-perl-5.837       52        10258          0       10258
HTML-Parser-3.68         7          883       1972        2855
HTML-Tagset-3.20         1          139          0         139
URI-1.56                52         2715          0        2715
                     -----        -----      -----       -----
TOTAL LWP & friends    112        13995       1972       15967

versus

Distribution        .pm files Lines (Perl) Lines (C)  Total Lines
-----------------   --------- ------------ ----------  -----------
HTTP-Lite-2.3            1          634          0         634
HTTP-Tiny-0.007          1          603          0         603

Wow! Both HTTP::Lite and HTTP::Tiny accomplish simple HTTP tasks with less than 4% of the SLOC of LWP and its non-core dependencies.

What about memory usage? Here's the "null" case that shows memory usage of just loading the three client modules:

  VSZ   RSS COMMAND
30440  5512 perl -MLWP::UserAgent -e 1 while 1
26700  3960 perl -MHTTP::Tiny -e 1 while 1
26040  3168 perl -MHTTP::Lite -e 1 while 1

HTTP::Lite is smallest, but it also doesn't have the features or conformance I need.

That's not much of a real-world test, so let's try something else -- downloading the 950K CPAN 02packages.details.txt.gz file. Here is the test code for HTTP::Tiny and LWP::UserAgent side by side:

# http-tiny-mirror.pl                  # lwp-useragent-mirror.pl
#!/usr/bin/env perl                    #!/usr/bin/env perl                                 
use strict;                            use strict;                                         
use warnings;                          use warnings;                                       
use HTTP::Tiny;                        use LWP::UserAgent;                                 
                                                                                           
my ($url, $file) = @ARGV;              my ($url, $file) = @ARGV;                           
die unless $url & $file;               die unless $url & $file;                            
                                                                                           
my $http = HTTP::Tiny->new;            my $http = LWP::UserAgent->new;                     
my $res = $http->mirror($url, $file);  my $res = $http->mirror($url, $file);               
die "Failed!\n"                        die "Failed!\n"                                     
  unless $res->{success};                unless $res->is_success;                          
                                                                                      
1 while 1;                             1 while 1;                                          

What about HTTP::Lite? It doesn't have a mirror method and doesn't do redirection, which I get for free with HTTP::Tiny and LWP::UserAgent. I'd have to write dozens of lines of code to emulate that for a "fair" test, so I skipped it.

Let's add those two programs (after downloading the 02packages file to files with different names) to our memory benchmarks:

  VSZ   RSS COMMAND
50916  9824 perl ./lwp-useragent-mirror.pl [...]
35376  4348 perl ./http-tiny-mirror.pl [...]
30440  5512 perl -MLWP::UserAgent -e 1 while 1
26700  3960 perl -MHTTP::Tiny -e 1 while 1
26040  3168 perl -MHTTP::Lite -e 1 while 1

HTTP::Tiny wins -- not by a huge amount in absolute terms, admittedly, but if you don't need the extra features that LWP::UserAgent offers, HTTP::Tiny might just be all you need.

Thank you, Christian, for HTTP::Tiny!

This entry was posted in cpan, perl programming, toolchain and tagged , . Bookmark the permalink. Both comments and trackbacks are currently closed.

12 Comments

  1. Posted January 11, 2011 at 9:30 pm | Permalink

    Cool. I love the ::Tiny movement, though I really seldom use the ::Tiny modules. It just feels that Perl programmers are solving real-world problems, with real-world constraints (like low memory/low CPU situation).

  2. zloyrusskiy
    Posted January 11, 2011 at 9:56 pm | Permalink

    There is Mojo::Client module without non-Core dependencies, why can't you use it?

    • Posted January 11, 2011 at 10:42 pm | Permalink

      Because it contains 14 "Mojo" dependencies and uses more memory.

      68388 13700 perl -MMojo::Client -e 1 while 1

  3. Posted January 11, 2011 at 10:08 pm | Permalink

    I like that you can write a whole one-liner without sigils

    perl -MHTTP::Tiny -E 'say HTTP::Tiny->new->get(shift)->{content}' http://www.whatismyip.org/

    With HTTP::Lite, we need a name for the instance

    perl -MHTTP::Lite -E '$h = HTTP::Lite->new; $h->request(shift); say $h->body' http://www.whatismyip.org/

  4. Christian Hansen
    Posted January 12, 2011 at 4:34 am | Permalink
  5. Clinton
    Posted January 12, 2011 at 10:03 am | Permalink

    You wouldn't consider adding the ability to POST as well? I'd happily add support for HTTP::Tiny to ElasticSearch.pm, but it needs to be able to POST

    clint

    • Posted January 12, 2011 at 11:58 am | Permalink

      It does support POST, but you have to prepare the content yourself. It doesn't do any escaping of key/value pairs at least not yet. That gets tricky when you start considering locales/Unicode, so it's left to the user. But there is a generic "request" method that works for POST.

      $response = $http->request('POST', $url, { 
          content => $content,
          headers => { 'content-type' => 'application/x-www-form-urlencoded' },
      });
      

  6. Jerome
    Posted January 12, 2011 at 7:36 pm | Permalink

    Thanks very much; already incorporated for a module suite I'm about to release. The LWP dependencies have gotten rather silly lately for a quick single file download.

    Ability to manually manipulate the headers in this was a life-saver, too, since I have to download from what appears to be a non-RFC-compliant webserver (shame on them).

  7. Kaare Rasmussen
    Posted January 28, 2011 at 7:33 am | Permalink

    Catalyst::Model::Rest (http://search.cpan.org/dist/Catalyst-Model-REST/) just switched to using HTTP::Tiny :-)

  8. Posted February 19, 2012 at 8:28 am | Permalink

    Thanks!

    Small typo in example:

    reponse -> response

    Also, I suggest a "use strict", although I recognize this is a style issue.

    • Posted February 19, 2012 at 10:15 pm | Permalink

      Thanks for catching that. I've fixed it.

      FWIW, I generally "use strict" in all real programs and recommend that others do as well unless they have good reason not to. I tend to omit it from short examples because I think it clutters and distracts from the point of the example. (You'll note that both "real-world test" examples include it.)

2 Trackbacks

© 2009-2014 David Golden All Rights Reserved