Why HTTP::Tiny?

I've recently been collaborating with Christian Hansen on HTTP::Tiny, a minimalist, HTTP/1.1 client library for Perl. For basic web client tasks like grabbing a single page or mirroring a file, it does the job in a fraction of the code that would be needed to install LWP::UserAgent. Because it has no non-core dependencies, it is ideal for what I need to get CPAN.pm to bootstrap itself with pure Perl.

Here's a quick look at how you would use it for the two common tasks I mentioned:

    use HTTP::Tiny;
    my $http = HTTP::Tiny->new;
    my $response;

    # get a single page
    $response = $http->get('http://example.com/');
    die "Failed!\n" unless $response->{success};
    print $response->{content};

    # mirror a file
    $response = $http->mirror('http://example.com/file.tar.gz', 'file.tar.gz');
    die "Failed!\n" unless $response->{success};
    print "Unchanged!\n" if $response->{status} eq '304';

The example above is almost the same as you'd get using LWP::UserAgent with one big exception. HTTP::Tiny returns the response as a hash reference rather than as an object. Just like LWP::UserAgent, the mirror method will send an If-Modified-Since header for an existing file to skip downloading if the file is unchanged. HTTP::Tiny doesn't (yet) handle query parameters -- you have to prepare those yourself, but for simple downloads, you don't generally need those anyway.

Where did HTTP::Tiny come from? When I was working on getting CPAN.pm to support a pure-Perl HTTP bootstrap, I started with HTTP::Lite. When I discussed it on #p5p, Christian Hansen pointed out a number of serious shortcomings and decided that it would be easier for him to write a new, lightweight HTTP/1.1 client from scratch rather than try to redo the plumbing in HTTP::Lite.

The result is HTTP::Tiny and I've been collaborating with Christian to get it ready for use by CPAN.pm and ready for the Perl core. Unlike HTTP::Lite, HTTP::Tiny is a conditionally conforming HTTP/1.1 client. It supports both redirection and mirroring, which HTTP::Lite does not, both of which are important features for a CPAN client.

As a "Tiny" module, HTTP::Tiny achieves its HTTP/1.1 conformance in a just a fraction the code required for LWP::UserAgent and its non-core dependents. Don't get me wrong -- LWP::UserAgent is a great piece of software. But sometimes -- like for the Perl core or for a fatpacked application -- something much smaller and simpler will do just as well.

Let's see what a difference there is. CPANdeps shows us all of LWP::UserAgent's dependencies and highlights the ones that are non-core (as of Perl 5.10.1):

• URI
• HTML::Parser
• HTML::Tagset

I used David A. Wheeler's SLOCCOUNT to count lines of code in each of the distributions containing these modules as well as for HTTP::Lite and HTTP::Tiny (based on the "soon-to-be" 0.007 release). In all cases, I excluded files in t/ and examples directories. (Lines of Perl includes any programs distributed with the distribution.)

Here are the results:

Distribution        .pm files Lines (Perl) Lines (C)  Total Lines
-----------------   --------- ------------ ----------  -----------
libwww-perl-5.837       52        10258          0       10258
HTML-Parser-3.68         7          883       1972        2855
HTML-Tagset-3.20         1          139          0         139
URI-1.56                52         2715          0        2715
                     -----        -----      -----       -----
TOTAL LWP & friends    112        13995       1972       15967

versus

Distribution        .pm files Lines (Perl) Lines (C)  Total Lines
-----------------   --------- ------------ ----------  -----------
HTTP-Lite-2.3            1          634          0         634
HTTP-Tiny-0.007          1          603          0         603

Wow! Both HTTP::Lite and HTTP::Tiny accomplish simple HTTP tasks with less than 4% of the SLOC of LWP and its non-core dependencies.

What about memory usage? Here's the "null" case that shows memory usage of just loading the three client modules:

  VSZ   RSS COMMAND
30440  5512 perl -MLWP::UserAgent -e 1 while 1
26700  3960 perl -MHTTP::Tiny -e 1 while 1
26040  3168 perl -MHTTP::Lite -e 1 while 1

HTTP::Lite is smallest, but it also doesn't have the features or conformance I need.

That's not much of a real-world test, so let's try something else -- downloading the 950K CPAN 02packages.details.txt.gz file. Here is the test code for HTTP::Tiny and LWP::UserAgent side by side:

# http-tiny-mirror.pl                  # lwp-useragent-mirror.pl
#!/usr/bin/env perl                    #!/usr/bin/env perl                                 
use strict;                            use strict;                                         
use warnings;                          use warnings;                                       
use HTTP::Tiny;                        use LWP::UserAgent;                                 
                                                                                           
my ($url, $file) = @ARGV;              my ($url, $file) = @ARGV;                           
die unless $url & $file;               die unless $url & $file;                            
                                                                                           
my $http = HTTP::Tiny->new;            my $http = LWP::UserAgent->new;                     
my $res = $http->mirror($url, $file);  my $res = $http->mirror($url, $file);               
die "Failed!\n"                        die "Failed!\n"                                     
  unless $res->{success};                unless $res->is_success;                          
                                                                                      
1 while 1;                             1 while 1;                                          

What about HTTP::Lite? It doesn't have a mirror method and doesn't do redirection, which I get for free with HTTP::Tiny and LWP::UserAgent. I'd have to write dozens of lines of code to emulate that for a "fair" test, so I skipped it.

Let's add those two programs (after downloading the 02packages file to files with different names) to our memory benchmarks:

  VSZ   RSS COMMAND
50916  9824 perl ./lwp-useragent-mirror.pl [...]
35376  4348 perl ./http-tiny-mirror.pl [...]
30440  5512 perl -MLWP::UserAgent -e 1 while 1
26700  3960 perl -MHTTP::Tiny -e 1 while 1
26040  3168 perl -MHTTP::Lite -e 1 while 1

HTTP::Tiny wins -- not by a huge amount in absolute terms, admittedly, but if you don't need the extra features that LWP::UserAgent offers, HTTP::Tiny might just be all you need.

Thank you, Christian, for HTTP::Tiny!