My second week of Dancer, now with queues and transactional email

A couple weeks ago, I wrote about my initial efforts with Dancer, Xslate and Bootstrap. Last week, I added the ability to send password reset emails. In the process, I've learned how to write Dancer plugins.

Designing a password reset system

Since I've never done it before, I decided to reinvent the wheel and make it work the way I think it should.

I broke the problem down into three big chunks:

  1. generate and store a random token
  2. send the token to a user's registered email address
  3. receive the token and prompt a password change

Then I made a couple additional architectural decisions.

First, I decided to make the token system generic, as I expect there will be other things I'll use tokens for, such as email address confirmation. Regardless of the action that happens when the token is received, the first two steps above are nearly identical. All I need to do is make sure that tokens have a type and can store whatever additional data is needed to complete the action.

Second, I decided to break email sending out of the web app itself. Instead, an email is generated within the app and dropped into an asynchronous message queue. A separate program monitors the queue and sends the emails. In addition to insulating the web app from latency from sending email, a message queue makes unit testing a lot easier. All I need to do is see if the web app dropped the right message into the queue. I don't have to mock up an email delivery system.

Creating the token model

I decided it was sufficient to have a token be a URL-friendly, base64-encoded random value associated with a username, a "type" (e.g. password reset), an expiration, and an arbitrary "value" field.

For what it's worth, here's the trivial code for generating a random value:

use MIME::Base64 qw/encode_base64url/;
use Data::Entropy::Algorithms qw/rand_bits/;

$token = encode_base64url( rand_bits(192) ),

One of the reasons for having a generic "value" field is that I'm using MongoDB for my data store, so any JSON-serializable data can go in there "for free". If I wasn't using a document-based data store, I'd have to think more carefully about the value field semantics, or I'd have to serialize to/from the field with JSON or Sereal or something like that.

Speaking of MongoDB, I've been pretty pleased with Mongoose as a MongoDB->Moose mapper and the related Dancer::Plugin::Mongoose. I can specify my model classes and their associated database connection parameters directly in the Dancer config.yml file. If I were using a relational database, I'd probably look into Dancer::Plugin::DBIC, instead.

Loose coupling

Since I didn't want to send the email directly from the application, I needed a message queue. Since I'm using MongoDB and have other (backend) reasons for using it for other message queues, I decided to use it here as well. Otherwise, I might have explored Amazon SQS, Gearman, or Redis. Generally, my approach is to use a small number of versatile tools that I can develop expertise in rather than spread my expertise across a giant toolbox. This, of course, is why Perl is my #1 tool.

I had already written MongoDBx::Queue, so that was done. What I needed was to get Dancer to use it. A Dancer plugin for it would need to do a few useful things:

  1. Gather config data for (one or more) queues
  2. Instantiate a singleton for the life of the app
  3. Extend the Dancer DSL to provide access.

Again, I prefer loose coupling and frameworks, so instead of writing Dancer::Plugin::MongoDBx::Queue, instead I wrote Dancer::Plugin::Queue, which is a generic queue interface. Then I wrote Dancer::Plugin::Queue::MongoDB to implement the generic mechanism using MongoDBx::Queue.

Other Dancers who might favor other message queue systems just need to write similar implementation plugins and then message queues become an interchangeable component, just like template systems and session management. Loose coupling for the win!

Here is a slightly simplified version of the resulting code:

# generate reset token
my $token = schema("token")->new(
  user => $user->username,
  type => 'p',            # password reset type
);
$token->save;

# queue the reset email
queue("mx_out")->add_msg(
  {
    to      => $user->email,
    from    => 'support@example.com',
    subject => 'Did you forget your password?',
    body    => template(
      'emails/password_reset',
      {
        username    => $user->username,
        token_url   => uri_for( '/confirm/' . $token->token ),
      },
    ),
  }
);

Once the reset email data goes into the queue, it waits for a separate worker process to retrieve the message and send it. At the moment, I'm using the Postmark email service to send my transactional emails. The worker is a pretty short Perl program that polls the message queue, retrieves new messages and hands them off via WWW::Postmark.

While I was at it...

When working on Dancer::Plugin::Queue, I realized what I was doing was similar to a lot of other plugins I had examined. In the case of D::P::Queue, I had the added step of creating a role to define the generic interface, but leaving that aside, a lot of plugins are doing this:

  1. Loading a class
  2. Loading some config options
  3. Creating a singleton

It's ridiculous to do that for any particular CPAN module you want to use within Dancer, so I wrote it generically as Dancer::Plugin::Adapter.

If I weren't committed to using WWW::Postmark via a message queue, this is how I could use it directly within a Dancer app with Dancer::Plugin::Adapter:

In the config.yml:

plugins:
  Adapter:
    postmark:
      class: WWW::Postmark
      options: POSTMARK_API_TEST

In the application:

use Dancer::Plugin::Adapter;

get '/send_email' => sub {
  eval {
    service("postmark")->send(
      from    => 'me@domain.tld',
      to      => 'you@domain.tld, them@domain.tld',
      subject => 'an email message',
      body    => "hi guys, what's up?"
    );
  };
  return $@ ? "Error: $@" : "Mail sent";
};

As long as the module needs only static data to initialize, Dancer::Plugin::Adapter does all the repetitive work. In my (not so) humble opinion, that makes a lot of useful CPAN modules trivial to use as singletons within Dancer. Enjoy!

Pulling it together

Once I could generate and send the reset email, I needed a handler to respond to someone clicking on the reset link. Most of the work was doing some rudimentary validation on the submitted token -- ensuring it was valid, not expired, and so on.

I decided to treat a password reset token as a one-shot, password-equivalent-login and existing-password-revocation. I also track that the login happened via token, so that the password change form and logic can skip requiring the current password.

Here is a simplified version of that code:

get '/confirm/:token' => sub {
  unless ( params->{token} =~ /^[a-zA-Z0-9_=-]{32}$/ ) {
    return template 'error' => { error => "Invalid token" };
  }

  my $token = schema("token")->find_token( params->{token} );
  if ( $token ) {
    $token->delete; # one-shot token, so delete from database
  }
  else {
    return template 'error' => { error => "Token not found" };
  }
  
  my $user = schema("user")->find_user( $token->user );
  unless ($user) {
    return template 'error' => { error => "Token has invalid user" };
  }

  if ( time() > $token->expiration ) {
    return template 'error' => { error => "Token has expired" };
  }

  if ( $token->type eq 'p' ) { # password reset
    session user  => $user->username; # treat as logged in
    session token => 1;               # note they arrived via token
    $user->scramble_password;
    $user->save;
    redirect '/change_password';
  }
  else {
    return template 'error' => { error => "Token not recognized" };
  }
};

You can see how the token confirmation logic is nearly completely generic. I can add additional token types as needed, just with different logic for each.

Summary

After another week of work, the application continues to take shape. Here's what I got done:

  • Wrote and shipped a message queue plugin system and MongoDB implmentation
  • Wrote and shipped a generic CPAN module adapter plugin
  • Added a password reset feature that generates reset tokens and emails users
  • Added a password reset token handler

It's still only the rough outline of an application, but Dancer feels less foreign and I'm about ready to get past basic user-account housekeeping and into the real feature set of the application.

This entry was posted in perl programming and tagged , , , . Bookmark the permalink. Both comments and trackbacks are currently closed.

11 Comments

  1. Posted December 18, 2012 at 6:41 pm | Permalink

    FWIW, using Email::Sender::Transport::Test makes testing email sending really easy. You just need a little voodoo in the test file like this:

    BEGIN { $ENV{EMAIL_SENDER_TRANSPORT} = 'Test' }

    Then later you can get the transport:

    my $transport = Email::Sender::Simple->default_transport();

    And call $transport->deliveries() to see what your app tried to send.

    • Posted December 18, 2012 at 6:59 pm | Permalink

      My (minor) objections to that are twofold. (1) it's not loosely coupled anymore and (2) it's not testing delivery, it's just testing the Email::Sender API.

      Testing that Email::Sender was called in certain circumstances is still valuable test coverage, but I can get that from inspecting the queue just as easily.

      On the delivery worker side, I could see using Email::Sender::Transport::Test, perhaps, if I wasn't using a web service for delivery.

  2. Alex
    Posted December 18, 2012 at 9:59 pm | Permalink

    I send mail from my webapp via Postfix. Postfix manages its own message queue and delivery from the app's point of view is instant.

    For testing, in the Postfix configuration I have set up special email account using the "pipe" transport. Any mail sent to that address is delivered to a script which simply dumps its input to a temp file, where I can inspect it.

    • Posted December 18, 2012 at 10:46 pm | Permalink

      That's another way to do it. I have specific reasons for not relying on a mail transport agent running on my web app server, however, so that wasn't an option for me.

  3. Michael Peters
    Posted December 19, 2012 at 11:04 am | Permalink

    I wonder if you'd have problems in the future scaling out your web/queue pair if your queue is also tied to your web framework (Dancer). In our current system we have so much happening in the queue that we have multiple machines just doing that (along with multiple web servers). Will your setup be problematic if you need to scale the queue out?

    • Posted December 19, 2012 at 12:15 pm | Permalink

      The queue isn't tied to the framework. It's maintained as a collection within MongoDB. For testing, that could be a local process. For production, it's a replicated cluster.

      The plugin just provides convenient access to an object to manage the queue. It's no different than using Amazon's SQS or another messaging service. (It wouldn't be hard to write an implementation for Dancer::Plugin::Queue that uses Amazon::SQS::Simple instead.)

  4. Martin
    Posted December 24, 2012 at 4:05 pm | Permalink

    Thanks for the great article. Loading up a queue from a web app is very useful.

  5. Posted December 25, 2012 at 6:35 am | Permalink

    > generate and store a random token
    A simple option would be to use Data::UUID, wouldn’t it?

    • Posted December 25, 2012 at 7:26 am | Permalink

      UUID's are not designed to be cryptographically secure. Data::UUID is pretty bad -- it doesn't generate random "version 4" UUID's at all, plus what randomness it does use gets its bits from C's rand() function. See my comparison of UUID generators for more details.

      Given that, I'd rather use a module like Data::Entropy, which is explicitly designed to provide the best randomness possible.

  6. Posted September 5, 2013 at 4:09 am | Permalink

    Thanks for your series on Dancer and friends. I'd love to know the purpose of

    $user->scramble_password;

    in your example code above. Does this invalidate the user's password such that when they click the token, they can no longer subsequently log in with their old password? If so, is this done purely for security?

    • Posted September 5, 2013 at 6:22 am | Permalink

      Yes. Though I haven't fully implemented it yet, the idea is that the session contains crypto_hash( $plaintext_password ) and the database contains crypto_hash( crypto_hash( $plaintext_password ) ). That way, a password change can invalidate all logged in sessions, even when a session is stored entirely browser-side. (And a database compromise can't be used to generate a valid session, either.)

One Trackback

© 2009-2014 David Golden All Rights Reserved