Here's the story: I'm a recent convert to Dist::Zilla. I really like it -- if I had more time I'd write a longer post about how easy it makes your life as a developer.
The problem is that it can make it too easy. I was writing a Dist::Zilla plugin to check "xt" tests before release. Unfortunately, the @Classic plugin that I added to a test includes the UploadToCPAN plugin -- which, if you have a .pause file with your credentials, will helpfully upload modules straight to CPAN.
You can probably see where this is going. Yes, I ran my tests and promptly uploaded a test distribution called "Foo-1.23" to CPAN. Thankfully, the "Foo" namespace is already claimed, so it's an unauthorized distribution and doesn't get indexed. I've already scheduled it for deletion, so hopefully, by the time you read this, it will just be a bad memory.
But, my mistake aside, Dist::Zilla rocks. If you develop for CPAN, you should really give it a try.
Just be very careful when you type 'dzil release'.