plack + starman + Inline::C = bloody stinking fast

My image management system for work is really cranking. It is now hosting 22 million images on a single server. Mind you it’s a beefy server, but that’s still a lot of images.  What still surprises me is the performance of plack + starman + inline::C.

Let’s say that you need to write a piece of software that stores millions upon millions of images.  Well, you can do that pretty easily with a standard web service and back it with something like Amazon::S3. But, in my experience, is that rarely enough. Usually you need to be able to modify the images somehow. People want a PNG when you are storing a JPG, people want a thumbnail, they want it cropped, they want it resized, etc…

You might reach for Image::Magick or some standard tool. Please don’t, that is very very slow.  How slow? Let me give you an example. For a sample image, just a nice normal book cover, this is how long it takes for convert on the command line:

-bash-3.2$ time convert 1156067855.jpg -resize 50% output.jpeg

real 0m0.383s
user 0m0.348s
sys 0m0.032s

That’s not too bad, but that’s quite a while.  Convert (one of the Image magick tools) is a very generalized tool. It’s not just a resizer, and it’s going to do lots of stuff to get ready to resize, when really, you just need a freaking resize. This is where you can use a specialzied image processing library to get bloody stinking fast performance. I use the awesome library leptonica.  So, using some of the sample code (prog/scaletest1.c) we can see that to do the same resize takes a considerably smaller amount of time:

-bash-3.2$ time ./scaletest1 1156067855.jpg .5 .5 output.jpeg

real 0m0.138s
user 0m0.116s
sys 0m0.022s

So, how can we use this in Perl?  Well, through the awesomeness that is the Inline::C module. Using it, allows us to throw together some nice code that lets you use leptonica’s low level image maniuplation primitives right from a Moose object, and let’s you take advantage of all the performance you could ever want:

So how fast is it? Well, here’s an example of using curl to retrieve a resized image from a plack/starman web service:

-bash-3.2$ time curl -O https://SECRET_HOST/2011-04-01/Images/front_cover/x200/sku/1156067855.jpg
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5774 100 5774 0 0 25600 0 –:–:– –:–:– –:–:– 5638k

real 0m0.233s
user 0m0.033s
sys 0m0.004s

So the entire web stack added 1/10th of a second. Not bad at all.  This web service, hosting 22+ million images, running on a single server, is serving 10s of millions of requests per day, every single day, in production for over a year, with lots of 9’s of reliability. So yeah, using a little bit of creativity and some fun perl modules, you can make some ridiculously fast performing services.

This entry was posted in pack, perl, tech. Bookmark the permalink.

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s