Nginx image processing server with OpenResty and Lua
Today I'll be showing you how to create a fast on the fly image processing server. The whole system can be created in less than 100 lines of code.
We'll be using OpenResty, an enhanced distribution of Nginx. We'll also need to write a little bit of Lua to get all the functionality we want. Lastly, we'll be using this Lua ImageMagick binding. If you're not familiar with any of these that’s OK, I'll be showing you how to get everything running from scratch.
The code
Because the entire system isn’t that many lines of code I'll show everything first and you can read on if you want to learn more about how it works.
You can also find the code in this Git repository.
nginx.conf
# These three directives should be tweaked for production
error_log stderr notice;
daemon off;
events { }
http {
include /usr/local/openresty/nginx/conf/mime.types;
server {
listen 80;
location @image_server {
content_by_lua_file "serve_image.lua";
}
location ~ ^/images/(?<sig>[^/]+)/(?<size>[^/]+)/(?<path>.*\.(?<ext>[a-z_]*))$ {
root cache;
set_md5 $digest "$size/$path";
try_files /$digest.$ext @image_server;
}
}
}
serve_image.lua
local sig, size, path, ext =
ngx.var.sig, ngx.var.size, ngx.var.path, ngx.var.ext
local secret = "hello_world" -- signature secret key
local images_dir = "images/" -- where images come from
local cache_dir = "cache/" -- where images are cached
local function return_not_found(msg)
ngx.status = ngx.HTTP_NOT_FOUND
ngx.header["Content-type"] = "text/html"
ngx.say(msg or "not found")
ngx.exit(0)
end
local function calculate_signature(str)
return ngx.encode_base64(ngx.hmac_sha1(secret, str))
:gsub("[+/=]", {["+"] = "-", ["/"] = "_", ["="] = ","})
:sub(1,12)
end
if calculate_signature(size .. "/" .. path) ~= sig then
return_not_found("invalid signature")
end
local source_fname = images_dir .. path
-- make sure the file exists
local file = io.open(source_fname)
if not file then
return_not_found()
end
file:close()
local dest_fname = cache_dir .. ngx.md5(size .. "/" .. path) .. "." .. ext
-- resize the image
local magick = require("magick")
magick.thumb(source_fname, size, dest_fname)
ngx.exec(ngx.var.request_uri)
What’s an image processing server?
An image processing server is a web application that is concerned with taking an image path along with a set of manipulation instructions and returning the manipulated image.
A good example is user avatar images. If you let your users upload their own images then the images probably come in a handful of different sizes and formats. When displaying the image you might have it on many different pages and require many different sizes. In order to avoid resizing up front you can use an image processing server to get the image sizes you want on demand just by requesting a special URL.
Additionally, if the URL is requested multiple times the resized image should be cached so it can be returned to the user instantly.
Initial thoughts
The first step to this project is to design a URL structure. In this tutorial I'll use the following format:
/images/SIGNATURE/SIZE/PATH
Given an image, leafo.jpg
, and a desired size, 100x100
, we might request
the URL:
/images/abcd123/100x100/leafo.png
You'll notice I've included a section for a signature in the URL. We'll be using some basic cryptography to ensure a stranger can’t request images of any size. This is an important thing to consider as image processing can take a lot of CPU power. If someone were to write a malicious script that iterates over a large quantity of image sizes they could max out your CPU in an attempt to perform a Denial-of-service attack.
The signature is the result of a cryptographic function run on a portion of the URL (the size and path) and a secret key. To verify that the URL of the resized image is valid you need to perform a simple assertion:
assert(calculate_signature("100x100/leafo.png") == "abcd123")
Lastly, it’s worth mentioning image sources and caching. For simplicity the images will be loaded directly from local disk. Although it’s perfectly possible to load them from external places, like S3 or other URLs, it wont be covered in this tutorial.
Modified images will be cached to disk, and cache expiration wont be covered. This is perfectly fine for most cases.
Installation requirements
You can download the latest version of OpenResty from here: http://openresty.org/#Download
Installation is simple, after extracting the archive just run
$ ./configure --with-luajit
$ make
$ make install
You can find more detailed installation instructions on the official site.
On my system this places OpenResty at /usr/local/openresty/
, so when it comes
time to run Nginx we'll be running: /usr/local/openresty/nginx/sbin/nginx
OpenResty comes with Lua, so the last component is the ImageMagick binding.
If you're familiar with Lua already, you can use LuaRocks to do the install:
luarocks install magick
Nginx configuration
Our Nginx configuration is concerned with serving cached images or executing our Lua script for un-cached images. It’s listed in full at the top of the post, but here I'll step all the pieces explain their roles.
error_log stderr notice;
daemon off;
events { }
These are basic settings that I use for doing Nginx configuration development. I leave most settings default but I disable the daemon and make sure information is printed to standard out. When deploying your Nginx application you'll want to spend some time adding some additional directives to make sure it can run as fast as possible.
http {
include /usr/local/openresty/nginx/conf/mime.types;
The http
block defines our HTTP settings, the only thing to be done here is
include the mime.types
file. This is a file that comes with Nginx. When Nginx
serves a file from disk is uses the mime types configuration file to correctly
set the Content-type
header.
server {
listen 80;
The server block configures our server. For illustrative purposes I've
bound to port 80 (even though port 80 is the default). In the server block we
declare location
blocks, which are destinations for requests.
location @image_server {
content_by_lua_file "serve_image.lua";
}
The first location defined is called @image_server
. The @
signifies a named
location. A named location can not be externally accessed by any URL, it can
only be called upon by other locations. We'll execute this location when
the file we want doesn’t exist in the cache.
location ~ ^/images/(?<sig>[^/]+)/(?<size>[^/]+)/(?<path>.*\.(?<ext>[a-z_]*))$ {
This location matches our image URLs. The regular expression here is quite big so let’s step through it.
Following the word location
is ~
, this instructs Nginx to perform case
sensitive regular expression matching against the incoming request path.
As I mentioned before our URL structure is:
/images/SIGNATURE/SIZE/PATH
And the regular expression is:
^/images/(?<sig>[^/]+)/(?<size>[^/]+)/(?<path>.*\.(?<ext>[a-z_]*))$
^/images/
will match the left hand side of the path. Following that is a
series of named capture groups that match each part of the request path
separated by /
. Although it would be possible to shorten the regular
expression by avoiding the named captures, I decided to use them because
they'll help with the readability of our script.
$(?<sig>[^/]+)
is a named capture group. It captures as many characters as it
can that aren’t /
and assigns it to the name sig
. The size
named capture
works the same.
(?<path>.*\.(?<ext>[a-z_]*))$
captures the path of our image, which is
everything else in the URL. It contains an inner named capture group to extract
the extension of the image.
Named capture groups are interesting in Nginx becuase their results can be used
directly as variables. For example, if we were using the echo
module, we
could just dump the extension of the request using echo $ext;
in the Nginx
configuration.
Now the body of the location:
root cache;
set_md5 $digest "$size/$path";
try_files /$digest.$ext @image_server;
This will only execute if the regular expression from above matches.
root cache;
This sets the directory of the cache where files will be searched using
try_files
.
set_md5 $digest "$size/$path";
This calculates a hash of the image path and size. The MD5 digest is used as the name of the file in the cache.
try_files /$digest.$ext @image_server;
Finally we use try_files
to either load the existing image from the cache or
pass the request off to the @image_server
location we defined earlier.
Lua script
Our Lua script runs when the @image_server
location is executed. Its job is
to verify the signature, ensure the image exists, then resize and serve the
image.
The script should be saved in the current directory of Nginx, normally next to
the nginx.conf
. I've also called the script serve_image.lua
. It must match
what is referenced in the configuration.
local sig, size, path, ext =
ngx.var.sig, ngx.var.size, ngx.var.path, ngx.var.ext
local secret = "hello_world" -- signature secret key
local images_dir = "images/" -- where images come from
local cache_dir = "cache/" -- where images are cached
The first step is to set some variables that will be used. The named capture group variables are pulled in as local variables to make their access more convenient.
local function return_not_found(msg)
ngx.status = ngx.HTTP_NOT_FOUND
ngx.header["Content-type"] = "text/html"
ngx.say(msg or "not found")
ngx.exit(0)
end
If an invalid URL is accessed the server should gracefully show a 404 message.
The function return_not_found
sets the correct status code, prints a
message and exits.
local function calculate_signature(str)
return ngx.encode_base64(ngx.hmac_sha1(secret, str))
:gsub("[+/=]", {["+"] = "-", ["/"] = "_", ["="] = ","})
:sub(1,12)
end
This is the function that signs our URL using the secret key. I've opted to take the first 12 characters of the base64 encoded result of the HMAC-SHA1.
Additionally I use gsub
to translate characters that have special meanings in
URLs to avoid any potential URL encoding issues.
Now that everything has been declared we can continue on with the logic of the file.
if calculate_signature(size .. "/" .. path) ~= sig then
return_not_found("invalid signature")
end
Here we verify that the signature is correct. If it’s not what we expect based on the rest of the URL then a 404 is returned.
local source_fname = images_dir .. path
-- make sure the file exists
local file = io.open(source_fname)
if not file then
return_not_found()
end
file:close()
These lines of check for the existence the file. In Lua we can check if a file is readable by trying to open it. If the file can’t be opened we abort. We don’t need to read the file here so we close.
local dest_fname = cache_dir .. ngx.md5(size .. "/" .. path) .. "." .. ext
-- resize the image
local magick = require("magick")
magick.thumb(source_fname, size, dest_fname)
dest_fname
is set to the same hashed name we searched for in our Nginx
configuration. The file can be picked up automatically by Nginx try_files
on any subsequent requests.
Now that the request has been verified it’s time to do the resize. We pass the
size string directly into Magick’s thumb
function. This gives us nice syntax
for various types of resizes and crops, like 100x100
for a resize, or
10x10+5+5
for a crop.
ngx.exec(ngx.var.request_uri)
Now that the image is written we are ready to display it to the browser. Here
I've trigger a request to the current location, request_uri
. Normally this
would trigger a loop error but, because we've written the cached file,
try_files
will return the file and skip the Lua script.
Running the server
Now were ready to try it out. We'll run Nginx isolated in its own directory.
This directory should start out with nginx.conf
, serve_image.lua
, and an
images
directory.
Before starting the server you should place some images in the images
directory.
You should be inside of the directory where we want the server to run before running the following commands.
Create the cache directory:
$ mkdir cache
Initialize some files our configuration requires for starting:
$ mkdir logs
$ touch logs/error.log
Now start the server:
$ /usr/local/openresty/nginx/sbin/nginx -p "$(pwd)" -c "nginx.conf"
Assuming the server has started we can now access the server. For example, if
you have an image leafo.jpg
you might resize it by going to the following
URL: http://localhost/images/LMzEhc_nPYwX/80x80/leafo.jpg.
Final notes
That’s all there is to it. With some minor tweaks to the initialization in
nginx.conf
your server is ready to go live.
There are a couple additional things you could also do:
If you already have an Nginx installation you could integrate this code into it so you don’t have to run separate Nginx processes.
If you are using the image server with another web application you'll need to
write the calculate_signature
function inside of your application so you can
generate valid URLs.
If you're concerned about the cache taking up too much space with unused image sizes you could look into creating a system that deletes unused cached entries.
Thanks for reading, leave a comment if you any suggestions or are confused about anything.
Related projects
A Lua module that provides LuaJIT FFI to MagickWand, the image library included in Image Magick.