One World Trade Center

Create image server with nginx + lua (Openresty) + graphicsmagick (Part I)

Update: Part two of this tutorial is now published.

Ever since I’ve read about Jetpack Photon and its features, I wanted to make something similar for my personal use and with no restrictions that it has (i.e., must have Jetpack installed and connected to WordPress.com). After a bit of searching around and stumbling on this article, it was decided for me, I must create my own. Since I did not know anything about Lua, and only have seen it used few times (Lightroom plugins, game plugins, nginx module ngx_lua), I needed to quickly learn some basics and start experimenting with it. It was not as hard of a language as it might be seen initially, plus I had a goal and drive to learn it quickly.

Few weekends and multiple rewrites (at least 5 or 6 I would guess), I have achieved level of quality that I would deem acceptable for public use and sharing. It is not as polished as it should be but it is trade-off between time and effort, and I had to draw the line at some point. If there will be an interest, it would be possible to place it on GitHub and take it from there, do let me know.

In this part I will go over preparation of the server and all of the necessary software that will be required to run this service. In the second part, I will share NGINX configuration and Lua code for performing actual image resizing and optimization.

Goals for implementation of image processing server

  • Optimize images for minimum download size and highest quality, have ability to apply further “google pagespeed module” optimizations
  • Perform resizing on-the-fly and cache resized and optimised images indefinitely
  • Have ability to sign and secure URLs, if needed; optionally allow for URL expiration
  • Allow for extensibility and flexibility to add functionality in the future
  • Lightweight, able to serve thousands of images per second, once cached
  • Full integration with nginx (my web server of choice), no requirement for fcgi or external processing
  • No restrictions on usage and terms of use
  • And most of all, have a lot of fun learning and experimenting

Software requirements (possibly incomplete list)

  • Linux base server, Centos 7 in my case (link)
  • VIM, the best editor on earth (nuff said…)
  • Development tools, i.e., gcc, make, cmake; you might need to install cmake separately
  • GraphicsMagick libraries and its dev headers (link)
  • Lua and its dev headers (link), small and versatile programming language
  • Openresty source (link), fast application server based on nginx and lua
  • Luarocks source (link), package manager for Lua modules

Preparing software

For this specific server I used Centos 7 with EPEL repositories (not sure if needed but guess that you will). I am sure it would be easily doable on any other modern distro, I just happened to have Centos 7 running and ready to be used. Here is (possibly incomplete) list of commands to get your Centos 7 installation ready. If in the process of installation, configuration or compilation you will find missing dependencies, search for them with yum search <dependency name>, to find the package that you can install via sudo yum install . Likely, you will need *-devel part of the package, which will pull main package as dependencies when installing. Login to your server, make sure you have root privileges (can check with sudo -l command), and execute the following in order. You will get prompted about dependencies, etc.

 sudo yum install epel-release
 sudo yum group install "Development Tools"
 sudo yum install cmake
 sudo yum install GraphicsMagick-devel
 sudo yum install lua
 sudo yum install pcre-devel
 sudo yum install openssl-devel
 sudo yum install gd-devel
 sudo yum install curl-devel
 sudo yum install nasm
 sudo yum install vim-enhanced

This should be sufficient to compile and configure server to our needs, and as mentioned above, it is not too hard to resolve dependencies (as already mentioned in the paragraph above.

Download necessary components

Before we start, lets prepare a directory where you can keep the full collection of downloaded sources as well as necessary “configure” scripts, will be useful in the future to perform update of the software. I like to have a “software” folder on one of the partitions with a lot of spare disk-space. My directory tree would look something like this:

 /data                   # Partition with plenty of spare capacity
 /data/software
 /data/software/web      # Web software, will use this one
 /data/software/dns      # DNS software
 /data/software/drivers  # Drivers, i.e., NVIDIA
 /data/software/graphics # Image converters, optimizer, also used
 /data/software/random   # Anything else

Once you have decided on the directory structure and ready lets login to your server via ssh and create the structure. I will be using above mentioned structure as an example:

 sudo mkdir -p /data/software/web #"-p" to make hierarchy
 sudo mkdir /data/software/graphics
 cd /data/software
 sudo chown -R $USER:$USER . #That is a dot in the end
 cd web

Openresty

This is the main component that will be the core of the service, thanks to kind people for creating this all-in-one bundle of goodness.

 curl -L -O http://openresty.org/download/ngx_openresty-1.7.10.1.tar.gz
 tar zxvf ngx_openresty-1.7.10.1.tar.gz
 cd ngx_openresty-1.7.10.1

Do check for the latest version of the openresty software by going to http://openresty.org and navigating to the downloads section. Also not a bad idea to read through the changelog, to know what has been modified and improved.

Next it would be good to create a shell script to execute configure script, will be very helpful for future updates.

 vim config.sh

If you are not comfortable with VIM, feel free to use any other editor of your choice, but I promise it is really good to know how to use it and get comfortable with its commands.

When you just open file in vim, you should be in a command mode, each key-stroke has a meaning and you should not just start typing. Here is a simple recapture of the basic ones:

Quick VIM reference

There are thousands more but I think I will stop here, also check http://www.catswhocode.com/blog/130-essential-vim-commands for more commands, or just search on google.

Once you opened a new file “config.sh” and placed you vim into paste mode (:set paste), entered insert mode (i), copy/paste the following:

#!/bin/sh
./configure --prefix=/opt/openresty \
            --sbin-path=/opt/openresty/sbin/nginx \
            --conf-path=/opt/openresty/etc/nginx.conf \
            --error-log-path=/opt/openresty/log/error.log \
            --http-log-path=/opt/openresty/log/access.log \
            --pid-path=/opt/openresty/run/nginx.pid \
            --lock-path=/opt/openresty/run/nginx.lock \
            --http-client-body-temp-path=/opt/openresty/cache/client_temp \
            --http-proxy-temp-path=/opt/openresty/cache/proxy_temp \
            --http-fastcgi-temp-path=/opt/openresty/cache/fastcgi_temp \
            --http-uwsgi-temp-path=/opt/openresty/cache/uwsgi_temp \
            --http-scgi-temp-path=/opt/openresty/cache/scgi_temp \
            --user=openresty \
            --group=openresty \
            --with-http_addition_module \
            --with-http_ssl_module \
            --with-http_realip_module \
            --with-http_addition_module \
            --with-http_sub_module \
            --with-http_dav_module \
            --with-http_flv_module \
            --with-http_mp4_module \
            --with-http_gunzip_module \
            --with-http_gzip_static_module \
            --with-http_random_index_module \
            --with-http_secure_link_module \
            --with-http_stub_status_module \
            --with-http_auth_request_module \
            --with-http_image_filter_module \
            --with-pcre-jit \
            --with-file-aio \
            --with-ipv6 \
            --with-http_spdy_module \
            --with-luajit \
            --with-lua51=/usr \
            --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' \
            -j2
make -j2
sudo make install
sudo mkdir /opt/openresty/cache

Exit into a command mode (Esc) and write file to disk while quitting (:wq).

Here I assume that you are fine with placing your openresty server in /opt directory, place it anywhere you like and just modify above tree. Also, I assume that user and group “openresty” exist, or just create new system account for openresty like so:

 groupadd -r openresty
 useradd -r -g openresty -s /sbin/nologin -d /opt/openresty -c "openresty user" openresty

Once all of the above is done, you can start configuring and compiling your openresty installation:

 chmod +x config.sh
 ./config.sh

Note any errors and act accordingly. You still might be missing some dependencies that I have forgotten to mention above.

Luarocks

Next we will install luarocks, to allow for easy Lua modules installation and dependencies management. Check for the latest version of luarocks, instructions are for the current version at this time.

 cd /data/software/web
 curl -L -O http://keplerproject.github.io/luarocks/releases/luarocks-2.2.0.tar.gz
 tar -xzvf luarocks-2.2.0.tar.gz
 cd luarocks-2.2.0/
 ./configure --prefix=/opt/openresty/luajit \
               --with-lua=/opt/openresty/luajit/ \
               --lua-suffix=jit-2.1.0-alpha \
               --with-lua-include=/opt/openresty/luajit/include/luajit-2.1
 make && sudo make install && sudo make bootstrap
#To check that luarocks are installed and executable
 /opt/openresty/luajit/bin/luarocks --version
#Should see similar output
/opt/openresty/luajit/lib/luarocks/rocks/luarocks/2.2.0-1/bin/luarocks 2.2.0
LuaRocks main command-line interface

This should be sufficient for us to be able to install all of the necessary Lua modules in the next part.

Image compressors and optimizer software

As a final step, we will need to download, configure, compile and install image optimization software: Gifsicle, Mozjpeg and Leanify (which could probably replace both of Gifsicle and Mozjpeg, plus more).

Gifsicle

I found Gifsicle to be one of the best optimizers for gifs that I could find for linux CLI. It is possible that there are better tools that produce smaller sized gifs but differences are marginal at best. It is also fairly fast at what it does and I chose to use it for this purpose.

I feel that we can be more experimental in this project and will use the latest available source on git https://github.com/kohler/gifsicle.git. I assume again that you have git installed on your system, if not: sudo yum install git

 cd /data/software/graphics
 git clone -b master https://github.com/kohler/gifsicle.git gifsicle-master
 cd gifsicle-master
 ./bootstrap.sh #watch for any missing tools, i.e., autoconf; you might need to install anything missing
 ./configure --prefix=/opt/openresty \
               --disable-gifview \
               --disable-gifdiff
 make && sudo make install
 /opt/openresty/bin/gifsicle --version #Check if the binary is in place and working
LCDF Gifsicle 1.87
Copyright (C) 1997-2014 Eddie Kohler
This is free software; see the source for copying conditions.
There is NO warranty, not even for merchantability or fitness for a
particular purpose.

And that is all for gif optimizer, lets move to the next one (arguably more used and important), JPEG optimizer.

Mozjpeg

This one was a bit of a discovery for me on my quest to find “the best” jpeg optimizer, initially I was going with jpegoptim, and only later I have discovered that Mozilla created a better (produces smaller jpegs) optimizer that we will be using for this project.

Lets start from download of the latest release from GitHub (3.0 at the time of writing): https://github.com/mozilla/mozjpeg/releases

 cd /data/software/graphics
 curl -O -L https://github.com/mozilla/mozjpeg/releases/download/v3.0/mozjpeg-3.0-release-source.tar.gz
 tar zxvf mozjpeg-3.0-release-source.tar.gz
 cd mozjpeg
 ./configure --prefix=/opt/openresty #making installation base in openresty root directory
 make -j$(($(nproc)+1)) #$(($(nproc)+1)) should give you number of processors + 1
 ./jpegtran -h #just to check that executable is compiled and works
 sudo make install

And that should be it for installation of mozjpeg optimizer.

Leanify (optional and will not be used directly at this time)

Finally, only after I was finished (and satisfied) with additional optimizers and their results, I have discovered this tool that combines all of the above and more. I have not integrated it into this project but feel free to experiment and replace above optimizers with this one. Would be really nice if you share your code as well.

First thing we do is clone Leanify’s git repository: https://github.com/JayXon/Leanify.git

 cd /data/software/graphics
 git clone -b master https://github.com/JayXon/Leanify.git leanify-master
 cd leanify-master
 make -j$(($(nproc)+1))
 ./leanify #again, just checking that compilation of executable was ok
 sudo cp leanify /opt/openresty/bin/ #I believe it is statically compiled binary ball

And that is all folks, we have all of the necessary software compiled and ready for implementation of the image processing server.

Summary

In this part we have covered installation of all the necessary software that will be used for creation of imaging server. I attempted to give sufficient level of details that will allow you complete this part with no significant issues, please leave feedback below if you find something wrong or missing, and will make sure to fix it. In the next part, which I am planning to finish and publish shortly, I will go over installation of Lua software modules, NGINX configuration, Lua code that will be embedded in the NGINX and used for processing of images.

Thank you for reading and getting this far, job well done!

Disclaimer
Copyright (c) 2015, Ian Matyssik
All rights reserved.

THIS TUTORIAL IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS TUTORIAL, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Ian Matyssik is geek to the bone with love for art and nature. In a free time likes walking with his dog and taking pictures.
%d bloggers like this: