Using OpenResty in enterprise grade environments

Wild OpenResty bird

OpenResty® is a full-fledged web platform that integrates our enhanced version of the Nginx core, our enhanced version of LuaJIT, many carefully written Lua libraries, lots of high quality 3rd-party Nginx modules, and most of their external dependencies. It is designed to help developers easily build scalable web applications, web services, and dynamic web gateways.

By taking advantage of various well-designed Nginx modules (most of which are developed by the OpenResty team themselves), OpenResty® effectively turns the nginx server into a powerful web app server, in which the web developers can use the Lua programming language to script various existing nginx C modules and Lua modules and construct extremely high-performance web applications that are capable to handle 10K ~ 1000K+ connections in a single box.

It has been around 2 years now that I’ve been in charge of the development of OpenResty based microservice in Luminate Security.

OpenResty the main entry point to our SaaS service. It is serving thousands of requests per second and so far has proven to be a very reliable framework for developing smart reverse proxies.

Among the things OpenResty is good for are:

• Authentication — Validate cookies
• Authorisation — Fetch session information based on the cookie and enforce access to different locations based on the session data
• Smart load balancing — This is where nginx shines. It has a-lot of very cool load balancing capabilities. When combined with OpenResty you can extend it as much as you want and make it do a lot of heavy lifting for you.

I inherited this project from a colleague, and I must say the first time I looked at the source code of the project all the different lua and conf files I was quite overwhelmed. Now, 2 years later, I feel like I have made enough mistakes to have a good understanding of the best way to create maintainable and testable enterprise level applications using OpenResty.

In this blog post I will share some tips and methods that I use, hopefully you will find them useful.

Lets define what an enterprise grade software is

What is that enterprise software you speak of?

In my opinion enterprise grade software must be:

1. Written in an easy to understand and to maintain fashion — KISS. This is by far the most important point. It should be easy enough so no one calls you when you are on vacation.
2. Well organized — Utilizing code reuse and being broken down to small modules.
3. Well tested — This includes unit tests. There are different approaches to unit tests. Personally, I strive for high code coverage. Normally NGINX is a critical path on the way to your backend. Every single line of code must be tested. Also make sure you actually test the stuff that runs in production. This is especially true when writing “prove” tests (more on this later).

Tips and tricks of achieving enterprise grade in an OpenResty project

conf file hell

conf files are the main way to configure nginx and OpenResty.

Unfortunately, by nature, conf files are not very organized. It’s essentially a text file that defines various settings directives and blocks. It is very easy to make this a conf file a complete spaghetti mess. This is especially true if your project started as a POC. You hack around a configuration file and keep using it as your proxy grows.

Don’t make your conf files look like this

I recommend splitting it into logical blocks and utilizing the include directive as much as possible to minimize copy-pasting. This will help you a great deal when writing tests for the conf files.

Think of just like writing code. If you have the same thing twice and it’s not included (or a function), you need to move it to a separate file and use it though an include instead of copy-pasting.

For example, try to define common location directive configurations in a separate file and reuse them when applicable.

location /admincp {
 include verify-admin-session.conf;
}
location /private-docs {
 include verify-admin-session.conf;
}

Unfortunately, there is no way to define a function that would represent a dynamic configuration block. A simple way around that is to use a variable which value you set before including the conf file that affects the behavior of the code flow inside the included conf file.

Using this technique will also serve you during the testing phase.

location /admincp {
 set $minimal_user_level 3; 
 include verify-admin-session.conf;
}
location /private-docs {
 set $minimal_user_level 2;
 include verify-admin-session.conf;
}

Example of verify-admin-session.conf

access_by_lua_block {
  if ngx.var.current_user_level < ngx.var.minimal_user_level then
    ngx.exit(ngx.HTTP_FORBIDDEN)
  end
}

Using environment variables inside conf files

Unfortunately, it’s not possible to use environment variables inside nginx directives. Say you have an env var that defines the listen port of your nginx proxy, can’t do it using nginx conf syntax.

The way to solve it is by means of pre-processing the config file prior to nginx boot.

server {
    listen __SERVER_PORT__ default_server;
    ...
}

In the preprocess step you will use sed to replace __SERVER_PORT__ with the environment variable of your choice.

Here’s an example of a Dockerfile that utilizes the method described above:

FROM openresty/openresty:xenial
ARG SERVER_PORT
ENV SERVER_PORT ${SERVER_PORT:-8080}
COPY nginx.conf /usr/local/openresty/conf/nginx.conf
COPY nginx_loader.sh /nginx_loader.sh
CMD [ "sh", "-c", "/nginx_loader.sh" ]

nginx_loader.sh

# nginx_loader.sh
sed -i "s/__SERVER_PORT__/${SERVER_PORT}/g" /usr/local/openresty/conf/nginx.conf
# start nginx
/usr/local/openresty/bin/openresty -g "daemon off;"

Maintaining good LUA coding conventions

Firstly, for all your code I recommend using Luacheck. Its a static lua linter that enforces good coding practices for writing Lua code. Most IDEs that support Lua have it built in. We run it as a part of our build pipeline. A commit that contains Luacheck violations will fail building and be rejected.

The newer versions of OpenResty have some built in warnings printed out for bad lua code, so keep an eye out for those. They will mostly warn you about using outdated badly written lua modules.

By the way, my favorite IDE with Lua/OpenResty support is IntellJ IDEA with the excellent OpenResty Lua Support and the Lua by sylvanaar plugins.

Structuring LUA modules for easy unit testing

The basic structure of a lua module is something like this

-- mymodule.lua
local _M = {}
function _M.do_stuff() 
end
return _M

Your modules will probably be dependant upon other modules. I like to manage the includes by placing them under the _M object. This way when testing its easy to mock calls to those modules by replacing their functionality.

-- mymodule2.lua
local _M = {
    file_helper = require("file_helper"),
    session_checker = require("session_checker)
}
function _M.do_stuff() 
    if not session_checker.is_session_valid() then
        return nil
    end
    _M.file_helper.write_to_file("hello world")
    return true
end
return _M

I also try to always return values from functions (both for success and failures). This aids me later during unit tests.

Testing

I believe in achieving maximum code coverage through tests. I don’t like surprises. This is especially true for a critical path that a reverse proxy normally is.

I use the following tools for testing:

1. busted — is an excellent TDD testing framework for Lua. It is very easy to use and has intuitive syntax especially for those coming from other programming languages such as Ruby and JavaScript. It also comes with a mocking module called spy which is great. It works great with OpenResty as well with some minor adjustments.
2. test-nginx (prove) — another excellent tool that lets you test conf files. It has a funky syntax that takes some getting used to, but once you understand it, you will realise its a powerful tool for testing nginx configuration files. I use it all the time as a blackbox testing tool for conf files.

Example lua module test

local module = require("mymodule")
require 'busted.runner'()
-- a trick to work around resty's limitations of modifying these variables in test
setmetatable(ngx,{status=0, arg={}})
describe("mymodule", function()
    before_each(function()
        ngx.var = {}
        ngx.req = {}
        ngx.ctx = {}
ngx.log = spy.new(function() end)
        ngx.print = spy.new(function() end)
    end)
describe("#do_stuff", function()
        it("should return nil if session is invalid", function()
            -- mock external module function
            module.session_checker.is_session_valid = function()    return false end
            assert.are.equal(module.do_stuff(), nil)
        end)
    end)
end)

In order to run lua tests we use the same container configuration as our production image, just with a different entry point.

To run the above test inside of openresty/openresty:xenial docker container you need to run the following command resty -I /path/to/all/lua/files mymodule.test.lua

nginx prove tests

This is a little confusing test framework for someone who is not coming from a Perl background. It has an excellent integration with nginx and once you get the gist of it, it becomes a very powerful tool in your testing flow.

I won’t go into much detail about it, you will have to look at nginx tests by yourself in this repository. In essence, it allows you to test different nginx configuration blocks. Send requests and validate response body or log output.

If you followed my tips and split your conf files into logical blocks, your testing experience will be much better. Just make sure not to copy paste blocks from your nginx file to the test file. This will eventually make them desynchronized and your test files won’t be actually testing what is running in production.

Afterwords

Hope this blog post was helpful to you. If you have any questions feel free to reach out via comments.