Production environment for web deployment - part 1: Nginx

This post focuses at the Nginx web server and is part one of a series explaining how to setup a production environment in Linux for web deployment. I will be using Debian 7.0 64-bit but same principles apply at the new stable Debian 7.8 or any other Linux distribution.

The following software is used in the setup:


Edited on 2015-04-24: Click here for a research paper written in the Dutch language comparing Nginx's performance between different numbers of worker_processes, multi_accept on or off, and with epoll or select() and poll().

Nginx or Apache

nginx - pronounced as engine x - is an open-source web server and reverse proxy written by the Russian software engineer Igor Sysoey. Since its first public release in 2004 by Nginx, Inc, effort has been spent at current important matters like performance, concurrency, and memory consumption. These design goals have been tackled early during the development what eventually turned out in mature software known as fast, trusted, and light-weight.

Nginx is event-driven single-threaded by architecture with as goal to reach higher performance than process- or thread-based architectures. Its architecture is asynchronous, non-blocking, and single-threading with support of several event notification mechanisms implemented in modern operating systems.

Apache however implements a traditional thread-per-connection architecture, where as the name already suggests every connection is handled in its own thread. Because of the use of blocking in- and output operations - blocking the execution path during requests and responses - it uses multiple threads for concurrency.

One advantage of using an event-driven single-threaded architecture is that available memory and computing power is used more efficiently as every thread requires its own thread control block, user- and kernel stack. For matters like memory consumption and CPU utilization this can be detrimental in multi-threaded high-concurrency applications.

Personally I prefer Nginx over Apache because of its performance and capability to cache and operate as reverse proxy or load balancer. Besides I find it easier to configure.

If you are interested in thread-based and event-driven server architectures or programming, the following links give well-explained information about the topic:

  1. Concurrent Programming for Scalable Web Architectures - Chapter 4.2: Server Architectures
  2. Concurrent Programming for Scalable Web Architectures - Chapter 4.3: The Case of Threads vs. Events

Installing and configuring Nginx

[email protected]:/$ apt-get update  
[email protected]:/$ apt-get install nginx  

Worker processes

Nginx's tasks can be split over multiple worker processes to operate at multiple cores reaching higher CPU utilization. In the most optimal condition every worker process runs at its own physical core.

Finding the number of cores

[email protected]:/$ grep --count processor /proc/cpuinfo  

Add the worker_processes directive to the top of the file, make sure it's outside any blocks.

worker_processes 2;  

Connection handling

For the best performance the multi_accept directive should be enabled in combination with usage of the epoll I/O event notification mechanism.

Enabling multi_accept causes Nginx to attempt to immediately accept as many sockets as soon as possible.

To use select(), the developer needs to initialize and fill up several structures with the descriptors and the events to monitor, and then call select(). These descriptors only exist during the execution of a single select() call and are being overwritten by the kernel upon return, so they have to be recreated all the time. This can result in a lot of overhead, particularly in environments with large numbers of connections and relatively few new events occurring. Also with the introduction of HTTP 1.1 and persistent connections its very common for connections to kept open for a specific duration of time. The list of events to be scanned can grow gigantic harming the whole performance of the system. This problem is called the c10k problem.

Epoll - Linux's variant of Unix's kqueue - is a scalable event notification mechanism to replace the select() and poll() kernel calls (which are still very common). With epoll only one structure has to be prepared enclosing all file descriptors to be monitored. This structure is shared persistently with the user-space and only contains descriptors that have events available.

The worker_connections directive sets the number of connections a worker process can maintain simultaneously. The maximum number of clients is the multiplication of the worker_connections and the worker_processes value, but can not exceed the system or worker's maximum number of allowed descriptors.

The worker's descriptor limit can be set by using the worker_rlimit_nofile directive. Make sure to set it high enough or else you risk getting the following error: "24: Too many open files".


worker_rlimit_nofile 2048;

events {  
    use epoll;
    multi_accept on;
    worker_connections 1024;

Server tokens

Avoid security by obscurity

Its a good practice for security reasons to hide server tokens that contain the Nginx version number. This is especially the case when running an outdated version of Nginx that might be vulnerable to exploitation.


http {  
    server_tokens off;

Tcp_nodelay, tcp_nopush, and sendfile

Enabling the tcp_nodelay directive enforces sockets to be created with the TCP_NODELAY option that disables the Nagle buffering algorithm implemented by the kernel. Its goal is to increase bandwidth efficiency at the expense of latency by combining (with delays up too 500ms) small outgoing messages. In general it should only be used in applications:

  • that don't write small buffers of a couple bytes, because with this enabled each small buffer will be transmitted in a single packet resulting in poor performance;
  • or that write frequent small bursts of information without requiring an immediate response, where timely delivery of data is required.

Those points might seem contradictory but today every possible reduction in latency is preferable, which benefits the bandwidth because data in the network pipe will stay there for a shorter duration of time. But, if we all do this for the advantage of bandwidth, doesn't it mean we should actually be using the algorithm because wasn't that what is was intentionally designed for? The thing is we cannot make sure software won't write small buffers, if it does and its the only thing to be written it can delay up to 500ms, which is far worse for performance perspective.

Because we would like to avoid small buffers for reasons explained above we should attempt Nginx to send its HTTP response headers in one packet. This can be done by enabling the tcp_nopush directive that enforces sockets to be created with the TCP_NOPUSH option causing the kernel not to send out partial frames.

The sendfile directive enables the use of the sendfile() system call which is used for transferring data from one to another descriptor. All done inside the kernel so it only requires one context-switch instead of multiple for reading the source descriptor, handling the buffer, and writing it to the destination descriptor. Enabling this option when serving static files can drastically increase performance because file content can be transferred directly into the socket.


http {  
    sendfile on;
    tcp_nodelay on;
    tcp_nopush on;

Buffer sizes

The following buffer sizes describe the maximum size of data that can be stored in memory (a buffer). It is actually far less important than other articles are trying to make you believe. We just have to make sure the buffers are large enough to store its data, if not the buffers will be written to a disk file so the full-length data can be used. Disk operations are slower than memory so we would like to avoid that.

The client_header_buffer_size directive sets the buffer size for reading the request header. For most requests a size of 1k is large enough. When it is common for requests to be larger, for example in web applications that use a lot of cookies, it is best to increase its size. If this only happens accidentally we should keep it at 1k, but always make sure it never exceeds the value of the large_client_header_buffers directive or else a 414 (Request-URI Too Large) error is returned. This directives sets the buffer size for large requests, and how many of those buffers may be allocated for each request.

The buffer size for the request body, containing form submissions and file uploads, can be set with the client_body_buffer_size directive. If you expect a lot of uploads a higher value comes out in favor. The client_max_body_size directive sets the maximum SIZE of the request body. If the request exceeds this size a 413 (Request Entity Too Large) error is returned. So if you plan on receiving a lot of uploads, use appropriate values.


http {  
    client_header_buffer_size 1k;
    large_client_header_buffers 2 1k;
    client_body_buffer_size 8k;

    # client_max_body_size 8m;


Compression can be used to significantly reduce the size of transmitted data, requiring less bandwidth. Because compression happens at run time by the CPU it can also add considerable overhead, which can negatively affect performance. Files that are already compressed by default should be avoided because compressing with a level that offers the wanted efficiency will probably not decrease its file size any further.

Acceptable values as compression level range from one till nine, where lower values are faster by offering less compression. Most compression is achieved at level one and two, so using a higher value wouldn't benefit its costs.


http {  
    gzip on;
    gzip_http_version 1.0;
    gzip_disable      "MSIE [1-6]\.(?!.*SV1)";

    gzip_comp_level 2;
    gzip_min_length 2024;
    gzip_proxied expired no-cache no-store private auth;
    gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/javascript application/xml;

File structure

Nginx's configuration files are placed in /etc/nginx. I prefer to have two subdirectories called sites-available and sites-enabled. The sites-available directory will contain all server configurations, one for each server. The sites-enabled directory should contain symlinks to the enabled server configurations and is to be included in the nginx.conf file. The configurations in the sites-enabled directory are loaded by Nginx. This structure makes it fairly easy to disable a server without removing its configuration.

[email protected]:/$ cd /etc/nginx  
[email protected]:/etc/nginx$ mkdir conf.d sites-enabled sites-available  

Append the following include directives to the bottom of the file, above the closing bracket from the http block. Also make sure the nginx.conf file doesn't contain any server configuration blocks, if it does remove them. We define those later in a separate file inside the sites-available directory.

http {  
    include /etc/nginx/conf.d/*;
    include /etc/nginx/sites-enabled/*;

Server configuration

Name-based Virtual Hosts and the default server

As you probably know only program is able to listen on a port at the same, and yet most servers are able to host multiple websites on the same server, IP address, and port. This concept is called Name-based Virtual Hosting and works by reading the Host header field which is required in all HTTP/1.1 requests. This value is then compared to the server names set in the configurations and the request is processed accordingly.

So by defining multiple server configurations with each a unique server name we can host multiple websites at the same server. When its value does not match any server name, or the request does not contain this header field at all, then Nginx will route the request to the default server for this port. The default server can be explicitly set by using the default_server parameter in a listen directive.

For more information see "How nginx processes a request" at

It returns a 444 status code because I prefer not to have any website visible when the server is being accessed directly by its IP address. Of course its also possible to add the default_server parameter to one of the website configurations.

server {  
    listen       80  default_server;

    server_name  _;
    return       444;

Like explained this configuration can be enabled by linking the file into the sites-enabled directory.

[email protected]:/etc/nginx$ ln -s /etc/sites/available/default /etc/sites-enabled/  

Access logs

As default every request is being logged to a disk file which impacts performance a lot. We should turn access logging off or at least reduce it amount of logging. To turn access logging completely off use the value off for the access_log directive. Though personally I prefer to write access logs (and especially error logs) for analytic purposes but am not interested about static or missing files.


server {  
    log_not_found off;

    location ~ \.(css|cur|gif|ico|jp?g|js|png|svg|ttf|txt|xml)$ {
        access_log off;

Cache headers

It's possible to set expire headers for files that don't change and are served regularly.

server {  
    location ~ \.(css|cur|gif|ico|jp?g|js|png|svg|ttf|txt|xml)$ {
        expires max;
        add_header Pragma public;
        add_header Cache-Control "public, must-revalidate, proxy-revalidate";