Nginx Performance Tuning

Use case: Nginx on Linux

NOTE: This page only contains Linux specific TCP tunings.

Example /etc/sysct.conf and the drop-in snippet /etc/sysctl.d/*

NOTE: Adjust values to suite specific needs / system (some of the values were stolen from XenServer 6.5 SP1 dom0).

# max number of file handles kernel will allocate

# tweak limits.conf(5) to adjust user file descriptor limit

# pam_limits imposed cap = nproc * nofile should be no greater than fs.file-max kernel cap

fs.file-max = 1024000

# socket listen() backlog

net.core.somaxconn = 65535

# max number of packets queued on the INPUT side

# when the interface receives packets faster than kernel can process

# NIC specific

# To achieve C1M set ii to 65535

net.core.netdev_max_backlog = 5000

# maximum ancillary buffer size allowed per socket

# ancillary data is a sequence of struct cmsghdr structures with appended data

net.core.optmem_max=20480

# default receive / send socket buffer for all protocols

net.core.rmem_default = 16384

net.core.wmem_default = 32768

# max receive / send socket buffer for all protocols

net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

# enable window scaling as defined in RFC132

# the window scale value can be 0 to 14 for each direction independently

# both sides must send the option in their SYN segments to enable window scaling in either direction

net.ipv4.tcp_window_scaling = 1

# low pressure max (in pages)

# net.ipv4.tcp_mem = low pressure max

net.ipv4.tcp_mem = 1572864 1835008 2097152

# min default max size of receive buffer used by TCP - increase by factor of 4

net.ipv4.tcp_rmem = 16384 349520 16777216

# min default max size of send buffer used by TCP

net.ipv4.tcp_wmem = 16384 349520 16777216

# max number of timewait sockets

net.ipv4.tcp_max_tw_buckets = 524288

# maximal number of TCP sockets not attached to any user file handle held by system

# If number is exceeded orphaned connections are reset immediately and warning is printed

# only to prevent simple DoS attachs, MUST not rely on it

# each orphan eats up to ~64K of unswappable memory

# net.ipv4.tcp_max_orphans=262144

# net.ipv4.tcp_orphan_retries = 0

# max number of remembered connection requests which have not received an ACK

# from a connecting client - generic purose -> 8096

net.ipv4.tcp_max_syn_backlog = 65536

# to achieve C1M

# net.ipv4.tcp_max_sync_backlog = 524288

# prevent SYN flood attach

net.ipv4.tcp_syncookies = 1

net.ipv4.tcp_synack_retries = 2 # default 5

net.ipv4.tcp_syn_retries = 2 # default 6

# define local port range for TCP and UDP

net.ipv4.ip_local_port_range = 1024 65000

# length of time an orphaned (no longer referenced by any application) connection

# will remain in the FIN_WAIT_2 state before it is aborted at the local end

net.ipv4.tcp_fin_timeout = 15

# reuse TIME-WAIT sockets for new connections when it is safe from protocol viewpoint

# be careful with NAT

# net.ipv4.tcp_tw_reuse = 0

# congestion control algorithm to be used for new connections bbr to replace cubic ;-)

net.ipv4.tcp_congestion_control = bbr

# TCP will not cache metrics on closing connections (in route cache)

# reused to set initial conditions for new connections

net.ipv4.tcp_no_metrics_save = 1

# TCP Fast Open enabled by default in 3.13+

net.ipv4.tcp_fastopen = 1

# reduce TIME_WAIT from the 120s default to 30-60s

net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30

# reduce FIN_WAIT from the 120s default to 30-60s

net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 30

# TCP Keepalive - try NOT to change unless you know what you are doing

net.ipv4.tcp_keepalive_intvl = 30

net.ipv4.tcp_keepalive_probes = 3

net.ipv4.tcp_keepalive_time = 120

# maybe

net.ipv4.tcp_abort_on_overflow =1

References

• • Nginx tuning for best performance
    https://github.com/denji/nginx-tuning

  • Nginx Tuning for best performance Gist
    https://gist.github.com/denji/8359866

  • Tuning Nginx for Performance
    https://www.nginx.com/blog/tuning-nginx/

  • Performance Tuning - Tips & Tricks

  • https://www.nginx.com/blog/performance-tuning-tips-tricks/

  • Kernel Documentation

  • https://www.kernel.org/doc/Documentation/sysctl/

  • https://www.kernel.org/doc/Documentation/sysctl/fs.txt

  • https://www.kernel.org/doc/Documentation/sysctl/net.txt

  • https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

  • C1M and Nginx

  • https://speakerdeck.com/dctrwatson/c1m-and-nginx

  • https://gist.github.com/dctrwatson/0b3b52050254e273ff11

  • https://www.youtube.com/watch?v=yL4Q7D4ynxU

  • Tuning nginx

  • https://www.nginx.com/blog/tuning-nginx/

  • curl main developer Daniel Stenberg

  • TCP Tuning for HTTP (draft)

  • Linode - Configure Nginx for optimized performance

  • https://www.linode.com/docs/websites/nginx/configure-nginx-for-optimized-performance

  • Brendan Gregg - Performance Tuning EC2 Instances

  • http://www.slideshare.net/brendangregg/performance-tuning-ec2-instances

• http://www.revsys.com/12days/nginx-tuning/

  • Xen - Network Throughput and Performance Guide

  • http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide

  • Lessons learned tuning TCP and nginx in EC2

  • http://engineering.chartbeat.com/2014/01/02/part-1-lessons-learned-tuning-tcp-and-nginx-in-ec2/

  • http://engineering.chartbeat.com/2014/02/12/part-2-lessons-learned-tuning-tcp-and-nginx-in-ec2/

  • Slides: http://www.slideshare.net/chartbeat/tuning-tcp-and-nginx-on-ec2

• http://dak1n1.com/blog/12-nginx-performance-tuning/

  • http://dak1n1.com/blog/14-http-load-generate/

• http://www.speedguide.net/articles/linux-tweaking-121

  • Nginx Performance Tuning

  • - https://www.maxcdn.com/blog/nginx-application-performance-optimization/

  • - https://www.nginx.com/blog/tuning-nginx/

  • RHEL / CentOS / Fedora Network Performance Tuning Guide

• https://access.redhat.com/solutions/29455

• https://accedian.com/blog/tcp-receive-window-everything-need-know/

• Nginx Security Hardening (HTTP Headers)
  https://www.keycdn.com/blog/http-security-headers