# Debugging performance under high load **URL:** \ **Category:** NGINX\ **Created:** [February 17, 2013, 12:53pm UTC](https://www.ruby-forum.com/t/debugging-performance-under-high-load/226668 "2013-02-17T12:53:04Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![fluffypony](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/e47c2d/32.png) [@fluffypony](https://www.ruby-forum.com/u/fluffypony)\ **Post date:** [February 17, 2013, 12:53pm UTC](https://www.ruby-forum.com/t/debugging-performance-under-high-load/226668/1 "2013-02-17T12:53:04Z")
Hi all, I have a reasonably beefy VPS (16gb RAM, 4x vCores) running Ubuntu 12.04 LTS on a 1GigE line that is basically uncontested at the moment. Speed tests on the box show reasonably high bandwidth available up and down (VirtIO isn’t on at the moment, but that doesn’t seem to be affecting it). When doing a load test on a static object via HTTPS (apachebench on a 100kb image) with a concurrency of 1000 I’m seeing pretty poor performance - 450 requests per second, about 4.5mbps traffic, and an average of about 2.2s per request. Monitoring the server in htop I’m not seeing the memory even twitch above 570mb (out of 16gb) and an overall processor usage of like 25% per core, if that much. My config is fairly standard - this is a static file, after all, so it’s not even touching php-fpm. I have my hard and soft ulimits raised to 100k for the www-data user. I have my worker\_processes set to 4, worker\_rlimit\_nofile set to 100k, and worker\_connections set to 2048. multi\_accept is on and epoll is on. I have a keepalive timeout of 2. For the purposes of this test I have a self-signed cert on the server, the ssl\_protocols are set to SSLv2 SSLv3 TLSv1; and the ssl\_ciphers are set to RC4:HIGH:!aNULL:!MD5:!kEDH;. Suggestions? How do I debug the poor performance so I at least know what to fix? Is there a way to step through exactly what is happening in a request under load to see where it’s being delayed? I’d like to get it up to at least 1k RPS if not more, and I believe the server and the bandwidth are up to the task. FP Posted at Nginx Forum: > **[Debugging performance under high load](https://forum.nginx.org/read.php?2,236260,236260#msg-236260)** > > Hi all, I have a reasonably beefy VPS (16gb RAM, 4x vCores) running Ubuntu 12.04 LTS on a 1GigE line that is basically uncontested at the moment. Speed tests on the box show reasonably high bandwidth available up and down (VirtIO isn't on at the... ---
**Author:** ![fluffypony](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/e47c2d/32.png) [@fluffypony](https://www.ruby-forum.com/u/fluffypony)\ **Post date:** [February 18, 2013, 8:03pm UTC](https://www.ruby-forum.com/t/debugging-performance-under-high-load/226668/2 "2013-02-18T20:03:19Z")
Hello! On Sun, Feb 17, 2013 at 3:52 AM, fluffypony wrote: > How do I debug the poor > performance so I at least know what to fix? Is there a way to step through > exactly what is happening in a request under load to see where it’s being > delayed? I’d like to get it up to at least 1k RPS if not more, and I believe > the server and the bandwidth are up to the task. We’ve been using the Flame Graph tools to profile our online Nginx on Linux in production. It is a great tool to find out which part (be it a function or a code path) is hot and slow (on various levels like the kernelspace, the C level in userspace, or even high levels on scripting languages like Lua). See the ngx-sample-bt tool in my Nginx Systemtap Toolkit: ``` https://github.com/agentzh/nginx-systemtap-toolkit#ngx-sample-bt ``` There’s no need to recompile or restart your Nginx for the live profiling. Just ensure that your Nginx executable is not stripped (the DWARF debug symbols should be enabled by Nginx by default). Another prerequisite to use tools in my Nginx Systemtap Toolkit is that you have a working systemtap installation in your Linux system, see the documentation for details: ``` https://github.com/agentzh/nginx-systemtap-toolkit#prerequisites ``` Best regards, -agentzh