Overhead when enabling debug?

Server NGINX
1

i have a question regarding the --with-debug - option; do i have to
expect
much overhead,
when compiling nginx with that option, but have it disabled per default?

i’d like to have it at hand for debugging certain issues every then and
there.

regards,

mex

Posted at Nginx Forum:

2

From what I’ve seen here there is a small <5% overhead when compiled
with
debug and 5-10% while debug logging is active. I want my 5% :slight_smile: so I make
2
versions.

Posted at Nginx Forum:

3

Hello!

On Fri, Sep 27, 2013 at 09:16:39AM -0400, mex wrote:

i have a question regarding the --with-debug - option; do i have to expect
much overhead,
when compiling nginx with that option, but have it disabled per default?

i’d like to have it at hand for debugging certain issues every then and
there.

While there is some overhead, it’s actually nontrivial to show it.
E.g. here are http_load results of a “return 204” test on my
laptop (5 tests for each --with-debug and without debug, 10
seconds each test), analyzed by ministat(1):

x bench.debug

  • bench.nodebug
    ±-----------------------------------------------------------------------------+
    |x xx + + + + x x +|
    | ||M___A____M_______A|______| |
    ±-----------------------------------------------------------------------------+
    N Min Max Median Avg
    Stddev
    x 5 7159.63 7754.85 7454.49 7503.202
    240.43415
  • 5 7503.91 8148.11 7566.35 7671.328
    269.26662
    No difference proven at 95.0% confidence

That is, tests done aren’t enough to show the difference, if any.

Overall, I would recommend to don’t bother and to compile --with-debug
if there are chances you’ll need it.

–
Maxim D.
http://nginx.org/en/donation.html

4

Hello!

On Fri, Sep 27, 2013 at 6:16 AM, mex wrote:

i have a question regarding the --with-debug - option; do i have to expect
much overhead,
when compiling nginx with that option, but have it disabled per default?

We avoid enabling debug logs in production because when we really need
to debug online issues, debug logs are just too expensive to emit and
analyze (consider the box is currently experiencing an L7 attack).

Also, --with-debug compiles in extra code paths which could have
bugs and consume some extra CPU cycles even when you’re not further
enabling it in nginx.conf.

Also, you’ll never never have enough debug logs for answer those
really hard questions in production.

We only enable it for every day Nginx related development. For online
trouble shooting, we’re relying on systemtap to do low overhead
dynamic tracing on various software stack levels, as demonstrated by
my Nginx Systemtap Toolkit:

https://github.com/agentzh/nginx-systemtap-toolkit

and also all those samples in my stap++ project:

https://github.com/agentzh/stapxx#samples

Hope these help.

Best regards,
-agentzh

5

hi maxim,

thanx, thats what i did expected.

i did installed the --with-debug - enabled version on site with ~ 2000
rp/s
during daytime, nothing to see so far.

thanx for checking!

regards,

mex

Posted at Nginx Forum:

6

hi agentzh,

your points are valid, but i talk about heisenbugs and the ability
to monitor a certain ip; you know, theres WTF??? - errors :slight_smile:

please note, on the infrastructure i talk about we have usually
debug-logs disabled, and the bottleneck is usually the app-servers.

but thanx for your answer, i’ll invest some time and check your
toolchains,
especially systemtap. is systemtap included in openresty? looks like the
perfect tool to create some nagios-plugins upon.

regards,

mex

Posted at Nginx Forum:

7

Thank you agentz, that looks amazing. I will be including that in the
next
server software push.

And also Maxim, thank you for you for taking the time to prepare those
figures. I am going to my own testing and presuming that holds with our
usecase / modules I will be deploying --with-debug myself. Its
definitely
worth it for <5% any day.

Regards,
Mathew

On Sat, Sep 28, 2013 at 5:01 AM, Yichun Z. (agentzh)

8

Hello!

On Sat, Oct 5, 2013 at 2:31 AM, mex wrote:

your points are valid, but i talk about heisenbugs and the ability
to monitor a certain ip; you know, theres WTF??? - errors :slight_smile:

It’s trivial to do with dynamic tracing tools like systemtap and
dtrace. You don’t even need to reload Nginx for this or parse any log
data files.

We also have great success in tracing down the real cause of those
rare timeout requests in our production nginx servers :wink: Again, we
didn’t change the Nginx configuration file nor reload the server.

please note, on the infrastructure i talk about we have usually
debug-logs disabled, and the bottleneck is usually the app-servers.

For our online systems, Nginx is the application server :slight_smile:

but thanx for your answer, i’ll invest some time and check your toolchains,
especially systemtap. is systemtap included in openresty? looks like the
perfect tool to create some nagios-plugins upon.

systemtap is the tool framework that can answer almost any questions
that can be formulated in its scripting language :slight_smile: The real-world
questions may involve many software layers at once, like involving
Nginx and Linux kernel’s TCP/IP stack (and even the LuaJIT VM) at the
same time. And systemtap can associate events happening at different
layers of the software stack easily and efficiently.

The biggest selling point is that you don’t have to modify the
software nor the software’s configuration yourself to make things work
and you can always aggregate data at the data source, saving a lot of
resources in dumping, storing, and parsing the raw logging data.

Best regards,
-agentzh