Forum: NGINX Overhead when enabling debug?

2974d09ac2541e892966b762aad84943?d=identicon&s=25 mex (Guest)
on 2013-09-27 15:17
(Received via mailing list)
i have a question regarding the --with-debug - option; do i have to
expect
much overhead,
when compiling nginx with that option, but have it disabled per default?

i'd like to have it at hand for debugging certain issues every then and
there.




regards,


mex

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,243201,243201#msg-243201
2974d09ac2541e892966b762aad84943?d=identicon&s=25 itpp2012 (Guest)
on 2013-09-27 15:43
(Received via mailing list)
From what I've seen here there is a small <5% overhead when compiled
with
debug and 5-10% while debug logging is active. I want my 5% :) so I make
2
versions.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,243201,243202#msg-243202
A8108a0961c6087c43cda32c8616dcba?d=identicon&s=25 Maxim Dounin (Guest)
on 2013-09-27 16:19
(Received via mailing list)
Hello!

On Fri, Sep 27, 2013 at 09:16:39AM -0400, mex wrote:

> i have a question regarding the --with-debug - option; do i have to expect
> much overhead,
> when compiling nginx with that option, but have it disabled per default?
>
> i'd like to have it at hand for debugging certain issues every then and
> there.

While there is some overhead, it's actually nontrivial to show it.
E.g. here are http_load results of a "return 204" test on my
laptop (5 tests for each --with-debug and without debug, 10
seconds each test), analyzed by ministat(1):

x bench.debug
+ bench.nodebug
+------------------------------------------------------------------------------+
|x                     xx   + +  +  +       x  x                              +|
|        |__________|___M___A____M_______A____|_______________|                |
+------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg
Stddev
x   5       7159.63       7754.85       7454.49      7503.202
240.43415
+   5       7503.91       8148.11       7566.35      7671.328
269.26662
No difference proven at 95.0% confidence


That is, tests done aren't enough to show the difference, if any.

Overall, I would recommend to don't bother and to compile --with-debug
if there are chances you'll need it.

--
Maxim Dounin
http://nginx.org/en/donation.html
37f3ea777f96500b332a1a89d6027897?d=identicon&s=25 Yichun Zhang (agentzh) (Guest)
on 2013-09-27 21:32
(Received via mailing list)
Hello!

On Fri, Sep 27, 2013 at 6:16 AM, mex wrote:
> i have a question regarding the --with-debug - option; do i have to expect
> much overhead,
> when compiling nginx with that option, but have it disabled per default?
>

We avoid enabling debug logs in production because when we really need
to debug online issues, debug logs are just too expensive to emit and
analyze (consider the box is currently experiencing an L7 attack).

Also, --with-debug compiles in extra code paths which *could* have
bugs and consume some extra CPU cycles even when you're not further
enabling it in nginx.conf.

Also, you'll never never have enough debug logs for answer those
really hard questions in production.

We only enable it for every day Nginx related development. For online
trouble shooting, we're relying on systemtap to do low overhead
dynamic tracing on various software stack levels, as demonstrated by
my Nginx Systemtap Toolkit:

    https://github.com/agentzh/nginx-systemtap-toolkit

and also all those samples in my stap++ project:

    https://github.com/agentzh/stapxx#samples

Hope these help.

Best regards,
-agentzh
5fdada893aecf361d226b53067e276ff?d=identicon&s=25 SplitIce (Guest)
on 2013-09-29 13:59
(Received via mailing list)
Thank you agentz, that looks amazing. I will be including that in the
next
server software push.

And also Maxim, thank you for you for taking the time to prepare those
figures. I am going to my own testing and presuming that holds with our
usecase / modules I will be deploying --with-debug myself. Its
definitely
worth it for <5% any day.

Regards,
Mathew


On Sat, Sep 28, 2013 at 5:01 AM, Yichun Zhang (agentzh)
2974d09ac2541e892966b762aad84943?d=identicon&s=25 mex (Guest)
on 2013-10-05 11:17
(Received via mailing list)
hi maxim,


thanx, thats what i did expected.

i did installed the --with-debug - enabled version on site with ~ 2000
rp/s
during daytime, nothing to see so far.

thanx for checking!



regards,

mex

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,243201,243451#msg-243451
2974d09ac2541e892966b762aad84943?d=identicon&s=25 mex (Guest)
on 2013-10-05 11:32
(Received via mailing list)
hi agentzh,


your points are valid, but i talk about heisenbugs and the ability
to monitor a certain ip; you know, theres WTF??? - errors :)

please note, on the infrastructure i talk about we have usually
debug-logs disabled, and the bottleneck is usually the app-servers.


but thanx for your answer, i'll invest some time and check your
toolchains,
especially systemtap. is systemtap included in openresty? looks like the
perfect tool to create some nagios-plugins upon.




regards,


mex

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,243201,243452#msg-243452
37f3ea777f96500b332a1a89d6027897?d=identicon&s=25 Yichun Zhang (agentzh) (Guest)
on 2013-10-05 20:26
(Received via mailing list)
Hello!

On Sat, Oct 5, 2013 at 2:31 AM, mex wrote:
> your points are valid, but i talk about heisenbugs and the ability
> to monitor a certain ip; you know, theres WTF??? - errors :)
>

It's trivial to do with dynamic tracing tools like systemtap and
dtrace. You don't even need to reload Nginx for this or parse any log
data files.

We also have great success in tracing down the real cause of those
rare timeout requests in our production nginx servers ;) Again, we
didn't change the Nginx configuration file nor reload the server.

> please note, on the infrastructure i talk about we have usually
> debug-logs disabled, and the bottleneck is usually the app-servers.
>

For our online systems, Nginx *is* the application server :)

> but thanx for your answer, i'll invest some time and check your toolchains,
> especially systemtap. is systemtap included in openresty? looks like the
> perfect tool to create some nagios-plugins upon.
>

systemtap is the tool framework that can answer almost *any* questions
that can be formulated in its scripting language :) The real-world
questions may involve many software layers at once, like involving
Nginx and Linux kernel's TCP/IP stack (and even the LuaJIT VM) at the
same time. And systemtap can associate events happening at different
layers of the software stack easily and efficiently.

The biggest selling point is that you don't have to modify the
software nor the software's configuration yourself to make things work
and you can always aggregate data at the data source, saving a lot of
resources in dumping, storing, and parsing the raw logging data.

Best regards,
-agentzh
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.