i have a question regarding the --with-debug - option; do i have to
expect
much overhead,
when compiling nginx with that option, but have it disabled per default?
i’d like to have it at hand for debugging certain issues every then and
there.
From what I’ve seen here there is a small <5% overhead when compiled
with
debug and 5-10% while debug logging is active. I want my 5% so I make
2
versions.
On Fri, Sep 27, 2013 at 09:16:39AM -0400, mex wrote:
i have a question regarding the --with-debug - option; do i have to expect
much overhead,
when compiling nginx with that option, but have it disabled per default?
i’d like to have it at hand for debugging certain issues every then and
there.
While there is some overhead, it’s actually nontrivial to show it.
E.g. here are http_load results of a “return 204” test on my
laptop (5 tests for each --with-debug and without debug, 10
seconds each test), analyzed by ministat(1):
x bench.debug
bench.nodebug
±-----------------------------------------------------------------------------+
|x xx + + + + x x +|
| ||M___A____M_______A|______| |
±-----------------------------------------------------------------------------+
N Min Max Median Avg
Stddev
x 5 7159.63 7754.85 7454.49 7503.202
240.43415
5 7503.91 8148.11 7566.35 7671.328
269.26662
No difference proven at 95.0% confidence
That is, tests done aren’t enough to show the difference, if any.
Overall, I would recommend to don’t bother and to compile --with-debug
if there are chances you’ll need it.
i have a question regarding the --with-debug - option; do i have to expect
much overhead,
when compiling nginx with that option, but have it disabled per default?
We avoid enabling debug logs in production because when we really need
to debug online issues, debug logs are just too expensive to emit and
analyze (consider the box is currently experiencing an L7 attack).
Also, --with-debug compiles in extra code paths which could have
bugs and consume some extra CPU cycles even when you’re not further
enabling it in nginx.conf.
Also, you’ll never never have enough debug logs for answer those
really hard questions in production.
We only enable it for every day Nginx related development. For online
trouble shooting, we’re relying on systemtap to do low overhead
dynamic tracing on various software stack levels, as demonstrated by
my Nginx Systemtap Toolkit:
your points are valid, but i talk about heisenbugs and the ability
to monitor a certain ip; you know, theres WTF??? - errors
please note, on the infrastructure i talk about we have usually
debug-logs disabled, and the bottleneck is usually the app-servers.
but thanx for your answer, i’ll invest some time and check your
toolchains,
especially systemtap. is systemtap included in openresty? looks like the
perfect tool to create some nagios-plugins upon.
Thank you agentz, that looks amazing. I will be including that in the
next
server software push.
And also Maxim, thank you for you for taking the time to prepare those
figures. I am going to my own testing and presuming that holds with our
usecase / modules I will be deploying --with-debug myself. Its
definitely
worth it for <5% any day.
Regards,
Mathew
On Sat, Sep 28, 2013 at 5:01 AM, Yichun Z. (agentzh)
your points are valid, but i talk about heisenbugs and the ability
to monitor a certain ip; you know, theres WTF??? - errors
It’s trivial to do with dynamic tracing tools like systemtap and
dtrace. You don’t even need to reload Nginx for this or parse any log
data files.
We also have great success in tracing down the real cause of those
rare timeout requests in our production nginx servers Again, we
didn’t change the Nginx configuration file nor reload the server.
please note, on the infrastructure i talk about we have usually
debug-logs disabled, and the bottleneck is usually the app-servers.
For our online systems, Nginx is the application server
but thanx for your answer, i’ll invest some time and check your toolchains,
especially systemtap. is systemtap included in openresty? looks like the
perfect tool to create some nagios-plugins upon.
systemtap is the tool framework that can answer almost any questions
that can be formulated in its scripting language The real-world
questions may involve many software layers at once, like involving
Nginx and Linux kernel’s TCP/IP stack (and even the LuaJIT VM) at the
same time. And systemtap can associate events happening at different
layers of the software stack easily and efficiently.
The biggest selling point is that you don’t have to modify the
software nor the software’s configuration yourself to make things work
and you can always aggregate data at the data source, saving a lot of
resources in dumping, storing, and parsing the raw logging data.
Best regards,
-agentzh
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.