As many of us will have discovered over the course of our careers,
centralised/non-local logging is an important part of any non-trivial
infrastructure.
I’m aware of different patches that add syslogging capabilities to
different Nginx versions, but I’ve yet to see an official description
of how we should achieve non-local logging. Preferably syslog,
personally speaking, but anything scalable, supportable, debug-able
and sane would, I feel, be acceptable to the wider community.
I’m aware of at least the following options, but I feel they’re all
lacking to some degree:
log to local disk and syslog/logstash/rsync them off: undesirable
due to the management overhead of the additional logging process/logic
the wasted disk I/O when creating the per-request logs
use post_action to hit a logging endpoint after each request: would
add overhead to both configuration and network traffic; post_action
has been described as “a bit of a hack”, IIRC
use syslog patches: not “official”, hence troublesome to debug
on-list; rightly or wrongly, the usual response of “please replicate
problem without 3rd party patches” would cause problems when debugging
production systems
use another 3rd-party logging protocol, e.g. statsd, redis: as
similarly unsupportable as syslog patches
I wonder if someone could officially comment on the potential for
resolving this gap in Nginx’s feature set? Or perhaps I’m missing
something …
On Wed, Jun 6, 2012 at 3:39 PM, Jonathan M. [email protected] wrote:
and sane would, I feel, be acceptable to the wider community.
has been described as “a bit of a hack”, IIRC
resolving this gap in Nginx’s feature set? Or perhaps I’m missing
something …
Hi!
I think this thread from a few days ago might answer some of your
questions :
We use log_by_lua to aggregate data, and then fetch the aggregated
data from another location.
In turns this feeds into a bigger centralized system (OpenTSDB).
This way we don’t have to output one log line, or hit one logging
endpoint for each request that goes through.
One interesting thing we’ve noticed while developing this is that even
an error in our logging code really doesn’t affect serving pages, so
this might be the safest use case for 3rd party modules you could
imagine.
Yes, it was glancing over this thread that made me think I needed to
raise this issue kinda sorta formally, and ask for some official
recognition of this relatively important issue.
We use log_by_lua to aggregate data, and then fetch the aggregated
data from another location.
In turns this feeds into a bigger centralized system (OpenTSDB).
This way we don’t have to output one log line, or hit one logging
endpoint for each request that goes through.
One interesting thing we’ve noticed while developing this is that even
an error in our logging code really doesn’t affect serving pages, so
this might be the safest use case for 3rd party modules you could
imagine.
Interesting, thank you. [ I’ve so far avoided enabling lua in my
production Nginx instances as it’s almost /too/ powerful. If our
systems team (or devs!) get the idea we can do /anything/ we like in
the reverse proxy layer in front of our app servers, I’m concerned
we’ll start to add too much logic in there that should actually live
in the apps ]
I’ll definitely take a look at log_by_lua, however I still think we
should get the centralised logging issue recognised officially as a
area where Nginx is deficient, along with suggestions for bringing it
up to spec. Nginx is a vital part of many people’s infrastructures
now: it’s a shame it’s behind the (devops’y!) times in this area.
Currently your only other official alternative to get data, besides
the access_log is the stub_status module : Module ngx_http_stub_status_module
Appreciated but, not being per-request, it isn’t really in the same
space as the other things we’re discussing.
On Wed, Jun 06, 2012 at 11:39:30PM +0100, Jonathan M. wrote:
and sane would, I feel, be acceptable to the wider community.
I’m aware of at least the following options, but I feel they’re all
lacking to some degree:
log to local disk and syslog/logstash/rsync them off: undesirable
due to the management overhead of the additional logging process/logic
the wasted disk I/O when creating the per-request logs
This is, actually, recommended solution. It’s main advantage is
that nginx will continue to serve requests even if something
really bad will happen with your “non-trivial infrastructure”.
Basically it will work till the server is alive.
[…]
use another 3rd-party logging protocol, e.g. statsd, redis: as
similarly unsupportable as syslog patches
I haven’t actually used it (as I prefer local logging solution,
see above), but I tend to like the basic concept behind the
module. At least it resolves the main problem with usual syslog
interfaces (i.e. blocking).
I realize that this is another 3rd party module, so falls into the
same ‘not officially supported’ category, but I figured I’d mention it
as something to add to the centralized logging bag of tricks. I’ve
used it and it’s pretty nice, though obviously you need an sflow
server somewhere to send to.
I just wanted to add a couple of comments on using sFlow as the logging
protocol - there are also sFlow plugins for Apache and Tomcat, allowing
you to get the same data from all the HTTP server pools.
On the analyzer side, the HTTP sFlow export also includes performance
counters that can be trended with Graphite or Ganglia. You can also use
sflowtool to convert the binary sFlow datagrams to ASCII combined
logfile format that can be fed into any log analyzer.
Mark M. wrote in post #1063450:
> * use another 3rd-party logging protocol, e.g. statsd, redis: as
> similarly unsupportable as syslog patches
I realize that this is another 3rd party module, so falls into the
same ‘not officially supported’ category, but I figured I’d mention it
as something to add to the centralized logging bag of tricks. I’ve
used it and it’s pretty nice, though obviously you need an sflow
server somewhere to send to.
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
