Forum: NGINX Syslog

Hi all, I'm new here.

Since I need my nginx to be integrated with syslog, I looked at the 
unofficial
patches (udpremote and syslog patch).

I saw I can have syslog logging with these, but I can not set a custom 
program
identifier per vhost, so that I can easily distinguish in the syslog 
daemon and
make it save the right log in the right file, making syslog manage
location(remote/local) and rotation of the logs.


Since the patches do not permit that, I think I'm going to write yet 
another
syslog module/patch.

So I have a couple of questions:
 - why weren't the patches/modules merged into the main ngix project? 
Any
criticism on those patches so that I don't make the same mistakes?

 - Do you think it's better to support syslog in the 
access_log/error_log
directives, just like the patch, or to have a different error_syslog /
access_syslog directives?

 - Do you have any preference in the format of the directive? (ex:
"access_syslog host:local6:notice" instead of "syslog_access host local6 
notice"
etc...)


Thank you for any reply in advance,
Luke

On Thursday 26 April 2012 17:14:06 Luke wrote:
[...]
> So I have a couple of questions:
>  - why weren't the patches/modules merged into the main ngix project? Any
> criticism on those patches so that I don't make the same mistakes?
>

http://trac.nginx.org/nginx/ticket/95

 wbr, Valentin V. Bartenev

On 26/04/2012 15:22, Valentin V. Bartenev wrote:
> http://trac.nginx.org/nginx/ticket/95
>
>  wbr, Valentin V. Bartenev
I saw that patch, and I understand that syslog(3) does block, but since 
I need
to set one program identifier per vhost, I wouldn't be using syslog(3), 
but
rather reimplementing it (already done, taken a lot from glib, bsd 
license).

Using udp connections would make the call non-blocking, and should be 
safe
enough on localhost.

Then again, other solutions can be found, such as having a dedicated 
thread send
the message to syslog, so that logging calls are non-blocking.

Or I could write on a fifo buffer/file the messages and have the 
dedicated
thread send the messages to syslog (date is set on the sent message, not 
at the
time the message arrives to syslog, so no problem with that).

Luke

Hello!

On Thu, Apr 26, 2012 at 03:14:06PM +0200, Luke wrote:

>
> Since the patches do not permit that, I think I'm going to write yet another
> syslog module/patch.
>
> So I have a couple of questions:
>  - why weren't the patches/modules merged into the main ngix project? Any
> criticism on those patches so that I don't make the same mistakes?

The main problem with syslog in general is that it's interface
blocks, and if syslogd daemon is dead/can't cope with load nginx
won't be able to work.  Additionally, it's slow/resource
consuming.  Writing normal files is much faster and more reliable.

Given the above, it's highly unlikely that syslog patches of any
quality will be accepted.

Maxim Dounin

On 26/04/2012 16:03, Maxim Dounin wrote:
> Hello!
> The main problem with syslog in general is that it's interface
> blocks, and if syslogd daemon is dead/can't cope with load nginx
> won't be able to work.  Additionally, it's slow/resource
> consuming.  Writing normal files is much faster and more reliable.
>
> Given the above, it's highly unlikely that syslog patches of any
> quality will be accepted.
>
> Maxim Dounin
Thank you for your reply, but as I said, I already have reimplemented 
syslog(3),
and with the aid of a udp or fifo with a thread things can be made 
completely
asynchronous.
Or I could use an aio_write() instead of a write() to write to the local 
syslog
socket, although a fifo is more secure for syslog crashes/hangups (which 
I've
never seen, anyway).



Anyway, let's assume for a second that I can get file-like performance.
The questions that remains are 2:
1- What format do you like more? I'm inclined towards something like
"syslog_access ident:facility:priority:host:port:connectiontype"
2- How should I benchmark it all?

Luke

The performance of syslog depends heavily on what implementation you are
using. For example, rsyslog has been measured at 1M logs/sec under ideal
conditions (and I've used it at over 100K logs/sec). rsyslog is now the
default syslog on just about every linux distro. This isn't like the old
sysklogd that was could only process one message at a time and blocked
everything until it finished processing the message (limiting it to a
few hundred logs/sec, even on really good disks)

I can understand why you would not want to make logging to syslog be the
default, but why are you so opposed to having it as an option?

yes, if the syslog daemon dies, things writing to it will block, but if
your partition that you are writing logfiles to fills up you will block
as well. Deciding how to log, and if blocking on another daemon is
acceptable should be up to the system administrator. Sometimes getting a
copy of the logs elsewhere in near real-time is worth reducing the peak
requests/sec that a box can handle

The other approach you could allow someone to submit would be to sent
the logs out in real-time via UDP, this will mean that logs are lost if
the syslog daemon goes down, but nginx won't block

David Lang

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,225811,228603#msg-228603

Maybe you can try the patch from Tengine
(http://tengine.taobao.org/document/http_log.html). We implement the
syslog protocol which indeed use the UDP socket. It will not block
Nginx any more.

2012/7/16 dlang <nginx-forum@nginx.us>:

Just thought i would add that support for error logging over syslog 
would
be great. Udp would be even better as we could send it straight out to 
the
remote storage and skip the local disk (io). If you need a test for your
module once you write it just let me know.

Although i plan on taking a look at tengines solution since im already
using some of his patches.