Logging to pipe

Server NGINX
1

Hi,

I’d like to use syslog-ng for centralized logging. I looked through
the archive and found that the reason that logging to a device or pipe
not being supported is because of speed concerns (too slow).

How slow is it? If I create a pipe with mkfifo and then write my
nginx logs to it, is performance going to be terrible?

Joe

2

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Joe Van D. wrote:
| Hi,
|
| I’d like to use syslog-ng for centralized logging. I looked through
| the archive and found that the reason that logging to a device or pipe
| not being supported is because of speed concerns (too slow).
|
| How slow is it? If I create a pipe with mkfifo and then write my
| nginx logs to it, is performance going to be terrible?
|
| Joe
|
We used the named pipes for a while in our setup. I saw syslog using up
30% cpu time on our quadcore 1950 dells (2.6 Ghz intel) when pushing
them to max load.

Later I wrote a patch which you can find it here
http://article.gmane.org/gmane.comp.web.nginx.english/5296. This patch
disables the use of named pipes, but performance with syslog is just as
bad. The bottleneck isn’t nginx offcourse but syslog.

Regards
Marlon de Boer
System A. hyves.nl
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkhE8+QACgkQQ6KLqYkbN3cxpACfYhtiH2/Cp+kdPDNYUtdlB3Jp
9+UAnjva6MoD8T+LRt/mHXB/tcdbA48L
=nk9R
-----END PGP SIGNATURE-----

3

Marlon de Boer wrote:

| How slow is it? If I create a pipe with mkfifo and then write my
disables the use of named pipes, but performance with syslog is just
as bad. The bottleneck isn’t nginx offcourse but syslog.

So there is no extra load on the nginx machine, just on the syslog
server?

Are you using syslog-ng? Buffering disk writes? Any fancy filters?

There was a spread module some time back for apache:
http://www.schlossnagle.org/~george/mod_log_spread/

Looked interesting…

Any other suggestions for reliable remote logging?

Ed W

4

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ed W wrote:
| So there is no extra load on the nginx machine, just on the syslog server?
Syslog has to run local on the machine and the 30% cpu time I measured
while benchmarking was local while syslog was reading from /dev/log. So
there is an extra load locally. Be aware that this 30% is restricted to
our test and could be quite different on your hardware.
|
| Are you using syslog-ng? Buffering disk writes? Any fancy filters?
We are using syslog, with some filters to split the access and the error
logging.

destination nginx_access { file("/var/log/nginx/access_log"); };
destination nginx_error { file("/var/log/nginx/error_log"); };

filter f_nginx_access { program(“nginx”) and level(notice); };
filter f_nginx_error { program(“nginx”) and level(crit); };

log { source(src); filter(f_nginx_access); destination(nginx_access); };
log { source(src); filter(f_nginx_error); destination(nginx_error); };

and something like

log { source(src); filter(nginx_access); destination(loghost); };
|
| Any other suggestions for reliable remote logging?
See syslog rules above, using tcp for the syslog loghost and clients
should ensure that your messages always reach the loghost if you’re
network is functioning correct. Also check the load and the disk-io on
the central logserver that it can still keep up with your clients
sending messages.

Regards
Marlon de Boer
System A. www.hyves.nl
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkhG9JYACgkQQ6KLqYkbN3eqjQCfUWnwn+6Mr1WM5QHP+VZ4scpt
RKkAnRkQQdBk6nt/G1r0zb4hKjKJBTZB
=I4i6
-----END PGP SIGNATURE-----