# Blackbox Logs **URL:** \ **Category:** NGINX\ **Created:** [June 17, 2010, 6:00pm UTC](https://www.ruby-forum.com/t/blackbox-logs/190324 "2010-06-17T18:00:21Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![alangano](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/94ad74/32.png) [@alangano](https://www.ruby-forum.com/u/alangano)\ **Post date:** [June 17, 2010, 6:00pm UTC](https://www.ruby-forum.com/t/blackbox-logs/190324/1 "2010-06-17T18:00:21Z")
I attempted to get a discussion going on [ServerFault.com](http://ServerFault.com) regarding “blackbox” logs and Nginx. It didn’t get the response I imagined, and I was told to ask here. Here is an article about “blackbox” logs in Apache by Chris Josephes from 2004. > **[Radar - O’Reilly](https://www.oreilly.com/radar/)** > > Now, next, and beyond: Tracking need-to-know trends at the intersection of business and technology I’ve always kept these logs around on my Apache servers and they have come in handy. I’m trying to recreate them for Nginx. Here is the Apache format. LogFormat “%a/%S %X %t "%r" %s/%\>s %{pid}P/%{tid}P %T/%D %I/%O/%B” blackbox Here’s what Nginx blackbox format would look like, the unmapped Apache directives have question marks after their names. access\_log blackbox ‘$remote\_addr/$remote\_port X? [$time\_local] “$request”’ ‘s?/$status $pid/0 T?/D? I?/$bytes\_sent/ $body\_bytes\_sent’ Here’s a table of the variables I’ve been able to map from the Nginx documentation. %a = $remote\_addr - The IP address of the remote client. %S = $remote\_port - The port of the remote client. %X = ? - Keep alive status. %t = $time\_local - The start time of the request. %r = $request - The first line of request containing method verb, path and protocol. %s = ? - Status before any redirections. %\>s = $status - Status after any redirections. %{pid}P = $pid - The process id. %{tid}P = N/A - The thread id, which is non-applicable to Nignx. %T = ? - The time in seconds to handle the request. %D = $request\_time - The time in milliseconds to handle the request. %I = ? - The count of bytes received including headers. %O = $bytes\_sent - The count of bytes sent including headers. %B = $body\_bytes\_sent - The count of bytes sent excluding headers, but with a 0 for none instead of ‘-’. So, I’m missing %I bytes received and %X keep alive status. The timings are off, but I don’t mind. Nginx doesn’t to milliseconds like Apache, but rather seconds with millisecond precision. Making the log format exactly like the Apache blackbox log format would be a nice exercise, but it is not imperative. If Nginx can do simple adjustments using multiplication that would be cool, but I haven’t tried it yet, nor have I seen any examples of it. Alan G. - [http://twitter.com/bigeasy](http://twitter.com/bigeasy) ---
**Author:** ![alangano](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/94ad74/32.png) [@alangano](https://www.ruby-forum.com/u/alangano)\ **Post date:** [June 18, 2010, 11:15pm UTC](https://www.ruby-forum.com/t/blackbox-logs/190324/2 "2010-06-18T23:15:27Z")
Maxim D. wrote: > Actually, most important thing to log is $upstream\_response\_time. > So it’s not a good idea to log something in an “apache compatible” > format as you are going to log wrong metrics in this case. To this point, I’m really looking to create something that is _conceptually_ compatible to the log format described in the original article, a log format geared towards generalized performance and security auditing, rather than hit counting. There is no need to record thread id, in Nginx, quite obviously. If there was a legacy of scripts for the “blackbox” log format described by Chris Josephes, that might be a different matter, but there are not. My own little Nginx “blackbox” format will include $upstream\_response\_time as you suggest. Thank you very much for filling in the blanks for me. Alan G. - [http://twitter.com/bigeasy](http://twitter.com/bigeasy) ---
**Author:** ![alangano](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/94ad74/32.png) [@alangano](https://www.ruby-forum.com/u/alangano)\ **Post date:** [June 18, 2010, 1:31pm UTC](https://www.ruby-forum.com/t/blackbox-logs/190324/3 "2010-06-18T13:31:44Z")
Hello! On Thu, Jun 17, 2010 at 10:59:27AM -0500, Alan G. wrote: > have come in handy. I’m trying to recreate them for Nginx. Here is > $body\_bytes\_sent’ > Here’s a table of the variables I’ve been able to map from the Nginx > documentation. > %a = $remote\_addr - The IP address of the remote client. > %S = $remote\_port - The port of the remote client. > %X = ? - Keep alive status. Logging $sent\_http\_connection should be enough, but due to a bug it currently always evaluates to “closed” during request logging. Patch below. > %t = $time\_local - The start time of the request. > %r = $request - The first line of request containing method verb, > path and protocol. > %s = ? - Status before any redirections. There is no such thing in nginx. > %\>s = $status - Status after any redirections. > %{pid}P = $pid - The process id. > %{tid}P = N/A - The thread id, which is non-applicable to Nignx. > %T = ? - The time in seconds to handle the request. > %D = $request\_time - The time in milliseconds to handle the request. > %I = ? - The count of bytes received including headers. $request\_length Note that it’s _including_ headers, while documentation says (in russian) something like “length of request body”. > examples of it. > Actually, most important thing to log is $upstream\_response\_time. > So it’s not a good idea to log something in an “apache compatible” > format as you are going to log wrong metrics in this case. Maxim D. p.s. diff --git a/src/http/ngx\_http\_request.c b/src/http/ngx\_http\_request.c — a/src/http/ngx\_http\_request.c +++ b/src/http/ngx\_http\_request.c @@ -2435,10 +2435,10 @@ ngx\_http\_set\_keepalive(ngx\_http\_request\_ } } - ngx\_http\_free\_request(r, 0); - r-\>keepalive = 0; - ngx\_http\_free\_request(r, 0); - c-\>data = hc; ---
**Author:** ![alangano](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/94ad74/32.png) [@alangano](https://www.ruby-forum.com/u/alangano)\ **Post date:** [June 20, 2010, 11:27pm UTC](https://www.ruby-forum.com/t/blackbox-logs/190324/4 "2010-06-20T23:27:23Z")
Hello! On Fri, Jun 18, 2010 at 04:14:21PM -0500, Alan G. wrote: > is no need to record thread id, in Nginx, quite obviously. If there > was a legacy of scripts for the “blackbox” log format described by > Chris Josephes, that might be a different matter, but there are not. > My own little Nginx “blackbox” format will include > $upstream\_response\_time as you suggest. Just a quick note: You may also want to include $upstream\_addr and $upstream\_status for configurations with more than one upstream server, and $upstream\_cache\_status for configurations using proxy/fastcgi cache. Other things to consider: $gzip\_ratio, $pipe. Maxim D.