# Log Parsing - Near Real Time **URL:** \ **Category:** NGINX\ **Created:** [August 1, 2011, 3:54pm UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162 "2011-08-01T15:54:15Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\ **Post date:** [August 1, 2011, 3:54pm UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/1 "2011-08-01T15:54:15Z")
I’m looking for a near real-time script to parse log files and insert interesting data into a db. Does anyone know of an existing script to do this? John ---
**Author:** ![John\_Macleod](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/a87d85/32.png) [@John\_Macleod](https://www.ruby-forum.com/u/John_Macleod)\ **Post date:** [August 1, 2011, 4:24pm UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/2 "2011-08-01T16:24:38Z")
I beleive the only way to do this is to custom write a script that reads the log files and then issues a reload signal to nginx (something like logrotate). If anyone does know of a script id be interested to learn too as I do this on my servers using a hacked togeather php script and logrotate (on crontab every minute) ---
**Author:** ![John\_Macleod](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/a87d85/32.png) [@John\_Macleod](https://www.ruby-forum.com/u/John_Macleod)\ **Post date:** [August 1, 2011, 4:40pm UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/3 "2011-08-01T16:40:08Z")
> I’m looking for a near real-time script to parse log files and insert > interesting data into a db. > Does anyone know of an existing script to do this? You can check/try [http://www.splunk.com](http://www.splunk.com) rr ---
**Author:** ![John\_Macleod](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/a87d85/32.png) [@John\_Macleod](https://www.ruby-forum.com/u/John_Macleod)\ **Post date:** [August 1, 2011, 4:58pm UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/4 "2011-08-01T16:58:06Z")
My app has a request that opens the log file, fseeks to the end, backs up as many bytes as it takes to get to the size the log file was on the last similar request by that user, and runs a regex over the novel part to get interesting metrics before closing the file. Since this happens less than once per minute, I have not done anything fancy to optimize. - Randy ---
**Author:** ![John\_Macleod](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/a87d85/32.png) [@John\_Macleod](https://www.ruby-forum.com/u/John_Macleod)\ **Post date:** [August 1, 2011, 6:25pm UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/5 "2011-08-01T18:25:12Z")
You could use nginx-sflow-module (binary logging over UDP). The “sflowtool” program can convert this back to a continuous feed of common log file format at the collector. This way you can receive from multiple servers all sending to the same UDP port. You can also apply random 1-in-N sampling at source as an efficient data-reduction measure if required. Neil M. ---
**Author:** ![John\_Macleod](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/a87d85/32.png) [@John\_Macleod](https://www.ruby-forum.com/u/John_Macleod)\ **Post date:** [August 1, 2011, 5:49pm UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/6 "2011-08-01T17:49:23Z")
You could use something like syslog-ng that can “tail” the log file and run them through a script. – Brian A. ---
**Author:** ![John\_Macleod](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/a87d85/32.png) [@John\_Macleod](https://www.ruby-forum.com/u/John_Macleod)\ **Post date:** [August 1, 2011, 8:58pm UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/7 "2011-08-01T20:58:30Z")
An alternative is to tail -F (aka. “–follow=name --retry”) the log file and pipe the output into a script. This allows you to parse the entries as they come in and rotate the log file as often as you want independently of the parsing script. Regards, Dennis ---
**Author:** ![John\_Macleod](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/a87d85/32.png) [@John\_Macleod](https://www.ruby-forum.com/u/John_Macleod)\ **Post date:** [August 1, 2011, 9:13pm UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/8 "2011-08-01T21:13:18Z")
I cobbled something like this together with open source tools and have been using it on hundreds of servers… pls contact me offline if you’d like a copy 🙂 -Harold On Mon, Aug 1, 2011 at 2:57 PM, Dennis J. \ **Author:** ![John\_Macleod](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/a87d85/32.png) [@John\_Macleod](https://www.ruby-forum.com/u/John_Macleod)\ **Post date:** [August 2, 2011, 1:14pm UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/9 "2011-08-02T13:14:26Z") You can apply “split” to the output of "tail"j to generate files of with the same number of lines, that you would process with the program that extracts the intereresting data from them: NAME split - split a file into pieces SYNOPSIS split [OPTION]… [INPUT [PREFIX]] DESCRIPTION Output fixed-size pieces of INPUT to PREFIXaa, PREFIXab, …; default size is 1000 lines, and default PREFIX is `x’. With no INPUT, or when INPUT is -, read standard input. ``` Mandatory arguments to long options are mandatory for short ``` options too. ``` -a, --suffix-length=N use suffixes of length N (default 2) -b, --bytes=SIZE put SIZE bytes per output file -C, --line-bytes=SIZE put at most SIZE bytes of lines per output file -d, --numeric-suffixes use numeric suffixes instead of alphabetic -l, --lines=NUMBER put NUMBER lines per output file --verbose print a diagnostic just before each output file is opened --help display this help and exit --version output version information and exit SIZE may have a multiplier suffix: b 512, kB 1000, K 1024, MB ``` 1000_1000, M 1024_1024, GB 1000_1000_1000, G 1024_1024_1024, and so on for T, P, E, Z, Y. ---
**Author:** ![John\_Macleod](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/a87d85/32.png) [@John\_Macleod](https://www.ruby-forum.com/u/John_Macleod)\ **Post date:** [August 2, 2011, 4:24am UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/10 "2011-08-02T04:24:21Z")
On Tue, Aug 2, 2011 at 3:08 AM, Harold Sinclair [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)wrote: > I cobbled something like this together with open source tools and have been > using it on hundreds of servers.. pls contact me offline if you’d like a > copy 🙂 Or you can share it on github to make it open to anyone. ---
**Author:** ![John\_Macleod](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/a87d85/32.png) [@John\_Macleod](https://www.ruby-forum.com/u/John_Macleod)\ **Post date:** [August 2, 2011, 3:09am UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/11 "2011-08-02T03:09:36Z")
sflow would be great it it was open source and had an easily customizable server (perl/python/bash or PHP) ---
**Author:** ![John\_Macleod](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/a87d85/32.png) [@John\_Macleod](https://www.ruby-forum.com/u/John_Macleod)\ **Post date:** [August 4, 2011, 7:00pm UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/12 "2011-08-04T19:00:11Z")
Not sure what you mean about sFlow needing to be open source? Here are links to the relevant open-source projects: [http://nginx-sflow-module.googlecode.com](http://nginx-sflow-module.googlecode.com) [http://host-sflow.sourceforge.net](http://host-sflow.sourceforge.net) [http://www.inmon.com/technology/sflowTools.php](http://www.inmon.com/technology/sflowTools.php) With a more complete “developer resources” description here: > **[Developer resources](https://blog.sflow.com/2010/01/developer-resources.html)** > > (photo from sFlow.org ) The challenge of providing  data center visibility  and  control  requires that all network devices integrate the i... If you use sflowtool to turn sFlow-HTTP into common-log format at the collector, that opens up a whole ecosystem of open-source perl/python/bash/PHP tools for the analysis, such as AWStats. > **[AWStats - Open Source Log File Analyzer for advanced statistics (GNU GPL)](https://awstats.sourceforge.io/)** > > AWStats Official Web Site - Compile and generate advanced graphical web, ftp or mail statistics with a logfile analysis (For IIS, Apache,... distributed under GNU GPL). The sFlow-HTTP feed also sends performance counters every N seconds. I don’t yet know of an open-source adaptor to feed that into something like Nagios, Ganglia or Graphite, but I know there are options to do that with the sFlow-HOST performance counters so it shouldn’t be hard to add. In fact, Ganglia now has native support for the sFlow-HOST counters. [Ganglia 3.2.0 Released « Ganglia Monitoring System](http://ganglia.info/?p=430) This sFlow-HOST ([http://host-sflow.sourceforge.net](http://host-sflow.sourceforge.net)) part is helpful because it provides telemetry on the underlying CPU/mem/disk/network stats in a light and scalable way, and supports zero-config (DNS-SD) to make sFlow easier to roll out on a large cluster/farm. Neil ---
**Author:** ![John\_Macleod](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/a87d85/32.png) [@John\_Macleod](https://www.ruby-forum.com/u/John_Macleod)\ **Post date:** [August 5, 2011, 2:30am UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/13 "2011-08-05T02:30:28Z")
On Aug 4, 2011, at 12:17 PM, Dennis J. wrote: > The problem is that sFlow currently lacks practical Documentation and a library > that can be used to develop agents and collectors. The [http://sflow.org](http://sflow.org) website has some overview documentation, links to developer tools, plus agent and collector source code. > It took me a while to realize why I couldn’t find a collector daemon that I > could set up to use with the sflowtool or sFlowTrend. These tools _are_ the > collectors. I would have expected to find some kind of management daemon akin to > the snmp world. Open Source collectors include Ganglia, ntop and pmacct, but there are so many possible applications for this data that no single collector-package is going to encompass them all. I think there may be some collectors that load data into relational database tables but that may not always make sense for what you want to do. Hence the starting point is usually a C, Perl or Java decoder that unpacks the data for you as a real-time feed, then you are free to do whatever you want. > sFlow looks really interesting but it is unnecessarily obscure and the developer > resources could use a face lift and present things in a way that introduces > concepts, term and methodology to new-comers. > Put out a C library with an agent and a collector API and throw the code on > github and you no doubt will see a pickup in interest from developers. Each agent is very different because it gets embedded in the device or application that it is monitoring, so the best thing is probably just to list some of the open-source projects: [http://nginx-sflow-module.googlecode.com](http://nginx-sflow-module.googlecode.com) [http://mod-sflow.googlecode.com](http://mod-sflow.googlecode.com) > **[GitHub - sflow/memcached: memcached development tree](https://github.com/sflow/memcached)** > > memcached development tree > **[Host sFlow](https://sourceforge.net/projects/host-sflow/)** > > Download Host sFlow for free. The Host sFlow project has moved to http://sflow.net > **[Open vSwitch](http://www.openvswitch.org/)** Perhaps there should be a page on [sflow.org](http://sflow.org) with this list? Would that have been helpful for you? If you have more questions about sFlow in general then it might make sense to post them to the sFlow mailing list instead: > **[sFlow - Google Groups](https://groups.google.com/g/sflow)** But getting back to the question of real-time monitoring of nginx servers, the nginx-sflow-module is a complete sFlow agent that offers centralized, real-time monitoring of large clusters. sflowtool can turn the feed into a piped stream of ASCII CLF data, so it represents one way to avoid all that log-file tailing. Neil ---
**Author:** ![John\_Macleod](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/a87d85/32.png) [@John\_Macleod](https://www.ruby-forum.com/u/John_Macleod)\ **Post date:** [August 5, 2011, 8:30am UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/14 "2011-08-05T08:30:21Z")
On 01/08/2011 14:53, John Macleod wrote: > I’m looking for a near real-time script to parse log files and insert > interesting data into a db. > > Does anyone know of an existing script to do this? I don’t think anyone said rsyslog yet? Logs directly into a database if you want, optionally passing through some kind of parser first. It has spooling to disk in case target cant keep up and semi reliable network modes. I heard someone say that Fedora had switched their default syslog to rsyslog (confirmation?) so hopefully it’s not too niche for you… (obviously I believe it can read directly from a file or pipe…) Alternatively ask nginx to log to some local fifo and write your own spooler? Beware blocking if you don’t keep up though… Finally note that there have been some previous patches to nginx to add syslog logging. I personally believe this is useful for many classes of problem, but I believe that the position by Igor is that it’s considered too slow to keep up with nginx? (I think I have seen Thrift patches in the past also?). I personally don’t use nginx to the limit and would love to see syslog logging in standard nginx, even if it limited maximum performance… Perhaps if there are a core of similar interested users we could interest (or pay) Igor to consider adding such a feature, caveat the limitations? Good luck Ed W P.S. This link has some suggestions on logging to a fifo and catching the output in syslog-ng [http://www.spinics.net/lists/centos/msg109747.html](http://www.spinics.net/lists/centos/msg109747.html) ---
**Author:** ![John\_Macleod](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/a87d85/32.png) [@John\_Macleod](https://www.ruby-forum.com/u/John_Macleod)\ **Post date:** [August 4, 2011, 9:17pm UTC](https://www.ruby-forum.com/t/log-parsing-near-real-time/210162/15 "2011-08-04T21:17:58Z")
The problem is that sFlow currently lacks practical Documentation and a library that can be used to develop agents and collectors. It took me a while to realize why I couldn’t find a collector daemon that I could set up to use with the sflowtool or sFlowTrend. These tools _are_ the collectors. I would have expected to find some kind of management daemon akin to the snmp world. sFlow looks really interesting but it is unnecessarily obscure and the developer resources could use a face lift and present things in a way that introduces concepts, term and methodology to new-comers. Put out a C library with an agent and a collector API and throw the code on github and you no doubt will see a pickup in interest from developers. Regards, Dennis