Forum: NGINX NGINX - traffic log / graphing / billing per destination ip

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
D0a98f6304ff122ac9b487ff8275d8bb?d=identicon&s=25 Payam Chychi (Guest)
on 2009-03-08 02:49
(Received via mailing list)
Hi Guys,

I've spend some time going through google trying to find an elegant
solution for traffic logging/ graphing(rrd) / billing(based on rrd?)
but have come up empty handed. Right now here is what I am doing:

I have a span port that monitors all traffic. On the system connected
to the span port I run an iptables + few scripts ive made that simply
adds the dst_ip address into iptables and then the other scripts run
through the output of  iptables and keeps learns counter values. There
is another script that runs in the background that compares data from
previous run to the current run and logs the data; it then injects it
into RRD which allows me to graph (and also use for billing)... This
is fine for now but its simply not a very elegant solution. Id like
for someone to be able to support the system if i ever choose to leave
the company.

Thanks in advance,
7b488e1be2db29c8489d80d59818b17e?d=identicon&s=25 Jeffrey Lim (jfl)
on 2009-03-08 06:05
(Received via mailing list)
On Sun, Mar 8, 2009 at 9:40 AM, Payam Chychi <pchychi@gmail.com> wrote:
> Hi Guys,
>
> I've spend some time going through google trying to find an elegant
> solution for traffic logging/ graphing(rrd) / billing(based on rrd?)
> but have come up empty handed. Right now here is what I am doing:
>
> <snip>

how about log parsing? would that help?

-jf

--
In the meantime, here is your PSA:
"It's so hard to write a graphics driver that open-sourcing it would not
help."
    -- Andrew Fear, Software Product Manager, NVIDIA Corporation
http://kerneltrap.org/node/7228
F5a6ed477b109fe6acc11a5a8f87e7e8?d=identicon&s=25 mike (Guest)
on 2009-03-08 07:00
(Received via mailing list)
i've had to resort to a mixture of basic logging and then processing
those logs later on. it -seems- to be reliable enough for now.

in my nginx.conf:

log_format traffic
'$http_host|$bytes_sent|$time_local|$remote_addr|$request_uri';
access_log /var/log/nginx/traffic traffic;

then a php script i run once per night at like 12:05am, it grabs the
stats from 3 webservers

it even parses the host and allows for URI processing to alter the
host to push the data from one host to another (like i do a couple
things to correlate bandwidth from one site based on the uri or the
host)

*** YOUR MILEAGE MAY VARY ***

#!/usr/local/bin/php -q
<?php

$servers = Array('web01', 'web02', 'web03');

$daysback = 1;
for($x=$daysback; $x>0; $x--) {
        $day = date('Ymd', time()-(60*60*24*$x));
        echo "$day\n";
        runstats($day);
}
system("rm -f /tmp/log??????");

function runstats($day) {
        global $servers;

        $tmpfile = tempnam('/tmp', 'log');
        $logfile = '/var/log/nginx/traffic.'.$day;

        $hosts = array();
        $bytes = array();
        $transfer = array();

        foreach($servers as $server) {
                echo "\t{$server}\n";
                system("/usr/bin/scp {$server}:{$logfile} {$tmpfile}
2>/dev/null");
                if(file_exists($tmpfile)) {
                        $fp = fopen($tmpfile, 'r');
                        while(!feof($fp)) {
                                list($host, $bytes, $date, $ip, $uri)
= fgetcsv($fp, 1024, '|');
                                $host = preg_replace('/^www\./', '',
$host);
                                $host = preg_replace('/\.$/', '',
$host);
                                if(strstr($host, ':')) {
                                        $host = rtrim($host, ':80');
                                        $host = rtrim($host, ':443');
                                }
                                $host = strtolower($host);
                                if(substr($uri, 0, 21) ==
'/rating/video/get.php') {
                                        if(strstr($host, 'foo3')) {
                                                $host = 'foo.com';
                                        } elseif(strstr($host, 'foo2'))
{
                                                $host = 'foo.com';
                                        }
                                }
                                if(!isset($requests[$host])) {
                                        $requests[$host] = 1;
                                } else {
                                        $requests[$host]++;
                                }
                                if(!isset($transfer[$host])) {
                                        $transfer[$host] = $bytes;
                                } else {
                                        $transfer[$host] =
$transfer[$host] + $bytes;
                                }
                                $hosts[$host] = 1;

                        }
                        fclose($fp);
                        unlink($tmpfile);
                }
        }

        $day = date('Y-m-d', strtotime($day));
        db_query("DELETE FROM stats WHERE day='{$day}'");
        foreach(array_keys($hosts) as $host) {
                if(!empty($host)) {
                        $vhost = db_escape($host);
                        $t = isset($transfer[$host]) ? $transfer[$host]
: 0;
                        $r = isset($requests[$host]) ? $requests[$host]
: 0;
                        db_query("REPLACE INTO stats
(vhost,day,bytes,requests) VALUES('$vhost', '$day', $t, $r)");
                }
        }
}
?>
This topic is locked and can not be replied to.