Hit counter persistent variables

Hello,

Is there a way to have hit counter variables that would save every hour
or persist across restarts?

I’m working on a photography site that I expect to be serving a huge
number of thumbnail (static) images. I don’t need full logging of the
thumbnails so I’ve turned if off for that location but it sure would be
nice to keep some tracking of how many are served and perhaps at what
rate. I thought maybe just a counter variable that increments on each
hit in that location but it wouldn’t be too useful unless it could save
to file from time to time, or perhaps append a count to a file every
hour so there is a time series log.

I saw there is a traffic logging RRD module but I wonder if that is
going to impact the ability of handling high activity rates? If that
were turned on for small static images (target: 4k each, 25 /page, 200+
users online) would it slow things down quite a bit? What would be an
upper limit on users “active” in such a scenario?

Chris :slight_smile:

Chris S. ha scritto:

hit in that location but it wouldn’t be too useful unless it could save
to file from time to time, or perhaps append a count to a file every
hour so there is a time series log.

You can write an Nginx module that use, as an example, gdbm to store
data, protected with a lock.

Manlio P.

Thanks for the suggestion. I don’t really want to go to that extent and
I’d wonder if updating a database would just slow the static files down.
What I had in mind is just incrementing a variable. I suppose I could
write a simple module that maintains a few in memory varibles. Just
wondered if there was an easier way.

Maybe write a log format that just adds a single char for each hit. Then
later I can find hit count by checking the length of that log file? And
a cron once an hour that stores the file length in another file and
deletes the log so it repeats. But is it possible to just write one char
to a log file? Maybe log format like,

log_format hits ‘x’;

Just an idea. Would that be very fast?

Chris.

On Wed, Aug 20, 2008 at 09:40:42PM +0700, Chris S. wrote:

to a log file? Maybe log format like,

log_format hits ‘x’;

Just an idea. Would that be very fast?

The simple log is better than db because it has lesser overhead.
The single byte is not possible because nginx adds new line, but
you just need to divide the size by two. The access_log should be
buffered:

  access_log  path/to/log   hits  buffer=128k;

Is shared memory Ok for that? And there can be a tool view the stats.

2008/8/20 Igor S. [email protected]:

Just to answer my own question in last message,

Yes. That simple log file works. However, it writes 2 bytes per hit. One
for the x and one for \n. Well, thats ok.

But can someone who really knows the internals tell me if a single x log
format would be very fast on say 1000+ requests/second or if it would
degrade how many thumbnails/second noticeably?

Chris :slight_smile:

On Wed, Aug 20, 2008 at 09:53:08PM +0700, Chris S. wrote:

Just to answer my own question in last message,

Yes. That simple log file works. However, it writes 2 bytes per hit. One
for the x and one for \n. Well, thats ok.

But can someone who really knows the internals tell me if a single x log
format would be very fast on say 1000+ requests/second or if it would
degrade how many thumbnails/second noticeably?

I suspect that on 1000 r/s you do not even see noticeable difference in
top.

Chris S. ha scritto:

Thanks for the suggestion. I don’t really want to go to that extent and
I’d wonder if updating a database would just slow the static files down.

Yes, it will slow down.

What I had in mind is just incrementing a variable. I suppose I could
write a simple module that maintains a few in memory varibles. Just
wondered if there was an easier way.

But do you need to keep a global counter, or a counter for each file?
I was suggesting a database assuming you want to know how many time each
single thumbnail was server.

Maybe write a log format that just adds a single char for each hit. Then
later I can find hit count by checking the length of that log file? And
a cron once an hour that stores the file length in another file and
deletes the log so it repeats.

In this case I think it is better to use shared memory.
You open a file (whose name is in configration), make sure its size is
sizeof(atomic_t), mmap it, cast the memory address to atomic_t* and
then use atomic_fetch_add(*v, 1) to increment the counter.

Each time you want to know the hit count, you just have to do the same
operation, reading the atomic_t* variable from mapped memory (this
should be an atomic operation).

This solution should be very fast, but you need to write a custom module
and a custom tool for reading data.

[…]

Manlio P.

Excellent. Thank you for the idea about buffering too.
Chris :slight_smile:

I would like this for per-Host: header byte and http request counts…

Then I could produce daily reports of traffic and http requests by Host:
header

I had this going with Zeus but don’t really have any options with
nginx right now besides turning on access log and churning that data
(which would be gigs of data per day…)

If anyone wants to make a simple stats module that does that, I’d
throw in some $$ - just email me.

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs