Statistics logging

Hello,

I am building a web application that will require a lot of statistic
logging. Each request could generate between 2 and 6 database inserts.
It seems a bit of an overhead to have this running in the main
application. Is something like this worth creating a stats logging
process (with backgroundrb for example). Also, is this something that
could be saved in a separate database since it’s pretty irrelevant
from the main collection of data (just to run marketing statistics
on).

Or am I just overthinking all of this?

Any thoughts on this? Have you dealt with something similar before?

Thanks,
-carl


EPA Rating: 3000 Lines of Code / Gallon (of coffee)

On Friday 09 February 2007, Carl L. wrote:

I am building a web application that will require a lot of statistic
logging. Each request could generate between 2 and 6 database
inserts. It seems a bit of an overhead to have this running in the
main application.

I think your options mainly depend on exactly how the inserts are
triggered. If there’s any request-related processing involved you may
have to do it in the controller(s), hopefully in a filter. If you can
handle the statistic gathering at the model level, then observers may
be for you. If the statistics are related purely to operations on the
data, you may even get away with a database trigger.

Michael


Michael S.
mailto:[email protected]
http://www.schuerig.de/michael/

I’d also add that you batch the inserts instead of inserting into the
database on every logged event.

[email protected] wrote:

I’d also add that you batch the inserts instead of inserting into the
database on every logged event.

Yes, especially if you’re expecting high traffic.

You could store as yaml and possibly avoid the later inserts all
together. Or store as yaml and have a nice hash ready for your batch.


Chris M.
Web D.

Could you elaborate on this please?

I would do all my logging throughout each page into some stats
collector object, then the stats collector object has a write method
of some sort called in an after_filter that sends all the inserts at
once? I would have to write SQL directly I"m guessing or is there a
way to do batch inserts into multiple tables with AR?

Thanks,
-carl

On 2/9/07, Chris M. [email protected] wrote:


Chris M.
Web D.
http://chriscodes.com


EPA Rating: 3000 Lines of Code / Gallon (of coffee)

Carl L. wrote:

Sounds good. Although you lost me at insert to multiple tables at once,
don’t know enough about what you’re trying to do.
It really depends on how many stats you’re logging, and the requirements
for reading them, whether it’s “overthinking” or if my suggestion would
be to “little”.

Since you said it was irrelevant to your app data, I was thinking just
to store it in a yaml file, and load/display it from your “marketing
app”. That way it’s separated. You could do something like

stats = {
‘user’ => ‘Tom’, ‘time’ => Time.now
‘uri’ => request[‘REQUEST_URI’],
‘browser’ => request[‘HTTP_USER_AGENT’] }
stat_file = File.new(“#{RAILS_ROOT}/stats/stats.yml”, “w”)
stat_file << stats.to_yaml
stat_file.close

Now you’ve got your stats in a simple yaml file that can be read like

yaml_stats = YAML.load(File.open(“#{RAILS_ROOT}/stats/stats.yml”))

Then displayed, or manipulated as you wish, and when you wish.
That’s obviously a really basic example.
You could do something more interesting like run a BackgrounDRB process
that loads the yaml file, does your inserts, removes the file, does a
backflip with a twist, then starts over, etc.
If the stats must go in a separate table, or separate database all
together, I’d keep the logic to do that within BackgroundRB, or even a
separate “stats-for-marketing” rails app. Your main app writes the
files, and backgrounDRB or other app reads them and writes to the
database.

Depending on your actual requirements, and if writing to a file is an
option, you could simply write to a csv file that marketing could open
in Excel. Then no extra code is needed to read the stats, and you’re on
your way to finishing early!

HTH


Chris M.
Web D.