Forum: Ruby SQLite as a Log?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
4299e35bacef054df40583da2d51edea?d=identicon&s=25 James Gray (bbazzarrakk)
on 2009-02-13 00:01
(Received via mailing list)
In a current project we're discussing using SQLite for all application
logging.

This is a heavy multiprocessing environment, which was one of our
reasons for considering it.  We also love the idea of structured
queries helping us debug issues:  show me any logged errors between
these times.

The minus of course is a slight loss in transparency.  For example,
you can't just tail a log file when you want.

Anyway, I would love to hear from anyone who has done this.  Did you
like it?  What were the downsides I am missing?

Thanks in advance for any information you can provide.

James Edward Gray II
47b1910084592eb77a032bc7d8d1a84e?d=identicon&s=25 Joel VanderWerf (Guest)
on 2009-02-13 00:19
(Received via mailing list)
James Gray wrote:
> Anyway, I would love to hear from anyone who has done this.  Did you
> like it?  What were the downsides I am missing?
>
> Thanks in advance for any information you can provide.
>
> James Edward Gray II

IIRC one writer blocks the entire sqlite db. You could avoid this using
a gatekeeper server for writing to the log.

Re tailing log files: I avoid that anyway, even with file logging,
because it doesn't know about log rotation. The alternative I've found
helpful is to set up a drb service that lets a process listen for log
entries.
4d5b5dd4e263d780a5dfe7ac8b8ac98c?d=identicon&s=25 Tim Pease (Guest)
on 2009-02-13 00:29
(Received via mailing list)
On Feb 12, 2009, at 3:59 PM, James Gray wrote:

>
> Anyway, I would love to hear from anyone who has done this.  Did you
> like it?  What were the downsides I am missing?
>
> Thanks in advance for any information you can provide.
>

Have you looked into using CouchDB as the logging destination instead?

I've started a branch of my logging gem that allows you to do this,
but I've had to table it due to other obligations.

The goal is to just dump everything to CouchDB with some mnemonics on
the messages. Then you create views to see all messages for a given
application. All error messages in the past week, etc.

Blessings,
TwP
851acbab08553d1f7aa3eecad17f6aa9?d=identicon&s=25 Ken Bloom (Guest)
on 2009-02-13 01:11
(Received via mailing list)
On Thu, 12 Feb 2009 18:17:34 -0500, Joel VanderWerf wrote:

>> can't just tail a log file when you want.
>
> Re tailing log files: I avoid that anyway, even with file logging,
> because it doesn't know about log rotation. The alternative I've found
> helpful is to set up a drb service that lets a process listen for log
> entries.

tail -F is useful following a log across log rotations. It follows the
file by filename, not by file descriptor. (I learned this trick
debugging
an embedded system that rotated its logs every 5 minutes or so.)

--Ken
851acbab08553d1f7aa3eecad17f6aa9?d=identicon&s=25 Ken Bloom (Guest)
on 2009-02-13 01:13
(Received via mailing list)
On Thu, 12 Feb 2009 17:59:53 -0500, James Gray wrote:

> In a current project we're discussing using SQLite for all application
> logging.

Consider a concurrent RDBMS instead if you can deploy it in your
environment.

If you want to tail the log, then it shouldn't be too hard to write a
script to do it. For a project I run, to look at the last 15 log
entries,
I have a shell function:

atlog ()
{
    $sql atman -t -e 'select * from atman_log order by time desc limit
15'
}

(Where $sql starts with "mysql" and then adds login information)

If you want to do continuous monitoring, you could use a loop (maybe
write a ruby script) that checks every 2 seconds or something, and
selects all of the log entries that are more recent than the last check.
C3a0d479bd322818d766a7e869c6c5b2?d=identicon&s=25 Matt Williams (Guest)
on 2009-02-13 03:28
(Received via mailing list)
On Fri, 2009-02-13 at 07:59 +0900, James Gray wrote:
> In a current project we're discussing using SQLite for all application
> logging.

Have you considered splunk?  That way you can do queries and event
correlation easily.

They have a community version which allows you to collect 500MB of logs
a day...
4299e35bacef054df40583da2d51edea?d=identicon&s=25 James Gray (bbazzarrakk)
on 2009-02-13 04:20
(Received via mailing list)
On Feb 12, 2009, at 5:17 PM, Joel VanderWerf wrote:

> The alternative I've found helpful is to set up a drb service that
> lets a process listen for log entries.

That's very interesting.  Thanks for the tip.

James Edward Gray II
4299e35bacef054df40583da2d51edea?d=identicon&s=25 James Gray (bbazzarrakk)
on 2009-02-13 04:23
(Received via mailing list)
On Feb 12, 2009, at 5:27 PM, Tim Pease wrote:

> Have you looked into using CouchDB as the logging destination instead?

On Feb 12, 2009, at 6:08 PM, Ken Bloom wrote:

> Consider a concurrent RDBMS instead if you can deploy it in your
> environment.

I appreciate the suggestions but this is for installed software, so I
want to keep the dependencies pretty light.  That's why I'm thinking
of SQLite.

James Edward Gray II
4299e35bacef054df40583da2d51edea?d=identicon&s=25 James Gray (bbazzarrakk)
on 2009-02-13 04:25
(Received via mailing list)
On Feb 12, 2009, at 8:26 PM, Matt Williams wrote:

> a day...
I need a local solution.  Thanks for the idea though.

James Edward Gray II
C3a0d479bd322818d766a7e869c6c5b2?d=identicon&s=25 Matt Williams (Guest)
on 2009-02-13 05:49
(Received via mailing list)
On Fri, 2009-02-13 at 12:22 +0900, James Gray wrote:
> > They have a community version which allows you to collect 500MB of
> > logs
> > a day...
>
>
> I need a local solution.  Thanks for the idea though.
>

I saw you wanted to keep it light, so yeah, it is probably not the best
solution.
However, you can log to a local database -- as I understand it, that's
what we're doing at work...

Another option would be to use syslog & batch load (every N minutes)
into sqlite....

Good Luck!

Matt
Ae36591847393e58ff189704f5eb18f2?d=identicon&s=25 Jeremy Hinegardner (Guest)
on 2009-02-13 16:53
(Received via mailing list)
On Fri, Feb 13, 2009 at 08:27:42AM +0900, Tim Pease wrote:
>> The minus of course is a slight loss in transparency.  For example, you
> I've started a branch of my logging gem that allows you to do this, but
> I've had to table it due to other obligations.
>
> The goal is to just dump everything to CouchDB with some mnemonics on the
> messages. Then you create views to see all messages for a given
> application. All error messages in the past week, etc.

I'll have to take a look at that and possibly do something similar with
Amalgalite.  Maybe an Amalgalite Appender?  That might also fill what
James is
thinking about.

enjoy,

-jeremy
4299e35bacef054df40583da2d51edea?d=identicon&s=25 James Gray (bbazzarrakk)
on 2009-02-13 18:09
(Received via mailing list)
On Feb 13, 2009, at 9:50 AM, Jeremy Hinegardner wrote:

> I'll have to take a look at that and possibly do something similar
> with
> Amalgalite.  Maybe an Amalgalite Appender?  That might also fill
> what James is thinking about.

Yeah, that would be really neat.

James Edward Gray II
This topic is locked and can not be replied to.