Problems with Ferret in RForum


#1

Hi,

I have been using Ferret on ruby-forum.com for a few days. While the
searching works with reasonable performance, there are a lot of problems
related to indexing. Sometimes the process hangs with 100% CPU usage,
sometimes it leaves Lockfiles in the directory (causing other processes
to fail while one ist still working), sometimes it even crashes with a
segfault.

I’m going to try to use only one indexing process to solve the locking
problems. A process that looks periodically for posts that need to be
indexed is probably the most robust solution.

Andreas


#2

Andreas S. wrote:

I’m going to try to use only one indexing process to solve the locking
problems. A process that looks periodically for posts that need to be
indexed is probably the most robust solution.

You must’ve tried this already, just checking - did you set :auto_flush
=> true while creating your index? The documentation says it prevents
locking when multiple processes are accessing the index.

Vamsee.


#3

Vamsee K. wrote:

Andreas S. wrote:

I’m going to try to use only one indexing process to solve the locking
problems. A process that looks periodically for posts that need to be
indexed is probably the most robust solution.

You must’ve tried this already, just checking - did you set :auto_flush
=> true while creating your index?

Yes.

The documentation says it prevents
locking when multiple processes are accessing the index.

Locking usually works correctly, but sometimes I get lockfiles that
are never removed. What I don’t understand is why one of the proceses is
still able to write to the index then, while the others aren’t.


#4

Hi, Andreas,

maybe a stupid guess: Maybe because this one process is the locking one?

Regards
Jan


#5

Jan P. wrote:

Hi, Andreas,

maybe a stupid guess: Maybe because this one process is the locking one?

Regards
Jan

I don’t know. I have two fastcgi processes, and it seems that they are
actually both able to write to the index (at least I never got
exceptions when I created a post in the forum), only the third process
(mail receiver) isn’t.

Andreas


#6

Hi Andreas,

I am using a modified version of your Ferret code in my application,
which is still in development mode, so no actual user testing has been
done. I also encountered the locking problem. What I did was remove the
auto_flush => true and handle flushing myself. So in the create_doc
method I placed:

index << doc
index.flush

at the end and removed your last line ‘doc’. Which also made the update
method absolete, so I made update an alias of create_doc . I haven’t had
any locks during my testing, but I haven’t done any real world testing
yet. So I’m eager to know how you managed to fix it, once it is fixed.

Kind regards,

Nick

Andreas S. wrote:

Hi,

I have been using Ferret on ruby-forum.com for a few days. While the
searching works with reasonable performance, there are a lot of problems
related to indexing. Sometimes the process hangs with 100% CPU usage,
sometimes it leaves Lockfiles in the directory (causing other processes
to fail while one ist still working), sometimes it even crashes with a
segfault.

I’m going to try to use only one indexing process to solve the locking
problems. A process that looks periodically for posts that need to be
indexed is probably the most robust solution.

Andreas


#7

Nick S. wrote:

Hi Andreas,

I am using a modified version of your Ferret code in my application,
which is still in development mode, so no actual user testing has been
done. I also encountered the locking problem. What I did was remove the
auto_flush => true and handle flushing myself.

I tried that too, but had the same problem as before.

at the end and removed your last line ‘doc’. Which also made the update
method absolete, so I made update an alias of create_doc . I haven’t had
any locks during my testing, but I haven’t done any real world testing
yet. So I’m eager to know how you managed to fix it, once it is fixed.

I am now using a process that checks periodically for new/changed posts
and updates the index. This seems to be the best solution. You can find
the code in RForum SVN.

Andreas