Find_id_by_contents crashing

Ruby Apps Ferret
#1

Hi-

I asked this question over on Ruby Forum, but it never made it to this
list, so I’m posting it directly now-

===

I’ve got an index with over 11M entries. Both server and clients are
running Ferret gem 0.11.4.

If I do “UserMessage.find_id_by_contents(‘kenn*’, :limit => 49)”, it
works fine. If I do " UserMessage.find_id_by_contents(‘kenn*’, :limit
=> 50)", it throws an exception, like this:

UserMessage.find_id_by_contents ‘kenn*’, :limit => 50
IOError: IO Error occured at <except.c>:93 in xraise
Error occured in fs_store.c:293 - fsi_seek_i
seeking pos -1473943740:
from (druby://ferret.lingr.com:9009) /usr/lib/ruby/gems/1.8/
gems/ferret-0.11.4/lib/ferret/index.rb:411:in []' from (druby://ferret.lingr.com:9009) /usr/lib/ruby/gems/1.8/ gems/ferret-0.11.4/lib/ferret/index.rb:411:in[]’
from (druby://ferret.lingr.com:9009) /usr/lib/ruby/1.8/
monitor.rb:229:in synchronize' from (druby://ferret.lingr.com:9009) /usr/lib/ruby/gems/1.8/ gems/ferret-0.11.4/lib/ferret/index.rb:403:in[]’
… yada yada yada …

Any idea what I can do to recover from this? Rebuilding this index
would take a LOOOOOOOOOOOOONG time :slight_smile:

#2

Hi,

I’m not sure but the negative seeking pos could be caused by an overflow
of some sort. Maybe a large file problem? What does your index look like
with ‘ls -l’?

Could you try reproducing this on a copy of your index with a small
script with plain Ferret, and then retry with 0.11.6?

Cheers,
Jens

On Wed, Feb 06, 2008 at 09:33:00AM -0800, Danny B. wrote:

If I do “UserMessage.find_id_by_contents(‘kenn*’, :limit => 49)”, it
gems/ferret-0.11.4/lib/ferret/index.rb:411:in `[]’
http://www.lingr.com/help/about#danny

Ferret-talk mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ferret-talk

–
Jens Krämer
Finkenlust 14, 06449 Aschersleben, Germany
VAT Id DE251962952
http://www.jkraemer.net/ - Blog
http://www.omdb.org/ - The new free film database

#3

I’m not sure but the negative seeking pos could be caused by an
overflow
of some sort. Maybe a large file problem? What does your index look
like
with ‘ls -l’?

I’ve attached the listing below…

Could you try reproducing this on a copy of your index with a small
script with plain Ferret, and then retry with 0.11.6?

Will do- I’ll post the results here soon.

Best Regards,

Danny

===

[email protected]:/var/ferret/production/user_message$ ls -l
total 4041936
-rw------- 1 root root 3160 2007-08-10 07:01
_2ti3z.fdt.saved
-rw------- 1 root root 8 2007-08-10 07:01
_2ti3z.fdx.saved
-rw------- 1 root root 1326528 2008-02-07 01:17 _7bgxh_5d3.del
-rw------- 1 root root 3861431389 2008-01-27 08:36 _7bgxh.cfs
-rw------- 1 root root 10067 2008-02-03 11:17 _7dutf_5.del
-rw------- 1 root root 37559822 2008-01-28 22:19 _7dutf.cfs
-rw------- 1 root root 37273935 2008-01-30 06:41 _7g8n1.cfs
-rw------- 1 root root 11371 2008-01-31 17:30 _7g8n1_ox.del
-rw------- 1 root root 11559 2008-02-02 20:57 _7imep_2.del
-rw------- 1 root root 35903751 2008-01-31 14:44 _7imep.cfs
-rw------- 1 root root 36105008 2008-02-02 04:11 _7l096.cfs
-rw------- 1 root root 10511 2008-02-03 18:26 _7l096_g.del
-rw------- 1 root root 11659 2008-02-03 20:12 _7ne15_0.del
-rw------- 1 root root 36128119 2008-02-03 13:04 _7ne15.cfs
-rw------- 1 root root 34444491 2008-02-05 03:40 _7prs3.cfs
-rw------- 1 root root 36773717 2008-02-06 13:41 _7s5pu.cfs
-rw------- 1 root root 3380507 2008-02-06 18:49 _7seal.cfs
-rw------- 1 root root 3260942 2008-02-06 23:37 _7smva.cfs
-rw------- 1 root root 54 2008-02-07 04:55 _7svfx_0.del
-rw------- 1 root root 3590095 2008-02-07 04:01 _7svfx.cfs
-rw------- 1 root root 4030326 2008-02-07 06:32 _7t40k.cfs
-rw------- 1 root root 347713 2008-02-07 06:48 _7t4vf.cfs
-rw------- 1 root root 348108 2008-02-07 07:03 _7t5qa.cfs
-rw------- 1 root root 359057 2008-02-07 07:16 _7t6l5.cfs
-rw------- 1 root root 353181 2008-02-07 07:28 _7t7g0.cfs
-rw------- 1 root root 397144 2008-02-07 07:41 _7t8av.cfs
-rw------- 1 root root 343822 2008-02-07 07:52 _7t95q.cfs
-rw------- 1 root root 348366 2008-02-07 08:04 _7ta0l.cfs
-rw------- 1 root root 355600 2008-02-07 08:14 _7tavg.cfs
-rw------- 1 root root 365283 2008-02-07 08:26 _7tbqb.cfs
-rw------- 1 root root 34225 2008-02-07 08:28 _7tbte.cfs
-rw------- 1 root root 36869 2008-02-07 08:29 _7tbwh.cfs
-rw------- 1 root root 37095 2008-02-07 08:30 _7tbzk.cfs
-rw------- 1 root root 36708 2008-02-07 08:31 _7tc2n.cfs
-rw------- 1 root root 36468 2008-02-07 08:32 _7tc5q.cfs
-rw------- 1 root root 38334 2008-02-07 08:33 _7tc8t.cfs
-rw------- 1 root root 4941 2008-02-07 08:33 _7tc94.cfs
-rw------- 1 root root 5270 2008-02-07 08:34 _7tc9f.cfs
-rw------- 1 root root 5070 2008-02-07 08:34 _7tc9q.cfs
-rw------- 1 root root 6055 2008-02-07 08:34 _7tca1.cfs
-rw------- 1 root root 5695 2008-02-07 08:34 _7tcac.cfs
-rw------- 1 root root 0 2008-02-07 07:27 ferret-
write.lck
-rw------- 1 dburkes dburkes 16 2008-02-07 08:34 segments
-rw------- 1 root root 733 2008-02-07 08:34 segments_832jn

#4

On Thu, Feb 07, 2008 at 08:44:44AM -0800, Danny B. wrote:
[…]

[email protected]:/var/ferret/production/user_message$ ls -l
total 4041936
-rw------- 1 root root 3160 2007-08-10 07:01
_2ti3z.fdt.saved
-rw------- 1 root root 8 2007-08-10 07:01
_2ti3z.fdx.saved
-rw------- 1 root root 1326528 2008-02-07 01:17 _7bgxh_5d3.del
-rw------- 1 root root 3861431389 2008-01-27 08:36 _7bgxh.cfs
[…]

Ok, might be a large file issue. There has been at least one patch [1]
after 0.11.4 was released regarding LFS, so upgrading to 0.11.6 might
indeed help.

0.11.4 and 0.11.6 are index compatible, so no rebuild should be
necessary.

Btw, from your directory listing it looks like you’re running the DRb
server as root - please don’t do that.

Given proper write access to the index and log directories it’s not
necessary at all. I usually run it as the same (non-root) user that’s
running the Mongrels.

Cheers,
Jens

[1] http://ferret.davebalmain.com/trac/ticket/215

–
Jens Krämer
webit! Gesellschaft für neue Medien mbH
Schnorrstraße 76 | 01069 Dresden
Telefon +49 351 46766-0 | Telefax +49 351 46766-66
[email protected] | www.webit.de

Amtsgericht Dresden | HRB 15422
GF Sven Haubold

#5

Ok, might be a large file issue. There has been at least one patch [1]
after 0.11.4 was released regarding LFS, so upgrading to 0.11.6 might
indeed help.

0.11.4 and 0.11.6 are index compatible, so no rebuild should be
necessary.

Thanks, Jens! I upgraded to the 0.11.6 gem, and the problem went away.

Best Regards,

Danny