Ferret EOFError creating index


#1

I’ve been messing around with Ferret (no punn intended). After spending
some time testing it out (indexing to file), I decided to index about
10% of the data I want to eventually index. It took several hours to
complete the index on my local machine, but it was created without any
problems and after optimising it the searches returned results at the
sort of speed I was expecting. I then continuned to add a further 10% of
the overall data to the index and it all ran through fine for a few
hours until it bombed out with the error below.

I’ve been running this on my local machine and the first run was left
overnight, whereas the second run was going on while I was using my
machine for other things. Which leads me to suspect that it’s a
memory-related issue that made it bail out.

so I have 2 questions:

  1. does anyone know what caused the error or have any suggestions to
    prevent it in future?
  2. I can’t search on the index without getting errors so is the index
    recoverable or will I need to recreate the index again?

Thanks in advance,
Derek

EOFError

c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:178:in
refill' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:94:inread_byte’
c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:109:in
read_bytes' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:108:inupto’
c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:108:in
read_bytes' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_reader.rb:286:inget_norms_into’
c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_reader.rb:271:in
synchronize' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_reader.rb:271:inget_norms_into’
c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:341:in
merge_norms' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:338:ineach’
c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:338:in
merge_norms' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:334:ineach_with_index’
c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:in
each_with_index' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:ineach’
c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:in
each_with_index' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:ineach_with_index’
c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:334:in
merge_norms' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:49:inmerge’
c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:403:in
merge_segments' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:351:inflush_ram_segments’
c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:127:in
close' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:126:insynchronize’
c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:126:in
close' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index.rb:173:inclose’
c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index.rb:168:in
synchronize' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index.rb:168:inclose’


#2

Hi Derek,

I’m not sure why this happens. But it seems to only happen with the
pure Ruby version so I’d recommend waiting for the Windows port of the
C version. You’ll also be done in minutes rather than hours with about
100 times speed up of the indexing. I will get around to trying to fix
this bug in the pure Ruby version eventually but don’t hold your
breath. Patches welcome!

Cheers,
Dave