Forum: Ferret error when indexing in multiple processes at a time

Hi everybody,

I'm working on a search engine for a small niche. I've built a ruby
program that takes a URL as a single argument and then processes and
indexes that page.

Another small ruby application then creates a number of concurrent
processes of the indexer application.

The problem is that the first few pages index well, but after that every
add to the ferret index throws an error like either:
/usr/lib/ruby/gems/1.8/gems/ferret-0.11.6/lib/ferret/index.rb:285:in
`delete': File Not Found Error occured at <except.c>:93 in xraise
(Ferret::FileNotFoundError)
Error occured in fs_store.c:329 - fs_open_input
  tried to open "/home/wykthorr/index/_1.fdx" but it doesn't exist: <No
such file or directory>

  from
/usr/lib/ruby/gems/1.8/gems/ferret-0.11.6/lib/ferret/index.rb:285:in
`<<'
  from
/usr/lib/ruby/gems/1.8/gems/ferret-0.11.6/lib/ferret/index.rb:8:in
`synchrolock'
  from /usr/lib/ruby/1.8/monitor.rb:242:in `synchronize'
  from
/usr/lib/ruby/gems/1.8/gems/ferret-0.11.6/lib/ferret/index.rb:8:in
`synchrolock'
  from
/usr/lib/ruby/gems/1.8/gems/ferret-0.11.6/lib/ferret/index.rb:267:in
`<<'
  from ./index_utils.rb:67:in `add_to_index'
  from ./index_utils.rb:104:in `index_link'
  from indexpage.rb:24:in `process_element'
  from
/usr/lib/ruby/gems/1.8/gems/hpricot-0.8.1/lib/hpricot/traverse.rb:330:in
`each'
  from
/usr/lib/ruby/gems/1.8/gems/hpricot-0.8.1/lib/hpricot/traverse.rb:330:in
`search'
  from indexpage.rb:19:in `process_element'
  from indexpage.rb:58
  from
/usr/lib/ruby/gems/1.8/gems/hpricot-0.8.1/lib/hpricot/traverse.rb:330:in
`each'
  from
/usr/lib/ruby/gems/1.8/gems/hpricot-0.8.1/lib/hpricot/traverse.rb:330:in
`search'
  from indexpage.rb:57

or or or or or or or or or

/usr/lib/ruby/gems/1.8/gems/ferret-0.11.6/lib/ferret/index.rb:285:in
`delete': End-of-File Error occured at <except.c>:117 in xpop_context
(EOFError)
Error occured in store.c:216 - is_refill
  current pos = 0, file length = 0

  from
/usr/lib/ruby/gems/1.8/gems/ferret-0.11.6/lib/ferret/index.rb:285:in
`<<'
  from
/usr/lib/ruby/gems/1.8/gems/ferret-0.11.6/lib/ferret/index.rb:8:in
`synchrolock'
  from /usr/lib/ruby/1.8/monitor.rb:242:in `synchronize'
  from
/usr/lib/ruby/gems/1.8/gems/ferret-0.11.6/lib/ferret/index.rb:8:in
`synchrolock'
  from
/usr/lib/ruby/gems/1.8/gems/ferret-0.11.6/lib/ferret/index.rb:267:in
`<<'
  from ./index_utils.rb:67:in `add_to_index'
  from ./index_utils.rb:104:in `index_link'
  from indexpage.rb:24:in `process_element'
  from
/usr/lib/ruby/gems/1.8/gems/hpricot-0.8.1/lib/hpricot/traverse.rb:330:in
`each'
  from
/usr/lib/ruby/gems/1.8/gems/hpricot-0.8.1/lib/hpricot/traverse.rb:330:in
`search'
  from indexpage.rb:19:in `process_element'
  from indexpage.rb:58
  from
/usr/lib/ruby/gems/1.8/gems/hpricot-0.8.1/lib/hpricot/traverse.rb:330:in
`each'
  from
/usr/lib/ruby/gems/1.8/gems/hpricot-0.8.1/lib/hpricot/traverse.rb:330:in
`search'
  from indexpage.rb:57


Am I doing something wrong. Shouldn't there be mutiple processes using a
ferret index at the same time?

If you need some code please tell me and I'll post it