# Writing to ferret index from multiple processes

**URL:** <https://www.ruby-forum.com/t/writing-to-ferret-index-from-multiple-processes/71966>\
**Category:** Ruby\
**Created:** [December 18, 2005, 11:08pm UTC](https://www.ruby-forum.com/t/writing-to-ferret-index-from-multiple-processes/71966 "2005-12-18T23:08:59Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![andreas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/ee7513/32.png) [@andreas](https://www.ruby-forum.com/u/andreas)\
**Post date:** [December 18, 2005, 11:08pm UTC](https://www.ruby-forum.com/t/writing-to-ferret-index-from-multiple-processes/71966/1 "2005-12-18T23:08:59Z")

</div>

Hi,

what do I have to do to be able to write a ferret index from multiple  
processes at the same time?

I was indexing a lot of documents with a script when another process  
made a change to the index; suddenly all of the imported data was gone  
from the index, and the import script quit with the exception  
“Errno::ENOENT: No such file or directory - ./ferret\_index/\_1ah.fnm”.

auto\_flush =\> true didn’t help. Is there something else?

Andreas

---

<div class="post-metadata">

**Author:** ![andreas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/ee7513/32.png) [@andreas](https://www.ruby-forum.com/u/andreas)\
**Post date:** [December 19, 2005, 4:06am UTC](https://www.ruby-forum.com/t/writing-to-ferret-index-from-multiple-processes/71966/2 "2005-12-19T04:06:00Z")

</div>

Hi Andreas,

Can you show me some more code? How are you creating the index?  
Perhaps you are setting :create =\> true in which case it will  
overwrite the old index.

Dave

---

<div class="post-metadata">

**Author:** ![andreas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/ee7513/32.png) [@andreas](https://www.ruby-forum.com/u/andreas)\
**Post date:** [December 19, 2005, 8:01am UTC](https://www.ruby-forum.com/t/writing-to-ferret-index-from-multiple-processes/71966/3 "2005-12-19T08:01:23Z")

</div>

David B. wrote:

> Hi Andreas,
> 
> Can you show me some more code? How are you creating the index?  
> Perhaps you are setting :create =\> true in which case it will  
> overwrite the old index.
> 
> Dave

Oops. I am indeed using :create =\> true. I forgot that I set it because  
create\_if\_missing did not work.

I removed it, but now there is a different problem. When I change the  
index while the indexing script is running, it quits, but with another  
error message:

316  
317  
318  
RuntimeError: docs out of order curent doc = 9 and previous doc = 17  
from  
/usr/local/lib/ruby/gems/1.8/gems/ferret-0.3.2/lib/ferret/index/segment\_merger.rb:276:in  
`append_postings' from /usr/local/lib/ruby/gems/1.8/gems/ferret-0.3.2/lib/ferret/index/segment_merger.rb:262:in`append\_postings’  
from  
/usr/local/lib/ruby/gems/1.8/gems/ferret-0.3.2/lib/ferret/index/segment\_merger.rb:240:in  
`merge_term_info' from /usr/local/lib/ruby/gems/1.8/gems/ferret-0.3.2/lib/ferret/index/segment_merger.rb:215:in`merge\_term\_infos’  
from  
/usr/local/lib/ruby/gems/1.8/gems/ferret-0.3.2/lib/ferret/index/segment\_merger.rb:176:in  
`merge_terms' from /usr/local/lib/ruby/gems/1.8/gems/ferret-0.3.2/lib/ferret/index/segment_merger.rb:48:in`merge’  
from  
/usr/local/lib/ruby/gems/1.8/gems/ferret-0.3.2/lib/ferret/index/index\_writer.rb:403:in  
`merge_segments' from /usr/local/lib/ruby/gems/1.8/gems/ferret-0.3.2/lib/ferret/index/index_writer.rb:371:in`maybe\_merge\_segments’  
from  
/usr/local/lib/ruby/gems/1.8/gems/ferret-0.3.2/lib/ferret/index/index\_writer.rb:161:in  
`add_document' from /usr/local/lib/ruby/1.8/monitor.rb:229:in`synchronize’  
from  
/usr/local/lib/ruby/gems/1.8/gems/ferret-0.3.2/lib/ferret/index/index\_writer.rb:159:in  
`add_document' from /usr/local/lib/ruby/gems/1.8/gems/ferret-0.3.2/lib/ferret/index/index.rb:270:in`\<\<’  
from /usr/local/lib/ruby/1.8/monitor.rb:229:in `synchronize' from /usr/local/lib/ruby/gems/1.8/gems/ferret-0.3.2/lib/ferret/index/index.rb:238:in`\<\<’  
from ./app/models/search\_ferret.rb:38:in `update’  
from (irb):1

---

<div class="post-metadata">

**Author:** ![andreas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/ee7513/32.png) [@andreas](https://www.ruby-forum.com/u/andreas)\
**Post date:** [December 19, 2005, 11:07am UTC](https://www.ruby-forum.com/t/writing-to-ferret-index-from-multiple-processes/71966/4 "2005-12-19T11:07:43Z")

</div>

I’m not to sure about this one. Are you by any chance explicitely  
deleting the lock files when your app starts up? I’ve seen a few  
people do that. The only way I can see doc numbers getting out of  
order is if you delete the lock files. Any chance I could look at more  
of your code? Is this for RForum? Perhaps I could check it out of svn.  
Anyway, I hope I can help you out with this.

Dave

PS: If you are interested you should join the Ferret mailing list. You  
seem to be doing some more advanced stuff judging from the bugs you’re  
finding. 😉

---

<div class="post-metadata">

**Author:** ![andreas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/ee7513/32.png) [@andreas](https://www.ruby-forum.com/u/andreas)\
**Post date:** [December 19, 2005, 2:20pm UTC](https://www.ruby-forum.com/t/writing-to-ferret-index-from-multiple-processes/71966/5 "2005-12-19T14:20:35Z")

</div>

David B. wrote:

> I’m not to sure about this one. Are you by any chance explicitely  
> deleting the lock files when your app starts up?

No.

> I’ve seen a few  
> people do that. The only way I can see doc numbers getting out of  
> order is if you delete the lock files. Any chance I could look at more  
> of your code? Is this for RForum? Perhaps I could check it out of svn.

It is for RForum. You can see the the code here:  
[http://rforum.andreas-s.net/trac/file/trunk/app/models/search\_ferret.rb](http://rforum.andreas-s.net/trac/file/trunk/app/models/search_ferret.rb)

My indexing script simply fetches all the posts from the database and  
calls Post.search\_handler.update(post) for each one. If another process  
calls the update method while this script is running, I am getting the  
exception. If you need more information to reproduce the problem, please  
let me know.

> PS: If you are interested you should join the Ferret mailing list. You  
> seem to be doing some more advanced stuff judging from the bugs you’re  
> finding. 😉

I didn’t know there was a list. I will definetely join it.

Thanks for fixing the other bugs so quickly.

Andreas

---

<div class="post-metadata">

**Author:** ![andreas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/ee7513/32.png) [@andreas](https://www.ruby-forum.com/u/andreas)\
**Post date:** [December 19, 2005, 8:29pm UTC](https://www.ruby-forum.com/t/writing-to-ferret-index-from-multiple-processes/71966/6 "2005-12-19T20:29:30Z")

</div>

David B. wrote:

> Hey Andreas,
> 
> The latest version of RForum still has :create =\> true so I’m guessing  
> you haven’t checked in your latest changes. Could you let me know when  
> you have?

I have checked it in.

---

<div class="post-metadata">

**Author:** ![andreas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/ee7513/32.png) [@andreas](https://www.ruby-forum.com/u/andreas)\
**Post date:** [December 19, 2005, 8:45pm UTC](https://www.ruby-forum.com/t/writing-to-ferret-index-from-multiple-processes/71966/7 "2005-12-19T20:45:06Z")

</div>

Andreas S. wrote:

> David B. wrote:
> 
> > Hey Andreas,
> > 
> > The latest version of RForum still has :create =\> true so I’m guessing  
> > you haven’t checked in your latest changes. Could you let me know when  
> > you have?
> 
> I have checked it in.

Btw, I tried it again on another machine, and couldn’t reproduce the  
“docs out of order” exception, but instead I got  
RuntimeError: could not obtain lock:  
./ferret\_index/ferret-f62496686e637eca67e933a9cdc5eb21write.lock

---

<div class="post-metadata">

**Author:** ![andreas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/ee7513/32.png) [@andreas](https://www.ruby-forum.com/u/andreas)\
**Post date:** [December 19, 2005, 4:56pm UTC](https://www.ruby-forum.com/t/writing-to-ferret-index-from-multiple-processes/71966/8 "2005-12-19T16:56:55Z")

</div>

Hey Andreas,

The latest version of RForum still has :create =\> true so I’m guessing  
you haven’t checked in your latest changes. Could you let me know when  
you have?

Cheers,  
Dave

---

<div class="post-metadata">

**Author:** ![andreas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/ee7513/32.png) [@andreas](https://www.ruby-forum.com/u/andreas)\
**Post date:** [December 20, 2005, 2:47am UTC](https://www.ruby-forum.com/t/writing-to-ferret-index-from-multiple-processes/71966/9 "2005-12-20T02:47:06Z")

</div>

Hi Andreas,

This is what I would expect to happen. What machine where you running  
it on the first time. Whatever it was, Ferret’s locking mechanism must  
not work.

Anyway, to avoid this problem you need to make sure the batch process  
doesn’t keep the lock for too long (about 5 seconds). I would change  
the rebuild index method to use an IndexWriter or switch auto\_flush to  
false. This should speed the reindexing up. I’d also add a pause in  
there so other processes can get a hold of the lock if they need to.  
Since you are flushing explicitly you may as well set auto\_flush to  
false anyway.

def index  
@index ||= Index::Index.new(:path =\> @path,  
#:auto\_flush =\>true \<= don’t use this  
anymore  
:default\_search\_field =\> [‘subject’],  
:key =\> [‘id’, ‘class’])  
end

# update will continue to work, handling the flushing explicitly

def update(post)  
index \<\< create\_doc(post)  
index.flush  
end

# batch\_update will keep the IndexWriter open between updates

# so it will run much faster

def batch\_update(post)  
index \<\< create\_doc(post)  
end

# define a flush method for use with the batch\_update method

def flush  
index.flush  
end

Then in your process that is doing the reindex I’d use the  
batch\_update method and I might even add some pauses in there.  
Something like this;  
MAX\_ADDS\_BEFORE\_FLUSH = 10  
def rebuild\_index  
i = 0  
Post.find\_all\_by\_deleted(0).each do |post|  
self.update(post)  
i += 1  
if (i % MAX\_ADDS\_BEFORE\_FLUSH) == 0  
self.flush  
sleep(0.5)  
end  
end  
end

These are just ideas. You’ll probably come up with something better. I  
think the best solution is just to keep the Ferret index in sync with  
the database so that you don’t need to reindex everything.

Let me know what kind of system you were running it on the first time  
to get the documents out of order error. I’ll see if I can find out  
why the locking wasn’t working.

Cheers,  
Dave