Updating Index Is Very Slow

Hi,

I am experiencing very poor performance when updating my index. For
example, to update the index for 10 documents, it is taking 3 to 4
seconds. My index is currently very small… with probably less than
100 docs in it.

I have created my index as follows:

GIFT_INDEX = Index::Index.new(:path => “#{index_dir}/gift”, :key =>
‘id’, :auto_flush => true)

and I have an after_save filter in my model as follows:

def update_index
INDEX << self.to_doc
end

Is there anything I can do to improve this performance?

Thanks,
Tom

On 3/1/06, Tom D. [email protected] wrote:

‘id’, :auto_flush => true)

and I have an after_save filter in my model as follows:

def update_index
INDEX << self.to_doc
end

Is there anything I can do to improve this performance?

Hi Tom,

That sounds very slow. How large are the documents?

The first thing you can do is turn off auto_flush. That should
substantially speed things up. If you only have one thread you won’t
need auto_flush. If you have more then one thread then I’d suggest
having a dedicated indexing thread (and again you won’t need
auto_flush). If things are still too slow after that, I’m nearly
finished with the C rewrite of ferret. A linux version should be out
some time next week. This will be at least 10 times as fast.

Hope that helps,

Cheers,
Dave

That’s great, David. Is the C version of ferret similar enough to the
current version so that someone could develop on windows or osx with
the current version of ferret and deploy on linux with cFerret?

Hi Dave,

That is good news about the Linux version. I tried turning off
autoflush, but that did not appear to have a very noticable
difference.

I just added the following benchmark code around the index update:
def update_index
Gift.benchmark(“updating index”) do
INDEX << self.to_doc
end
end

I am attaching a trace of running this against 28 docs, which is the
entire index. The documents are not large, as you will see from the
trace. Each index update is averaging around .4 seconds. Does that
seem acceptable? I am hoping this turns out to be some sort of
configuration error.

Tom

Hi Tom,

It does seem rather slow. The unit tests are adding a few hundred
documents and they’re finishing in under ten seconds on my machine.
I’d suggest you don’t worry about it too much though as the new
version of Ferret will solve any performance problems you’re having.

Cheers,
Dave

On 3/1/06, Gary E. [email protected] wrote:

That’s great, David. Is the C version of ferret similar enough to the
current version so that someone could develop on windows or osx with
the current version of ferret and deploy on linux with cFerret?

It’s similar enough that it won’t take long to port but it won’t be
exactly the same. The pure ruby version is being phased out, but I do
plan have a windows version of cferret. Hopefully a keen windows
developer will lend a hand. Unfortunately I don’t have the microsoft C
compiler.

Andreas, do you just mean performance problems or other bugs as well?

David B. wrote:

I’d suggest you don’t worry about it too much though as the new
version of Ferret will solve any performance problems you’re having.

I’m looking forward to trying the new version, as I have quite a lot of
problems with the current version on ruby-forum.com.

Gary E. wrote:

Andreas, do you just mean performance problems or other bugs as well?

Other bugs as well, occasional crashes, stale lockfiles, exploding index
file size (see bugtracker on Ferret website).

On 3/3/06, Andreas S. [email protected] wrote:

David B. wrote:

I’d suggest you don’t worry about it too much though as the new
version of Ferret will solve any performance problems you’re having.

I’m looking forward to trying the new version, as I have quite a lot of
problems with the current version on ruby-forum.com.

Once the new version is out I’ll be able to be a lot more responsive
when it comes to fixing bugs and addressing other problems. I should
warn though that the new version is an alpha release and it’s going to
take some work before it’s even as stable as the current ruby version.
It’ll get there though.