Forum: Ferret Updating Index Is Very Slow

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Tom D. (Guest)
on 2006-03-01 15:47
(Received via mailing list)
Hi,

I am experiencing very poor performance when updating my index.  For
example, to update the index for 10 documents, it is taking 3 to 4
seconds.  My index is currently very small... with probably less than
100 docs in it.

I have created my index as follows:

GIFT_INDEX = Index::Index.new(:path => "#{index_dir}/gift", :key =>
'id', :auto_flush => true)

and I have an after_save filter in my model as follows:

def update_index
    INDEX << self.to_doc
end


Is there anything I can do to improve this performance?

Thanks,
Tom
David B. (Guest)
on 2006-03-01 15:58
(Received via mailing list)
On 3/1/06, Tom D. <removed_email_address@domain.invalid> wrote:
> 'id', :auto_flush => true)
>
> and I have an after_save filter in my model as follows:
>
> def update_index
>     INDEX << self.to_doc
> end
>
>
> Is there anything I can do to improve this performance?

Hi Tom,

That sounds very slow. How large are the documents?

The first thing you can do is turn off auto_flush. That should
substantially speed things up. If you only have one thread you won't
need auto_flush. If you have more then one thread then I'd suggest
having a dedicated indexing thread (and again you won't need
auto_flush). If things are still too slow after that, I'm nearly
finished with the C rewrite of ferret. A linux version should be out
some time next week. This will be at least 10 times as fast.

Hope that helps,

Cheers,
Dave
Gary E. (Guest)
on 2006-03-01 16:19
(Received via mailing list)
That's great, David. Is the C version of ferret similar enough to the
current version so that someone could develop on windows or osx with
the current version of ferret and deploy on linux with cFerret?
David B. (Guest)
on 2006-03-01 16:52
(Received via mailing list)
On 3/1/06, Gary E. <removed_email_address@domain.invalid> wrote:
> That's great, David. Is the C version of ferret similar enough to the
> current version so that someone could develop on windows or osx with
> the current version of ferret and deploy on linux with cFerret?
>

It's similar enough that it won't take long to port but it won't be
exactly the same. The pure ruby version is being phased out, but I do
plan have a windows version of cferret. Hopefully a keen windows
developer will lend a hand. Unfortunately I don't have the microsoft C
compiler.
Tom D. (Guest)
on 2006-03-01 16:57
(Received via mailing list)
Hi Dave,

That is good news about the Linux version.  I tried turning off
autoflush, but that did not appear to have a very noticable
difference.

I just added the following benchmark code around the index update:
  def update_index
    Gift.benchmark("updating index") do
      INDEX << self.to_doc
    end
  end

I am attaching a trace of running this against 28 docs, which is the
entire index.  The documents are not large, as you will see from the
trace.  Each index update is averaging around .4 seconds.  Does that
seem acceptable?  I am hoping this turns out to be some sort of
configuration error.

Tom
David B. (Guest)
on 2006-03-02 03:53
(Received via mailing list)
Hi Tom,

It does seem rather slow. The unit tests are adding a few hundred
documents and they're finishing in under ten seconds on my machine.
I'd suggest you don't worry about it too much though as the new
version of Ferret will solve any performance problems you're having.

Cheers,
Dave
Andreas S. (Guest)
on 2006-03-03 13:43
David B. wrote:
> I'd suggest you don't worry about it too much though as the new
> version of Ferret will solve any performance problems you're having.

I'm looking forward to trying the new version, as I have quite a lot of
problems with the current version on ruby-forum.com.
Gary E. (Guest)
on 2006-03-03 15:35
(Received via mailing list)
Andreas, do you just mean performance problems or other bugs as well?
Andreas S. (Guest)
on 2006-03-03 16:43
Gary E. wrote:
> Andreas, do you just mean performance problems or other bugs as well?


Other bugs as well, occasional crashes, stale lockfiles, exploding index
file size (see bugtracker on Ferret website).
David B. (Guest)
on 2006-03-03 19:56
(Received via mailing list)
On 3/3/06, Andreas S. <removed_email_address@domain.invalid> wrote:
> David B. wrote:
> > I'd suggest you don't worry about it too much though as the new
> > version of Ferret will solve any performance problems you're having.
>
> I'm looking forward to trying the new version, as I have quite a lot of
> problems with the current version on ruby-forum.com.

Once the new version is out I'll be able to be a lot more responsive
when it comes to fixing bugs and addressing other problems. I should
warn though that the new version is an alpha release and it's going to
take some work before it's even as stable as the current ruby version.
It'll get there though.
This topic is locked and can not be replied to.