Forum: Ferret Is indexing slower?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
C1012b1169c4cc64a2050b5f218641de?d=identicon&s=25 Matt Schnitz (Guest)
on 2007-03-05 03:17
(Received via mailing list)
Hi -  I upgraded to Ferret 0.11.3 from 0.10.13.

I used to index 10,000 records in 10 secs.  Now it takes 13 minutes.
(That's a factor of ~75x)

Did something change in the flush semantics, or something?


Thanks!


Schnitz
B5e329ffa0cc78efbfc7ae2d084c149f?d=identicon&s=25 David Balmain (Guest)
on 2007-03-06 04:48
(Received via mailing list)
On 3/5/07, Matt Schnitz <matt@mattschnitz.com> wrote:
> Hi -  I upgraded to Ferret 0.11.3 from 0.10.13.
>
> I used to index 10,000 records in 10 secs.  Now it takes 13 minutes.
> (That's a factor of ~75x)
>
> Did something change in the flush semantics, or something?

Hi Matt,

The opening of an index takes a little longer now. I guess if you have
the index set to :auto_flush then it could take a fair bit longer but
I didn't expect it to take that much longer. Unfortunately this
slowdown was a price I had to pay to prevent the segfault and
FileNotFound errors that people where getting. Having said that, you
shouldn't have :auto_flush set when you are batch indexing anyway.

If you send me a benchmark which approximates what you are doing, I'd
be happy to take a look at it for you and tell you how to make it
faster or add a fix to Ferret if the problem does happen to be at this
end.

Cheers,
Dave
C1012b1169c4cc64a2050b5f218641de?d=identicon&s=25 Matt Schnitz (Guest)
on 2007-03-06 06:04
(Received via mailing list)
Figured it out.  It's interesting, academically.

I was flushing every time I added something to the index.  I forget
exactly
why I thought that was a good idea, but that's what I was doing.

Apparently, that's a bad idea under 0.11.  Worked fine under 0.10, not
so
fine under 0.11.  Like, 75x less fine.

Now I'm just flushing after every batch.  It's back to reasonable,
5-seconds-for-10,000-small-records performance.

Thanks for all your help, Dave.


Schnitz
This topic is locked and can not be replied to.