Search result inconsistencies due to indexing

I seem to be having problems with getting my searcher to be consistent
while indexing. I am running the latest version of ferret (0.10.13) and
I am using the Searchable plugin. Currently the way it indexes is by
using a callback in the model of either after_update or after_create to
index the fields that I have setup to be indexed.

Right now I update the index about once every 4 or 5 minutes. The
problem occurs when I do a search and either no results ( or a very
small amount like 2) are returned for particular keywords that
previously had plenty of results. I am thinking that it has something
to do with indexing or possibly that Searchable does not optimize after
it adds to the index.

I currently have about 10,000 records in my index and am fairly new to
ferret. Any assistance would be helpful. Thanks.

Andy

On Wed, Oct 25, 2006 at 06:41:14AM +0200, William Minkstein wrote:

I seem to be having problems with getting my searcher to be consistent
while indexing. I am running the latest version of ferret (0.10.13) and
I am using the Searchable plugin. Currently the way it indexes is by
using a callback in the model of either after_update or after_create to
index the fields that I have setup to be indexed.

huh, I didn’t ever hear about that plugin before - the DrB remote
indexing stuff is quite interesting indeed.

Right now I update the index about once every 4 or 5 minutes. The
problem occurs when I do a search and either no results ( or a very
small amount like 2) are returned for particular keywords that
previously had plenty of results. I am thinking that it has something
to do with indexing or possibly that Searchable does not optimize after
it adds to the index.

Did you contact the plugin author about this ?

cheers,
Jens


webit! Gesellschaft für neue Medien mbH www.webit.de
Dipl.-Wirtschaftsingenieur Jens Krämer [email protected]
Schnorrstraße 76 Tel +49 351 46766 0
D-01069 Dresden Fax +49 351 46766 66

Jens K. wrote:

On Wed, Oct 25, 2006 at 06:41:14AM +0200, William Minkstein wrote:

I seem to be having problems with getting my searcher to be consistent
while indexing. I am running the latest version of ferret (0.10.13) and
I am using the Searchable plugin. Currently the way it indexes is by
using a callback in the model of either after_update or after_create to
index the fields that I have setup to be indexed.

huh, I didn’t ever hear about that plugin before - the DrB remote
indexing stuff is quite interesting indeed.

The url for author’s page is: http://searchable.rubyforge.org/ There
are things about it that are very convenient.

Right now I update the index about once every 4 or 5 minutes. The
problem occurs when I do a search and either no results ( or a very
small amount like 2) are returned for particular keywords that
previously had plenty of results. I am thinking that it has something
to do with indexing or possibly that Searchable does not optimize after
it adds to the index.

Did you contact the plugin author about this ?

Yes I did contact the author about this. I checked the Searchable code
that indexes records and it seems to be pretty consistent with how
people are indexing with just Ferret itself. I guess I was wondering if
anyone else experienced having inconsistent search results after
updating or adding records to their index.

cheers,
Jens


webit! Gesellschaft f�r neue Medien mbH www.webit.de
Dipl.-Wirtschaftsingenieur Jens Kr�mer [email protected]
Schnorrstra�e 76 Tel +49 351 46766 0
D-01069 Dresden Fax +49 351 46766 66

Andy

On 10/25/06, William (Andy) Minkstein [email protected] wrote:

The url for author’s page is: http://searchable.rubyforge.org/ There
are things about it that are very convenient.

Hehe. I hadn’t seen this either. Seth Fitzsimmons, if you’re out
there, nice work. I’m going to be adding a DRb server to Ferret soon.
I’ll definitely be checking out your code. If you’d like to
contribute, please do. :slight_smile:

Yes I did contact the author about this. I checked the Searchable code
that indexes records and it seems to be pretty consistent with how
people are indexing with just Ferret itself. I guess I was wondering if
anyone else experienced having inconsistent search results after
updating or adding records to their index.

I haven’t seen this problem in version 0.10.13. The way I would go
about debugging it, though, is to store all the fields in the index.
Then if you look at a certain document in the index and it contains
the data you’re searching for but doesn’t get matched in the search
results it is a bug. In this case you can send me a zipped up copy of
the index and I’ll fix the problem. Otherwise I’m not sure there’s
much else we can do (unless you can give me ssh access to your
server).

Cheers,
Dave

Hey David,

Thanks for the quick response. Sorry about the delay in responding, we
had a deadline we had to meet for this app we’re developing and that has
been taking up most of my life.

I am having a lot of trouble reproducing the bug I first reported. I
talked to the writer of the Searchable plugin (Seth) and he is stumped
as well. It seems that creating an index from scratch works fine and
the bug only occurs when documents are added or deleted.

A weird thing happens though once searchable starts updating the index.
It creates folders inside the index directory called development and
production. Seth claims that this is not desired behavior but I guess I
was wondering if having files or directories like that in the index
directory would cause it to behave strangely?

Another question I have is can certain characters in the text that is
being indexed cause problems when trying to retrieve search results?
Should I be filtering out any non-alphanumerics? Not sure if that
matters at all but like I said I am new to this.

Andy

David B. wrote:

On 10/25/06, William (Andy) Minkstein [email protected] wrote:

The url for author’s page is: http://searchable.rubyforge.org/ There
are things about it that are very convenient.

Hehe. I hadn’t seen this either. Seth Fitzsimmons, if you’re out
there, nice work. I’m going to be adding a DRb server to Ferret soon.
I’ll definitely be checking out your code. If you’d like to
contribute, please do. :slight_smile:

Yes I did contact the author about this. I checked the Searchable code
that indexes records and it seems to be pretty consistent with how
people are indexing with just Ferret itself. I guess I was wondering if
anyone else experienced having inconsistent search results after
updating or adding records to their index.

I haven’t seen this problem in version 0.10.13. The way I would go
about debugging it, though, is to store all the fields in the index.
Then if you look at a certain document in the index and it contains
the data you’re searching for but doesn’t get matched in the search
results it is a bug. In this case you can send me a zipped up copy of
the index and I’ll fix the problem. Otherwise I’m not sure there’s
much else we can do (unless you can give me ssh access to your
server).

Cheers,
Dave