Forum: Rails deployment sphinx vs ferret

26ca684555abbdb44f01bd1e165e649b?d=identicon&s=25 Vince W. (vince)
on 2008-01-04 17:28
(Received via mailing list)
I've got a smallish site with not a ton of data at the moment.. but
all that could change at some point so I'd like to plan with that in
mind.  Currently I'm deployed on an nginx/mongrel stack that works
quite well.  My site uses Ferret for search and it's ok.. the big
problem is that some terms don't show up as expected.. especially if
there are apostrophes, plurals, etc involved.

I've got two choices that I see... pony up the O'reilly mini-pdf and
tweak ferret settings or scrap ferret and go with Sphinx (and hope it
handles cases like this better).  I'm not sure how much time the
latter would take me but, assuming that I'm going to spend somewhere
around 40 hours anyway, which route would you all recommend?

Thanks for your time,
Vince
A82ba1167f4d4a8d1de63820e576a87f?d=identicon&s=25 Robby Russell (Guest)
on 2008-01-04 17:34
(Received via mailing list)
On Jan 4, 2008, at 8:26 AM, Vince Wadhwani wrote:

> handles cases like this better).  I'm not sure how much time the
> latter would take me but, assuming that I'm going to spend somewhere
> around 40 hours anyway, which route would you all recommend?
>

We've used ferret on past projects... and now use sphinx. We're not
likely going back to ferret. ;-)

Robby

--
Robby Russell
Founder and Executive Director

PLANET ARGON, LLC
Design, Development, and Hosting with Ruby on Rails

http://www.planetargon.com/
http://www.robbyonrails.com/

+1 503 445 2457
+1 877 55 ARGON [toll free]
+1 815 642 4068 [fax]
71af5d93d5666f58c0f3d7fc7bd1b528?d=identicon&s=25 Ericson Smith (Guest)
on 2008-01-04 17:54
(Received via mailing list)
If you consider using Postgresql, then tsearch2 is awesome. Its built
into
the latest version of Postgresql.

- Ericson Smith
CTO
http://www.funadvice.com
2f9a03aa0fcfe945229cb6126eda2cb2?d=identicon&s=25 Philip Hallstrom (Guest)
on 2008-01-04 18:23
(Received via mailing list)
>> latter would take me but, assuming that I'm going to spend somewhere
>> around 40 hours anyway, which route would you all recommend?
>>
>
> We've used ferret on past projects... and now use sphinx. We're not
> likely going back to ferret. ;-)

Can you elaborate on why?  I'm mostly just curious :)

To the parent...

the ferret PDF booklet is pretty full of good information
if you stick with ferret.  I don't however remember if it discusses how
to
handle words with apostrophes in it.  It does talk about how to hand
plurals via the StemFilter though.

http://ferret.davebalmain.com/api/classes/Ferret/A...

-philip
6076c22b65b36f5d75c30bdcfb2fda85?d=identicon&s=25 Ezra Zygmuntowicz (Guest)
on 2008-01-04 21:37
(Received via mailing list)
On Jan 4, 2008, at 11:41 AM, Philip Hallstrom wrote:

>>> it
> To the parent...
>
> the ferret PDF booklet is pretty full of good information
> if you stick with ferret.  I don't however remember if it discusses
> how to
> handle words with apostrophes in it.  It does talk about how to hand
> plurals via the StemFilter though.
>
> http://ferret.davebalmain.com/api/classes/Ferret/A...
>
> -philip


  Ferret is unstable in production. Segfaults, corrupted indexes
galore. We've switched around 40 clients form ferret to sphinx and
solved their problems this way. I will never use ferret again after
all the problems I have seen it cause peoples production apps.

  Plus sphinx can reindex many many times faster then ferret and uses
less cpu and memory as well.

Cheers-
- Ezra Zygmuntowicz
-- Founder & Software Architect
-- ezra@engineyard.com
-- EngineYard.com
38a02bf7121a81be5be6f3d488ce23b5?d=identicon&s=25 Alexey Verkhovsky (Guest)
on 2008-01-04 22:10
(Received via mailing list)
> Ferret is unstable in production
Very true.

A decent search option is Lucene via acts_as_solr plugin.
I never used Sphynx though. Can anyone with firsthand experience of
both Lucene and Sphynx give their opinion?

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]
2f9a03aa0fcfe945229cb6126eda2cb2?d=identicon&s=25 Philip Hallstrom (Guest)
on 2008-01-04 22:30
(Received via mailing list)
>>>> handles cases like this better).  I'm not sure how much time the
>>
>
>   Ferret is unstable in production. Segfaults, corrupted indexes
> galore. We've switched around 40 clients form ferret to sphinx and
> solved their problems this way. I will never use ferret again after
> all the problems I have seen it cause peoples production apps.

Huh.  I must be lucky.  Or not have that much to index (true) or users
don't complain about not finding anything (probably very true)

:-)

I'll have t ogive sphinx a go next time around... thanks ezra
67d3880bbfaafdb2a2108b7c9a64df47?d=identicon&s=25 Adrian Madrid (Guest)
on 2008-01-04 22:36
(Received via mailing list)
Ferret has been very unstable for us. It is unfortunate because it seems
like it would be more customizable than Sphinx. But I must admit that I
like
that Sphinx can take the data by itself from MySQL and index it really
fast.
AEM

On Jan 4, 2008 1:37 PM, Ezra Zygmuntowicz <ezmobius@gmail.com> wrote:

> >>> there are apostrophes, plurals, etc involved.
> >> likely going back to ferret. ;-)
> >
>
> >
>


--
Adrian Esteban Madrid
Lead Developer, Prefab Markets
http://www.prefabmarkets.com
6076c22b65b36f5d75c30bdcfb2fda85?d=identicon&s=25 Ezra Zygmuntowicz (Guest)
on 2008-01-04 22:40
(Received via mailing list)
On Jan 4, 2008, at 1:09 PM, Alexey Verkhovsky wrote:

>
>> Ferret is unstable in production
> Very true.
>
> A decent search option is Lucene via acts_as_solr plugin.
> I never used Sphynx though. Can anyone with firsthand experience of
> both Lucene and Sphynx give their opinion?
>
> --
> Alexey Verkhovsky


  We have a bunch of clients using solr as well. In general it is more
powerful then sphinx but a lot slower to reindex and querey. Also it
uses 50 times the memory of sphinx. If you have a box or vm to put
SOLR on by itself then it is a good option as well. but if sphinx can
do everything you need from a a search indexer then it is a way better
option cost wise.

Cheers-
- Ezra Zygmuntowicz
-- Founder & Software Architect
-- ezra@engineyard.com
-- EngineYard.com
30b3f6355fad8461e37de7891753d810?d=identicon&s=25 John Leach (Guest)
on 2008-01-04 22:41
(Received via mailing list)
On Fri, 2008-01-04 at 12:37 -0800, Ezra Zygmuntowicz wrote:
>   Ferret is unstable in production. Segfaults, corrupted indexes
> galore. We've switched around 40 clients form ferret to sphinx and
> solved their problems this way. I will never use ferret again after
> all the problems I have seen it cause peoples production apps.
>

Just out of interest, were corrupted indexes seen even with only one
process writing to the index (via DRb as is recommended)?  Multiple
writers are unsupported and cause these kinds of problems.

Segfaults were quite common in older version too, but it's settled down
now and I've had it rather stable in a few small production sites
(though I'm not talking Twitter-like load :).

John.
--
http://www.brightbox.co.uk - UK Ruby on Rails hosting
6076c22b65b36f5d75c30bdcfb2fda85?d=identicon&s=25 Ezra Zygmuntowicz (Guest)
on 2008-01-04 22:55
(Received via mailing list)
On Jan 4, 2008, at 1:41 PM, John Leach wrote:

> writers are unsupported and cause these kinds of problems.
>
> Segfaults were quite common in older version too, but it's settled
> down
> now and I've had it rather stable in a few small production sites
> (though I'm not talking Twitter-like load :).
>
> John.
> --
> http://www.brightbox.co.uk - UK Ruby on Rails hosting




  Yes we have tried every way possible of running ferret, by itself,
drb server etc. I really like ferrets interface and integration with
rails but unfortunately it causes nothing but problems for so many
people that I cannot recommend it with a straight face. Not meaning to
bash on the ferret devs here at all, just stating what I've seen
across hundreds of deployments.


Cheers-
- Ezra Zygmuntowicz
-- Founder & Software Architect
-- ezra@engineyard.com
-- EngineYard.com
30b3f6355fad8461e37de7891753d810?d=identicon&s=25 John Leach (Guest)
on 2008-01-04 23:15
(Received via mailing list)
On Fri, 2008-01-04 at 11:26 -0500, Vince Wadhwani wrote:
> latter would take me but, assuming that I'm going to spend somewhere
> around 40 hours anyway, which route would you all recommend?

Hi Vince,

They're different tools really.  I've found the flexibility of Ferret to
be really quite awesome. I can (in Ruby):

 * set boost values independently per field and per record
 * write custom text tokenizers, stemmers and stop lists (and use
different ones per field even)
 * highlight matches in results using the same engine that does the
searching
 * manage my own indexes, merging them at will, or just merging results
from them.
 * Index content generated on the fly, without having to store it in my
sql database (pull in all the associated tags for a post as you index it
for example).
 * Store original data in the index (though most people use it to index
an SQL database anyway).
 * other awesome stuff I can't remember right now.

Looking at the documentation for Sphinx (and it's usual usage, with
MySQL), many (if not all) of those features are missing.  But Sphinx is
reportedly quicker, supports distributed searching, and appears to be
undergoing more development that Ferret is at the moment so I think it
depends on your needs.

I'd recommend you ask on the Ferret mailing list about your search
result issues though - I'm surprised you're having problems with that.
I'm sure it can be solved.

John.
--
http://www.brightbox.co.uk - UK Ruby on Rails hosting
Ddeec660d0ddef3647786f7adb228a97?d=identicon&s=25 Luca Mearelli (Guest)
on 2008-01-05 15:48
(Received via mailing list)
> > A decent search option is Lucene via acts_as_solr plugin.
> > I never used Sphynx though. Can anyone with firsthand experience of
> > both Lucene and Sphynx give their opinion?
> >
...
>         We have a bunch of clients using solr as well. In general it is more
> powerful then sphinx but a lot slower to reindex and querey. Also it
> uses 50 times the memory of sphinx. If you have a box or vm to put
> SOLR on by itself then it is a good option as well. but if sphinx can
> do everything you need from a a search indexer then it is a way better
> option cost wise.

I don't have first hand experiences with sphinx, but i can confirm
that given a decent hw setup solr (with acts_as_solr) is really good
(not only in terms of performance but also of flexibility, and
functionality). We used it for miojob.it and it powers almost any
aspect of that site, which is built around faceted browsing of job
postings and has a only a few spots where caching was appropriate
without sweating under a traffic which is in the multi hundred K hits
per day (i don't have the real numbers)

Anyhow given the lower system requirements, I'd like to give a try to
sphinx to see what can it do!

cheers,
Luca Mearelli

http://spazidigitali.com - http://kiaraservice.com
545c91af71715620b6863ec1e330d90d?d=identicon&s=25 Jérémie Bordier (ahfeel)
on 2008-01-07 09:20
(Received via mailing list)
I've been using Ferret since it's beginning, I'm also the french
translator
of the Ferret Shortcut's for O'Reilly, and i can tell one thing: Don't
use Ferret.
It's really unstable and the development has stopped a while ago...
That's
really sad because it was really an AWESOME product but it never
reached
a stable state.

I've experienced also huge problems with act_as_solr, so finally i'd
just
say "use Sphinx". That's for me the safier decision.

--
Jérémie 'ahFeel' BORDIER
90ebe8da17aabd36cc30d9f96a530e6f?d=identicon&s=25 James Herdman (biocsnerd)
on 2008-01-07 16:07
(Received via mailing list)
I've been humming and hawing all weekend about whether or not to put
in the time to use Sphinx, and I guess the mountain of evidence is
clear: I'll be moving my project over to Sphinx today.

James
2ebf42f8625b86c0e91ae7bd4e2bef52?d=identicon&s=25 Raymond O'Connor (nappin)
on 2008-01-08 10:00
Ya we use ferret right now on our site.  It's ok, but it does segfault
about once a week.  It's not a huge deal I suppose, but doesn't make me
feel good.  Right now I'm evaluating switching to solr or sphinx.  It
would be nice to have the 'more like this' ability that AAF/Ferret has.
I didn't really see this feature with sphinx.  We would also like to be
able to write a custom sort method, which I haven't been able to do with
ferret.  I see there's an ability to do that with sphinx which looks
nice.

Anyways, can anyone recommend a sphinx plugin for Rails?
There's 3 so far that I found.  acts_as_sphinx, ultrasphinx, and
sphinctor. Are they all actively updated?

Thanks,
Ray
D7576a4091fcfdebd69845a6808c6bc1?d=identicon&s=25 Frederico Araujo (brfsa)
on 2008-01-12 20:52
(Received via mailing list)
Ultrasphinx is awesome... I use it for many sites.
and as well I have some capistrano+ultrasphinx recipes.
http://frederico-araujo.com/2007/12/7/capistrano-2...


Sphinx is not as complete, but almost, as SOLR...

but all I can say is that sphinx itself is a piece of art software.

It indexes REALLY fast, 5 seconds, 25,000 records database,
I have a cron job that each hour it updates the index.

/ultrasphinx/production.conf'...
indexing index 'complete'...
collected 25088 docs, 10.4 MB
sorted 1.7 Mhits, 100.0% done
total 25088 docs, 10409184 bytes
total 5.361 sec, 1941487.86 bytes/sec, 4679.33 docs/sec


FERRET is in my second choice only because shared hosts won't support
sphinx....

what a sad thing :(


On Jan 8, 4:00 pm, Raymond O'Connor <ruby-forum-incom...@andreas-
29f82ebe1801087f04de6aaae92e19ea?d=identicon&s=25 pat (Guest)
on 2008-01-19 00:03
(Received via mailing list)
I'm not sure about acts_as_sphinx and sphinctor being actively
updated, but I can confirm that both Ultrasphinx and Thinking Sphinx
(my own plugin - http://ts.freelancing-gods.com) are regularly updated
- and under the hood they both use the same Ruby Sphinx client -
Riddle (http://riddle.freelancing-gods.com - again, mine - sorry for
blowing my own trumpet), which I've been keeping up to date to match
the recent releases of Sphinx.

Evan's and my plugins do a lot of the same things, just different
approaches, so, with as little bias as possible, I think either can do
the job for you. I can't speak for the other two plugins though, as
it's been so long since I've looked into them.

Cheers

--
Pat
e: pat@freelancing-gods.com    || m: 0413 273 337
w: http://freelancing-gods.com || p: 03 9386 0928
discworld: http://ausdwcon.org || skype: patallan

On Jan 8, 8:00 pm, Raymond O'Connor <ruby-forum-incom...@andreas-
979594059cb47947217d6dc458ccd64c?d=identicon&s=25 Samuel Dakin (jeffct)
on 2008-01-19 00:18
(Received via mailing list)
I've been playing with Ferret for awhile. I actually get corrupted
indexes just running in development. I'm close to deploying an app
that uses ferret and some of the things I've heard really worry me.
Haven't had a chance to test the drb server though, but the whole idea
of that bothers me too.

How difficult would it be to change over to Sphinx?
67d3880bbfaafdb2a2108b7c9a64df47?d=identicon&s=25 Adrian Madrid (Guest)
on 2008-01-19 00:40
(Received via mailing list)
On Jan 18, 2008 4:17 PM, Jeff <jeff.cabaniss@gmail.com> wrote:

> ...
> How difficult would it be to change over to Sphinx?
>

That would really depend on how you hooking up with Ferret and if you
were
using any advanced features. My guess is that it shouldn't be too hard
to
switch.

--
Adrian Esteban Madrid
Lead Developer, Prefab Markets
http://www.prefabmarkets.com
29f82ebe1801087f04de6aaae92e19ea?d=identicon&s=25 Pat Allan (Guest)
on 2008-01-19 00:51
(Received via mailing list)
On 19/01/2008, at 10:17 AM, Jeff wrote:

> How difficult would it be to change over to Sphinx?


The overall process? Not hard, with the caveat Adrian mentioned (ie:
advanced Ferret features).

But keep in mind Sphinx does not allow updating fields of index
records (Ferret does) - you have to re-index to get the latest changes
into Sphinx. There are ways around this, to some extent - delta
indexes, containing just the recent changes - but it doesn't seem to
be critical to everyone.

Essentially, though:
- Choose a sphinx plugin, and install it.
- Set up the configuration and indexes, either manually, or within
your models (depending on the plugin)
- Install sphinx
- Index your data
- Switch your ferret-specific search calls to use the sphinx plugin's
search calls.
- Start the sphinx daemon (searchd)
- Confirm everything works

Or something along those lines. I'm sure the EngineYard crew have a
better idea though.

--
Pat
e: pat@freelancing-gods.com    || m: 0413 273 337
w: http://freelancing-gods.com || p: 03 9386 0928
discworld: http://ausdwcon.org || skype: patallan
979594059cb47947217d6dc458ccd64c?d=identicon&s=25 Samuel Dakin (jeffct)
on 2008-01-19 00:54
(Received via mailing list)
No advanced features. Did everything through acts_as_ferret. Other
than that, added one method for pagination from ferret searches.

But Sphinx is good to go in production? What about UTF-8? Ferret
seemed to do that out of the box, I just tested some non-latin
characters and was surprised
to find ferret indexed them all properly.
160992dfd434ff7055465446b9423bd5?d=identicon&s=25 Peter Vandenabeele (petervandenabeele)
on 2008-01-22 22:46
Ericson Smith wrote:
> If you consider using Postgresql, then tsearch2 is awesome. Its built
> into the latest version of Postgresql.

How would you do the integration into Rails 2 ?

I tried the acts_as_tsearch plugin

  http://code.google.com/p/acts-as-tsearch/

and the first line of the example works, but it really
does not seem to be ready for prime time to me and at
this moment ...

Thanks for any insights,

Peter Vandenabeele
(new to rails)
82476266af9d460415d8f1fc16bb54ed?d=identicon&s=25 Jarkko Laine (jarkko)
on 2008-01-23 08:22
(Received via mailing list)
On 22.1.2008, at 23.46, Peter Vandenabeele wrote:

>
> and the first line of the example works, but it really
> does not seem to be ready for prime time to me and at
> this moment ...

I haven't used the plugin, but interfacing with tsearch2 is easy
enough so you can write your own in a day:
http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/

We did that back in early '06 and since talking with tsearch2 is
basically normal SQL, all you have to do is to write a custom finder
method.

I have no idea how the performance compares to other engines but I
find it pretty cool that everything happens transparently inside the
database so you have one less process to monitor and keep fresh. So if
you're using PostgreSQL, it should definitely be worth a shot. It's
been around forever, so it should be void of most pediatric diseases.

Cheers,
//jarkko

--
Jarkko Laine
http://jlaine.net
http://dotherightthing.com
http://www.railsecommerce.com
http://odesign.fi
Ae9dcb9298a637ca497cb5b581e2ee11?d=identicon&s=25 Jens Krämer (Guest)
on 2008-01-25 17:07
(Received via mailing list)
Hi!

> Ferret is unstable in production. Segfaults, corrupted indexes
> galore. We've switched around 40 clients form ferret to sphinx and
> solved their problems this way. I will never use ferret again after
> all the problems I have seen it cause peoples production apps.

I'd really like anybody experiencing problems like this to contact me
or even
better the ferret-talk mailing list about such problems. I have
several sites using
Ferret with DRb server runs rock solid there. I must admit that
they're relatively low
traffic, but high load is nothing that will make Ferret crash or
currupt indexes, if you
use it in the right way (say, one process accessing the index).
Without doubt there
are cases when Ferret will segfault, i.e. because of platform specific
problems, poor
argument checking and error handling in the C code and so on, but they
may be
circumvented most of the time. Not nice, but acts_as_ferret already
does most of this
for you.

I also did some load tests with acts_as_ferret's DRb server a while
ago, where it handled
> 30 mixed indexing and search requests per second from multiple client processes for 
hours,
and no crash or index corruption (index size was 7GB at the end of the
run) happened.

So to summarize: it's definitely possible to have a stable Ferret
setup, before you take on the
work to switch to something else why not drop me a line and I'll be
happy to have a look at your
problem.

However from what I've read here I'll be sure to check out Sphinx soon
so I know what you're
talking about here ;-)

Cheers,
Jens
26ca684555abbdb44f01bd1e165e649b?d=identicon&s=25 Vince W. (vince)
on 2008-01-25 18:24
(Received via mailing list)
> I'd really like anybody experiencing problems like this to contact me

I had trouble with it using version 0.11.6.  I was having intermittent
problems every time I tagged a store.  When I removed my rescue I
found it was ferret (don't have the exact error on me.. sorry).
Stepping back to 0.11.3 seems to have resolved this (this is the last
version I can remember that worked for me somewhat reliably).  With
0.11.6, removing my index solves it temporarily (6 or 7 tag actions)
but then it comes back.

Feel free to move this to the ferret talk list, I'll go check on it
there.

-Vince

support independent business -- http://www.buyindie.net/
979594059cb47947217d6dc458ccd64c?d=identicon&s=25 Samuel Dakin (jeffct)
on 2008-01-29 22:10
(Received via mailing list)
I took a good look at Sphinx and Ultrasphinx, even tried
implementation in my app. Unfortunately these were show stoppers for
me:

- No real integration with activerecord (plugin just generates sql
statements outside of the context of AR. Therefore you can't really
use your own custom model methods as fields... as far as I could tell)
- No wildcards at all (Sphinx doesn't support them)
- No automatic updates - must rebuild entire index using cron jobs.
Again using straight SQL, not the current state of your models

On the contrary, I could see Sphinx being very appropriate for certain
types of apps... but these were important features for my particular
use (especially wildcards)
160992dfd434ff7055465446b9423bd5?d=identicon&s=25 Peter Vandenabeele (petervandenabeele)
on 2008-01-29 23:28
Jeff Cc wrote:
> - No wildcards at all (Sphinx doesn't support them)

Do you mean the "*" feature (prefix*  and *infix*) ? Where
the search term "program*" matches the database text
"program", "programmer", "programs" ...

Those work for me in version sphinx-0.9.8-svn-r1065 and
sphinx-0.9.8-svn-r1112 ... I have done quite some testing
on r0165 (still testing the newest r1112) and that seems
to work OK for me. Set the "enable_star" to 1 and set a
min_prefix_leng or a min_infix_leng.

> - No automatic updates - must rebuild entire index using cron jobs.

Indeed. But automatic rotation of indexes seems to work OK.
Indexing on my dataset takes 15 seconds (37000 records,
28 MByte) on a desktop PC.

> Again using straight SQL, not the current state of your models

At least in one (limited) test, I have just used the :include
feature of ultrasphinx and that automatically created the SQL
for the  sphinx configuration file. So, if I understand well,
that did use the AR model ?

From:
http://blog.evanweaver.com/files/doc/fauna/ultrasp...

* Including a field from an association

Use the :include key.

Accepts an array of hashes.

  :include => [{:association_name => 'category', :field => 'name', :as
=> 'category_name'}]

Each should contain an :association_name key (the association name for
the included model), a :field key (the name of the field to include),
and an optional :as key (what to name the field in the parent).

So in my Model for jobs, that is as simple as e.g.:

class Job < ActiveRecord::Base

  is_indexed :fields => [
    'title']
  :include => [
    {:association_name => 'employer', :field => 'name'}]

  belongs_to :employer
...

The config file for sphinx that is calculated by ultrasphinx then has
automatically calcuated by ultrasphinx:

...
sql_query = SELECT (jobs.id * 1 + 0) AS id, 'Job' AS class, 0 AS
class_id, jobs.
title AS title, employer.name AS name, ...
...
index complete
{
  source = jobs
  charset_type = utf-8
  charset_table = 0..9, A..Z->a..z, -, _, &, a..z,
U+410..U+42F->U+430..U+44F, ... and a lot more ...
  min_word_len = 2
  # min_infix_len = 4
  stopwords =
  enable_star = 1
  path = /var/sphinx//sphinx_index_complete
  docinfo = extern
  morphology = none
  min_prefix_len = 4
}

All of this seems to work for me (no production experience yet ...).
29f82ebe1801087f04de6aaae92e19ea?d=identicon&s=25 Pat Allan (Guest)
on 2008-01-30 01:26
(Received via mailing list)
On 1/29/2008, "Peter Vandenabeele" <ruby-forum-incoming@andreas-s.net>
wrote:

>on r0165 (still testing the newest r1112) and that seems
>to work OK for me. Set the "enable_star" to 1 and set a
>min_prefix_leng or a min_infix_leng.

Yup, works for me too - it's just not turned on by default. The
enable_star feature has been around in at least the last 4 releases of
0.9.8. Fairly certain it's not in 0.9.7 though (the last 'production'
release).

>
>> - No automatic updates - must rebuild entire index using cron jobs.
>
>Indeed. But automatic rotation of indexes seems to work OK.
>Indexing on my dataset takes 15 seconds (37000 records,
>28 MByte) on a desktop PC.

Thinking Sphinx has delta indexes, which keep track of changes between
explicit indexes. I know Evan's working on adding something like this
to UltraSphinx as well.

The super small delta indexes means they get indexed really quickly,
straight after a model is updated.

>
>> Again using straight SQL, not the current state of your models
>

You're correct that UltraSphinx doesn't support model methods (as
opposed to standard attributes) are not accessible for index generation
- that's the same with Thinking Sphinx and perhaps all of the other
plugins as well.

Because you're dealing with MySQL directly when the data is indexed,
there's no instantiation of models (and no Ruby at all), so it's not
really an option. If the data you want isn't available somewhere in the
database, you're out of luck.

Ferret uses model methods, I believe, if that's an option available to
you.

>Use the :include key.
>So in my Model for jobs, that is as simple as e.g.:
>
Thinking Sphinx equivalent (just to provide a comparison):

class Job < ActiveRecord::Base

  define_index do |index|
    index.includes.title
    index.includes.employer.name
  end

  # ...
end

Cheers

--
Pat
545c91af71715620b6863ec1e330d90d?d=identicon&s=25 Jérémie Bordier (ahfeel)
on 2008-01-30 10:33
(Received via mailing list)
Hi Jens,

It's been a long time ;) Hope you're doing well.
I have something to say about that all: Even if you find a right way
of making Ferret quite *"stable"*, the development has stopped for
more than a year now, leaving a LOT of bugs unsolved.
Ferret has no future for the moment, and considering builduing website
on it's top is like doing extrem sports on a just recovered broken
leg..

Cheers,
Jérémie
979594059cb47947217d6dc458ccd64c?d=identicon&s=25 Samuel Dakin (jeffct)
on 2008-01-30 21:55
(Received via mailing list)
> Do you mean the "*" feature (prefix*  and *infix*) ? Where
> the search term "program*" matches the database text
> "program", "programmer", "programs" ...
>
> Those work for me in version sphinx-0.9.8-svn-r1065 and
> sphinx-0.9.8-svn-r1112 ... I have done quite some testing
> on r0165 (still testing the newest r1112) and that seems
> to work OK for me. Set the "enable_star" to 1 and set a
> min_prefix_leng or a min_infix_leng.
>

No that's the stemming feature I believe.. it just changes prefixes
and suffixes on words and is language dependent. Awesome feature
however. Not sure how (or if) Ferret implements it.

What I meant was just straight wildcards as in a MySQL LIKE clause,
example:  "*@gmail.com" to find all emails @gmail.com
979594059cb47947217d6dc458ccd64c?d=identicon&s=25 Samuel Dakin (jeffct)
on 2008-01-30 22:20
(Received via mailing list)
Are you sure Ferret development has stopped? According to the Ferret
trac, last change to the trunk was only a few weeks ago, and last tag
(0.11.6) was dated Nov 28 2007, only two months ago. I also see the
developer replying to tickets just this month. Am I missing something
here?
160992dfd434ff7055465446b9423bd5?d=identicon&s=25 Peter Vandenabeele (petervandenabeele)
on 2008-01-30 22:22
Jeff Cc wrote:
>> Do you mean the "*" feature (prefix*  and *infix*) ? Where
>> the search term "program*" matches the database text
>> "program", "programmer", "programs" ...
>>
>> Those work for me in version sphinx-0.9.8-svn-r1065 and
>> sphinx-0.9.8-svn-r1112 ... I have done quite some testing
>> on r0165 (still testing the newest r1112) and that seems
>> to work OK for me. Set the "enable_star" to 1 and set a
>> min_prefix_leng or a min_infix_leng.
>
> No that's the stemming feature I believe.. it just changes prefixes
> and suffixes on words and is language dependent. Awesome feature
> however. Not sure how (or if) Ferret implements it.
>
> What I meant was just straight wildcards as in a MySQL LIKE clause,
> example:  "*@gmail.com" to find all emails @gmail.com

I have the impression the enable_star _is_ really the feature that does
allow
search for "*@gmail.com" to find all emails @ gamil.com  (if you add the
'@' sign to the char table actually ... (which is another problem, since
'@' also has a special meaning as a field indicator for field specific
search).
For the enable star the user must explicitely give a '*'. WIthout a '*'
the match is only for "exact match". I give an example at the end of
my blog: (http://www.vandenabeele.com/Ultrasphinx-performance) where
I tested with and without the enable_star feature and always without
stemming
(since I had not stemmer for the Duthch language).

0.001 sec [ext/0/rel 1409 (0,20)] [complete] c
0.001 sec [ext/0/rel 1409 (0,20)] [complete] c*
0.000 sec [ext/0/rel 35 (0,20)] [complete] co
0.000 sec [ext/0/rel 35 (0,20)] [complete] co*
0.000 sec [ext/0/rel 5 (0,20)] [complete] com
0.000 sec [ext/0/rel 5 (0,20)] [complete] com*
0.000 sec [ext/0/rel 10 (0,20)] [complete] comp
0.003 sec [ext/0/rel 5343 (0,20)] [complete] comp*
0.000 sec [ext/0/rel 0 (0,20)] [complete] compl
0.000 sec [ext/0/rel 1473 (0,20)] [complete] compl*
0.000 sec [ext/0/rel 0 (0,20)] [complete] comple
0.000 sec [ext/0/rel 1214 (0,20)] [complete] comple*
0.000 sec [ext/0/rel 0 (0,20)] [complete] complet
0.000 sec [ext/0/rel 793 (0,20)] [complete] complet*
0.000 sec [ext/0/rel 458 (0,20)] [complete] complete
0.000 sec [ext/0/rel 642 (0,20)] [complete] complete*
0.000 sec [ext/0/rel 30 (0,20)] [complete] completed
0.000 sec [ext/0/rel 30 (0,20)] [complete] completed*
0.000 sec [ext/0/rel 0 (0,20)] [complete] completel
0.000 sec [ext/0/rel 130 (0,20)] [complete] completel*
0.000 sec [ext/0/rel 10 (0,20)] [complete] completely.

What happens is that with less than 4 characters, the * has no effect,
but from 4 characters on, the * expands to all words that match the same
first 4 letters. And that is an interesting feature the major public
search engines do not offer. At this time, with the relatively small
database I expect initially for our project (< 10 MByte or so), it
should not be a problem to keep indices with start expansion after 4
letters in memory.

An issue that I still have is that a final '.' of a sentence is attached
to the index data and so not found without attaching a '.' or '*' to the
search term.

++++

I solved the '.' issue in the meanwhile with a crude solution of
removing the '.' character from the char_table list (which causes other
problems ...).

The stemming will e.g. 'companies' and 'company' to a stem of 'compani'
(both in the search term and in the database index), without the user
needing to add a special * to the search. so any combination of
'company' and 'companies' will match.

HTH,

Peter
545c91af71715620b6863ec1e330d90d?d=identicon&s=25 Jérémie Bordier (ahfeel)
on 2008-01-30 23:00
(Received via mailing list)
Hi Jeff,

The 0.11.6 release has only a LITTLE bugfix. Also, closing tickets as
"wont fix" is easy. Have a look at the latest MEANINGFUL changesets,
long time ago :)

Cheers,
Jérémie
72cb32060772721a561890385172355e?d=identicon&s=25 lamyseba (Guest)
on 2008-03-17 00:46
(Received via mailing list)
On 30 jan, 22:58, "ahF...@gmail.com" <Ahf...@gmail.com> wrote:
> Hi Jeff,
>
> The 0.11.6 release has only a LITTLE bugfix. Also, closing tickets as
> "wont fix" is easy. Have a look at the latest MEANINGFUL changesets,
> long time ago :)
>
> Cheers,
> Jérémie

Seems like the maintenance is back, just have a look at the trac
timeline, dbalmain seems to be the new maintenance guy
545c91af71715620b6863ec1e330d90d?d=identicon&s=25 Jérémie Bordier (ahfeel)
on 2008-03-18 11:48
(Received via mailing list)
On Mar 17, 12:45 am, lamyseba <lamys...@free.fr> wrote:
>
> Seems like the maintenance is back, just have a look at the trac
> timeline, dbalmain seems to be the new maintenance guy

Lol, dbalmain has always been the one and only coder of Ferret, he's
back but his current commits only fixes a very little bugs and add
more features such as compression with zlib... Adding feature instead
of fixing existing and CORE ones ? Have fun guys :)

I REALLY hope ferret to become stable one day since it's the most
flexible and easy to use product i've seen for this job, but don't use
it in production now..

Jérémie.
Fff557ffbe33c6aab8aa17e1489fdaea?d=identicon&s=25 Harry Seldon (harry_seldon)
on 2008-03-26 20:29
Jérémie Bordier wrote:
> On Mar 17, 12:45 am, lamyseba <lamys...@free.fr> wrote:
>>
>> Seems like the maintenance is back, just have a look at the trac
>> timeline, dbalmain seems to be the new maintenance guy
>
> Lol, dbalmain has always been the one and only coder of Ferret, he's
> back but his current commits only fixes a very little bugs and add
> more features such as compression with zlib... Adding feature instead
> of fixing existing and CORE ones ? Have fun guys :)
>
> I REALLY hope ferret to become stable one day since it's the most
> flexible and easy to use product i've seen for this job, but don't use
> it in production now..
>
> J�r�mie.

Hi,

That is a very interesting thread. I am currently deploying a rails app
with aaf. I have many troubles and basically I cannot have it working. I
am surprised because everything was so simple in development.

I must admit that I understand nothing to the DRB server. (I am learning
this new thing.)
My app is on a shared host. I do not even understand if the drb server
can run on it...

When I run : script/ferret_server -e production start
I get : starting ferret server...
That is all
But when I stop it (script/ferret_server -e production start) I get :
ferret_server doesn't appear to be running
I guess it is not normal (can someone confirm please ?)...

Then when I do script/console production
Article.rebuild_index
I get the first time :
DRb::DRbConnError: druby://ferret.myhost.com:9010 - #<SocketError:
getaddrinfo: Name or service not known>
And if I do it a second time :
LoadError: Expected article.rb to define Article

Whatever bad I am, this is just an awful behavior for a software, sorry
to say that because I loved aaf in dev.

(I tried a chmod -R 777 index without success)
For info I am deploying with Capistrano, in case it rings a bell to
someone.

My options :
more help from my host, I am currently discussing with them
help from you about the aaf configuration
  but even if I make it work, from what I have read here I should not
build my app with it...
try another search engine : sphinx
  but I read from brfsa "FERRET is in my second choice only because
shared hosts won't support sphinx...." Can I have any precision on that
? Or alternatively for those of you on a shared host how do you manage
your search ?

Finally, I am listening to your suggestions about :
the web host : which ones allow a search engine such as
ferret/sphinx/other ?
how to configure aaf ?
Which plugins for the engine ? (Do not worry I will read again the whole
thread!)

Thx !
H
67d3880bbfaafdb2a2108b7c9a64df47?d=identicon&s=25 Adrian Madrid (Guest)
on 2008-03-26 21:30
(Received via mailing list)
Don't even try running ferret on a shared host. I don't think you really
have any other option but MySQL fulltext indexes in a shared hosting
environment.

AEM

On Wed, Mar 26, 2008 at 1:29 PM, Harry Seldon <
ruby-forum-incoming@andreas-s.net> wrote:

> > of fixing existing and CORE ones ? Have fun guys :)
> with aaf. I have many troubles and basically I cannot have it working. I
> But when I stop it (script/ferret_server -e production start) I get :
>
>  but even if I make it work, from what I have read here I should not
> how to configure aaf ?
>
>
> --
> Posted via http://www.ruby-forum.com/.
>
> >
>


--
Adrian Esteban Madrid
Lead Developer, Prefab Markets
http://www.prefabmarkets.com
Fff557ffbe33c6aab8aa17e1489fdaea?d=identicon&s=25 Harry Seldon (harry_seldon)
on 2008-03-26 21:36
Heartbreaking !

Adrian Madrid wrote:
> Don't even try running ferret on a shared host. I don't think you really
> have any other option but MySQL fulltext indexes in a shared hosting
> environment.
>
> AEM
160992dfd434ff7055465446b9423bd5?d=identicon&s=25 Peter Vandenabeele (petervandenabeele)
on 2008-03-26 21:43
Adrian Madrid wrote:
> Don't even try running ferret on a shared host. I don't think you really
> have any other option but MySQL fulltext indexes in a shared hosting
> environment.

You might take a look at tsearch2 on postgresql (for a shared host
solution).
IIRC, it only requires special indexes in the database, but no daemon
process (like e.g. sphinx does). This was mentioned higher up in this
thread too, by
Ericson Smith.

I did some experiments with tsearch2 and it worked OK (but then I
switched to sphinx, mainly because MySQL is more common as a Rails
back-end and because a clean and full plug-in (Ultrasphinx) was
available). In older versions of Postgresql it is a plug-in, since 8.2
(IIRC) it is built-in by default.

HTH,

Peter
67d3880bbfaafdb2a2108b7c9a64df47?d=identicon&s=25 Adrian Madrid (Guest)
on 2008-03-26 21:49
(Received via mailing list)
Running memory hungry/long running processes in shared hosting
environments
is sure to get you kicked out quickly. Is there something that fulltext
mysql indexes won't give you that you desperately need? If MySQL won't
cut
it then you probably need to move into a VPS.


AEM

On Wed, Mar 26, 2008 at 2:36 PM, Harry Seldon <
ruby-forum-incoming@andreas-s.net> wrote:

> Posted via http://www.ruby-forum.com/.
>
> >
>


--
Adrian Esteban Madrid
Lead Developer, Prefab Markets
http://www.prefabmarkets.com
Fff557ffbe33c6aab8aa17e1489fdaea?d=identicon&s=25 Harry Seldon (harry_seldon)
on 2008-03-26 21:51
Indeed, but unfortunately I am under MySQL and not willing to change
right now...

About ferret on a shared host there is this solution which could be a
temporary solution.
http://boonedocks.net/mike/archives/151-Rails-acts...

H

Peter Vandenabeele wrote:
> Adrian Madrid wrote:
>> Don't even try running ferret on a shared host. I don't think you really
>> have any other option but MySQL fulltext indexes in a shared hosting
>> environment.
>
> You might take a look at tsearch2 on postgresql (for a shared host
> solution).
> IIRC, it only requires special indexes in the database, but no daemon
> process (like e.g. sphinx does). This was mentioned higher up in this
> thread too, by
> Ericson Smith.
>
> I did some experiments with tsearch2 and it worked OK (but then I
> switched to sphinx, mainly because MySQL is more common as a Rails
> back-end and because a clean and full plug-in (Ultrasphinx) was
> available). In older versions of Postgresql it is a plug-in, since 8.2
> (IIRC) it is built-in by default.
>
> HTH,
>
> Peter
Fff557ffbe33c6aab8aa17e1489fdaea?d=identicon&s=25 Harry Seldon (harry_seldon)
on 2008-03-26 22:13
@Adrian

"Is there something that fulltext mysql indexes won't give you that you
desperately need? If MySQL won't cut it then you probably need to move
into a VPS."

Well that is a good question I was wondering about. And basically the
answer is that it was so easy to run aaf that it is a pity to go without
it to search in different models, for different fields.

By the way I do not really understand why ferret could not use the db to
write its index (performance issue?). At least the db knows how not to
corrupt a file system.

H
7cc7da7ccf794c3602258dae32de3260?d=identicon&s=25 Adhiraj Rankhambe (adhiraj)
on 2008-04-03 07:55
Hey I have used Ferret, Sphinx and Solr all three of them in development
as well as production environments.

If you want to stay out of all this debate about which search engine to
use, avoid troubleshooting your search feature and make it zero
maintenance and still get a great speed at indexing and searching(pros
and cons), I would suggest you to go for Solr and acts_as_solr plugin.

I have compiled some points that I came across during my experience with
RoR till date.

Ferret:-
Advantages:
1. Easy to implement.
2. Indexing on ActiveRecord save - It hooks up with the life cycle of an
object.
Disadvantages:-
1. Corrupts indexes if used with Transactions in your apps because of
its after_update filter.(It updates the index before the actual save to
the database)
2. Unstable on the production server if you use some load balancing
techniques like round-robbin scheme and you have instances of mongrel on
different machines.
(Added burden to use a separate dRB server)
3. Faster at indexing but slower at searching.

Sphinx:-
Advantages:-
1. Great at speed of indexing and searching.
2. Its at the database level so just one copy of indexes unlike ferret.
Disadvantages:-
1. Difficult to integrate as compared to Ferret or Solr.
2. You have to write a lot of sql code in the configuration file for
indexing and searching data.
3. Not hooked with the ActiveRecord save or the life cycle of an object,
so you need a cron job to rebuild the index periodically.

Solr:-
Advantages:-
1. Easy to implement
2. Runs on a separate Java server(Solr server), so just one copy of
indexes.
3. Hooked up with the object life cycle, so index update with
ActiveRecord save.
4. Good speed at indexing and searching
5. No gem required, no engine installation......just get the
Acts_as_solr plugin.
6. In-built support for highlighting search keywords like you see in
Google Search and many more advanced features.
7. NONE of the disadvantages mentioned above
Disadvantages:-
1. It costs you just some extra memory but not an unbearable amount
though.(I would say that now-a-days memory is cheaper, so you can afford
it)

I personally would suggest you to go for Acts_As_Solr plugin.

You could also refer to the following links:-
http://bloggingrails.wordpress.com/2007/05/31/impl...
http://blog.aisleten.com/2007/04/14/getting-starte...

If you decide to use Acts_as_solr on windows, this would be helpful:-
http://www.webonrails.com/2007/09/13/acts_as_solr-...
29f82ebe1801087f04de6aaae92e19ea?d=identicon&s=25 Pat Allan (Guest)
on 2008-04-03 11:22
(Received via mailing list)
On 03/04/2008, at 4:55 PM, Adhiraj Rankhambe wrote:

> Sphinx:-
> Advantages:-
> 1. Great at speed of indexing and searching.
> 2. Its at the database level so just one copy of indexes unlike
> ferret.
> Disadvantages:-
> 1. Difficult to integrate as compared to Ferret or Solr.

Arguable, but each to their own.

> 2. You have to write a lot of sql code in the configuration file for
> indexing and searching data.

This very much depends on the plugin you use. I'm reasonably sure this
isn't required for Ultrasphinx, and it's definitely not for Thinking
Sphinx (my own plugin, as mentioned earlier in this thread - yes, I've
got some level of bias).

> 3. Not hooked with the ActiveRecord save or the life cycle of an
> object,
> so you need a cron job to rebuild the index periodically.

Yeah, that's pretty much true. Both of the above plugins support delta
indexes, so model changes are automatically put into the live indexes,
but regular periodic reindexing is still needed.

> Solr:-
<snip>
> Disadvantages:-
> 1. It costs you just some extra memory but not an unbearable amount
> though.(I would say that now-a-days memory is cheaper, so you can
> afford
> it)

2. It's Java - which is extra overhead for some people - I certainly
don't use any other Java tools, and I've not dealt with Java since
Uni. Again, each to their own, but that may push non-Java people away
from Solr.

Cheers

--
Pat
e: pat@freelancing-gods.com    || m: 0413 273 337
w: http://freelancing-gods.com || p: 03 9386 0928
discworld: http://ausdwcon.org || skype: patallan
7cc7da7ccf794c3602258dae32de3260?d=identicon&s=25 Adhiraj Rankhambe (adhiraj)
on 2008-04-10 11:48
> 2. It's Java - which is extra overhead for some people - I certainly
> don't use any other Java tools, and I've not dealt with Java since
> Uni. Again, each to their own, but that may push non-Java people away
> from Solr.


You don't need to know Java to use the acts_as_solr plugin.

You just install the plugin and build the index for the first time.
Thenonwards, you just have to start the solr server by issuing a
command:
rake solr:start.

Now tell me where's Java?

That's pretty much it.
7cc7da7ccf794c3602258dae32de3260?d=identicon&s=25 Adhiraj Rankhambe (adhiraj)
on 2008-04-10 11:53
>> 2. You have to write a lot of sql code in the configuration file for
>> indexing and searching data.
>
> This very much depends on the plugin you use. I'm reasonably sure this
> isn't required for Ultrasphinx, and it's definitely not for Thinking
> Sphinx (my own plugin, as mentioned earlier in this thread - yes, I've
> got some level of bias).


Ultrasphinx works only on Rails 2.0.
29f82ebe1801087f04de6aaae92e19ea?d=identicon&s=25 Pat Allan (Guest)
on 2008-04-10 17:10
(Received via mailing list)
On 10/04/2008, at 7:48 PM, Adhiraj Rankhambe wrote:
> rake solr:start.
>
> Now tell me where's Java?
>
> That's pretty much it.

Sorry - first off, I have complete ignorance about acts_as_solr and
Solr. My Java comment was in reference to the latter though, since you
mentioned you can run it on 'a separate Java server', I assumed that's
all it runs in.

> Ultrasphinx works only on Rails 2.0.
Granted, this is a problem if you're shoehorning Sphinx into an
existing app - but I'm guessing most people starting new projects
would be using 2.0 (or even edge in preparation for 2.1?)

Cheers

--
Pat
This topic is locked and can not be replied to.