Scaling full text indexing(ferret vs solr vs hyperstraier)


#1

Hi,

Does any have experience scaling full text search in RoR?

Right now our project is running a simple setup with ferret and
acts_as_ferret. We are thinking about deploying a feature that would
send 50x more search requests.

So we probably have to rethink our solution. How do services like
search.twitter.com (the former Summize) use?

Or in what direction should I look?


Thanks,
M.


#2

Marcelo B. wrote:

Right now our project is running a simple setup with ferret and
acts_as_ferret. We are thinking about deploying a feature that would
send 50x more search requests.

With Ferret you can scale reads horizontally: you can have multiple read
servers on a single index. You can only have one write server on a
single index or you’ll risk data corruption.

Another strategy is partitioning: having separate indices for buckets of
data. Each index could run on it’s own server or cluster of servers.


Roderick van Domburg
http://www.nedforce.com


#3

With Ferret you can scale reads horizontally: you can have multiple read
servers on a single index. You can only have one write server on a
single index or you’ll risk data corruption.

Another strategy is partitioning: having separate indices for buckets of
data. Each index could run on it’s own server or cluster of servers.

Would it be easier to scale with hyperestraier or something else?


M.


#4

On Sun, Mar 8, 2009 at 1:14 PM, Marcelo B. removed_email_address@domain.invalid
wrote:

Does any have experience scaling full text search in RoR?

One option that worked very well for me is ultrasphinx.

http://blog.evanweaver.com/files/doc/fauna/ultrasphinx/files/README.html

IIRC, 2 limitations of ultrasphinx are:

  • new entries can only be found after reindexing
    (full reindexing or delta indexing)
  • you need a separate sphinx process somewhere on a server
    (if you run a shared hosting system, this may be an issue)

If you can live with those 2 limitations, ultrasphinx is a very good
candidate.

HTH,

Peter