# Performance before and after optimization

**URL:** <https://www.ruby-forum.com/t/performance-before-and-after-optimization/119131>\
**Category:** Ferret\
**Created:** [November 3, 2007, 1:50pm UTC](https://www.ruby-forum.com/t/performance-before-and-after-optimization/119131 "2007-11-03T13:50:38Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alex\_Neth](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/b487fb/32.png) [@Alex\_Neth](https://www.ruby-forum.com/u/Alex_Neth)\
**Post date:** [November 3, 2007, 1:50pm UTC](https://www.ruby-forum.com/t/performance-before-and-after-optimization/119131/1 "2007-11-03T13:50:38Z")

</div>

I have an index with a few hundred thousand records. The index is  
generally very fast, with sub 100ms responses. However if I start  
adding records, it gets extremely slow, up to over 2 seconds per  
query. This is true even if I am not currently indexing until I  
optimize the  
index.

In order to work around this, I index in bulk and immediately  
optimize. This is not ideal for the performance of my site.

Unfortunately, contrary to what Dave Balmain seems to say here:  
[OSDIR](http://osdir.com/ml/lang.ruby.ferret.general/2006-08/msg00037.html) ,  
the index seems to be locked for reading during optimization.

So I have two questions:

1. Why does the performance degrade so badly after adding just a few  
records, unless I optimize the index? Can I avoid this?

2. Can I keep a second index so that it doesn’t get locked during  
optimization and then switch to the optimized index? Perhaps the index  
is not really locked and it is just using all the CPU? (I am using a  
single CPU server)?

Thanks for any help.

-Alex

---

<div class="post-metadata">

**Author:** ![Alex\_Neth](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/b487fb/32.png) [@Alex\_Neth](https://www.ruby-forum.com/u/Alex_Neth)\
**Post date:** [November 5, 2007, 11:29am UTC](https://www.ruby-forum.com/t/performance-before-and-after-optimization/119131/2 "2007-11-05T11:29:31Z")

</div>

On Sat, Nov 03, 2007 at 08:49:17PM +0800, Alex Neth wrote:  
[…]

> 1. Can I keep a second index so that it doesn’t get locked during  
> optimization and then switch to the optimized index? Perhaps the index  
> is not really locked and it is just using all the CPU? (I am using a  
> single CPU server)?

If you’re already indexing in batches, keeping a second read-only index  
for  
searching is a good idea. rsync is useful to keep the search-index up to  
date in this case.

To check if CPU usage is a problem, try lowering the optimizing process’  
priority and see how it goes.

Jens

–  
Jens Krämer  
[http://www.jkraemer.net/](http://www.jkraemer.net/) - Blog  
[http://www.omdb.org/](http://www.omdb.org/) - The new free film database