# (ultra)sphinx reindexing eats up a lot of memory **URL:** \ **Category:** Deployment\ **Created:** [September 24, 2008, 12:11am UTC](https://www.ruby-forum.com/t/ultra-sphinx-reindexing-eats-up-a-lot-of-memory/149354 "2008-09-24T00:11:14Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![dxk3355](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/d6d6ee/32.png) [@dxk3355](https://www.ruby-forum.com/u/dxk3355)\ **Post date:** [September 24, 2008, 12:11am UTC](https://www.ruby-forum.com/t/ultra-sphinx-reindexing-eats-up-a-lot-of-memory/149354/1 "2008-09-24T00:11:14Z")
Hello, On my 512MB slice (Slicehost) running 3 thins and a Postgres database, I had set up a delta indexing scheme that executes “rake ultrasphinx index:delta” task every minute via cron. The server was unstable (database process got killed every day or so). I isolated the problem to that cron task (eating up too much memory). After temporarily disabling it, the server was stable. I wonder if some of you also had similar problems and if the solution is to simply use a bigger slice… Thank you! david ---
**Author:** ![dxk3355](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/d6d6ee/32.png) [@dxk3355](https://www.ruby-forum.com/u/dxk3355)\ **Post date:** [September 24, 2008, 4:05am UTC](https://www.ruby-forum.com/t/ultra-sphinx-reindexing-eats-up-a-lot-of-memory/149354/2 "2008-09-24T04:05:32Z")
I haven’t used the UltraSphinx plugin, but Sphinx itself has a mem\_limit configuration in its conf file to cap the amount of RAM that the indexer will use. Roberto [blog.idearise.com](http://blog.idearise.com) On Tue, Sep 23, 2008 at 10:11 PM, D. Krmpotic \< ---
**Author:** ![dxk3355](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/d6d6ee/32.png) [@dxk3355](https://www.ruby-forum.com/u/dxk3355)\ **Post date:** [September 24, 2008, 10:54am UTC](https://www.ruby-forum.com/t/ultra-sphinx-reindexing-eats-up-a-lot-of-memory/149354/3 "2008-09-24T10:54:02Z")
Roberto, thank you for the answer. I checked and saw that I already have reduced mem\_limit for sphinx indexer to 64MB. It seems that even this is too much for my slice, when it shouldn’t be… Check this (I’m baffled): Mem: 523008k total, 470204k used, 52804k free, 1896k buffers Swap: 0k total, 0k used, 0k free, 41524k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2218 david 15 0 204m 108m 1936 S 0 21.2 2:22.51 thin 2223 david 15 0 201m 104m 2100 S 0 20.5 1:33.93 thin 2210 david 15 0 130m 60m 1352 S 0 11.8 0:04.31 thin 13651 david 16 0 108m 34m 1464 S 0 6.8 0:01.52 ruby 2102 mysql 15 0 162m 32m 1524 S 0 6.4 0:07.48 mysqld 2378 root 16 0 40608 6512 856 S 0 1.2 0:02.34 munin-node 2472 postgres 16 0 99.6m 4836 2168 S 0 0.9 0:00.85 postgres 13653 postgres 16 0 99.1m 4064 1876 S 0 0.8 0:00.03 postgres 11600 david 16 0 19760 3384 940 S 0 0.6 0:00.12 bash 2476 postgres 16 0 99.9m 3356 812 S 0 0.6 0:00.39 postgres … … So out of 512MB, only 52MB is free and that’s why indexer reported “Cannot allocate memory”. Can you please help me understand how can 90% of memory be used if first 5 lines in TOP total to 63% (the rest is negligible). So 3 thins, nginx, mysql and postgres should not consume 90% of 512MB and the percentages in the table confirm this. But still I only have 52MB left (!?). Thank you for any clarifications. David Roberto R wrote: > I haven’t used the UltraSphinx plugin, but Sphinx itself has a mem\_limit > configuration in its conf file to cap the amount of RAM that the indexer > will use. > > Roberto > [blog.idearise.com](http://blog.idearise.com) > > On Tue, Sep 23, 2008 at 10:11 PM, D. Krmpotic \< ---
**Author:** ![dxk3355](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/d6d6ee/32.png) [@dxk3355](https://www.ruby-forum.com/u/dxk3355)\ **Post date:** [September 24, 2008, 2:12pm UTC](https://www.ruby-forum.com/t/ultra-sphinx-reindexing-eats-up-a-lot-of-memory/149354/4 "2008-09-24T14:12:16Z")
I don’t know anything about Slicehost - but surely you should have some swap space? Cheers, Tony 2008/9/24 D. Krmpotic [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > 2476 postgres 16 0 99.9m 3356 812 S 0 0.6 0:00.39 postgres > > > [blog.idearise.com](http://blog.idearise.com) – Tony White ---
**Author:** ![dxk3355](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/d6d6ee/32.png) [@dxk3355](https://www.ruby-forum.com/u/dxk3355)\ **Post date:** [September 24, 2008, 2:30pm UTC](https://www.ruby-forum.com/t/ultra-sphinx-reindexing-eats-up-a-lot-of-memory/149354/5 "2008-09-24T14:30:20Z")
Yes, that’s true (and strange). It should start swapping if nothing else… but still… swapping is just as bad … and 512MB should be enough for my needs… the problem is finding out what is eating the rest of the memory. BTW - free reports this: free -m total used free shared buffers cached Mem: 510 470 40 0 11 45 -/+ buffers/cache: 413 97 Swap: 0 0 0 This is quite a problem for me and I hope someone can suggest a solution! Thanx! david Tony White wrote: > I don’t know anything about Slicehost - but surely you should have > some swap space? > > Cheers, > Tony > > 2008/9/24 D. Krmpotic [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > > > 2476 postgres 16 0 99.9m 3356 812 S 0 0.6 0:00.39 postgres > > > > > [blog.idearise.com](http://blog.idearise.com) > > – > Tony White ---
**Author:** ![dxk3355](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/d6d6ee/32.png) [@dxk3355](https://www.ruby-forum.com/u/dxk3355)\ **Post date:** [September 24, 2008, 6:54pm UTC](https://www.ruby-forum.com/t/ultra-sphinx-reindexing-eats-up-a-lot-of-memory/149354/6 "2008-09-24T18:54:01Z")
Hi, It looks like there is no swap (???). On Wed, Sep 24, 2008 at 12:30 PM, D. Krmpotic \< ---
**Author:** ![dxk3355](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/d6d6ee/32.png) [@dxk3355](https://www.ruby-forum.com/u/dxk3355)\ **Post date:** [September 24, 2008, 6:29pm UTC](https://www.ruby-forum.com/t/ultra-sphinx-reindexing-eats-up-a-lot-of-memory/149354/7 "2008-09-24T18:29:51Z")
D. Krmpotic wrote: > free -m > total used free shared buffers > cached > Mem: 510 470 40 0 11 > 45 > -/+ buffers/cache: 413 97 > Swap: 0 0 0 > > This is quite a problem for me and I hope someone can suggest a > solution! Thanx! Linux aggressively caches files using available free memory (and even sometimes swapping out lesser used programs for more cache), consequently “free” memory is not just the reported unused memory, but unused + buffers + cache. So, when you looked at top, it said 52M free, but the effective free memory is 97M as reported by that second line from free. Your VM does appear to have swap disabled, which can be a good idea if you have a very well tuned and controlled memory profile, but quite a problem if not. Yes it’s best never to swap, but it might be preferable to go slow instead of fail completely when you run out. > Can you please help me understand how can 90% of memory be used if > first > 5 lines in TOP total to 63% (the rest is negligible). So 3 thins, > nginx, > mysql and postgres should not consume 90% of 512MB and the > percentages > in the table confirm this. But still I only have 52MB left (!?). Well, as we just discussed, you actually have about 80% used, not 90%, when you reclaim the OSes cache slush, Don’t discount all those little processes – you likely have several dozen to a hundred 1-5M processes below your top five – it adds up. Looking back at your earlier top output, I also notice you have Pg running with about 90M shared buffers (inferred from the virtual memory size), but it normally uses only 2-4M RSS: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2472 postgres 16 0 99.6m 4836 2168 S 0 0.9 0:00.85 postgres 13653 postgres 16 0 99.1m 4064 1876 S 0 0.8 0:00.03 postgres 2476 postgres 16 0 99.9m 3356 812 S 0 0.6 0:00.39 postgres It’s possible that the sphinx indexing operation, which queries postgres directly, causes Pg to temporarily allocate substantially more memory then the normal working set to serve the re-index requests, and that could be another source of your memory over-commit. If your principle database is postgres, I gotta wonder why you have a 32M mysqld running too? 2102 mysql 15 0 162m 32m 1524 S 0 6.4 0:07.48 mysqld -Devin ---
**Author:** ![dxk3355](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/d6d6ee/32.png) [@dxk3355](https://www.ruby-forum.com/u/dxk3355)\ **Post date:** [September 25, 2008, 7:36pm UTC](https://www.ruby-forum.com/t/ultra-sphinx-reindexing-eats-up-a-lot-of-memory/149354/8 "2008-09-25T19:36:34Z")
Yeah, without any swap… you’re going to have some problems. Surprised that you don’t have any enabled by default. (at least 2x the amount of RAM to start… ) On Wed, Sep 24, 2008 at 9:02 AM, Roberto R [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > else… but still… swapping is just as bad … and 512MB should be > > -/+ buffers/cache: 413 97 > > > > > some swap space? > > > > > > > – Robby R. Chief Evangelist, Partner PLANET ARGON, LLC design // development // hosting > **[Planet Argon](https://www.planetargon.com/)** > > Planet Argon is a Ruby on Rails development agency specializing in maintenance and support for existing Rails apps. Based in Portland, OR, Planet Argon has been a member of the Rails community since 2004. [http://www.robbyonrails.com/](http://www.robbyonrails.com/) aim: planetargon +1 503 445 2457 +1 877 55 ARGON [toll free] +1 815 642 4068 [fax] ---
**Author:** ![dxk3355](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/d6d6ee/32.png) [@dxk3355](https://www.ruby-forum.com/u/dxk3355)\ **Post date:** [September 25, 2008, 7:39pm UTC](https://www.ruby-forum.com/t/ultra-sphinx-reindexing-eats-up-a-lot-of-memory/149354/9 "2008-09-25T19:39:46Z")
in the meantime I did enable it… no idea how it got disabled… but still… do you think it’s ok that so little memory is left on boot and reindexing would have to start swapping? Robby R. wrote: > Yeah, without any swap… you’re going to have some problems. > Surprised that you don’t have any enabled by default. (at least 2x the > amount of RAM to start… ) > > On Wed, Sep 24, 2008 at 9:02 AM, Roberto R [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) > wrote: > > > > else… but still… swapping is just as bad … and 512MB should be > > > -/+ buffers/cache: 413 97 > > > > > > > some swap space? > > > > > > > > > > > – > Robby R. > Chief Evangelist, Partner > > PLANET ARGON, LLC > design // development // hosting > > [http://www.planetargon.com/](http://www.planetargon.com/) > [http://www.robbyonrails.com/](http://www.robbyonrails.com/) > aim: planetargon > > +1 503 445 2457 > +1 877 55 ARGON [toll free] > +1 815 642 4068 [fax] ---
**Author:** ![dxk3355](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/d6d6ee/32.png) [@dxk3355](https://www.ruby-forum.com/u/dxk3355)\ **Post date:** [September 26, 2008, 1:12am UTC](https://www.ruby-forum.com/t/ultra-sphinx-reindexing-eats-up-a-lot-of-memory/149354/10 "2008-09-26T01:12:57Z")
for those interested how swap got disabled… it was probably during an upgrade to 8.04 (Ubuntu). I enabled the swap partition with SWAPON command, but discovered that upon reboot it got disabled again… I found out the problem (UUIDs for sda2 in /etc/fstab were wrong…): :~$ cat /etc/fstab proc /proc proc defaults 0 0 # /dev/sda1 – converted during upgrade to edgy UUID=9c65b16b-2082-4121-9c78-d5fc6e407ade / ext3 defaults,errors=remount-ro 0 1 # /dev/sda2 – converted during upgrade to edgy UUID=ff26b2f9-e08c-42d8-97fa-9a5e37f9a711 none swap sw 0 0 The correct UUID: :~$ cat /etc/blkid.tab /dev/sda1 /dev/sda2 So I updated the UUID in fstab and the swap partition settings survive the reboot now! david D. Krmpotic wrote: > in the meantime I did enable it… no idea how it got disabled… > > but still… do you think it’s ok that so little memory is left on boot > and reindexing would have to start swapping? > > Robby R. wrote: > > > Yeah, without any swap… you’re going to have some problems. > > Surprised that you don’t have any enabled by default. (at least 2x the > > amount of RAM to start… ) > > > > On Wed, Sep 24, 2008 at 9:02 AM, Roberto R [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) > > wrote: > > > > > > else… but still… swapping is just as bad … and 512MB should be > > > > -/+ buffers/cache: 413 97 > > > > > > > > > some swap space? > > > > > > > > > > > > > > > – > > Robby R. > > Chief Evangelist, Partner > > > > PLANET ARGON, LLC > > design // development // hosting > > > > [http://www.planetargon.com/](http://www.planetargon.com/) > > [http://www.robbyonrails.com/](http://www.robbyonrails.com/) > > aim: planetargon > > > > +1 503 445 2457 > > +1 877 55 ARGON [toll free] > > +1 815 642 4068 [fax] ---
**Author:** ![dxk3355](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/d6d6ee/32.png) [@dxk3355](https://www.ruby-forum.com/u/dxk3355)\ **Post date:** [February 23, 2009, 1:13pm UTC](https://www.ruby-forum.com/t/ultra-sphinx-reindexing-eats-up-a-lot-of-memory/149354/11 "2009-02-23T13:13:40Z")
I would like to know how to use cron with sphinx. I would be kind enough to assist me? Thanks, Emeka