Forum: Rails deployment thin server recently started hanging

I have been running nginx/thin for about a month now with no issues.
For some reason today my first thin server on port 3000 has been sucking
up all the cpu and a lot of memory and if you try a make a web request
it either takes a very very long time to come back or times out.

if i restart thin the server works for a bit but then has the same
problem.  thin has locked up at least 5 times today.

i haven't made any changes to the site and i can't find any errors in
the nginx, ruby production or thin logs.  anyone have any ideas?

thin 0.8.2 codename Double Margarita
ruby 1.8.6 (2007-09-24 patchlevel 111) [x86_64-linux]

under ubuntu hardy 8.04

thanks!

Scott,

How many thins do you have?

What is the load average?

How many request are hitting your site?

--
Aníbal Rojas
http://rubycorner.com
http://hasmanydevelopers.com
http://anibal.rojas.com.ve

On Sun, Sep 7, 2008 at 7:59 PM, Scott Kulik
<ruby-forum-incoming@andreas-s.net> wrote:
> the nginx, ruby production or thin logs.  anyone have any ideas?
> >
>



--
Aníbal

Aníbal Rojas wrote:
> Scott,
> 
> How many thins do you have?
> 
> What is the load average?
> 
> How many request are hitting your site?
> 
> --
> An�bal Rojas
> http://rubycorner.com
> http://hasmanydevelopers.com
> http://anibal.rojas.com.ve
> 
> On Sun, Sep 7, 2008 at 7:59 PM, Scott Kulik
> <ruby-forum-incoming@andreas-s.net> wrote:
>> the nginx, ruby production or thin logs.  anyone have any ideas?
>> >
>>
> 
> 
> 
> --
> An�bal

2 thin clients with default settings.

the load average is usually very low...definitely less than 1.  when it 
has issues it gets up to 3 or 4.

about 20,000 page views a day with an average of 25 users on at a time.

i also tried switching to mongrel but i'm still having the same issues 
with the rails server hanging.

i wonder if this means it is somehow related to nginx since i havent 
changed any of the rails code recently.

VPS? Dedicated Server? How much RAM and CPU do you have?

Take a look at log/production.log how much time does your app need to
service a request?  Also check:

  -  Look for a bottleneck at DB level (time spend in queries)
  -  Do you have any particular slow controller/action?
  -  Does the time required to service a request is short when
servicing a low number of request and grows a lot at peak moments?

--
Aníbal

On Sun, Sep 7, 2008 at 10:38 PM, Scott Kulik

Aníbal Rojas wrote:
> VPS? Dedicated Server? How much RAM and CPU do you have?
> 
> Take a look at log/production.log how much time does your app need to
> service a request?  Also check:
> 
>   -  Look for a bottleneck at DB level (time spend in queries)
>   -  Do you have any particular slow controller/action?
>   -  Does the time required to service a request is short when
> servicing a low number of request and grows a lot at peak moments?
> 
> --
> Aníbal
> 
> On Sun, Sep 7, 2008 at 10:38 PM, Scott Kulik

this is a 512MB RAM and 1024MB swap slice from slicehost.com.

4 64-bit AMD cpu's.

some of index views on controllers pages take .5 - 1 second to load.

the weird thing is that now i can't even get my web services to stay up 
for 10 minutes without it starting to timeout which makes me think 
something weird is going on.  this site has been running for well over a 
month without any crashes.  do you think this could be related to the 
hardware underlying on the virtual slice?

the site is wlodb.com if you want to take a look at how long it takes 
(unless i just restarted the services).

how should i check how long the DB queries are taking?  should i turn on 
verbose logging in mysql?  I have on slow query logging but haven't seen 
any today.

thanks for the help so far!
i have even tried to switch to a mongrel cluster but have the same 
results.

Scott,

>> On Sun, Sep 7, 2008 at 10:38 PM, Scott Kulik
>
> this is a 512MB RAM and 1024MB swap slice from slicehost.com.
>
> 4 64-bit AMD cpu's.

   Does this slice also runs the database?

> some of index views on controllers pages take .5 - 1 second to load.

   This is the problem, having request that take up to a second to be
serviced is too much. You are getting a lot of queued requests.

> the weird thing is that now i can't even get my web services to stay up
> for 10 minutes without it starting to timeout which makes me think
> something weird is going on.  this site has been running for well over a
> month without any crashes.  do you think this could be related to the
> hardware underlying on the virtual slice?

Your hardware should be fine for a moderated load (whatever "moderated" 
is)
You need to pin point why your request are taking so long to be 
serviced.

> the site is wlodb.com if you want to take a look at how long it takes
> (unless i just restarted the services).

:-)

> how should i check how long the DB queries are taking?  should i turn on
> verbose logging in mysql?  I have on slow query logging but haven't seen
> any today.

With INFO (default for production) you should be ok, look at this 
sample:

Completed in 0.06489 (15 reqs/sec) | Rendering: 0.05481 (84%) | DB:
0.00109 (1%) | 200 OK [http://:/logi
n]

If things are OK, time spent on DB should be a little fraction of the
total and rendering should take most of the time.

You need to assure _at least_ a a dozen request per second (in my
experience) to offer a smooth operation.

> thanks for the help so far!

You are welcome.

> i have even tried to switch to a mongrel cluster but have the same
> results.

Thin and Mongrel, are architected in different ways. You real problem
is the slow actions.

You can try adding another thin/mongrel but while you have
controller/actions taking up to a second to service a request you will
be in trouble.

--
Aníbal

ok thanks, i'll take a look at optimizing the controllers that are
performing slower.  thanks for all the help!

You are welcome, good luck

--
Aníbal Rojas
http://hasmanydevelopers.com
http://rubycorner.com
http://anibal.rojas.com.ve

On Tue, Sep 9, 2008 at 10:48 AM, Scott Kulik
<ruby-forum-incoming@andreas-s.net> wrote:
>
> ok thanks, i'll take a look at optimizing the controllers that are
> performing slower.  thanks for all the help!
> --
> Posted via http://www.ruby-forum.com/.
>
> >
>



--
Aníbal