Forum: Rails deployment thin server recently started hanging

Debcfd8f3f533800e2711c704825f9c0?d=identicon&s=25 Scott Kulik (kuliksco)
on 2008-09-08 02:29
I have been running nginx/thin for about a month now with no issues.
For some reason today my first thin server on port 3000 has been sucking
up all the cpu and a lot of memory and if you try a make a web request
it either takes a very very long time to come back or times out.

if i restart thin the server works for a bit but then has the same
problem.  thin has locked up at least 5 times today.

i haven't made any changes to the site and i can't find any errors in
the nginx, ruby production or thin logs.  anyone have any ideas?

thin 0.8.2 codename Double Margarita
ruby 1.8.6 (2007-09-24 patchlevel 111) [x86_64-linux]

under ubuntu hardy 8.04

thanks!
C381828d1907912eab30cbe38d5ea245?d=identicon&s=25 Aníbal Rojas (Guest)
on 2008-09-08 03:56
(Received via mailing list)
Scott,

How many thins do you have?

What is the load average?

How many request are hitting your site?

--
Aníbal Rojas
http://rubycorner.com
http://hasmanydevelopers.com
http://anibal.rojas.com.ve

On Sun, Sep 7, 2008 at 7:59 PM, Scott Kulik
<ruby-forum-incoming@andreas-s.net> wrote:
> the nginx, ruby production or thin logs.  anyone have any ideas?
> >
>



--
Aníbal
Debcfd8f3f533800e2711c704825f9c0?d=identicon&s=25 Scott Kulik (kuliksco)
on 2008-09-08 05:08
Aníbal Rojas wrote:
> Scott,
>
> How many thins do you have?
>
> What is the load average?
>
> How many request are hitting your site?
>
> --
> An�bal Rojas
> http://rubycorner.com
> http://hasmanydevelopers.com
> http://anibal.rojas.com.ve
>
> On Sun, Sep 7, 2008 at 7:59 PM, Scott Kulik
> <ruby-forum-incoming@andreas-s.net> wrote:
>> the nginx, ruby production or thin logs.  anyone have any ideas?
>> >
>>
>
>
>
> --
> An�bal

2 thin clients with default settings.

the load average is usually very low...definitely less than 1.  when it
has issues it gets up to 3 or 4.

about 20,000 page views a day with an average of 25 users on at a time.

i also tried switching to mongrel but i'm still having the same issues
with the rails server hanging.

i wonder if this means it is somehow related to nginx since i havent
changed any of the rails code recently.
C381828d1907912eab30cbe38d5ea245?d=identicon&s=25 Aníbal Rojas (Guest)
on 2008-09-08 14:03
(Received via mailing list)
VPS? Dedicated Server? How much RAM and CPU do you have?

Take a look at log/production.log how much time does your app need to
service a request?  Also check:

  -  Look for a bottleneck at DB level (time spend in queries)
  -  Do you have any particular slow controller/action?
  -  Does the time required to service a request is short when
servicing a low number of request and grows a lot at peak moments?

--
Aníbal

On Sun, Sep 7, 2008 at 10:38 PM, Scott Kulik
Debcfd8f3f533800e2711c704825f9c0?d=identicon&s=25 Scott Kulik (kuliksco)
on 2008-09-08 14:39
Aníbal Rojas wrote:
> VPS? Dedicated Server? How much RAM and CPU do you have?
>
> Take a look at log/production.log how much time does your app need to
> service a request?  Also check:
>
>   -  Look for a bottleneck at DB level (time spend in queries)
>   -  Do you have any particular slow controller/action?
>   -  Does the time required to service a request is short when
> servicing a low number of request and grows a lot at peak moments?
>
> --
> Aníbal
>
> On Sun, Sep 7, 2008 at 10:38 PM, Scott Kulik

this is a 512MB RAM and 1024MB swap slice from slicehost.com.

4 64-bit AMD cpu's.

some of index views on controllers pages take .5 - 1 second to load.

the weird thing is that now i can't even get my web services to stay up
for 10 minutes without it starting to timeout which makes me think
something weird is going on.  this site has been running for well over a
month without any crashes.  do you think this could be related to the
hardware underlying on the virtual slice?

the site is wlodb.com if you want to take a look at how long it takes
(unless i just restarted the services).

how should i check how long the DB queries are taking?  should i turn on
verbose logging in mysql?  I have on slow query logging but haven't seen
any today.

thanks for the help so far!
i have even tried to switch to a mongrel cluster but have the same
results.
C381828d1907912eab30cbe38d5ea245?d=identicon&s=25 Aníbal Rojas (Guest)
on 2008-09-08 16:46
(Received via mailing list)
Scott,

>> On Sun, Sep 7, 2008 at 10:38 PM, Scott Kulik
>
> this is a 512MB RAM and 1024MB swap slice from slicehost.com.
>
> 4 64-bit AMD cpu's.

   Does this slice also runs the database?

> some of index views on controllers pages take .5 - 1 second to load.

   This is the problem, having request that take up to a second to be
serviced is too much. You are getting a lot of queued requests.

> the weird thing is that now i can't even get my web services to stay up
> for 10 minutes without it starting to timeout which makes me think
> something weird is going on.  this site has been running for well over a
> month without any crashes.  do you think this could be related to the
> hardware underlying on the virtual slice?

Your hardware should be fine for a moderated load (whatever "moderated"
is)
You need to pin point why your request are taking so long to be
serviced.

> the site is wlodb.com if you want to take a look at how long it takes
> (unless i just restarted the services).

:-)

> how should i check how long the DB queries are taking?  should i turn on
> verbose logging in mysql?  I have on slow query logging but haven't seen
> any today.

With INFO (default for production) you should be ok, look at this
sample:

Completed in 0.06489 (15 reqs/sec) | Rendering: 0.05481 (84%) | DB:
0.00109 (1%) | 200 OK [http://:/logi
n]

If things are OK, time spent on DB should be a little fraction of the
total and rendering should take most of the time.

You need to assure _at least_ a a dozen request per second (in my
experience) to offer a smooth operation.

> thanks for the help so far!

You are welcome.

> i have even tried to switch to a mongrel cluster but have the same
> results.

Thin and Mongrel, are architected in different ways. You real problem
is the slow actions.

You can try adding another thin/mongrel but while you have
controller/actions taking up to a second to service a request you will
be in trouble.

--
Aníbal
Debcfd8f3f533800e2711c704825f9c0?d=identicon&s=25 Scott Kulik (kuliksco)
on 2008-09-08 17:18
ok thanks, i'll take a look at optimizing the controllers that are
performing slower.  thanks for all the help!
C381828d1907912eab30cbe38d5ea245?d=identicon&s=25 Aníbal Rojas (Guest)
on 2008-09-08 17:39
(Received via mailing list)
You are welcome, good luck

--
Aníbal Rojas
http://hasmanydevelopers.com
http://rubycorner.com
http://anibal.rojas.com.ve

On Tue, Sep 9, 2008 at 10:48 AM, Scott Kulik
<ruby-forum-incoming@andreas-s.net> wrote:
>
> ok thanks, i'll take a look at optimizing the controllers that are
> performing slower.  thanks for all the help!
> --
> Posted via http://www.ruby-forum.com/.
>
> >
>



--
Aníbal
This topic is locked and can not be replied to.