[URGENT] Need help getting rails app back up desperately

I’m desperate. A few hours ago I was checking my server because we are
running a survey for a new client and everything looked good for that
app but I noticed another Rails app was quite large (twice as normal)
so I decided it was time for a restart. Big mistake. Upon restart the
application gave me a 500 error. Checking the logs I found a stack
level too deep error that after hours I still haven’t been able to
figure out. I need help desperately to get this app back on track. Has
anybody had a problem like this before? It looks like AC’s helpers.rb
and layout.rb are playing ping-pong over inherited
(inherited_without_layout and inherited_without_helper) and after a
while the stack is full and ruby calls it quits. PLEASE, HELP! This is
an application for one of our biggest clients and I need to get it
going ASAP. If you have any ideas please throw them my way. And if you
are confident you can solve this problem quickly for a fee we can
definitely work it out. My cell is 801-816-1870 and you can call me
anytime. And I do mean anytime. I really need to get this back up.
Here’s the log:

stack level too deep
/usr/local/lib/ruby/gems/1.8/gems/actionpack-1.12.3/lib/action_controller/helpers.rb:124:in
inherited_without_layout' /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.12.3/lib/action_controller/layout.rb:183:ininherited_without_helper’
/usr/local/lib/ruby/gems/1.8/gems/actionpack-1.12.3/lib/action_controller/helpers.rb:124:in
inherited_without_layout' /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.12.3/lib/action_controller/layout.rb:183:ininherited_without_helper’
/usr/local/lib/ruby/gems/1.8/gems/actionpack-1.12.3/lib/action_controller/helpers.rb:124:in
inherited_without_layout' /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.12.3/lib/action_controller/layout.rb:183:ininherited_without_helper’
/usr/local/lib/ruby/gems/1.8/gems/actionpack-1.12.3/lib/action_controller/helpers.rb:124:in
inherited_without_layout' /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.12.3/lib/action_controller/layout.rb:183:ininherited_without_helper’

### ### ### this last two lines repeat about 2200 times each in the

same order ### ### ###
/usr/local/lib/ruby/gems/1.8/gems/actionpack-1.12.3/lib/action_controller/helpers.rb:124:in
inherited_without_layout' /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.12.3/lib/action_controller/layout.rb:183:ininherited_without_helper’
/usr/local/lib/ruby/gems/1.8/gems/actionpack-1.12.3/lib/action_controller/helpers.rb:124:in
inherited' /home/iflo/domains/infocenter.iflo.com/current/app/controllers/application.rb:3 /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/dependencies.rb:140:inload’
/usr/local/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/dependencies.rb:56:in
require_or_load' /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/dependencies.rb:30:independ_on’
/usr/local/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/dependencies.rb:85:in
require_dependency' /usr/local/lib/ruby/gems/1.8/gems/rails-1.1.4/lib/dispatcher.rb:69:inprepare_application’
/usr/local/lib/ruby/gems/1.8/gems/rails-1.1.4/lib/dispatcher.rb:37:in
dispatch' /usr/local/lib/ruby/gems/1.8/gems/rails-1.1.4/lib/fcgi_handler.rb:150:inprocess_request’
/usr/local/lib/ruby/gems/1.8/gems/rails-1.1.4/lib/fcgi_handler.rb:54:in
process!' /usr/local/lib/ruby/site_ruby/1.8/fcgi.rb:600:ineach_cgi’
/usr/local/lib/ruby/site_ruby/1.8/fcgi.rb:597:in each_cgi' /usr/local/lib/ruby/gems/1.8/gems/rails-1.1.4/lib/fcgi_handler.rb:53:inprocess!’
/usr/local/lib/ruby/gems/1.8/gems/rails-1.1.4/lib/fcgi_handler.rb:23:in
`process!’
/home/iflo/domains/infocenter.iflo.com/current/public/dispatch.fcgi:24


Adrian Esteban Madrid
[email protected]
Lead Developer
Hyperxmedia, Inc.
Mobile 801-815-1870
Office 801-566-0670

Not sure I can help, but going to be difficult to tell just from a log.
What’s on (and around):

app/controllers/application.rb:3

Chris,

Thanks for replying. Here’s the start of app/controllers/application.rb:

The filters added to this controller will be run for all controllers

in the application.

Likewise will all the methods added be available for all controllers.

class ApplicationController < ActionController::Base

So the third line is just the declaration of the class. BTW, I have
not changed this file in weeks so I’m not too worried about its
contents. Nevertheless, if this is what it takes I will play with it
uncommenting things until I can figure out what is going on.

I have been googling this for so long now and I have been able to find
only one other iincident in the mailing list that has the exact same
error. I have tried to contact the guy that posted about it but I
haven’t herad back from him yet. Here’s the link:

http://groups.google.de/group/rubyonrails/browse_thread/thread/27550ec41209f6e8/22d614d839f0cf6b

I have even commented out all the code inside the
ApplicationController class and still gives me that error. The funny
thing is that even if I do ‘script/console’ I get:

[[email protected] current]# script/console
Loading development environment.
/home/iflo/domains/infocenter.iflo.com/current/vendor/rails/actionpack/lib/action_controller/helpers.rb:123:in
`inherited_without_layout’:SystemStackError: stack level too deep

Something tells me that I royally screwed. Have you seen something
like this situation happen to you? Any ideas would help.

Sincerely,

Adrian M.

Adrian M. wrote:

Something tells me that I royally screwed. Have you seen something
like this situation happen to you? Any ideas would help.

Have you checked that you’re actually running the copy of Ruby that you
think you’re running, and that the hashbang lines in dispatch.* point to
that copy of Ruby?

This sounds to me like something the the system has changed.

–Al Evans

Al,

Thanks for replying. I haven’t recompiled ruby if that is what you mean.

dispatch.fcgi -> #!/usr/local/bin/ruby
]# which ruby -> /usr/local/bin/ruby

Does that clarify it?

AEM

Hi Adrian

Here are a few suggestions/questions:

Did you upgrade Rails on your machine since the application was
launched? If so, you might want to go back and freeze Rails to the
version the application was developed with

What OS are you using? A search on ‘rails stack level too deep’ suggests
that not all Linux versions has played well with Rails at all times.

What plugins do you use? Some plugins lags behind the Rails release
schedule.

Have you tried starting the application with webrick or mongrel? If this
works it suggests your setup is not 100% ok.

Good luck
Kasper

Adrian M. wrote:

I’m desperate. A few hours ago I was checking my server because we are

Kasper,

I tried freezing to the latest (1.1.4) but I will try freezing back to
see if that fix it. Should have thought of that. UPDATE: went back one
by to to 1.1.0 and still got that same error but on 1.1.0 got another
one in rails/railties/lib/initializer: undefined method
cache_template_extensions. Here the same error I got every time I
started the console after freezing to another version:

vendor/rails/actionpack/lib/action_controller/helpers.rb:123:in
`inherited_without_layout’:SystemStackError: stack level too deep

I’m running it on Red Hat Enterprise Linux ES release 4 (Nahant Update
3) both in a production and testing server. It’s been running just
fine for over a year now. I must say this is the first time that I
can’t seem to be able to get it back up.

I have only a few plugins: acts_as_ordered_list and nested_set

I have tried running it with mongrel and get the same problem. Heck,
even trying script/console gets me the same error.

Thanks for your pointers,

AEM

Adrian M. wrote:

Here’s the log:

stack level too deep

Things to check:

  • Are you using recursive calls?
  • Have you renamed any tables in the DB recently? Are any corrupt?

Looks like your using Apache/2.0.52 (Red Hat), is that with FastCGI,
Mongrel, or?

Can you copy to app to your local box and run it via Webrick or Mongrel?

Adrian M. wrote:

error. I have tried to contact the guy that posted about it but I
haven’t herad back from him yet. Here’s the link:

http://groups.google.de/group/rubyonrails/browse_thread/thread/27550ec41209f6e8/22d614d839f0cf6b

Heres the follow up to that guy’s question:
http://www.thescripts.com/forum/thread183322.html

Bob,

Thanks for replying. I have not changed much code in the last while
and it was running fine until I restarted it so I would say there are
no recursive calls. The funny thing is that it nevers gets to do any
of my code, it dies when calling the application controller which,
btw, I’ve cleared of all my code and still does this.

This is running apache just as proxy for lighttpd running 5 processes.
I wish I could try this in my own computer but it is a large set of 13
applications sharing the db/sessions/etc. Kida complex to replicate.
Although I do have a full mirror server, testing, that I’ve been
playing with trying to figure it out.

Thanks again,

AEM

Have you tried creating a brand new rails application (eg: rails
testapp) and just seeing if that will work? If not, then I suggest
switching to another user on the system and trying to create and run
another new rails application.

I had a similarly inexplicable (although with a different error)
problem a few months ago and the steps above led me to the answer:
Running a new app as a different user solved the problem. I had a
.gem directory in my user directory which somehow had gotten
corrupted.

Good luck.

-Payton

Bob,

Thanks for the link but I think I got it wrong. That link takes me to
a DB2 discussion.

Thanks again,

AEM

Do you have nightly backups of your account? Can you just go back to
before the restart?

The problem is knowing exactly what made the code (be it in gems or
somewhere) go corrupt and when it happenned and then you know where to
get the backup from (what date). The system has been running for days
in production mode so it did not know there was a problem until it was
restarted.

AEM

You’re welcome :slight_smile:

Is it possible that the log files somehow has grown too large for the
filesystem to handle? I once ran into a 2GB limit.

Kasper

Adrian M. wrote:

Kasper,

I tried freezing to the latest (1.1.4) but I will try freezing back to
see if that fix it. Should have thought of that. UPDATE: went back one

Thanks for your pointers,

AEM

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs