DRb server crashing

Ruby Apps Ferret
1

I’m having a problem where the DRb server seems to be disappearing
(crashing?) and I don’t know how to track down the cause or work around
the problem.

Unfortunately I haven’t found a way to reproduce the problem, but it
seems to happen fairly often (maybe once a day). Other then that ferret
seems to be working well.

I can’t seem to find any trace of the crash, neither of the
ferret_server.out (blank) and ferret_server.log (tail -n 100 after a
crash attached below) seem to hold any hints to the problem.

So my questions are.

  1. Has anyone seen this behavior before, how did you work around it?
  2. If no one has seen it, any ideas of where I should look to track it
    down?
  3. Last if I can’t fix it how much work would it be to just turn off
    ferret when the server was down. That would mean indexing and searching
    wouldn’t work, but as it now stands no one can post to my site when the
    problem occurs.

Thanks,
Jesse

----------- typical crash that I see in my rails app --------------
A DRb::DRbConnError occurred in comments#create:

druby://127.0.0.1:40869 - #<Errno::ECONNREFUSED: Connection refused -
connect(2)>
/usr/local/lib/ruby/1.8/drb/drb.rb:736:in `open’

Request:

  • URL:
    http://127.0.0.1:40860/forums/writeroom/topics/18_WR_1_document_model/comments
  • Parameters: {“topic_id”=>“18_WR_1_document_model”, “commit”=>“Create
    and Save”, “action”=>“create”, “controller”=>“comments”,
    “forum_id”=>“writeroom”, “comment”=>{“comment”=>“Perhaps there is a way
    to let the two document models coexist. What if you could open, save and
    close documents the standard way, but let WriteRoom handle all open
    documents internally until they are manually closed? Even between
    sessions? I imagine this could be achieved by using an internal
    database, or perhaps a folder with "working" files under Application
    Support. This would also serve as a simple versioning system – you open
    your file, manipulate it in WriteRoom (perhaps during the course of many
    sessions) and then save the changes when your ready.\r\nThis would
    accomplish the same thing as WR 1.0 did, but would be invisible to those
    who want the familiar old file-system. I also believe this would bring
    back the "room" in WriteRoom.\r\n\r\nJust a thought.”,
    “parent_id”=>“”}}
  • Rails root: /home/jessegr/apps/blocks/releases/20070627174655

Session:

  • session id: “6c6a8092f55f2b5160197f33ec616e74”
  • data: {“flash”=>{}, :user=>15}

Environment:

  • CONTENT_LENGTH : 872

  • CONTENT_TYPE : application/x-www-form-urlencoded

  • GATEWAY_INTERFACE : CGI/1.2

  • HTTP_ACCEPT : /

  • HTTP_ACCEPT_ENCODING : gzip, deflate

  • HTTP_ACCEPT_LANGUAGE : sv-se

  • HTTP_CONNECTION : Keep-Alive

  • HTTP_CONTENT_LENGTH : 872

  • HTTP_CONTENT_TYPE : application/x-www-form-urlencoded

  • HTTP_COOKIE :
    _blocks_session_id=6c6a8092f55f2b5160197f33ec616e74;
    __utmz=26552955.1182412857.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none);
    __utmc=26552955; __utmb=26552955;
    auth_token=4618c42bf7a93f8d74e9bef8eedb1c9fbfb37690;
    __utma=26552955.823564440.1182412857.1183021598.1183026292.14

  • HTTP_HOST : 127.0.0.1:40860

  • HTTP_MAX_FORWARDS : 10

  • HTTP_REFERER :
    http://75.126.217.82/forums/writeroom/topics/18_WR_1_document_model/comments/new

  • HTTP_USER_AGENT : Mozilla/5.0 (Macintosh; U; PPC Mac OS X;
    sv-se) AppleWebKit/419.2.1 (KHTML, like Gecko) Safari/419.3

  • HTTP_VERSION : HTTP/1.1

  • HTTP_X_FORWARDED_FOR : 213.185.4.64

  • HTTP_X_FORWARDED_HOST : 75.126.217.82

  • HTTP_X_FORWARDED_SERVER: www.jesse-grosjean-temp.com

  • PATH_INFO :
    /forums/writeroom/topics/18_WR_1_document_model/comments

  • RAW_POST_DATA : [FILTERED]

  • REMOTE_ADDR : 213.185.4.64

  • REQUEST_METHOD : POST

  • REQUEST_PATH :
    /forums/writeroom/topics/18_WR_1_document_model/comments

  • REQUEST_URI :
    /forums/writeroom/topics/18_WR_1_document_model/comments

  • SCRIPT_NAME : /

  • SERVER_NAME : 127.0.0.1

  • SERVER_PORT : 40860

  • SERVER_PROTOCOL : HTTP/1.1

  • SERVER_SOFTWARE : Mongrel 1.0.1

  • Process: 4894

  • Server : spurgeon

Backtrace:

/usr/local/lib/ruby/1.8/drb/drb.rb:736:in open' /usr/local/lib/ruby/1.8/drb/drb.rb:729:in each’
/usr/local/lib/ruby/1.8/drb/drb.rb:729:in open' /usr/local/lib/ruby/1.8/drb/drb.rb:1189:in initialize’
/usr/local/lib/ruby/1.8/drb/drb.rb:1169:in new' /usr/local/lib/ruby/1.8/drb/drb.rb:1169:in open’
/usr/local/lib/ruby/1.8/drb/drb.rb:1085:in method_missing' /usr/local/lib/ruby/1.8/drb/drb.rb:1103:in with_friend’
/usr/local/lib/ruby/1.8/drb/drb.rb:1084:in method_missing' [RAILS_ROOT]/vendor/plugins/acts_as_ferret/lib/remote_index.rb:31:in <<’
[RAILS_ROOT]/vendor/plugins/acts_as_ferret/lib/instance_methods.rb:73:in
ferret_update' [RAILS_ROOT]/app/models/topic.rb:61:in update_cached_fields’
[RAILS_ROOT]/app/models/topic.rb:45:in update_cached_comment_fields' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/associations/association_proxy.rb:128:in send’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/associations/association_proxy.rb:128:in
method_missing' [RAILS_ROOT]/app/models/comment.rb:62:in update_cached_fields’
[RAILS_ROOT]/app/models/comment.rb:43:in after_update' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:352:in send’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:352:in
callback' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:271:in update_without_timestamps’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/timestamp.rb:38:in
update' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/base.rb:1959:in create_or_update_without_callbacks’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:243:in
create_or_update' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/base.rb:1693:in save_without_validation’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/validations.rb:848:in
save_without_transactions' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:105:in save’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:59:in
transaction' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/query_cache.rb:66:in send’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/query_cache.rb:66:in
method_missing' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:77:in transaction’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:97:in
transaction' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:105:in save’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:117:in
rollback_active_record_state!' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:105:in save’
[RAILS_ROOT]/app/models/comment.rb:35:in after_create' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:352:in send’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:352:in
callback' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:257:in create_without_timestamps’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/timestamp.rb:29:in
create' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/base.rb:1959:in create_or_update_without_callbacks’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:243:in
create_or_update' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/base.rb:1693:in save_without_validation’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/validations.rb:848:in
save_without_transactions' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:105:in save’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:59:in
transaction' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/query_cache.rb:66:in send’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/query_cache.rb:66:in
method_missing' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:77:in transaction’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:97:in
transaction' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:105:in save’
[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:117:in
rollback_active_record_state!' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:105:in save’
[RAILS_ROOT]/app/controllers/comments_controller.rb:52:in create' [RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:104:in call’
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:104:in
respond_to' [RAILS_ROOT]/app/controllers/comments_controller.rb:51:in create’
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/base.rb:1136:in
send' [RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/base.rb:1136:in perform_action_without_filters’
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/filters.rb:713:in
call_filters' [RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/filters.rb:752:in perform_action_without_benchmark’
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:68:in
perform_action_without_rescue' /usr/local/lib/ruby/1.8/benchmark.rb:293:in measure’
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:68:in
perform_action_without_rescue' [RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/rescue.rb:133:in perform_action_without_caching’
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/caching.rb:668:in
perform_action' [RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/query_cache.rb:99:in cache’
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/caching.rb:667:in
perform_action' [RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/base.rb:494:in send’
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/base.rb:494:in
process_without_filters' [RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/filters.rb:747:in process_without_session_management_support’
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/session_management.rb:122:in
process' [RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/base.rb:346:in process’
[RAILS_ROOT]/vendor/rails/railties/lib/dispatcher.rb:39:in dispatch' /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/rails.rb:78:in process’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/rails.rb:76:in
synchronize' /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/rails.rb:76:in process’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:618:in
process_client' /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:617:in each’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:617:in
process_client' /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:736:in run’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:736:in
initialize' /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:736:in new’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:736:in
run' /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:720:in initialize’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:720:in
new' /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:720:in run’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/configurator.rb:271:in
run' /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/configurator.rb:270:in each’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/configurator.rb:270:in
run' /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/bin/mongrel_rails:127:in run’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/command.rb:211:in
run' /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.0.1/bin/mongrel_rails:243 /usr/local/bin/mongrel_rails:16:in load’
/usr/local/bin/mongrel_rails:16

----------- ferret_server.out -----------------
EMPTY

----------- ferret_server.log -----------------
call index method: highlight with [13, “Topic”, “jesse”,
{:field=>:ferret_name, :excerpt_length=>150, :num_excerpts=>1,
:pre_tag=>“”, :post_tag=>“”}]
call index method: highlight with [13, “Topic”, “jesse”,
{:field=>:ferret_content, :excerpt_length=>150, :num_excerpts=>1,
:pre_tag=>“”, :post_tag=>“”}]
call index method: highlight with [17, “Topic”, “jesse”,
{:field=>:ferret_name, :excerpt_length=>150, :num_excerpts=>1,
:pre_tag=>“”, :post_tag=>“”}]
call index method: highlight with [17, “Topic”, “jesse”,
{:field=>:ferret_content, :excerpt_length=>150, :num_excerpts=>1,
:pre_tag=>“”, :post_tag=>“”}]
call index method: add with [{:ferret_name=>“test”,
:class_name=>“Topic”, :ferret_content=>“test”, :id=>19}]
call index method: add with [{:ferret_name=>“test”,
:class_name=>“Topic”, :ferret_content=>“test”, :id=>19}]
call index method: remove with [19, “Topic”]
call index method: add with [{:ferret_name=>“test”,
:class_name=>“Topic”, :ferret_content=>“test”, :id=>19}]
jessegr@spurgeon [~/apps/blocks/current/log]#

2

On Thu, Jun 28, 2007 at 01:39:27PM +0200, Jesse G. wrote:

crash attached below) seem to hold any hints to the problem.

So my questions are.

  1. Has anyone seen this behavior before, how did you work around it?
  2. If no one has seen it, any ideas of where I should look to track it
    down?

Difficult given that there are no log entries at all. Try to find out if
the crash is related to special queries or indexing requests, or happens
during idle time. You might use monit or something like this to track
down when exactly the DRb server went away, and automatically launch a
new one.

  1. Last if I can’t fix it how much work would it be to just turn off
    ferret when the server was down. That would mean indexing and searching
    wouldn’t work, but as it now stands no one can post to my site when the
    problem occurs.

I’ll implement handling of these errors in aaf soon (hopefully this
weekend).

Jens

–
Jens Krämer
webit! Gesellschaft für neue Medien mbH
Schnorrstraße 76 | 01069 Dresden
Telefon +49 351 46766-0 | Telefax +49 351 46766-66
[email protected] | www.webit.de

Amtsgericht Dresden | HRB 15422
GF Sven Haubold, Hagen Malessa