Production trouble

Hello everybody,

I must say that putting rails in production has been a rather
frustrating experience. It maybe just me of course…

At first I simulated the deployment process on my home computer. I
wrote up deploy.rb, played with it until it worked right, deployed at
least a dozen times locally (into another directory) and finally
decided to deploy to my production machine.

The first thing that happened on the remote machine is that
mongrel_cluster started mongrel instances in the development mode. So,
I had to put environment: production into mongrel_cluster.yml. OK,
that worked fine. So now I had 4 instances of mongrel running in the
production mode. I could connect to each of them on their respective
ports no problem. So far so good.

Then I started configuring Apache. In my setup, Apache listens on 9000
and forwards the requests to a balancer (using the standard issue
proxy balancer setup). The problem is that when I access Apache on
9000, the application starts in the development mode! How could that
be? If I modify environment.rb and force the production mode, it works
OK.

But why the heck is all this defaulting to the development mode?

Thanks for any tips…

Sergei

I’ve heard of this happening on some shared hosts. The only solution
I’ve ever heard of is to force the environment.rb into production
mode.

That said, I have to say it – Use Passenger (modrails) if you have
the access to install it. I’ve had zero problems since I’ve converted
my servers to it.


TekWiz

Never heard of Passenger, but I’m eager to look into it. I’m glad I’m
not the only one though. How do I force the environment only on the
remote machine? I mean I don’t want to take environment.rb from under
subversion. What’s the clean way of doing it?

I would like to see your mongrel_cluster.yml file which needs to be
different from dev to test to production… Its best to create this
file on the fly when deploying through capistrano. (Using an ERB
template )

I’m not sure how you could get to a development rails through apache
if your mongrels was started in production mode… Is the standard
proxy setup proxypass / balancer://mongrels ?

Hello,

Here’s my mongrel_setup.yml:


log_file: /var/www/bugtracker/current/log/mongrel.log
port: 2600
pid_file: /var/www/bugtracker/current/tmp/pids/mongrel.pid
servers: 5
environment: production

I’m not sure how you could get to a development rails through apache
if your mongrels was started in production mode…

It must be something I did wrong. I’m going to keep investigating.

Is the standard proxy setup proxypass / balancer://mongrels ?

Yep, exactly.

Yes, they respond individually in production mode, but when you do it
through the balancer I see the development environment. Something is
definitely off somewhere in my configuration. But I have a couple of
ideas to check on.

I’ll be back within a few days with an update.

Thanks for the tips!

hi,

I would recommend that you add a few more lines but I don’t think they
would matter to the issues that you have BUT it would make your setup
more secure…


port: 2600
servers: 5
address: 127.0.0.1
cwd: /var/www/bugtracker/current
pid_file: tmp/pids/olex-mongrel.pid
log_file: log/mongrel.log
user: mongrel
group: mongrel
environment: production

If you have links installed on the server or just use wget you should
be able to hit 2600-2604 and get a “production” rails environment…
and your balancer would look something like this:

ProxyPreserveHost On
<Proxy balancer://mongrels>
BalancerMember http://127.0.0.1:2600 max=1 timeout=120 acquire=1
BalancerMember http://127.0.0.1:2601 max=1 timeout=120 acquire=1
BalancerMember http://127.0.0.1:2602 max=1 timeout=120 acquire=1
BalancerMember http://127.0.0.1:2603 max=1 timeout=120 acquire=1
BalancerMember http://127.0.0.1:2604 max=1 timeout=120 acquire=1

Ok, I know what’s going on but I can’t seem to be able to fix it.
What’s happening is that Apache is not forwarding requests for the
application to the balancer. And I can’t figure out why. Here’s my
Apache config (sorry if it’s too verbose):

<Proxy balancer://bugtracker>
BalancerMember http://127.0.0.1:2600
BalancerMember http://127.0.0.1:2601
BalancerMember http://127.0.0.1:2602
BalancerMember http://127.0.0.1:2603
BalancerMember http://127.0.0.1:2604

ExtendedStatus On

<VirtualHost 127.0.0.1:9000>
ServerName bugtracker
DocumentRoot /var/www/bugtracker/current/public

<Location /server-status>
SetHandler server-status

<Location /balancer-manager>
SetHandler balancer-manager

<Directory /var/www/bugtracker/current/public">
Options FollowSymLinks
# AllowOverride None
Order allow,deny
Allow from all

RewriteEngine On
RewriteLog /var/log/apache2/myapp_rewrite_log
RewriteLogLevel 9 # This has no effect – nothing is written to the
log for some reason

RewriteRule .* balancer://bugtracker%{REQUEST_URI} [P,QSA,L]

RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
RewriteCond %{SCRIPT_FILENAME} !maintenance.html
RewriteRule ^.*$ /system/maintenance.html [L]

Rewrite index to check for static

RewriteRule ^/$ /index.html [QSA]

Rewrite to check for Rails cached page

RewriteRule ^([^.]+)$ $1.html [QSA]

Redirect all non-static requests to cluster

RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f

RewriteRule ^/bugtracker(.*)$ balancer://bugtracker%{REQUEST_URI}
[P,QSA,L]

AddOutputFilterByType DEFLATE text/html text/plain text/xml
application/xml application/xhtml+xml text/javascript text/css
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

ErrorLog /var/log/apache2/rails_errors.log
CustomLog /var/log/apache2/rails.log combined
/

I took the config from here:
http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you
and modified the appropriate stuff.

Whenever I try something like “links http://localhost:9000”, I get
“Forbidden”. http://localhost:9000/server-status and
http://localhost:9000/balancer-manager don’t work either.

Oh, by the way, with AllowOverride None uncommented, none of the
directives in .htaccess under public were allowed – as expected I
think. I listened on ports 2600 through 2604 and nothing came in. So
that’s how I know for sure that the requests were not forwarded. The
reason why it works at all (even though in the development mode) is
because apache interprets dispatch.fcgi thanks to the DocumentRoot in
the config.

Any ideas?

Alright, I got it all working correctly. It was all because of haste.
I made a bunch of errors in the apache config and I paid for it… But
at least I was helped by you guys. Thanks very much! Here’s my final
config (no ProxyPass).

<VirtualHost 127.0.0.1:9000>
ServerName bugtracker
DocumentRoot /var/www/bugtracker/current/public

RewriteEngine On

<Proxy balancer://bugtracker>
BalancerMember http://127.0.0.1:2600
BalancerMember http://127.0.0.1:2601
BalancerMember http://127.0.0.1:2602
BalancerMember http://127.0.0.1:2603
BalancerMember http://127.0.0.1:2604

RewriteEngine On
RewriteLog /var/log/apache2/myapp_rewrite.log
RewriteLogLevel 9

RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
RewriteCond %{SCRIPT_FILENAME} !maintenance.html
RewriteRule ^.*$ /system/maintenance.html [L]

Rewrite index to check for static

RewriteRule ^/$ /index.html [QSA]

Rewrite to check for Rails cached page

RewriteRule ^([^.]+)$ $1.html [QSA]

Redirect all non-static requests to cluster

RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ balancer://bugtracker%{REQUEST_URI} [P,QSA,L]

AddOutputFilterByType DEFLATE text/html text/plain text/xml
application/xml application/xhtml+xml text/javascript text/css
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html <Proxy *>
Order deny,allow
Allow from all

Custom log file locations

ErrorLog /var/log/apache2/rails_errors.log
CustomLog /var/log/apache2/rails.log combined

Notice that in my original config I made a mistake in this line:
RewriteRule ^/(.*)$ balancer://bugtracker%{REQUEST_URI} [P,QSA,L].

It was (and I don’t know what made me do that!):
RewriteRule ^/(.bugtracker*)$ balancer://bugtracker%{REQUEST_URI}
[P,QSA,L].

My only question is, why specify “AllowOverride None” at all in the
options for the public directory? Curiously, I got no problems with it
locally, but I had to comment it out when deploying remotely. Sorry
for frequent posting. I’m doing it as I go :slight_smile:

I forgot to take out the line "RewriteRule .*
balancer://bugtracker%{REQUEST_URI}
[P,QSA,L] " – I was just trying to force everything to the balancer.
The rewrite log is finally working and here’s what I get there on
“links http://localhost:9000/tickets

127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768]
[rid#83e37d0/initial] (2) init rewrite engine with requested uri /
tickets
127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768]
[rid#83e37d0/initial] (3) applying pattern ‘.*’ to uri ‘/tickets’
127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768]
[rid#83e37d0/initial] (2) rewrite ‘/tickets’ → ‘balancer://bugtracker/
tickets’
127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768]
[rid#83e37d0/initial] (2) forcing proxy-throughput with
balancer://bugtracker/tickets
127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768]
[rid#83e37d0/initial] (1) go-ahead with proxy request proxy:balancer://
bugtracker/tickets [OK]

But the browser still show the forbidden page and nothing comes in on
ports 2600 through 2604.

This worked:

<VirtualHost 127.0.0.1:9000>
ServerName bugtracker
DocumentRoot /var/www/bugtracker/current/public

RewriteEngine On

<Proxy balancer://bugtracker>
BalancerMember http://127.0.0.1:2600
BalancerMember http://127.0.0.1:2601
BalancerMember http://127.0.0.1:2602
BalancerMember http://127.0.0.1:2603
BalancerMember http://127.0.0.1:2604

Redirect all non-static requests to thin

RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ balancer://bugtracker%{REQUEST_URI} [P,QSA,L]

ProxyPass / balancer://bugtracker/
ProxyPassReverse / balancer://bugtracker/
ProxyPreserveHost on

<Proxy *>
Order deny,allow
Allow from all

Custom log file locations

ErrorLog /var/log/apache2/rails_errors.log
CustomLog /var/log/apache2/rails.log combined

Found that config here:
http://articles.slicehost.com/2008/5/9/ubuntu-hardy-apache-rails-and-mongrels

I was missing these lines of course:

ProxyPass / balancer://bugtracker/
ProxyPassReverse / balancer://bugtracker/
ProxyPreserveHost on

But the example at
http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you/
didn’t have them either… And it did work on my local machine. What
was my mistake precisely you think?