Running Mercurial using Nginx as http server

Hi,

I am having some troubles getting Mercurial (the distributed Version
Control System) to work with nginx. It uses a cgi script (hgweb.cgi)
to launch mercurial which is a python program.

I have set up the cgi support in nginx using this documentation page:
http://wiki.codemongers.com/NginxSimpleCGI

This setup with simple scripts works fine.

Now if I want to move to using Mercurial, when I call the cgi script
that triggers mercurial, I get the following Python error in my web
browser:
By the way seeing this error message means that the cgi script is
being processed, so cgi does work. It’s the cgi->python that goes
wrong.

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/var/hg/hgweb.cgi in ()
34 from mercurial.hgweb.hgweb_mod import hgweb
35 import mercurial.hgweb.wsgicgi as wsgicgi
36
37 application = hgweb(“/var/hg/repo1”, “repository name”)
38 wsgicgi.launch(application)
wsgicgi = <proxied module ‘wsgicgi’>, wsgicgi.launch = <function
launch at 0xb7c8f304>, application = <mercurial.hgweb.hgweb_mod.hgweb
object at 0xb7db050c>
/var/lib/python-support/python2.5/mercurial/hgweb/wsgicgi.py in
launch(application=<mercurial.hgweb.hgweb_mod.hgweb object at
0xb7db050c>)
62 return write
63
64 result = application(environ, start_response)
65 try:
66 for data in result:
result undefined, application = <mercurial.hgweb.hgweb_mod.hgweb
object at 0xb7db050c>, environ = {‘CONTENT_LENGTH’: ‘’,
‘CONTENT_TYPE’: ‘’, ‘DOCUMENT8ROOT’: ‘/usr/local/nginx/html’,
‘DOCUMENT_ROOT’: ‘/var/hg’, ‘DOCUMENT_URI’: ‘/hgweb.cgi’, ‘FCGI_ROLE’:
‘RESPONDER’, ‘GATEWAY_INTERFACE’: ‘CGI/1.1’, ‘HISTCONTROL’:
‘ignoreboth’, ‘HOME’: ‘/home/thomas’, ‘HTTP_ACCEPT’:
‘text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,/;q=0.5’,
…}, start_response = <function start_response at 0xb7c8f41c>

<type ‘exceptions.TypeError’>: ‘hgweb’ object is not callable
args = (“‘hgweb’ object is not callable”,)
message = “‘hgweb’ object is not callable”

I don’t know Pyhton, but it seems that hgweb which is a module is not
imported thus not callable.

In the Mercurial documentation page, it says: “Remember that the web
server runs the CGI program without any of the environment variables
that you take for granted in an interactive sessions.”

So how can I tell Python where to look the modules for? This should be
set in the cgi script I guess, but how to specify it?

I tried inside hgweb.cgi:

import sys
sys.path.insert(0, “/usr/lib/python2.5”)
sys.path.insert(0, “/usr/lib/python-support”)
sys.path.insert(0, “/usr/local/lib/python2.5”)
sys.path.insert(0, “/var/lib/python-support/python2.5/mercurial/hgweb”)
sys.path.insert(0,
“/usr/share/python-support/mercurial/mercurial/hgweb”)
sys.path.append(‘/var/lib/python-support/python2.5/mercurial/hgweb’)

import os
os.environ[“PYTHONPATH”] = “/usr/lib/python2.5”
os.environ[“PYTHONPATH”] = “/usr/lib/python-support”
os.environ[“PYTHONPATH”] = “/usr/local/lib/python2.5”

But I still get the exact same error message. Are there some Python
programmers that could help me out? I am using Ubuntu 7.10 server
edition.

On Sat, 2008-01-26 at 18:27 +0100, Thomas wrote:

Hi,

I am having some troubles getting Mercurial (the distributed Version
Control System) to work with nginx. It uses a cgi script (hgweb.cgi)
to launch mercurial which is a python program.

I have set up the cgi support in nginx using this documentation page:
http://wiki.codemongers.com/NginxSimpleCGI

This setup with simple scripts works fine.

This information is mostly for Perl scripts.

Now if I want to move to using Mercurial, when I call the cgi script
that triggers mercurial, I get the following Python error in my web
browser:
By the way seeing this error message means that the cgi script is
being processed, so cgi does work. It’s the cgi->python that goes
wrong.

I’d recommend not using CGI. There are other ways to deploy Mercurial
(as a quick Google search will tell you).

http://anil.recoil.org/blog/articles/2006/12/27/mercurial-fastcgi-module

http://www.selenic.com/pipermail/mercurial/2005-August/003640.html

Regards,
Cliff

El Sat, 26 Jan 2008 11:14:55 -0800
Cliff W. [email protected] escribió:

I’d recommend not using CGI. There are other ways to deploy
Mercurial (as a quick Google search will tell you).

I would go for mod_wsgi. There is a wiki page which show how to set-up
Apache (http://www.selenic.com/mercurial/wiki/index.cgi/mod_wsgi), and
I has been able of setting up a test repository served with Nginx using
Manlio P.'s mod_wsgi for Nginx.

First, do some version control with Mercurial:

$ mkdir /tmp/hg-test && cd /tmp/hg-test && hg-init
$ echo ‘Hello world’ > readme.txt
$ hg add readme.txt && hg ci -m’Test commit’

Cast some magic words in the Nginx configuration file:

include wsgi_vars
location / {
wsgi_pass /tmp/hg-test.wsgi;
}

Create /tmp/hg-test.wsgi with the following contents:

from mercurial.hgweb.hgwebdir_mod import hgwebdir
from mercurial.hgweb.request import wsgiapplication
application = wsgiapplication(lambda:hgwebdir(“/tmp/hgwebdir.config”))

Finally, create /tmp/hgwebdir.config, contents should look like the
following:

[web]
style = gitweb
[collections]
/tmp = /tmp

(Re)Start Nginx and point your browser to the webserver… hgweb should
appear in front of you :smiley:

Of course, you should adjust paths to your particular setup, I
used /tmp for testing, but it is not a safe setting!

Regards,

-Adrian

Hi Adrian

Thanks for your assistance. I will postpone nginx+mercurial to next
week, I currently have some other stuff to work on. But I will
definitely make it work using your guidelines. Also here is the
documentation page from nginx about mod_swgi:
http://wiki.codemongers.com/NginxNgxWSGIModule

I post it here for later reference.

Hello guys,

I have heard some horror stories about mod_wsgi. Anyway I was finally
able to make 99% mercurial work with nginx: I can browse the changes
in my webbrowser, I can push using ssh and I can pull using http.

The problem I have is that I am unable to push changes to the
published repository, I get the following error message:
"http://myserver:83/ does not appear to be an hg repository!

The hgweb.cgi file has been renamed index.cgi and is living inside
/var/www/mercurial. nginx redirects incoming cgi requests to a fastcgi
process and executes /var/www/mercurial/index.cgi without any problem.
It has ownership thomas:www-data and chmod 775.

nginx uses www-data:www-data

The published/central repository lives at: /var/hg/repo1, has
ownership thomas:www-data and chmod 775

My personal repository is located at /var/hg/repo3, has ownership
thomas:thomas and chmod 755

When I want to push to that repository using http, I issue the
following command: hg push http://myserver:83

And in response I get:

pushing to http://ubuntu:83
searching for changes
abort: ‘http://ubuntu:83/’ does not appear to be an hg repository!

However, if my personal repository is synced with the central
repository, if I try to push (even tough I know there are no changes)
I get:

pushing to http://ubuntu:83
searching for changes
no changes found

Where could the problem come from? I tried enabling http
authentication in nginx, so I type my username/password, but
afterwards I still get the same error.

On Jan 29, 2008 12:49 PM, Marcin K.

How does it push?
webdav?

I never tried mercurial pushing over http.

I would go for mod_wsgi.
(…)

One more option is to reverse-proxy hg serve…

No I don’t think it uses webdav.

I was thinking, could it problably be my hgweb.cgi file that is simply
incapable of handling pushes over http?

Here is my hgweb.cgi file that I have renamed to index.cgi:

#!/usr/bin/env python

import cgitb, os, sys
cgitb.enable()

from mercurial import hgweb

h = hgweb.hgweb("/var/hg/repo1", “My nice Rails appilcation depot”)
h.run()

As you can see it is rather small :-/

Thomas ha scritto:

Hello guys,

I have heard some horror stories about mod_wsgi.

mod_wsgi for Nginx?
What horror stories?

[…]

Manlio P.

Hurray!

I have used the following hgweb.cgi file: wget
http://www.selenic.com/repo/hg/raw-file/tip/hgweb.cgi


#!/usr/bin/env python

An example CGI script to use hgweb, edit as necessary

adjust python path if not a system-wide install:

#import sys
#sys.path.insert(0, “/path/to/python/lib” )

enable importing on demand to reduce startup time

from mercurial import demandimport; demandimport.enable()

send python tracebacks to the browser if an error occurs:

import cgitb
cgitb.enable()

If you’d like to serve pages with UTF-8 instead of your default

locale charset, you can do so by uncommenting the following lines.

Note that this will cause your .hgrc files to be interpreted in

UTF-8 and all your repo files to be displayed using UTF-8.

#import os
#os.environ[“HGENCODING”] = “UTF-8”

from mercurial.hgweb.hgweb_mod import hgweb
import mercurial.hgweb.wsgicgi as wsgicgi

application = hgweb(“/var/hg/repo1”, “repository name” )
wsgicgi.launch(application)

And I have the following error in my web browser:
http://img301.imageshack.us/img301/7619/pythontracedo9.png

I think it has to do with the sys.path.insert(0, “/path/to/python/lib”
) not well set. On ubuntu 7.10 what would the correct setting be? I
have tried many paths without success.

I have installed Apache to test my mercurial setup, and in 30s I was
able to push through http using apache.

It’s only a matter of correctly configuring nginx, which I haven’t found
yet…

I have decided to install mercurial from scratch and by compiling from
the source using this documentation page:
http://www.selenic.com/mercurial/wiki/index.cgi/UnixInstall

It all goes almost well until: $ hg debuginstall.

If I run it as a regular user, I get:

Traceback (most recent call last):
File “/usr/bin/hg”, line 11, in
mercurial.commands.run()
AttributeError: ‘module’ object has no attribute ‘run’

And if I run it as root:

Checking encoding (UTF-8)…
Checking extensions…
Checking templates…
Checking patch…
Checking merge helper…
Checking commit editor…
Checking username…
No username found, using ‘root@ubuntu’ instead
(specify a username in your .hgrc file)
No problems detected

Is that okay?

Now browsing the repository has come back finally!

So I try pushing through http, and I get the error as a regular user:
$hg push http://ubuntu:83

Traceback (most recent call last):
File “/usr/bin/hg”, line 11, in
mercurial.commands.run()
AttributeError: ‘module’ object has no attribute ‘run’

And as root: $sudo hg push http://ubuntu:83

pushing to http://ubuntu:83
searching for changes
abort: ‘http://ubuntu:83/’ does not appear to be an hg repository!

Well I am back to square one, and actually things have gotten even
worse, because now I need to push being root.

I am totally lost with that issue.

I will roll back to my previous setup using the ubuntu mercurial
package and the old hgweb.cgi file, and I’ll try to correct the
permissions or something like that. If nothing works I’ll have to move
on to another DVCS as I won’t let nginx down.

Thomas ha scritto:

mercurial.commands.run()
Checking commit editor…
Checking username…
No username found, using ‘root@ubuntu’ instead
(specify a username in your .hgrc file)
No problems detected

Is that okay?

No.
It seems that you have a problem with your Mercurial installation

You should ask for help on the mercurial mailing list.

I am totally lost with that issue.

Please read Mercurial documentation; this is certainly not a problem
with the web server.

For the last problem you should at least post the nginx config file!

[…]

Manlio P.

Well with further tests here are my conclusions:

With Apache hgweb and hgwebdir work flawlessly. I have 2 repositories
published and they all work perfectly.

Now with nginx I have tested different cgi/fastcgi setups and kept the
exact same nginx and mercurial configuration files, only the
redirection to the correct cgi/fastcgi server was changed in the
nginx.conf file (fatcgi_pass xxxxxxxxxxx)

  1. Using this CGI setup: http://wiki.codemongers.com/NginxSimpleCGI
    Browsing with hgweb OK
    Browsing with hgwebdir was not tested
    Pushing through http crashes the socket.

  2. Using this fastcgi setup: nginx.eu
    Browsing with hgweb OK
    Browsing with hgwebdir doesn’t show any repository and there is no CSS
    decoration
    Pushing through http returns “does not appear to be an hg repository”.
    Pulling and cloning through http both work

  3. Using the fasctgi setup that uses spawn-fcgi found on the mailing
    list with a spawn on 127.0.0.1:10008. The link:
    http://www.selenic.com/pipermail/mercurial/2007-August/014240.html
    Browsing with hgweb shows just as much as browsing with hgwebdir: no
    repository
    Browsing with hgwebdir shows no repository and no CSS decoration
    Pushing through http returns: “operation not supported over http”
    If I open a socket to connect to, I get in /var/log/nginx/error.log:
    “111: connection refused”

Therefore it seems that it is the fastcgi setup that is going wrong.

It’s a shame that the Python via Flup/FastCGI/WSGI doesn’t exist yet
on the nginx documentation page.

I need to make a slight adjustement to my previous email.

With apache, only hgweb.cgi allows for http pushing. With hgwebdir.cgi
I get: “operation not supported over http”

Today after a hard reboot and purging some files, mercurial has gone
back to a normal behavior. I don’t need to sudo to issue mercurial
commands.

I have been playing with different fastcgi setups with more or less
success.

Here are my nginx.conf and fasctgi.conf files:

Nginx.conf:

user www-data www-data;
worker_processes 1;

error_log /var/log/nginx/error.log;

pid /var/run/nginx.pid;

events {
worker_connections 1024;
use epoll;
}

http {
index index.html;
include /etc/nginx/mime.types;
default_type application/octet-stream;

log_format main '$remote_addr - $remote_user [$time_local] $status ’
'“$request” $body_bytes_sent “$http_referer” ’
‘“$http_user_agent” “http_x_forwarded_for”’;
#access_log logs/access.log main;

sendfile on;
tcp_nopush on;
tcp_nodelay on;

keepalive_timeout 75 20;

ignore_invalid_headers on;

PhpMyAdmin listening server

server {
listen 81;
#listen 192.168.1.1:8080;

server_name  ubuntu;

location / {
  root /var/www/phpmyadmin;
  index index.php;
}

location ~ \.php$ {
  include       /etc/nginx/fastcgi.conf;
  fastcgi_pass  127.0.0.1:10005;
  fastcgi_index index.php;
  fastcgi_param SCRIPT_FILENAME 

/var/www/phpmyadmin$fastcgi_script_name;
fastcgi_param DOCUMENT_ROOT $document_root;
}
} # End of the server directive

FastCGI server from nginx.eu

server {
listen 82;
#listen 192.168.1.1:8080;

server_name  ubuntu;

location / {
  root /var/www/mercurial;
  index index.cgi;
}

location ~ \.cgi($|/) {
  include       /etc/nginx/fastcgi.conf;

  set $script     $uri;
  set $path_info  "";

  if ($uri ~ "^(.+\.php)(/.+)") {
    set  $script     $1;
    set  $path_info  $2;
  }

  #root /var/hg;

  proxy_redirect     off;
  proxy_set_header   Host             $host;
  proxy_set_header   X-Real-IP        $remote_addr;
  proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
  fastcgi_pass  unix:/var/run/nginx/nginx-fcgi.sock;

fastcgi_pass_header Authorization;

fastcgi_intercept_errors off;

  fastcgi_param SCRIPT_FILENAME 

/var/www/mercurial$fastcgi_script_name;

fastcgi_param SCRIPT_NAME $fastcgi_script_name; # Don’t

put it for PHP but keep it otherwise
#fastcgi_param DOCUMENT_ROOT /var/hg;
fastcgi_param PATH_INFO $path_info; # Super important
or Python freaks out
}
} # End of the server directive

server {
listen 83;
#listen 192.168.1.1:8080;

server_name  ubuntu;

location / {
  root /var/www/mercurial;
  index index.cgi;
}

location ~ \.cgi($|/) {

auth_basic “Restricted area”;

auth_basic_user_file /etc/nginx/htpasswd; #htpasswd must be

created using Apache’s htpasswd program
include /etc/nginx/fastcgi.conf;

  set $script     $uri;
  set $path_info  "";

  if ($uri ~ "^(.+\.php)(/.+)") {
    set  $script     $1;
    set  $path_info  $2;
  }

  #root /var/hg;

  proxy_redirect     off;
  proxy_set_header   Host             $host;
  proxy_set_header   X-Real-IP        $remote_addr;
  proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
  fastcgi_pass  unix:/var/run/nginx/fcgi.sock;
  #fastcgi_pass  127.0.0.1:10008;
  fastcgi_pass_header Authorization;
  fastcgi_intercept_errors off;
  fastcgi_param SCRIPT_FILENAME 

/var/www/mercurial$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name; # Don’t
put it for PHP but keep it otherwise
#fastcgi_param DOCUMENT_ROOT /var/hg;
fastcgi_param PATH_INFO $path_info; # Super important
or Python freaks out
}
} # End of the server directive
} # End of the config file


And fastcgi.conf:

/etc/nginx/fastcgi.conf

fastcgi_param PATH_INFO $fastcgi_script_name;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx;

Required otherwise the .cgi content is served as plain text

fastcgi_param QUERY_STRING $query_string;

Required for dealing with POST requests

fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;

fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;

PHP only, required if PHP was built with --enable-force-cgi-redirect

#fastcgi_param REDIRECT_STATUS 200;

Here is my mercurial apache configuration file:

receive requests on all interfaces, use IP:port instead

NameVirtualHost *

Following directives only apply to the * virtual host

<VirtualHost *>

Used for creating redirection URLs

ServerName ubuntu

The directory from which httpd will serve files

DocumentRoot /var/www/mercurial

Aplly directives only to the / directory and its subdirectories

Ressource to look for when client requests an index by specifying /

You don’t need this rule it is useless

DirectoryIndex index.cgi

treat .cgi files as CGI programs

AddHandler cgi-script .cgi

Follow symbolic links in that directory

Options ExecCGI FollowSymLinks

First Allow directives are evalutated then the Deny

Order allow,deny

All hosts are allowed access

Allow from all

AuthUserFile /etc/nginx/htpasswd

AuthName “Depot Mercurial”

AuthType Basic

Require valid-user

activate rewrite module

RewriteEngine on

Rewrite / to index.cgi

If you remove this rule, you need to type http://ubuntu/index.cgi

in firefox

Pushing to http with: hg push http://ubuntu:90 still works

RewriteRule ^$ index.cgi [L]

RewriteCond precedes a RewriteRule

You don’t need the following rules everything still works

RewriteCond %{REQUEST_FILENAME} !-f

RewriteRule (.*) index.cgi/$1 [QSA,L]

------------------------------------------------------------------------------------- I tried reducing it to its minimal unti mercurial doesn't work anymore.

How does it look like?

Again, using the exact same configuration files and different fastcgi
servers, I have had different behaviors for mercurial. And
apache+mercurial works perfectly.

Thomas ha scritto:

Today after a hard reboot and purging some files, mercurial has gone
back to a normal behavior. I don’t need to sudo to issue mercurial
commands.

I have been playing with different fastcgi setups with more or less success.

Here are my nginx.conf and fasctgi.conf files:

Nginx.conf:

[…]

  index index.cgi;
}

I don’t understand this location.

fastcgi_param SCRIPT_NAME $fastcgi_script_name; # Don’t

put it for PHP but keep it otherwise
#fastcgi_param DOCUMENT_ROOT /var/hg;
fastcgi_param PATH_INFO $path_info; # Super important
or Python freaks out
}
} # End of the server directive

What are you trying to do here?
It seems you have mixed CGI and PHP stuff.

Nginx does not supports CGI, and Python is not PHP.

And why are you mixing configuration from the http proxy module and
fastcgi module?

If you want to expose the Mercurial repository at the root directory,
then the configuration is very simple (untested):

location / {
set $path_info “”;
if ($fastcgi_script_name ~ “^(/.+)$”) {
set $path_info $1;
}

 include /etc/nginx/fastcgi_params;
 fastcgi_param   SCRIPT_NAME     "";
 fastcgi_param   PATH_INFO       $path_info;

 limit_except  GET {
     satisfy_any  on;
     deny   all;

     auth_basic            "closed site";
     auth_basic_user_file  conf/htpasswd;

 }

fastcgi_pass unix:/var/run/nginx/nginx-fcgi.sock;

}

This will allow POST requestes, but only from authenticated users.
Note that you have to properly configure Mercurial for allowing push on
HTTP.

[…]

Manlio P.

Hi guys,

A little follow up, I took Marcin K. advice and decided to
proxy to hg serve. it took me 20 minutes to setup correctly, and now I
am up and running with my repository accessible through http and with
push enabled for only authenticated users.

Best regards,

On Tue, Jan 29, 2008 at 12:49 PM, Marcin K.