Nginx as reverse caching proxy for CentOS repositories

Server NGINX
1

Dear list,

In an attempt to create an offline copy for my homenetwork to quickly
install CentOS I wanted to use nginx as a reverse caching proxy so that
nginx stores all the files the that the CentOS installer requests after
the first time.

However, after most of the startup files (product.img, stage2.img etc)
are downloaded and cached succesfully, it bails on the chkconfig rpm
(which happens to be the first file it actually downloads for the
install, from the CentOS/ folder.

I’m running nginx/0.8.34, compiled with a standard ./configure (only
using the prefix, so the rest is left as default).

The following relevant changes to the default configuration were made:

proxy_cache_path /storage/cache/proxy_temp/ levels=1:2 keys_zone=one:10m
inactive=100d max_size=1000M;
proxy_temp_path /storage/cache/;

     location / {
  proxy_pass        http://mirror.oxilion.nl;
         proxy_cache             one;
         proxy_cache_key "$scheme$host$request_uri";
         proxy_cache_valid       200  100d;
     }

As far as I could google, this seemed only fair. Basic caching works as
a charm, but it somehow chokes on the chkconfig rpm. However, when I use
my proxy_pass mirror (and doing so, bypass nginx), it works like a
charm. So, the mirror is up2date and up&running.

I’ve tried emptying my cache several times, and it doesn’t matter.

This will be unable to troubleshoot without some debug. I’ve enabled
caching debug and traced the (requesting) HTTP headers

T 192.168.1.113:33652 -> 192.168.1.104:80 [AP]
GET /centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm
HTTP/1.1…Accept-Encoding: identity…Range: bytes=440-25087…Connection:
close…Host: 192.168.1.104…User-agent:
urlgrabber/3.1.0 yum/3.2.22…

Accesslog:

192.168.1.113 - - [13/Mar/2010:19:16:05 +0100] “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” 200
162614 “-” “urlgrabber/3.1.0 yum/3.2.22”
192.168.1.113 - - [13/Mar/2010:19:16:06 +0100] “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” 200
143352 “-” “urlgrabber/3.1.0 yum/3.2.22”
192.168.1.113 - - [13/Mar/2010:19:16:06 +0100] “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” 200
133216 “-” “urlgrabber/3.1.0 yum/3.2.22”
192.168.1.113 - - [13/Mar/2010:19:16:06 +0100] “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” 200
162614 “-” “urlgrabber/3.1.0 yum/3.2.22”
192.168.1.113 - - [13/Mar/2010:19:16:06 +0100] “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” 200
162614 “-” “urlgrabber/3.1.0 yum/3.2.22”
192.168.1.113 - - [13/Mar/2010:19:16:06 +0100] “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” 200
162614 “-” “urlgrabber/3.1.0 yum/3.2.22”
192.168.1.113 - - [13/Mar/2010:19:16:06 +0100] “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” 200
123238 “-” “urlgrabber/3.1.0 yum/3.2.22”
192.168.1.113 - - [13/Mar/2010:19:16:06 +0100] “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” 200
162614 “-” “urlgrabber/3.1.0 yum/3.2.22”
192.168.1.113 - - [13/Mar/2010:19:16:06 +0100] “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” 200
152040 “-” “urlgrabber/3.1.0 yum/3.2.22”
192.168.1.113 - - [13/Mar/2010:19:16:06 +0100] “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” 200
128872 “-” “urlgrabber/3.1.0 yum/3.2.22”

It tries to redownload the file several times and seems to give up after
a few failed attempts.

Cache log:

***14/Mar/2010:13:56:12 +0100 MISS Cache-Control: - Expires: - “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” (200)
“urlgrabber/3.1.0 yum/3.2.22”
***14/Mar/2010:13:56:12 +0100 MISS Cache-Control: - Expires: - “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” (200)
“urlgrabber/3.1.0 yum/3.2.22”
***14/Mar/2010:13:56:12 +0100 HIT Cache-Control: - Expires: - “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” (200)
“urlgrabber/3.1.0 yum/3.2.22”
***14/Mar/2010:13:56:13 +0100 HIT Cache-Control: - Expires: - “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” (200)
“urlgrabber/3.1.0 yum/3.2.22”
***14/Mar/2010:13:56:13 +0100 HIT Cache-Control: - Expires: - “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” (200)
“urlgrabber/3.1.0 yum/3.2.22”
***14/Mar/2010:13:56:13 +0100 HIT Cache-Control: - Expires: - “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” (200)
“urlgrabber/3.1.0 yum/3.2.22”
***14/Mar/2010:13:56:13 +0100 HIT Cache-Control: - Expires: - “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” (200)
“urlgrabber/3.1.0 yum/3.2.22”
***14/Mar/2010:13:56:13 +0100 MISS Cache-Control: - Expires: - “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” (200)
“urlgrabber/3.1.0 yum/3.2.22”
***14/Mar/2010:13:56:13 +0100 HIT Cache-Control: - Expires: - “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” (200)
“urlgrabber/3.1.0 yum/3.2.22”
***14/Mar/2010:13:56:13 +0100 HIT Cache-Control: - Expires: - “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” (200)
“urlgrabber/3.1.0 yum/3.2.22”

The file does exists, and downloads succesfully when I simply feed this
link to firefox or any other tool.

Anaconda (the CentOS installer) gives me a
“chkconfig-1.3.30.1-2.i386.rpm cannot be opened. This is due to a
missing file, a corrupt package, or corrupt media.” in the installer
screen, with no other options than retry and reboot.

For what it’s worth; I’ve also tried nginx-0.7.65 with a comparable
config, but to no avail.

Any hints or pointers I might have missed? I’ve tried several
configoptions, but none of them seem to help. If you have a comparable
setup, I’d love to see the relevant config parts.

Thanks,

Wouter

2

Hi again,

To add something, from the error log:

2010/03/14 21:31:31 [info] 19695#0: *23 client closed prematurely
connection, client: 192.168.1.113, server: localhost, request: “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1”,
host: “192.168.1.104”
2010/03/14 21:31:31 [info] 19695#0: *24 client closed prematurely
connection (104: Connection reset by peer), client: 192.168.1.113,
server: localhost, request: “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1”,
host: “192.168.1.104”
2010/03/14 21:31:31 [info] 19695#0: *25 client closed prematurely
connection (104: Connection reset by peer), client: 192.168.1.113,
server: localhost, request: “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1”,
host: “192.168.1.104”
2010/03/14 21:31:31 [info] 19695#0: *26 client closed prematurely
connection (104: Connection reset by peer), client: 192.168.1.113,
server: localhost, request: “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1”,
host: “192.168.1.104”
2010/03/14 21:31:31 [info] 19695#0: *27 client closed prematurely
connection (104: Connection reset by peer), client: 192.168.1.113,
server: localhost, request: “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1”,
host: “192.168.1.104”
2010/03/14 21:31:32 [info] 19695#0: *28 client closed prematurely
connection (104: Connection reset by peer), client: 192.168.1.113,
server: localhost, request: “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1”,
host: “192.168.1.104”
2010/03/14 21:31:32 [info] 19695#0: *29 client closed prematurely
connection (104: Connection reset by peer), client: 192.168.1.113,
server: localhost, request: “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1”,
host: “192.168.1.104”
2010/03/14 21:31:32 [info] 19695#0: *30 client closed prematurely
connection (104: Connection reset by peer), client: 192.168.1.113,
server: localhost, request: “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1”,
host: “192.168.1.104”

I tried the tips on proxy_buffering from
Hanging byte-range requests for remote backend and

proxy_ignore_client_abort on;
fastcgi_ignore_client_abort on;

from "client closed prematurely connection..." - NGINX - Ruby-Forum .

Could it be so that, as suggested on the first link, only a partial file
is received, or sent to the client ?

Wouter

3

Hello!

On Sun, Mar 14, 2010 at 02:03:45PM +0100, Wouter Schoot wrote:

In an attempt to create an offline copy for my homenetwork to
quickly install CentOS I wanted to use nginx as a reverse caching
proxy so that nginx stores all the files the that the CentOS
installer requests after the first time.

However, after most of the startup files (product.img, stage2.img
etc) are downloaded and cached succesfully, it bails on the
chkconfig rpm (which happens to be the first file it actually
downloads for the install, from the CentOS/ folder.

[…]

This will be unable to troubleshoot without some debug. I’ve enabled
caching debug and traced the (requesting) HTTP headers

T 192.168.1.113:33652 -> 192.168.1.104:80 [AP]
GET /centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm
HTTP/1.1…Accept-Encoding: identity…Range:
bytes=440-25087…Connection: close…Host: 192.168.1.104…User-agent:
urlgrabber/3.1.0 yum/3.2.22…

Most likely your client (incorrectly) assumes that range request
must by satisfied, and bails out due to getting full reply (which
happens when url in question isn’t yet cached).

Looking into client code may shed some light on the problem.

[…]

Maxim D.

4

Hi Maxim,

Thanks for your reply!

I do however have replies which state there is a cache-hit, so there’s
no need to refetch the file:

***14/Mar/2010:21:31:32 +0100 HIT Cache-Control: - Expires: - “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” (200)
“urlgrabber/3.1.0 yum/3.2.22”
***14/Mar/2010:21:31:32 +0100 HIT Cache-Control: - Expires: - “GET
/centos/5.4/os/i386/CentOS/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1” (200)
“urlgrabber/3.1.0 yum/3.2.22”

(using the following log-line: log_format cache '***$time_local ’
'$upstream_cache_status - $upstream_status - ’
'Cache-Control: $upstream_http_cache_control ’
'Expires: $upstream_http_expires ')

Most likely your client (incorrectly) assumes that range request
must by satisfied, and bails out due to getting full reply (which
happens when url in question isn’t yet cached).

Looking into client code may shed some light on the problem.

So it probably lies somewhere in the Range header. Is there some setting
I can use in nginx to override or obey whatever the CentOS installer
wants? A client-hack in nginx’ config, so to say?

But, now that I think of it… When nginx has the file cached (HIT
cachestatus), how does nginx’ reply to the client differ from the
original one… When I directly fetch the rpm from the backend mirror
(during install), there’s no problem at all. Only when I put nginx in
between, with or without the rpm in cache, it fails on me.

Wouter

5

Hi,

Is there anyone who can shed a light on this issue? It somehow seems to
misnegotiate the range headers. Can I somehow (on nginx) ignore this and
make it all work so that the reverse caching proxy will work for my
installs?

Wouter

6

Hello!

On Thu, Mar 18, 2010 at 07:30:02PM +0100, Wouter Schoot wrote:

Could this somehow be related to http 1.1 to the client and 1.0 to
the backend? Will that mess up the Range header? Any idea’s ?

Have you actually tried looking into your client’s code as
I’ve already suggested?

Given the fact that there are multiple requests with Range header
and some of them replied with 200 (and full content) it’s pretty
obvious that client which doesn’t handle 200 correctly will get
garbled result. And you can’t do anything with it without
actually fixing client (or stopping it somehow from using range
requests).

Maxim D.

7

Hi Maxim,

Thanks for replying,

Have you actually tried looking into your client’s code as
I’ve already suggested?

Given the fact that there are multiple requests with Range header
and some of them replied with 200 (and full content) it’s pretty
obvious that client which doesn’t handle 200 correctly will get
garbled result. And you can’t do anything with it without
actually fixing client (or stopping it somehow from using range
requests).

I understand what you’re saying. However, since “my client” is the
CentOS/RedHat installer (Anaconda is used during that installer to
download and install the *.rpm files), I’m not going to be able to modfy
it’s code I think. And besides, I’m not that good at reading code.

I’m now, since I’m quite sure it is indeed the installer that goes
wrong, looking for a way to work around it. Perhaps some server-side
voodoo or (as I tried tonight) another proxy in between that filters
some of the headers (I tried pound, but to no avail yet).

Wouter

8

On 18-3-2010 21:51, Maxim D. wrote:

Have you actually tried looking into your client’s code as
I’ve already suggested?

It seems it’s a known “bug”:
https://bugzilla.redhat.com/show_bug.cgi?id=530900

9

Could this somehow be related to http 1.1 to the client and 1.0 to the
backend? Will that mess up the Range header? Any idea’s ?