Proxy uploading

Server NGINX
#1

Hi,

Currently using Nginx 0.7.65-1 on Ubuntu 10.04. I need to have my
application upload files to another backend, so I thought I could use
proxy pass, and have a certain URL be proxied to another machine
(proxy_pass http://upstreamserver.com:8080/API/import) . The URL would
be http//portalvm/API/upload/ that I upload to.

Otherwise proxy to Apache running on 127.0.0.1:8000.

What is happening is that it is indeed proxying but I am getting a 404
error from the upstreamserver.com even though the URL that I am using
looks correct. The logs (error.log in Debug mode) are saying:

http proxy header:
"POST
/API/import/raw?transferid=unique321&name=p158cr86gf1i34t099c1gkn1vh51.tmp
HTTP/1.0^M
Authorization: Basic base64string^M
Host: p upstreamserver.com:8080^M
Connection: close^M
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8^M
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8^M
Accept-Language: en,en-us;q=0.7,sv;q=0.3^M
Accept-Encoding: gzip,deflate^M
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7^M
Content-Type: application/octet-stream^M
RunAs: username^M
INDEX: 0^M
Referer: http://portalvm/my/uploadpage/^M
Content-Length: 2809716^M
Cookie: sessionid=09520e8dfd27d9ee86781b928ed20689^M
Pragma: no-cache^M
Cache-Control: no-cache^M
^M
"

Then there are a lot of logs streaming the file to the upstream server
such as:

http upstream request:
“/API/import/raw?transferid=unique321&name=p158csbronq6718k215au1ctp1ulo1.tmp”
2010/07/28 17:57:39 [debug] 3950#0: *4 http upstream send request
handler
2010/07/28 17:57:39 [debug] 3950#0: *4 http upstream send request
2010/07/28 17:57:39 [debug] 3950#0: *4 read: 9, 0000000000C9B640, 8192,
0
2010/07/28 17:57:39 [debug] 3950#0: *4 chain writer buf fl:0 s:725
2010/07/28 17:57:39 [debug] 3950#0: *4 chain writer buf fl:0 s:345
2010/07/28 17:57:39 [debug] 3950#0: *4 chain writer buf fl:0 s:8192
2010/07/28 17:57:39 [debug] 3950#0: *4 chain writer in: 0000000000C9AD88
2010/07/28 17:57:39 [debug] 3950#0: *4 writev: 9262
2010/07/28 17:57:39 [debug] 3950#0: *4 chain writer out:
0000000000000000

Then at the end::

2010/07/28 17:58:09 [debug] 3950#0: *4 http upstream process header
2010/07/28 17:58:09 [debug] 3950#0: *4 malloc: 0000000000CECF20:4096
2010/07/28 17:58:09 [debug] 3950#0: *4 recv: fd:15 260 of 4096
2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy status 404 “404 Not
Found”
2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header: “X-Powered-By:
Servlet/2.5”
2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header: “Server: Sun
GlassFish Enterprise Server v2.1.1”
2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header: “Content-Type:
text/plain”
2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header:
“Content-Length: 57”
2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header: “Date: Wed, 28
Jul 2010 15:58:09 GMT”
2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header: “Connection:
close”
2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header done
2010/07/28 17:58:09 [debug] 3950#0: *4 malloc: 0000000000CEDF30:4096
2010/07/28 17:58:09 [debug] 3950#0: *4 HTTP/1.1 404 Not Found^M
Server: nginx/0.7.65^M
Date: Wed, 28 Jul 2010 15:58:09 GMT^M
Content-Type: text/plain^M
Transfer-Encoding: chunked^M
Connection: keep-alive^M
X-Powered-By: Servlet/2.5^M
Content-Encoding: gzip^M

What I can’t understand is the headers look correct, and so does the
HTTP upstream request URL. In fact if I try and use the same headers and
URL in a util that lets me post to the server it creates an empty file.

Any idea on why I am getting a 404?

Thanks,

Tim.

In my nginx.conf I have this (base64string - obviously changed):

http {
include /etc/nginx/mime.types;
gzip on;
gzip_comp_level 2;
gzip_proxied any;
gzip_disable msie6;
gzip_types text/plain text/css application/x-javascript text/xml
application/xml application/xml+rss text/javascript;

upstream portalvm {
    server 127.0.0.1:8000;
}

server {
    listen 80;

    location ~ ^/(favicon.ico|robots.txt|sitemap.xml)$ {
        alias /opt/media/$1;
        expires 30d;
    }
    location /sitemedia {
        alias /opt/media/;
        expires 30d;
    }
    location /API/import {
       error_log /var/log/nginx/error.log debug;
       client_max_body_size 1000M;
       proxy_pass http://upstreamserver.com:8080/API/import;
       proxy_hide_header Referer;
       proxy_hide_header Cookie;
       proxy_pass_header Content-Length;
       proxy_set_header Authorization "Basic base64string";
    }
    location / {
        proxy_pass http://portalvm;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        add_header X-Handled-By $upstream_addr;
    }
#2

On Thu, Jul 29, 2010 at 11:45:50AM +0200, Tim Child wrote:

Authorization: Basic base64string^M
Host: p upstreamserver.com:8080^M

I see strange host name: “p upstreamserver.com:8080”.

Content-Length: 2809716^M
2010/07/28 17:57:39 [debug] 3950#0: *4 http upstream send request handler

2010/07/28 17:58:09 [debug] 3950#0: *4 http proxy header done

    }

It’s better to write this as three locations:

     location = /favicon.ico {
         root /opt/media;
         expires 30d;
     }

     location = /robots.txt {
         root /opt/media;
         expires 30d;
     }

     location = /sitemap.xml {
         root /opt/media;
         expires 30d;
     }

    location /sitemedia {
        alias /opt/media/;
        expires 30d;            
    }
    location /API/import {
       error_log /var/log/nginx/error.log debug;
       client_max_body_size 1000M; 
       proxy_pass http://upstreamserver.com:8080/API/import;

These directives hide/pass header from upstream to client:

       proxy_hide_header Referer;
       proxy_hide_header Cookie;
       proxy_pass_header Content-Length;

       proxy_set_header Authorization "Basic base64string";
    }
    location / {
        proxy_pass http://portalvm;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;      
        add_header X-Handled-By $upstream_addr;      
    }

I do not understand, what you want to proxy:
“/API/upload/” or “/API/import/” ?

–
Igor S.
http://sysoev.ru/en/

#3

On 29 Jul 2010, at 14:49, Igor S. wrote:

http proxy header:
"POST /API/import/raw?transferid=unique321&name=p158cr86gf1i34t099c1gkn1vh51.tmp HTTP/1.0^M
Authorization: Basic base64string^M
Host: p upstreamserver.com:8080^M

I see strange host name: “p upstreamserver.com:8080”.

Actually that is my bad, as I didn’t want to expose the server I was
using to the whole world.

Host: upstreamserver.com:8080

        root /opt/media;
        expires 30d;        
    }

Will do.

These directives hide/pass header from upstream to client:
I don’t need as far as I know to send the Cookie and Referer to the
upstream server, but the content-length is needed for the file I think.

       add_header X-Handled-By $upstream_addr;      
   }

I do not understand, what you want to proxy:
“/API/upload/” or “/API/import/” ?

Again my bad, I mean’t /API/import/

I have actually got it working when I removed a header “INDEX” that was
getting sent.

Thanks,

Tim.

#4

Great! Thanks for the help.

#5

On Thu, Jul 29, 2010 at 03:11:13PM +0200, Tim Child wrote:

These directives hide/pass header from upstream to client:

I don’t need as far as I know to send the Cookie and Referer to the upstream server, but the content-length is needed for the file I think.

      proxy_hide_header Referer;
      proxy_hide_header Cookie;
      proxy_pass_header Content-Length;

If you do not want to pass Cookie and Referer to the upstream, then you
should use:

  proxy_set_header   Cookie    "";
  proxy_set_header   Referer   "";

–
Igor S.
http://sysoev.ru/en/