Help, I need write a nginx module

Server NGINX
1

I need to write a nginx module to enhance the performance of our
service.The following is what our service doing,

1. When getting the http request from the client,  our service send

http request to another server to validate the identification of the
client.

2. If the validation passed, then send http request to another

server to get the metainfo of the file requested by the client.

3. If the metainfo is retrieved,  use the metainfo to pull file data

from backend stroage using http GET.

I've read the guide of nginx module developement, but that's not

enough, these days, I’ve been reading the code of nginx. It’s a hard
job, i confess.
I think, I should write a handler module, and using upstream to
interact with other services, as far as i know , upstream send the
response directly to the client, the only thing I can do is defining an
“input_filter” of upstream, I’m wondering whether “input_filter” can be
used to initiate another upstream request.

 Any one can help ? Thanks in advance!

Posted at Nginx Forum:

2

I don’t think this is really necessary. You can use proxy upstream or
fastcgi to connect to php (or perl or whatever), which can then do
authentication, and send back a reproxy-url header, which will cause
nginx to send the user the contents of whatever url you send back,
which can be a local file or a remote http link. this is very fast and
efficient. I have no problems proxying out over 300 megabit of traffic
from a single server in this manner, with minimal ram use, and most of
the cpu time used being taken up by the network card’s interrupts.

the only part i’m fuzzy on is the metadata. how exactly does that fit in
here?

3

Well, we divide file into blocks, each block is 4M, and files are
divided into blocks before storing them in backend storage.

The metadata here including the filename, filesize, and sha256 hash of
the blocks.

Our service need to assemble the blocks on the fly and pipelining them
to the client.

Seems subrequests are needed to send multiple requests to the backend
storage server and then assemble the response.

Is that correct?

Posted at Nginx Forum:

4

On Fri, Sep 25, 2009 at 04:49:33AM -0400, fuji246 wrote:

Well, we divide file into blocks, each block is 4M, and files are divided into blocks before storing them in backend storage.

The metadata here including the filename, filesize, and sha256 hash of the blocks.

Our service need to assemble the blocks on the fly and pipelining them to the client.

Seems subrequests are needed to send multiple requests to the backend storage server and then assemble the response.

Is that correct?

Yes, subrequests allow to assemble the response in right order.

5

On Fri, Sep 25, 2009 at 02:35:37AM -0400, fuji246 wrote:

I need to write a nginx module to enhance the performance of our service.The following is what our service doing,

1. When getting the http request from the client,  our service send http request to another server to validate the identification of the client.

2. If the validation passed, then send http request to another server to get the metainfo of the file requested by the client.

3. If the metainfo is retrieved,  use the metainfo to pull file data from backend stroage using http GET.

I've read the guide of nginx module developement, but that's not enough, these days, I've been reading the code of nginx. It's a hard job, i confess.
I think, I should write a handler module, and using upstream to interact with other services, as far as i know , upstream send the response directly to the client,  the only thing I can do is defining an "input_filter" of upstream, I'm wondering whether "input_filter" can be used to initiate another upstream request.

It’s not easy task.
However, you may try to use X-Accel-Redirect:

location / {
    proxy_pass   http://auth;
    # it returns "X-Accel-Redirect: /meta/..." for valid users
}

location /meta/ {
    proxy_pass   http://meta;
    # it returns "X-Accel-Redirect: /storage/..."
}

location /storage/ {
    proxy_pass   http://storage;
}
6

This is something I’d like to do as well. How do subrequests work?

7

I just can’t believe it.

I‘ve got it.

Thanks for your relay, Igor,

Thanks for your excellent piece of art.

Posted at Nginx Forum: