Re: aggregating multiple upstream responses before finalizing the down stre am request

---------- Forwarded Message ----------
[email protected]… <[email protected]…> writes:

B) the headend nginx server (which has my module in it…) receives the request and passes the request on to three or more upstream servers… upstream1, upstream2, and upstream3

Sorry if I don’t interpret correctly your idea but you really mean you want to issue the same request on each of the upstream servers in a parallel way?

That is correct… the single request from downstream is to be sent to a
plurality of upstream servers in a parallel manner such that there would
be 2 or more pending upstream connections for the given downstream
request.

D) during this time, the headend server is waiting for all three upstream responses

You mean actively waiting or waiting for an event on the upstream connection? If you have to do an active wait waiting for all of the upstream servers to reply it will be catastrophic because you already divided by 3 the processing power of the uptreams servers and in case of the failure of one server latency will be added due to error processing for each request done!
The requests would be non-blocking. Since nginx already does not block
on upstream requests, I simply want to add the facility for multiple
upstream requests - I’m looking at either: daisy chaining the requests
(request->upstream->second upstream->third upstream->etc… where the
processing of multiples upstreams is confined to the upstream module) or
making the request->upstream in the http module into an array of
upstreams (i.e. request->*upstream)

Question: do you think that multiple parallel upstream requests are possible as a module and not a branch of the nginx source?

What would be interesting AMHO would be a heartbeat protocol checking upstream health, may be as a thread inside Nginx and manipulating upstream status (it’s not so easy to do because there could be multiple workers and critical section issues). But sadly, no light heartbeat protocol is defined and normalized.

The focus of my module is to -compare- results from multiple parallel
servers, not monitoring upstream server health.
The closer-to-real-world-example is:
old server A with old crappy hard to maintain (hmmm Apache anyone)
database app
new server B with new shiny happy easy to configure (nginx) front
end to new shiny database app
new server C with nginx (and my fault tolerance module) acting as proxy
to servers A and B
If the process of migrating the database app from server A to B breaks
the results returned by the app, then tell me with a POST to my own
server… cause we gotta know if the two apps are not returning the same
data…
When the new server B is --verified-- to be equivilent to old server A,
then we can retire server A — but not before.

Question: Is there a mechanism by which I can inject a request into the nginx
queue as if it came from a downstream client?

Yes, I’ve done it, but for now it’s Linux [recent kernel] only by using signalfd() and a special event module based on the epoll one with circular buffers, messaging and dedicated worker threads.

It seems to me that I could send the request to be injected to the port
I’m listening on… duh. But I’m incurring the overhead of going out to
the tcp/ip stack and OS and then returning. Surely there is a way of
chaining a new request into the queue without having to resort to that.
I know this is a dumb question but it is puzzling me.

Thanks for any comments on my endeavor. I have studied Evan M.'s guide to module development and have followed the forum intently when internals have been discussed. I am still learning nginx internals though…

Yes, me too! My advice: take your time!

Question: Is the any documentation or better yet, flow graphs, describing how requests are handled?

That’s why we all finished to land on this list :wink:

Thanks for the response so far!

[email protected]… <[email protected]…> writes:

server… cause we gotta know if the two apps are not returning the
same data…
When the new server B is --verified-- to be equivilent to old server A,
then we can retire server A — but not before. Â

Well, that way I understand, but I think it’s still a waste of
ressources, why
don’t you put a X-Header on the application server like
X-My-Application-Version=“x.y.z” and issue an HTTP HEAD request on a
special url
each hour by a script outside Nginx to do the monitoring, except if
there are
some points I don’t understand, it make no sense for me to monitor
something
like this for each request, since it’s not request related.

It seems to me that I could send the request to be injected to the port
I’m listening on… duh. But I’m incurring the overhead of going out to
the tcp/ip stack and OS and then returning. Surely there is a way of
chaining a new request into the queue without having to resort to that.Â
I know this is a dumb question but it is puzzling me.Â

It’s not a dumb question; it’s a complex question. The main problem is
to get
back the context when you have the event and without interfering with
the normal
processing of Nginx; it takes me weeks to find and validate a clean
approach.
The idea of spoofing, injecting or to fake data in a software like Nginx
is IMHO
very dangerous.

Best regards.

François Battail <[email protected]…> writes:

Well, that way I understand, but I think it’s still a waste of ressources,why
don’t you put a X-Header on the application server like
X-My-Application-Version=“x.y.z” and issue an HTTP HEAD request on a special
url
each hour by a script outside Nginx to do the monitoring, except if there are
some points I don’t understand, it make no sense for me to monitor something
like this for each request, since it’s not request related.

Server monitoring is not the target of the this module…
Every request is important…

The point of this nginx module/modification is to:

A) act as a proxy sitting in front of a legacy application server and a
new
application server

The two application servers MUST return the same results for a given GET
request…

B) compare the responses from the legacy application and that of the new
application server… If the responses are different, then the instance
should
be logged for future review of the new application server’s development.

===============

Imagine you have a crappy apache server sitting on top of a bunch of
badly
written php/python scripts which are talking to very expensive and very
slow
oracle databases which are costing you an enormous amount of money…

Now imagine that you have an nginx server with fastcgi to a well written
php/python set of scripts which in turn front a mysql database…

Your company DEPENDS ON the old crappy system and has hundreds of
gigabytes of
data sitting in the oracle database…

Changing the look and feel of the web application is NOT AN OPTION
because
employees are dumb and your call center will be flooded with calls about
“how
do I do this…? I did it yesterday… but it doesn’t work today…”

SO… you keep running the crappy server because it is the gold standard
by
which you measure the accuracy of the new servers…

Accuracy is not just measured in appearance but the actual information
being
provided. We are porting away from an oracle database with embedded sql
triggers which are now more easily maintained python scripts…

The information returned from the old server “is always considered
correct”…
the information returned from the new server “is always considered
suspect”…

Yet testing the new server in a sterile environment is not an option
because
there is no way to test all aspects of the system without real world
data…

So you write a proxy which sends all requests (GET/POST/HEAD/DELETE) to
both
the old and new servers, thus testing the new application while
simultaneously
keeping the old server up to date.

In the process, you get metrics on how much better the new system is
than the
old which makes sure you get a pay check next week. You also get a
clear log
of all differences between you work-in-progress new system and the old
gold
standard.

It seems to me that I could send the request to be injected to the port
I’m listening on… duh. But I’m incurring the overhead of going out to
the tcp/ip stack and OS and then returning. Surely there is a way of
chaining a new request into the queue without having to resort to that.Â
I know this is a dumb question but it is puzzling me.Â

It’s not a dumb question; it’s a complex question. The main problem is to get
back the context when you have the event and without interfering with the
normal
processing of Nginx; it takes me weeks to find and validate a clean approach.
The idea of spoofing, injecting or to fake data in a software like Nginx is
IMHO
very dangerous.

I agree that that can be dangerous in a server in the wild…

In this case, it allows for a testing module to be an adjunct to the
compare
module described…

Still - the question remains… is it possible to have multiple upstream
requests for a given downstream request?

  • Daniel

I believe it could be possible, but you will have to face with a very complex
problem: the HTTP scheme: request -> reply will be broken and Nginx is an
asynchronous server so it’s not so easy to manage errors when parallel
requests are evolved.

True…

Yet, as I understand the code…

A request comes in… nginx determines it is to be forwarded to an
upstream
server… the request is appended with request->upstream and the chain
of
responsibility leads to the upstream module connecting to the upstream
server. The request is held in the queue until the upstream module
releases
buffer chains and/or finalizes the request… the upstream connection
receives
the upstream response and stores them in it’s own buffer chain.

The process is asynchronous since as chunked responses from the upstream
arrive and when the open request is examined again, the upstream module
transfers from one to the other…

I’m thinking that I should look at using the upstream round robin module
as my
code base… Essentially I am letting multiple upstream connections
be
created and the round robin module already has the code for dealing with
failed connections.

Thanks for the CRC concept… The batch mode doesn’t work for this
problem,
but a SHA1 of the two responses makes comparision much faster.

Thanks for your input - it has helped me look at the problem with a
different
perspective.

Daniel

Daniel Chapiesky <[email protected]…> writes:

Still - the question remains… is it possible to have multiple upstream
requests for a given downstream request?

I believe it could be possible, but you will have to face with a very
complex
problem: the HTTP scheme: request -> reply will be broken and Nginx is
an
asynchronous server so it’s not so easy to manage errors when parallel
requests
are evolved.
But you will not have to hack Nginx with a module but rather writing an
application using Nginx as a web server component…

Why don’t you calculate a CRC on the old application server reply and
log it
and then check (off line in batch mode) the request and the CRC with the
new
application server, may be it would help to focus on problems and it’s
very easy
to do (an X-Header would do the job for example)?
So far, no more idea.

Best regards.

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs