Forum: NGINX Nginx postgres problem

2974d09ac2541e892966b762aad84943?d=identicon&s=25 arunh (Guest)
on 2014-03-04 16:15
(Received via mailing list)
Hello All,

I am newbie to Nginx and I am using Nginx with PostgreSQL database.

When there is a http request to the server I want to extract one of the
input variables in the request,
query the Database and retrun the result, the variable inside the
postgres_query is empty.

But when I try to print the value of "$1" variable and also after
assigning
it to the variable "$tenantID" I do get the value.

I guess in the eval block the postgres_escape is not getting set.

Can some one throw some light on this issue if I am doing something
wrong.

Also is there any supporting module to use the echo statements inside
the
postgres block.

Thanks,
Arun

Request :
http://localhost/tenantservices/tenantID/service/u...

location ~ ^/tenantservices/(.*)\/(.*)\/(.*)\/(.*)
{
echo teh complete request uri is :$request_uri;
echo The first parameter is :$1;
echo the second aprameter is :$2;

echo the third parameter is :$3;
echo the fourth param is :$4;

    set $tenantID $1;
    set $serviceName $2;
    set $userID $3;
    set $endPointName $4;
echo After setting the tenantID:$tenantID;

    eval_subrequest_in_memory  off;
    eval $max_no_clusters
   {
        default_type 'text/plain';
        postgres_pass   tenantdatabase;
        postgres_escape $tenantURID $tenantID;
        postgres_query   "SELECT max_no_clusters FROM load_balancer
WHERE
id=$tenantURID";
        postgres_output  value  ;
   }

  echo MaxNoOfclusters is : $max_no_clusters;
}

Log message on Nginx:
2014/03/04 15:46:05 [error] 31995#0: *152 postgres: "postgres_output
value"
received 0 value(s) instead of expected single value in location
"/eval_155080700" while processing result from PostgreSQL database,

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,248077,248077#msg-248077
37f3ea777f96500b332a1a89d6027897?d=identicon&s=25 Yichun Zhang (agentzh) (Guest)
on 2014-03-05 00:19
(Received via mailing list)
Hello!

On Tue, Mar 4, 2014 at 7:15 AM, arunh wrote:
> But when I try to print the value of "$1" variable and also after assigning
> it to the variable "$tenantID" I do get the value.
> I guess in the eval block the postgres_escape is not getting set.
>

The "eval" directive always runs before the directives of the standard
ngx_rewrite module. No matter in what order you write them down in
nginx.conf.

As a co-maintainer of ngx_postgres, I recommend you use ngx_lua module
to communicate with ngx_postgres.

Basically you can use rewrite_by_lua to inject some Lua code and use
ngx.location.capture to issue a subrequest to your postgres location.

Regards,
-agentzh
2974d09ac2541e892966b762aad84943?d=identicon&s=25 arunh (Guest)
on 2014-03-05 17:39
(Received via mailing list)
Hello Yichun,

Thank you very much for the reply.

I have used ngx.location.capture for extracting the values from postgres
database as suggested by you.
I had to make a series of postgres calls so I used content_by_lua along
by
sharing the varialbes as I had to query tables from multiple databases.

I have now got the needed nodeIP and port ultimately. But I am not able
to
redirect the url.
I have used rewrite_by_lua in the end to redirect to the new url, but it
does not seem to be happening.

If I use http rewrite module again the values of the nodeIP and port
would
be empty.

Could you please suggest how to redirect to the new url with the
extracted
nodeIP and port?

Thanks in advance,
Arun

CODE:


location /postgresrewrite
{

       rewrite ^ "http://$nodeIP:$port/$inputURI";
}


location ~ ^/tenantservices/(.*)\/(.*)\/(.*)\/(.*)
{
        set $inputURI $request_uri;
        set $tenantURI $1;
        set $serviceName $2;
        set $userID $3;
        set $endPointName $4;

        set $id '';
        set $tenantID '';
        set $instanceUUID '';
        set $nodeIP '';d
        set $port '';

        set $redirectURL '';

         content_by_lua '
            res = ngx.location.capture(
                    "/postgrestenantkey", { share_all_vars = true } );
           ngx.var.id = res.body;
           ngx.print(res.body);
           ngx.print(ngx.var.id);

           res1=ngx.location.capture(
                 "/postgrestenantid", { share_all_vars = true } );
           ngx.var.tenantID = res1.body;
           ngx.print(res1.body);
           ngx.print(ngx.var.tenantID);


           res2=ngx.location.capture(
                 "/postgresinstanceid", { share_all_vars = true } );
           ngx.var.instanceUUID = res2.body;
           ngx.print(res2.body);
           ngx.print(ngx.var.instanceUUID);

           res3=ngx.location.capture(
                 "/postgresnodeip", { share_all_vars = true } );
           ngx.var.nodeIP = res3.body;
           ngx.print(res3.body);
           ngx.print(ngx.var.nodeIP);

           res4=ngx.location.capture(
                 "/postgresport", { share_all_vars = true } );
           ngx.var.port = res4.body;
           ngx.print(res4.body);
           ngx.print(ngx.var.port);
           ';

           rewrite_by_lua '
           res5=ngx.location.capture(
                 "/postgresrewrite", { share_all_vars = true } );
           ';

}

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,248077,248147#msg-248147
37f3ea777f96500b332a1a89d6027897?d=identicon&s=25 Yichun Zhang (agentzh) (Guest)
on 2014-03-05 20:58
(Received via mailing list)
Hello!

On Wed, Mar 5, 2014 at 8:39 AM, arunh wrote:
> I had to make a series of postgres calls so I used content_by_lua along by
> sharing the varialbes as I had to query tables from multiple databases.
>

No, you should use rewrite_by_lua exclusively for all the
ngx.location.capture calls here I think.

> I have now got the needed nodeIP and port ultimately. But I am not able to
> redirect the url.
> I have used rewrite_by_lua in the end to redirect to the new url, but it
> does not seem to be happening.

rewrite_by_lua always runs before content_by_lua. Because NGINX's
"rewrite" phase always runs before the "content" phase.

Just put all your subrequest calls in the same code chunk as your
redirect Lua code in rewrite_by_lua.

What kind of redirect do you really need? 301? 302? or internal
redirect? See

    https://github.com/chaoslawful/lua-nginx-module#ngxredirect

    https://github.com/chaoslawful/lua-nginx-module#ngxexec

Speaking of running order, I suggest you read my NGINX tutorials to
prevent such confusions:
http://openresty.org/download/agentzh-nginx-tutori...

BTW, you're recommended to join the openresty-en mailing list and
discuss such things there:
https://groups.google.com/group/openresty-en You might get faster and
more responses on that list.

Best regards,
-agentzh
37f3ea777f96500b332a1a89d6027897?d=identicon&s=25 Yichun Zhang (agentzh) (Guest)
on 2014-03-05 21:04
(Received via mailing list)
Hello!

On Wed, Mar 5, 2014 at 8:39 AM, arunh wrote:
> location /postgresrewrite
> {
>        rewrite ^ "http://$nodeIP:$port/$inputURI";
> }
>
[...]
>            rewrite_by_lua '
>            res5=ngx.location.capture(
>                  "/postgresrewrite", { share_all_vars = true } );
>            ';
>

Forgot to mention that redirects in a subrequest won't affect its
parent requests. They're separate requests anyway. Try thinking about
it :)

Use either ngx.req.set_uri() or ngx.exec() in rewrite_by_lua. (If you
use content_by_lua, then you can only use ngx.exec for internal
redirects).

See

    https://github.com/chaoslawful/lua-nginx-module#ng...

    https://github.com/chaoslawful/lua-nginx-module#ngxexec

Also, it is a bad idea to enable "share_all_vars" due to potential bad
side-effects that lead to weird issues that are extremely hard to
debug. Just share the NGINX variables you want to share.

Best regards,
-agentzh
2974d09ac2541e892966b762aad84943?d=identicon&s=25 arunh (Guest)
on 2014-03-05 23:22
(Received via mailing list)
Hello Yichun,

The use case I am trying to test is the client sends a request to nginx
server with some parameters say "/a/b/c/d":
http://NginxHost/a/b/c/d

Depending on the parameters a,b,c and d I will get the IP and port of
the
destination server (by communicating with postgres) where the request
must
be redirected to ie the new url is of the form:
http://IP:port/a/b/c/d.

Using both ngx.redirect and nginx.exec() are giving errors.
I tried to redirect the url to "www.google.com" using ngx.redirect
inside
rewrite_by_lua. Even I get the same error.

You had mentioned that " redirects in a subrequest won't affect its
parent
requests." Does that mean that I cannot change the url inside the
rewrite_by_lua module?

I cannot use http rewrite module as this will be executed before the
rewrite_by_lua code is executed.

Please suggest what can be done to redirect the parent url to new url.

Thank you,
Arun


ERROR LOG:

CODE:
location /postgresrewrite
{
        rewrite ^ "http://www.google.co.in";
}
ngx.exec("/postgresrewrite");
LOG:
2014/03/05 22:59:42 [error] 8129#0: *374 lua entry thread aborted:
runtime
error: [string "rewrite_by_lua"]:32: attempt to call ngx.exec after
sending
out response headers
stack traceback:
coroutine 0:
        [C]: in function 'exec'
        [string "rewrite_by_lua"]:32: in function <[string
"rewrite_by_lua"]:1> while sending to client, client: xx.xx.xx.xx,
server:
localhost, request: "GET
/tenantservices/ArunTenant.com/Service1/13717e3b-c32d-4172-a316-74857b1237e1/httpSoapProvider
HTTP/1.1", host: "yy.yy.yy.yy"

CODE:
return ngx.redirect("http://www.google.co.in");
LOG:
2014/03/05 22:23:05 [error] 5397#0: *366 lua entry thread aborted:
runtime
error: [string "rewrite_by_lua"]:32: attempt to call ngx.redirect after
sending out the headers
stack traceback:
coroutine 0:
        [C]: in function 'redirect'
        [string "rewrite_by_lua"]:32: in function <[string
"rewrite_by_lua"]:1> while sending to client, client: xx.xx.xx.xx,
server:
localhost, request: "GET
/tenantservices/ArunTenant.com/Service1/13717e3b-c32d-4172-a316-74857b1237e1/httpSoapProvider
HTTP/1.1", host: "yy.yy.yy.yy"

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,248077,248161#msg-248161
37f3ea777f96500b332a1a89d6027897?d=identicon&s=25 Yichun Zhang (agentzh) (Guest)
on 2014-03-05 23:33
(Received via mailing list)
Hello!

On Wed, Mar 5, 2014 at 2:21 PM, arunh wrote:
> Depending on the parameters a,b,c and d I will get the IP and port of the
> destination server (by communicating with postgres) where the request must
> be redirected to ie the new url is of the form:
> http://IP:port/a/b/c/d.
>

You need either a 301/302 redirect or a proxy_pass to go to external
target.

Below is an example:

    rewrite_by_lua '
        local new_target = "http:/IP:port/a/b/c/d"
        return ngx.redirect(new_target)
    ';

> Using both ngx.redirect and nginx.exec() are giving errors.
> I tried to redirect the url to "www.google.com" using ngx.redirect inside
> rewrite_by_lua. Even I get the same error.
>

Please provide a minimal but still complete example that can reproduce
the issue.

> You had mentioned that " redirects in a subrequest won't affect its parent
> requests." Does that mean that I cannot change the url inside the
> rewrite_by_lua module?
>

You created a subrequest with ngx.location.capture and you used the
"rewrite" directive in the location targeted by the subrequest in the
hope that it will change the parent request calling
ngx.location.capture.

What you need is to initiate redirects in your *main* request. So you
should not use ngx.location.capture for a redirect. I never say you
cannot perform redirect in rewrite_by_lua. Do not get me wrong.

> I cannot use http rewrite module as this will be executed before the
> rewrite_by_lua code is executed.
>

No, as I've said, you should use the Lua API in rewrite_by_lua to
perform redirects.

> 2014/03/05 22:59:42 [error] 8129#0: *374 lua entry thread aborted: runtime
> error: [string "rewrite_by_lua"]:32: attempt to call ngx.exec after sending
> out response headers

The error clearly indicates the problem. You should not send the
response header before doing redirects (note that, ngx.say, ngx.print,
ngx.flush all trigger sending out the response header automatically,
so avoid them).

Regards,
-agentzh
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.