Forum: NGINX shell environment variables in "include"-directive not working

Hy everybody,
i'm doing my first steps with nginx/0.8.36 and trying to get *NIX shell
environment variables working inside the configuration files. Sadly,
it's seems they aren't working inside the "include"-directive! :(



My current (very basic) configuration looks like this:

# user and group to run as
user $USER $USER;

# pid of nginx master process
pid /nginx/$INSTANCE/run/nginx.pid;

# number of nginx workers
worker_processes 2;

# number of worker connection
events {
     worker_connections 1024;
}

http {
     # pull in mime-types
     include /nginx/INSTANCE/nginx/conf/mime.types;

     # set a default type for the rare situation that
     # nothing matches from the mimie-type include
     #
     default_type  application/octet-stream;

     # logging in server-context
     #
     access_log /nginx/$INSTANCE/nginx/logs/access.log
     error_log /nginx/$INSTANCE/nginx/logs/error.log
}


When starting the server i got the following messages:
[alert]: could not open error log file: open()
"/var/log/nginx/error.log" failed (13: Permission denied)
2010/05/13 00:16:38 [warn] 27977#0: the "user" directive makes sense
only if the master process runs with super-user privileges, ignored in
/nginx/wally1-dach-static/nginx/conf/nginx.conf:2
2010/05/13 00:16:38 [emerg] 27977#0: open()
"/nginx/$INSTANCE/nginx/conf/mime.types" failed (2: No such file or
directory) in /nginx/wally1-dach-static/nginx/conf/nginx.conf:18



The [emerg] indicates, that the $INSTANCE environment variable isn't
expanded, whereas the "user" and "pid" directive doesn't raise an
exception??

Cheers,
Markus

Hello!

On Thu, May 13, 2010 at 12:49:21AM +0200, Markus Grobelin wrote:

> Hy everybody,
> i'm doing my first steps with nginx/0.8.36 and trying to get *NIX
> shell environment variables working inside the configuration files.
> Sadly, it's seems they aren't working inside the
> "include"-directive! :(

nginx doesn't have syntax for expanding environment variables in
configuration file.  Syntax $var used for runtime per-request
variables (supported by some directives, support explicitly noted
in directive descriptions).

[...]

> The [emerg] indicates, that the $INSTANCE environment variable isn't
> expanded, whereas the "user" and "pid" directive doesn't raise an
> exception??

Because there is no syntax error in user "$USER" and config parser
has nothing against it.  As you aren't running as root nginx
just prints warning about being non-root and forgets about it.
Under root you should see:

[emerg]: getpwnam("$USER") failed in /path/to/nginx.conf:line

(unless you actually have "$USER" in your /etc/passwd)

Similar thing with pid.  It's syntactically correct and will only
produce error when nginx will try to create pid file.  If you
happen to cleanup other critical config errors you should see
something like this on startup:

[emerg]: open() "/nginx/$INSTANCE/run/nginx.pid" failed (2: No such file 
or directory)

(again, unless you actually have "/nginx/$INSTANCE/run/" directory)

Maxim Dounin

On Thu, May 13, 2010 at 12:49:21AM +0200, Markus Grobelin wrote:

> user $USER $USER;
> }
>      # logging in server-context
> only if the master process runs with super-user privileges, ignored in 
> /nginx/wally1-dach-static/nginx/conf/nginx.conf:2
> 2010/05/13 00:16:38 [emerg] 27977#0: open() 
> "/nginx/$INSTANCE/nginx/conf/mime.types" failed (2: No such file or 
> directory) in /nginx/wally1-dach-static/nginx/conf/nginx.conf:18
> 
> 
> 
> The [emerg] indicates, that the $INSTANCE environment variable isn't 
> expanded, whereas the "user" and "pid" directive doesn't raise an 
> exception??

As Maxim already said, nginx does not support environment variables.
However, your configuration can be set by running

   nginx -p /nginx/$INSTANCE/ -g "user $USER $USER;"

and using relative paths in configuration:

      pid        run/nginx.pid;
      access_log nginx/logs/access.log;
      error_log  nginx/logs/error.log;


--
Igor Sysoev
http://sysoev.ru/en/

On 05/13/2010 02:29 AM, Maxim Dounin wrote:

> nginx doesn't have syntax for expanding environment variables in
> configuration file.
>    
Too bad... I thought it is a common thing for oos server software :(

Recently I built up a farm of several Tomcat & Apache httpd Instances.
It's just the frontend part of the application. There is a business
layer and backend, too. I wanted to migrate some http functionality
stuff - delivering static media, cache some userphotos - to nginx. Think
of multiple instances across multiple hosts (30 and counting)  for all
the stages (dev,qa, prelive,live) one want to have when deploy a big
web-application.

It's at charming to unify system- and daemon-configuration via
placeholders (environment variables). Convention over Configuration
enables a system administrators live to be as stressless as possible ;)

Is there a plan to add this feature?

Hello!

On Fri, May 14, 2010 at 01:11:55PM +0200, Markus Grobelin wrote:

> - to nginx. Think of multiple instances across multiple hosts (30
> and counting)  for all the stages (dev,qa, prelive,live) one want to
> have when deploy a big web-application.
> 
> It's at charming to unify system- and daemon-configuration via
> placeholders (environment variables). Convention over Configuration
> enables a system administrators live to be as stressless as possible
> ;)
> 
> Is there a plan to add this feature?

Not really.  But there are some well-known tools[1] which will do it
for you easily enough.

[1] http://www.opengroup.org/onlinepubs/9699919799/uti...

Maxim Dounin

Hmm. Maybe you're right. The following one will do the trick, if your
environment variables are prefixed with "ENV_", e.g. ENV_foo=bar, in the
.env file:

     . conf/instance.env
     env_vars="`env | grep 'ENV_'`"

     find nginx/conf/ -type f -iname '*.conf' | while read file
     do
         for var in $env_vars; do
             vkey="`echo $var | cut -f1 -d=`"
             vvalue="`echo $var | cut -f2 -d=`"
             sed -e "s/\$${vkey}/`echo $vvalue`/g" -i $file
          done
     done

... so i can have configfiles like:

server {
     listen $ENV_httpdip:$ENV_port;

     server_name media$ENV_domainsuffix;

     # doc root
     root $ENV_instance/htdocs/superoperty;

     access_log $ENV_instance/logs/static/media.access.log;
     error_log $ENV_instance/logs/static/media.error.log;
}

Cheers,
Markus Grobelin

Just wanted to respond that I had the similar need, and hopefully a 
better solution. We too have dev, staging, prod, and the configurations 
are similar, but not identical.
We have hundreds of servers, and the variances in files means we need to 
have a much longer runway for new sysadm recruits to get up to speed, 
and increases the opportunity for errors caused by divergence - 
especially in active/active or active/passive configurations in 
production.

Next - the link to sed which is a HORRIBLE idea to my OCD brain.  sed is 
a line parser, it fundamentally doesn't understand the hierarchy of an 
nginx conf file(s). The opportunity for something to go horribly 
horribly wrong and be very difficult to troubleshoot (especially with 
high degrees of automation/scripting in other areas of our platform) 
scares the crap out of me. Using sed to parse old style unix files where 
everything is on a single line and all lines are identical = great, 
using it to parse a nested configuration file where each line is 
different is a horrid idea (even with excellent naming conventions).
I am actually not aware of any valid NGINX hierarchy parser, and since 
it uses a very non standard, highly complex syntax I think it would be 
reasonably challenging to write one which could read, modify, write back 
the same file with comments, etc.  (I bet a lot of people do)

AND -- something as a simple __HOSTNAME__ or #IFHOSTNAME would be 
*amazingly* useful for a variety of situations.
I also wanted to share to others who may also have the same issues why 
user defined environment variables are a horrible idea too and why the 
devs are so resistant to them.   When trying to hot-swap executables 
since the environment won't carry across pids -- this is a cool feature 
nginx has, and using environment variables would break that.
So stop asking for environment variables -- instead we should be 
thinking about a few well known variables that can be interpolated at 
config parsing time would be useful for those of us with hundreds of 
servers to manage the 1-2% variances between each servers without 
needing an elaborate custom build script. one config file to bind them 
all and KISS.

**SO** I may write a plugin to do that in the future, but alas, no time. 
So I submit the slightly ghetto work around I devised.

First use host file trickery (this is IMHO pretty common for 
dev/prod/staging) to alias well known role based names "ex: api, www, 
etc"

Use symlinking with hostname to include the proper files based on 
hostname. Modify your init.d script so it links, or touches empty files 
when a corresponding file doesn't exist.

I'm not going to provide examples here because inevitably your 
environment will be different than mine, but I can tell you that nginx 
will load a empty (zero byte) include file with no issues.
Of course environment variables like hostname in a shell script is 
trivial.

So when you bake it all together - in the nginx.conf file:

include "some-role.conf"
include "lotsofcustom-roles/*.conf"
include "yetanother-role.conf"

** those includes point at symlinked or zero byte files.

In the /etc/init.d/nginx script do something like:

/bin/rm -f $NGINXROOT/conf/some-role.conf
/bin/rm -f $NGINXROOT/conf/lotsofcustom-roles
/bin/rm -f $NGINXROOT/conf/yetanother-role.conf
if [ -f "$NGINXROOT/conf/some-role-$HOSTNAME.conf" ] ; then
   ln -s "$NGINXROOT/conf/some-role-$HOSTNAME.conf" 
$NGINXROOT/conf/some-role.conf"
else
   touch $NGINXROOT/conf/some-role.conf
fi
if [ -d "$NGINXROOT/conf/lotsofcustom-roles-$HOSTNAME" ] ; then
   ln -sd "$NGINXROOT/conf/lotsofcustom-roles-$HOSTNAME" 
$NGINXROOT/lotsofcustom-roles"
else
   mkdir $NGINXROOT/conf/lotsofcustom-roles
   touch $NGINXROOT/conf/lotsofcustom-roles/nothing-to-see-here.conf
fi
if [ -f "$NGINXROOT/conf/yetanother-role-$HOSTNAME.conf" ] ; then
   ln -s "$NGINXROOT/conf/yetanother-role-$HOSTNAME.conf" 
$NGINXROOT/conf/some-role.conf"
else
   touch $NGINXROOT/conf/yetanother-role.conf
fi

Obviously this is highly dependent on exactly what you want to 
accomplish, but I think it strikes a much nicer balance than use a sed 
machete to hack through a config file using regular expressions it 
doesn't understand and accidentally clobbering something you didn't 
intend to.

Regards,

-Brian Horakh
Chief Technical Guy
anyCommerce





Maxim Dounin wrote in post #911875:
> Hello!
>
> On Thu, May 13, 2010 at 12:49:21AM +0200, Markus Grobelin wrote:
>
>> Hy everybody,
>> i'm doing my first steps with nginx/0.8.36 and trying to get *NIX
>> shell environment variables working inside the configuration files.
>> Sadly, it's seems they aren't working inside the
>> "include"-directive! :(
>
> nginx doesn't have syntax for expanding environment variables in
> configuration file.  Syntax $var used for runtime per-request
> variables (supported by some directives, support explicitly noted
> in directive descriptions).
>
> [...]
>
>> The [emerg] indicates, that the $INSTANCE environment variable isn't
>> expanded, whereas the "user" and "pid" directive doesn't raise an
>> exception??
>
> Because there is no syntax error in user "$USER" and config parser
> has nothing against it.  As you aren't running as root nginx
> just prints warning about being non-root and forgets about it.
> Under root you should see:
>
> [emerg]: getpwnam("$USER") failed in /path/to/nginx.conf:line
>
> (unless you actually have "$USER" in your /etc/passwd)
>
> Similar thing with pid.  It's syntactically correct and will only
> produce error when nginx will try to create pid file.  If you
> happen to cleanup other critical config errors you should see
> something like this on startup:
>
> [emerg]: open() "/nginx/$INSTANCE/run/nginx.pid" failed (2: No such file
> or directory)
>
> (again, unless you actually have "/nginx/$INSTANCE/run/" directory)
>
> Maxim Dounin