Shell environment variables in "include"-directive not working

Markus_Grobelin · May 13, 2010, 12:49am

Hy everybody,
i’m doing my first steps with nginx/0.8.36 and trying to get *NIX shell
environment variables working inside the configuration files. Sadly,
it’s seems they aren’t working inside the “include”-directive!

My current (very basic) configuration looks like this:

user and group to run as

user $USER $USER;

pid of nginx master process

pid /nginx/$INSTANCE/run/nginx.pid;

number of nginx workers

worker_processes 2;

number of worker connection

events {
worker_connections 1024;
}

http {
# pull in mime-types
include /nginx/INSTANCE/nginx/conf/mime.types;

 # set a default type for the rare situation that
 # nothing matches from the mimie-type include
 #
 default_type  application/octet-stream;

 # logging in server-context
 #
 access_log /nginx/$INSTANCE/nginx/logs/access.log
 error_log /nginx/$INSTANCE/nginx/logs/error.log

}

When starting the server i got the following messages:
[alert]: could not open error log file: open()
“/var/log/nginx/error.log” failed (13: Permission denied)
2010/05/13 00:16:38 [warn] 27977#0: the “user” directive makes sense
only if the master process runs with super-user privileges, ignored in
/nginx/wally1-dach-static/nginx/conf/nginx.conf:2
2010/05/13 00:16:38 [emerg] 27977#0: open()
“/nginx/$INSTANCE/nginx/conf/mime.types” failed (2: No such file or
directory) in /nginx/wally1-dach-static/nginx/conf/nginx.conf:18

The [emerg] indicates, that the $INSTANCE environment variable isn’t
expanded, whereas the “user” and “pid” directive doesn’t raise an
exception??

Cheers,
Markus

Markus_Grobelin · May 13, 2010, 2:30am

Hello!

On Thu, May 13, 2010 at 12:49:21AM +0200, Markus Grobelin wrote:

Hy everybody,
i’m doing my first steps with nginx/0.8.36 and trying to get *NIX
shell environment variables working inside the configuration files.
Sadly, it’s seems they aren’t working inside the
“include”-directive!

nginx doesn’t have syntax for expanding environment variables in
configuration file. Syntax $var used for runtime per-request
variables (supported by some directives, support explicitly noted
in directive descriptions).

[…]

The [emerg] indicates, that the $INSTANCE environment variable isn’t
expanded, whereas the “user” and “pid” directive doesn’t raise an
exception??

Because there is no syntax error in user “$USER” and config parser
has nothing against it. As you aren’t running as root nginx
just prints warning about being non-root and forgets about it.
Under root you should see:

[emerg]: getpwnam("$USER") failed in /path/to/nginx.conf:line

(unless you actually have “$USER” in your /etc/passwd)

Similar thing with pid. It’s syntactically correct and will only
produce error when nginx will try to create pid file. If you
happen to cleanup other critical config errors you should see
something like this on startup:

[emerg]: open() “/nginx/$INSTANCE/run/nginx.pid” failed (2: No such file
or directory)

(again, unless you actually have “/nginx/$INSTANCE/run/” directory)

Maxim D.

Markus_Grobelin · May 13, 2010, 8:43am

On Thu, May 13, 2010 at 12:49:21AM +0200, Markus Grobelin wrote:

user $USER $USER;
}
# logging in server-context
only if the master process runs with super-user privileges, ignored in
/nginx/wally1-dach-static/nginx/conf/nginx.conf:2
2010/05/13 00:16:38 [emerg] 27977#0: open()
“/nginx/$INSTANCE/nginx/conf/mime.types” failed (2: No such file or
directory) in /nginx/wally1-dach-static/nginx/conf/nginx.conf:18

The [emerg] indicates, that the $INSTANCE environment variable isn’t
expanded, whereas the “user” and “pid” directive doesn’t raise an
exception??

As Maxim already said, nginx does not support environment variables.
However, your configuration can be set by running

nginx -p /nginx/$INSTANCE/ -g “user $USER $USER;”

and using relative paths in configuration:

  pid        run/nginx.pid;
  access_log nginx/logs/access.log;
  error_log  nginx/logs/error.log;

–
Igor S.
http://sysoev.ru/en/

Markus_Grobelin · May 14, 2010, 1:12pm

On 05/13/2010 02:29 AM, Maxim D. wrote:

nginx doesn’t have syntax for expanding environment variables in
configuration file.

Too bad… I thought it is a common thing for oos server software

Recently I built up a farm of several Tomcat & Apache httpd Instances.
It’s just the frontend part of the application. There is a business
layer and backend, too. I wanted to migrate some http functionality
stuff - delivering static media, cache some userphotos - to nginx. Think
of multiple instances across multiple hosts (30 and counting) for all
the stages (dev,qa, prelive,live) one want to have when deploy a big
web-application.

It’s at charming to unify system- and daemon-configuration via
placeholders (environment variables). Convention over Configuration
enables a system administrators live to be as stressless as possible

Is there a plan to add this feature?

Markus_Grobelin · May 14, 2010, 2:57pm

Hello!

On Fri, May 14, 2010 at 01:11:55PM +0200, Markus Grobelin wrote:

to nginx. Think of multiple instances across multiple hosts (30
and counting) for all the stages (dev,qa, prelive,live) one want to
have when deploy a big web-application.

It’s at charming to unify system- and daemon-configuration via
placeholders (environment variables). Convention over Configuration
enables a system administrators live to be as stressless as possible

Is there a plan to add this feature?

Not really. But there are some well-known tools[1] which will do it
for you easily enough.

[1] sed

Maxim D.

Markus_Grobelin · May 15, 2010, 3:12pm

Hmm. Maybe you’re right. The following one will do the trick, if your
environment variables are prefixed with “ENV_”, e.g. ENV_foo=bar, in the
.env file:

 . conf/instance.env
 env_vars="`env | grep 'ENV_'`"

 find nginx/conf/ -type f -iname '*.conf' | while read file
 do
     for var in $env_vars; do
         vkey="`echo $var | cut -f1 -d=`"
         vvalue="`echo $var | cut -f2 -d=`"
         sed -e "s/\$${vkey}/`echo $vvalue`/g" -i $file
      done
 done

… so i can have configfiles like:

server {
listen $ENV_httpdip:$ENV_port;

 server_name media$ENV_domainsuffix;

 # doc root
 root $ENV_instance/htdocs/superoperty;

 access_log $ENV_instance/logs/static/media.access.log;
 error_log $ENV_instance/logs/static/media.error.log;

}

Cheers,
Markus Grobelin

Markus_Grobelin · February 24, 2013, 12:02am

Just wanted to respond that I had the similar need, and hopefully a
better solution. We too have dev, staging, prod, and the configurations
are similar, but not identical.
We have hundreds of servers, and the variances in files means we need to
have a much longer runway for new sysadm recruits to get up to speed,
and increases the opportunity for errors caused by divergence -
especially in active/active or active/passive configurations in
production.

Next - the link to sed which is a HORRIBLE idea to my OCD brain. sed is
a line parser, it fundamentally doesn’t understand the hierarchy of an
nginx conf file(s). The opportunity for something to go horribly
horribly wrong and be very difficult to troubleshoot (especially with
high degrees of automation/scripting in other areas of our platform)
scares the crap out of me. Using sed to parse old style unix files where
everything is on a single line and all lines are identical = great,
using it to parse a nested configuration file where each line is
different is a horrid idea (even with excellent naming conventions).
I am actually not aware of any valid NGINX hierarchy parser, and since
it uses a very non standard, highly complex syntax I think it would be
reasonably challenging to write one which could read, modify, write back
the same file with comments, etc. (I bet a lot of people do)

AND – something as a simple HOSTNAME or #IFHOSTNAME would be
amazingly useful for a variety of situations.
I also wanted to share to others who may also have the same issues why
user defined environment variables are a horrible idea too and why the
devs are so resistant to them. When trying to hot-swap executables
since the environment won’t carry across pids – this is a cool feature
nginx has, and using environment variables would break that.
So stop asking for environment variables – instead we should be
thinking about a few well known variables that can be interpolated at
config parsing time would be useful for those of us with hundreds of
servers to manage the 1-2% variances between each servers without
needing an elaborate custom build script. one config file to bind them
all and KISS.

SO I may write a plugin to do that in the future, but alas, no time.
So I submit the slightly ghetto work around I devised.

First use host file trickery (this is IMHO pretty common for
dev/prod/staging) to alias well known role based names “ex: api, www,
etc”

Use symlinking with hostname to include the proper files based on
hostname. Modify your init.d script so it links, or touches empty files
when a corresponding file doesn’t exist.

I’m not going to provide examples here because inevitably your
environment will be different than mine, but I can tell you that nginx
will load a empty (zero byte) include file with no issues.
Of course environment variables like hostname in a shell script is
trivial.

So when you bake it all together - in the nginx.conf file:

include “some-role.conf”
include “lotsofcustom-roles/*.conf”
include “yetanother-role.conf”

** those includes point at symlinked or zero byte files.

In the /etc/init.d/nginx script do something like:

/bin/rm -f $NGINXROOT/conf/some-role.conf
/bin/rm -f $NGINXROOT/conf/lotsofcustom-roles
/bin/rm -f $NGINXROOT/conf/yetanother-role.conf
if [ -f “$NGINXROOT/conf/some-role-$HOSTNAME.conf” ] ; then
ln -s “$NGINXROOT/conf/some-role-$HOSTNAME.conf”
$NGINXROOT/conf/some-role.conf"
else
touch $NGINXROOT/conf/some-role.conf
fi
if [ -d “$NGINXROOT/conf/lotsofcustom-roles-$HOSTNAME” ] ; then
ln -sd “$NGINXROOT/conf/lotsofcustom-roles-$HOSTNAME”
$NGINXROOT/lotsofcustom-roles"
else
mkdir $NGINXROOT/conf/lotsofcustom-roles
touch $NGINXROOT/conf/lotsofcustom-roles/nothing-to-see-here.conf
fi
if [ -f “$NGINXROOT/conf/yetanother-role-$HOSTNAME.conf” ] ; then
ln -s “$NGINXROOT/conf/yetanother-role-$HOSTNAME.conf”
$NGINXROOT/conf/some-role.conf"
else
touch $NGINXROOT/conf/yetanother-role.conf
fi

Obviously this is highly dependent on exactly what you want to
accomplish, but I think it strikes a much nicer balance than use a sed
machete to hack through a config file using regular expressions it
doesn’t understand and accidentally clobbering something you didn’t
intend to.

Regards,

-Brian Horakh
Chief Technical Guy
anyCommerce

Maxim D. wrote in post #911875:

Hello!

On Thu, May 13, 2010 at 12:49:21AM +0200, Markus Grobelin wrote:

Hy everybody,
i’m doing my first steps with nginx/0.8.36 and trying to get *NIX
shell environment variables working inside the configuration files.
Sadly, it’s seems they aren’t working inside the
“include”-directive!

nginx doesn’t have syntax for expanding environment variables in
configuration file. Syntax $var used for runtime per-request
variables (supported by some directives, support explicitly noted
in directive descriptions).

[…]

The [emerg] indicates, that the $INSTANCE environment variable isn’t
expanded, whereas the “user” and “pid” directive doesn’t raise an
exception??

Because there is no syntax error in user “$USER” and config parser
has nothing against it. As you aren’t running as root nginx
just prints warning about being non-root and forgets about it.
Under root you should see:

[emerg]: getpwnam("$USER") failed in /path/to/nginx.conf:line

(unless you actually have “$USER” in your /etc/passwd)

Similar thing with pid. It’s syntactically correct and will only
produce error when nginx will try to create pid file. If you
happen to cleanup other critical config errors you should see
something like this on startup:

[emerg]: open() “/nginx/$INSTANCE/run/nginx.pid” failed (2: No such file
or directory)

(again, unless you actually have “/nginx/$INSTANCE/run/” directory)

Maxim D.