Add support for setting worker processes and CPU affinity automatically

Hi,

It’s a little bit inconvenient to set the CPU affinities with
many-core-system. For example, to fully utilize a 16-core Linux box, we
have to write something like this:
worker_processes 16;
worker_cpu_affinity 1000000000000000 0100000000000000 0010000000000000
0001000000000000 0000100000000000 0000010000000000 0000001000000000
0000000100000000 0000000010000000 0000000001000000 0000000000100000
0000000000010000 0000000000001000 0000000000000100 0000000000000010
0000000000000001;

This patch will make life much easier:
worker_processes auto;
worker_cpu_affinity auto;

Please see the attachment for more detail.

Regards,

Hi Joshua,

nice work indeed, thanks!

Do you have a practical example when such explicit binding
does better work than OS scheduler?


Maxim K.
+7 (910) 4293178
http://nginx.com/

Hi Maxim,

On Fri, Jan 13, 2012 at 12:07 AM, Maxim K. [email protected]
wrote:

http://nginx.com/

Thanks!
Actually we did a test against the CPU affinity feature before and found
it
was a little bit better than what the OS scheduler did, not too much
though. This is as expected since there will be less CPU cache missing
with
this feature turned on.

Regards,

On Thursday 12 January 2012 19:37:15 Joshua Z. wrote:

This patch will make life much easier:
worker_processes auto;
worker_cpu_affinity auto;

Please see the attachment for more detail.

You use “ngx_ncpu” global variable and for Linux you set it by
sysconf(_SC_NPROCESSORS_ONLN).

On freeBSD and MacOS systems it set by sysctls(hw.ncpu), and it is equal
to 1 on others.

Which means that your “auto” settings are very inaccurate. Sometimes the
“worker_processes” directive will be equal to 2 * cores or more,
sometimes
it will be equal to 1, and if you’re lucky, then it will be equal to
real
CPU cores.

So, I don’t think that this patch makes life easier. In fact, it creates
a
false sense of optimal worker_processes setting. Even if ngx_ncpu would
be
always equal to real hardware cores number, it wouldn’t be optimal for
every
use cases.

wbr, Valentin V. Bartenev

Setting CPU affinity is available on Linux systems only. Please
correct me if I’m wrong.

That’s wrong.

Just some start points:

FreeBSD: cpuset_setaffinity(2)
Solaris: probably better to start from processor_bind(2) man page.
HP/UP: pset_bind(2), psrset(1m)
AIX: bindprocessor()

Sure other systems with more or less advances *MP support have a
similar functionality.


Maxim K.
+7 (910) 4293178
http://nginx.com/

Hi,

Setting CPU affinity is available on Linux systems only. Please
correct me if I’m wrong.

That’s wrong.

I believe that Joshua meant that workers CPU affinity in nginx
works only on Linux right now (which is true).

Best regards,
Piotr S. < [email protected] >

Hi,

On Fri, Jan 13, 2012 at 1:53 AM, Valentin V. Bartenev [email protected]
wrote:

On Thursday 12 January 2012 19:37:15 Joshua Z. wrote:

[snip]

Thank you very much for the review.

You use “ngx_ncpu” global variable and for Linux you set it by

sysconf(_SC_NPROCESSORS_ONLN).

On freeBSD and MacOS systems it set by sysctls(hw.ncpu), and it is equal
to 1 on others.

Setting CPU affinity is available on Linux systems only. Please correct
me
if I’m wrong.

Which means that your “auto” settings are very inaccurate. Sometimes the
“worker_processes” directive will be equal to 2 * cores or more, sometimes
it will be equal to 1, and if you’re lucky, then it will be equal to real
CPU cores.

This is mostly caused by the hyper-threading feature of some CPUs. I
don’t
think this is an issue if hyper-threading is turned on. Sure we need to
find a better way to detect the number of CPUs on a system.

So, I don’t think that this patch makes life easier. In fact, it creates a
false sense of optimal worker_processes setting. Even if ngx_ncpu would be
always equal to real hardware cores number, it wouldn’t be optimal for
every
use cases.

It’s not ‘optimal’, but ‘automatic’. This patch still supports setting
worker processes and CPU affinities mannually as what the standard Nginx
does :wink:

Regards,

On Friday 13 January 2012 14:04:11 Joshua Z. wrote:

sysconf(_SC_NPROCESSORS_ONLN).

On freeBSD and MacOS systems it set by sysctls(hw.ncpu), and it is equal
to 1 on others.

Setting CPU affinity is available on Linux systems only. Please correct me
if I’m wrong.

You’re right.

But I generally said about “ngx_ncpu”, that also affects “auto” setting
in your
patch for the “worker_processes” directive.

Which means that your “auto” settings are very inaccurate. Sometimes the
“worker_processes” directive will be equal to 2 * cores or more,
sometimes it will be equal to 1, and if you’re lucky, then it will be
equal to real CPU cores.

This is mostly caused by the hyper-threading feature of some CPUs. I don’t
think this is an issue if hyper-threading is turned on. Sure we need to
find a better way to detect the number of CPUs on a system.

Yeap, we need to find a better way. And I don’t think the solution will
be simple
and it is worth to be implemented in a web server.

So, I don’t think that this patch makes life easier. In fact, it creates
a false sense of optimal worker_processes setting. Even if ngx_ncpu
would be always equal to real hardware cores number, it wouldn’t be
optimal for every
use cases.

It’s not ‘optimal’, but ‘automatic’. This patch still supports setting
worker processes and CPU affinities mannually as what the standard Nginx
does :wink:

So, when you need some automation for configuration and deployment, the
better
solution would be using proper software for that task, isn’t it? Like
Puppet,
for example.

wbr, Valentin V. Bartenev

On 1/13/12 2:27 PM, Piotr S. wrote:

Hi,

Setting CPU affinity is available on Linux systems only. Please
correct me if I’m wrong.

That’s wrong.

I believe that Joshua meant that workers CPU affinity in nginx
works only on Linux right now (which is true).

Ah, I see, thanks.

I just wonder how useful this option in real environment.


Maxim K.
+7 (910) 4293178
http://nginx.com/

Hi,

On Fri, Jan 13, 2012 at 6:33 PM, Valentin V. Bartenev [email protected]
wrote:

On Friday 13 January 2012 14:04:11 Joshua Z. wrote:

[snip]

So, when you need some automation for configuration and deployment, the
better
solution would be using proper software for that task, isn’t it? Like
Puppet,
for example.

OK.
If this patch would not be accepted, could you please also review the
first
one?
Thank you.

Regards,

Hello!

On Fri, Jan 13, 2012 at 09:08:25PM +0800, Joshua Z. wrote:

So, when you need some automation for configuration and deployment, the
better
solution would be using proper software for that task, isn’t it? Like
Puppet,
for example.

OK.
If this patch would not be accepted,

I wouldn’t say it won’t be accepted at all, I think we need
something like this, but probably not by default.

could you please also review the first one?

I’m currently looking into it.

Maxim D.

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs