Nginx slow at streaming flv or strange behavior

Server NGINX
1

Hi all,

I have installed nginx 0.8.35 (flv streaming and h264 module enabled) on
my production video streaming server to see if it can perform better
than lighttpd.

However, lighttpd outperforms nginx

 + speed
 + throughput

Here is my settings

  • CentOS release 5.3 (Final) with Lustre 1.8 patch

  • Kernel 2.6.18-92.1.17.el5_lustre.1.8.0smp x64

  • lighttpd-1.5.0

    server.network-backend = “gthread-aio”
    server.max-read-threads = 120
    server.stat-cache-engine = “fam”
    server.event-handler = “linux-sysepoll”
    server.max-worker = 25

  • nginx 0.8.35

    worker_processes 50; # I have tested with 4, 8, 20 processes
    worker_connections 2048;
    sendfile on;
    tcp_nopush on;
    tcp_nodelay off;

  • iptraf is used to monitor network activity in real time

All the video files is LustreFS based, network mounted. The real data is
stored on 6 storage servers (Lustre OSS)

I have configured 2 1-GB network cards: one for outgoing data (streaming
via port 80) and the another for Lustre client, reading data from
storage servers

nginx

When I turned nginx on, I saw strange things on network activity ( I
used iptraf to see network activity per network cards)

  • Incoming data: data read from storage servers: 250 - 290 MBit/s
    (better than lighttpd - see below)
  • Outgoing data: data sent to clients (flash players, VLC …): 90 -
    140 Mbit/s (much worse than lighttpd - see below)

It means that nginx can read more data from disk than the throughput it
can send to clients.

At 1300 - 1400 concurrent connections, it is expected to wait 42 - 51 s
before my VLC player can create a new streaming connection.

lighttpd

After that I turned off nginx and switched on lighttpd. The network
activity seemed to be much more logical.

  • Incoming data: data read from storage servers: 180 - 220 MBit/s
  • Outgoing data: data sent to clients (flash players, VLC …): 190 -
    230 Mbit/s

It means that lighttpd can send everything it read from the disk to
clients.

At 1300 - 1400 concurrent connections, it is expected to wait 16 - 23 s
before my VLC player can create a new streaming connection. Much faster
than in nginx.

I can not use sendfile on lighttpd because it is extremely slow.

Conclusion

25 lighttpd worker processes outperforms 50 nginx processes in term of

  • response time
  • throughput

==========

What I really want to replace lighttpd with nginx is that lighttpd

  • crashed many time a day
  • caused very high load average.
  • unable to killall because it can become zombie after that. Every
    lighttpd processes switches to S status while streaming. nginx switches
    to D status.

But I still don’t know why nginx is slower than lighttpd on sending data
to clients while it can read more data from disk than the counterpart.

Any thought?

Regards,

Hoang

Posted at Nginx Forum:

2

Hello!

On Thu, Apr 22, 2010 at 04:54:05AM -0400, vnjug wrote:

  • nginx 0.8.35
    All the video files is LustreFS based, network mounted. The real
    data is stored on 6 storage servers (Lustre OSS)

I have configured 2 1-GB network cards: one for outgoing data
(streaming via port 80) and the another for Lustre client,
reading data from storage servers

You may want to use proxy instead of network filesystem (as nginx
normally blocks on file IO requests, which likely to block the
whole worker process for a relatively long time in case of network
filesystem).

Alternatively you may try to use aio in nginx, but

  1. This doesn’t help with mp4 streaming much, see below.

  2. It has some known not-yet-fixed bugs which may cause socket
    leak.

  3. It requires O_DIRECT under Linux and this may not be best
    choise.

…): 90 - 140 Mbit/s (much worse than lighttpd - see below)

It means that nginx can read more data from disk than the
throughput it can send to clients.

Mp4 streaming module is known to have non-optimal working model:
it get time position from client and then scans mp4 file for
coresponding file position. This may cause very strange effects.
Could you please confirm that you are seeing the same picture with
flv streaming only?

[…]

But I still don’t know why nginx is slower than lighttpd on
sending data to clients while it can read more data from disk
than the counterpart.

Any thought?

You may try to play with:

  1. Using sendfile_max_chunk (if not yet).

  2. Switching sendfile off and tuning output_buffers instead (i.e.
    use small number of big buffers, something like “output_buffers 1
    512k;”). This is known to perform better when serving large files
    as it reduces number of requests to disks. Alternative aproach
    would be to tune your os sendfile() to do reads of appropriate
    sizes, but I’m not sure it’s possible under Linux.

  3. Using aio, see above. Though I suppose it don’t help with mp4
    streaming as it uses blocking read()'s by itself.

  4. Using directio and read_ahead directives.

  5. Using proxy instead of network filesystem (and tuning proxy
    module).

Maxim D.