Forum: NGINX varnish?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Ilan B. (Guest)
on 2009-03-30 03:10
(Received via mailing list)
I just came across varnish.  We serve a lot of MP3 files on a regular
basis.  Does it make sense to have Varnish cache those and serve them
out of
cache?  Does anyone have any experience with Varnish and Nginx installs?
Are they complimentary or exclusionary?

Thanks
Kiril A. (Guest)
on 2009-03-30 04:04
(Received via mailing list)
You can check also http://www.ncache.org
Cliff W. (Guest)
on 2009-03-30 07:35
(Received via mailing list)
On Sun, 2009-03-29 at 19:00 -0400, Ilan B. wrote:
> I just came across varnish.  We serve a lot of MP3 files on a regular
> basis.  Does it make sense to have Varnish cache those and serve them
> out of cache?  Does anyone have any experience with Varnish and Nginx
> installs?  Are they complimentary or exclusionary?

They are complimentary.  That being said, I don't think it makes much
sense to cache static files with an application.   Your OS kernel
already caches disk accesses and running another application will cut
into that cache memory.   Varnish makes more sense for dynamic content.

Regards,
Cliff
russlndr (Guest)
on 2009-04-14 00:28
(Received via mailing list)
My company is a heavy user of varnish cache since 2006. We have 6
varnish servers with 550 GB of RAM caching static pictures for our
community. At peak each server is handling 40k active connections per
second.

If you only have one server you should use a dedicated static server and
let the file system cache(linux caches last read data in ram), but it
won't scale beyond that server. But I assume you have many GBs of mp3s
and need more than one cache. Our setup have pools of varnish servers
delivering photos from RAM, and we distribute traffic between the
varnishes by hash method so one server only serve his part of the photos
and caches them in memory.

Varnish is also a very hardware friendly software. It makes smokeping
look nice. :)

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,615,1007#msg-1007
CryptWizard (Guest)
on 2009-04-14 03:09
(Received via mailing list)
How did you get 61 GB of RAM in one server?
Jeff E. (Guest)
on 2009-04-14 10:48
(Received via mailing list)
On Mon, Apr 13, 2009 at 1:55 AM, CryptWizard 
<removed_email_address@domain.invalid>
wrote:
> How did you get 61 GB of RAM in one server?
>

64 bit architecture supports a theoretical limit of 16 exabytes of RAM.

http://en.wikipedia.org/wiki/64-bit
russlndr (Guest)
on 2009-04-14 11:17
(Received via mailing list)
HP Blades with AMD CPU support up to 128 GB of RAM. We have 3 x AMD with
128 GB and 3 x Intel 64 GB. The new intel HP blades support 144 GB RAM.

At peak each varnish server is handling 40k established connections, not
active connections as I wrote by mistake. About 7000 to 8000 hits per
second.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,615,1011#msg-1011
Barry A. (Guest)
on 2009-04-14 20:08
(Received via mailing list)
On Apr 13, 2009, at 10:05 AM, russlndr wrote:

> HP Blades with AMD CPU support up to 128 GB of RAM. We have 3 x AMD
> with 128 GB and 3 x Intel 64 GB. The new intel HP blades support 144
> GB RAM.
>
> At peak each varnish server is handling 40k established connections,
> not active connections as I wrote by mistake. About 7000 to 8000
> hits per second.

Are you also using nginx in this setup?  We are using nginx + varnish
for http://gravatar.com (nginx --> varnish --> apache ) and are
running into issues with the nginx boxes getting CPU bound (softirq,
not user).  We are only able to get about 8000 req/sec (1k est
concurrent TCP connections) per nginx server before this happens.
Anyone else seeing this at high concurrency?  We have already done the
obvious stuff like disable connection tracking, and are using
irqbalance to push the interrupt handlers from eth1 and eth0 onto
separate cores.  We are running Debian Linux Sarge (2.6.18) 64-bit.
Any suggestions?
Victor I. (Guest)
on 2009-04-15 02:55
@Barry A.

Why don't you guys just use nginx to serve the files and let the file
system and OS hande caching of static data? Just curious?
Barry A. (Guest)
on 2009-04-15 06:24
(Received via mailing list)
On Apr 14, 2009, at 6:55 PM, Victor I. wrote:

> @Barry A.
>
> Why don't you guys just use nginx to serve the files and let the file
> system and OS hande caching of static data? Just curious?

Not sure exactly what you mean, but nothing is "static files" --
everything is generated dynamically.
Floren M. (Guest)
on 2009-04-15 07:51
(Received via mailing list)
Hi Kiril,

> -----Original Message-----
> From: Kiril A. [mailto:kupokomapa-
> removed_email_address@domain.invalid]
> Posted At: Sunday, March 29, 2009 7:57 PM
> Posted To: gmane.comp.web.nginx.english
> Conversation: varnish?
> Subject: Re: varnish?
>
> You can check also http://www.ncache.org

Isn't better to use the built-in cache, available into 0.7.5x branch?
Thanks for the link, though. I haven't got the time to look at Igor's
new
cache system.

Floren
Victor I. (Guest)
on 2009-04-15 21:43
@Barry A.

Sorry, I should of been more specific. I also follow the varnish-misc
list, and I have read that gravatar uses varnish specifically for their
thumbnails.  That is what I meant by static data. So the question still
stands, won't it make more sense to avoid the double buffer problem and
just use filesystem cache?
Barry A. (Guest)
on 2009-04-15 23:22
(Received via mailing list)
On Apr 15, 2009, at 1:43 PM, Victor I. wrote:

> @Barry A.
>
> Sorry, I should of been more specific. I also follow the varnish-misc
> list, and I have read that gravatar uses varnish specifically for
> their
> thumbnails.  That is what I meant by static data. So the question
> still
> stands, won't it make more sense to avoid the double buffer problem
> and
> just use filesystem cache?

Yes, except the vast majority images are generated dynamically, so
they don't exist on a filesystem anywhere :)
zepolen (Guest)
on 2009-04-16 05:09
(Received via mailing list)
On Wed, Apr 15, 2009 at 10:14 PM, Barry A.
<removed_email_address@domain.invalid> wrote:
>
> Yes, except the vast majority images are generated dynamically, so they
> don't exist on a filesystem anywhere :)
>

Can't proxy_store help you with that?

I've been using it with nginx to read thumbnails from an origin server
and save them to local disk, works pretty well.
Phillip B Oldham (Guest)
on 2009-04-16 11:32
(Received via mailing list)
Attachment: phill.vcf (0 Bytes)
Barry A. wrote:
> Yes, except the vast majority images are generated dynamically, so
> they don't exist on a filesystem anywhere :)
Wouldn't it be faster to have the image generation script push the file
to a memcached store on creation, and have nginx check this store before
falling back to the generation script?
--

*Phillip B Oldham*
The Activity People
removed_email_address@domain.invalid 
<mailto:removed_email_address@domain.invalid>

------------------------------------------------------------------------

*Policies*

This e-mail and its attachments are intended for the above named
recipient(s) only and may be confidential. If they have come to you in
error, please reply to this e-mail and highlight the error. No action
should be taken regarding content, nor must you copy or show them to
anyone.

This e-mail has been created in the knowledge that Internet e-mail is
not a 100% secure communications medium, and we have taken steps to
ensure that this e-mail and attachments are free from any virus. We must
advise that in keeping with good computing practice the recipient should
ensure they are completely virus free, and that you understand and
observe the lack of security when e-mailing us.
Barry A. (Guest)
on 2009-04-16 18:55
(Received via mailing list)
On Apr 15, 2009, at 8:58 PM, zepolen wrote:

> On Wed, Apr 15, 2009 at 10:14 PM, Barry A. <removed_email_address@domain.invalid
> > wrote:
>>
>> Yes, except the vast majority images are generated dynamically, so
>> they
>> don't exist on a filesystem anywhere :)
>>
>
> Can't proxy_store help you with that?

Maybe, but at 30,000 req/sec, reading/writing content from disk is too
slow.
Barry A. (Guest)
on 2009-04-16 19:01
(Received via mailing list)
On Apr 16, 2009, at 3:20 AM, Phillip B Oldham wrote:

> Barry A. wrote:
>>
>> Yes, except the vast majority images are generated dynamically, so
>> they don't exist on a filesystem anywhere :)
> Wouldn't it be faster to have the image generation script push the
> file to a memcached store on creation, and have nginx check this
> store before falling back to the generation script?

Maybe, but speed isn't the main issue here, the requests are served
generally "fast enough"  Its scaling a single box to serve a huge
number of requests/sec.

BTW, I enabled http keep alives in nginx, and it seems to have reduced
softirq by about 5% which is pretty good.  I guess setting up and
tearing down all of those TCP connections has a little bit of
overhead.  We have sacrificed a little bit of RAM to deal with the
huge number of established connections (50k ish), but thats ok.  I'm
wondering if there are even more optimizations like this which could
reduce sofirq even more.
zepolen (Guest)
on 2009-04-18 01:08
(Received via mailing list)
On Thu, Apr 16, 2009 at 5:48 PM, Barry A. <removed_email_address@domain.invalid>
wrote:
> Maybe, but at 30,000 req/sec, reading/writing content from disk is too slow.

Have you tried it? Typically only the first access will be slow, after
that it should be cached by the file system.
This topic is locked and can not be replied to.