How to use multiple IP as proxied connection IP

ZhangSTony · August 29, 2009, 9:25am

Hi

Currently, my Nginx service IP is used as both VIP (service IP) and
proxied connection source IP ( I use Nginx as pure proxy server), since
I use close connection to user client, and close connection is used
between Nginx and backend squid, those two situation means that there
are many time_wait connections hung over there, and proxy connection
port of that VIP is not that enough, is there any way to use multiple IP
or dedicated IPs as proxied IP to increase the usable connection port.
Thx a lot!

Thx
Jie

Tony ZHANG

Tudou.com - Tech - OP - Site
Email: [email protected]
Better Performance, Better Experience
[cid:[email protected]]

ZhangSTony · August 29, 2009, 12:06pm

On Sat, 2009-08-29 at 15:17 +0800, Zhang,Tony wrote:

Currently, my Nginx service IP is used as both VIP (service IP) and
proxied connection source IP ( I use Nginx as pure proxy server),
since I use close connection to user client, and close connection is
used between Nginx and backend squid, those two situation means that
there are many time_wait connections hung over there, and proxy
connection port of that VIP is not that enough, is there any way to
use multiple IP or dedicated IPs as proxied IP to increase the usable
connection port. Thx a lot!

If you haven’t done that yet, you certainly want to shorten tcp
timeouts first.

ZhangSTony · August 31, 2009, 8:59am

Hi,

Have you enlarged the ephemeral port range of your operating system?
http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html

On Sat, Aug 29, 2009 at 7:51 PM, Zhang,Tony[email protected] wrote:

Tudou.com - Tech - OP - Site
On Sat, 2009-08-29 at 15:17 +0800, Zhang,Tony wrote:
timeouts first.
Version: 8.5.409 / Virus Database: 270.13.71/2330 - Release Date: 08/27/09 18:02:00

No virus found in this outgoing message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.71/2330 - Release Date: 08/27/09 18:02:00

This e-mail is confidential. It may also be legally privileged.If you are not the addressee you may not copy, forward, disclose or use any part of it. If you have received this message in error,please delete it and all copies from your system and notify the sender immediately by return e-mail.Internet communications cannot be guaranteed to be timely,secure, error or virus-free. The sender does not accept liability for any errors or omissions.

Regards,

ZhangSTony · August 31, 2009, 9:12am

Hi

Yes, I have done that yet

net.ipv4.ip_local_port_range = 1025 65000

almost largest port range

Thx
Jie

Tony ZHANG

Tudou.com - Tech - OP - Site
Email: [email protected]
Better Performance, Better Experience

-----ÓÊ¼þÔ¼þ-----
·¢¼þÈË: [email protected] [mailto:[email protected]] ´ú±í Joshua Z.
·¢ËÍÊ±¼ä: 2009Äê8ÔÂ31ÈÕ 14:48
ÊÕ¼þÈË: [email protected]
Ö÷Ìâ: Re: ´ð¸´: How to use multiple IP as proxied connection IP

Hi,

Have you enlarged the ephemeral port range of your operating system?
http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html

On Sat, Aug 29, 2009 at 7:51 PM, Zhang,Tony[email protected] wrote:

Tudou.com - Tech - OP - Site
On Sat, 2009-08-29 at 15:17 +0800, Zhang,Tony wrote:
timeouts first.
Version: 8.5.409 / Virus Database: 270.13.71/2330 - Release Date: 08/27/09 18:02:00

No virus found in this outgoing message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.71/2330 - Release Date: 08/27/09 18:02:00

This e-mail is confidential. It may also be legally privileged.If you are not the addressee you may not copy, forward, disclose or use any part of it. If you have received this message in error,please delete it and all copies from your system and notify the sender immediately by return e-mail.Internet communications cannot be guaranteed to be timely,secure, error or virus-free. The sender does not accept liability for any errors or omissions.

Regards,

–
Joshua Z.

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.71/2336 - Release Date:
08/30/09 17:51:00

No virus found in this outgoing message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.71/2336 - Release Date:
08/30/09 17:51:00

This e-mail is confidential. It may also be legally privileged.If you
are not the addressee you may not copy, forward, disclose or use any
part of it. If you have received this message in error,please delete it
and all copies from your system and notify the sender immediately by
return e-mail.Internet communications cannot be guaranteed to be
timely,secure, error or virus-free. The sender does not accept liability
for any errors or omissions.

ZhangSTony · August 29, 2009, 2:00pm

Hi

Kernel timeout has been shortened before, but still many time_wait

Thx
Jie

Tony ZHANG

Tudou.com - Tech - OP - Site
Email: [email protected]
Better Performance, Better Experience

-----é‚®ä»¶åŽŸä»¶-----
å‘ä»¶äºº: [email protected] [mailto:[email protected]] ä»£è¡¨ Miros?aw
Jaworski
å‘é€æ—¶é—´: 2009å¹´8æœˆ29æ—¥ 16:52
æ”¶ä»¶äºº: [email protected]
ä¸»é¢˜: Re: How to use multiple IP as proxied connection IP

On Sat, 2009-08-29 at 15:17 +0800, Zhang,Tony wrote:

Currently, my Nginx service IP is used as both VIP (service IP) and
proxied connection source IP ( I use Nginx as pure proxy server),
since I use close connection to user client, and close connection is
used between Nginx and backend squid, those two situation means that
there are many time_wait connections hung over there, and proxy
connection port of that VIP is not that enough, is there any way to
use multiple IP or dedicated IPs as proxied IP to increase the usable
connection port. Thx a lot!

If you haven’t done that yet, you certainly want to shorten tcp
timeouts first.

–
Miroslaw “Psyborg” Jaworski
GCS/IT d- s+:+ a C++$ UBI++++$ P+++$ L- E— W++(+++)$ N++ o+ K- w-- O-
M- V- PS+ PE++ Y+ PGP t 5? X+ R++ !tv b++(+++) DI++ D+ G e* h++ r+++ y?

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.71/2330 - Release Date:
08/27/09 18:02:00

No virus found in this outgoing message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.71/2330 - Release Date:
08/27/09 18:02:00

This e-mail is confidential. It may also be legally privileged.If you
are not the addressee you may not copy, forward, disclose or use any
part of it. If you have received this message in error,please delete it
and all copies from your system and notify the sender immediately by
return e-mail.Internet communications cannot be guaranteed to be
timely,secure, error or virus-free. The sender does not accept liability
for any errors or omissions.

ZhangSTony · August 31, 2009, 10:13am

Hi

net.ipv4.tcp_fin_timeout = 1

about 50000 time_wait totally (2 nginx lb servers), peak time, maybe
more.

That’s why I think ports are not enough

Thx
Jie

Tony ZHANG

Tudou.com - Tech - OP - Site
Email: [email protected]
Better Performance, Better Experience

-----ÓÊ¼þÔ¼þ-----
·¢¼þÈË: [email protected] [mailto:[email protected]] ´ú±í Joshua Z.
·¢ËÍÊ±¼ä: 2009Äê8ÔÂ31ÈÕ 15:39
ÊÕ¼þÈË: [email protected]
Ö÷Ìâ: Re: ´ð¸´: ´ð¸´: How to use multiple IP as proxied connection IP

Hi,

Then how many TIME_WAITs were there? And your TIME_WAIT value?

2009/8/31 Zhang,Tony [email protected]:

Jie
·¢ËÍÊ±¼ä: 2009Äê8ÔÂ31ÈÕ 14:48

Better Performance, Better Experience

proxied connection source IP ( I use Nginx as pure proxy server),
–
No virus found in this outgoing message.

Cheers!

–
Joshua Z.

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.71/2336 - Release Date:
08/30/09 17:51:00

No virus found in this outgoing message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.71/2336 - Release Date:
08/30/09 17:51:00

This e-mail is confidential. It may also be legally privileged.If you
are not the addressee you may not copy, forward, disclose or use any
part of it. If you have received this message in error,please delete it
and all copies from your system and notify the sender immediately by
return e-mail.Internet communications cannot be guaranteed to be
timely,secure, error or virus-free. The sender does not accept liability
for any errors or omissions.

ZhangSTony · August 31, 2009, 9:45am

Hi,

Then how many TIME_WAITs were there? And your TIME_WAIT value?

2009/8/31 Zhang,Tony [email protected]:

Jie
·¢ËÍÊ±¼ä: 2009Äê8ÔÂ31ÈÕ 14:48

Better Performance, Better Experience

proxied connection source IP ( I use Nginx as pure proxy server),
–
No virus found in this outgoing message.

Cheers!

ZhangSTony · August 31, 2009, 11:54am

On Mon, 2009-08-31 at 16:05 +0800, Zhang,Tony wrote:

Hi

net.ipv4.tcp_fin_timeout = 1

about 50000 time_wait totally (2 nginx lb servers), peak time, maybe more.

That’s why I think ports are not enough

Check with some state expire listing tool whether you really use
this 1 second timeout.

If you do - you must have pretty interesting numbers. Can you share some
information about your hardware and traffic ( bandwidth, requests per
minute/hour/day ), OS tuning and nginx config?