Thanks to all, The problem seems to be fixed by changing the NAT ip to something else and than back. It does seem much like NAT exhaustion even though the f/w claims only 13K session for two dynamic NATs and about 20 static ones. What I don't get is why there is consistency in opening sites. Why does facebook open all the time and store.apple.com barely opens all the time. I'd say if it would be NAT exhaustion, they would all behave the same way meaning open and then not open and then open again. It is solved for the time being. Again, thanks to all. ----- Andrey Gordon [andrey.gordon@gmail.com] On Tue, Feb 9, 2010 at 5:34 PM, Andrey Gordon <andrey.gordon@gmail.com>wrote:
I don't know, that's true. I don't where to find that info in this particular firewall would be a more correct statement. and my f/w guy is not much help either. It definitely looks to me like a NATting issue, but what I don't understand is why the same sites (e.g. facebook) loads fine consistently and others don't. NAT exhaustion would not allow that, imo.
This is the only relevant info I was able to find in the box:
andrey.gordon@PA-2050-Bos> show session info
------------------------------------------------------------------------------- number of sessions supported: 262143 number of active sessions: 6799 number of active TCP sessions: 5906 number of active UDP sessions: 889 number of active ICMP sessions: 4 number of active BCAST sessions: 0 number of active MCAST sessions: 0 number of predict sessions: 1884 session table utilization: 2% number of sessions created since system bootup: 142823265 Packet rate: 5920/s Throughput: 45871 Kbps
-------------------------------------------------------------------------------
----- Andrey Gordon [andrey.gordon@gmail.com]
On Tue, Feb 9, 2010 at 5:31 PM, Nathan Ward <nward@daork.net> wrote:
You don't know how many NAT sessions are open though, right?
This is where I'd start looking, if you do or not is up to you.
On 10/02/2010, at 11:26 AM, Andrey Gordon wrote:
Well, if I understand NATting right, I should be able to have at least 65000 sessions per NAT address to one destination. Am I wrong? the firewall is rated for 260K sessions.
----- Andrey Gordon [andrey.gordon@gmail.com]
On Tue, Feb 9, 2010 at 5:22 PM, Nathan Ward <nward@daork.net> wrote:
13,000 sessions could be your problem - perhaps you are running out of NAT state table space.
On 10/02/2010, at 11:18 AM, Andrey Gordon wrote:
Not 100% sure. I have more than one NAT address on that firewall two of which are dynamic: student and business. It's the student one that's broken. Now, with that said, the Palo Alto firewall shows 13,000 session in progress. Even the f/w guy does not know how to check out the session count per NATted IP.
----- Andrey Gordon [andrey.gordon@gmail.com]
On Tue, Feb 9, 2010 at 5:08 PM, Nathan Ward <nward@daork.net> wrote:
How many users do you have behind your NAT?
On 10/02/2010, at 11:04 AM, Andrey Gordon wrote:
Thx to all the folks replying off the list.
The more I trouble shoot the more I'm convinced that it's not the sites that are doing rate-limiting. I went to a website of one of my previous employers (a small company). Chances of them having a fancy reverse proxy with some sort of black list filtering are slim to none, yet their site barely opens up as well.
Must be something that either my firewall device is doing (which is what is doing the NATting) or I don't' know what else. I'm working with my firewall guy since f/w is his domain and I have no clue about that vendor of the firewalls (PaloAlto).
Thanks all for the suggestions. I'll keep digging.
----- Andrey Gordon [andrey.gordon@gmail.com]
On Tue, Feb 9, 2010 at 4:56 PM, Jay Hennigan <jay@west.net> wrote:
Andrey Gordon wrote:
> Can't find my IP on any of the black lists. Don't have any proxies. Sites > that behave poorly are consistent. That is to say that facebook.com , > apple.com would always come up without an issue, but cnn.com, > forever21.com(i know, don't ask, students), > store.apple.com would consistently take forever to come up. > > Just wanted to check of rate-limiting web clients is a common practice > nowdays in the industry. If it's not, it's probably an unlikely cause of > my > troubles... >
It could be that the problem sites have some form of load balancer that has an issue keeping state on multiple sessions from the same IP.
You mentioned that changing the source IP fixed it. Is this a temporary fix that breaks after several users access the sites from the new IP?
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
!DSPAM:22,4b71e13583451376319610!
On Tue, 09 Feb 2010 17:44:01 EST, Andrey Gordon said:
It does seem much like NAT exhaustion even though the f/w claims only 13K session for two dynamic NATs and about 20 static ones. What I don't get is why there is consistency in opening sites. Why does facebook open all the time and store.apple.com barely opens all the time.
This sounds like possibly a hash table with a spectacularly poor hash function, causing most of your entries to be in only a few hash buckets. You hit one of the 497 buckets that has 0 or 1 or 3 entries, it works great. You hit one of 3 buckets that has 4,000+ entries in it, things suck. (You Linux geeks can quit smirking - Linux had a very similar issue in its networking stack not so long ago). Never underestimate the ability of vendor engineers to write hilariously poor code: http://thedailywtf.com/Articles/Else-where.aspx You really gotta assume that your firewall code (or any other code, for that matter) was written by that programmer until proved otherwise.
On Tue, 2010-02-09 at 17:44 -0500, Andrey Gordon wrote:
What I don't get is why there is consistency in opening sites. Why does facebook open all the time and store.apple.com barely opens all the time. I'd say if it would be NAT exhaustion, they would all behave the same way meaning open and then not open and then open again.
My guess the fault drives some SSL/TLS sessions through some loadbalancers mad, but not all :) Gord
participants (3)
-
Andrey Gordon -
gordon b slater -
Valdis.Kletnieks＠vt.edu