I'm running short on theories and options, so I thought I would see if any other ISPs have seen this problem on your network(s). If so, what was the cure? mjr -----Original Message----- The Unknown problem. Symptoms: At random times dialup, dedicated, & internal network users are unable to pass TCP traffic to off network sites. ICMP and UDP appears to be uneffected by the outage which lasts anywhere from 2 to 5 minutes. The problem appears to be wide spread with similar reports from WVNET and other ISPS. nTelos is experiencing a similar problem but we have yet to confirm it is the same. Problem has changed in it's action but remained similar enough to consider it the same problem. Effected Platforms: Windows 2000 Pro, XP Home, XP Pro, & 2003 Server. Uneffected Platforms: Unix, MacOS (?) History: During the week of 2/9/04 the call center started recieving reports of users being unable to connect to sites off the CityNet network. Sites hosted on the internal network are uneffected by the outage. Initally it was thought to be a Internet Explorer problem possably caused by the KB832894 / IE SP1 or other updates but after further investigation it was found that Mozilla users were encountering the same problem. After several days of testing it was determined that during the outage any TCP session started on any port would fail. TCP sessions started before the outage continue to work and show no ill effects from the outage. After logging connection attempts at various intervals on many machines there appears to be no sort of pattern in the outages. Most machines encounter the problem, some more than others and a few do not encounter it at all. The duration and frequency of the outage is very fluid. During an outage, we can verify that the packet does seem to leave and reenter the network: Mar 5 22:28:04 pittpa-chaswv-ds3 17083: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3376) -> 216.41.224.3(80), 1 packet Mar 5 22:28:09 pittpa-chaswv-ds3 17084: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.14.174(3376), 1 packet Mar 5 22:28:09 pittpa-chaswv-ds3 17085: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3378) -> 216.41.224.3(80), 1 packet Mar 5 22:28:09 pittpa-chaswv-ds3 17086: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.14.174(3378), 1 packet Mar 5 22:33:24 pittpa-chaswv-ds3 17089: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.14.174(3378), 7 packets Mar 5 22:33:24 pittpa-chaswv-ds3 17090: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.14.174(3376), 17 packets Mar 5 22:33:58 pittpa-chaswv-ds3 17092: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3378) -> 216.41.224.3(80), 7 packets Mar 5 22:33:58 pittpa-chaswv-ds3 17093: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3376) -> 216.41.224.3(80), 18 packets Mar 5 00:58:30 pittpa-clarwv-ds3 16062: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3183) -> 216.41.224.3(80), 1 packet Mar 5 00:58:30 pittpa-clarwv-ds3 16063: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3183), 1 packet Mar 5 01:03:28 pittpa-clarwv-ds3 16067: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3217) -> 216.41.224.3(80), 1 packet Mar 5 01:03:28 pittpa-clarwv-ds3 16068: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3217), 1 packet Mar 5 01:03:34 pittpa-clarwv-ds3 16069: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3228) -> 216.41.224.3(80), 1 packet Mar 5 01:03:34 pittpa-clarwv-ds3 16070: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3228), 1 packet Mar 5 01:03:39 pittpa-clarwv-ds3 16072: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3239) -> 216.41.224.3(80), 1 packet Mar 5 01:03:47 pittpa-clarwv-ds3 16073: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3183) -> 216.41.224.3(80), 74 packets Mar 5 01:04:13 pittpa-clarwv-ds3 16075: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3183), 72 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16078: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3218) -> 216.41.224.3(80), 4 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16079: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3217) -> 216.41.224.3(80), 3 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16080: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3221) -> 216.41.224.3(80), 19 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16081: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3228) -> 216.41.224.3(80), 5 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16082: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3229) -> 216.41.224.3(80), 6 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16083: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3236) -> 216.41.224.3(80), 9 packets Mar 5 01:08:47 pittpa-clarwv-ds3 16084: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3233) -> 216.41.224.3(80), 12 packets Mar 5 01:08:47 pittpa-clarwv-ds3 16085: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3239) -> 216.41.224.3(80), 21 packets Mar 5 01:09:12 pittpa-clarwv-ds3 16087: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3239), 19 packets Mar 5 01:09:12 pittpa-clarwv-ds3 16088: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3228), 4 packets Mar 5 01:09:12 pittpa-clarwv-ds3 16089: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3217), 2 packets Mar 5 01:09:12 pittpa-clarwv-ds3 16091: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3218), 3 packets Mar 5 01:09:13 pittpa-clarwv-ds3 16092: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3221), 17 packets Mar 5 01:09:13 pittpa-clarwv-ds3 16093: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3229), 5 packets Mar 5 01:09:13 pittpa-clarwv-ds3 16094: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3236), 7 packets Mar 5 01:09:13 pittpa-clarwv-ds3 16096: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3233), 9 packets Network analysis showed significant amounts of spoofed multicast traffic and odd arp traffic. 10:17:16.416222 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 278 10:17:16.421886 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 334 10:17:16.423873 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 262 10:17:16.426948 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 254 10:17:16.432095 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 298 10:17:16.435921 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 274 10:17:16.439959 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 328 10:17:16.445317 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 326 10:17:16.449688 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 330 10:17:16.463537 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 322 Steps were taken to elminiate the spoofed traffic on the routers and access servers in the form of ACLs and filter lists. Neither have eliminiated the problem... but to what extent they might have helped has yet to be determined. The problem is still occuring, for some users the duration of the outage seems to have shortened other users notice no difference. It is not yet known if the filtering on the routers and access servers or the conversion to the 10.x.x.x network has made any difference. We should have a better idea in the upcoming days. What Doesn't help: Removing windows updates. Turning off XP firewall. Searching for malware. (SpyBot-SD, Adaware) Virus scanning (Various softwares) Specifying dns servers. Reinstalling windows.
On Fri, 2004-03-12 at 21:17, Riley, Marty wrote:
10:17:16.416222 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 278
This is UPnP discovery. Take a look here: http://www.nthelp.com/upnpscrewup.htm http://www.linuxsa.org.au/mailing-list/2002-11/1134.html I see a lot of unicast UPnP traffic on my networks. UPnP seems like a train wreck waiting to happen, to me. It would be interesting to see what happens if one of your users turns UPnP off on their host. Just a shot in the dark. -- James H. Edwards Routing and Security At the Santa Fe Office: Internet at Cyber Mesa jamesh@cybermesa.com noc@cybermesa.com
On Fri, 12 Mar 2004, James Edwards wrote:
I see a lot of unicast UPnP traffic on my networks. UPnP seems like a train wreck waiting to happen, to me.
Yep. Giving insecure PC's the power to change firewall settings. Doesn't sound like the cleverest idea. I have a firewall, my computer can't be a zombie. Yes, I click on every attachment I see and install every program any random web site offers me, but I have a firewall so my computer can't be a zombie :-( But it does demostrate that people really, really want to run their applications no matter how we try to stop them. Instead of blocking people from running their applications, can we figure out better ways for them to run them safely?
That reads more like a person who is customer centric with an acceptable idea... -Henry Sean Donelan <sean@donelan.com> wrote: On Fri, 12 Mar 2004, James Edwards wrote:
I see a lot of unicast UPnP traffic on my networks. UPnP seems like a train wreck waiting to happen, to me.
Yep. Giving insecure PC's the power to change firewall settings. Doesn't sound like the cleverest idea. I have a firewall, my computer can't be a zombie. Yes, I click on every attachment I see and install every program any random web site offers me, but I have a firewall so my computer can't be a zombie :-( But it does demostrate that people really, really want to run their applications no matter how we try to stop them. Instead of blocking people from running their applications, can we figure out better ways for them to run them safely?
Any exploits out there yet? http://www.cisco.com/en/US/customer/products/products_security_advisory09186...
MessageThis in reply to the earlier thread Weird Problems? Well barring that, I've seen simuliar issues, maybe not the exact same timings but. I've noticed a couple of things while working with a roll out of Active-Directory and a recent upgrade to I.E 6.0 for the user base. Since there were several thousand users involved some of the issues were simply bad configs/drivers/etc. However one of the stats I have noticed is that in certain situations where a system is connected to a Cisco 3548, and the client is running in an Auto detect (AD/AN) mode that things are horendiously slow during boot up, and at various times seem to hang unexplainably. It seemed corrected by setting the client to 100/Full, but not in all cases. Lots of HTTP complaints still remain about accessing webpages etc. but never consistant. This of course is a pretty fresh problem and is still in my queue for research to start this Monday. As well, we've found that there was an odd bug with Cisco's 6513s and their 48 port 10/100/1000 line cards. This was using the latest IOS/CAT software at the time. Again not sure if its a documented problem but, several users were unable to Telnet or FTP to systems that teminated to the 6513, oddly we were able to icmp echo and pass HTTP. After sometime and 2 TACs I found that there was a bug regarding these items and real small packets (I Think less than 64bits??) being passed thru the 6513 and got an RMA for new Line cards. Again, perhaps nothing to do with your situation. Since the Nix systems and non-Doze seem not to have an issue, perhaps you can enlighten me with further Sniffs/Captures of these events directly? As soon as I get some more data/Captures on my end from the problems I'm seeing I can forward those apon request so as to keep S/N ratio down on Nanog (: Cheers, -Joe ----- Original Message ----- From: Riley, Marty To: nanog@merit.edu Sent: Friday, March 12, 2004 11:17 PM Subject: FW: hey had eric sent you
Bit hard by same bug. What version of code are you running on the 6513 8.1(2) fixes the bug on the 6x48 line cards. What happens is that packets of 64 bytes or less are silently dropped. Replacing linecards will not help unless there is another bug of which I am not aware. With a little digging I can dredge up the relevant DDTS. Scott C. McGrath On Sat, 13 Mar 2004, joe wrote:
MessageThis in reply to the earlier thread Weird Problems?
Well barring that, I've seen simuliar issues, maybe not the exact same timings but. I've noticed a couple of things while working with a roll out of Active-Directory and a recent upgrade to I.E 6.0 for the user base. Since there were several thousand users involved some of the issues were simply bad configs/drivers/etc. However one of the stats I have noticed is that in certain situations where a system is connected to a Cisco 3548, and the client is running in an Auto detect (AD/AN) mode that things are horendiously slow during boot up, and at various times seem to hang unexplainably. It seemed corrected by setting the client to 100/Full, but not in all cases. Lots of HTTP complaints still remain about accessing webpages etc. but never consistant. This of course is a pretty fresh problem and is still in my queue for research to start this Monday. As well, we've found that there was an odd bug with Cisco's 6513s and their 48 port 10/100/1000 line cards. This was using the latest IOS/CAT software at the time. Again not sure if its a documented problem but, several users were unable to Telnet or FTP to systems that teminated to the 6513, oddly we were able to icmp echo and pass HTTP. After sometime and 2 TACs I found that there was a bug regarding these items and real small packets (I Think less than 64bits??) being passed thru the 6513 and got an RMA for new Line cards. Again, perhaps nothing to do with your situation. Since the Nix systems and non-Doze seem not to have an issue, perhaps you can enlighten me with further Sniffs/Captures of these events directly? As soon as I get some more data/Captures on my end from the problems I'm seeing I can forward those apon request so as to keep S/N ratio down on Nanog (:
Cheers, -Joe
----- Original Message ----- From: Riley, Marty To: nanog@merit.edu Sent: Friday, March 12, 2004 11:17 PM Subject: FW: hey had eric sent you
participants (7)
-
Henry Linneweh -
James Edwards -
joe -
Ken Yeo -
Riley, Marty -
Scott McGrath -
Sean Donelan