- Test - lists.nanog.org

RE: Router / Protocol Problem
by Mike Walter 07 Sep '06

07 Sep '06

Good morning everyone. I just wanted to say thanks for all the help. I did discover the problem this morning and I should be hit with a herring. I upgraded the IOS on the router with the issue to match the other router and the problem was still there. So I tested and noticed the following line in the logs, since I was on console it popped up right in front of me. Sep 7 06:50:20.697 EST: %SEC-6-IPACCESSLOGP: list 166 denied tcp 69.50.222.8(25) -> 69.4.74.14(2421), 4 packets What is this I thought? What is my ACL 166 doing this? I thought I tested removing all access-lists from interfaces with the original problem came up. Apparently not. Here is my ACL 166, the first line is what was being matched. Apparently some how this connection is being matched via NBAR for good old Code Red. access-list 166 deny ip any any dscp 1 log access-list 166 deny tcp any any eq sunrpc access-list 166 deny tcp any any eq 135 access-list 166 deny tcp any any eq 137 access-list 166 deny tcp any any eq 138 access-list 166 deny tcp any any eq 139 access-list 166 deny tcp any any eq 445 access-list 166 deny tcp any any eq 5554 access-list 166 deny tcp any any eq 9996 access-list 166 deny tcp any any eq 1025 access-list 166 deny udp any any eq 1434 access-list 166 deny udp any any eq 135 access-list 166 deny udp any any eq netbios-ns access-list 166 deny udp any any eq netbios-dgm access-list 166 deny udp any any eq netbios-ss access-list 166 deny udp any any eq 445 access-list 166 deny icmp any any redirect access-list 166 deny ip 127.0.0.0 0.255.255.255 any access-list 166 deny ip 10.0.0.0 0.255.255.255 any access-list 166 deny ip 172.16.0.0 0.15.255.255 any access-list 166 deny ip 192.168.0.0 0.0.255.255 any access-list 166 permit ip any any class-map match-any http-hacks match protocol http url "*default.ida*" match protocol http url "*cmd.exe*" match protocol http url "*root.exe*" policy-map mark-inbound-http-hacks class http-hacks set ip dscp 1 I have always had this on my FE0/0 as an outbound ACL, well atleast since Code Red came about: ip access-group 166 out. Now I have two questions. Is that not a good idea to have this on FE0/0 out? Second, why the heck would a smtp connection be matched via my http-hacks class-map? Thanks again everyone, Mike -----Original Message----- From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Rodney Dunn Sent: Wednesday, September 06, 2006 8:45 PM To: Christopher L. Morrow Cc: Rodney Dunn; Mike Walter; Hank Nussbacher; Justin M. Streiner; nanog(a)merit.edu Subject: Re: Router / Protocol Problem Then that proves it's not a local router problem then. :) On Wed, Sep 06, 2006 at 07:49:26PM +0000, Christopher L. Morrow wrote: > On Wed, 6 Sep 2006, Rodney Dunn wrote: > > > > > Get a sniffer trace. Packets on the wire prove what's going on. > > provided the packets get back to him, it seems his problem is traffic > getting back to him :( so probably no packets will be on the wire > (none in question atleast)...

4 3

RE: Router / Protocol Problem
by Jeff Jirsa 07 Sep '06

07 Sep '06

On Sep 6, 2006, at 9:04 AM, Mike Walter wrote: > Recently with no changes to my network, I have been having problems > connecting to certain websites and mail servers. I am always able > to ping the sites and trace route without error. If I telnet to > port 80 or port 25 it does not connect. If I login to my router > and telnet sourcing my each of Internet Providers ports, I am able > to get to the sites. I have talked with all the providers and none Check a packet dump and see if your affected boxes are sending SWE (SYN with ECN enabled) instead of plain SYN packets. Some firewalls (at least default m0n0wall and older PIX) reject and dump ECN syn packets while allowing pure syns through. If your affected peer has you running through some weird filter that's dropping SWE packets, this would cause symptoms exactly as you're seeing - ping is fine, traceroute is fine, but TCP sessions never complete the handshake (as the receiving side never got the first SYN). - J

1 0

RE: Router / Protocol Problem
by Hank Nussbacher 07 Sep '06

07 Sep '06

At 07:27 AM 07-09-06 -0400, Mike Walter wrote: Best moved to cisco-nsp. -Hank Nussbacher http://www.interall.co.il >Good morning everyone. I just wanted to say thanks for all the help. I >did discover the problem this morning and I should be hit with a >herring. I upgraded the IOS on the router with the issue to match the >other router and the problem was still there. So I tested and noticed >the following line in the logs, since I was on console it popped up >right in front of me. > >Sep 7 06:50:20.697 EST: %SEC-6-IPACCESSLOGP: list 166 denied tcp >69.50.222.8(25) -> 69.4.74.14(2421), 4 packets > >What is this I thought? What is my ACL 166 doing this? I thought I >tested removing all access-lists from interfaces with the original >problem came up. Apparently not. Here is my ACL 166, the first line is >what was being matched. Apparently some how this connection is being >matched via NBAR for good old Code Red. > >access-list 166 deny ip any any dscp 1 log >access-list 166 deny tcp any any eq sunrpc >access-list 166 deny tcp any any eq 135 >access-list 166 deny tcp any any eq 137 >access-list 166 deny tcp any any eq 138 >access-list 166 deny tcp any any eq 139 >access-list 166 deny tcp any any eq 445 >access-list 166 deny tcp any any eq 5554 >access-list 166 deny tcp any any eq 9996 >access-list 166 deny tcp any any eq 1025 >access-list 166 deny udp any any eq 1434 >access-list 166 deny udp any any eq 135 >access-list 166 deny udp any any eq netbios-ns >access-list 166 deny udp any any eq netbios-dgm >access-list 166 deny udp any any eq netbios-ss >access-list 166 deny udp any any eq 445 >access-list 166 deny icmp any any redirect >access-list 166 deny ip 127.0.0.0 0.255.255.255 any >access-list 166 deny ip 10.0.0.0 0.255.255.255 any >access-list 166 deny ip 172.16.0.0 0.15.255.255 any >access-list 166 deny ip 192.168.0.0 0.0.255.255 any >access-list 166 permit ip any any > >class-map match-any http-hacks >match protocol http url "*default.ida*" >match protocol http url "*cmd.exe*" >match protocol http url "*root.exe*" > >policy-map mark-inbound-http-hacks >class http-hacks >set ip dscp 1 > >I have always had this on my FE0/0 as an outbound ACL, well atleast >since Code Red came about: ip access-group 166 out. > >Now I have two questions. Is that not a good idea to have this on FE0/0 >out? Second, why the heck would a smtp connection be matched via my >http-hacks class-map? > >Thanks again everyone, > >Mike > >-----Original Message----- >From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of >Rodney Dunn >Sent: Wednesday, September 06, 2006 8:45 PM >To: Christopher L. Morrow >Cc: Rodney Dunn; Mike Walter; Hank Nussbacher; Justin M. Streiner; >nanog(a)merit.edu >Subject: Re: Router / Protocol Problem > > >Then that proves it's not a local router problem then. :) > >On Wed, Sep 06, 2006 at 07:49:26PM +0000, Christopher L. Morrow wrote: > > On Wed, 6 Sep 2006, Rodney Dunn wrote: > > > > > > > > Get a sniffer trace. Packets on the wire prove what's going on. > > > > provided the packets get back to him, it seems his problem is traffic > > getting back to him :( so probably no packets will be on the wire > > (none in question atleast)... > > +++++++++++++++++++++++++++++++++++++++++++ > This Mail Was Scanned By Mail-seCure System > at the Tel-Aviv University CC.

4 3

RE: Router / Protocol Problem
by Mike Walter 07 Sep '06

07 Sep '06

Sorry, I am running iBGP. I just swapped out the NPE225 engine to a NPE400 and 512MB and have not seen a change yet. I am still unable to reach the sites. I am going to give it a while and sometime soon reboot the other router. I removed the single /24 today out the one connection to see if that would change anything as well. Mike -----Original Message----- From: Hank Nussbacher [mailto:hank@efes.iucc.ac.il] Sent: Wednesday, September 06, 2006 12:07 PM To: Mike Walter Cc: Justin M. Streiner; nanog(a)merit.edu Subject: RE: Router / Protocol Problem On Wed, 6 Sep 2006, Mike Walter wrote: > > Thanks for everyone's great input. Here are answers to Justin's > questions. > > #1 - 12.3.6a - 7204VXR (NPE400) 512MB - 200+ MB free > #2 - 12.2.15T5 - cisco 7204VXR (NPE225) - 256MB (I have a NPE400 - 512MB > I want to swap in) - 23MB Free (Issue?) > > Full Routes from all peers. No internal routing protocol as of yet, all > static routes. Getting ready to implement OSPF. I have not rebooted > the routers as a test. I have CEF on both routers. I have had some > customers complaining about slowness. No internal routing protocol? Not even iBGP? How do the 2 routers exchange info? How do the internal systems know which router to exit from? Or are they both independent? I assume you are AS26241 and peer with 3356, 4323 and 6181. I also assume you should be announcing your 2 prefixes: 69.4.64.0/20 216.68.104.0/21 but you have deaggregated a single /24 - 69.4.71.0/24 which has sent 34 BGP updates in the past 24 hours (which might be ok). So, it is a bit hard to debug this with only partial info. Regards, Hank Nussbacher http://www.interall.co.il

3 3

Cisco IOS GRE issue (fwd)
by Gadi Evron 06 Sep '06

06 Sep '06

---------- Forwarded message ---------- Date: Wed, 6 Sep 2006 18:33:44 +0200 From: FX <fx(a)phenoelit.de> To: full-disclosure(a)lists.grok.org.uk, bugtraq(a)securityfocus.com Subject: Cisco IOS GRE issue Phenoelit Advisory <wir-haben-auch-mal-was-gefunden #0815 +----> [ Title ] Cisco Systems IOS GRE decapsulation fault [ Authors ] FX <fx(a)phenoelit.de> Phenoelit Group (http://www.phenoelit.de) Advisory http://www.phenoelit.de/stuff/CiscoGRE.txt [ Affected Products ] Cisco IOS Tested on: C3550 IOS 12.1(19) Cisco Bug ID: CSCuk27655, CSCea22552, CSCei62762 CERT Vu ID: <not assinged> [ Vendor communication ] 07.07.05 Initial Notification, gaus(a)cisco.com 27.07.05 PSIRT realized that nobody took this bug, Paul Oxman took over 28.07.05 Paul successfully reproduces the issue 04.08.05 Paul notifies FX about availabe fixes 05.08.05 Paul notifies FX about new side effects discovered by Cisco 06.09.06 Final advisory going public as coordinated release *Note-Initial notification by phenoelit includes a cc to cert(a)cert.org by default [ Overview ] Cisco Systems IOS contains a bug when parsing GRE packets with GRE source routing information. A specially crafter GRE packet can cause the router to reuse packet packet data from unrelated ring buffer memory. The resulting packet is reinjected in the routing queues. [ Description ] The GRE protocol according to RFC1701 supports source routing different from the one known in IPv4. An optional header is added to the GRE header containing Source Route Entries for further routing. GRE header: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |C|R|K|S|s|Recur| Flags | Ver | Protocol Type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Checksum (optional) | Offset (optional) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Key (optional) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sequence Number (optional) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Routing (optional) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ When a specially crafted GRE packet with routing information is received by a Cisco IOS device, the offset field is not verified to point inside the packet but is subtracted from what appears to be a short integer holding the overall length of the IP packet, causing an overflow of the same. This causes other memory contents of the packet ring buffers to be interpreted as the payload IP packet and reinjected into the routing queue with fairly large length information: GRE decapsulated IP 0.3.74.0->0.0.1.30 (len=65407, ttl=39) GRE decapsulated IP 176.94.8.0->0.0.0.0 (len=64904, ttl=0) GRE decapsulated IP 0.15.31.193->176.94.8.0 (len=64894, ttl=237) GRE decapsulated IP 128.42.131.220->128.0.3.74 (len=64884, ttl=128) The outer IP packet must come from the configured tunnel source and be sent to the configured tunnel destination IP address. By carefully filling the ring buffers with legitimate traffic like ICMP, containing an IP header at the right offset, an attacker can create IP packets with large length values inside IOS. PSIRT believes this cannot be done, Phenoelit differs on that. [ Example ] Internet Protocol, Src Addr: 85.158.1.110 (85.158.1.110), Dst Addr: 198.133.219.25 (198.133.219.25) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 Total Length: 28 Identification: 0xaffe (45054) Flags: 0x00 Fragment offset: 0 Time to live: 30 Protocol: GRE (0x2f) Header checksum: 0xf409 (correct) Source: 85.158.1.110 (85.158.1.110) Destination: 198.133.219.25 (198.133.219.25) Generic Routing Encapsulation (IP) Flags and version: 0x4000 0... .... .... .... = No checksum .1.. .... .... .... = Routing ..0. .... .... .... = No key ...0 .... .... .... = No sequence number .... 0... .... .... = No strict source route .... .000 .... .... = Recursion control: 0 .... .... 0000 0... = Flags: 0 .... .... .... .000 = Version: 0 Protocol Type: IP (0x0800) Checksum: 0x0000 Offset: 99 [ Notes ] IOS implements GRE source routing as forwarding of the inner IP packet. Thus, a Source Route Entry of 255.255.255.255 will cause IOS to resend the GRE packet to the specified address according to the routing table (all in this case) on the appropriate interface (all in this case). The source address of the new packet will be the router's IP address, the destination address according to the received packet. This can be used to circumvent Access Control Lists with GRE. [ Solution ] Stop using GRE. There is no way in IOS to turn off source routing for GRE tunnels. To correct the parsing issue, try to install an IOS version containing the fixes CSCuk27655 or CSCea22552 or CSCei62762. [ end of file ($Revision: 1.3 $) ] -- FX <fx(a)phenoelit.de> Phenoelit (http://www.phenoelit.de) 672D 64B2 DE42 FCF7 8A5E E43B C0C1 A242 6D63 B564

1 0

RE: Router / Protocol Problem
by Mike Walter 06 Sep '06

06 Sep '06

Thanks for everyone's great input. Here are answers to Justin's questions. #1 - 12.3.6a - 7204VXR (NPE400) 512MB - 200+ MB free #2 - 12.2.15T5 - cisco 7204VXR (NPE225) - 256MB (I have a NPE400 - 512MB I want to swap in) - 23MB Free (Issue?) Full Routes from all peers. No internal routing protocol as of yet, all static routes. Getting ready to implement OSPF. I have not rebooted the routers as a test. I have CEF on both routers. I have had some customers complaining about slowness. Mike Walter, MCP Systems Administrator 3z.net a PCD Company http://www.3z.net "When Success is the Only Solution think 3z.net" Voice (859) 331-9004 Fax (859) 578-3522 -----Original Message----- From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Justin M. Streiner Sent: Wednesday, September 06, 2006 9:42 AM To: nanog(a)merit.edu Subject: Re: Router / Protocol Problem On Wed, 6 Sep 2006, Mike Walter wrote: > I normally would not post to the group, but I am 100% stumped and have > talked with peers with no luck. > > I have (2) Cisco 7204 Routers running BGP with 3 peers and HSRP. I am > not doing anything special with BGP, pretty much a default config that > has not changed in years. Please provide details on both your default config and the hardware you're using. You say you have two Cisco 7204s - are these straight '04s, or 7204VXRs? What NPE(s) are you using, and how much memory is on them? The BGP you're getting from your peers - are you getting full routes from any of them? Do you have CEF enabled on these routers? What IOS version(s) are running on these routers? What else are they doing besides slinging BGP routes? Does the problem go away for a while if you reboot one router or the other? Without knowing any of this, it sounds like you might have NPE-225, -300, or -400 with 256 MB of RAM and you are running into memory exhaustion issues from carrying full routes. That's been a pretty popular topic on this list and others like cisco-nsp in the last 12 months :) At a minimum, what do the output of "show mem summary" and "show ip bgp sum" from each router show you? Have you seen other performance problems lately, such as things getting mysteriously slower, beyond the rachability issues you mentioned above? If so, check if CEF is still running (if it was configured in the first place). When a 7200 gets dangerously low on free memory and CEF is running, it may cannibalize the IP CEF process to try to conserve memory. Earlier 12.0 releases did this - I don't know if newer ones still do it. jms

3 2

BGP-Inspect update...
by Manish Karir 06 Sep '06

06 Sep '06

All, Just wanted to provide people with a quick update on the BGP-Inspect project. The BGP-Inspect website (http://bgpinspect.merit.edu) provides people with an easy to use, fast, querable interface to a subset of the data collected at routeviews. Since the last update we have added 3 additional "views", i.e. peers whose data you can query individually. These new peers are: UUNET-MCI(157.130.10.233), Verio-CA(129.250.0.11) and France-Telecom-NYC(193.251.245.6). What this means is that we are extracting information for these 3 peers in addition to the 5 that we did previously. The website now hosts data going back to Aug 1 2005, and the database has grown to over 400GB. We continue to see a moderate amount of usage with people running queries on a continual basis. Roughly speaking queries from outside North America seem to outnumber queries from within(probably an artifact of the data the site provides). One question I had for the community is just how useful is it to provide live access to data going back upto a year. Has anyone actually used such historical data in the past? The original goal of the project was to house data for upto 6 months, we are now upto 13 months. If there is not much use for it perhaps we can age out data older than say 1 year? Comments welcome(offlist). thanks -manish

1 0

Router / Protocol Problem
by Mike Walter 06 Sep '06

06 Sep '06

I normally would not post to the group, but I am 100% stumped and have talked with peers with no luck. I have (2) Cisco 7204 Routers running BGP with 3 peers and HSRP. I am not doing anything special with BGP, pretty much a default config that has not changed in years. Recently with no changes to my network, I have been having problems connecting to certain websites and mail servers. I am always able to ping the sites and trace route without error. If I telnet to port 80 or port 25 it does not connect. If I login to my router and telnet sourcing my each of Internet Providers ports, I am able to get to the sites. I have talked with all the providers and none can find a problem. If I shut down one specific peer, everything works fine. So I keep thinking it was that peers problem some how. I have tested with just that peer up and I still can not connect. However, when talking with that peer, they are able to telnet from their network to the sites I can not reach. I don't know what else to check besides shutting down that peer. Which since it is under a 3 year contract, not an option. That isn't the real solution anyhow. Can anyone shed some light on or off-list? Thanks, Mike Walter

4 3

RE: Router / Protocol Problem
by Mike Walter 06 Sep '06

06 Sep '06

One more thing, I can successfully do a tcptraceroute if that matters. Mike Walter ________________________________ From: tony sarendal [mailto:dualcyclone@gmail.com] Sent: Wednesday, September 06, 2006 9:32 AM To: Mike Walter Cc: nanog(a)merit.edu Subject: Re: Router / Protocol Problem On 06/09/06, Mike Walter <mwalter(a)3z.net> wrote: I normally would not post to the group, but I am 100% stumped and have talked with peers with no luck. I have (2) Cisco 7204 Routers running BGP with 3 peers and HSRP. I am not doing anything special with BGP, pretty much a default config that has not changed in years. Recently with no changes to my network, I have been having problems connecting to certain websites and mail servers. I am always able to ping the sites and trace route without error. If I telnet to port 80 or port 25 it does not connect. If I login to my router and telnet sourcing my each of Internet Providers ports, I am able to get to the sites. I have talked with all the providers and none can find a problem. If I shut down one specific peer, everything works fine. So I keep thinking it was that peers problem some how. I have tested with just that peer up and I still can not connect. However, when talking with that peer, they are able to telnet from their network to the sites I can not reach. I don't know what else to check besides shutting down that peer. Which since it is under a 3 year contract, not an option. That isn't the real solution anyhow. Can anyone shed some light on or off-list? Give your peer a /32 to install on their access router, verify that return path is via them and have them do connectivity tests to your problem sites. If that checks out you step by step through it. Ask to be moved to a different access router, next change your hardware. /Tony -- Tony Sarendal - tony(a)polarcap.org IP/Unix -= The scorpion replied, "I couldn't help it, it's my nature" =-

1 0

RE: NNTP feed.
by John van Oppen 06 Sep '06

06 Sep '06

They might as aleron used to offer it. That comes with the disclaimer that I have never tried it. John :) -----Original Message----- From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Drew Weaver Sent: Tuesday, September 05, 2006 9:10 AM To: nanog(a)merit.edu Subject: NNTP feed. Does anyone know if cogent offers NNTP feeds to their DIA customers? Before we take the plunge we need to know and the sales fellas werent able to tell me this. -Drew

13 15