June 2008 - Test - lists.nanog.org

Types of packet modifications allowed for networks
by Jean-François Mezei 02 Jun '08

02 Jun '08

I would like any pointers to good documents that outline what sort of packet modifications are allowed (in terms of Internet culture/policies) by networks. Notably: For a transit network (neither sending or destination IPs belong to the network) For the sending network (originating IP belongs to that network) For the destination network (destination IP belongs to that network). Obviously, every router will change/decrement the TTL (and recalculate the header checksum) in the IP header. Are there other fields that are routinely changed at every hop ? Would it also be correct to state that any network along the way would have the right to fragment a packet in two or more pieces ? Or would that only be the destination network needing to fragment a packet to fit the last mile (PPP dialup or PPPoE ) in cases where MTU negotiations failed ? Are there sacred rules documented anywhere about not modifying anything else in the packets during transit ? Or has there never been any formal documentation on this because it was so obvious nobody was allowed to modify packets in transit ? (I an in Canada and currently participating in CRTC proceedings about the Bell throttling issue, and would like to consider the angle where the CRTC might decides Bell has the right to manage PPPoE packets as if they were TCPIP packets (by looking inside). Thanks in advance for any hints/pointers on this issue.

6 5

Comcast - Stuck route in Chicago directing MN traffic via Denver
by Eric Spaeth 02 Jun '08

02 Jun '08

For the last couple weeks there has been a route stuck in the Chicago wan/core that is directing some Minnesota-bound traffic through Denver, even though Chicago and the Roseville, MN aggregation remain up and directly connected. This has the dual benefit of unnecessarily increasing the load on Comcast's internal backbone as well as increasing latency for Minnesota subscribers connecting to "east of the Mississippi" destinations by ~20ms. I'm hoping Comcast engineers read this list, or someone in the carrier community can help poke one of their Comcast contacts to help get this resolved. Thanks in advance! "Wedged" route - 76.113.128.0/17 Correct route - 69.180.128.0/18 Example trace from Chicago source to 76.113.128.0/17: ========================================= traceroute to 76.113.128.1 (76.113.128.1), 30 hops max, 40 byte packets 1 69.65.40.62 (69.65.40.62) 0.542 ms 0.511 ms 0.508 ms 2 so2-0-0-0.er1.Chi1.Servernap.net (69.39.239.169) 1.632 ms 1.642 ms 2.121 ms 3 ge-6-20.car1.Chicago1.Level3.net (4.79.65.49) 1.605 ms 1.608 ms 1.619 ms 4 ae-2-54.edge1.Chicago2.Level3.net (4.68.101.115) 1.604 ms 1.602 ms 1.600 ms 5 COMCAST-IP.edge1.Chicago2.Level3.net (4.71.248.26) 2.735 ms 2.741 ms 2.739 ms 6 pos-0-8-0-0-cr01.denver.co.ibone.comcast.net (68.86.85.114) 27.284 ms 27.398 ms 27.387 ms 7 te-9-4-ar02.roseville.mn.minn.comcast.net (68.86.91.154) 44.177 ms * * 8 te-0-2-0-5-ar03.roseville.mn.minn.comcast.net (68.87.174.73) 28.352 ms 28.352 ms 28.349 ms 9 te-2-1-ur01.sims.mn.minn.comcast.net (68.87.174.74) 28.826 ms * * 10 te-8-3-ur02.sims.mn.minn.comcast.net (68.87.174.78) 28.959 ms * * 11 te-2-1-ur01.newport.mn.minn.comcast.net (68.87.174.82) 29.267 ms * te-2-1-ur01.newport.mn.minn.comcast.net (68.87.174.82) 28.700 ms 12 c-76-113-128-1.hsd1.mn.comcast.net (76.113.128.1) 28.638 ms 28.673 ms 28.667 ms ========================================= Example trace from Chicago source to working route 69.180.128.0/18 ========================================= traceroute to 69.180.130.1 (69.180.130.1), 30 hops max, 40 byte packets 1 69.65.40.62 (69.65.40.62) 0.482 ms 0.450 ms 0.446 ms 2 so2-0-0-0.er1.Chi1.Servernap.net (69.39.239.169) 1.595 ms 2.082 ms 2.082 ms 3 ge-6-20.car1.Chicago1.Level3.net (4.79.65.49) 1.568 ms 1.569 ms 1.579 ms 4 ae-2-52.edge1.Chicago2.Level3.net (4.68.101.51) 1.562 ms 1.563 ms 1.560 ms 5 COMCAST-IP.edge1.Chicago2.Level3.net (4.71.248.22) 2.708 ms 2.713 ms 2.711 ms 6 te-0-1-0-7-ar03.roseville.mn.minn.comcast.net (68.87.174.21) 13.144 ms 11.919 ms 11.877 ms 7 68.87.174.22 (68.87.174.22) 11.824 ms * * 8 te-8-3-ur02.brooklynpark.mn.minn.comcast.net (68.87.174.26) 12.333 ms * * 9 te-2-1-ur01.newhope.mn.minn.comcast.net (68.87.174.30) 12.012 ms * * 10 c-3-0-ubr02.newhope.mn.minn.comcast.net (69.180.130.1) 11.963 ms 12.018 ms 11.973 ms ========================================= -Eric

1 1

NANOG 42 Lightning Talk submission reminder...
by Joel Jaeggli 02 Jun '08

02 Jun '08

Greetings, Lightning talk submission remains open until Tuesday June 3rd. Submissions can be made on the NANOG PC website by logging in as or creating a speaker account: http://www.nanogpc.org A lightning talk is a very short presentation or speech by any attendee on any topic relevant to the NANOG audience. These are limited to ten minutes; this will be strictly enforced. If you have a topic that's timely, interesting, or even a crackpot idea you want to share, we encourage you to consider presenting it. The Program Committee will decide which submissions are relevant (using criteria based on the NANOG mailing list AUP) and choose the best ones to be presented. Use of slides is optional. All slides must be in PDF or Powerpoint format, and will be loaded in advance onto the speaker laptop on the podium. Thanks Joel

1 0

Update: NANOG 43 PGP signing party.
by Joel Jaeggli 02 Jun '08

02 Jun '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The keysigning sessions are going to be during the morning breaks during the general session, and will be located in the Gleason/Roebling rooms. Monday June 2nd 11:00-11:30 Tuesday June 3rd 11:00-11:30 If you plan to participate there is still time up until tomorrow to add your key to the keyring at: http://biglumber.com/x/web?ev=19916 And come to one or both sessions with some form of government issued photo ID. If you have any further questions, feel free to contact me via email or corner one of the people with the pgp signing dots since they mostly know the score. While printouts will probably be available at the sessions, feel free to add your key to the keyring right up to the time of the last keysigning event. thanks joel _______________________________________________ NANOG mailing list NANOG(a)nanog.org http://mailman.nanog.org/mailman/listinfo/nanog -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFIQ+FV8AA1q7Z/VrIRAuN6AJ4hlfoRX/B2lFC5xlLV+nX1jOnuhgCdE8Me vmxenQEVkrzrcT6waUiN3zk= =Oaxb -----END PGP SIGNATURE-----

1 0

IOS Rookit: the sky isn't falling (yet)
by Nicolas FISCHBACH 02 Jun '08

02 Jun '08

I finally got to see Topo's presentation this week-end at PH-Neutral and discuss it with him and FX. Given that the slides aren't online yet [1], that Core hasn't published Topo's technical paper on their website [2] yet either, and that I'm done replying to direct inquiries about it [3], here's a summary of the IOS rootkit saga and its impact on the Service Provider community (from my point of view :) Topo spent a lot of time (and if you ever loaded an IOS image in IDA you know what I'm talking about) analyzing strings and functions in IOS. In his proof of concept he located the code doing the password check and adds a trampoline to his backdoor code (by saving paramaters, glueing the two codes together, doing the "new" password check and returning properly to the main code path). Nice lesson on 101 hooking on IOS. The (oversimplified) modus operandi is pretty straight forward: take an image, decompress it, have his tool locate the function and later patch it, add his code by overwriting large strings, (re)compress the image and (re)calculate/fix the checksums. Pretty neat. The fact that he doesn't do basic binary patching makes the approach portable and not architecture, version or feature set specific. This image then needs to be uploaded to the router and the device need to be reloaded. This backdoor is persistent (vs the old backdoor trick using the TCL shell [4] which wasn't - or if you want to turn it into a non-volatile one it was easy to detect as in clear text in the startup/running configuration). An alternative approach is to use gdb on the router (and combine it with a TCL script to make it easier) and patch on the fly. This is non-persistent, but some people don't wan't to leave traces as large as an IOS image behind :) Or another alternative approach: network boot the router via TFTP. At the end of the day this is nothing new from a rootkit technology point of view, but it's in the IOS/router world. He deserves credit to actually have researched this in deep and managed to make it work (it's much more difficult to achieve this on a mostly undocumented and large binary than on common OSes). Respect. What's the best way to actually test this when you don't have the HW you ask ? Dynamips [9] is the answer. As long as the rootkit isn't too advanced and e.g. also hooks the write/copy functions (e.g. an attacker could store the image diff on the system and play a "proper" memory dump or proper IOS back when you write core/copy to TFTP) then FX's CIR[7] is the forensics tool of choice. On platforms where the IOS image is stored on an external flash card forensics may be easier. Here's [8] a "screenshot" of CIR vs Topo. So what's the impact today ? Topo's proof of concept doesn't bypass ACLs (rACLs, VTY ACLs), AAA, etc [yet], requires enable rights, a new image and a reload (or enable only if you do gdb-on-the-fly patching). In summary it's "noisy" and unless you bought the router on an auction site and/or download IOS from "alternative" sources) you should notice (or probably deserve to get owned :) See the Cisco PSIRT response for best current practices on securing routers [10] and my old forensics presentation [3]. In the past FX [5] and Mike Lynn [6] proved that code execution is doable. This is a different approach. Can it be combined ? Probably. It is much more complex ? Yes. Is it going to be architecture specific ? Probably. Future developments ? I'm surprised people still focus on the IOS side of things and don't attack the bootrom code as it's smaller and usually never changed unless you bring in some new/unsupported hardware/features. IOS-XR is probably going to become a target too as it makes some of these things easier [11] but code signing may have to be broken/bypassed first. This has been done on other devices, so it's just one more layer to attack. An alternative rootkit ? Privilege level 16 used by the Lawful Intercept [12] feature could be abused to do some of this too. Or the other way around: use a "patched" IOS to keep an eye on Law Enforcement's operations on the router as privilege level 15 doesn't allow it and the only alternative is to sniff the traffic export. I've probably missed some stuff (and got some stuff wrong), but this summary became way too long already and it's late. Feedback welcome! [1] Dragos should post them soon here: http://www.eusecwest.com/ [2] Watch http://www.coresecurity.com/?module=ContentMod&action=news&id=papers [3] Google "IOS rootkit" used to return the presentation below as first hit "Cisco Router Forensics" - http://www.securite.org/presentations/secip/ [4] http://seclists.org/bugtraq/2007/Nov/0384.html [5] http://www.phenoelit-us.org/ultimaratio/index.html http://www.milw0rm.com/exploits/77 [6] http://cryptome.org/lynn-cisco.pdf [7] http://cir.recurity.com/ [8] http://www.securite.org/nico/XP/CIRvsTopo.jpg [9] http://www.ipflow.utc.fr/index.php/Cisco_7200_Simulator [10] http://www.cisco.com/en/US/products/products_security_response09186a0080997… [11] http://lists.darklab.org/pipermail/darklab/2005-August/000029.html [12] http://www.cisco.com/en/US/docs/ios/12_0s/feature/guide/lawf_int.html Nico. -- Nicolas FISCHBACH Senior Manager - Network Engineering/Security - COLT Telecom e:(nico@securite.org) w:<http://www.securite.org/nico/>

18 40

Emerg data recovery recommdnations?
by david raistrick 02 Jun '08

02 Jun '08

guys, wrong place, I know, but network down is network down no matter which side of the cable it falls on... Can anyone give any solid recommendations for a data recovery service who can fly to our site to extract data from a f'ed up RAID array? It's an absolute emergency (for us, of course). offlist please. .d --- david raistrick http://www.netmeister.org/news/learn2quote.html drais(a)icantclick.org http://www.expita.com/nomime.html

1 0

ISP Security BOF: NANOG 43
by Danny McPherson 31 May '08

31 May '08

Folks, Here's the agenda for the ISP security BOF tomorrow afternoon. There's still some space left on the agenda so if there's something you'd like to discuss (or hear someone else discuss) drop Stefan and I an email. Thanks, see you in Brooklyn! -danny & Stefan --- Sunday, June 1 - 4-530 P Salon F/G ISP Security BOF: NANOG 43 http://www.nanog.org/mtg-0806/abstracts.php?sp=Fouant Moderators: Stefan Fouant/Neustar Danny McPherson/Arbor ISC SIE Update http://sie.isc.org/ Paul Vixie/ISC ICANN SSAC Update http://www.icann.org/committees/security/ Paul Vixie & Danny McPherson/SSAC DNS OARC Update/Meeting Pointers http://public.oarci.net/dns-operations/workshop-2008/ https://oarc.isc.org/register.php Keith Mitchell & Paul Vixie/OARC ISP Security POC Intros & Open Microphone ---

1 0