April 2009 - Test - lists.nanog.org

Level3 funkiness
by J. Oquendo 15 Apr '09

15 Apr '09

Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail. One of my machines on AT&T: # traceroute level3.net traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets 4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms 21.477 ms 5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms 16.393 ms 6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019 ms 10.833 ms 7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms 14.474 ms 8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166 ms 9.725 ms 9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616 ms 16.275 ms 10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms ae-74-74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms ae-84-84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms 11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms 12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms 45.774 ms 13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115 ms 38.135 ms 14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms 67.937 ms 15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) 75.797 ms !H (From Texas through Above.net) $ traceroute level3.net|tail -n 1 traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets 11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H * ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H Confirmed it can't be reached from Travelers Ins, The Hartford, none of my connections. Anyone else seeing issues? I'm seeing drop off from clients going through their Atlanta interconnects with Charter and two other providers, which I can't make sense of. I DO KNOW they experienced some sort of issue with a TDM switch or so they said... Very broad statements: "We know teh interwebs are down please stand by" I know websites are one thing, but the chances of the website going down, a TDM switch being wacky and now clients traversing their networks complaining all at once seems a little out of the ordinary. =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP "It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett 227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E

13 18

RE: Level3 funkiness
by Dave Larter 15 Apr '09

15 Apr '09

Tracing to www.level3.net (4.68.95.28) dies at 4.68.94.1 for me as well. -----Original Message----- From: Andy Vance [mailto:avance@hq.speakeasy.net] Sent: Wednesday, April 15, 2009 3:55 PM To: Dave Larter; J. Oquendo; nanog(a)nanog.org Subject: RE: Level3 funkiness I'm not having any issues from either east coast or west coast in reaching Level-3. traceroute to www.level3.net (4.68.95.28), 30 hops max, 38 byte packets 1 5.ge-0-2-0.cr1.wdc1.speakeasy.net (66.92.159.252) 0.337 ms 0.269 ms 0.255 ms 2 100.ge-0-0-0.cr2.wdc1.speakeasy.net (69.17.83.66) 0.278 ms 0.264 ms 0.256 ms 3 ge-4-1-440.car2.Washington1.Level3.net (166.90.148.49) 0.932 ms 0.858 ms 0.822 ms 4 vlan89.csw3.Washington1.Level3.net (4.68.17.190) 14.051 ms 0.975 ms 1.001 ms 5 ae-82-82.ebr2.Washington1.Level3.net (4.69.134.153) 1.340 ms 1.127 ms 1.385 ms 6 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 18.646 ms 18.612 ms 18.519 ms 7 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 18.925 ms 19.119 ms 18.712 ms 8 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 47.165 ms 53.793 ms 54.239 ms 9 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 43.107 ms 43.287 ms ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) 42.558 ms 10 4.68.94.1 (4.68.94.1) 43.116 ms 43.128 ms 43.224 ms Cheers, Andy -----Original Message----- From: Dave Larter [mailto:dave@stayonline.com] Sent: Wednesday, April 15, 2009 12:46 PM To: J. Oquendo; nanog(a)nanog.org Subject: RE: Level3 funkiness Yes, I die on your hop14 with TWTelecom -----Original Message----- From: J. Oquendo [mailto:sil@infiltrated.net] Sent: Wednesday, April 15, 2009 3:36 PM To: nanog(a)nanog.org Subject: Level3 funkiness Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail. One of my machines on AT&T: # traceroute level3.net traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets 4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms 21.477 ms 5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms 16.393 ms 6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019 ms 10.833 ms 7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms 14.474 ms 8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166 ms 9.725 ms 9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616 ms 16.275 ms 10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms ae-74-74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms ae-84-84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms 11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms 12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms 45.774 ms 13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115 ms 38.135 ms 14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms 67.937 ms 15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) 75.797 ms !H (From Texas through Above.net) $ traceroute level3.net|tail -n 1 traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets 11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H * ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H Confirmed it can't be reached from Travelers Ins, The Hartford, none of my connections. Anyone else seeing issues? I'm seeing drop off from clients going through their Atlanta interconnects with Charter and two other providers, which I can't make sense of. I DO KNOW they experienced some sort of issue with a TDM switch or so they said... Very broad statements: "We know teh interwebs are down please stand by" I know websites are one thing, but the chances of the website going down, a TDM switch being wacky and now clients traversing their networks complaining all at once seems a little out of the ordinary. =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP "It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett 227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E

2 1

SIP - perhaps botnet? anyone else seeing this?
by Leland E. Vandervort 15 Apr '09

15 Apr '09

Hi All, Over the past couple of days we have been seeing an exponential increase (about 200-fold) in the amount of UDP SIP Control traffic in our netflow data. The past 24 hours, for example, has shown a total of nearly 300 GB of this traffic incoming and over 400 GB outgoing -- this despite the fact that we do not host any SIP services ourselves, and currently to my knowledge, we have no hosting customers running any kind of SIP services. (Total RTP traffic for 24 hours is only in the region of 150 Kb -- so a vast inbalance between control and RTP) The local sources/destinations of the traffic are within our hosting space, but are spread across a wide range of hosts (i.e. nothing really related to a single or handful of hosts). Additionally over the past couple of days we have seen an increase of mails to our abuse desk for "brute force" attempts against a number of SIP services... possibly directly related to this traffic. Is anyone aware of a new variant or modus-operandi of botnets in circulation in the past couple of days which attempt to exploit SIP services? Has anyone else notice a significant increase in this kind of traffic? Thanks Leland

8 11

Re: Network SLA
by Saqib Ilyas 15 Apr '09

15 Apr '09

Hmmm. Good point. Perhaps the Internet traffic gets only a small share of the link capacity and the rest is reserved for corporate clients' VPN traffic etc. I was thinking more along the lines of corporate SLAs, not for Internet traffic. On Wed, Apr 15, 2009 at 4:05 PM, Rod Beck <Rod.Beck(a)hiberniaatlantic.com>wrote: > Congestion is more common than you think. And by the way, if congestion > is not a problem in Pakistan, then why is the VoIP qualit so poor? > > :) > > Roderick S. Beck > Director of European Sales > Hibernia Atlantic > 13-15, rue Sedaine, 75011 Paris > http://www.hiberniaatlantic.com > Wireless: 1-212-444-8829. > French Landline: 33+1+4355+8224 > French Wireless: 33-6-14-33-48-97. > AOL Messenger: GlobalBandwidth > rod.beck(a)hiberniaatlantic.com > rodbeck(a)erols.com > ``Unthinking respect for authority is the greatest enemy of truth.'' Albert > Einstein. > > > > -----Original Message----- > From: Saqib Ilyas [mailto:msaqib@gmail.com <msaqib(a)gmail.com>] > Sent: Wed 4/15/2009 11:22 AM > To: nanog(a)nanog.org > Subject: Re: Network SLA > > I talked to the NOC personnel at a small (compared to North American > standards) ISP in Pakistan. They said that their core links are operating > at > less than 50% utilization most of the time. Under such conditions, > violating > SLA conditions in the core is unlikely. If such is also the case with most > service providers in the North America as well, then why would they even > use > active measurement such as iPerf or BRIX or Cisco IP SLAs before signing an > SLA? > Thanks and best regards > > On Thu, Feb 19, 2009 at 8:50 PM, Saqib Ilyas <msaqib(a)gmail.com> wrote: > > > Greetings > > I am curious to know about any tools/techniques that a service provider > > uses to assess an SLA before signing it. That is to say, how does an > > administrator know if he/she can meet what he is promising. Is it based > on > > experience? Are there commonly used tools for this? > > Thanks and best regards > > -- > > Muhammad Saqib Ilyas > > PhD Student, Computer Science and Engineering > > Lahore University of Management Sciences > > > > > > -- > Muhammad Saqib Ilyas > PhD Student, Computer Science and Engineering > Lahore University of Management Sciences > > -- Muhammad Saqib Ilyas PhD Student, Computer Science and Engineering Lahore University of Management Sciences

3 2

ACLs vs. full firewalls
by Michael Helmeste 15 Apr '09

15 Apr '09

Hi all, One of the duties of my current place of employ is reorganizing the network. We have a few Catalyst 6500 series L3 switches, but currently do all packet filtering (and some routing) using a software based firewall. Don't ask me, I didn't design it :) Current security requirements are only based on TCP and non-stateful UDP src/dst net/port filtering, and so my suggestion was to use ACLs applied on the routed interface of each VLAN. There was some talk of using another software based firewall or a Cisco FWSM card to filter traffic at the border, mostly for management concerns. We expect full 1 gig traffic levels today, and 10 gig traffic levels in the future. I view ACLs as being a cheap, easy to administrate solution that scales with upgrades to new interface line speeds, where a full stateful firewall isn't necessary. However, I wanted to get other opinions of what packet filtering solutions people use in the border and in the core, and why. What's out there, and why do you guys use it? How do you feel about the scalability, performance, security, and manageability of your solution? What kind of traffic levels do you put through it?

13 14

Re: Fiber cut in SF area
by Murphy, Brian S CTR USAF ACC 83 NOS/Det 4 15 Apr '09

15 Apr '09

I haven't seen any mention of the possible use of FSO (Free Space Optics) by the provider to restore some reasonable amount of connectivity during an outage due to a fiber cut. I would expect that having 2 or 3 pairs of FSO boxes to provide a "reduced failover capacity" in metro areas would be a reasonable measure to ensure service for extended physical (fiber break, cut, backhoe) outages - although not necessarily for power. Yes, it would take some time to roll them out and set them up, but less time than the crew working the splices, and the folks handling the FSO boxes should be different from the fiber splice truck roll crew. Note that a power outage would not allow microwave to be an effective remediation method either. Plus, FSO's use of lasers (vice microwaves) means no issues with spectrum (AFAIK). Granted, they have limited distance and require LoS, but using two or more pairs can probably handle the 80% situation in the metro (unless there is data to indicate otherwise). murph ------------------------------------- Date: Tue, 14 Apr 2009 15:57:52 -0700 From: Roy <r.engehausen(a)gmail.com> Subject: Re: Fiber cut in SF area To: JC Dill <jcdill.lists(a)gmail.com> Cc: nanog(a)nanog.org Message-ID: <49E514F0.7000009(a)gmail.com> Content-Type: text/plain; charset=ISO-8859-1 JC Dill wrote: > Gino Villarini wrote: >> Good points, some variables are dependant on the network infrastructure >> of the wireless provider. Localy, the main 2 providers have a >> "copper/fiber independent" networks. >> >> > I'm pretty sure the WISPs in the Santa Cruz and Gilroy/Morgan Hill > areas were all also taken offline due to the fiber cut. (Roy, can you > verify, for south county?) Anyone in those areas who relied on a WISP > as a backup to their fiber/copper link found that their "redundant" > system wasn't really redundant after all. > > You may want to check (verify) how your 2 main providers handle their > backhaul. > > jc > > It based on where the WISP fiber feed was located but in general they were all down. There were some special edge cases that stayed up fed from distant mountain tops. It didn't seem to matter who your upstream ISP was, they were all gone.

3 2

Network SLA
by Saqib Ilyas 15 Apr '09

15 Apr '09

Greetings I am curious to know about any tools/techniques that a service provider uses to assess an SLA before signing it. That is to say, how does an administrator know if he/she can meet what he is promising. Is it based on experience? Are there commonly used tools for this? Thanks and best regards -- Muhammad Saqib Ilyas PhD Student, Computer Science and Engineering Lahore University of Management Sciences

11 16

Anyone from Intelligence Network Online?
by Justin Ream 15 Apr '09

15 Apr '09

Hi - I wanted to see if anyone is here from Intelligence Network Online - I suspect an old AS number and a /16 of yours is being hijacked by a spam gang operating in downtown LA and wanted to get some confirmation. -Justin

1 0

RE: Fiber cut in SF area
by Gino Villarini 15 Apr '09

15 Apr '09

Here in my area most of business outfits that require maximum availability of Internet or WAN conenctions have implemented dual connections from dual providers, most with a fiber/copper main and a fixed wireless backup. This trend goes from banks to Mcdonalds Gino A. Villarini gav(a)aeronetpr.com Aeronet Wireless Broadband Corp. tel 787.273.4143 fax 787.273.4145 -----Original Message----- From: Jorge Amodio [mailto:jmamodio@gmail.com] Sent: Tuesday, April 14, 2009 11:21 AM To: nanog(a)nanog.org Subject: Re: Fiber cut in SF area > "Earth is a single point of failure." On top of that, one basic principle of telecommunications: No matter how much diversity and path redundancy, tons of concrete or titanium sealed fiber vaults you have, in the data exchange between points A and B there will be always two single points of failure: A and B. IMHO, this thread is getting way off topic, boring and useless. Fiber cut is over, there will be many more, move on ... Cheers Jorge

6 5

RE: Fiber cut in SF area
by Gino Villarini 15 Apr '09

15 Apr '09

Good points, some variables are dependant on the network infrastructure of the wireless provider. Localy, the main 2 providers have a "copper/fiber independent" networks. Gino A. Villarini gav(a)aeronetpr.com Aeronet Wireless Broadband Corp. tel 787.273.4143 fax 787.273.4145 -----Original Message----- From: Deepak Jain [mailto:deepak@ai.net] Sent: Tuesday, April 14, 2009 4:36 PM To: Gino Villarini; Jorge Amodio; nanog(a)nanog.org Subject: RE: Fiber cut in SF area I don't mean to jump in here and state the obvious, but wireless links are not a panacea. At least a few folks have presented that fiber grooming has affected their *region*. It's not difficult to imagine that wherever the "head" link side (or agg point) of these regional wireless networks is... probably coincides with a fiber network or other telecom POP. You are just moving where your last mile vulnerabilities are (slightly.. as you are picking up multiple power vulnerabilities, Line of Sight, and other things along the way). In the example of a tornado or other weather disturbance, wireless links are subject to fade just as much as any kind of aerial wired asset. Deepak Jain AiNET > -----Original Message----- > From: Gino Villarini [mailto:gav@aeronetpr.com] > Sent: Tuesday, April 14, 2009 4:12 PM > To: Jorge Amodio; nanog(a)nanog.org > Subject: RE: Fiber cut in SF area > > Here in my area most of business outfits that require maximum > availability of Internet or WAN conenctions have implemented dual > connections from dual providers, most with a fiber/copper main and a > fixed wireless backup. This trend goes from banks to Mcdonalds > > > Gino A. Villarini > gav(a)aeronetpr.com > Aeronet Wireless Broadband Corp. > tel 787.273.4143 fax 787.273.4145 > > -----Original Message----- > From: Jorge Amodio [mailto:jmamodio@gmail.com] > Sent: Tuesday, April 14, 2009 11:21 AM > To: nanog(a)nanog.org > Subject: Re: Fiber cut in SF area > > > "Earth is a single point of failure." > > On top of that, one basic principle of telecommunications: > > No matter how much diversity and path redundancy, tons of concrete or > titanium sealed fiber vaults you have, in the data exchange between > points A and B there will be always two single points of failure: A > and B. > > IMHO, this thread is getting way off topic, boring and useless. > > Fiber cut is over, there will be many more, move on ... > > Cheers > Jorge

5 4