- NANOG - lists.nanog.org

22 Jan '01

Hi All! I'm trying to get a feeling for how often ISPs announce a prefix from different origin ASes intentionally.... (and why...) I asked this question on my peering survey (*grin*), but I thought i'd hit a wider audience... Thanks. -abha ;)

4 4

Re: Wierd Route
by Sean Donelan 22 Jan '01

22 Jan '01

I hate to tell you ... In many cases, the folks in the NOC are primarily there to watch red lights and green lights. Even if you had the secret NOC phone number, it wouldn't do you much good. In a large provider the NOC has soo many rules and procedures, the NOC-level techs aren't even allowed to log onto core routers to look. And forget about making any changes to a router configuration. Even if the NOC folks wanted to help, they aren't allowed to. Whether its a large provider or a small provider, they both seem to have about the same number of people with the expertise and authority to fix "interesting" problems. It might be called 4th level support, or backbone engineering, or senior engineers. It has less to do with technical clue than management clue. I've often found a person in a NOC who understood the problem, and even understood what was required to fix it, but due to decisions by his management he wasn't allowed to fix it. There are management culture differences between the various providers which affect how well their employees can fix things. On Mon, 22 January 2001, Adrian Chadd wrote: > <off-topic> > > .. and since various companies keep hiring bright university students > right off the bat, they don't get to spend time learning how the net > works by working in a NOC. > > Which means, finding clueful people to staff a NOC is getting more > difficult each day. > > Thanks guys. :-) > > </offtopic>

2 1

Re: Wierd Route
by Richard A. Steenbergen 22 Jan '01

22 Jan '01

On Mon, Jan 22, 2001 at 03:05:20AM -0500, Joe Budion wrote: > > This one takes the cake for weirdest route I've ever seen: 10 p9-0.crtntx1-br1.bbnplanet.net (4.24.10.213) 48 msec 48 msec 44 msec 11 p15-0.crtntx1-br2.bbnplanet.net (4.24.10.114) 48 msec 48 msec 48 msec 12 p2-0.crtntx1-cr8.bbnplanet.net (4.24.8.198) 48 msec 48 msec 48 msec 13 p5-0.toucham.bbnplanet.net (4.24.117.66) 68 msec 68 msec 68 msec 14 den-edge-18.inet.qwest.net (205.171.16.34) 68 msec 68 msec 68 msec 15 208.46.255.14 120 msec 120 msec 120 msec 16 63.237.115.200 120 msec 120 msec 120 msec ner-routes>sh ip bgp 205.171.16.34 BGP routing table entry for 205.171.16.32/30, version 0 Paths: (1 available, no best path) 3908 4.24.117.66 (inaccessible) from 4.0.4.20 (4.24.0.207) Origin IGP, metric 0, localpref 100, valid, internal Community: 1:1001 209:209 209:30357 3908:900 Originator : 4.24.0.207, Cluster list: 4.0.4.20, 4.24.0.58 ner-routes>sh ip bgp 63.237.115.20 BGP routing table entry for 63.237.115.0/27, version 0 Paths: (1 available, no best path) 3908 4.24.117.66 (inaccessible) from 4.0.4.20 (4.24.0.207) Origin IGP, metric 5, localpref 100, valid, internal Community: 1:1001 209:209 209:30357 3908:900 Originator : 4.24.0.207, Cluster list: 4.0.4.20, 4.24.0.58 Gotte filters? -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

5 5

Re: Wierd Route
by Jon Stanley 22 Jan '01

22 Jan '01

not particularly when you take a look at it. Misbehaved origin AS: BGP routing table entry for 63.237.115.192/27, version 3368964 Paths: (3 available, best #3) Not advertised to any peer 5056 3561 3908 167.142.3.6 from 167.142.3.6 (167.142.3.6) Origin IGP, localpref 100, valid, external 1221 3561 3908 203.62.248.4 from 203.62.248.4 (203.62.248.4) Origin IGP, localpref 100, valid, external 1221 3561 3908 203.62.252.21 from 203.62.252.21 (203.62.252.21) Origin IGP, localpref 100, valid, external, best route-views.oregon-ix.net> A router will take the most specific path that it knows about to a particular destination. Since parts of the Internet are seeing a /27 here, it's winding up going all over the place. It would appear that AS3561 is allowing such small advertisements from AS3908, and then providers that dont have filtering at the border based on prefix length get injected this very specific route. Yet more examples of what happens with a lack of filtering........ On Mon, 22 Jan 2001, Joe Budion wrote: > > This one takes the cake for weirdest route I've ever seen: > > #3 sl-gw5-kc-5-0-0.sprintlink.net (144.232.132.45): TTL Exceeded, ttl=253, > 41 ms > #4 sl-bb20-kc-0-0.sprintlink.net (144.232.23.1): TTL Exceeded, ttl=252, 40 > ms > #5 sl-bb20-fw-13-0.sprintlink.net (144.232.9.254): TTL Exceeded, ttl=251, 50 > ms > #6 Unavailable (144.232.18.162): TTL Exceeded, ttl=250, 50 ms > #7 dal-core-02.inet.qwest.net (205.171.25.49): TTL Exceeded, ttl=249, 50 ms > #8 hou-core-01.inet.qwest.net (205.171.5.169): TTL Exceeded, ttl=248, 50 ms > #9 hou-edge-07.inet.qwest.net (205.171.23.14): TTL Exceeded, ttl=247, 50 ms > #10 a6-0-6.crtntx1-ba2.bbnplanet.net (4.24.147.21): TTL Exceeded, ttl=241, > 80 ms > #11 p10-0.crtntx1-br2.bbnplanet.net (4.24.10.189): TTL Exceeded, ttl=242, 81 > ms > #12 p2-0.crtntx1-cr8.bbnplanet.net (4.24.8.198): TTL Exceeded, ttl=241, 80 > ms > #13 p5-0.toucham.bbnplanet.net (4.24.117.66): TTL Exceeded, ttl=240, 100 ms > #14 den-edge-18.inet.qwest.net (205.171.16.34): TTL Exceeded, ttl=239, 101 > ms > #15 Unavailable (63.145.64.62): TTL Exceeded, ttl=238, 150 ms > #16 SLCNTWEB01 (63.237.115.200): Echo Reply, ttl=110, 150 ms > > Statistics: Out 14, in 14, loss 0%, times (min/avg/max) 40/76/150 ms > > joe > >

1 0

Re: Inter-provider communications (Re: nobody @home)
by Dan Hollis 22 Jan '01

22 Jan '01

On Sun, 21 Jan 2001, Jared Mauch wrote: > On Sun, Jan 21, 2001 at 09:41:49PM -0500, Steve Sobol wrote: > > I'm just waiting for one of the big NSPs to be sued due to complicity in > > an attack. That, essentially, is what we're dealing with. > I don't see that happening anytime. If it's big enough to sue over > the law enforcement agencies get involved and they tend to get the > attention required to stop these attacks on the isp/nsp side. By the time law enforcement has to be involved to convince a tier1 to shut off their ddos sources, it's far past the point of complicity and the preventable monetary damages have already occurred. You can bet someones going to get sued. Sadly, it's probably going to take a high profile lawsuit to get the tier1s to shape up their act. -Dan

3 3

Re: Inter-provider communications (Re: nobody @home)
by Patrick Greenwell 22 Jan '01

22 Jan '01

On Sun, 21 Jan 2001, Jared Mauch wrote: > On Sun, Jan 21, 2001 at 09:41:49PM -0500, Steve Sobol wrote: > > > > Patrick Greenwell wrote: > > > > > If a provider won't help their own paying customers, what chance is their > > > that they are going to help anyone that isn't their customer? > > > > I'm just waiting for one of the big NSPs to be sued due to complicity in > > an attack. That, essentially, is what we're dealing with. > > I don't see that happening anytime. If it's big enough to sue over > the law enforcement agencies get involved and they tend to get the > attention required to stop these attacks on the isp/nsp side. I think you are absolutely right, but not solely for that reason. Anyone big enough to play "my lawyers can beat up your lawyers" for real is likely to garner the attention of the provider and avoid litigation. So here we are with the Internet being ruled by lawyers.

1 0

Re: Inter-provider communications (Re: nobody @home)
by Patrick Greenwell 22 Jan '01

22 Jan '01

On Sun, 21 Jan 2001, Richard A. Steenbergen wrote: > On Sun, Jan 21, 2001 at 11:11:55AM -0800, Patrick Greenwell wrote: > > > > One large provider of hosting services who shall remain nameless in > > the hopes they will become more helpful through private discussion > > recently told one of my clients that placing RFC-1918 filters within > > their borders(the client was being DDOS'd in part from machines within > > that providers network) was "against policy" and they wouldn't do it. > > > > I shudder to think what they tell non-customers(if they even talk to > > them at all.) > > How would placing RFC1918 filters on that providers borders I said "within" not "on". To their credit the do have filters on their borders. > In all fairness, many large providers have a legitimate point when > refusing to deploy just any customer-request filter. Sure, but I'd hardly characterize this as some random request for just any customer-requested filter. They(the client) were undergoing a pretty significant DDOS attack in part being accomplished from within their(the hosting providers) network against another customer within their network, and the request (RFC-1918 filtering) isn't vodoo. > With most large hosting providers, what cisco markets as "core" routers > are required for customer aggregation. ACLs can have a serious impact on > performance and stability on these routers. And deploying filters "on > their borders" is a time consuming, performance impacting, perl-powered > mess. Huh? First, see above regarding "within" and "on." Second, RFC-1918 filtering is hardly a "time-consuming", "perl-powered" mess. > Why should they go through this for your 1Mbps of normal paid traffic > just so you can get on irc and taunt the packet kids with your "large > provider filters"? The client in question pushes a near constantly sustained 100Mbit and has no need, desire, and would see no value in tauting script kiddies on IRC. They're an attractive target simply because they are a high-profile site. If a provider won't help their own paying customers, what chance is their that they are going to help anyone that isn't their customer? Awaiting obligatory comments about moving away from said provider, SLAs, etc.... :-)

2 1

RE: Second day of rolling blackouts starts
by Sean Donelan 22 Jan '01

22 Jan '01

On Thu, 18 January 2001, "Steven J. Sobol" wrote: > Is anyone seeing lots of routing oddities? I'm not able to get to a lot > of sites that I normally can, that are hosted in different places; and I'm > wondering if some providers are routing around California outages. Not that I know of. There is something squirrely going on with the root name servers, but I haven't figured it out if it is just my location or more widespread.

6 6

Re: Inter-provider communications (Re: nobody @home)
by Basil Kruglov 21 Jan '01

21 Jan '01

On Sun, Jan 21, 2001 at 02:42:58PM -0500, Richard A. Steenbergen wrote: > How would placing RFC1918 filters on that providers borders help prevent > attacks originating from that providers network? Perhaps they should have > been busy tracing the attacker on their network? <not exactly related, but...> if you do host an irc server, or shell, or anything related to chat/irc... at some point attacks will become normal day to day events, no known NSP (to me) will be tracking those... no resources, time, most importantly -will- to justify it... "you're getting those daily, even if we do track this one down for you it won't make a bit of a difference" as for RFC1918, out-of-bgp table attacks, smurf/fraggle perhaps (?), new "loose" unicast RPF for Cisco might help, but only if big boys start putting this at edge (customer aggregation routers) to prevent stupid customers from spoofing, and at borders (peering points) to deny crap from peers... (if I'm not mistaken Verio, one of many who's doing this at some places, please do correct me if I'm wrong.) > In all fairness, many large providers have a legitimate point when > refusing to deploy just any customer-request filter. With most large > hosting providers, what cisco markets as "core" routers are required for > customer aggregation. ACLs can have a serious impact on performance and > stability on these routers. And deploying filters "on their borders" is a > time consuming, performance impacting, perl-powered mess. Why should they > go through this for your 1Mbps of normal paid traffic just so you can get > on irc and taunt the packet kids with your "large provider filters"? Yes.. yes... yes... personally I'd pay for that traffic, I can even work in 'no-filter' environment, but only if big boys work together to track and shut down those attacks, shut down smurf amps if they happen to be their customers/downstreams, etc.. "We provide transit to sh*tload of people, and we are not required to do any tracebacks, filtering...work with your upstream, bye bye.."; I've been getting this quite a lot lately... And what if my upstream is too damn tired or have no will to deal with it? Getting transit to someone else is not an option, simply no one is going to host or deal with you if you or your downstreams are getting 3-5, sometimes more attacks *daily*. I only see two options 1) get rid of all DoS-getting applications/clients, 2) peering, more peering, and even more peering never advertising prefixes that are getting attacked to certain companies that can't control their customers. > I've been accused of being anti-Cisco, but the simple fact of the matter > is that if you have a Juniper with an IP2 you will be able to filter > things that would make a GSR puke all over itself. "Cisco powered network" > be damned. Oh and just a quick note, just because someone has the > technically ability or the willingness to filter packets does not mean > they will be able to filter it well or stop the attacks. Oh yes... crisco sucks when it comes to performance. -Basil

1 0

Inter-provider communications (Re: nobody @home)
by Sean Donelan 21 Jan '01

21 Jan '01

On Sat, 20 January 2001, Dan Hollis wrote: > Youre lucky. Ive noticed an increasing trend that the NOC phone numbers > listed on corporate web pages come back as disconnected/RNA, and the > emails bounce "no such user". Real cute, especially when you're being > ddos'd. > > whois data is even worse. I suspect upwards of 25% of whois contact data > is worthless. This is not new, or even an increasing trend. I used to contact between 80 and 100 different NOCs every six months. I would call every NOC contact phone number, and ask them to confirm it was the correct phone number. Its amazing how many NOC's didn't know their own number. And when I told them what number I had dialed, didn't know that number was connected to their NOC. The phone number churn was a consistent 30%, even with contact information exchanged in accordance with written peer agreements. Every six months it would take me almost a week to reach 90 or so NOCs, and get the current contact information. In the case of a couple of NOCs, in three years I was never able to reach anyone at the NOC willing to confirm the information. I used to post the results of my "drill" on NANOG. IOPS set up a contact list for its members, and for a while was doing monthly drills. But since IOPS's keeps it a secret, I don't know how successful they've been. About two years ago, I proposed to all major providers we set up a set of permanent, private communication links between all the NOCs. The "phone number" would be your ASN(s). The system would be based on the same technology used by the NAWAS and NTA. Which lets you set up an instant conferences. I can set up the system so those providers who want to talk in "private," have conferences as small as two parties or as large as everyone on the network, while monitoring multiple circuits at the same time. 10 providers thought it was a good idea, but the two largest providers never responded to any inquiries. I can bring the idea out of mothballs again, if folks are interested.

4 4