Best way to deal with bad advertisements?
Hi! I'm going to ask the rest of the NANOG community for their thoughts/opinions on a problem that's been plaguing us periodically that we haven't been able to find a satisfactory solution for yet. There's an ISP back on the East Coast that has been periodically advertising more specific routes for /24's out of our CIDR blocks and black-holing the traffic within their network. We've called all the listed numbers for their technical, admin, billing, and any other contacts we can find, and haven't been able to reach a human; we've left messages of various levels of nastyness, from very sugary on up to vaguely threatening. In every case, including the current one, it's been more than 24 hours, and they still haven't made any response to the problem; in fact, I just got paged by our NOC early this morning informing me they've stolen another one of our /24's. As you can well imagine, all the customers on those blocks are _very_ unhappy. Each time this happens, we end up with dissatisfied customers, many of whom leave, deciding that we're too unstable, and can't provide quality network connectivity, even though to the best of my knowledge, there's nothing we can do to prevent these people from stealing our blocks. My question to the NANOG community is twofold and simple: Am I overlooking some solution that would allow us to 'negate' their advertisement of our blocks (205.159.193.0/24 and 207.88.102.0/24 in this case) and secondly, is there a formal process within the community to seek recompense, or formal action against a clueless and net-unfriendly ISP, perhaps one as simple as the net equivalent of Mennonite 'shunning'? Or are we simply out of luck, and have to simply tell our customers "Sorry, everyone is at the mercy of the morons who can steal IP blocks simply by advertising more specific routes with higher weights?" It's getting really tempting to advertise the networks they have their nameservers on from *our* network with a weight of 65535, just to get them to call us back. :-( :-( Anyhow, enough frustrated venting, I *am* very interested in what the community feels is the best policy to follow in situations like this. Thanks again! Matt Petach Network Engineer (writing from home)
There's an ISP back on the East Coast that has been periodically advertising more specific routes for /24's out of our CIDR blocks and black-holing the traffic within their network.
We've called all the listed numbers for their technical, admin, billing, and any other contacts we can find, and haven't been able to reach a human; we've left messages of various levels of nastyness, from very sugary on up to vaguely threatening. In every case, including the current one, it's been more than 24 hours, and they still haven't made any response to the problem; in fact, I just got paged by our NOC early this morning informing me they've stolen another one of our /24's.
In this case, the very first thing you should probably do is to start announcing the more specific /24s to match their advertisements! Depending on AS-PATH length (how various nets hear your announcements vs. theirs) this may solve the immediate problem, allowing you to hunt them down and kill them at your leisure.
As you can well imagine, all the customers on those blocks are _very_ unhappy. Each time this happens, we end up with dissatisfied customers, many of whom leave, deciding that we're too unstable, and can't provide quality network connectivity, even though to the best of my knowledge, there's nothing we can do to prevent these people from stealing our blocks.
My question to the NANOG community is twofold and simple: Am I overlooking some solution that would allow us to 'negate' their advertisement of our blocks (205.159.193.0/24 and 207.88.102.0/24 in this case) and secondly, is there a formal process within the community to seek recompense, or formal action against a clueless and net-unfriendly ISP, perhaps one as simple as the net equivalent of Mennonite 'shunning'?
1) Announce *your own* routes more specifically. This may lose you ANS connectivity, though. 2) Announce *their* routes more specifically. Especially the routes for their web, news, and dns servers. I've never had to do this, but it came very close once. A major backbone provider had a customer that was announcing our own most critical /24 (that we normally advertise as a /23) and the NOC staff was unable to get anyone to put an internal-to-their-net filter on it. They had to spend a few hours to contact their customer to get them to stop announcing it! It was quite frustrating, and if announcing the /24 more specifically ourselves hadn't solved the problem (which it did, except for the customers of said major backbone, which we really don't get that many complaints about when they are unreachable) the next step would have been to announce one of their /24s - or to take it to NANOG. 3) You can post to NANOG and other lists in an attempt to embarrass/ get someone who knows the jokers to poke them.
Or are we simply out of luck, and have to simply tell our customers "Sorry, everyone is at the mercy of the morons who can steal IP blocks simply by advertising more specific routes with higher weights?"
Are there higher weights involved?
It's getting really tempting to advertise the networks they have their nameservers on from *our* network with a weight of 65535, just to get them to call us back. :-( :-(
No weights are necessary; the more specific route wins.
Anyhow, enough frustrated venting, I *am* very interested in what the community feels is the best policy to follow in situations like this.
Thanks again!
Matt Petach Network Engineer (writing from home)
Avi
There's an ISP back on the East Coast that has been periodically advertising more specific routes for /24's out of our CIDR blocks and black-holing the traffic within their network.
We've called all the listed numbers for their technical, admin, billing, and any other contacts we can find, and haven't been able to reach a human; we've left messages of various levels of nastyness, from very sugary on up to vaguely threatening. In every case, including the current one, it's been more than 24 hours, and they still haven't made any response to the problem; in fact, I just got paged by our NOC early this morning informing me they've stolen another one of our /24's.
In this case, the very first thing you should probably do is to start announcing the more specific /24s to match their advertisements! Depending on AS-PATH length (how various nets hear your announcements vs. theirs) this may solve the immediate problem, allowing you to hunt them down and kill them at your leisure.
The downside to this is that we go from advertising /16's out, to advertising a fleet of /24's out, most of which would be filtered by Sprint's ever-lovin' CIDR-forcing wall. I agree with Sprint, and Sean, but in this case it pretty much makes it hard for us to force the issue by dropping to the same or smaller sized announcement. Good thought, though! Even if it does result in going from 2 /16 announcements to 512 /24 announcements in the process, growing the routing tables, and generally making everyone else unhappy as well. *sigh* There really MUST be some nice way of handling lame ISP's like this.
As you can well imagine, all the customers on those blocks are _very_ unhappy. Each time this happens, we end up with dissatisfied customers, many of whom leave, deciding that we're too unstable, and can't provide quality network connectivity, even though to the best of my knowledge, there's nothing we can do to prevent these people from stealing our blocks.
My question to the NANOG community is twofold and simple: Am I overlooking some solution that would allow us to 'negate' their advertisement of our blocks (205.159.193.0/24 and 207.88.102.0/24 in this case) and secondly, is there a formal process within the community to seek recompense, or formal action against a clueless and net-unfriendly ISP, perhaps one as simple as the net equivalent of Mennonite 'shunning'?
1) Announce *your own* routes more specifically. This may lose you ANS connectivity, though.
And Sprint, and anyone else that filters small specifics.
2) Announce *their* routes more specifically. Especially the routes for their web, news, and dns servers. I've never had to do this, but it came very close once. A major backbone provider had a customer that was announcing our own most critical /24 (that we normally advertise as a /23) and the NOC staff was unable to get anyone to put an internal-to-their-net filter on it. They had to spend a few hours to contact their customer to get them to stop announcing it! It was quite frustrating, and if announcing the /24 more specifically ourselves hadn't solved the problem (which it did, except for the customers of said major backbone, which we really don't get that many complaints about when they are unreachable) the next step would have been to announce one of their /24s - or to take it to NANOG.
I took that step last night, and was advised to remove it by those more in tune with legal issues. I guess it's not considered "nice" to sink to the same level as your attacker, and play dirty. :-} So, I removed my announcement of their backbone shortly after putting it in place. 'Twould have been sweet revenge, though...
3) You can post to NANOG and other lists in an attempt to embarrass/ get someone who knows the jokers to poke them.
Or are we simply out of luck, and have to simply tell our customers "Sorry, everyone is at the mercy of the morons who can steal IP blocks simply by advertising more specific routes with higher weights?"
Are there higher weights involved?
Nominally higher, but I won't vouschafe that it's intentional--it could be more of a simple matter of by default, they've set their weights at a level that happens to be higher than our default level. I try to avoid assuming culpability in cases where it may just be coincidence. Occam's razor, and all that sort of thing.
It's getting really tempting to advertise the networks they have their nameservers on from *our* network with a weight of 65535, just to get them to call us back. :-( :-(
No weights are necessary; the more specific route wins.
I know, I actually did it briefly yesterday out of sheer frustration, and then pulled it back out when counselled that even self-defense isn't always a sure win if the laywers get involved.
Anyhow, enough frustrated venting, I *am* very interested in what the community feels is the best policy to follow in situations like this.
Thanks again!
Matt Petach Network Engineer (writing from home)
Avi
Again, my thanks for you feedback and support! Matt Petach
In this case, the very first thing you should probably do is to start announcing the more specific /24s to match their advertisements! Depending on AS-PATH length (how various nets hear your announcements vs. theirs) this may solve the immediate problem, allowing you to hunt them down and kill them at your leisure.
The downside to this is that we go from advertising /16's out, to advertising a fleet of /24's out, most of which would be filtered by Sprint's ever-lovin' CIDR-forcing wall. I agree with Sprint, and Sean, but in this case it pretty much makes it hard for us to force the issue by dropping to the same or smaller sized announcement.
Good thought, though! Even if it does result in going from 2 /16 announcements to 512 /24 announcements in the process, growing the routing tables, and generally making everyone else unhappy as well.
We got hit by some guys from Poland (which made them uncontactable by phone). Try this: * Continue to advertise your /16s * Advertise more specifics of only the routes from the closest AS to the trouble. Ensure they also filter the bad routes as do all the AS's between that and the customer. * If anyone filters the more specifics, you will be *fine* as they will also be filtering the bogon more-specifics, but will still hear your /16s. * Take the AS path to the bogon route. Complain to those in the AS path in reverse order, till you get to someone default free. You might also want to enlist the help of your upstream. Alex Bligh Xara Networks
We got hit by some guys from Poland (which made them uncontactable by phone).
Try this:
* Continue to advertise your /16s * Advertise more specifics of only the routes from the closest AS to the trouble. Ensure they also filter the bad routes as do all the AS's between that and the customer. * If anyone filters the more specifics, you will be *fine* as they will also be filtering the bogon more-specifics, but will still hear your /16s.
Ah, true :) I knew there was a reason we were fine just announcing matching more specifics.
* Take the AS path to the bogon route. Complain to those in the AS path in reverse order, till you get to someone default free. You might also want to enlist the help of your upstream.
Alex Bligh Xara Networks
Avi
Hi Matt,
In this case, the very first thing you should probably do is to start announcing the more specific /24s to match their advertisements! Depending on AS-PATH length (how various nets hear your announcements vs. theirs) this may solve the immediate problem, allowing you to hunt them down and kill them at your leisure.
The downside to this is that we go from advertising /16's out, to advertising a fleet of /24's out, most of which would be filtered by Sprint's ever-lovin' CIDR-forcing wall.
If your more specific networks are filtered, then wouldn't the evil ISP's be filtered as well? This would be a large problem only if you gain transit from Sprint....
I agree with Sprint, and Sean, but in this case it pretty much makes it hard for us to force the issue by dropping to the same or smaller sized announcement.
Well, I'm not sure that the two entities can be put in the same sentence any more, but you can always leave the less specific /16 in there while you attempt to advertise the more speciic.
Good thought, though! Even if it does result in going from 2 /16 announcements to 512 /24 announcements in the process, growing the routing tables, and generally making everyone else unhappy as well.
I'd rather have happy workable customers and an unhappy community in the short run, than unhappy unworkable customers and a happy community. I think your letter will raise the awareness of this kind of problem. Of course we all know it's possible, but it's not a problem that we've had to deal with on a malicious level. ? I do assume that there's no doubt the evil-isp is doing this maliciously?
*sigh* There really MUST be some nice way of handling lame ISP's like this.
One thing you could do is coordinate with largerish ISPs to filter the incorect network from the affected peering sessions. While this is a stopgap fix, and not one to be repeated, I don't think you'd have problems getting it done w/ MCI, UUnet, AGIS, etc...
1) Announce *your own* routes more specifically. This may lose you ANS connectivity, though.
And Sprint, and anyone else that filters small specifics.
Again, not if you leave the /23 or /16s in place... Then you just revert to the pre-action situation. It is again important to note that if your announcement would be blocked by mask length policies, then the evil-isp would be as well.
2) Announce *their* routes more specifically.
Ouch, that's playing as dirty as them. Can't recommend it unless it's life or death...
I took that step last night, and was advised to remove it by those more in tune with legal issues. I guess it's not considered "nice" to sink to the same level as your attacker, and play dirty. :-}
Aiyeee.
3) You can post to NANOG and other lists in an attempt to embarrass/ get someone who knows the jokers to poke them.
I recommend this, show traceroutes, RR entries, InterNIC assignments, routing table dumps, and state the problem clearly. You can bet the appropriate folks will poke them. In summary, whatever they do to hit you, do for yourself in self-defense. Don't advertise their networks, just advertise your networks as specifically as needed. Continue to raise the ante by involving more appropriate folks, and provide specific documentation to those involved of what happened when. It sounds like a war to me. Try to find middle ground with them, there must be SOME reason they are after your cidr space. Perhaps you can negotiate a fix? I doubt it will be to long before a standard as-path list looks like this: .... ip as-path 10 deny EVIL-ISP1-AS ip as-path 10 deny EVIL-ISP2-AS ip as-path 10 deny EVIL-ISP3-AS .... In this age of global routing, with no central body, politicking and negotiation are your best tools for solution. There's no overseeing body to go to. You can gain allies, but it's up to you. Good luck, and count most folks here as allies. $.02 -alan
Hi Matt,
In this case, the very first thing you should probably do is to start announcing the more specific /24s to match their advertisements! Depending on AS-PATH length (how various nets hear your announcements vs. theirs) this may solve the immediate problem, allowing you to hunt them down and kill them at your leisure.
The downside to this is that we go from advertising /16's out, to advertising a fleet of /24's out, most of which would be filtered by Sprint's ever-lovin' CIDR-forcing wall.
If your more specific networks are filtered, then wouldn't the evil ISP's be filtered as well?
This would be a large problem only if you gain transit from Sprint....
Bingo. We buy transit from Sprint.
I agree with Sprint, and Sean, but in this case it pretty much makes it hard for us to force the issue by dropping to the same or smaller sized announcement.
Well, I'm not sure that the two entities can be put in the same sentence any more, but you can always leave the less specific /16 in there while you attempt to advertise the more speciic.
Good point; I'd forgotten you can have both advertised, and those that hear the /16, and no /24's will honor it, those that hear the /24's get to worry about the weights independantly of the /16's advertisement.
Good thought, though! Even if it does result in going from 2 /16 announcements to 512 /24 announcements in the process, growing the routing tables, and generally making everyone else unhappy as well.
I'd rather have happy workable customers and an unhappy community in the short run, than unhappy unworkable customers and a happy community.
Agreed; if this were something I felt would be resolved within 24 hours, it wouldn't even really be worth mentioning. When it starts moving on 48 hours, I start worrying about whether or not we're going to start showing up in the Top 50 lists. :-)
I think your letter will raise the awareness of this kind of problem. Of course we all know it's possible, but it's not a problem that we've had to deal with on a malicious level.
? I do assume that there's no doubt the evil-isp is doing this maliciously?
This is the third time they've done this. The first two times we chalked up to ignorance and stupidity. This time, though, we're not as willing to give them the benefit of the doubt.
*sigh* There really MUST be some nice way of handling lame ISP's like this.
One thing you could do is coordinate with largerish ISPs to filter the incorect network from the affected peering sessions. While this is a stopgap fix, and not one to be repeated, I don't think you'd have problems getting it done w/ MCI, UUnet, AGIS, etc...
1) Announce *your own* routes more specifically. This may lose you ANS connectivity, though.
And Sprint, and anyone else that filters small specifics.
Again, not if you leave the /23 or /16s in place... Then you just revert to the pre-action situation. It is again important to note that if your announcement would be blocked by mask length policies, then the evil-isp would be as well.
Not since we connect through sprint, and they don't. :-(
2) Announce *their* routes more specifically.
Ouch, that's playing as dirty as them. Can't recommend it unless it's life or death...
I know. Tempting though it is...
I took that step last night, and was advised to remove it by those more in tune with legal issues. I guess it's not considered "nice" to sink to the same level as your attacker, and play dirty. :-}
Aiyeee.
In thinking about it, I realized I didn't want to be the one listed as having escalated the cold war prefix race resulting in a flood of GIFS illustrating the death of the net. :-)
3) You can post to NANOG and other lists in an attempt to embarrass/ get someone who knows the jokers to poke them.
I recommend this, show traceroutes, RR entries, InterNIC assignments, routing table dumps, and state the problem clearly. You can bet the appropriate folks will poke them.
We did this last time, in complaining to MCI, their upstream provider, and MCI responded in record time, putting in a temporary filter for those blocks in less than 36 hours. That helped for about 30 seconds, before we found that they then announced the same blocks through a second connection which hadn't shown up as a path previously when we did a 'show ip bgp 205.158.193.0 255.255.255.0 l' With the advent of more and more multihomed networks, there's more and more paths that need to be filtered to stop an invalid announcement like this; perhaps the concept of a routing database that can only be updated by authoratative contacts isn't that unreasonable anymore. Gone are the days when you can just trust that everyone else will "do the right thing" in maintaining the well-being of the net. Just as the Guardian project came to the InterNIC, perhaps a similar check-and-verify proceedure needs to be put in place before new routing announcements are added or believed. I miss the older, more democratic days of the net, but it seems the overall level of knowledge and skill is dropping, forcing more and more levels of checks and balances to prevent abuse either through stupidity and ignorance, or malicious intent.
In summary, whatever they do to hit you, do for yourself in self-defense. Don't advertise their networks, just advertise your networks as specifically as needed. Continue to raise the ante by involving more appropriate folks, and provide specific documentation to those involved of what happened when. It sounds like a war to me. Try to find middle ground with them, there must be SOME reason they are after your cidr space. Perhaps you can negotiate a fix?
I doubt it will be to long before a standard as-path list looks like this:
.... ip as-path 10 deny EVIL-ISP1-AS ip as-path 10 deny EVIL-ISP2-AS ip as-path 10 deny EVIL-ISP3-AS ....
In this age of global routing, with no central body, politicking and negotiation are your best tools for solution. There's no overseeing body to go to. You can gain allies, but it's up to you. Good luck, and count most folks here as allies.
$.02
-alan
Thanks! It's a new ballgame for geeks and engineers more at home in a telco closet hunched over a laptop cabled into a console port; I'm not as familiar with politics and the subtle tactics of diplomacy, negotiation, and power brokering. I can see it's time I started learning some new tricks, and prepared for the new way of doing business on the net. Thanks again for the support. I'm sorry to see that the days of simple trust may be coming to a close, but we've all invested money into this creation, and we can't really afford to let others bring it down, whether by SYN attacks, domain name theivery, or IP theivery. Matt
Hi Matt,
In this case, the very first thing you should probably do is to start announcing the more specific /24s to match their advertisements! Depending on AS-PATH length (how various nets hear your announcements vs. theirs) this may solve the immediate problem, allowing you to hunt them down and kill them at your leisure.
The downside to this is that we go from advertising /16's out, to advertising a fleet of /24's out, most of which would be filtered by Sprint's ever-lovin' CIDR-forcing wall.
If your more specific networks are filtered, then wouldn't the evil ISP's be filtered as well?
This would be a large problem only if you gain transit from Sprint....
Bingo. We buy transit from Sprint.
It's the other way around: You are safe. Because you buy transit from Sprint, Sprint WILL hear your more specifics. Sprint will NOT hear more specifics from peers, customers of peers, customers of customers of peers, etc...
I agree with Sprint, and Sean, but in this case it pretty much makes it hard for us to force the issue by dropping to the same or smaller sized announcement.
Well, I'm not sure that the two entities can be put in the same sentence any more, but you can always leave the less specific /16 in there while you attempt to advertise the more speciic.
Good point; I'd forgotten you can have both advertised, and those that hear the /16, and no /24's will honor it, those that hear the /24's get to worry about the weights independantly of the /16's advertisement.
Specificity wins.
Matt
Avi
On Sat, 28 Sep 1996, Matthew Petach wrote:
I think your letter will raise the awareness of this kind of problem. Of course we all know it's possible, but it's not a problem that we've had to deal with on a malicious level.
? I do assume that there's no doubt the evil-isp is doing this maliciously?
This is the third time they've done this. The first two times we chalked up to ignorance and stupidity.
This time, though, we're not as willing to give them the benefit of the doubt.
I don't believe you. If you were as confident as you say you are that this is an evil ISP you would have just said: Evilnet Inc. is blackholing my routes. I've sent mail to techie@evilnet.inc, bigboss@evilnet.inc and chambermaid@evilnet.inc and nobody returns my mail. I phoned them at 1-888-555-2222 and left voicemail, I faxed them at 1-888-555-1111 and I don't get any response. In this way you accomplish the following: 1) clear identification of the problem, i.e. blackholed routes 2) clear identification of who seems to be causing the problem 3) clear identification of the contact means that you tried and the results or lack thereof obtained. As a result, somebody who happens to know that Joe Bloe is the techie at EvilNet can call Joe at home and say, "Hey Joe, did you know that so-and-so doesn't like what you are doing and can't get a hold of you by email or telephone. Maybe you better fix this...". Or it could be Evilnet's upstream who contacts them. Or somebody could email you Evilnet's secret "human" NOC phone number, or whatever.
We did this last time, in complaining to MCI, their upstream provider, and MCI responded in record time, putting in a temporary filter for those blocks in less than 36 hours.
That helped for about 30 seconds, before we found that they then announced the same blocks through a second connection which hadn't shown up as a path previously when we did a 'show ip bgp 205.158.193.0 255.255.255.0 l'
Trying to solve a social problem with technology often results in this kind of thing.
I miss the older, more democratic days of the net, but it seems the overall level of knowledge and skill is dropping, forcing more and more levels of checks and balances to prevent abuse either through stupidity and ignorance, or malicious intent.
I think you are jumping to conclusions here by assuming it is due to stupidity, ignorance or malicious intent. I strongly suspect that it is due to lack of information and work overload. Lack of information is subtly but significantly different from stupidity and ignorance and you yourself are contributing to Evilnet's lack of information by withholding important information about the problem. Shine the light of day on the problem and it will soon clear up. Throw all the relevant information into the "public" NANOG mailing list pool and numerous avenues for action will open up. Michael Dillon - ISP & Internet Consulting Memra Software Inc. - Fax: +1-604-546-3049 http://www.memra.com - E-mail: michael@memra.com
On Sat, 28 Sep 1996, Matthew Petach wrote:
I think your letter will raise the awareness of this kind of problem. Of course we all know it's possible, but it's not a problem that we've had to deal with on a malicious level.
? I do assume that there's no doubt the evil-isp is doing this maliciously?
This is the third time they've done this. The first two times we chalked up to ignorance and stupidity.
This time, though, we're not as willing to give them the benefit of the doubt.
I don't believe you. If you were as confident as you say you are that this is an evil ISP you would have just said:
*grin* Well, since this is the first time sending a report of something like this to NANOG, I didn't know there was a form I was supposed to fill out first. :-)
Evilnet Inc. is blackholing my routes. I've sent mail to techie@evilnet.inc, bigboss@evilnet.inc and chambermaid@evilnet.inc and nobody returns my mail. I phoned them at 1-888-555-2222 and left voicemail, I faxed them at 1-888-555-1111 and I don't get any response.
JVNC.net is blackholing my routes. We've called their NOC, their techsupport, and everyone listed in the whois listing.
In this way you accomplish the following:
1) clear identification of the problem, i.e. blackholed routes
2) clear identification of who seems to be causing the problem
3) clear identification of the contact means that you tried and the results or lack thereof obtained.
As a result, somebody who happens to know that Joe Bloe is the techie at EvilNet can call Joe at home and say, "Hey Joe, did you know that so-and-so doesn't like what you are doing and can't get a hold of you by email or telephone. Maybe you better fix this...". Or it could be Evilnet's upstream who contacts them. Or somebody could email you Evilnet's secret "human" NOC phone number, or whatever.
And if they want examples of the problems, here's a traceroute from Stanford University, where I happen to have an account, over to one of our customers. nyx.Stanford.EDU> traceroute pdm.xo.com traceroute to pdm.xo.com (205.158.193.246): 1-30 hops, 38 byte packets 1 ceras-gateway.Stanford.EDU (36.190.0.1) 2.69 ms 1.60 ms 1.59 ms 2 Core-gateway.Stanford.EDU (171.64.2.1) 3.16 ms 2.17 ms 2.4 ms 3 SUNet-Gateway.Stanford.EDU (171.64.1.34) 3.29 ms 2.62 ms 2.57 ms 4 su-pr1.bbnplanet.net (198.31.10.3) 2.19 ms 2.26 ms 2.50 ms 5 paloalto-mci.bbnplanet.net (131.119.0.202) 3.16 ms 3.46 ms 3.12 ms 6 borderx1-hssi2-0.SanFrancisco.mci.net (204.70.158.101) 115 ms 68.5 ms 10.0 ms 7 border3-fddi-0.SanFrancisco.mci.net (204.70.2.163) 4.78 ms 5.17 ms 6.55 ms 8 santa-clara.west.cix.net (149.20.64.1) 107 ms 205 ms 224 ms 9 jvnc-cix.west.cix.net (149.20.6.2) 86.2 ms (ttl=243!) 97.1 ms (ttl=243!) 90.5 ms (ttl=243!) 10 unclesam-ser1.jvnc.net (130.94.15.249) 86.5 ms (ttl=244!) 84.6 ms (ttl=244!) 95.1 ms (ttl=244!) 11 liberty-ser3-2.jvnc.net (130.94.11.250) 160 ms 90.3 ms 146 ms 12 bcn-hq.jvnc.net (130.94.40.253) 90.7 ms 135 ms 96.0 ms 13 204.70.179.110 (204.70.179.110) 85.0 ms (ttl=20!) 85.7 ms (ttl=20!) 86.8 ms (ttl=20!) 14 jc-bcn.jvnc.net (130.94.52.2) 93.4 ms (ttl=19!) 93.6 ms (ttl=19!) 96.6 ms (ttl=19!) 15 dialogic-gateway.jvnc.net (130.94.56.50) 108 ms (ttl=18!) 105 ms (ttl=18!) 96.5 ms (ttl=18!) 16 146.152.224.250 (146.152.224.250) 103 ms (ttl=17!) 113 ms (ttl=17!) 106 ms (ttl=17!) 17 146.152.241.249 (146.152.241.249) 175 ms (ttl=16!) 185 ms (ttl=16!) 181 ms (ttl=16!) 18 146.152.160.1 (146.152.160.1) 177 ms (ttl=49!) 195 ms (ttl=49!) 183 ms (ttl=49!) 19 146.152.160.249 (146.152.160.249) 174 ms (ttl=16!) 175 ms (ttl=16!) 179 ms (ttl=16!) 20 146.152.160.1 (146.152.160.1) 179 ms (ttl=49!) 176 ms (ttl=49!) 181 ms (ttl=49!) 21 146.152.160.249 (146.152.160.249) 177 ms (ttl=16!) 177 ms (ttl=16!) 184 ms (ttl=16!) 22 146.152.160.1 (146.152.160.1) 180 ms (ttl=49!) 180 ms (ttl=49!) 193 ms (ttl=49!) 23 146.152.160.249 (146.152.160.249) 188 ms (ttl=16!) 192 ms (ttl=16!) 199 ms (ttl=16!) 24 146.152.160.1 (146.152.160.1) 180 ms (ttl=49!) 182 ms (ttl=49!) 192 ms (ttl=49!) 25 146.152.160.249 (146.152.160.249) 190 ms (ttl=16!) 183 ms (ttl=16!) 197 ms (ttl=16!) 26 146.152.160.1 (146.152.160.1) 218 ms (ttl=49!) 189 ms (ttl=49!) 182 ms (ttl=49!) 27 146.152.160.249 (146.152.160.249) 186 ms (ttl=16!) 199 ms (ttl=16!) 186 ms (ttl=16!) 28 146.152.160.1 (146.152.160.1) 187 ms (ttl=49!) 188 ms (ttl=49!) 188 ms (ttl=49!) 29 146.152.160.249 (146.152.160.249) 190 ms (ttl=16!) 199 ms (ttl=16!) 213 ms (ttl=16!) 30 146.152.160.1 (146.152.160.1) 186 ms (ttl=49!) 189 ms (ttl=49!) 187 ms (ttl=49!) nyx.Stanford.EDU> This is the same traceroute AFTER we put in the more specific routes, to override their bogus announcement: nyx.Stanford.EDU> traceroute 205.158.193.82 traceroute to vn.com (205.158.193.82): 1-30 hops, 38 byte packets 1 ceras-gateway.Stanford.EDU (36.190.0.1) 2.13 ms 1.61 ms 2.13 ms 2 Core-gateway.Stanford.EDU (171.64.1.1) 2.74 ms 1.62 ms 1.82 ms 3 SUNet-Gateway.Stanford.EDU (171.64.1.34) 2.80 ms 2.23 ms 2.13 ms 4 su-pr1.bbnplanet.net (198.31.10.3) 6.47 ms 2.16 ms 1.79 ms 5 paloalto-br2.bbnplanet.net (4.0.1.90) 3.54 ms 3.14 ms 2.64 ms 6 sanjose1-br3.bbnplanet.net (4.0.1.14) 6.65 ms 3.19 ms 4.1 ms 7 mae-west.agis.net (198.32.136.21) 299 ms 21.8 ms 7.45 ms 8 santaclara.santanap.agis.net (206.62.13.249) 9.60 ms (ttl=246!) 7.55 ms (ttl=246!) 9.30 ms (ttl=246!) 9 internex.santanap.agis.net (206.62.13.18) 6.81 ms (ttl=249!) 7.32 ms (ttl=249!) 12.0 ms (ttl=249!) 10 area-1-rtr-fddi.InterNex.Net (205.158.0.2) 9.31 ms (ttl=248!) 13.8 ms (ttl=248!) 13.0 ms (ttl=248!) 11 milpitas01-S0.POP.InterNex.Net (205.158.2.26) 208 ms (ttl=247!) 38.9 ms (ttl=247!) 87.3 ms (ttl=247!) 12 Milpitas01-Max1.POP.InterNex.Net (205.158.3.68) 64.1 ms (ttl=55!) 44.8 ms (ttl=55!) 24.2 ms (ttl=55!) 13 Milpitas01-rtr.POP.InterNex.Net (205.158.3.65) 20.9 ms (ttl=247!) 18.0 ms (ttl=247!) 35.5 ms (ttl=247!) 14 Milpitas01-Max1.POP.InterNex.Net (205.158.3.68) 28.5 ms (ttl=55!) 23.0 ms (ttl=55!) 42.0 ms (ttl=55!) [ ... ] nyx.Stanford.EDU> (It's a dialup customer, so it makes it to the Max, and then bounces back and forth to the cisco and back to the max a bunch of times, but that's where it SHOULD be, within OUR network)
We did this last time, in complaining to MCI, their upstream provider, and MCI responded in record time, putting in a temporary filter for those blocks in less than 36 hours.
That helped for about 30 seconds, before we found that they then announced the same blocks through a second connection which hadn't shown up as a path previously when we did a 'show ip bgp 205.158.193.0 255.255.255.0 l'
Trying to solve a social problem with technology often results in this kind of thing.
It's a bit of an uphill road for an old network engineer to shift from technology solutions to social engineering solutions, but I think I'll figure out the requirements soon enough, given incentives like this.
I miss the older, more democratic days of the net, but it seems the overall level of knowledge and skill is dropping, forcing more and more levels of checks and balances to prevent abuse either through stupidity and ignorance, or malicious intent.
I think you are jumping to conclusions here by assuming it is due to stupidity, ignorance or malicious intent. I strongly suspect that it is due to lack of information and work overload. Lack of information is subtly but significantly different from stupidity and ignorance and you yourself are contributing to Evilnet's lack of information by withholding important information about the problem.
Shine the light of day on the problem and it will soon clear up. Throw all the relevant information into the "public" NANOG mailing list pool and numerous avenues for action will open up.
Hm. Well, I can list the whois entry for jvnc.net to list the phone numbers of the contact people. Is there a master list of AS #'s with ROUTING contacts, rather than the fuzzy admin type contacts that get listed in whois? Right about now, I'm still searching for information myself, but I'll put as much forward as I can.
Michael Dillon - ISP & Internet Consulting Memra Software Inc. - Fax: +1-604-546-3049 http://www.memra.com - E-mail: michael@memra.com
Matt Petach still learning all the nuances of this social troubleshooting...
Hi Matt,
Is there a master list of AS #'s with ROUTING contacts, rather than the fuzzy admin type contacts that get listed in whois? Right about now, I'm still searching for information myself, but I'll put as much forward as I can.
Obviously there is the AS info at the internic, plus various registration entries for various networks in the various RRs. One thing that those of us interested (sean@dra,loco/jonh@isi,etc...) in inter-provider communication have asked for is a common place to air connectivity and routing problems. Craig Labovitz@Merit is doing a darned good job of making this a reality. Another thing we've asked for is a more real time updated contact list. However, having just left a company where I was the contact for most all routing, I don't see an easy way for my name to be swept away. So, the answer to your question is not really, but it would be nice if there was a more highly utilized central authority for such things. I liked what chris@nap.net said about self regulation and all. I trust the political types take it to heart. -alan
In this case, the very first thing you should probably do is to start announcing the more specific /24s to match their advertisements! Depending on AS-PATH length (how various nets hear your announcements vs. theirs) this may solve the immediate problem, allowing you to hunt them down and kill them at your leisure.
The downside to this is that we go from advertising /16's out, to advertising a fleet of /24's out, most of which would be filtered by Sprint's ever-lovin' CIDR-forcing wall. I agree with Sprint, and Sean, but in this case it pretty much makes it hard for us to force the issue by dropping to the same or smaller sized announcement.
Good thought, though! Even if it does result in going from 2 /16 announcements to 512 /24 announcements in the process, growing the routing tables, and generally making everyone else unhappy as well.
Only advertise the /24s that they're announcing of yours. And if you need to get them into Sprint, see if a multi- homed Sprint customer will temporarily shove them into Sprint and static them back to you via another provider/connection.
*sigh* There really MUST be some nice way of handling lame ISP's like this.
1) Announce *your own* routes more specifically. This may lose you ANS connectivity, though.
I meant ANS connectivity because of RADB issues, but yes, anyone who filters small announcements in your space won't see you.
I took that step last night, and was advised to remove it by those more in tune with legal issues. I guess it's not considered "nice" to sink to the same level as your attacker, and play dirty. :-}
No, but if it went on for 12 hours, I very well might do so.
Avi
Again, my thanks for you feedback and support!
Matt Petach
Sure, good luck. And if you're going for the shunning effect, tell us all who it is that you're having trouble with. Avi
In message <199609281903.PAA06201@netaxs.com>, Avi Freedman writes:
1) Announce *your own* routes more specifically. This may lose you ANS connectivity, though.
I meant ANS connectivity because of RADB issues, but yes, anyone who filters small announcements in your space won't see you.
Avi, If there is a route object in the RADB for the /24 we will set policy according to the origin AS of the RO. If you can't get rid of a bogus RADB RO, ask us we can create exceptions for AS based policy on a per prefix basis. If there is no RO, we won't listen to the /24 so we won't hear the bogon, just the /16. This is temporarily moot since we have 2 routers in AS1673 that can't handle the policy filters. The bottom line is the IRR helps keep connectivity in the face of this sort of problem where you are claiming it hurts. Curtis
Curtis,
If there is a route object in the RADB for the /24 we will set policy according to the origin AS of the RO. If you can't get rid of a bogus RADB RO, ask us we can create exceptions for AS based policy on a per prefix basis. If there is no RO, we won't listen to the /24 so we won't hear the bogon, just the /16. This is temporarily moot since we have 2 routers in AS1673 that can't handle the policy filters.
The bottom line is the IRR helps keep connectivity in the face of this sort of problem where you are claiming it hurts.
Is there any way of determining which routes ANS are filtering out (other than noticing customer X can't reach you all of a sudden), or trying to filter out as is the case currently. How about a daily mailing which would encourage people to register routes in the appropriate IRR as well as allow us to preempt our slower learning BGP downstreams into warning them they are about to have a problem when they start using a new block they haven't registered a routing policy (yep, they *should* know this by now). Or a web site. Even if it just does the equivalent of the Digex l/g. Your NOC are normally v. helpful, but I'm sure it would save you some man hours. Alex Bligh Xara Networks
In message <199609302131.WAA08565@diamond.xara.net>, "Alex.Bligh" writes:
Curtis,
If there is a route object in the RADB for the /24 we will set policy according to the origin AS of the RO. If you can't get rid of a bogus RADB RO, ask us we can create exceptions for AS based policy on a per prefix basis. If there is no RO, we won't listen to the /24 so we won't hear the bogon, just the /16. This is temporarily moot since we have 2 routers in AS1673 that can't handle the policy filters.
The bottom line is the IRR helps keep connectivity in the face of this sort of problem where you are claiming it hurts.
Is there any way of determining which routes ANS are filtering out (other than noticing customer X can't reach you all of a sudden), or trying to filter out as is the case currently.
"whois -h whois.ans.net as690" or "whois -h whois.ra.net as690". The RA copy is updated each time we deploy a set of config files. Then look for route objects.
How about a daily mailing which would encourage people to register routes in the appropriate IRR as well as allow us to preempt our slower learning BGP downstreams into warning them they are about to have a problem when they start using a new block they haven't registered a routing policy (yep, they *should* know this by now).
I've tried to code stuff like this but I'd prefer not to send mail to people asking them to register something leaking from and aggregate or when an aggregate would better solve the problem. It seems that most people that know how to form an aggregate also know enough to register the aggregate. We haven't done anything because of lack of time and wanting to wait until we had automated something that could give quality advice rather than spray mail to a few thousand unregistered /24s asking them to register when we should be asking their provider to aggregate them.
Or a web site. Even if it just does the equivalent of the Digex l/g.
Your NOC are normally v. helpful, but I'm sure it would save you some man hours.
Alex Bligh Xara Networks
Good suggestions. Thanks, Curtis
In message <199609281903.PAA06201@netaxs.com>, Avi Freedman writes:
1) Announce *your own* routes more specifically. This may lose you ANS connectivity, though.
I meant ANS connectivity because of RADB issues, but yes, anyone who filters small announcements in your space won't see you.
Avi,
If there is a route object in the RADB for the /24 we will set policy according to the origin AS of the RO. If you can't get rid of a bogus RADB RO, ask us we can create exceptions for AS based policy on a per prefix basis. If there is no RO, we won't listen to the /24 so we won't hear the bogon, just the /16. This is temporarily moot since we
Yes, I overlooked this; ANS and Sprint would both not be affected by bogus announcements; ANS in any space and Sprint in >= 205/8 space (or wherever the Sean-filters start).
have 2 routers in AS1673 that can't handle the policy filters.
The bottom line is the IRR helps keep connectivity in the face of this sort of problem where you are claiming it hurts.
Yes... My thinking was fuzzy, you are of course correct.
Curtis
Avi
In message <199609281508.IAA02192@falcon.netflight.com>, Matthew Petach writes:
Hi!
I'm going to ask the rest of the NANOG community for their thoughts/opinions on a problem that's been plaguing us periodically that we haven't been able to find a satisfactory solution for yet.
There's an ISP back on the East Coast that has been periodically advertising more specific routes for /24's out of our CIDR blocks and black-holing the traffic within their network.
We've called all the listed numbers for their technical, admin, billing, and any other contacts we can find, and haven't been able to reach a human; we've left messages of various levels of nastyness, from very sugary on up to vaguely threatening. In every case, including the current one, it's been more than 24 hours, and they still haven't made any response to the problem; in fact, I just got paged by our NOC early this morning informing me they've stolen another one of our /24's.
As you can well imagine, all the customers on those blocks are _very_ unhappy. Each time this happens, we end up with dissatisfied customers, many of whom leave, deciding that we're too unstable, and can't provide quality network connectivity, even though to the best of my knowledge, there's nothing we can do to prevent these people from stealing our blocks.
My question to the NANOG community is twofold and simple: Am I overlooking some solution that would allow us to 'negate' their advertisement of our blocks (205.159.193.0/24 and 207.88.102.0/24 in this case) and secondly, is there a formal process within the community to seek recompense, or formal action against a clueless and net-unfriendly ISP, perhaps one as simple as the net equivalent of Mennonite 'shunning'?
Or are we simply out of luck, and have to simply tell our customers "Sorry, everyone is at the mercy of the morons who can steal IP blocks simply by advertising more specific routes with higher weights?"
It's getting really tempting to advertise the networks they have their nameservers on from *our* network with a weight of 65535, just to get them to call us back. :-( :-(
Anyhow, enough frustrated venting, I *am* very interested in what the community feels is the best policy to follow in situations like this.
Thanks again!
Matt Petach Network Engineer (writing from home)
A good solution would be for providers to only accept routes registered in a routing database (the IRR) from those authorized to send them with hierarchical authorization within the database (as implemented by RIPE) and strong authentication (PGP as implemented by the RA) and top level authorization based on IANA or delegated address registry assignments. But you've heard this before. The best any one provider can do is to accurately populate the IRR and if possible (based on the limitations of their routers) put the IRR data into use in defining filters. Curtis
participants (7)
-
Alan Hannan -
Alex.Bligh -
Avi Freedman -
Curtis Villamizar -
Matthew Petach -
Michael Dillon -
randyï¼ psg.com