RE: YouTube IP Hijacking
Which means that, by advertising routes more specific than the ones they are poisoning, it may well be possible to restore universal connectivity to YouTube.
-----Original Message----- From: Michael Smith [mailto:msmith@internap.com] Sent: Sunday, February 24, 2008 1:23 PM To: neil.fenemor@fx.net.nz; Tomas L. Byrnes Cc: will@harg.net; nanog@merit.edu Subject: Re: YouTube IP Hijacking
Exactly... They inadvertently made the details of their oppression more readily apparent...
----- Original Message ----- From: owner-nanog@merit.edu <owner-nanog@merit.edu> To: Tomas L. Byrnes <tomb@byrneit.net> Cc: Will Hargrave <will@harg.net>; nanog@merit.edu <nanog@merit.edu> Sent: Sun Feb 24 16:00:35 2008 Subject: Re: YouTube IP Hijacking
While they are deliberately blocking Youtube nationally, I suspect the wider issue has no malice, and is a case of poorly constructed/ implemented outbound policies on their part, and poorly constructed/ implemented inbound polices on their upstreams part.
On 25/02/2008, at 9:49 AM, Tomas L. Byrnes wrote:
Pakistan is deliberately blocking Youtube.
http://politics.slashdot.org/article.pl?sid=08/02/24/1628213
Maybe we should all block Pakistan.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]
Of Will Hargrave Sent: Sunday, February 24, 2008 12:39 PM To: nanog@nanog.org Subject: Re: YouTube IP Hijacking
Sargun Dhillon wrote:
So, it seems that youtube's ip block has been hijacked by a more specific prefix being advertised. This is a case of IP hijacking, not case of DNS poisoning, youtube engineers doing something stupid, etc. For people that don't know. The router will try to get the most specific prefix. This is by design, not by accident.
You are making the assumption of malice when the more
On Behalf likely cause is
one of accident on the part of probably stressed NOC staff at 17557.
They probably have that /24 going to a gateway walled garden box which replies with a site saying 'we have banned this', and that /24 route is leaking outside of their AS via PCCW due to dodgy filters/communities.
Will
Neil Fenemor FX Networks
On Sun Feb 24, 2008 at 01:49:00PM -0800, Tomas L. Byrnes wrote:
Which means that, by advertising routes more specific than the ones they are poisoning, it may well be possible to restore universal connectivity to YouTube.
Well, if you can get them in there.... Youtube tried that, to restore service to the rest of the world, and the announcements didn't propogate. Simon
I figured as much, but it was worth a try. Which touches on the earlier discussion of the null routing of /32s advertised by a special AS (as a means of black-holing DDOS traffic). It seems to me that a more immediately germane matter regarding BGP route propagation is prevention of hijacking of critical routes. Perhaps certain ASes that are considered "high priority", like Google, YouTube, Yahoo, MS (at least their update servers), can be trusted to propagate routes that are not aggregated/filtered, so as to give them control over their reachability and immunity to longer-prefix hijacking (especially problematic with things like MS update sites).
-----Original Message----- From: Simon Lockhart [mailto:simon@slimey.org] Sent: Sunday, February 24, 2008 2:07 PM To: Tomas L. Byrnes Cc: Michael Smith; neil.fenemor@fx.net.nz; will@harg.net; nanog@merit.edu Subject: Re: YouTube IP Hijacking
On Sun Feb 24, 2008 at 01:49:00PM -0800, Tomas L. Byrnes wrote:
Which means that, by advertising routes more specific than the ones they are poisoning, it may well be possible to restore universal connectivity to YouTube.
Well, if you can get them in there.... Youtube tried that, to restore service to the rest of the world, and the announcements didn't propogate.
Simon
Tomas L. Byrnes wrote:
Perhaps certain ASes that are considered "high priority", like Google, YouTube, Yahoo, MS (at least their update servers), can be trusted to propagate routes that are not aggregated/filtered, so as to give them control over their reachability and immunity to longer-prefix hijacking (especially problematic with things like MS update sites).
Not to stir up a huge debate here, but if I were a day trader, I could live without YouTube for a day, but not e*trade or Ameritrade as it would be my livelihood. If I were an eBay seller, why would I care about YouTube? You get the idea. What makes Google, YouTube, Yahoo, MS, etc more important? More importantly, why is PCCW not prefix filtering their downstreams? Certainly AS17557 cannot be trusted without a filter. Randy
-----Original Message----- From: Simon Lockhart [mailto:simon@slimey.org] Sent: Sunday, February 24, 2008 2:07 PM To: Tomas L. Byrnes Cc: Michael Smith; neil.fenemor@fx.net.nz; will@harg.net; nanog@merit.edu Subject: Re: YouTube IP Hijacking
On Sun Feb 24, 2008 at 01:49:00PM -0800, Tomas L. Byrnes wrote:
Which means that, by advertising routes more specific than the ones they are poisoning, it may well be possible to restore universal connectivity to YouTube.
Well, if you can get them in there.... Youtube tried that, to restore service to the rest of the world, and the announcements didn't propogate.
Simon
I'm sure we can all find a list of "critical infrastructure" ASes that could be trusted to peer via the "high priority" AS. I'd say that the criteria should be: 1: Hosted at a Tier 1 provider. 2: Within a jurisdiction where North American operators have a good chance of having the law on their side in case of any network outage caused by the entity. 3: Considered highly competent technically. 4: With state of the art security and operations. OTOH: I would say that, until today, those who advocate not engaging in any kind of ethnic or political profiling would have considered 17557, as a national telco, a trusted route source.
-----Original Message----- From: Randy Epstein [mailto:repstein@chello.at] Sent: Sunday, February 24, 2008 4:15 PM To: Tomas L. Byrnes; 'Simon Lockhart' Cc: 'Michael Smith'; neil.fenemor@fx.net.nz; will@harg.net; nanog@merit.edu Subject: RE: YouTube IP Hijacking
Tomas L. Byrnes wrote:
Perhaps certain ASes that are considered "high priority", like Google, YouTube, Yahoo, MS (at least their update servers), can be trusted to propagate routes that are not aggregated/filtered, so as to give them control over their reachability and immunity to longer-prefix hijacking (especially problematic with things like MS update sites).
Not to stir up a huge debate here, but if I were a day trader, I could live without YouTube for a day, but not e*trade or Ameritrade as it would be my livelihood. If I were an eBay seller, why would I care about YouTube? You get the idea. What makes Google, YouTube, Yahoo, MS, etc more important?
More importantly, why is PCCW not prefix filtering their downstreams? Certainly AS17557 cannot be trusted without a filter.
Randy
-----Original Message----- From: Simon Lockhart [mailto:simon@slimey.org] Sent: Sunday, February 24, 2008 2:07 PM To: Tomas L. Byrnes Cc: Michael Smith; neil.fenemor@fx.net.nz; will@harg.net; nanog@merit.edu Subject: Re: YouTube IP Hijacking
Which means that, by advertising routes more specific
On Sun Feb 24, 2008 at 01:49:00PM -0800, Tomas L. Byrnes wrote: than the ones
they are poisoning, it may well be possible to restore universal connectivity to YouTube.
Well, if you can get them in there.... Youtube tried that, to restore service to the rest of the world, and the announcements didn't propogate.
Simon
On Feb 24, 2008, at 7:36 PM, Tomas L. Byrnes wrote:
I'm sure we can all find a list of "critical infrastructure" ASes that could be trusted to peer via the "high priority" AS. I'd say that the criteria should be:
1: Hosted at a Tier 1 provider.
That is a silly requirement. (I am sorry, I tried hard to find a nicer way to say this, but I really feel strongly about this.)
2: Within a jurisdiction where North American operators have a good chance of having the law on their side in case of any network outage caused by the entity.
This is also a bit strange. Do your users never attach to a host outside the USofA?
3: Considered highly competent technically.
Here we agree.
4: With state of the art security and operations.
I think we agree, but I wouldn't have said it like that. -- TTFN, patrick
OTOH: I would say that, until today, those who advocate not engaging in any kind of ethnic or political profiling would have considered 17557, as a national telco, a trusted route source.
-----Original Message----- From: Randy Epstein [mailto:repstein@chello.at] Sent: Sunday, February 24, 2008 4:15 PM To: Tomas L. Byrnes; 'Simon Lockhart' Cc: 'Michael Smith'; neil.fenemor@fx.net.nz; will@harg.net; nanog@merit.edu Subject: RE: YouTube IP Hijacking
Tomas L. Byrnes wrote:
Perhaps certain ASes that are considered "high priority", like Google, YouTube, Yahoo, MS (at least their update servers), can be trusted to propagate routes that are not aggregated/filtered, so as to give them control over their reachability and immunity to longer-prefix hijacking (especially problematic with things like MS update sites).
Not to stir up a huge debate here, but if I were a day trader, I could live without YouTube for a day, but not e*trade or Ameritrade as it would be my livelihood. If I were an eBay seller, why would I care about YouTube? You get the idea. What makes Google, YouTube, Yahoo, MS, etc more important?
More importantly, why is PCCW not prefix filtering their downstreams? Certainly AS17557 cannot be trusted without a filter.
Randy
-----Original Message----- From: Simon Lockhart [mailto:simon@slimey.org] Sent: Sunday, February 24, 2008 2:07 PM To: Tomas L. Byrnes Cc: Michael Smith; neil.fenemor@fx.net.nz; will@harg.net; nanog@merit.edu Subject: Re: YouTube IP Hijacking
Which means that, by advertising routes more specific
On Sun Feb 24, 2008 at 01:49:00PM -0800, Tomas L. Byrnes wrote: than the ones
they are poisoning, it may well be possible to restore universal connectivity to YouTube.
Well, if you can get them in there.... Youtube tried that, to restore service to the rest of the world, and the announcements didn't propogate.
Simon
This candidate list of requirements is for route sources that North American Operators should trust to propagate long prefix routes, nothing more, nothing less. In that context, some of your comments don't really make sense. Perhaps you might like to propose criteria you would find useful in setting a level of trust, or some alternative method to avoid a recurrence of a site that is widely visited being black holed through another ISP advertising a more specific route? Specifically: In place of item 1, what criteria would you propose for the route source? Item 2: in this context, is specific to the needs of North American Network Operators accepting long prefix routes. I am not advocating not accepting routes from the ROW, just not very specific ones. It's entirely possible for North American Operators to rely on law enforcement in say, the EU and Australia. Item 3: Glad we agree on something. Item 4: How would you have said it? I think it would be better to propose some constructive ideas as to how we can avoid what happened today from recurring, and also deal with the issue of hijacked IP space in general.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Patrick W. Gilmore Sent: Sunday, February 24, 2008 5:43 PM To: nanog@merit.edu Cc: Patrick W. Gilmore Subject: Re: YouTube IP Hijacking
On Feb 24, 2008, at 7:36 PM, Tomas L. Byrnes wrote:
I'm sure we can all find a list of "critical infrastructure" ASes that could be trusted to peer via the "high priority" AS. I'd say that the criteria should be:
1: Hosted at a Tier 1 provider.
That is a silly requirement.
(I am sorry, I tried hard to find a nicer way to say this, but I really feel strongly about this.)
2: Within a jurisdiction where North American operators have a good chance of having the law on their side in case of any network outage caused by the entity.
This is also a bit strange. Do your users never attach to a host outside the USofA?
3: Considered highly competent technically.
Here we agree.
4: With state of the art security and operations.
I think we agree, but I wouldn't have said it like that.
-- TTFN, patrick
OTOH: I would say that, until today, those who advocate not engaging in any kind of ethnic or political profiling would have considered 17557, as a national telco, a trusted route source.
-----Original Message----- From: Randy Epstein [mailto:repstein@chello.at] Sent: Sunday, February 24, 2008 4:15 PM To: Tomas L. Byrnes; 'Simon Lockhart' Cc: 'Michael Smith'; neil.fenemor@fx.net.nz; will@harg.net; nanog@merit.edu Subject: RE: YouTube IP Hijacking
Tomas L. Byrnes wrote:
Perhaps certain ASes that are considered "high priority", like Google, YouTube, Yahoo, MS (at least their update servers), can be trusted to propagate routes that are not aggregated/filtered, so as to give them control over their reachability and immunity to longer-prefix hijacking (especially problematic with things like MS update sites).
Not to stir up a huge debate here, but if I were a day trader, I could live without YouTube for a day, but not e*trade or Ameritrade as it would be my livelihood. If I were an eBay seller, why would I care about YouTube? You get the idea. What makes Google, YouTube, Yahoo, MS, etc more important?
More importantly, why is PCCW not prefix filtering their downstreams? Certainly AS17557 cannot be trusted without a filter.
Randy
-----Original Message----- From: Simon Lockhart [mailto:simon@slimey.org] Sent: Sunday, February 24, 2008 2:07 PM To: Tomas L. Byrnes Cc: Michael Smith; neil.fenemor@fx.net.nz; will@harg.net; nanog@merit.edu Subject: Re: YouTube IP Hijacking
Which means that, by advertising routes more specific
On Sun Feb 24, 2008 at 01:49:00PM -0800, Tomas L. Byrnes wrote: than the ones
they are poisoning, it may well be possible to restore universal connectivity to YouTube.
Well, if you can get them in there.... Youtube tried that, to restore service to the rest of the world, and the announcements didn't propogate.
Simon
This candidate list of requirements is for route sources that North American Operators should trust to propagate long prefix routes, nothing more, nothing less.
All operators already have some kind of criteria which they use to decide whether or not to trust a particular source of routes whether long prefixes or short. You are suggesting that these operators should give up this role to a trusted third party so that al North American network operators share fate in terms of BGP trust relationships. It seems that you feel this is an improvement since some network operators have very lax policies and trust people that they shouldn't. In that case, I wonder whether these operators would even bother joining such a shared-fate arrangement. But the big downside is for the operators who have carefully honed filtering policies and who are careful about who they trust. For them there is no upside to joining a shared-fate arrangement, and a potential downside if management decides that their internal BGP filtering practices can now be made more lax. And, of course, the shared fate arrangement is now a single point of failure and a very juicy target for bad guys to attack. The real solution to the YouTube issue is for people to pressure other network operators to raise their game and pay attention to how they manage their BGP trust relationships and filter announcements. In addition, more people need to get involved in information sharing arrangements like Routing Registries, MyASN, alert services and so on. None of these things create a single point of failure and some of them would be useful even if your Super AS is created. Because all of this activity is done by humans, even your Super AS can make mistakes so it would be bad for people to trust it just because it is big. Alert services, RRs, MyASN, etc., can protect against human failures whether in the Super AS or Pakistan Telecom.
Perhaps you might like to propose criteria you would find useful in setting a level of trust, or some alternative method to avoid a recurrence of a site that is widely visited being black holed through another ISP advertising a more specific route?
Haven't you noticed that the definition of "widely visited site" changes regularly, and often quite abruptly? How much traffic did YouTube get 3 years ago? Facebook? MySpace? There is no shortcut for eternal vigilance, i.e. manage your BGP relationships don't just configure and forget.
Item 2: in this context, is specific to the needs of North American Network Operators accepting long prefix routes. I am not advocating not accepting routes from the ROW, just not very specific ones. It's entirely possible for North American Operators to rely on law enforcement in say, the EU and Australia.
In case you hadn't noticed, there is no North American law enforcement agency and no North American courts and no North American laws outside of NAFTA. So I'm not sure what you are getting at here. Do you want to reopen NAFTA negotiations to include Internet peering?
I think it would be better to propose some constructive ideas as to how we can avoid what happened today from recurring, and also deal with the issue of hijacked IP space in general.
The tools and techniques are out there. All that is needed is for people to follow best practices. Seems to me that educational activity would be more productive than building castles in the sky. --Michael Dillon
On Mon, Feb 25, 2008 at 10:12:47AM -0000, michael.dillon@bt.com wrote:
In case you hadn't noticed, there is no North American law enforcement agency and no North American courts and no North American laws outside of NAFTA. So I'm not sure what you are getting at here. Do you want to reopen NAFTA negotiations to include Internet peering?
the law process within NAFTA is no more than a delay tactic used by various big business and big government to defer any real resolution to the problems. the laws of Canada, Mexico and the US are still largely seperate, and the laws of one do not necessarily follow in another. -- Jim Mercer jim@reptiles.org +971 55 410-5633 "I'm Prime Minister of Canada, I live here and I'm going to take a leak." - Lester Pearson in 1967, during a meeting between himself and President Lyndon Johnson, whose Secret Service detail had taken over Pearson's cottage retreat. At one point, a Johnson guard asked Pearson, "Who are you and where are you going?"
the laws of Canada, Mexico and the US are still largely seperate, and the laws of one do not necessarily follow in another.
Not to mention other North American countries such as France(1), Bermuda, Cuba, Haiti, etc., etc. --Michael Dillon (1) The islands of St. Pierre and Miquelon, Martinique, Guadeloupe P.S. The point of this is that there is nothing nice and neat about the environment in which the Internet operates therefore we really can't expect to find any nice and neat solutions to the messy problems we run across. All we can do is to limit the damage, detect the aberrations, and put them right as quickly as we can. The public has voted and their opinion is that 5 nines does not matter, best effort is good enough, if it really is *BEST* effort.
michael.dillon@bt.com wrote:
Haven't you noticed that the definition of "widely visited site" changes regularly, and often quite abruptly? How much traffic did YouTube get 3 years ago? Facebook? MySpace? There is no shortcut for eternal vigilance, i.e. manage your BGP relationships don't just configure and forget.
I hear that computers can be programed to create and maintain a list of sites one's own customers visit frequently. The same computer (so I'm told) could also determine which AS has authority to announce the IPs for each site. The same computer (so I'm told) could also watch for announcements from other ASs for the IPs from this up-to-date "widely visited sites" list, and send out an alert or otherwise take action when such an announcement was seen. It would be wonderful if someone would write and share code to do just this. jc
On Sun, 24 Feb 2008 20:42:51 -0500 "Patrick W. Gilmore" <patrick@ianai.net> wrote:
4: With state of the art security and operations.
I think we agree, but I wouldn't have said it like that.
How about state-of-the-art routing security? Seriously -- a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold. Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done. --Steve Bellovin, http://www.cs.columbia.edu/~smb
On Feb 25, 2008, at 12:31 AM, Steven M. Bellovin wrote:
Seriously -- a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold.
As much as I love to make fun of Avi + Vinny, as7007 was not really the problem, Sprint running custom cisco code which wouldn't let go of prefixes after the router announcing them had been turned off for more than an hour was. Plus, problems which happened over 10 years ago is a bit far back in Internet time. :)
Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done.
I would argue the answer to your question is "not yet", as we haven't done anything yet. We can argue whether that is the "right" answer, but it is still THE answer. -- TTFN, patrick
On Mon, 25 Feb 2008, Steven M. Bellovin wrote:
How about state-of-the-art routing security?
The problem is what is the actual trust model? Are you trusting some authority to not be malicious or never make a mistake? There are several answers to the malicious problem. There are fewer answers to never making a mistake problem. The state of the art routing security proposals let the "trusted" securely make mistakes. At one time or another, I think every router vendor, every ASN operator, every RIR, and so on has made a mistake at some time. Yeah, I know some of those mistakes may have actually been malicious, but so far the mistakes have outnumbered the malicious. If someone comes up with the anti-mistake routing protocol ...
On Mon, 25 Feb 2008 01:49:51 -0500 (EST) Sean Donelan <sean@donelan.com> wrote:
On Mon, 25 Feb 2008, Steven M. Bellovin wrote:
How about state-of-the-art routing security?
The problem is what is the actual trust model?
Are you trusting some authority to not be malicious or never make a mistake?
There are several answers to the malicious problem.
There are fewer answers to never making a mistake problem.
The state of the art routing security proposals let the "trusted" securely make mistakes. At one time or another, I think every router vendor, every ASN operator, every RIR, and so on has made a mistake at some time.
Yeah, I know some of those mistakes may have actually been malicious, but so far the mistakes have outnumbered the malicious.
If someone comes up with the anti-mistake routing protocol ...
Right. Everyone makes mistakes, but not everyone is malicious. And the RIRs and the big ISPs are *generally* more clueful than the little guys and the newcomers. Note also that secured BGP limits the kinds of mistakes people can make. If I have a certificate from my RIR for 192.0.2.0/24, I can't neither announce 10.0.0.0/8 nor delegate it to you, no matter how badly I type. Secured BGP still strikes me as a net win. --Steve Bellovin, http://www.cs.columbia.edu/~smb
Right. Everyone makes mistakes, but not everyone is malicious. And the RIRs and the big ISPs are *generally* more clueful than the little guys and the newcomers. Note also that secured BGP limits the kinds of mistakes people can make. If I have a certificate from my RIR for 192.0.2.0/24, I can't neither announce 10.0.0.0/8 nor delegate it to you, no matter how badly I type. Secured BGP still strikes me as a net win.
I suspect that a major part of the problem with implementing Secured BGP is that it is put forth as a solution that you implement in your routers. Network Operators are very careful about the stuff that goes into routers, even to the extent that many of them do not use SSH to manage them. Instead, they run SSH on trusted and secured servers inside their PoPs and configure their routers to only accept telnet sessions from those trusted and secured servers. Is there some way of deploying a solution like Secure BGP without actually requiring that it go into the routers? Perhaps something that allows the routers to still maintain BGP sessions that can withdraw routes, or announce routes which were recently withdrawn, but require a separate encrypted session between two servers, each one in a trust relationship with one of the BGP speaking routers, to handle announcements of new routes? Pushing this task off to a server that does not have packet-forwarding duties also allows for flexible interfaces to network management systems including the possibility of asking for human confirmation before announcing a new route. --Michael Dillon
michael.dillon@bt.com wrote: [..]
Pushing this task off to a server that does not have packet-forwarding duties also allows for flexible interfaces to network management systems including the possibility of asking for human confirmation before announcing a new route.
There is no (direct) requirement for most of these solutions to do it in the router that forwards actual packets, just add a special BGP box for this. This box then 'verifies' if the update looks OK. When the update looks fishy, it can either, depending on what you want either notify your favourite $nocmonkey to look at it and/or at least instruct the real routers to not use that path. You can take (S-)BGP(-S) for verification, but you can also use IRR data or whatever source you have for stating 'this prefix from there over this path is trusted', compare against that and voila, you got a report when the assumed vectors don't match and you can at least react to them. These kind of systems already exist, see previous emails, but clearly not too many actually make use of them, now that is too bad for your customers who couldn't see their lolcats or worse who couldn't reach their stock house for quickly selling their shares before that company went down the drain completely... Greets, Jeroen
Is there some way of deploying a solution like Secure BGP without actually requiring that it go into the routers?
The IETF SIDR wg (shameless plug as I'm wg co-chair) is working on a way to say with strong assurance who holds what prefixes, and therefore who can authorize the origination of what prefixes. This could be used in creating filter lists, answering customer request (please announce this for me...), checking the RIB out-of-band, etc. Such info is also the foundation of any yet proposed mechanism for doing in-band bgp security (S-BGP, soBGP, psBGP, SPV, etc., etc.), but the sidr work by itself does not need to be done in the router. Maybe some of you could take a look and comment. Look for the drafts at http://www.ietf.org/html.charters/sidr-charter.html --Sandy
On Sun, Feb 24, 2008 at 10:49 PM, Sean Donelan <sean@donelan.com> wrote:
On Mon, 25 Feb 2008, Steven M. Bellovin wrote:
How about state-of-the-art routing security?
The problem is what is the actual trust model?
Are you trusting some authority to not be malicious or never make a mistake?
There are several answers to the malicious problem.
There are fewer answers to never making a mistake problem.
[snip] +5, Insightful. The focus thus far seems to have been on establishing security on the basis of trusted senders (SPF for BGP, if you'll pardon my rather crude analogy). The impact of a mistake-based failure in a trusted scenario could actually be quite a bit higher than what we've seen thus far: 1) if data (e.g. routes) from a "trusted" source is allowed into a network (or used as a basis for decision-making) with minimal screening, attackers will immediately shift focus to spoofing trusted sources, just as they currently do in other scenarios; 2) the impact of a typo or other operator error when operating in "trusted mode" is much higher than that of mistakes from non-trusted sources (if 17557's upstream had trusted a little less - by not automatically propagating any new routes that showed up at the front door - the current incident could very well have amounted to little more than a log entry somewhere, and perhaps an email). I think what you and Steve Bellovin had to say about anti-mistake protocol and belt-and-suspenders should be garnering at least as much consideration as prevention of malicious attacks/forgeries/etc., considering the percentage of outages caused by operator error vs those caused by malice ... -- darkuncle@{gmail.com,darkuncle.net} || 0x5537F527 http://darkuncle.net/pubkey.asc for public key
At 05:31 AM 25-02-08 +0000, Steven M. Bellovin wrote:
Seriously -- a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold.
Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done.
"we've been warning that this could happen *again*" - this is happening every day - just look to: http://cs.unm.edu/~karlinjf/IAR/prefix.php?filter=most http://cs.unm.edu/~karlinjf/IAR/subprefix.php?filter=most for samples. Thing is - these prefix hijacks are not big ticket sites like Youtube or Microsoft or Cisco or even whitehouse.gov - but rather just sites that never make it onto the NANOG radar. -Hank
On Feb 25, 2008, at 2:32 AM, Hank Nussbacher wrote:
At 05:31 AM 25-02-08 +0000, Steven M. Bellovin wrote:
Seriously -- a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold.
Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done.
"we've been warning that this could happen *again*" - this is happening every day - just look to: http://cs.unm.edu/~karlinjf/IAR/prefix.php?filter=most http://cs.unm.edu/~karlinjf/IAR/subprefix.php?filter=most for samples. Thing is - these prefix hijacks are not big ticket sites like Youtube or Microsoft or Cisco or even whitehouse.gov - but rather just sites that never make it onto the NANOG radar.
How many of those would be stopped with transit providers filtering their downstreams? Which doesn't require rolling out a new technology like SBGP. And, I would argue, if we cannot even get transit providers to filter their downstreams, there is no way in hell we can get transit providers to filter on some RR or doing authentication on individual prefixes. Let's start with the easy stuff. Walk before run and all that. -- TTFN, patrick
This is a very interesting site. However, I notice that, in the "all in the last 24 hours" it doesn't show the YouTube hijack. It does have a lot of entries for 17557, most recently on 2/17. How reliable is this system?
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Hank Nussbacher Sent: Sunday, February 24, 2008 11:33 PM To: Steven M. Bellovin; nanog@merit.edu Subject: Re: YouTube IP Hijacking
At 05:31 AM 25-02-08 +0000, Steven M. Bellovin wrote:
Seriously -- a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold.
Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done.
"we've been warning that this could happen *again*" - this is happening every day - just look to: http://cs.unm.edu/~karlinjf/IAR/prefix.php?filter=most http://cs.unm.edu/~karlinjf/IAR/subprefix.php?filter=most for samples. Thing is - these prefix hijacks are not big ticket sites like Youtube or Microsoft or Cisco or even whitehouse.gov - but rather just sites that never make it onto the NANOG radar.
-Hank
Tomas: It's primarily a proof of concept site, to show that such an idea would be useful, but it has been running for over a year now and discovered many interesting hijacks (such as eBay/google/etc..). You're right that there is a glaring ommission, which is yesterday's youtube hijack. This is due to a bug in the sub-prefix lookup code (which can cause the IAR to miss some sub-prefix hijacks), which I'm currently fixing. Once that is done I'll rerun the IAR over yesterday's logs and it will show up. Josh On Mon, Feb 25, 2008 at 10:37 AM, Tomas L. Byrnes <tomb@byrneit.net> wrote:
This is a very interesting site. However, I notice that, in the "all in the last 24 hours" it doesn't show the YouTube hijack. It does have a lot of entries for 17557, most recently on 2/17.
How reliable is this system?
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Hank Nussbacher Sent: Sunday, February 24, 2008 11:33 PM To: Steven M. Bellovin; nanog@merit.edu Subject: Re: YouTube IP Hijacking
At 05:31 AM 25-02-08 +0000, Steven M. Bellovin wrote:
Seriously -- a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold.
Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done.
"we've been warning that this could happen *again*" - this is happening every day - just look to: http://cs.unm.edu/~karlinjf/IAR/prefix.php?filter=most<http://cs.unm.edu/%7Ekarlinjf/IAR/prefix.php?filter=most> http://cs.unm.edu/~karlinjf/IAR/subprefix.php?filter=most<http://cs.unm.edu/%7Ekarlinjf/IAR/subprefix.php?filter=most> for samples. Thing is - these prefix hijacks are not big ticket sites like Youtube or Microsoft or Cisco or even whitehouse.gov - but rather just sites that never make it onto the NANOG radar.
-Hank
A bit of administrativia: This thread generated over a hundred posts, many without operational relevance or by people who do not understand how operators, well, operate, or by people who really don't have any idea what's going on but feel like posting. I'd like to briefly summarize the important things that were said. If you would like to add something to the thread, make sure you read this post in entirety. Sorry if I didn't attribute every suggestion to a poster. Facts: * AS17557 announced more specific /24 to 3491, which propagated to wider internets * Chronology (by Martin@Renesys) http://www.renesys.com/blog/2008/02/pakistan_hijacks_youtube.shtml * Things suggested to possibly address the problem: ** IRR filtering (using IRRPT http://sourceforge.net/projects/irrpt/ to generate filter lists) ** Notification when origin of a given route changes http://www.cs.ucla.edu/~mohit/cameraReady/ladSecurity06.pdf http://www.ris.ripe.net/myasn.html http://cs.unm.edu/~karlinjf/IAR/index.php (from pgBGP) ** pgBGP to depref "suspicious routes" http://www.nanog.org/mtg-0606/pdf/josh-karlin.pdf (unclear the number of false positives that will adversely affect connectivity) ** sbgp/sobgp - require full authentication for each IP block, and thus unlikely to be implemented until certificate chains are in place, and vendors release code that does verification, and operators are happy enough running it. Other things addressed: * Fragility of Internet: ** Nobody brought up the important point - the BGP announcement filtering are only as secure as the weakest link. No [few?] peers or transits are filtering "large" ISPs (ones announcing few hundred routes and up). There are a great many of them, and it takes only one of them to mess up filtering a downstream customer for the route to be propagated. ** Paul Wall brought up the fact that even obviously bogus routes (1/8 and 100/7) were accepted by 99% of internet during an experiment. Will it take someone announcing 9/11 to get us to pay attention? (ok, bad joke) ** What I'd like to see discussed: Issues of filtering your transit downstream customers, who announce thousands of routes. Does *anyone* do it? * Typos vs Malicious announcements ** Some ways of "fixing" the problem (such as IRR filtering) only address the typos or unintentional announcements. There's full agreement that IRR is full of junk, which is not authenticated in any sort. ** Things like PHAS won't work if hijacker keeps the origin-AS same (by getting their upstream to establish session with different ASN) ** What I'd like to see discussed: Who (ICANN/RIRs/LIRs) is actively working on implementing "chain of trust" of IP space allocations? * Ways to address the issue without cooperation of 3491: ** Filtering anything coming out of 17557 ** Suggestions given: ** What I'd like to see discussed: Can an network operator, *today*, filter the "possibly bogus" routes from their peers, without manual intervention, and without false positives? * Yelling at people who don't filter ** Per above, 3491 isn't the only one who filters. In fact, claims were made that *nobody* filters "large enough" downstreams. (beyond aspath/maxpref) ** *please* do not post additional comments about pccw bad, etc. * Malicious vs mistaken on part of AS17557 and 3491: ** *please* do not post speculation unless you have facts to back it up. ** Any discussions of cyber-jihad are off-topic unless you can produce the fatwa to back it up.
On Feb 25, 2008, at 12:51 PM, Alex Pilosov wrote:
** Nobody brought up the important point - the BGP announcement filtering are only as secure as the weakest link. No [few?] peers or transits are filtering "large" ISPs (ones announcing few hundred routes and up). There are a great many of them, and it takes only one of them to mess up filtering a downstream customer for the route to be propagated.
Yes, that was my implicit point to Pekka. Even if you do everything feasible today (i.e., explicitly filter customers, some amount of policy for peers, and perhaps a few hacks for multi-homed customers), you're still pretty much screwed if someone announces your address space. Heck, you're as likely to accept the announcement as anyone.
** Paul Wall brought up the fact that even obviously bogus routes (1/8 and 100/7) were accepted by 99% of internet during an experiment.
I'm not sure why this would surprise anyone.
** What I'd like to see discussed: Issues of filtering your transit downstream customers, who announce thousands of routes. Does *anyone* do it?
Lots of folks do. The interesting bit is that even then, those same providers would accept perhaps even those customer routes from their peers implicitly.
* Typos vs Malicious announcements
** Some ways of "fixing" the problem (such as IRR filtering) only address the typos or unintentional announcements.
You mean as opposed to intentionally malice acts? Well, not completely. See Pekka's email, for example. Of course, it does vary widely across IRRs, etc..
There's full agreement that IRR is full of junk, which is not authenticated in any sort.
Mostly, though not completely.
** Things like PHAS won't work if hijacker keeps the origin-AS same (by getting their upstream to establish session with different ASN)
NO, that's not even necessary. Simple originate the route from the legit AS, and then transit it with the local AS as a transit AS. AS path manipulation is trivial.
** What I'd like to see discussed: Who (ICANN/RIRs/LIRs) is actively working on implementing "chain of trust" of IP space allocations?
* Ways to address the issue without cooperation of 3491: ** Filtering anything coming out of 17557
Bad idea.
** Suggestions given: ** What I'd like to see discussed: Can an network operator, *today*, filter the "possibly bogus" routes from their peers, without manual intervention, and without false positives?
Sure, if they want to dedicate an engineer to it, automate policy deployment and deal with brokenness by turning steam valves.
* Yelling at people who don't filter
That's been productive for over a decade now.
** Per above, 3491 isn't the only one who filters. In fact, claims were made that *nobody* filters "large enough" downstreams. (beyond aspath/maxpref)
Wrong. -danny
On Mon, 25 Feb 2008, Danny McPherson wrote:
** Paul Wall brought up the fact that even obviously bogus routes (1/8 and 100/7) were accepted by 99% of internet during an experiment.
I'm not sure why this would surprise anyone. To me and you, it's not surprising. To public, it might be. Even the majority of nanog attendees I think would be surprised.
** What I'd like to see discussed: Issues of filtering your transit downstream customers, who announce thousands of routes. Does *anyone* do it?
Lots of folks do. The interesting bit is that even then, those same providers would accept perhaps even those customer routes from their peers implicitly. Well, in this case, they *aren't* filtering! (unless I am misunderstanding what you are saying, due to repeated use of 'their').
** Things like PHAS won't work if hijacker keeps the origin-AS same (by getting their upstream to establish session with different ASN)
NO, that's not even necessary. Simple originate the route from the legit AS, and then transit it with the local AS as a transit AS. AS path manipulation is trivial. Oh yeah, d'oh! Thanks for correction. But that is also an important point against PHAS and IRRPT filtering - they are powerless against truly malicious hijacker (one that would register route in IRR, add the right origin-as to AS-SET, and use correct origin).
** What I'd like to see discussed: Who (ICANN/RIRs/LIRs) is actively working on implementing "chain of trust" of IP space allocations?
* Ways to address the issue without cooperation of 3491: ** Filtering anything coming out of 17557
Bad idea. Obviously :)
** Suggestions given: ** What I'd like to see discussed: Can an network operator, *today*, filter the "possibly bogus" routes from their peers, without manual intervention, and without false positives?
Sure, if they want to dedicate an engineer to it, automate policy deployment and deal with brokenness by turning steam valves. I'd hear to see who does it, and get them to present the "operational lessons" at the next nanog!
* Yelling at people who don't filter
That's been productive for over a decade now.
** Per above, 3491 isn't the only one who filters. In fact, claims were made that *nobody* filters "large enough" downstreams. (beyond aspath/maxpref)
Wrong. Likewise, I'd like to know who does this (names) and how can we get them to present best practices at the next nanog!
-alex
On Feb 25, 2008, at 1:22 PM, Alex Pilosov wrote:
Well, in this case, they *aren't* filtering! (unless I am misunderstanding what you are saying, due to repeated use of 'their').
What I'm saying is that best case today ISPs police routes advertised by their customers, yet they accept routes implicitly (including routes from address space that may belong to their customers) from peers. Seems a little hokey, eh?
Oh yeah, d'oh! Thanks for correction. But that is also an important point against PHAS and IRRPT filtering - they are powerless against truly malicious hijacker (one that would register route in IRR, add the right origin-as to AS-SET, and use correct origin).
Yep, pretty much.
Sure, if they want to dedicate an engineer to it, automate policy deployment and deal with brokenness by turning steam valves. I'd hear to see who does it, and get them to present the "operational lessons" at the next nanog!
Maybe Curtis V. would present what ANS was doing in 1994 :-) But now we've even got things like BGP route refresh, incrementally updatable filters, and BGP soft reconfiguration to ease the deployment burden. There have been two or three panels on this exact topic in the past, you can find them in the index of talks. Unfortunately, the problem hasn't changed at all. Perhaps we could just replay those video streams :-) -danny
I'd hear to see who does it, and get them to present the "operational lessons" at the next nanog!
On second thought, I guess one thing has changed considerably since 15 years ago. Rather than ~5000 monkeys with keyboard access to manipulate global routing tables, there are likely well North of 250,000 (>25k active ASNs * 10 meat computers per), which is surely well on the conservative side. The bottom line is [still] that ISPs should at least explicitly filter prefixes from customers and networks from which they provide transit services. -danny
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
There have been two or three panels on this exact topic in the past, you can find them in the index of talks. Unfortunately, the problem hasn't changed at all. Perhaps we could just replay those video streams :-)
My $.02 - http://www.getit.org/wordpress/?p=82 The irony to one of those, is that in NANOG 25 right before my session which pointed out this continued threat vector, we had Protecting the BGP Routes to Top Level DNS Servers http://www.nanog.org/mtg-0206/bush.html. -----BEGIN PGP SIGNATURE----- Version: PGP 8.1 iQA/AwUBR8M5ib/UEA/xivvmEQISnwCgojEcUx7dyMBQPP59gZjdLgQeqqMAoL0p seCMm+llF8tkr+pGf7LlyXrW =6jfG -----END PGP SIGNATURE-----
Alex Pilosov wrote:
Oh yeah, d'oh! Thanks for correction. But that is also an important point against PHAS and IRRPT filtering - they are powerless against truly malicious hijacker (one that would register route in IRR, add the right origin-as to AS-SET, and use correct origin).
With a decent LIR DB (like the RIPE DB) this is only possible if an hijacker breaks the authentication of the according database objects which is a pain in the a** _if_ the objects use a proper authentication scheme like PGP. -- Arnd
On 26/02/2008 12:06, "Arnd Vehling" <av@nethead.de> wrote: [...]
With a decent LIR DB (like the RIPE DB) this is only possible if an hijacker breaks the authentication of the according database objects which is a pain in the a** _if_ the objects use a proper authentication scheme like PGP.
I wonder what percentage of maintainers in the RIPE database only have PGP and/or X.509 auth schemes. I'd be surprised if it was as high as 5%. Leo
Leo Vegoda wrote:
On 26/02/2008 12:06, "Arnd Vehling" <av@nethead.de> wrote:
[...]
With a decent LIR DB (like the RIPE DB) this is only possible if an hijacker breaks the authentication of the according database objects which is a pain in the a** _if_ the objects use a proper authentication scheme like PGP.
I wonder what percentage of maintainers in the RIPE database only have PGP and/or X.509 auth schemes. I'd be surprised if it was as high as 5%.
True, but thats still better than having no authentication at all and its possible to require strong authentication on inetnum, route and AS objects. I just cant understand why LIR's like ARIN dont have any decent methods for this implemented in their DB. Or did this change recently? -- Arnd
On Mon, Feb 25, 2008, Alex Pilosov wrote:
A bit of administrativia:
This thread generated over a hundred posts, many without operational relevance or by people who do not understand how operators, well, operate, or by people who really don't have any idea what's going on but feel like posting.
I'd like to briefly summarize the important things that were said. If you would like to add something to the thread, make sure you read this post in entirety.
.. and, since I have 5 minutes to spare between disk array rebuilds, I bring you: http://nanog.cluepon.net/index.php/MailTopics For now it just links into Alex's summary post. I'll go trawl the archives now for more summaries. Adrian
Alex Pilosov ha scritto:
Facts:
* AS17557 announced more specific /24 to 3491, which propagated to wider internets
I think that they should use remote triggered blackhole filtering with no-export community. In this way they do the job with no impact on the rest of internet. Regards, Gianluca
On Tue, Feb 26, 2008 at 10:40 AM, hjan <hjan@libero.it> wrote:
I think that they should use remote triggered blackhole filtering with no-export community. In this way they do the job with no impact on the rest of internet.
so, certainly this isn't a bad idea, but given as an example: <http://www.secsup.org/CustomerBlackHole/> (Sorry not a perfect example, but illustrates my point) instead of: ip route my.offensive.material.0 255.255.255.0 Null0 tag 12345 the operator in question (person not place) types: ip route my.offensive.material.0 255.255.255.0 Null0 tag 1234 oops, a simple cut/paste mistake means that a route didn't get tagged properly, didn't get community tagged properly, didn't get set no-export and didn't get kept internally :( There is no SINGLE fix for this, there is a belt+suspenders approach: 1) Know what you are advertising (customer side of the puzzle) 2) Know what you are expecting to recieve (provider side of the puzzle) 3) plan for failures in both parts of this puzzle. -Chris
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Missing a tag in the trigger is why you put the Murphy Filters in the trigger router's route-map (the point you were getting at but being even more explicit). In my route map on the trigger router, I would not allow any static route triggers which did not have an exact match. I would also set the BGP advertisement to have - by default - the no-export community, a community range for all my triggers, and limit all my triggers to be below /24 (i.e /25 - /32). I then have three things to set my egress prefix filters to all my peers and customers: - comply with the default communities (no export) - filter all communities in my trigger range - filter anything in the /25 - /32 range. BTW - "Murphy Filters" is my term for policy filters which expect "Murphy's Law of Networking" to kick in. You have to expect human error. In addition to this, I would have my upstream mirror my filters. Life sucks when you advertise big blocks of the Internet and you become one giant sink hole (until you go congestion collapse, drop the BGP session and start flapping like crazy).
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Christopher Morrow Sent: Tuesday, February 26, 2008 8:59 AM To: hjan Cc: nanog@merit.edu Subject: Re: [admin] [summary] RE: YouTube IP Hijacking
On Tue, Feb 26, 2008 at 10:40 AM, hjan <hjan@libero.it> wrote:
I think that they should use remote triggered blackhole filtering with no-export community. In this way they do the job with no impact on the rest of internet.
so, certainly this isn't a bad idea, but given as an example:
<http://www.secsup.org/CustomerBlackHole/> (Sorry not a perfect example, but illustrates my point)
instead of: ip route my.offensive.material.0 255.255.255.0 Null0 tag 12345
the operator in question (person not place) types: ip route my.offensive.material.0 255.255.255.0 Null0 tag 1234
oops, a simple cut/paste mistake means that a route didn't get tagged properly, didn't get community tagged properly, didn't get set no-export and didn't get kept internally :(
There is no SINGLE fix for this, there is a belt+suspenders approach:
1) Know what you are advertising (customer side of the puzzle) 2) Know what you are expecting to recieve (provider side of the puzzle) 3) plan for failures in both parts of this puzzle.
-Chris
-----BEGIN PGP SIGNATURE----- Version: PGP 8.1 iQA/AwUBR8RSF7/UEA/xivvmEQJUKACfZB+typ7sIJMnDS+QrO0MqGED+CYAoKFC iBmY+pq0CohSIJwtu5pgzCJt =xiog -----END PGP SIGNATURE-----
The biggest problem here is that Cisco needs to change their defaults to require more configuration than router bgp X neighbor 1.2.3.4 remote-as A When that's the bar for the complexity required for setting up BGP, bad things WILL happen. Period. Cisco has taken all these years and not stepped up in providing any sort of a reasonable change to the marketplace as others have done. This hurts the industry as a whole, and hurts the perception that "we can't route". As for other problems with leadership, there's no good way to manage large configurations on the platforms, nor a reasonable size of NVRAM provided either. The list goes on and on, and I've communicated this more than once to the company. Nobody cares about this basic infrasturcture at Cisco, or at least nobody that can make something happen. Instead people care about what product is intruding on their turf and how to defend it instead of building a better product and improving things. Honestly, it's a lost cause and SP's don't account for any significant amount of revenue. If someone at Cisco cares to address these things, i'm interested in helping but it's clear that the head-in-the-sand policy by upper mgmt lives on and they'd rather fight amongst themselves and risk the industry as a whole because of their antics. - Jared (speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost). -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
for a list filled with network operators and engineers, the lot of you are quick to whip out lawyers and courts and international tribunals. perhaps I missed the message, but has anyone mentioned the direct economic impact of SFI? as a responsible network operator, would you peer with a network that didn't filter their customers and was consistently linked to route leakage issues? when is enough, enough (in general, not speaking to any specific network)? also, this: On Tue, Feb 26, 2008 at 10:21 AM, Jared Mauch <jared@puck.nether.net> wrote:
The biggest problem here is that Cisco needs to change their defaults to require more configuration than
router bgp X neighbor 1.2.3.4 remote-as A
When that's the bar for the complexity required for setting up BGP, bad things WILL happen. Period.
(speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost).
I remember the days of making rtconfig work properly in various situations (heck, do people still use that? Does it even do IPv6 right?) and building router configs out of both public and private IRR data. Perhaps if the entry barrier to building dynamically generated router configurations were lowered significantly (to the point of it being free, GUI, and multi-platform) then it may be used for new network designs by people starting off. Getting Cisco/Juniper/etc to push -that- as part of their best practices for network design would be quite helpful. The problem isn't that the router config is too easy Jared, its that there's no nice and easy way of doing it right from scratch that matches the sort of newbie network operators that exist today. For examples of what "new school" netops are like, visit isp-* lists. There's a lot of clue there, its just "different" and "haven't learnt from the old school experience" clue, and its amusing/sad to watch people make the same mistakes you all did in the 90s. :) (Where's vijay now when science for generated network configurations is needed?) Make the public tools better. Push the tools as best practice. ADrian
On 27/02/2008, at 11:39 AM, Adrian Chadd wrote:
(speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost).
I remember the days of making rtconfig work properly in various situations (heck, do people still use that? Does it even do IPv6 right?)
Yeah, we use it. And (following a bunch of patches we made a couple of months ago) we've convinced it to do IPv6 too. - mark -- Mark Newton Email: newton@internode.com.au (W) Network Engineer Email: newton@atdot.dotat.org (H) Internode Systems Pty Ltd Desk: +61-8-82282999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223
On Wed, Feb 27, 2008 at 10:09:19AM +0900, Adrian Chadd wrote:
(speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost).
Getting Cisco/Juniper/etc to push -that- as part of their best practices for network design would be quite helpful.
The problem isn't that the router config is too easy Jared, its that there's no nice and easy way of doing it right from scratch that matches the sort of newbie network operators that exist today. For examples of what "new school" netops are like, visit isp-* lists. There's a lot of clue there, its just "different" and "haven't learnt from the old school experience" clue, and its amusing/sad to watch people make the same mistakes you all did in the 90s. :)
(Where's vijay now when science for generated network configurations is needed?)
Make the public tools better. Push the tools as best practice.
The problem is that some of us have developed tools that are considered our companies "property", so we can't just go ahead and release it to the public. Who is gonna start the project to get this going? How do you integrate it with your existing provisioning system? I've regularly heard some of the larger telecoms quote times of ~2-3 years and $10m+ for any project like this. Not sure if those timelines ever were started. Perhaps this is something that renesys or cariden could market and sell? (just to name two nanog sponsors that have some sort of dataset or tools that could apply). I'd like to see this all cleaned up and get better. I track obvious leaks that should be caught by as-path filtering and proper policy here: http://puck.nether.net/bgp/leakinfo.cgi there's a stats page one can find so you can track the number of leaks/day that are seen, including the most common as-paths. if you're smart try appending ?days=3 on the end of the statistics cgi. - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
On Tue, Feb 26, 2008, Jared Mauch wrote:
The problem isn't that the router config is too easy Jared, its that there's no nice and easy way of doing it right from scratch that matches the sort of newbie network operators that exist today. For examples
The problem is that some of us have developed tools that are considered our companies "property", so we can't just go ahead and release it to the public.
I know. I even had to write some of that.
Who is gonna start the project to get this going? How do you integrate it with your existing provisioning system? I've regularly heard
I didn't say it had to be integrated into -existing- systems. I said there's no nice and easy way of doing it right now -from scratch-.
Perhaps this is something that renesys or cariden could market and sell? (just to name two nanog sponsors that have some sort of dataset or tools that could apply).
I'd like to see this all cleaned up and get better. I track obvious leaks that should be caught by as-path filtering and proper policy here:
http://puck.nether.net/bgp/leakinfo.cgi
there's a stats page one can find so you can track the number of leaks/day that are seen, including the most common as-paths.
if you're smart try appending ?days=3 on the end of the statistics cgi.
I can't see this linked off the NANOG wiki. So much clue out there, so little put into the Wiki for those like me to poke with a stick. ... and its there - http://nanog.cluepon.net/index.php/InternetTools Now comes the fun task of trying to scrape the last 12 months of NANOG mailing list posts to find all the interesting data and tool URLs which should really be in the Wiki. Adrian
Dear all, Discussions on the recent Youtube incident raised the question about availability of our projects PHAS (Prefix Hijack Alert System). http://phas.netsec.colostate.edu/ Unfortunately, the timing of the hijack coincided with our transitioning to the next stage of PHAS, thus it was unavailable at the time. We have switched back to the last stable version and the site is fully functional now. We apologize for the inconvenience. For people not familiar with PHAS, we analyze BGP updates received from different vantage points and maintain 3 sets for each prefix. 1. Origin set 2. Last hop set 3. Sub-prefix set Anyone may register with PHAS for the prefixes he/she wants to watch, and select the types of alarms of interest. Each time the set changes, an email is sent to the registered email addresses. If you want to get an idea of the alarms generated, you can register for one or more active prefixes that are constantly generating alarms as seen in http://phas.netsec.colostate.edu/stat.html For the youtube hijack case: 1. since a more specific prefix was observed for youtube's prefix, PHAS caught the incident as a "sub-prefix set change" and an alarm was generated. 2. PHAS does not rely on information from IRR, so any manipulations to IRR (or outdated entries) would not affect PHAS. 3. Some folks questioned whether PHAS would detect cases of hijack if origin AS was unchanged: from the above, one can see that PHAS catches any sub prefix announcements, and any changes to the last hop (i.e. next hop to origin AS). It is true that the current version of PHAS does not detect AS path manipulations beyond the last hop. We are developing solutions to this problem and hoping to combine the new solution into PHAS soon. Our recent results also show that the farther away from the origin the hijacker inserts his AS number, the less impact it would have on the Internet. For folks interested in how the impact of a hijack may vary depending on which prefix is involved and the hijacker's location, we have a paper in DSN 2007 with some interesting results. http://www.cs.ucla.edu/~mohit/cameraReady/hijack-dsn.pdf Thanks -Mohit
On Sat, Mar 01, 2008, Mohit Lad wrote:
Dear all,
Discussions on the recent Youtube incident raised the question about availability of our projects PHAS (Prefix Hijack Alert System). http://phas.netsec.colostate.edu/
Cute! Its in the Wiki. (And I'm going to keep saying this until I get fed up with the Wiki, Marty/Alex tells me to shut up, or others start dumping interesting stuff into the Wiki.) Adrian
On Feb 27, 2008, at 2:09 AM, Adrian Chadd wrote:
(speaking as someone who has built large ACLs/prefix-lists and has 6MB+ configs that can't be loaded on my routers. without vendor support those that want to do the right thing can't, so the game is lost).
I remember the days of making rtconfig work properly in various situations (heck, do people still use that? Does it even do IPv6 right?) and building router configs out of both public and private IRR data.
Perhaps if the entry barrier to building dynamically generated router configurations were lowered significantly (to the point of it being free, GUI, and multi-platform) then it may be used for new network designs by people starting off.
Getting Cisco/Juniper/etc to push -that- as part of their best practices for network design would be quite helpful.
The problem isn't that the router config is too easy Jared, its that there's no nice and easy way of doing it right from scratch that matches the sort of newbie network operators that exist today. For examples of what "new school" netops are like, visit isp-* lists. There's a lot of clue there, its just "different" and "haven't learnt from the old school experience" clue, and its amusing/sad to watch people make the same mistakes you all did in the 90s. :)
(Where's vijay now when science for generated network configurations is needed?)
Make the public tools better. Push the tools as best practice.
ADrian
Well there is a slight risk that the more you will improve the automated tools, the less net engineers will actually understand the reasons why it is done... That being said, all the filtering work is a significant part of every Network Engineer's work, and is not that big of a deal. Education is the art of repeating, and has always been. I'm not saying we should systematically point the ones making mistakes and make it public, what I'm saying is that the reason why mailing lists such as nanog exist is actually mutual education. I'm sure the people involved in this incident were (or now are) Nanog readers, and that they've understood the message. Still, what should be done is, I assume, centralizing the info in one single mail, kept for the record: - list the incident chronology - analyze what technical mistake lead to that - list -all- the ways to prevent it Another mean of education is including the analysis of this incident (troubleshooting, resolution, implementing means to avoid it) next Nanog agenda, which I already think is the case :) Greg VILLAIN
On Mon, Feb 25, 2008, Alex Pilosov wrote:
A bit of administrativia:
This thread generated over a hundred posts, many without operational relevance or by people who do not understand how operators, well, operate, or by people who really don't have any idea what's going on but feel like posting.
I'd like to briefly summarize the important things that were said. If you would like to add something to the thread, make sure you read this post in entirety.
[snip] http://arstechnica.com/news.ars/post/20080225-insecure-routing-redirects-you... There's a discussion page with feedback from a non-NANOG audience. Science, anyone? Adrian
On Mon, Feb 25, 2008 at 2:32 AM, Hank Nussbacher <hank@efes.iucc.ac.il> wrote:
"we've been warning that this could happen *again*" - this is happening every day - just look to: http://cs.unm.edu/~karlinjf/IAR/prefix.php?filter=most http://cs.unm.edu/~karlinjf/IAR/subprefix.php?filter=most
So, as someone that once subscribed to josh's alerts... for a large ISP, or an ISP with lots of customers using PA/PI space I got loads and loads of alerts about customer allocations popping out in other ASN's, it turns out those were almost always intended by the customer even... So I'm not sure that the above links are the best judge of this problem space. I agree though that the problem is there to some extent... much more often seen than the 1/2yr events we've noted on nanog-l over the last 8 years. -Chris
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Steven M. Bellovin How about state-of-the-art routing security?
Seriously -- a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold.
Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done.
BCPs stops this problem. soBGP may make it easier.
y'all, On Mon, Feb 25, 2008 at 06:49:35AM -0800, Barry Greene (bgreene) wrote:
Seriously -- a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold.
Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done.
BCPs stops this problem. soBGP may make it easier.
except in the most meaningless of interpretations, i don't see how BCPs solve this problem. Barry: did you mean: "If only everyone on the Internet would perfectly filter their customers prefix announcements on every connection, then everything would be fine?" Or perhaps: "If everyone would register all of their prefixes in some applicable routing registry with such a degree of accuracy that we could build customer and peer filters out of it then this problem would go away." ? both are true statements, but neither is ever going to happen (i'm not sure that sBGP or soBGP is going to happen ever either). in the mean time all we can do is watch and try to respond to events as quickly as they occur. In fact, we would all be better off if more people were just watching their prefix announcement progapations. Although this hijacking/blackholing was caught quickly, i have seen many other cases where the event lasted for hours (or days). by the way: my colleague Martin A. Brown (who spoke at NANOG on the Taiwan Earthquakes), has written a fairly detailed blog post on the this event. It includes a detailed timeline and some information for people who might find that useful. http://www.renesys.com/blog/2008/02/pakistan_hijacks_youtube_1.shtml Clearly this is not the first interesting accidental hijacking and certainly won't be the last. t. -- _____________________________________________________________________ todd underwood +1 603 643 9300 x101 renesys corporation general manager babbledog todd@renesys.com http://www.renesys.com/blog
On Sun, Feb 24, 2008 at 8:42 PM, Patrick W. Gilmore <patrick@ianai.net> wrote:
2: Within a jurisdiction where North American operators have a good chance of having the law on their side in case of any network outage caused by the entity.
This is also a bit strange. Do your users never attach to a host outside the USofA?
m.root-servers.net i.root-servers.net www.ripe.net www.apnic.net oops!
3: Considered highly competent technically.
Here we agree.
except that even the 'good guys' make mistakes. Belt + suspenders please... is it really that hard for a network service provider to have a prefix-list on their customer bgp sessions?? L3 does it, ATT does it, Sprint does it, as do UUNET/vzb, NTT, GlobalCrossing ... seriously, it's not that hard.
OTOH: I would say that, until today, those who advocate not engaging in any kind of ethnic or political profiling would have considered 17557, as a national telco, a trusted route source.
no, unless they had some recourse (SFP agreement?) for such behaviours... clearly said agreement wasn't in place so the PCCW folks REALLY should have had some belt+suspenders approach in place. As an aside, I'm against the 'golden prefixes' idea, because it quickly devolves into a pay-for-play game where in the end everyone pays a disproportionate amount :( -Chris
Christopher Morrow wrote:
On Sun, Feb 24, 2008 at 8:42 PM, Patrick W. Gilmore <patrick@ianai.net> wrote: except that even the 'good guys' make mistakes. Belt + suspenders please... is it really that hard for a network service provider to have a prefix-list on their customer bgp sessions?? L3 does it, ATT does it, Sprint does it, as do UUNET/vzb, NTT, GlobalCrossing ... seriously, it's not that hard.
I know of one tier-1 on that list that made a filtering mistake on one of my own circuits no more than a few months ago. Even some of the biggest players make mistakes. Justin
On Feb 25, 2008, at 1:22 AM, Christopher Morrow wrote:
except that even the 'good guys' make mistakes. Belt + suspenders please... is it really that hard for a network service provider to have a prefix-list on their customer bgp sessions?? L3 does it, ATT does it, Sprint does it, as do UUNET/vzb, NTT, GlobalCrossing ... seriously, it's not that hard.
What Chris said. Why isn't this a BCP that people can actually agree on? Wait, being a BCP is probably what killed BCP38 take-up so never mind :(
John Payne wrote:
On Feb 25, 2008, at 1:22 AM, Christopher Morrow wrote:
except that even the 'good guys' make mistakes. Belt + suspenders
please... is it really that hard for a network service provider to
have a prefix-list on their customer bgp sessions?? L3 does it, ATT
does it, Sprint does it, as do UUNET/vzb, NTT, GlobalCrossing ...
seriously, it's not that hard.
What Chris said. Why isn't this a BCP that people can actually agree on? Wait, being a BCP is probably what killed BCP38 take-up so never mind :(
Well... If you want to work on one, I'm willing help shepherd it through the process. We even have a working group setup for that purpose. joel
On Tue, Feb 26, 2008 at 7:17 PM, Joel Jaeggli <joelja@bogus.com> wrote:
John Payne wrote:
On Feb 25, 2008, at 1:22 AM, Christopher Morrow wrote:
except that even the 'good guys' make mistakes. Belt + suspenders
please... is it really that hard for a network service provider to
have a prefix-list on their customer bgp sessions?? L3 does it, ATT
does it, Sprint does it, as do UUNET/vzb, NTT, GlobalCrossing ...
seriously, it's not that hard.
What Chris said. Why isn't this a BCP that people can actually agree on? Wait, being a BCP is probably what killed BCP38 take-up so never mind :(
Well... If you want to work on one, I'm willing help shepherd it through the process. We even have a working group setup for that purpose.
proposal for work in GROW?
At 07:15 PM 24-02-08 -0500, Randy Epstein wrote:
More importantly, why is PCCW not prefix filtering their downstreams?
Why? - Lack of clue - Couldn't care less - No revenue Take your pick - or add your own reason. PCCW is not alone. They just happen to be the latest in a long line of ISPs that follow the same rules - their own. -Hank
On Feb 25, 2008, at 2:27 AM, Hank Nussbacher wrote:
At 07:15 PM 24-02-08 -0500, Randy Epstein wrote:
More importantly, why is PCCW not prefix filtering their downstreams?
Why?
- Lack of clue - Couldn't care less - No revenue
Take your pick - or add your own reason. PCCW is not alone. They just happen to be the latest in a long line of ISPs that follow the same rules - their own.
All good, er, bad reasons. Fixing the "filter your downstreams" problem is very important. It would also solve 90-something percent of the problems mentioned in this thread. E.g. as7007. :) -- TTFN, patrick
"Patrick W. Gilmore" <patrick@ianai.net> wrote
On Feb 25, 2008, at 2:27 AM, Hank Nussbacher wrote:
At 07:15 PM 24-02-08 -0500, Randy Epstein wrote:
More importantly, why is PCCW not prefix filtering their downstreams?
Why?
- Lack of clue - Couldn't care less - No revenue
Take your pick - or add your own reason. PCCW is not alone. They just happen to be the latest in a long line of ISPs that follow the same rules - their own.
All good, er, bad reasons. Fixing the "filter your downstreams" problem is very important. It would also solve 90-something percent of the problems mentioned in this thread. E.g. as7007. :)
I am in the APRICOT meeting in Taipei now, and met a guy from PCCW/AS3491. I have showed him this thread, and have suggested 1) validating prefixes from downstreams before accept, and 2) setting an inbound prefix-filter to their downstreams. regards, ----- Matsuzaki Yoshinobu <maz@iij.ad.jp> - IIJ/AS2497 INOC-DBA: 2497*629
At 06:17 PM 25-02-08 +0900, Matsuzaki Yoshinobu wrote:
All good, er, bad reasons. Fixing the "filter your downstreams" problem is very important. It would also solve 90-something percent of the problems mentioned in this thread. E.g. as7007. :)
I am in the APRICOT meeting in Taipei now, and met a guy from PCCW/AS3491. I have showed him this thread, and have suggested 1) validating prefixes from downstreams before accept, and 2) setting an inbound prefix-filter to their downstreams.
Now if only everyone here on NANOG were to do what Matsuzaki has done, and take the time to educate those less clueless, the world would be a better place. -Hank
Now if only everyone here on NANOG were to do what Matsuzaki has done, and take the time to educate those less clueless, the world would be a better place.
Its time that people responsible for BGP routing need to show that they have the skills and knowledge for it. Every ISP requesting an ASN from one of the LIR's should be required to make a test covering the neccessary skillsets. -- Arnd
On Tue, Feb 26, 2008 at 11:43:10AM +0100, Arnd Vehling <av@nethead.de> wrote a message of 12 lines which said:
Every ISP requesting an ASN from one of the LIR's should be required to make a test covering the neccessary skillsets.
Giving the rapid turnover of people in this industry, I'm not sure it would help. Two months after the test, may be all the clueful people will be gone. And what about the organizations which rely on external consultancy? Should it be forbidden?
Stephane Bortzmeyer wrote:
On Tue, Feb 26, 2008 at 11:43:10AM +0100, Arnd Vehling <av@nethead.de> wrote a message of 12 lines which said:
Every ISP requesting an ASN from one of the LIR's should be required to make a test covering the neccessary skillsets.
Giving the rapid turnover of people in this industry, I'm not sure it would help. Two months after the test, may be all the clueful people will be gone.
How about an ISP running BGB is required to have a number of "LIR certified" BGP people and is otherwise not allowed to peer *grin*
And what about the organizations which rely on external consultancy? Should it be forbidden?
No, the consultants can be "LIR certified" too. I just dont see why people need to have a damn license for "lots of stuff" but noone cares if the people running border routers have the neccessary qualifications. It just doesnt make sense and i am pretty sure that this _will_ change. -- Arnd
On Mon, Feb 25, 2008 at 09:27:41AM +0200, Hank Nussbacher <hank@efes.iucc.ac.il> wrote a message of 17 lines which said:
- Lack of clue - Couldn't care less - No revenue
Take your pick - or add your own reason. PCCW is not alone. They just happen to be the latest in a long line of ISPs that follow the same rules - their own.
No, most operators do filter BGP announcements. I know it, because I have read it on Cnet: http://www.news.com/8301-10784_3-9878655-7.html
That's because Hong Kong-based PCCW, which provides the Internet link to Pakistan Telecom, did not stop the misleading broadcast--which is what most large providers in the United States and Europe do.
On Feb 24, 2008, at 2:14 PM, Tomas L. Byrnes wrote:
I figured as much, but it was worth a try.
Which touches on the earlier discussion of the null routing of /32s advertised by a special AS (as a means of black-holing DDOS traffic).
It seems to me that a more immediately germane matter regarding BGP route propagation is prevention of hijacking of critical routes.
Perhaps certain ASes that are considered "high priority", like Google, YouTube, Yahoo, MS (at least their update servers), can be trusted to propagate routes that are not aggregated/filtered, so as to give them control over their reachability and immunity to longer-prefix hijacking (especially problematic with things like MS update sites).
That's just inviting the injection of forged AS routes to commit abuse. Owen
-----Original Message----- From: Simon Lockhart [mailto:simon@slimey.org] Sent: Sunday, February 24, 2008 2:07 PM To: Tomas L. Byrnes Cc: Michael Smith; neil.fenemor@fx.net.nz; will@harg.net; nanog@merit.edu Subject: Re: YouTube IP Hijacking
On Sun Feb 24, 2008 at 01:49:00PM -0800, Tomas L. Byrnes wrote:
Which means that, by advertising routes more specific than the ones they are poisoning, it may well be possible to restore universal connectivity to YouTube.
Well, if you can get them in there.... Youtube tried that, to restore service to the rest of the world, and the announcements didn't propogate.
Simon
Not if only trusted peers are allowed to advertise to that AS. It's the same mechanism proposed for blackholing on destination to dampen DOS a while back, except it is to prevent hijacking, and therefore doesn't run afoul of the AT&T patent (and now the prior art for this is in the public domain). It's also something that can be built using the existing infrastructure, and rough consensus.
-----Original Message----- From: Owen DeLong [mailto:owen@delong.com] Sent: Sunday, February 24, 2008 8:25 PM To: Tomas L. Byrnes Cc: Simon Lockhart; Michael Smith; neil.fenemor@fx.net.nz; will@harg.net; nanog@merit.edu Subject: Re: YouTube IP Hijacking
On Feb 24, 2008, at 2:14 PM, Tomas L. Byrnes wrote:
I figured as much, but it was worth a try.
Which touches on the earlier discussion of the null routing of /32s advertised by a special AS (as a means of black-holing DDOS
traffic).
It seems to me that a more immediately germane matter regarding BGP route propagation is prevention of hijacking of critical routes.
Perhaps certain ASes that are considered "high priority",
YouTube, Yahoo, MS (at least their update servers), can be
like Google, trusted to
propagate routes that are not aggregated/filtered, so as to give them control over their reachability and immunity to longer-prefix hijacking (especially problematic with things like MS update sites).
That's just inviting the injection of forged AS routes to commit abuse.
Owen
-----Original Message----- From: Simon Lockhart [mailto:simon@slimey.org] Sent: Sunday, February 24, 2008 2:07 PM To: Tomas L. Byrnes Cc: Michael Smith; neil.fenemor@fx.net.nz; will@harg.net; nanog@merit.edu Subject: Re: YouTube IP Hijacking
On Sun Feb 24, 2008 at 01:49:00PM -0800, Tomas L. Byrnes wrote:
Which means that, by advertising routes more specific
than the ones
they are poisoning, it may well be possible to restore universal connectivity to YouTube.
Well, if you can get them in there.... Youtube tried that, to restore service to the rest of the world, and the announcements didn't propogate.
Simon
It does sort of shed light on a sobering fact that some of the PCCW's of the world are not using proper filtering, and with a coordinated effort, someone could inject a large number of routes into the global routing table through them effectively taking offline much of the Internet. Anything more specific than a /24 would get blocked by many filters, so some of the "high target" sites may want to announce their mission critical IP space as /24 and avoid using prepends. If the PCCW's of the world are not going to sanity check inbound announcements from some of their peers, they should at least be prepending them to help fight abuse of this nature (accidental or not). Also, IANAL, but there seems to be a misconception of what AT&T's DDoS patent (application 20060031575) covers. The patent is not simply about blackholing an IP address, it claims "Such a selective black-holing scheme can be used to allow some traffic to continue in route to the IP address under attack, while other traffic is diverted." So simply blackholing everything destined to an IP address does not seem to conflict with the patent. As a side note, it will be interesting to see how the youtube posters respond to this. If Pakistan thought the site was offensive before, I doubt they will be amused at the backlash that will probably occur as the result of this. I have a feeling youtubers will be trying to 1up each other for most offensive video.
Rick Astley writes:
Anything more specific than a /24 would get blocked by many filters, so some of the "high target" sites may want to announce their mission critical IP space as /24 and avoid using prepends.
Good idea. But only the "high target" sites, please. If you're an unimportant site that nobody cares about, then DON'T DO THIS, ok? ;-) -- Simon.
Which means that, by advertising routes more specific than the ones they are poisoning, it may well be possible to restore universal connectivity to YouTube.
Well, if you can get them in there.... Youtube tried that, to restore service to the rest of the world, and the announcements didn't propogate.
Some of us block prefixes longer than /24 at our borders (even if our transit providers don't). Steinar Haug, Nethelp consulting, sthaug@nethelp.no
Not if the hijackers have advertised a /24. Anything you advertise more specific than /24 will be lost on many networks' filters. -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Tomas L. Byrnes Sent: Monday, 25 February 2008 8:49 AM To: Michael Smith; neil.fenemor@fx.net.nz Cc: will@harg.net; nanog@merit.edu Subject: RE: YouTube IP Hijacking Which means that, by advertising routes more specific than the ones they are poisoning, it may well be possible to restore universal connectivity to YouTube.
-----Original Message----- From: Michael Smith [mailto:msmith@internap.com] Sent: Sunday, February 24, 2008 1:23 PM To: neil.fenemor@fx.net.nz; Tomas L. Byrnes Cc: will@harg.net; nanog@merit.edu Subject: Re: YouTube IP Hijacking
Exactly... They inadvertently made the details of their oppression more readily apparent...
----- Original Message ----- From: owner-nanog@merit.edu <owner-nanog@merit.edu> To: Tomas L. Byrnes <tomb@byrneit.net> Cc: Will Hargrave <will@harg.net>; nanog@merit.edu <nanog@merit.edu> Sent: Sun Feb 24 16:00:35 2008 Subject: Re: YouTube IP Hijacking
While they are deliberately blocking Youtube nationally, I suspect the wider issue has no malice, and is a case of poorly constructed/ implemented outbound policies on their part, and poorly constructed/ implemented inbound polices on their upstreams part.
On 25/02/2008, at 9:49 AM, Tomas L. Byrnes wrote:
Pakistan is deliberately blocking Youtube.
http://politics.slashdot.org/article.pl?sid=08/02/24/1628213
Maybe we should all block Pakistan.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]
Of Will Hargrave Sent: Sunday, February 24, 2008 12:39 PM To: nanog@nanog.org Subject: Re: YouTube IP Hijacking
Sargun Dhillon wrote:
So, it seems that youtube's ip block has been hijacked by a more specific prefix being advertised. This is a case of IP hijacking, not case of DNS poisoning, youtube engineers doing something stupid, etc. For people that don't know. The router will try to get the most specific prefix. This is by design, not by accident.
You are making the assumption of malice when the more
On Behalf likely cause is
one of accident on the part of probably stressed NOC staff at 17557.
They probably have that /24 going to a gateway walled garden box which replies with a site saying 'we have banned this', and that /24 route is leaking outside of their AS via PCCW due to dodgy filters/communities.
Will
Neil Fenemor FX Networks
participants (38)
-
Aaron Glenn -
Adrian Chadd -
Alex Pilosov -
Arnd Vehling -
Barry Greene (bgreene) -
Campbell, Alex -
Christopher Morrow -
Danny McPherson -
Greg VILLAIN -
Hank Nussbacher -
hjan -
Jared Mauch -
JC Dill -
Jeroen Massar -
Jim Mercer -
Joel Jaeggli -
John Payne -
Josh Karlin -
Justin Shore -
Leo Vegoda -
Mark Newton -
Matsuzaki Yoshinobu -
michael.dillon＠bt.com -
Mohit Lad -
Owen DeLong -
Patrick W. Gilmore -
Randy Epstein -
Rick Astley -
sandy＠tislabs.com -
Scott Francis -
Sean Donelan -
Simon Leinen -
Simon Lockhart -
Stephane Bortzmeyer -
Steven M. Bellovin -
sthaug＠nethelp.no -
Todd Underwood -
Tomas L. Byrnes