so a few of us are still looking at routing through the anycast sunglasses. a particular probe is seeing instability [0] for k.root-servers.net [1]. so we hop on to a router nearby, and have some fun looking at things. we discover an anomaly which takes a while to sort out o some of the anycasted servers mark their announcements with the magic NO_EXPORT community in an attempt to localize their distribution (it would be good if a server in kenya did not take load from nyc) o they also have a server or two which does not so mark their announcements on the presumption that the rest of the world can get to those non-marking server(s) this last assumption is not safe o consider large providers p0 and p1 which are connected to k0 which announces with NO_EXPORT o there is also server k1 which is out there somewhere and announcing without NO_EXPORT o test point t0 is in a multi-homed asn connected to p0 and p1 o the routers in p0 and p1 at which t0's router is connected have their best path to k being the one to k0 o this obscures their path to k1 o and, as they obey k0's NO_EXPORT, they can not export any route to a k.root-servers.net server to t0 o so t0 sees no route to k.root-servers.net this implies that a non-trivial part of the net can not see anycast services for which some of the servers are marking their announcements as NO_EXPORT. note that we saw this from a multi-homed site in seattle, not from some more net-isolated probe. whoops! randy --- [0] - please remember, this is not that the servers are unstable, but rather that routing near the probe is unstable, and the anycasted prefixes are likely to show this more than those which are unicast. [1] - side note: we got misled for a few seconds by % host k.root-servers.org. k.root-servers.org has address 193.0.0.214 % host k.root-servers.net. k.root-servers.net has address 193.0.14.129
Randy Bush wrote:
so a few of us are still looking at routing through the anycast sunglasses. a particular probe is seeing instability [0] for k.root-servers.net [1]. so we hop on to a router nearby, and
o this obscures their path to k1
o and, as they obey k0's NO_EXPORT, they can not export any route to a k.root-servers.net server to t0
Isnt this the standard problem? Why does anycast have any special bearing on the problem? (perhaps it doesnt you just want someone to fix it) My amateurs understanding of this is that: Sites connected to providers who have chosen a path marked as NO_EXPORT as best over one not so marked will not get any route to that prefix from that provider. They better hope that they are connected to another provider who did not select as best path a NO_EXPORT marked prefix. A number of conclusions can possibly be made: NO_EXPORT is not safe to be used while trying to traffic engineer but maintain global connectivity. NO_EXPORT is only safe for more specific prefixes, as long as there is a less specific prefix that is still usuable. NO_EXPORT to a large provider with potential large numbers of single homed BGP customers who may not be taking a 0/0 (in an attempt to use SAV?) is probably not a good idea NO_EXPORT to large providers raises the probablity of there being sites who multihome to only those, therefore NO_EXPORT to multiple large providers is almost certainly dangerous. traffic engineering done by the core based upon instructions from the edge is dangerous.
On Mon, 31 Oct 2005, Joe Maimon wrote: > Isnt this the standard problem? Correct. > Why does anycast have any special bearing on the problem? It doesn't, except that Vixie decided to do it anyway with F, and it sounds like now K is doing it as well. I have no clue why. The problems with doing this had been clearly understood for a long time, the putative benefit is negliglble, and to the best of my knowledge, none of those of us who run large anycast networks commercially have ever found it beneficial to use NO_EXPORT in the general case. I guess folks just like to be different. > Sites connected to providers who have chosen a path marked as NO_EXPORT as > best over one not so marked will not get any route to that prefix from > that provider. They better hope that they are connected to another > provider who did not select as best path a NO_EXPORT marked prefix. Correct. > NO_EXPORT is not safe to be used while trying to traffic engineer but > maintain global connectivity. Correct. > NO_EXPORT is only safe for more specific prefixes, as long as there is a > less specific prefix that is still usuable. Correct. > NO_EXPORT to a large provider with potential large numbers of single homed > BGP customers who may not be taking a 0/0 (in an attempt to use SAV?) is > probably not a good idea Correct. > NO_EXPORT to large providers raises the probablity of there being sites > who multihome to only those, therefore NO_EXPORT to multiple large > providers is almost certainly dangerous. Correct. Which leaves the question of why F, and now K, appear to be trying to do it. -Bill
On 31-Oct-2005, at 17:49, Bill Woodcock wrote:
Which leaves the question of why F, and now K, appear to be trying to do it.
F's covering prefix, 192.5.5.0/24, is advertised to peers of F-root local nodes with NO_EXPORT. 192.5.5.0/24 is advertised to peers of AS 3557 without NO_EXPORT. A shorter prefix, 192.5.4.0/23, is also advertised without NO_EXPORT from AS 3557. In the cases where local nodes of F advertise 192.5.5.0/24 with NO_EXPORT to ASes which also provide transit for 3557, we have taken care to ensure that local policy in those ASes causes the 3557 route (a customer route) always to be selected in preference to that received from the local node (a peer route). In these cases the local node provides local backup service. A description of F's deployment strategy can be found in <http:// www.isc.org/pubs/tn/isc-tn-2003-1.html>. Joe
Randy's description of the issue with NO_EXPORT is correct. It has never appeared to be particularly widespread. It is not specific to anycast. You also describe the rationale correctly by saying "it would be good if a server in Kenya did not take load from nyc". I'll expand a little more on that. K does anycast with two objectives: primarily to increase robustness of the service in the face of serious load increases, secondarily provide better service to some local areas in the Internet topology. It is the secondary objective that poses the problem. We operate "local nodes" which are intended to serve only a local area. This is clearly a routing problem and routing policy is clearly the responsibility of ISPs. The local ISPs should make sure that routes of local nodes do not propagate far enough to cause unwanted load. Consequently we could just announce one route without doing anything to it and "wash our hands" as far as routing and network load is concerned. Server load is not a concern here because in the case of local nodes the network will saturate far more quickly than the server. This is a very clean solution. It keeps responsibility where it belongs and does not introduce extra complexity in BOP routing. I like it. ;-) However we try to be helpful and provide tools to the ISPs by tagging the routes from such "local nodes" with NO_EXPORT and we artificially lengthen the AS-paths to the global nodes in order to make the local nodes more attractive to ASes that hear both. The latter is problematic too because it can cause non-optimal node selection but does not lead to anything worse than that. Remember that these are nothing but hints to ISPs who determine their own routing policy and are responsible for it. So if someone barks at K for this it is the wrong tree for the most part. What should we do? Stop giving the hints? Add complexity by announcing another less specific covering prefix like F does? Any better ideas? We are currently in an evaluation phase of our anycast deployment. We are taking measurements and are analysing them. Early results: http://www.ripe.net/ripe/meetings/ripe-51/presentations/pdf/ripe51-anycast_k... We are also seriously considering to reduce the number of local nodes where this is feasible. This is a good time to hear good ideas. Let us have them. Daniel PS: Bill, I hope this also answers your question on why we do this. We have been doing it for a long time too. As I said: suggestions are welcome.
mornin' daniel:
You also describe the rationale correctly by saying "it would be good if a server in Kenya did not take load from nyc". I'll expand a little more on that. K does anycast with two objectives: primarily to increase robustness of the service in the face of serious load increases, secondarily provide better service to some local areas in the Internet topology. It is the secondary objective that poses the problem. We operate "local nodes" which are intended to serve only a local area.
when it is connected to global providers, this does not work. and do not count on the hope that small local provider p0 does not pass the marked prefix to a global provider - that's like saying 1918 prefixes will never leak. [ note: i have friends in kenya, and would be happy if this stuff would work well. this does not mean that i will pretend that it does. ]
This is clearly a routing problem and routing policy is clearly the responsibility of ISPs.
as you have deployed something that participates in the global routing mesh, this ploy should be embarrassing. as what you have deployed attempts to take clever advantage of a kinky, and not widely used (guess why!), feature of the global routing system, you would be polite to take responsibility for what happens.
What should we do?
at the core of the problem is the assumption that anycast will find the closest server. thus, if the service is deployed in many places in the topology and geography, each will only take local load. it is critical to note that this relies on an assumption of *very* topologically and geographically rich deployment. it also gets bitten by the abundance of providers with linear topologies with large geographic reach (but this will be a short-term problem as tony hain from cisco plans to abolish us as part of cisco's ipv6 marketing campaign:-).
Add complexity by announcing another less specific covering prefix like F does?
although this further descends into the dangerous purgatory of cleverness, you would probably be advised to do something such as this. otherwise, even if k connected directly to all of multi-homed t0's upstreams, by default, none of them would give t0 your prefix because it is poisoned. my naive view of your current deployment means that k can not be seen from any multi-homed sites unless one or more of their upstreams (recurse for tier-n) is even more clever and implements "t0 is our customer and we ignore NO_EXPORT toward customers," thus piling on yet another bit of cleverness, the implications of which we can discover in the next level of purgatory. randy
On 01.11 05:41, Randy Bush wrote:
mornin' daniel:
ev'nin randy: Of course the NCC takes resposibility for the K anycast deployment including the way we announce BGP routes to K. We also clearly describe and announce what we do. We cannot take responsibility for what others do with that routing information; we cannot because we have no control over and little way of knowing what they do. We are doing the best we can; hence this conversation. We are considering to add a covering prefix announced from global nodes relatively quickly. This should solve the particular problem and we cannot (yet) see any problems it would create. But this is more complex than the current state and thus brings us further away from salvation ;-). If there are reasons not to do this, please let us know. We are also considering seriously to treat "local" nodes and global nodes the same routing wise wherever possible. This will be done one-by-one wiht the proper announcements and concurrent measurements. My poersonal hope is that we can do this for all K nodes and thus remove all BGP cleverness that originates from us. This does not mean that all cleverness concerning K's routes would be removed though. Daniel
On Tue, 1 Nov 2005, Daniel Karrenberg wrote:
We are considering to add a covering prefix announced from global nodes relatively quickly. This should solve the particular problem and we cannot (yet) see any problems it would create. But this is more complex than the current state and thus brings us further away from salvation ;-). If there are reasons not to do this, please let us know.
We are also considering seriously to treat "local" nodes and global nodes the same routing wise wherever possible. This will be done one-by-one with the proper announcements and concurrent measurements. My personal hope is that we can do this for all K nodes and thus remove all BGP cleverness that originates from us. This does not mean that all cleverness concerning K's routes would be removed though.
Here's what we do on the PCH anycast network, to derive our "anycast cleverness" from the network topology rather than from BGP hacks. It seems to work. Other ways of doing this are presumably valid as well: We've got four global nodes (nodes that have transit, rather than just peering), in Hong Kong, Palo Alto, Ashburn, and London. We get transit from the same transit providers in all four locations. Our transit providers hot potato to us, so as long as their peers hot potato to them, those who can't get to one of our local nodes should get to the topologically closest global node (topology, of course, does not always match geography). We've then got a much larger number of local nodes, which look just like the global nodes except that they don't have any BGP transit. They're all at exchange points, they all peer openly, and we encourage our peers to peer with us at all common locations and to treat us like a normal peer. That means they don't announce us to their transit providers, but do generally announce us to their customers. Since this is the same thing that pretty much every network that's peering either globally or locally does, this doesn't require anything non-standard or hackish. Our peers and their customers see us at whatever set of nodes they connect to. Those who don't peer with us, and aren't customers of any networks who do, see us at a more limited set of locations. This does mean we have to turn down offers of donated BGP transit from time to time, and we did have to turn off one peer who decided we were a good cause and was determined to give us transit whether we wanted it or not. There have been a few times when we've found our routes being leaked (fortunately by networks with considerably longer AS paths to most of the world than our transit providers) and have had to turn down sessions until the filters got fixed. These are the same issues we had at a real intra-connected global network I used to work for; it's not anything special about anycast. The cases of suboptimal routing we see this leading to generally stem from networks that are unwilling to peer with us in their home markets but are eager to peer with us somewhere else. Their generally suggested way around this is that we should buy transit from them, and our response is that we aren't going to pay them to accept free services from us. Again, that's really a standard peering politics issue, and has nothing to do with anycast (and we're still generally closer to them than we would be with an arbitrary unicast location). The remaining theoretical concern that might be solved by NO_EXPORT would be a situation where a network closer to one of our global nodes was getting transit from a poorly connected network close to one of our local nodes. However, simple economics works against that. Connectivity to or from poorly connected regions tends to be really expensive, so it's unlikely that anybody is going to be hauling traffic over those links when they don't have to. My impression (and I think this is what Bill was saying yesterday as well) is that most of the weird routing problems that come up with anycast are a result of treating anycast as something different and special, which it doesn't need to be. -Steve
On Tue, 1 Nov 2005, Randy Bush wrote:
my naive view of your current deployment means that k can not be seen from any multi-homed sites unless one or more of their upstreams (recurse for tier-n) is even more clever and implements "t0 is our customer and we ignore NO_EXPORT toward customers," thus piling on yet another bit of cleverness, the implications of which we can discover in the next level of purgatory.
assuming we are talking about the well known community no-export, then i understand the problem. a better solution would be to peer only the anycast node, such that transit providers continue to propagate to the global internet (minus the peers seeing the shorter path). for wider distribution within the region, possibly using a transit provider for the anycast and use communities supported by the upstream to restrict announcement to its peers or upstreams or am i naive too? Steve
On 1-Nov-2005, at 14:19, Stephen J. Wilcox wrote:
or am i naive too?
I think you underestimate the tendencies of ISPs all over the world to leak peering routes towards their transit providers. Contrary to popular belief, leaks through peers in remote regions do not always result in huge AS_PATHs which are never selected by the rest of the network. For example, some of the most remote and poorly- connected ISPs that F is announced to from local nodes are transit customers of international, default-free carriers. Joe
On Tue, 1 Nov 2005, Joe Abley wrote:
On 1-Nov-2005, at 14:19, Stephen J. Wilcox wrote:
or am i naive too?
I think you underestimate the tendencies of ISPs all over the world to leak peering routes towards their transit providers.
Contrary to popular belief, leaks through peers in remote regions do not always result in huge AS_PATHs which are never selected by the rest of the network. For example, some of the most remote and poorly- connected ISPs that F is announced to from local nodes are transit customers of international, default-free carriers.
ok sure, but is this not just normal transit issues, these are not special because they are a) anycast b) root-servers? if any networks peers leak they should be reprimanded Steve
On 1-Nov-2005, at 15:15, Stephen J. Wilcox wrote:
ok sure, but is this not just normal transit issues, these are not special because they are a) anycast b) root-servers?
You're right -- these are normal issues that any multi-homed AS might see. The effectiveness of knuckle-rapping after the fact is not necessarily great, however, with respect to service uptime. Anybody who cares about their service availability might think around the subject and take appropriate steps to mitigate or avoid leaks. Alternatively, they might well consider the cure worse than the disease, and live with the occasional leak instead. I think there is sound, logical reasoning that can result in both conclusions, depending on the peculiarities of the service in general and the habits of local peers in particular (with a dash of personal preference and a sprinkling of past experience). It's the thinking part that is important. Diversity in approach is good, especially in the delivery of a single, critical service. Joe
On Tue, 1 Nov 2005, Stephen J. Wilcox wrote: > ok sure, but is this not just normal transit issues, these are not special > because they are a) anycast b) root-servers? Correct. > if any networks peers leak they > should be reprimanded Well, I might phrase that in a different way, but yes, if they do it enough to matter, the market will teach them the error of their ways. -Bill
Contrary to popular belief, leaks through peers in remote regions do not always result in huge AS_PATHs which are never selected by the rest of the network. For example, some of the most remote and poorly- connected ISPs that F is announced to from local nodes are transit customers of international, default-free carriers.
bingo! thanks to, among other techno-colonialist games, the usaid leland intiative, entire countries are direct non-bgp customers of a local telco monopoly which is a customer of a european or american telco monopoly. randy
We have considered the options carefully and decided to announce a covering /23 to get around the problem spotted by Randy and the folks in Oregon. We considered the /24+/25 soloution but decided against that because it makes little sense when we are considering to eventually remove as much of the BG cleverness as possible. See the attached announcement for details. Thanks to all people who took the time to make suggestions either privately or on list(s). This was both operational and useful. ;-) Daniel
On Mon, Oct 31, 2005 at 12:19:58PM -1000, Randy Bush wrote: Hi,
this implies that a non-trivial part of the net can not see anycast services for which some of the servers are marking their announcements as NO_EXPORT.
Is it an idea to have anycasted instances using NO_EXPORT announce /25's instead of /24's? That way you would still have global connectivity for the /24 and still respect the NO_EXPORT. Another possibility is for $LARGE_ISP to localpref the NO_EXPORTED down to $LOW value (some of you will go 'doh' now). Thanks, -- Sabri please do not throw salami pizza away
Is it an idea to have anycasted instances using NO_EXPORT announce /25's instead of /24's?
many many folk filter on /24, so the /25 would not be seen.
Another possibility is for $LARGE_ISP to localpref the NO_EXPORTED down to $LOW value
and then how will the down-preffed prefix be seen within $large_isp? local-pref is a very heavy hammer. randy, at the clue edge i guess
Maybe I'm missing something, but the core issue is that the NO- EXPORT'ed anycast instance has a higher localpref inside the AS it's being advertised to, and as such supressing the non-NO_EXPORT'ed prefix. The "exportable" prefix gets suppressed at a point on the network such that the peering routers never see it, so it never makes it out of that AS. Advertising the NO-EXPORT as a /25 (or two /25s) would serve the same purpose as tagging it NO-EXPORT, because as you say most peers filter on the /25. Incidentally it would obviate the need to assign it a higher localpref because it's a shorter prefix. However, unless I'm misunderstanding something, the /25 would not preempt the /24 advertisement, so the /24 would still make it out of the AS. Just make sure you don't have anything numbered x.x.x.127 on the block... On Nov 2, 2005, at 8:42 AM, Randy Bush wrote:
Is it an idea to have anycasted instances using NO_EXPORT announce /25's instead of /24's?
many many folk filter on /24, so the /25 would not be seen.
Isn't that the point? The existing /24 is tagged NO-EXPORT after all...
Another possibility is for $LARGE_ISP to localpref the NO_EXPORTED down to $LOW value
and then how will the down-preffed prefix be seen within $large_isp? local-pref is a very heavy hammer.
randy, at the clue edge i guess
-C
Maybe I'm missing something, but the core issue is that the NO- EXPORT'ed anycast instance has a higher localpref inside the AS it's being advertised to, and as such supressing the non-NO_EXPORT'ed prefix. The "exportable" prefix gets suppressed at a point on the network such that the peering routers never see it, so it never makes it out of that AS.
it willl lose the best path war at the point it is down-preffed. if that is done at the entrance, then the route is useless. if it is done at some intermediate points, it's hell to create and maintain a consistent net-wide config. randy
participants (9)
-
Bill Woodcock -
Chris Woodfield -
Daniel Karrenberg -
Joe Abley -
Joe Maimon -
Randy Bush -
Sabri Berisha -
Stephen J. Wilcox -
Steve Gibbard