Quick question about inbound route-selection
Howdy, Keep in mind I am basing this 'idea' off of fixed orbit's data which can sometimes be a bit out of date, etc. (in theory, and based upon number of peers, data): If you have a network with these upstream connections to the Internet you should see inbound traffic utilization in this order: AS Name --------- 3356 Level3 7018 ATT 3549 Global Crossing 4323 Time Warner Telecom 10796 TimeWarnerCable/RR I am trying to determine why I am seeing it in this order: 3356 Level3 4323 Time Warner Telecom 3549 Global Crossing 10796 TimeWarnerCable/RR 7018 ATT I suppose there is a certain level of convergence where these providers inter-connect, and also the source network of the traffic plays a big part of it, i.e. if most of the sources are directly connected to Level3, etc. I am mainly wondering why 7018 sends us such a little amount compared to even 10796. Also, with the providers already connected, if we added a new one, which one would (in your opinion) benefit us the most on spreading the inbound traffic out better? I realize that we can use communities, and prepends to control the inbound flow, I am just speaking from a purely natural standpoint. thanks, -Drew
On Thu, Jul 16, 2009 at 09:45:24AM -0400, Drew Weaver wrote:
Howdy,
Keep in mind I am basing this 'idea' off of fixed orbit's data which can sometimes be a bit out of date, etc.
Understatement. [snip]
I realize that we can use communities, and prepends to control the inbound flow, I am just speaking from a purely natural standpoint.
Since your inbound is someone else's outbound, presuming any kind of "natural flow" without accounting for the remote end's sending policies is unreasonable. Cheers, Joe -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
On Thu, Jul 16, 2009 at 09:45:24AM -0400, Drew Weaver wrote:
I realize that we can use communities, and prepends to control the inbound flow, I am just speaking from a purely natural standpoint.
I don't know where people are getting this "natural" bgp path selection concept from, but it is completely misguided and needs to be corrected before any more misinformation is spread. On the modern Internet, the vast majority of paths look pretty much the same across any major networks, even via metrics as irrelevent as "as-path hop length". A "natural" path selection would be based on such garbage data as "who has the lowest router id", "which network has the smallest numeric value in their igp cost scheme when setting MEDs", or the wonderfully non-deterministic "which path has been up the longest". I recently heard some complaints from a bunch of customers who were upset that they "couldn't send us any traffic using natural bgp", and they didn't want to "artificially alter bgp's best path selection" with route-maps and localprefs. After trying to explain that there was really no such thing as "natural bgp", and having it fall on deaf ears, I went to take a look at their routing tables to see what they were talking about. It turned out that we were sending them MED values based on our IGP costs while their other networks were sending them 0's, which was making the tie-breaking decision go the other way for the vast majority of the routes. The BGP best path selection algorithm is really nothing special, it provides almost no useful data for selecting between major well connected networks on the modern Internet, and if you refuse to alter any attributes you're going to end up with a giant mess of path selection which would be better accomplished by asking a magic 8ball. As for trying to determine where your inbound traffic is coming from by looking at natural bgp, this is absolutely impossible to do correctly. First off, your inbound is someone else's outbound, and the person sending the traffic outbound is in complete and total control. The vast majority of the traffic on the Internet is being picked by local-prefs based on policies like "what does this make/cost me monetarily" or "which major networks can I grab in a simple as-path regexp to balance some traffic". But even if you ignore all of that, the "natural" path selection is based on criteria which is specific to the other network or even to a specific session which you can't possibly know about remotely (e.g. their router id). -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
As for trying to determine where your inbound traffic is coming from by looking at natural bgp, this is absolutely impossible to do correctly. First off, your inbound is someone else's outbound, and the person sending the traffic outbound is in complete and total control. The vast majority of the traffic on the Internet is being picked by local-prefs based on policies like "what does this make/cost me monetarily" or "which major networks can I grab in a simple as-path regexp to balance some traffic". But even if you ignore all of that, the "natural" path selection is based on criteria which is specific to the other network or even to a specific session which you can't possibly know about remotely (e.g. their router id).
Another way to say what Richard is getting at (which was full of good information) is: Just because you aren't modifying what your BGP process sees, at this stage of the Internet's maturity, it is safe to assume almost everyone else is. Therefore, rather than pray for BGP to make a logical selection, even though its *probably* being fed prefs based on other people's engineering, you should take charge of the parts you can. HTH, Deepak Jain AiNET
On Thu, Jul 16, 2009 at 06:32:32PM -0400, Deepak Jain wrote:
As for trying to determine where your inbound traffic is coming from by looking at natural bgp, this is absolutely impossible to do correctly. First off, your inbound is someone else's outbound, and the person sending the traffic outbound is in complete and total control. The vast majority of the traffic on the Internet is being picked by local-prefs based on policies like "what does this make/cost me monetarily" or "which major networks can I grab in a simple as-path regexp to balance some traffic". But even if you ignore all of that, the "natural" path selection is based on criteria which is specific to the other network or even to a specific session which you can't possibly know about remotely (e.g. their router id).
I would actually disagree with that and go one step further. Look at content providers. They're not concerned about best path. They're not even concerned about shortest path. Since bandwidth consuming services are what they provide, they're interested in cheapest path as much as they are the shortest path.
Another way to say what Richard is getting at (which was full of good information) is:
Just because you aren't modifying what your BGP process sees, at this stage of the Internet's maturity, it is safe to assume almost everyone else is. Therefore, rather than pray for BGP to make a logical selection, even though its *probably* being fed prefs based on other people's engineering, you should take charge of the parts you can.
Take the traffic shaping products. They completely override the normal BGP mechanisms and force traffic out a given circuit. So as long as there is a usable route down that interface, it will get used whether the neighbor wants it or not. The long and short of it is that via MEDS, prepending, and your neighbor's community policies, you can *hint* where you want traffic to come in but ultimately you may have very little say in the matter. (Community exchanges are probably the best mechanism since the existance of them in your peer's network means they will be most likely to honor your hints.) As Deepak indicated, don't rely on the originally the protocol's best effort. Take control of your own world wherever you can. It's the only way to ensure a good measure of predictability. -Wayne --- Wayne Bouchard web@typo.org Network Dude http://www.typo.org/~web/
On Jul 16, 2009, at 7:05 PM, Wayne E. Bouchard wrote:
On Thu, Jul 16, 2009 at 06:32:32PM -0400, Deepak Jain wrote:
As for trying to determine where your inbound traffic is coming from by looking at natural bgp, this is absolutely impossible to do correctly. First off, your inbound is someone else's outbound, and the person sending the traffic outbound is in complete and total control. The vast majority of the traffic on the Internet is being picked by local- prefs based on policies like "what does this make/cost me monetarily" or "which major networks can I grab in a simple as-path regexp to balance some traffic". But even if you ignore all of that, the "natural" path selection is based on criteria which is specific to the other network or even to a specific session which you can't possibly know about remotely (e.g. their router id).
I would actually disagree with that and go one step further. Look at content providers. They're not concerned about best path. They're not even concerned about shortest path. Since bandwidth consuming services are what they provide, they're interested in cheapest path as much as they are the shortest path.
s/content providers/some content providers, some eyeball networks, some corporate networks, and some of just about every type of network/ OTOH: Some content providers measure latency, packet loss, and throughput and react accordingly to ensure the end users get adequate performance, no matter the path or cost. Interestingly, in either case, the 'natural' BGP selection is a poor predictor of inbound traffic. But then we already knew that. :) -- TTFN, patrick
Drew,
(in theory, and based upon number of peers, data): If you have a network with these upstream connections to the Internet you should see inbound traffic utilization in this order:
AS Name --------- 3356 Level3 7018 ATT 3549 Global Crossing 4323 Time Warner Telecom 10796 TimeWarnerCable/RR
In short (and not to repeat what others have said, but simply point out a different reason), the Internet is an example of a large anisotropic system. The implication for 'inbound traffic distribution' will not only depend in Neighbor-AS policies (upstream of your own AS, mind you), but equally (if not moreso) on the traffic matrix your users (or host systems, applications, etc) generate as a consequence of their activities. Almost entire decoupled from "pull heavy," "push heavy," or so-called "balanced" (in the bits/sec sense) traffic patterns, quite simply, "what you're doing" will influence the distribution. This will change over time, too, especially if the source of the traffic reaching you via 3356 experiences a change in a business relationship (174 and 3356 de-peer, again).
I am trying to determine why I am seeing it in this order:
3356 Level3 4323 Time Warner Telecom 3549 Global Crossing 10796 TimeWarnerCable/RR 7018 ATT
Netflow or sflow will likely shed light on "why?" with a higher degree of certainty than AS-AS adjacencies. Knowing both the rendezvous mechanism and the application inducing the flow(s) would be the first step to answering "why did that reach us via (3) and not some other edge we know exists?" Additionally, how apps discover both the network and content is a topic being explored by several researchers and operators, and may be relevant to your question. You may be able to tease out further data by considering these mechanisms as you go about monitoring. Dave Plonka is working on as much, but as of yet, I can't find a paper - only presentations [*]. Best, -Tk [*]: "Rendezvous-based Network Traffic Analysis" - http://www.cio.wisc.edu/events/lockdown/09/presentations.aspx#plonka
participants (7)
-
Anton Kapela -
Deepak Jain -
Drew Weaver -
Joe Provo -
Patrick W. Gilmore -
Richard A Steenbergen -
Wayne E. Bouchard