Load Balancing Multiple DS3s (outgoing) on a 7500
Does anyone know of an article, or documentation regarding load balancing the traffic on 3 or more FastEthernet interfaces on the outgoing direction? Right now we're running BGP internally, and the routes that are being chosen based upon the final BGP decision step or what I like to call the 'IP address tie breaker' which is not always optimal. We have a cisco 7500 that is connected to 4 other Cisco 7500s which each have 45Mbps ds3s to the Internet, we would like to load balance the outgoing traffic across all 4 of these 7500s, can anyone shine any advice my way? I noticed that there are instructions on Cisco's site regarding doing LB on 12000s. Anyways thanks in advance ;-) -Drew
On Mar 12, 2004, at 10:39 PM, Drew Weaver wrote:
Does anyone know of an article, or documentation regarding load balancing the traffic on 3 or more FastEthernet interfaces on the outgoing direction? Right now we're running BGP internally, and the routes that are being chosen based upon the final BGP decision step or what I like to call the 'IP address tie breaker' which is not always optimal. We have a cisco 7500 that is connected to 4 other Cisco 7500s which each have 45Mbps ds3s to the Internet, we would like to load balance the outgoing traffic across all 4 of these 7500s, can anyone shine any advice my way? I noticed that there are instructions on Cisco's site regarding doing LB on 12000s.
Load balancing with BGP is the same on any cisco router. Are you doing BGP with the routers on the other side of those DS3s? If you are, you will need their help in load balancing properly. Get them to allow you peering with a loopback interface and use equal cost static routes to do the load balancing to that loopback interface. -- TTFN, patrick
Patrick, I suspect that each FE goes to a different AS... On 3/12/04 7:27 PM, "Patrick W.Gilmore" <patrick@ianai.net> wrote:
On Mar 12, 2004, at 10:39 PM, Drew Weaver wrote:
Does anyone know of an article, or documentation regarding load balancing the traffic on 3 or more FastEthernet interfaces on the outgoing direction? Right now we're running BGP internally, and the routes that are being chosen based upon the final BGP decision step or what I like to call the 'IP address tie breaker' which is not always optimal. We have a cisco 7500 that is connected to 4 other Cisco 7500s which each have 45Mbps ds3s to the Internet, we would like to load balance the outgoing traffic across all 4 of these 7500s, can anyone shine any advice my way? I noticed that there are instructions on Cisco's site regarding doing LB on 12000s.
Load balancing with BGP is the same on any cisco router.
Are you doing BGP with the routers on the other side of those DS3s? If you are, you will need their help in load balancing properly. Get them to allow you peering with a loopback interface and use equal cost static routes to do the load balancing to that loopback interface.
On Mar 12, 2004, at 11:24 PM, joe mcguckin wrote:
I suspect that each FE goes to a different AS...
Yeppers. We've been corresponding privately, and you got it right (unlike me). He'll be okie. It's just a little difficult for BGP to "load balance" outbound bits when the bulk of the Internet these days is 2 AS hops away from each of four upstreams. Not impossible, but it doesn't happen by default either. -- TTFN, patrick
He'll be okie. It's just a little difficult for BGP to "load balance" outbound bits when the bulk of the Internet these days is 2 AS hops away from each of four upstreams. Not impossible, but it doesn't happen by default either.
I used to do this ages ago, I did it by setting MEDs on the incoming BGP prefixes, in my route-maps I arbitrarily gave some nets (/8s or smaller) lower med on one feed and higher on the others to influence path selection. I shy away from anything but the gentlest of tweaks so I personally wouldnt mess with as-path, localpref, weight etc Steve
On Mar 13, 2004, at 4:57 PM, Stephen J. Wilcox wrote:
He'll be okie. It's just a little difficult for BGP to "load balance" outbound bits when the bulk of the Internet these days is 2 AS hops away from each of four upstreams. Not impossible, but it doesn't happen by default either.
I used to do this ages ago, I did it by setting MEDs on the incoming BGP prefixes, in my route-maps I arbitrarily gave some nets (/8s or smaller) lower med on one feed and higher on the others to influence path selection.
I shy away from anything but the gentlest of tweaks so I personally wouldnt mess with as-path, localpref, weight etc
Yeah, probably a good idea not to use Weights, but not sure about AS_PATH. Nothing wrong with a prepend here or there, IMHO. :) But also nothing wrong with setting the MEDs if you like. Just be fore to have "always compare MED" on, or MEDs between multiple providers are not useful (which you obviously had set or this wouldn't work). I kinda like setting the origin code. No one pays attention to it, but it is in the selection criteria. that way you can use MEDs from the same provider and still influence routes between providers. -- TTFN, patrick
On 12 Mar 2004, at 23:24, joe mcguckin wrote:
Patrick,
I suspect that each FE goes to a different AS...
In that case, sample/count outbound traffic volumes by (prefix/AS/AS_PATH/something), sort the resulting list, and develop an import policy based on the top N entries which shares the traffic by tweaking some other attribute to avoid the last-resort tie-break. Or bypass the measurement part, and make wild guesses about where your traffic is going, and apply an import policy based on that. Either way, lather, rinse, repeat. There might be something relevant in the slot I did in this tutorial in Richmond Hill: http://www.nanog.org/mtg-0206/te.html Joe
On Fri, Mar 12, 2004 at 11:37:25PM -0500, Joe Abley wrote:
On 12 Mar 2004, at 23:24, joe mcguckin wrote:
I suspect that each FE goes to a different AS...
In that case, sample/count outbound traffic volumes by (prefix/AS/AS_PATH/something), sort the resulting list, and develop an import policy based on the top N entries which shares the traffic by tweaking some other attribute to avoid the last-resort tie-break.
The tool "ehnt" is pretty useful for generating a "top" style list of ASes in order of the amount of traffic you're sending their way. By the way, w/r/t to the tiebreaker stuff, note that (on Cisco devices) if you don't have "bgp bestpath compare-routerid" set, the route that was received first will be preferred. This minimizes route-flap, but can cause weird shifts in your traffic patterns when one bgp session or another goes down (credit goes to Mark Nagel for figuring out this one for me). -- "Since when is skepticism un-American? Dissent's not treason but they talk like it's the same..." (Sleater-Kinney - "Combat Rock")
On Sat, Mar 13, 2004 at 04:19:40PM -0800, Will Yardley wrote:
On Fri, Mar 12, 2004 at 11:37:25PM -0500, Joe Abley wrote:
On 12 Mar 2004, at 23:24, joe mcguckin wrote:
I suspect that each FE goes to a different AS...
In that case, sample/count outbound traffic volumes by (prefix/AS/AS_PATH/something), sort the resulting list, and develop an import policy based on the top N entries which shares the traffic by tweaking some other attribute to avoid the last-resort tie-break.
The tool "ehnt" is pretty useful for generating a "top" style list of ASes in order of the amount of traffic you're sending their way.
By the way, w/r/t to the tiebreaker stuff, note that (on Cisco devices) if you don't have "bgp bestpath compare-routerid" set, the route that was received first will be preferred. This minimizes route-flap, but can cause weird shifts in your traffic patterns when one bgp session or another goes down (credit goes to Mark Nagel for figuring out this one for me).
Also, check out: http://www.nanog.org/mtg-0206/feamster.html for some general guidelines, pitfalls, etc. (The paper linked from the presentation recently appeared in ACM Sigcomm CCR.) -Nick
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Joe Abley wrote: | | | On 12 Mar 2004, at 23:24, joe mcguckin wrote: | |> Patrick, |> |> I suspect that each FE goes to a different AS... | | | In that case, sample/count outbound traffic volumes by | (prefix/AS/AS_PATH/something), sort the resulting list, and develop an | import policy based on the top N entries which shares the traffic by | tweaking some other attribute to avoid the last-resort tie-break. | | Or bypass the measurement part, and make wild guesses about where your | traffic is going, and apply an import policy based on that. Either way, | lather, rinse, repeat. | | There might be something relevant in the slot I did in this tutorial in | Richmond Hill: | | http://www.nanog.org/mtg-0206/te.html | And products from folks like Proficient Networks and Routescience can automate the process for you. - -- ========= bep -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (MingW32) iD8DBQFAVgGrE1XcgMgrtyYRAo3xAJ4qwszZ/lXxMeMJ5jF2OD9LDaMR/QCeOjz+ a8Mzb383mIOoEE2J0IzVq+I= =4QaS -----END PGP SIGNATURE-----
Hi Drew - We have 6 backbones distributed across two 7507s and we messed around with a lot of different ways to make this happen. MEDs, Weights, manual BGP configurations every time one of the connections would get overloaded (even at 2am), you name it - we tried it, and in the end we determined that we needed something that could keep an eye on everything and do it automatically within guidelines I had set. In the end, we headed the route of performance-based routing optimization hardware. After testing many different vendors, we choose the RouteScience PathControl box to make my life (as well as the life of my lead backbone engineer) much, much simpler. About a month or two ago, there was quite a discussion on route-optimization hardware on the list including a lot of different ideas. If you do a search on the list for RouteScience or route optimization, you should hit the core of the discussion around the different platforms. If you need more info, feel free to contact me off-list. On Fri, 12 Mar 2004 22:39:16 -0500 Drew Weaver <drew.weaver@thenap.com> wrote:
Does anyone know of an article, or documentation regarding load balancing the traffic on 3 or more FastEthernet interfaces on the outgoing direction? Right now we're running BGP internally, and the routes that are being chosen based upon the final BGP decision step or what I like to call the 'IP address tie breaker' which is not always optimal. We have a cisco 7500 that is connected to 4 other Cisco 7500s which each have 45Mbps ds3s to the Internet, we would like to load balance the outgoing traffic across all 4 of these 7500s, can anyone shine any advice my way? I noticed that there are instructions on Cisco's site regarding doing LB on 12000s.
Anyways thanks in advance ;-)
-Drew
****************************************** Richard J. Sears Vice President American Digital Network ---------------------------------------------------- rsears@adnc.com http://www.adnc.com ---------------------------------------------------- 858.576.4272 - Phone 858.427.2401 - Fax ---------------------------------------------------- I fly because it releases my mind from the tyranny of petty things . . "Work like you don't need the money, love like you've never been hurt and dance like you do when nobody's watching."
Drew, Something that was just released that you might be interested in if you haven't already found an alternate solution. http://www.cisco.com/en/US/products/sw/iosswrel/ps5207/products_feature_guid... It's a new feature in 12.3(8)T. Rodney On Fri, Mar 12, 2004 at 10:39:16PM -0500, Drew Weaver wrote:
Does anyone know of an article, or documentation regarding load balancing the traffic on 3 or more FastEthernet interfaces on the outgoing direction? Right now we're running BGP internally, and the routes that are being chosen based upon the final BGP decision step or what I like to call the 'IP address tie breaker' which is not always optimal. We have a cisco 7500 that is connected to 4 other Cisco 7500s which each have 45Mbps ds3s to the Internet, we would like to load balance the outgoing traffic across all 4 of these 7500s, can anyone shine any advice my way? I noticed that there are instructions on Cisco's site regarding doing LB on 12000s.
Anyways thanks in advance ;-)
-Drew
participants (10)
-
Bruce Pinsky -
Drew Weaver -
Joe Abley -
joe mcguckin -
Nick Feamster -
Patrick W.Gilmore -
Richard J. Sears -
Rodney Dunn -
Stephen J. Wilcox -
Will Yardley