From time to time some have posted questions asking if BGP load balancers such as the old Routescience Pathcontrol device are still around, and if not what have others found to replace that function. I have used the Routescience device with much success 10 years ago when it first came on the market, but since then a full BGP feed has become much larger, Routescience has been bought by Avaya, then discontinued, and other competitors such as Sockeye, Netvmg have been acquired by other companies.
Doing some research on how load balancing can be accomplished in 2011, I have come across Cisco's performance routing feature, and features from load balancing companies such as F5's Link Controller. I have always found BGP to be easy to work with, and an elegant, simple solution to load balancing using a route-reflector configuration in which one BGP client (Routescience Pathcontrol in my background) learns the best route to destination networks, and then announces that best route to BGP border routers using common and widely understood BGP concepts such as communities and local pref, and found this to lead to a deterministic Internet routing architecture. This required a knowledge only of IETF standards (common BGP concepts and configurations), required no specialized scripting, or any other knowledge lying outside IETF boundaries, and it seemed reasonable to expect that network engineers should eagerly and enthusiastically want to master this technology, just as any other technology must be mastered to run high availability networks. So I am wondering if anyone has experience with implementing load balancing across multiple ISP links in 2011, and if there have been any comparisons between IETF standards-based methods using BGP, and other proprietary methods which may use a particular vendor's approach to solving the same problem, but involves some complexity with more variables to be plugged in to the architecture. David ________________________________ This communication, together with any attachments or embedded links, is for the sole use of the intended recipient(s) and may contain information that is confidential or legally protected. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, dissemination, distribution or use of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by return e-mail message and delete the original and all copies of the communication, along with any attachments or embedded links, from your system.
I've asked several times about this in the past; although I learned quickly to stop asking. It seems that the consensus has generally been that the best way to handle traffic engineering in networks where you have multiple full-feed up-streams is to do it manually (i.e. set preference for your top N AS/prefix destinations) or don't do it at all (let BGP figure it out..?). Suggesting that a "route optimization system" has any value generally makes people cranky. Thanks, -Drew -----Original Message----- From: Holmes,David A [mailto:dholmes@mwdh2o.com] Sent: Wednesday, December 14, 2011 2:07 PM To: nanog@nanog.org Subject: Multiple ISP Load Balancing
From time to time some have posted questions asking if BGP load balancers such as the old Routescience Pathcontrol device are still around, and if not what have others found to replace that function. I have used the Routescience device with much success 10 years ago when it first came on the market, but since then a full BGP feed has become much larger, Routescience has been bought by Avaya, then discontinued, and other competitors such as Sockeye, Netvmg have been acquired by other companies.
Doing some research on how load balancing can be accomplished in 2011, I have come across Cisco's performance routing feature, and features from load balancing companies such as F5's Link Controller. I have always found BGP to be easy to work with, and an elegant, simple solution to load balancing using a route-reflector configuration in which one BGP client (Routescience Pathcontrol in my background) learns the best route to destination networks, and then announces that best route to BGP border routers using common and widely understood BGP concepts such as communities and local pref, and found this to lead to a deterministic Internet routing architecture. This required a knowledge only of IETF standards (common BGP concepts and configurations), required no specialized scripting, or any other knowledge lying outside IETF boundaries, and it seemed reasonable to expect that network engineers should eagerly and enthusiastically want to master this technology, just as any other technology must be mastered to run high availability networks. So I am wondering if anyone has experience with implementing load balancing across multiple ISP links in 2011, and if there have been any comparisons between IETF standards-based methods using BGP, and other proprietary methods which may use a particular vendor's approach to solving the same problem, but involves some complexity with more variables to be plugged in to the architecture. David ________________________________ This communication, together with any attachments or embedded links, is for the sole use of the intended recipient(s) and may contain information that is confidential or legally protected. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, dissemination, distribution or use of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by return e-mail message and delete the original and all copies of the communication, along with any attachments or embedded links, from your system.
On Wed, Dec 14, 2011 at 2:28 PM, Drew Weaver <drew.weaver@thenap.com> wrote:
I've asked several times about this in the past; although I learned quickly to stop asking.
It seems that the consensus has generally been that the best way to handle traffic engineering in networks where you have multiple full-feed up-streams is to do it manually (i.e. set preference for your top N AS/prefix destinations) or don't do it at all (let BGP figure it out..?).
seems the feeling is that if you have multiple full feeds and need to loadshare, you really don't want (in most cases) ispa=500mbps + ispb=500mbps. you really want destinationA to be reached across the 'best path' (best ... in some form, distance? packetdrop%? jitter? cost?) you'll most likely have to tweak things in order to achieve what you want since only distance is really used in the stock bgp calculation (distance by as-hops, presuming you don't listen to closely to med from your providers)
Suggesting that a "route optimization system" has any value generally makes people cranky.
ha! :) -chris
Thanks, -Drew
-----Original Message----- From: Holmes,David A [mailto:dholmes@mwdh2o.com] Sent: Wednesday, December 14, 2011 2:07 PM To: nanog@nanog.org Subject: Multiple ISP Load Balancing
From time to time some have posted questions asking if BGP load balancers such as the old Routescience Pathcontrol device are still around, and if not what have others found to replace that function. I have used the Routescience device with much success 10 years ago when it first came on the market, but since then a full BGP feed has become much larger, Routescience has been bought by Avaya, then discontinued, and other competitors such as Sockeye, Netvmg have been acquired by other companies.
Doing some research on how load balancing can be accomplished in 2011, I have come across Cisco's performance routing feature, and features from load balancing companies such as F5's Link Controller. I have always found BGP to be easy to work with, and an elegant, simple solution to load balancing using a route-reflector configuration in which one BGP client (Routescience Pathcontrol in my background) learns the best route to destination networks, and then announces that best route to BGP border routers using common and widely understood BGP concepts such as communities and local pref, and found this to lead to a deterministic Internet routing architecture. This required a knowledge only of IETF standards (common BGP concepts and configurations), required no specialized scripting, or any other knowledge lying outside IETF boundaries, and it seemed reasonable to expect that network engineers should eagerly and enthusiastically want to master this technology, just as any other technology must be mastered to run high availability networks.
So I am wondering if anyone has experience with implementing load balancing across multiple ISP links in 2011, and if there have been any comparisons between IETF standards-based methods using BGP, and other proprietary methods which may use a particular vendor's approach to solving the same problem, but involves some complexity with more variables to be plugged in to the architecture.
David
________________________________ This communication, together with any attachments or embedded links, is for the sole use of the intended recipient(s) and may contain information that is confidential or legally protected. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, dissemination, distribution or use of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by return e-mail message and delete the original and all copies of the communication, along with any attachments or embedded links, from your system.
seems the feeling is that if you have multiple full feeds and need to loadshare, you really don't want (in most cases) ispa=500mbps + ispb=500mbps.
you really want destinationA to be reached across the 'best path' (best ... in some form, distance? packetdrop%? jitter? cost?) you'll most likely have to tweak things in order to achieve what you want since only distance is really used in the stock bgp calculation (distance by as-hops, presuming you don't listen to closely to med from your providers)
Yes, but performance from your network to $destination_AS via $ISPx can be variable and how do you know when it changes before someone starts complaining? There are traditionally two pieces involved with optimization. 1) "Cost" (Commitment/oversubscribe management and monitoring) 2) "Performance" Usually "cost control" is #1 so systems like that are configured so that as long as the traffic isn't breaking your commits or filling your pipes they will then optimize X number of your top prefixes for performance (based on what the system can see). The performance aspect is generally just sending basic probes in all directions towards a destination host and seeing which ones reply the fastest. Although obviously this only impacts traffic outbound from your AS. -Drew
Hi David, You might want to take a look at work happening in ALTO (http://tools.ietf.org/wg/alto/) Regards, Jeff -----Original Message----- From: Holmes,David A [mailto:dholmes@mwdh2o.com] Sent: Wednesday, December 14, 2011 11:07 AM To: nanog@nanog.org Subject: Multiple ISP Load Balancing
From time to time some have posted questions asking if BGP load balancers such as the old Routescience Pathcontrol device are still around, and if not what have others found to replace that function. I have used the Routescience device with much success 10 years ago when it first came on the market, but since then a full BGP feed has become much larger, Routescience has been bought by Avaya, then discontinued, and other competitors such as Sockeye, Netvmg have been acquired by other companies.
Doing some research on how load balancing can be accomplished in 2011, I have come across Cisco's performance routing feature, and features from load balancing companies such as F5's Link Controller. I have always found BGP to be easy to work with, and an elegant, simple solution to load balancing using a route-reflector configuration in which one BGP client (Routescience Pathcontrol in my background) learns the best route to destination networks, and then announces that best route to BGP border routers using common and widely understood BGP concepts such as communities and local pref, and found this to lead to a deterministic Internet routing architecture. This required a knowledge only of IETF standards (common BGP concepts and configurations), required no specialized scripting, or any other knowledge lying outside IETF boundaries, and it seemed reasonable to expect that network engineers should eagerly and enthusiastically want to master this technology, just as any other technology must be mastered to run high availability networks. So I am wondering if anyone has experience with implementing load balancing across multiple ISP links in 2011, and if there have been any comparisons between IETF standards-based methods using BGP, and other proprietary methods which may use a particular vendor's approach to solving the same problem, but involves some complexity with more variables to be plugged in to the architecture. David ________________________________ This communication, together with any attachments or embedded links, is for the sole use of the intended recipient(s) and may contain information that is confidential or legally protected. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, dissemination, distribution or use of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by return e-mail message and delete the original and all copies of the communication, along with any attachments or embedded links, from your system.
The best applications for analyzing paths, that I've seen, have been in-house development projects. So, admittedly, I don't have much experience with commercial products for route optimization. Projects I've seen that analyze "best" paths to Internet destinations via multiple ISPs add instrumentation to content-serving applications to log stream performance details to a database or log collection system along with a timestamp. Another database keeps a periodic log of RIB data that lists the specific next-hops out of the AS. Another log keeps a running log of UPDATEs.
From joining up all of this information, you can figure out the ISP you're taking to a destination (at a given time) and how the stream performed. Then, add some logic to inject routes to try out different next-hop ISPs for some destinations.
Then, compare the newer ISP-path to the older one and see which performs "best". Where "best" means something specific to your application (optimizing for latency, cost, etc.) Cheers, jof
On Wed, 14 Dec 2011, Holmes,David A wrote:
From time to time some have posted questions asking if BGP load balancers such as the old Routescience Pathcontrol device are still around, and if not what have others found to replace that function. I have used the Routescience device with much success 10 years ago when it first came on the market, but since then a full BGP feed has become much larger, Routescience has been bought by Avaya, then discontinued, and other competitors such as Sockeye, Netvmg have been acquired by other companies.
It's important to keep in mind what load-balancing is and isn't in this case. The terminology gap can be important because load-balancing (more accurately, load-sharing) in the context of internetwork traffic engineering is very different from load-balancing pools of servers in a data center. Some people can (and sometimes do) confuse the two, which can cause unrealistic expectations to be set :) Achieving a perfect split of network traffic across two or more upstream links rarely happens in the real world. General good practice is to put bandwidth where the network traffic wants to go, but that can be a moving target, and executives and accountants don't like those :) Traffic engineering still has a place on many networks, for a veriety of reasons (technical, financial, political, some combination of these), but as other posters have mentioned, it's often done manually, i.e. looking at Netflow reports, seeing where your traffic is going/coming from, adjusting BGP policies accordingly. Repeat as needed. Since traffic profiles can change over time, any policy tweaks that are put in place need to be reviewed periodically. Depending on how much prep work and planning you're willing to do, you can put a fairly rich set of internal BGP communities in place to control things like localpref, MEDs, selective prepends, and tagging outbound advertisements with provider-specific communities. With that kind of structure, you could control many aspects of your traffic engineering from a route server, rather than having to tinker with route policies on each outside-facing router. One caveat: If your route server crashes or has to be taken down for maintenance, the traffic patterns that were being tweaked by your policy framework could start to revert to the way the traffic would flow in its un-altered state, which could cause you some unintended headaches. jms
We have specific situations where we have successfully used the Avaya CNA tool (old Route Science Patch Control). Not for load balancing, but for sub second failover from primary to a backup paths over MPLS VPN's. This is done on our internal network where we have MPLS VPN's sometimes over multiple carriers where normal convergence times are 30 seconds to 1 minute across an external provider. It's not easy to setup initially, but once you get it setup and the kinks worked out I've been impressed with its ability to test a path and move traffic at the first hint of trouble. Jim -----Original Message----- From: Justin M. Streiner [mailto:streiner@cluebyfour.org] Sent: Wednesday, December 14, 2011 2:10 PM To: nanog@nanog.org Subject: Re: Multiple ISP Load Balancing On Wed, 14 Dec 2011, Holmes,David A wrote:
From time to time some have posted questions asking if BGP load balancers such as the old Routescience Pathcontrol device are still around, and if not what have others found to replace that function. I have used the Routescience device with much success 10 years ago when it first came on the market, but since then a full BGP feed has become much larger, Routescience has been bought by Avaya, then discontinued, and other competitors such as Sockeye, Netvmg have been acquired by other companies.
It's important to keep in mind what load-balancing is and isn't in this case. The terminology gap can be important because load-balancing (more accurately, load-sharing) in the context of internetwork traffic engineering is very different from load-balancing pools of servers in a data center. Some people can (and sometimes do) confuse the two, which can cause unrealistic expectations to be set :) Achieving a perfect split of network traffic across two or more upstream links rarely happens in the real world. General good practice is to put bandwidth where the network traffic wants to go, but that can be a moving target, and executives and accountants don't like those :) Traffic engineering still has a place on many networks, for a veriety of reasons (technical, financial, political, some combination of these), but as other posters have mentioned, it's often done manually, i.e. looking at Netflow reports, seeing where your traffic is going/coming from, adjusting BGP policies accordingly. Repeat as needed. Since traffic profiles can change over time, any policy tweaks that are put in place need to be reviewed periodically. Depending on how much prep work and planning you're willing to do, you can put a fairly rich set of internal BGP communities in place to control things like localpref, MEDs, selective prepends, and tagging outbound advertisements with provider-specific communities. With that kind of structure, you could control many aspects of your traffic engineering from a route server, rather than having to tinker with route policies on each outside-facing router. One caveat: If your route server crashes or has to be taken down for maintenance, the traffic patterns that were being tweaked by your policy framework could start to revert to the way the traffic would flow in its un-altered state, which could cause you some unintended headaches. jms E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
This is why I wish they would release it as open source or sell it to someone else, the product really did work well, the kernel in the underlying Linux is the biggest hurdle. Thanks, -Drew -----Original Message----- From: Rampley Jr, Jim F [mailto:jim.rampley@chartercom.com] Sent: Wednesday, December 14, 2011 3:42 PM To: Justin M. Streiner; nanog@nanog.org Subject: RE: Multiple ISP Load Balancing We have specific situations where we have successfully used the Avaya CNA tool (old Route Science Patch Control). Not for load balancing, but for sub second failover from primary to a backup paths over MPLS VPN's. This is done on our internal network where we have MPLS VPN's sometimes over multiple carriers where normal convergence times are 30 seconds to 1 minute across an external provider. It's not easy to setup initially, but once you get it setup and the kinks worked out I've been impressed with its ability to test a path and move traffic at the first hint of trouble. Jim -----Original Message----- From: Justin M. Streiner [mailto:streiner@cluebyfour.org] Sent: Wednesday, December 14, 2011 2:10 PM To: nanog@nanog.org Subject: Re: Multiple ISP Load Balancing On Wed, 14 Dec 2011, Holmes,David A wrote:
From time to time some have posted questions asking if BGP load balancers such as the old Routescience Pathcontrol device are still around, and if not what have others found to replace that function. I have used the Routescience device with much success 10 years ago when it first came on the market, but since then a full BGP feed has become much larger, Routescience has been bought by Avaya, then discontinued, and other competitors such as Sockeye, Netvmg have been acquired by other companies.
It's important to keep in mind what load-balancing is and isn't in this case. The terminology gap can be important because load-balancing (more accurately, load-sharing) in the context of internetwork traffic engineering is very different from load-balancing pools of servers in a data center. Some people can (and sometimes do) confuse the two, which can cause unrealistic expectations to be set :) Achieving a perfect split of network traffic across two or more upstream links rarely happens in the real world. General good practice is to put bandwidth where the network traffic wants to go, but that can be a moving target, and executives and accountants don't like those :) Traffic engineering still has a place on many networks, for a veriety of reasons (technical, financial, political, some combination of these), but as other posters have mentioned, it's often done manually, i.e. looking at Netflow reports, seeing where your traffic is going/coming from, adjusting BGP policies accordingly. Repeat as needed. Since traffic profiles can change over time, any policy tweaks that are put in place need to be reviewed periodically. Depending on how much prep work and planning you're willing to do, you can put a fairly rich set of internal BGP communities in place to control things like localpref, MEDs, selective prepends, and tagging outbound advertisements with provider-specific communities. With that kind of structure, you could control many aspects of your traffic engineering from a route server, rather than having to tinker with route policies on each outside-facing router. One caveat: If your route server crashes or has to be taken down for maintenance, the traffic patterns that were being tweaked by your policy framework could start to revert to the way the traffic would flow in its un-altered state, which could cause you some unintended headaches. jms E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
participants (7)
-
Christopher Morrow -
Drew Weaver -
Holmes,David A -
Jeff Tantsura -
Jonathan Lassoff -
Justin M. Streiner -
Rampley Jr, Jim F