Hello all. I am having a hard time trying to articulate why a Dual Home ISP should have full tables. My understanding has always been that full tables when dual homed allow much more control. Especially in helping to prevent Async routes. Am I crazy?
It’s pretty pointless for a small ISP to get full routes, because the BGP tables are so highly manipulated. It’s better to just get “company” routes for each upstream, and then use your own traffic engineering via prepending and static or policy routes to balance the outbound traffic the way you like. -mel
On Jan 24, 2020, at 8:40 AM, Brian <brian.bsi@gmail.com> wrote:
Hello all. I am having a hard time trying to articulate why a Dual Home ISP should have full tables. My understanding has always been that full tables when dual homed allow much more control. Especially in helping to prevent Async routes.
Am I crazy?
Honestly, this. Your only real choice is what of 2 pipes to chuck it out of. Full tables vs partial and a default don’t make the process much more intelligent for 1 site dual homed, and as mentioned routing policy will have more influence. -Ben
On Jan 24, 2020, at 8:47 AM, Mel Beckman <mel@beckman.org> wrote:
It’s pretty pointless for a small ISP to get full routes, because the BGP tables are so highly manipulated. It’s better to just get “company” routes for each upstream, and then use your own traffic engineering via prepending and static or policy routes to balance the outbound traffic the way you like.
-mel
On Jan 24, 2020, at 8:40 AM, Brian <brian.bsi@gmail.com> wrote:
Hello all. I am having a hard time trying to articulate why a Dual Home ISP should have full tables. My understanding has always been that full tables when dual homed allow much more control. Especially in helping to prevent Async routes.
Am I crazy?
We have full tables from 2 ISPs at just one datacenter, and it is nice in the case of partial reachability issues—If one ISP loses access to routes to a destination but the other one doesn’t, for example. For us, the decision to do full tables was easy, as we are running 2 MX150s which can very easily handle the load and convergence is still less than a minute or so. As far as optimal path goes, full tables really doesn't help us much, so we made sure to get matching speed circuits just to make things simple. We have AT&T and CenturyLink, and most things prefer CenturyLink as they are pretty well peered due to all of their acquisitions. It would be interesting to see a distribution plot of ASPATH length, I would bet that a huge chunk of our routes are only 2-3 hops away. /chris On 1/24/20, 10:56, "NANOG on behalf of Ben Cannon" <nanog-bounces@nanog.org on behalf of ben@6by7.net> wrote: Honestly, this. Your only real choice is what of 2 pipes to chuck it out of. Full tables vs partial and a default don’t make the process much more intelligent for 1 site dual homed, and as mentioned routing policy will have more influence. -Ben > On Jan 24, 2020, at 8:47 AM, Mel Beckman <mel@beckman.org> wrote: > > It’s pretty pointless for a small ISP to get full routes, because the BGP tables are so highly manipulated. It’s better to just get “company” routes for each upstream, and then use your own traffic engineering via prepending and static or policy routes to balance the outbound traffic the way you like. > > -mel > >> On Jan 24, 2020, at 8:40 AM, Brian <brian.bsi@gmail.com> wrote: >> >> >> Hello all. I am having a hard time trying to articulate why a Dual Home ISP should have full tables. My understanding has always been that full tables when dual homed allow much more control. Especially in helping to prevent Async routes. >> >> >> Am I crazy?
Dear Brian, On Fri, 24 Jan 2020 at 17:40, Brian <brian.bsi@gmail.com> wrote:
Hello all. I am having a hard time trying to articulate why a Dual Home ISP should have full tables. My understanding has always been that full tables when dual homed allow much more control. Especially in helping to prevent Async routes.
The advantage of receiving full routing tables from both providers is that in cases where one of the two providers is not yet fully converged, your routers will use the other provider for those missing destinations. This may happen during maintenance or router boot-up in your upstream’s network. Another advantage of receiving full routes is that you can manipulate LOCAL_PREF per destination, or compose routing policy based on per-route attributes such as BGP communities your upstreams set. It can happen that a provider is great for 99% of destinations, except a few - without full tables such granular traffic-engineering can be cumbersome. Virtually all internet routing is asymmetric, I wouldn’t consider that an issue. Am I crazy?
I dropped out of university, never completed my psychology studies, I fear I am unqualified to answer this question. ;-) Kind regards, Job
fre. 24. jan. 2020 18.23 skrev Job Snijders <job@instituut.net>:
On Fri, 24 Jan 2020 at 17:40, Brian <brian.bsi@gmail.com> wrote:
Am I crazy?
I dropped out of university, never completed my psychology studies, I fear I am unqualified to answer this question. ;-)
Education shopping, it is called by some. Chriztoffer
Dear Job and NANOG, Just wondering, wouldn't any of you guys consider using full tables in this case, for the ability to detect and avoid prefix hijacks (using RPKI/ROV or other means)? Of course, I'm focused on security, and I know this is often not a high priority for a real network manager who has many other considerations; just want to know. Thanks. -- Amir On Fri, Jan 24, 2020 at 12:27 PM Job Snijders <job@instituut.net> wrote:
Dear Brian,
On Fri, 24 Jan 2020 at 17:40, Brian <brian.bsi@gmail.com> wrote:
Hello all. I am having a hard time trying to articulate why a Dual Home ISP should have full tables. My understanding has always been that full tables when dual homed allow much more control. Especially in helping to prevent Async routes.
The advantage of receiving full routing tables from both providers is that in cases where one of the two providers is not yet fully converged, your routers will use the other provider for those missing destinations. This may happen during maintenance or router boot-up in your upstream’s network.
Another advantage of receiving full routes is that you can manipulate LOCAL_PREF per destination, or compose routing policy based on per-route attributes such as BGP communities your upstreams set. It can happen that a provider is great for 99% of destinations, except a few - without full tables such granular traffic-engineering can be cumbersome.
Virtually all internet routing is asymmetric, I wouldn’t consider that an issue.
Am I crazy?
I dropped out of university, never completed my psychology studies, I fear I am unqualified to answer this question. ;-)
Kind regards,
Job
Reading all the arguments one could generalize that choosing default/partial routes (instead of full feed) one is basically outsourcing all the control, convergence speed, security, etc.. to upstream providers. adam From: NANOG <nanog-bounces@nanog.org> On Behalf Of Amir Herzberg Sent: Monday, January 27, 2020 1:49 PM To: Job Snijders <job@instituut.net> Cc: NANOG <nanog@nanog.org> Subject: Re: Dual Homed BGP Dear Job and NANOG, Just wondering, wouldn't any of you guys consider using full tables in this case, for the ability to detect and avoid prefix hijacks (using RPKI/ROV or other means)? Of course, I'm focused on security, and I know this is often not a high priority for a real network manager who has many other considerations; just want to know. Thanks. -- Amir On Fri, Jan 24, 2020 at 12:27 PM Job Snijders <job@instituut.net <mailto:job@instituut.net> > wrote: Dear Brian, On Fri, 24 Jan 2020 at 17:40, Brian <brian.bsi@gmail.com <mailto:brian.bsi@gmail.com> > wrote: Hello all. I am having a hard time trying to articulate why a Dual Home ISP should have full tables. My understanding has always been that full tables when dual homed allow much more control. Especially in helping to prevent Async routes. The advantage of receiving full routing tables from both providers is that in cases where one of the two providers is not yet fully converged, your routers will use the other provider for those missing destinations. This may happen during maintenance or router boot-up in your upstream’s network. Another advantage of receiving full routes is that you can manipulate LOCAL_PREF per destination, or compose routing policy based on per-route attributes such as BGP communities your upstreams set. It can happen that a provider is great for 99% of destinations, except a few - without full tables such granular traffic-engineering can be cumbersome. Virtually all internet routing is asymmetric, I wouldn’t consider that an issue. Am I crazy? I dropped out of university, never completed my psychology studies, I fear I am unqualified to answer this question. ;-) Kind regards, Job
On 1/23/20 16:01, Brian wrote:
Hello all. I am having a hard time trying to articulate why a Dual Home ISP should have full tables. My understanding has always been that full tables when dual homed allow much more control. Especially in helping to prevent Async routes.
If you're multi-homed you'll have some asymmetric routes. Even if you aren't, they'll be asymmetric to your upstream. It's easy to manipulate which path traffic leaving your site takes, difficult or impossible to manipulate via which path it enters. If you have sufficient horsepower and memory in your routers, taking full tables won't hurt anything and will help in situations where one of your upstreams loses reachability to a destination. For belt-and-suspenders, take full routes plus a default.
Am I crazy?
Perhaps, but if so it has little if any relationship to the size of the BGP tables in your border routers. -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
Don't forget to connect to peering exchanges and take their full routes too if you can.
Full tables will not make much noticeable difference if you are not peering. However you want to make sure both links get used. It can be a 90%/10% split but 100%/0% is bad because then you may discover that the alternate path is actually broken the moment the primary fail. If you choose only default then you need to think about that. If you join any peering exchanges, full tables will be mandatory. Some parties will export prefixes and then expect a more specific prefix received from your transit to override a part of the space received via the peering. Regards Baldur fre. 24. jan. 2020 17.41 skrev Brian <brian.bsi@gmail.com>:
Hello all. I am having a hard time trying to articulate why a Dual Home ISP should have full tables. My understanding has always been that full tables when dual homed allow much more control. Especially in helping to prevent Async routes.
Am I crazy?
On Fri, 24 Jan 2020, Baldur Norddahl wrote:
Full tables will not make much noticeable difference if you are not peering. However you want to make sure both links get used. It can be a 90%/10% split but 100%/0% is bad because then you may discover that the alternate path is actually broken the moment the primary fail. If you choose only default then you need to think about that. If you join any peering exchanges, full tables will be mandatory. Some parties will export prefixes and then expect a more specific prefix received from your transit to override a part of the space received via the peering.
90/10 will suck when the link carrying 90% of your traffic needs more pipe and you have a ton of unused capacity on the other one. Full tables from both providers gives you more options to tune things (assuming outbound is your larger direction). If you're an eyeball provider and most of your traffic is inbound, your outbound traffic routing decisions aren't quite as relevant. Have those suggesting "multihoming with two partial feeds and default routes" forgotten peering pissing matches, long lasting inter-network capacity issues, or that certain "tier 1" providers don't even have/provide a full v6 table? If you're going to multihome, do it right, and get full routes from all your providers. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route StackPath, Sr. Neteng | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
lør. 25. jan. 2020 00.40 skrev Jon Lewis <jlewis@lewis.org>:
On Fri, 24 Jan 2020, Baldur Norddahl wrote:
Full tables will not make much noticeable difference if you are not peering. However you want to make sure both links get used. It can be a 90%/10% split but 100%/0% is bad because then you may discover that the alternate path is actually broken the moment the primary fail. If you choose only default then you need to think about that. If you join any peering exchanges, full tables will be mandatory. Some parties will export prefixes and then expect a more specific prefix received from your transit to override a part of the space received via the peering.
90/10 will suck when the link carrying 90% of your traffic needs more pipe and you have a ton of unused capacity on the other one. Full tables from both providers gives you more options to tune things (assuming outbound is your larger direction). If you're an eyeball provider and most of your traffic is inbound, your outbound traffic routing decisions aren't quite as relevant.
If your goal is to maximize your capacity, you should run a default route with equal cost multi path for perfect load balancing. Just beware that there is effectively no redundancy when exceeding the capacity of a single link. Also consider the typical two transits each connected to a separate router, each router handling a single circuit. I will wager that the majority of such dual homed organisations have no idea that those two routers by default will make different routing decisions. You get more control but you also need the experience and talent to use it. For many it might be better to have a solution that is understood.
Have those suggesting "multihoming with two partial feeds and default routes" forgotten peering pissing matches, long lasting inter-network capacity issues, or that certain "tier 1" providers don't even have/provide a full v6 table?
The solution is to stay clear of tier 1 networks. Find a good local tier 3. Whatever you are going to do, they will do better. Regards Baldur
On 25/Jan/20 02:49, Baldur Norddahl wrote:
The solution is to stay clear of tier 1 networks. Find a good local tier 3. Whatever you are going to do, they will do better.
So all our transit comes from the top 7 "global" carriers. Yes, including Cogent :-). But that only accounts for about 15% of our overall traffic. The rest comes from peering. Mark.
On Sun, 16 Feb 2020 at 14:02, Mark Tinka <mark.tinka@seacom.mu> wrote:
But that only accounts for about 15% of our overall traffic. The rest comes from peering.
I'd really like to hear more datapoints from different eyeballs on this, like 60% local caches, of remaining traffic, 70% peered so transit = 0.4*0.3 = 12% I think this might be reasonable, but perhaps it's even less transit for eyeballs today? -- ++ytti
On 16/Feb/20 16:51, Saku Ytti wrote:
I'd really like to hear more datapoints from different eyeballs on this, like
60% local caches, of remaining traffic, 70% peered
so transit = 0.4*0.3 = 12%
I think this might be reasonable, but perhaps it's even less transit for eyeballs today?
So for us, we have quite a number of on-net caches, and the fill for them (or origin pull) is typically done via peering. Of the 85% we consider "peering traffic", around one-third of that is traffic from our on-net caches. Mark.
On Sun, Feb 16, 2020 at 12:45 PM Mark Tinka <mark.tinka@seacom.mu> wrote:
On 25/Jan/20 02:49, Baldur Norddahl wrote:
The solution is to stay clear of tier 1 networks. Find a good local tier 3. Whatever you are going to do, they will do better.
So all our transit comes from the top 7 "global" carriers. Yes, including Cogent :-).
But that only accounts for about 15% of our overall traffic. The rest comes from peering.
Mark.
From the perspective of someone just starting out being dual homed, this
will be very different. You are not going to get 7 transits and you are not going to be able to peer 85% of the traffic. That is why I advocate that it is better to buy transit from a middle tier company. Instead of getting a connection to just one so called global carrier, you get a package deal with connection to all of them and 85% peering one step removed. Plus many of the companies that the middle tier has a peering with, is something the tier 1 companies would refuse to peer (exception Hurricane Electric). Also while your company may not need dual connections to each transit, the situation is completely different from the perspective of a small dual homed customer of yours. That is a lot of paths that are lost if this customer where to experience a disruption to the connection to your network. This is especially true if there is an unbalance between the two chosen transit providers. Say the other provider is Cogent, which are famous for refusing to peer. That means that all those peers, unless they have a Cogent contract, they will need to find an indirect path to replace your peering. Of course I may also recommend to simply set your expectations modestly. Dual homing will get you redundancy but unless you line up all your ducks correctly, you should expect some brownouts in the case of a link failure. Simply tell the boss, that unless he wants to pay at least double in every way, there will be expected downtime in the order of 5 minuttes in the case of a link failure. Regards, Baldur
"you are not going to be able to peer 85% of the traffic" It depends. If you are an eyeball ISP and you join one of the major IXes, you'll be near 85%. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Baldur Norddahl" <baldur.norddahl@gmail.com> To: nanog@nanog.org Sent: Sunday, February 16, 2020 10:08:12 AM Subject: Re: Dual Homed BGP On Sun, Feb 16, 2020 at 12:45 PM Mark Tinka < mark.tinka@seacom.mu > wrote: On 25/Jan/20 02:49, Baldur Norddahl wrote:
The solution is to stay clear of tier 1 networks. Find a good local tier 3. Whatever you are going to do, they will do better.
So all our transit comes from the top 7 "global" carriers. Yes, including Cogent :-). But that only accounts for about 15% of our overall traffic. The rest comes from peering. Mark.
From the perspective of someone just starting out being dual homed, this will be very different. You are not going to get 7 transits and you are not going to be able to peer 85% of the traffic. That is why I advocate that it is better to buy transit from a middle tier company. Instead of getting a connection to just one so called global carrier, you get a package deal with connection to all of them and 85% peering one step removed. Plus many of the companies that the middle tier has a peering with, is something the tier 1 companies would refuse to peer (exception Hurricane Electric).
Also while your company may not need dual connections to each transit, the situation is completely different from the perspective of a small dual homed customer of yours. That is a lot of paths that are lost if this customer where to experience a disruption to the connection to your network. This is especially true if there is an unbalance between the two chosen transit providers. Say the other provider is Cogent, which are famous for refusing to peer. That means that all those peers, unless they have a Cogent contract, they will need to find an indirect path to replace your peering. Of course I may also recommend to simply set your expectations modestly. Dual homing will get you redundancy but unless you line up all your ducks correctly, you should expect some brownouts in the case of a link failure. Simply tell the boss, that unless he wants to pay at least double in every way, there will be expected downtime in the order of 5 minuttes in the case of a link failure. Regards, Baldur
We are seeing about 79% currently that is with one of our new Akamai PNIs in CHI and we peer at most major IXs across the US. Top 5 peers Netflix, Google, Akamai, Amazon and EdgeCast. (In order) Erich Kaiser The Fusion Network erich@gotfusion.net On Sun, Feb 16, 2020 at 10:11 AM Mike Hammett <nanog@ics-il.net> wrote:
"you are not going to be able to peer 85% of the traffic"
It depends. If you are an eyeball ISP and you join one of the major IXes, you'll be near 85%.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> ------------------------------ *From: *"Baldur Norddahl" <baldur.norddahl@gmail.com> *To: *nanog@nanog.org *Sent: *Sunday, February 16, 2020 10:08:12 AM *Subject: *Re: Dual Homed BGP
On Sun, Feb 16, 2020 at 12:45 PM Mark Tinka <mark.tinka@seacom.mu> wrote:
On 25/Jan/20 02:49, Baldur Norddahl wrote:
The solution is to stay clear of tier 1 networks. Find a good local tier 3. Whatever you are going to do, they will do better.
So all our transit comes from the top 7 "global" carriers. Yes, including Cogent :-).
But that only accounts for about 15% of our overall traffic. The rest comes from peering.
Mark.
From the perspective of someone just starting out being dual homed, this will be very different. You are not going to get 7 transits and you are not going to be able to peer 85% of the traffic. That is why I advocate that it is better to buy transit from a middle tier company. Instead of getting a connection to just one so called global carrier, you get a package deal with connection to all of them and 85% peering one step removed. Plus many of the companies that the middle tier has a peering with, is something the tier 1 companies would refuse to peer (exception Hurricane Electric).
Also while your company may not need dual connections to each transit, the situation is completely different from the perspective of a small dual homed customer of yours. That is a lot of paths that are lost if this customer where to experience a disruption to the connection to your network.
This is especially true if there is an unbalance between the two chosen transit providers. Say the other provider is Cogent, which are famous for refusing to peer. That means that all those peers, unless they have a Cogent contract, they will need to find an indirect path to replace your peering.
Of course I may also recommend to simply set your expectations modestly. Dual homing will get you redundancy but unless you line up all your ducks correctly, you should expect some brownouts in the case of a link failure. Simply tell the boss, that unless he wants to pay at least double in every way, there will be expected downtime in the order of 5 minuttes in the case of a link failure.
Regards,
Baldur
I just looked at the stats again we are actually at about 82%. Erich Kaiser The Fusion Network erich@gotfusion.net On Sun, Feb 16, 2020 at 11:18 AM Kaiser, Erich <erich@gotfusion.net> wrote:
We are seeing about 79% currently that is with one of our new Akamai PNIs in CHI and we peer at most major IXs across the US.
Top 5 peers Netflix, Google, Akamai, Amazon and EdgeCast. (In order)
Erich Kaiser The Fusion Network erich@gotfusion.net
On Sun, Feb 16, 2020 at 10:11 AM Mike Hammett <nanog@ics-il.net> wrote:
"you are not going to be able to peer 85% of the traffic"
It depends. If you are an eyeball ISP and you join one of the major IXes, you'll be near 85%.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> ------------------------------ *From: *"Baldur Norddahl" <baldur.norddahl@gmail.com> *To: *nanog@nanog.org *Sent: *Sunday, February 16, 2020 10:08:12 AM *Subject: *Re: Dual Homed BGP
On Sun, Feb 16, 2020 at 12:45 PM Mark Tinka <mark.tinka@seacom.mu> wrote:
On 25/Jan/20 02:49, Baldur Norddahl wrote:
The solution is to stay clear of tier 1 networks. Find a good local tier 3. Whatever you are going to do, they will do better.
So all our transit comes from the top 7 "global" carriers. Yes, including Cogent :-).
But that only accounts for about 15% of our overall traffic. The rest comes from peering.
Mark.
From the perspective of someone just starting out being dual homed, this will be very different. You are not going to get 7 transits and you are not going to be able to peer 85% of the traffic. That is why I advocate that it is better to buy transit from a middle tier company. Instead of getting a connection to just one so called global carrier, you get a package deal with connection to all of them and 85% peering one step removed. Plus many of the companies that the middle tier has a peering with, is something the tier 1 companies would refuse to peer (exception Hurricane Electric).
Also while your company may not need dual connections to each transit, the situation is completely different from the perspective of a small dual homed customer of yours. That is a lot of paths that are lost if this customer where to experience a disruption to the connection to your network.
This is especially true if there is an unbalance between the two chosen transit providers. Say the other provider is Cogent, which are famous for refusing to peer. That means that all those peers, unless they have a Cogent contract, they will need to find an indirect path to replace your peering.
Of course I may also recommend to simply set your expectations modestly. Dual homing will get you redundancy but unless you line up all your ducks correctly, you should expect some brownouts in the case of a link failure. Simply tell the boss, that unless he wants to pay at least double in every way, there will be expected downtime in the order of 5 minuttes in the case of a link failure.
Regards,
Baldur
Hello My numbers for the last 7 days: NL-IX 2107 22% STHIX 412 4% Cache (netflix+akamai) 3070 32% Telia 1118 12% Cogent 258 3% HE 2490 26% = 9455 100% The numbers being average (not peak or 95 percentile) inbound. Outbound is irrelevant for us. I suppose that if you consider Hurricane Electric as a poor mans peering service, we would be up there as well. This is for volumen however. Almost everything breaks if we loose the transits. It is not like you are 80% ok. Regards, Baldur On Sun, Feb 16, 2020 at 7:28 PM Kaiser, Erich <erich@gotfusion.net> wrote:
I just looked at the stats again we are actually at about 82%.
Erich Kaiser The Fusion Network erich@gotfusion.net
On Sun, Feb 16, 2020 at 11:18 AM Kaiser, Erich <erich@gotfusion.net> wrote:
We are seeing about 79% currently that is with one of our new Akamai PNIs in CHI and we peer at most major IXs across the US.
Top 5 peers Netflix, Google, Akamai, Amazon and EdgeCast. (In order)
Erich Kaiser The Fusion Network erich@gotfusion.net
On Sun, Feb 16, 2020 at 10:11 AM Mike Hammett <nanog@ics-il.net> wrote:
"you are not going to be able to peer 85% of the traffic"
It depends. If you are an eyeball ISP and you join one of the major IXes, you'll be near 85%.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> ------------------------------ *From: *"Baldur Norddahl" <baldur.norddahl@gmail.com> *To: *nanog@nanog.org *Sent: *Sunday, February 16, 2020 10:08:12 AM *Subject: *Re: Dual Homed BGP
On Sun, Feb 16, 2020 at 12:45 PM Mark Tinka <mark.tinka@seacom.mu> wrote:
On 25/Jan/20 02:49, Baldur Norddahl wrote:
The solution is to stay clear of tier 1 networks. Find a good local tier 3. Whatever you are going to do, they will do better.
So all our transit comes from the top 7 "global" carriers. Yes, including Cogent :-).
But that only accounts for about 15% of our overall traffic. The rest comes from peering.
Mark.
From the perspective of someone just starting out being dual homed, this will be very different. You are not going to get 7 transits and you are not going to be able to peer 85% of the traffic. That is why I advocate that it is better to buy transit from a middle tier company. Instead of getting a connection to just one so called global carrier, you get a package deal with connection to all of them and 85% peering one step removed. Plus many of the companies that the middle tier has a peering with, is something the tier 1 companies would refuse to peer (exception Hurricane Electric).
Also while your company may not need dual connections to each transit, the situation is completely different from the perspective of a small dual homed customer of yours. That is a lot of paths that are lost if this customer where to experience a disruption to the connection to your network.
This is especially true if there is an unbalance between the two chosen transit providers. Say the other provider is Cogent, which are famous for refusing to peer. That means that all those peers, unless they have a Cogent contract, they will need to find an indirect path to replace your peering.
Of course I may also recommend to simply set your expectations modestly. Dual homing will get you redundancy but unless you line up all your ducks correctly, you should expect some brownouts in the case of a link failure. Simply tell the boss, that unless he wants to pay at least double in every way, there will be expected downtime in the order of 5 minuttes in the case of a link failure.
Regards,
Baldur
On 16/Feb/20 20:47, Baldur Norddahl wrote:
This is for volumen however. Almost everything breaks if we loose the transits. It is not like you are 80% ok.
And this is an important point, as even if many of your on-net caches would origin via a peering session with its owner, some origins are only available via transit. Mark.
On 16/Feb/20 18:08, Baldur Norddahl wrote:
From the perspective of someone just starting out being dual homed, this will be very different. You are not going to get 7 transits and you are not going to be able to peer 85% of the traffic. That is why I advocate that it is better to buy transit from a middle tier company. Instead of getting a connection to just one so called global carrier, you get a package deal with connection to all of them and 85% peering one step removed. Plus many of the companies that the middle tier has a peering with, is something the tier 1 companies would refuse to peer (exception Hurricane Electric).
Also while your company may not need dual connections to each transit, the situation is completely different from the perspective of a small dual homed customer of yours. That is a lot of paths that are lost if this customer where to experience a disruption to the connection to your network.
This is especially true if there is an unbalance between the two chosen transit providers. Say the other provider is Cogent, which are famous for refusing to peer. That means that all those peers, unless they have a Cogent contract, they will need to find an indirect path to replace your peering.
Of course I may also recommend to simply set your expectations modestly. Dual homing will get you redundancy but unless you line up all your ducks correctly, you should expect some brownouts in the case of a link failure. Simply tell the boss, that unless he wants to pay at least double in every way, there will be expected downtime in the order of 5 minuttes in the case of a link failure.
Completely agreed, as I highlighted in my post at https://mailman.nanog.org/pipermail/nanog/2020-February/105953.html as a response to Adam's original query. Mark.
* Baldur Norddahl
If you join any peering exchanges, full tables will be mandatory. Some parties will export prefixes and then expect a more specific prefix received from your transit to override a part of the space received via the peering.
That would be a fundamentally flawed expectation, in my opinion. An AS that that advertises a prefix to its peers must be prepared to carry traffic to that entire prefix via that peering circuit. There is simply no guarantee that a more-specific prefix advertised somewhere else will make it into the RIBs and FIBs of all the peers of that AS. The AS might of course opt to do so anyway for traffic engineering purposes, but there is no assurance that it will actually work 100% of the time. When it doesn't, the AS in question would need to carry the traffic from the peering circuit across their own backbone. If the AS in question for some reason cannot do so, it would need to adjust its advertisements across the peering circuit so as to avoid falsely advertising reachability to unreachable destinations. Tore
lør. 25. jan. 2020 13.42 skrev Tore Anderson <tore@fud.no>:
* Baldur Norddahl
If you join any peering exchanges, full tables will be mandatory. Some parties will export prefixes and then expect a more specific prefix received from your transit to override a part of the space received via the peering.
That would be a fundamentally flawed expectation, in my opinion.
I do not disagree, however the real world sometimes works differently. Like anycast, people break the rules and gets away with it. In any case, this is from a recent personal experience. We had a problem that led us to drop full tables and run with a default for a while. Nobody noticed the difference, which is why I can confidently say that unless you need the full tables for something, the advantages are somewhat overstated. However one customer found a reachability problem. Turns out that a peer was exporting a /19 prefix through a peering session with us and at another site they exported a /24 from the same space with no routing between sites. Regards Baldur
I’m listening to the advice of others and taking it in…. For my ISP, I’ve had 2 or 3 internet uplinks for about 12 years now for 50,000 subs, and have only learned a default route on them. It’s been good up to this point. -Aaron
Interesting discussion. Besides the point about control which many people have made, I would like to point - it also depends on your geography and peering in that geography. Cannot generalise but typically in the US and Europe you will find reasonably good peering across larger operators and hence cases of your traffic taking really odd path (like going out of country and in forward / reverse direction) would be low. That may not be the case in certain other geographies like say Asia or Africa. In my home country India I find occasional cases of traffic leaking out of the country. I recently posted this trace taken from an Indian RIPE Atlas probes to a popular content site with frontend mostly on Akamai: https://atlas.ripe.net/measurements/23857778/#!probes Traceroute to www.hotstar.com (104.123.148.87), 48 byte packets 1 172.16.0.1 0.943ms 0.828ms 0.821ms 2 172.16.1.100 0.743ms * 13.019ms 3 103.145.6.4 AS139540 0.684ms 3.548ms 2.972ms 4 103.145.6.3 AS139540 5.035ms 17.172ms 0.664ms 5 59.162.52.161 59.162.52.161.static.vsnl.net.in AS4755 0.92ms 0.845ms 1.218ms 6 115.112.167.241 115.112.167.241.STATIC-Kolkata.vsnl.net.in AS4755 4.522ms 21.518ms 29.118ms 7 172.25.87.174 13.919ms 13.741ms 13.509ms 8 172.31.180.57 53.947ms 93.642ms 58.971ms 9 180.87.36.9 ix-ae-4-2.tcore1.cxr-chennai.as6453.net AS6453 54.4ms 53.752ms 53.721ms 10 180.87.36.41 if-ae-34-2.tcore1.svq-singapore.as6453.net AS6453 85.424ms 85.835ms 86.098ms 11 120.29.215.39 AS6453 86.247ms 85.818ms 86.203ms 12 49.45.4.87 AS64049 84.675ms 84.555ms 84.614ms 13 49.45.4.87 AS64049 83.941ms 83.835ms 84.163ms 14 103.198.140.185 AS64049 85.437ms 85.498ms 85.506ms 15 172.16.23.9 100.334ms 100.817ms 172.25.106.39 100.344ms 16 172.16.2.70 98.977ms 98.835ms 98.795ms 17 172.16.2.41 99.69ms 99.628ms 99.565ms 18 172.16.0.210 102.005ms 102.059ms 101.929ms 19 104.123.148.87 a104-123-148-87.deploy.static.akamaitechnologies.com AS55836 105.386ms 105.358ms 105.284ms Ignoring the DNS based mapping and associated challenges of Akamai and speaking purely from routing. Say if AS139540 had full table it would simply have picked direct route to AS55836 which is one of it's upstream ( https://bgp.he.net/AS139540#_graph4). Thus 100ms would be down to probably sub 20ms levels. Whether or not such issues affect you depends on whether upstreams or upstream's upstream peer in the region or not. In this case upstream AS4755 & AS55836 are not exchanging traffic locally. Furthermore their upstream AS6453 and AS64049 are also not exchanging traffic locally either. So if I have to decide, I will always pick full table in these cases. AS_PATH will help and one can always tweak for larger known cases. On Sun, Jan 26, 2020 at 10:11 AM Aaron Gould <aaron1@gvtc.com> wrote:
I’m listening to the advice of others and taking it in….
For my ISP, I’ve had 2 or 3 internet uplinks for about 12 years now for 50,000 subs, and have only learned a default route on them. It’s been good up to this point.
-Aaron
-- Anurag Bhatia anuragbhatia.com
On 1/23/2020 6:01 PM, Brian wrote:
Hello all. I am having a hard time trying to articulate why a Dual Home ISP should have full tables. My understanding has always been that full tables when dual homed allow much more control. Especially in helping to prevent Async routes.
Brian, you're correct that having more routing information will give you more control over routing decisions. However, for me, it's not about control/traffic engineering but about basic redundancy/availability. Taking full tables gives you visibility into whether an actual path exists (or does not exist) via each of your upstream providers. If you just take a default route originated by your peer you'll only have visibility that there's a connection between your router and your peer - not to the rest of the internet or to a specific destination past your peer. As Job said, maintenance, router reboots, and other upstream connectivity issues can cause an upstream peer to be missing routes that the other peer might have. In my experience, most folks multi-home for improved redundancy/availability. So for those that are multi-homed, I recommend taking full tables so that you can get the most benefit. This may come at an additional expense so I understand why some would choose to take a default route only. --Blake
On 1/23/20 6:01 PM, Brian wrote:
Hello all. I am having a hard time trying to articulate why a Dual Home ISP should have full tables. My understanding has always been that full tables when dual homed allow much more control. Especially in helping to prevent Async routes.
If you don't have full tables you will certainly have a default route somewhere, either set statically by you or advertised by your upstream. Accidents may happen: your ISP may blackhole a destination; sometimes they adjust loads, sometimes their redundancy is not properly set up or they may have an otherwise incorrect BGP configuration. Sometimes they just mess up. If you have full tables, you will simply stop getting the advertisements for the bad destinations from the bad ISP and your routers will take care of it because they will keep getting the advertisements from the other exit. If you don't have full tables and the mistake is in a destination for which the default route is used, your traffic will most probably be lost because they don't have enough information to know a different exit and you may get a call at midnight. It may or may not happened to me. ;-) Octavio.
participants (21)
-
Aaron Gould -
adamv0025＠netconsultings.com -
Amir Herzberg -
Anurag Bhatia -
Baldur Norddahl -
Ben Cannon -
Blake Hudson -
Brian -
Chriztoffer Hansen -
Cummings, Chris -
Gavin Henry -
Jay Hennigan -
Job Snijders -
Jon Lewis -
Kaiser, Erich -
Mark Tinka -
Mel Beckman -
Mike Hammett -
Octavio Alvarez -
Saku Ytti -
Tore Anderson