Re: [c-nsp] DNS amplification
On Sat, 16 Mar 2013, Robert Joosten wrote:
Hi,
Can anyone provide insight into how to defeat DNS amplification attacks? Restrict resolvers to your customer networks.
And deploy RPF
uRPF / BCP38 is really the only solution. Even if we did close all the open recursion DNS servers (which is a good idea), the attackers would just shift to another protocol/service that provides amplification of traffic and can be aimed via spoofed source address packets. Going after DNS is playing whack-a-mole. DNS is the hip one right now. It's not the only one available. Many networks will say "but our gear doesn't do uRPF, and maintaining an ACL on every customer port is too hard / doesn't scale." Consider an alternative solution. On a typical small ISP / small service provider network, if you were to ACL every customer (because your gear won't do uRPF), you might need hundreds or even thousands of ACLs. However, if you were to put output filters on your transit connections, allowing traffic sourced from all IP networks "valid" inside your network, you might find that all you need is a single ACL of a handful to several dozen entries. Having one ACL to maintain that only needs changing if you get a new IP allocation or add/remove a customer who has their own IPs really isn't all that difficult. As far at the rest of the internet is concerned, this solves the issue of spoofed IP packets leaving your network. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
yes - and it presumes your DNS servers are based on Linux and use IPTables. http://www.cryptonizer.com/dnsamp.html http://serverfault.com/questions/418810/public-facing-recursive-dns-servers-... http://sf-alpha.bjgang.org/wordpress/2013/01/iptables-for-common-dns-amplifi... these should give you a good idea of how to get started... On Sat, Mar 16, 2013 at 6:24 PM, Jon Lewis <jlewis@lewis.org> wrote:
On Sat, 16 Mar 2013, Robert Joosten wrote:
Hi,
Can anyone provide insight into how to defeat DNS amplification attacks?
Restrict resolvers to your customer networks.
And deploy RPF
uRPF / BCP38 is really the only solution. Even if we did close all the open recursion DNS servers (which is a good idea), the attackers would just shift to another protocol/service that provides amplification of traffic and can be aimed via spoofed source address packets. Going after DNS is playing whack-a-mole. DNS is the hip one right now. It's not the only one available.
Many networks will say "but our gear doesn't do uRPF, and maintaining an ACL on every customer port is too hard / doesn't scale."
Consider an alternative solution. On a typical small ISP / small service provider network, if you were to ACL every customer (because your gear won't do uRPF), you might need hundreds or even thousands of ACLs. However, if you were to put output filters on your transit connections, allowing traffic sourced from all IP networks "valid" inside your network, you might find that all you need is a single ACL of a handful to several dozen entries. Having one ACL to maintain that only needs changing if you get a new IP allocation or add/remove a customer who has their own IPs really isn't all that difficult. As far at the rest of the internet is concerned, this solves the issue of spoofed IP packets leaving your network.
------------------------------**------------------------------**---------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/**pgp<http://www.lewis.org/~jlewis/pgp>for PGP public key_________
-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
Yes, BCP38 is the solution. Now, how widely is deployed? Someone said in the IEPG session during the IETF86 that 80% of the service providers had done it? This raises two questions for me. One, is it really 80%, how to measure it? Second, if it were 80%, how come the 20% makes so much trouble and how to encourage it to deploy BCP38? (well, actually 4 questions :) Regards, as On 3/16/13 7:24 PM, Jon Lewis wrote:
On Sat, 16 Mar 2013, Robert Joosten wrote:
Hi,
Can anyone provide insight into how to defeat DNS amplification attacks? Restrict resolvers to your customer networks.
And deploy RPF
uRPF / BCP38 is really the only solution. Even if we did close all the open recursion DNS servers (which is a good idea), the attackers would just shift to another protocol/service that provides amplification of traffic and can be aimed via spoofed source address packets. Going after DNS is playing whack-a-mole. DNS is the hip one right now. It's not the only one available.
On Sun, Mar 17, 2013 at 11:33 AM, Arturo Servin <arturo.servin@gmail.com> wrote:
Yes, BCP38 is the solution.
Now, how widely is deployed?
Someone said in the IEPG session during the IETF86 that 80% of the service providers had done it?
right... sure.
This raises two questions for me. One, is it really 80%, how to measure it?
csail had a project for a while... spoofer project? <http://spoofer.csail.mit.edu/> I think the last I looked they reported ONLY 35% or so coverage of proper filtering. Looking at: <http://spoofer.csail.mit.edu/summary.php> though they report 86% non-spoofable, that seems very high to me.
Second, if it were 80%, how come the 20% makes so much trouble and how to encourage it to deploy BCP38?
some of the 20% seems to be very highspeed connected end hosts and at a 70:1 amplification ratio you don't need much bandwidth to fill a 1g pipe, eh? -chris
(well, actually 4 questions :)
Regards, as
On 3/16/13 7:24 PM, Jon Lewis wrote:
On Sat, 16 Mar 2013, Robert Joosten wrote:
Hi,
Can anyone provide insight into how to defeat DNS amplification attacks? Restrict resolvers to your customer networks.
And deploy RPF
uRPF / BCP38 is really the only solution. Even if we did close all the open recursion DNS servers (which is a good idea), the attackers would just shift to another protocol/service that provides amplification of traffic and can be aimed via spoofed source address packets. Going after DNS is playing whack-a-mole. DNS is the hip one right now. It's not the only one available.
They should publish the spoofable AS. Not for public shame but at least to show the netadmins that they are doing something wrong, or if they are trying to do the good think is not working. Or at least a tool to check for your ASN or netblock. /as On 3/17/13 1:35 PM, Christopher Morrow wrote:
On Sun, Mar 17, 2013 at 11:33 AM, Arturo Servin <arturo.servin@gmail.com> wrote:
Yes, BCP38 is the solution.
Now, how widely is deployed?
Someone said in the IEPG session during the IETF86 that 80% of the service providers had done it?
right... sure.
This raises two questions for me. One, is it really 80%, how to measure it?
csail had a project for a while... spoofer project? <http://spoofer.csail.mit.edu/>
I think the last I looked they reported ONLY 35% or so coverage of proper filtering. Looking at: <http://spoofer.csail.mit.edu/summary.php>
though they report 86% non-spoofable, that seems very high to me.
Second, if it were 80%, how come the 20% makes so much trouble and how to encourage it to deploy BCP38?
some of the 20% seems to be very highspeed connected end hosts and at a 70:1 amplification ratio you don't need much bandwidth to fill a 1g pipe, eh?
-chris
(well, actually 4 questions :)
Regards, as
On 3/16/13 7:24 PM, Jon Lewis wrote:
On Sat, 16 Mar 2013, Robert Joosten wrote:
Hi,
Can anyone provide insight into how to defeat DNS amplification attacks? Restrict resolvers to your customer networks.
And deploy RPF
uRPF / BCP38 is really the only solution. Even if we did close all the open recursion DNS servers (which is a good idea), the attackers would just shift to another protocol/service that provides amplification of traffic and can be aimed via spoofed source address packets. Going after DNS is playing whack-a-mole. DNS is the hip one right now. It's not the only one available.
On Sun, Mar 17, 2013 at 6:36 PM, Arturo Servin <arturo.servin@gmail.com> wrote:
They should publish the spoofable AS. Not for public shame but at least to show the netadmins that they are doing something wrong, or if they are trying to do the good think is not working.
Or at least a tool to check for your ASN or netblock.
I don't disagree, but I'd point out that there are likely easier places to do bcp38 than others in everyone's network(s)... So, 'I do bcp38' unqualified is not as helpful, especially when almost all consumer grade links are bcp38 by default, which is likely where a bunch of this measurement originates. (well, I suspect a bunch of it is from consumer-grade links anyway)
On Mar 17, 2013, at 8:55 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Sun, Mar 17, 2013 at 6:36 PM, Arturo Servin <arturo.servin@gmail.com> wrote:
They should publish the spoofable AS. Not for public shame but at least to show the netadmins that they are doing something wrong, or if they are trying to do the good think is not working.
Or at least a tool to check for your ASN or netblock.
I don't disagree, but I'd point out that there are likely easier places to do bcp38 than others in everyone's network(s)... So, 'I do bcp38' unqualified is not as helpful, especially when almost all consumer grade links are bcp38 by default, which is likely where a bunch of this measurement originates. (well, I suspect a bunch of it is from consumer-grade links anyway)
(Not sure how this made it from c-nsp to nanog, but ...) uRPF/BCP38 is an important part of a global solution. Similar to open-relays, smurf amplifiers, and other "badness" on the network, one must assist the global network by deploying it where it makes sense. Deploying it at your customer ports may make sense depending on your network. Deploying it on peers may also make sense. I think having a simple set of locations where people actually deploy it is critical, eg: Colocation Network Server Lans VPS Lans Static Routed Customer Edge This should be the default, and something I've pushed at my employer for years. If you do nothing, you can expect nothing as the result. If you attempt do so something, you can at least get an idea of where it's not coming from. At least target these easy edges of the network where there is some value. - Jared
On Sun, 17 Mar 2013, Arturo Servin wrote:
Now, how widely is deployed?
Someone said in the IEPG session during the IETF86 that 80% of the service providers had done it?
This raises two questions for me. One, is it really 80%, how to measure it?
Second, if it were 80%, how come the 20% makes so much trouble and how to encourage it to deploy BCP38?
You'd have to get access (cloud VM, dedicated server, etc.) on each network and see if you can successfully get spoofed packets out to another network. I seriously doubt those numbers though. I'd bet it's more like 80% of service providers are too embarrassed to admit they're not doing BCP38 filtering (or don't know what it is), and 20% are doing it on at least some parts of their network. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On 3/17/13, Jon Lewis <jlewis@lewis.org> wrote:
On Sun, 17 Mar 2013, Arturo Servin wrote:
You'd have to get access (cloud VM, dedicated server, etc.) on each network and see if you can successfully get spoofed packets out to another network.
If you have packet data about a sufficient number of different kinds of attacks per source network over a long period of time, at a specific attack/normal traffic sensor; you might be able to infer some information about which networks prevent spoofing, through a difference in the kind of attacks shown to be originating from all the networks. If spoofing is preferred, or used by other nodes involved in a particular attack, the networks that are concentrated sources of non-spoofing attack packets most likely, are places where spoofing prevention could be present -- and have altered attacker behavior. Possibly the presence of spoofed packets may be suggested by a sudden drastic difference in the average TTL versus legitimate traffic for a particular source network for packets with a particular source IP, correlated with the attack VS the remaining packet TTLs normally observed for legitimate traffic from that network. If you have a sufficiently massive number of traffic sensors, and massive data gathering infrastructure, close enough to the attacks, it may be possible to analyze the microsecond-level timing of packets, and the time sequence/order they arrive at various sensors (milliseconds delay/propagation rate of attacker nodes initiating), in order to provide a probability that spoofed packets came from certain networks. Then at that point, you might make some guesses about which networks implement BCP38 -- -JH
On Sun, Mar 17, 2013 at 7:04 PM, Jimmy Hess <mysidia@gmail.com> wrote:
If you have a sufficiently massive number of traffic sensors, and massive data gathering infrastructure, close enough to the attacks, it may be possible to analyze the microsecond-level timing of packets, and the time sequence/order they arrive at various sensors (milliseconds delay/propagation rate of attacker nodes initiating), in order to provide a probability that spoofed packets came from certain networks.
To get microsecond-level timing, you have to be so close that you're basically just peering with everyone. And at that point you can just look to see which fibers carry spoofed packets. Once you know an ISP hasn't implemented BCP38, what'st the next step? De-peering just reduces your own visibility into the problem. What if it's a transit provider, who can be legitimately expected to route for 0/0? Damian
On 3/17/13, Damian Menscher <damian@google.com> wrote:
Once you know an ISP hasn't implemented BCP38, what'st the next step? De-peering just reduces your own visibility into the problem. What if
In general, a hard problem, not directly solvable in any obvious way. It's similar to the question of what's the next step, after you identified a probable connectivity issue. Detection does not always grant you a way of preventing something. Ultimately, to improve matters with regards to BCP38, I believe you have to secure cooperation; cooperation can sometimes be achieved through persuasion (discussing/requesting/bargaining/begging), or coercing (bribing, threatening, seeking intervention from sponsors, regulators, other networks, or other authorities, public shaming). The recommended next-step would be the ones with the least harmful ramifications for all involved and the network that do have a chance of being effective, and more aggressive options reserved as possible backup plans. In some cases, extreme methods such as inserting offending network's AS into the middle of the AS path of outgoing announcements, possibly, so the spoofed source's upstream network omits reachability to the prefix under attack.... or maintaining peering, but blackholing traffic from that peer, to the local prefix under attack
it's a transit provider, who can be legitimately expected to route for 0/0?
Restricted peering can reduce the impact of the problem; in other words: maintain the peering, but strictly controlling the packets per second and octets per second volumes; traffic going over the peer link is sacrificed during attack, to protect the target. This may still be mutually beneficial for the peers. If the peer is such a transit provider, the problem is indeed hard, possibly not able to be mitigated.
Damian -- -JH
Arturo Servin wrote:
Yes, BCP38 is the solution.
It is not a solution at all, because it, instead, will promote multihomed sites bloats the global routing table. To really solve the problem in an end to end fashion, it is necessary to require IGPs carry information for the proper source address corresponding to each routing table entry in a *FULL* routing table, which must be delivered to almost, if not all, all the end systems. Masataka Ohta
In message <51469FAE.7030102@necom830.hpcl.titech.ac.jp>, Masataka Ohta writes:
Arturo Servin wrote:
Yes, BCP38 is the solution.
It is not a solution at all, because it, instead, will promote multihomed sites bloats the global routing table.
How does enforcing that source address entering your net from customers sites match thoses that have been allocated to them bloat the routing table? Now if you only accept address you have allocated to them by you then that could bloat the routing table but BCP 38 does NOT say to do that. Simlarly URP checking is not BCP 38. With SIDR each multi-homed customer could provide CERTs which proves they have been allocated a address range which could be feed into the acl generators as exceptions to the default rules. This is in theory automatible.
To really solve the problem in an end to end fashion, it is necessary to require IGPs carry information for the proper source address corresponding to each routing table entry in a *FULL* routing table, which must be delivered to almost, if not all, all the end systems.
How does that solve the problem?
Masataka Ohta
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
Mark Andrews wrote:
Yes, BCP38 is the solution.
It is not a solution at all, because it, instead, will promote multihomed sites bloats the global routing table.
How does enforcing that source address entering your net from customers sites match thoses that have been allocated to them bloat the routing table?
First of all, multihomed sites with its own global routing table entries bloats the global routing table, which is the major cause of global routing table bloat and is not acceptable. Then, the only solution is to let the multihomed sites have multiple prefixes, each of which is aggregated by each provider. But, then, all the end systems are required to choose the proper source addresses corresponding to destination addresses, which requires IGPs carry such information. See draft-ohta-e2e-multihoming-05 for details.
Now if you only accept address you have allocated to them by you then that could bloat the routing table but BCP 38 does NOT say to do that. Simlarly URP checking is not BCP 38.
That BCP 38 is narrowly scoped is not my problem.
With SIDR each multi-homed customer could provide CERTs which proves they have been allocated a address range which could be feed into the acl generators as exceptions to the default rules. This is in theory automatible.
The problem is not in individual ISPs but in the global routing table size.
How does that solve the problem?
In the end to end fashion. See draft-ohta-e2e-multihoming-05 for details. Masataka Ohta
On Mar 18, 2013, at 12:47 PM, Masataka Ohta wrote:
See draft-ohta-e2e-multihoming-05 for details.
See <http://datatracker.ietf.org/wg/lisp/> for an actual solution to the problem of routing-table bloat, which has nothing to do with BCP38/84. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
Dobbins, Roland wrote:
See draft-ohta-e2e-multihoming-05 for details.
See <http://datatracker.ietf.org/wg/lisp/> for an actual solution to the problem of routing-table bloat,
It is, by no means, a solution.
which has nothing to do with BCP38/84.
Locator ID separation has nothing to do with routing table bloat. Masataka Ohta
On Mar 18, 2013, at 9:45 PM, Masataka Ohta wrote:
Locator ID separation has nothing to do with routing table bloat.
You obviously haven't read through the materials. I'm done feeding trolls for the day. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
Dobbins, Roland wrote:
Locator ID separation has nothing to do with routing table bloat.
You obviously haven't read through the materials.
LISP merely attempts to replace BGP routing table bloat with something a lot worse than that, that is, a lot more serious routing table bloat of its mapping system. Masataka Ohta
On 19 March 2013 01:06, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp>wrote:
LISP merely attempts to replace BGP routing table bloat with something a lot worse than that, that is, a lot more serious routing table bloat of its mapping system.
I'm guessing you're not a fan of LISP, but in it's defense I'd say the mapping system is akin to DNS - a scalable, distributed, reliable database mapping services to locations. BGP certainly can't cope with unconstrained growth, we will need something better. Aled
On Tue, Mar 19, 2013 at 7:15 AM, Aled Morris <aledm@qix.co.uk> wrote:
On 19 March 2013 01:06, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp>wrote:
LISP merely attempts to replace BGP routing table bloat with something a lot worse than that, that is, a lot more serious routing table bloat of its mapping system.
I'm guessing you're not a fan of LISP, but in it's defense I'd say the mapping system is akin to DNS - a scalable, distributed, reliable database mapping services to locations.
BGP certainly can't cope with unconstrained growth, we will need something better.
and by 'bgp' you mean 'the implementation(s) of BGP on platforms today' There's nothing inherent in BGP that would not work with an unconstrained growth of the routing table, right? You just need enough bandwidth and interrupts to deal with updates.
On Mar 19, 2013, at 10:12 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
There's nothing inherent in BGP that would not work with an unconstrained growth of the routing table, right? You just need enough bandwidth and interrupts to deal with updates.
With enough thrust, pigs fly quite well. Landing can get messy though... Regards, -drc
On Tue, Mar 19, 2013 at 1:45 PM, David Conrad <drc@virtualized.org> wrote:
On Mar 19, 2013, at 10:12 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
There's nothing inherent in BGP that would not work with an unconstrained growth of the routing table, right? You just need enough bandwidth and interrupts to deal with updates.
With enough thrust, pigs fly quite well. Landing can get messy though...
I was being serious... the current 'bgp unconstrained dies' problem isn't such a problem if you have (today): 4-8 cores 16 gb ram ssd gigabit ethernet or as you'd call this, your desktop computer... trying to do this on a 600mhz mips with 512mb ram is, clearly, a problem. put modern hardware to work and it gets simpler. Yes, the above addresses getting/sending 'rib' data, it doesn't address programming a FIB, but rethinking the programming of the fib a bit could, I bet, even get us to a palatable point for a longer while, in a relatively short period of time. -chris
On Mar 19, 2013, at 1:50 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Tue, Mar 19, 2013 at 1:45 PM, David Conrad <drc@virtualized.org> wrote:
On Mar 19, 2013, at 10:12 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
There's nothing inherent in BGP that would not work with an unconstrained growth of the routing table, right? You just need enough bandwidth and interrupts to deal with updates.
With enough thrust, pigs fly quite well. Landing can get messy though...
I was being serious... the current 'bgp unconstrained dies' problem isn't such a problem if you have (today): 4-8 cores 16 gb ram ssd gigabit ethernet
or as you'd call this, your desktop computer... trying to do this on a 600mhz mips with 512mb ram is, clearly, a problem. put modern hardware to work and it gets simpler. Yes, the above addresses getting/sending 'rib' data, it doesn't address programming a FIB, but rethinking the programming of the fib a bit could, I bet, even get us to a palatable point for a longer while, in a relatively short period of time.
Try telling this to a vendor that uses these common components (eg: Juniper) We have had numerous performance issues that have been attributed to software defects they haven't observed on their 'modern' hardware, but is visible in the prior generation or few back of routing engines. There's also the problem of fitting the data in the appropriate SRAM on a linecard which is very expensive on a per-bit basis to purchase and on a per-watt basis to operate. This is why some folks have TCAM based platforms, which have their own tradeoffs. We all can't just forward with N*10GE interfaces in a x86_64 platform. That may work if your scale is small, but when you're quite large the dynamics change considerably. Also, a "modern router" might look like this: cisco ASR9K Series (Intel 686 F6M14S4) processor with 12582912K bytes of memory. Intel 686 F6M14S4 processor at 2134MHz, Revision 2.174 vs Cisco 7206VXR (NPE-G1) processor (revision B) with 983040K/65536K bytes of memory. Processor board ID 23671962 SB-1 CPU at 700MHz, Implementation 1025, Rev 0.2, 512KB L2 Cache Those that are confusing the two need to be sent to reeducation camp :) - Jared
On Tue, Mar 19, 2013 at 1:57 PM, Jared Mauch <jared@puck.nether.net> wrote:
On Mar 19, 2013, at 1:50 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Tue, Mar 19, 2013 at 1:45 PM, David Conrad <drc@virtualized.org> wrote:
On Mar 19, 2013, at 10:12 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
There's nothing inherent in BGP that would not work with an unconstrained growth of the routing table, right? You just need enough bandwidth and interrupts to deal with updates.
With enough thrust, pigs fly quite well. Landing can get messy though...
I was being serious... the current 'bgp unconstrained dies' problem isn't such a problem if you have (today): 4-8 cores 16 gb ram ssd gigabit ethernet
or as you'd call this, your desktop computer... trying to do this on a 600mhz mips with 512mb ram is, clearly, a problem. put modern hardware to work and it gets simpler. Yes, the above addresses getting/sending 'rib' data, it doesn't address programming a FIB, but rethinking the programming of the fib a bit could, I bet, even get us to a palatable point for a longer while, in a relatively short period of time.
Try telling this to a vendor that uses these common components (eg: Juniper)
you mean a vendor embeded in their current design? ok.
We have had numerous performance issues that have been attributed to software defects they haven't observed on their 'modern' hardware, but is visible in the prior generation or few back of routing engines.
There's also the problem of fitting the data in the appropriate SRAM on a linecard which is very expensive on a per-bit basis to purchase and on a per-watt basis to operate. This is why some folks have TCAM based platforms, which have their own tradeoffs.
right, these are design choices they made ~10 years ago and didn't upgrade along with the problem space. I'm really saying that we almost have to take a fresh slate look at the problem... 'if you could do things again, without the constraints of what we have today'
We all can't just forward with N*10GE interfaces in a x86_64 platform. That may work if your scale is small, but when you're quite large the dynamics change considerably.
sure thing.
Also, a "modern router" might look like this:
cisco ASR9K Series (Intel 686 F6M14S4) processor with 12582912K bytes of memory. Intel 686 F6M14S4 processor at 2134MHz, Revision 2.174
vs
Cisco 7206VXR (NPE-G1) processor (revision B) with 983040K/65536K bytes of memory. Processor board ID 23671962 SB-1 CPU at 700MHz, Implementation 1025, Rev 0.2, 512KB L2 Cache
Those that are confusing the two need to be sent to reeducation camp :)
yes, still all single threaded and single core and ... :( 32bit, woot :( so constrained to some extent on max-memory and address space. also, I don't think this is 'just that simple'... but the tools exist.
Composed on a virtual keyboard, please forgive typos. On Mar 19, 2013, at 13:57, Jared Mauch <jared@puck.nether.net> wrote:
On Mar 19, 2013, at 1:50 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Tue, Mar 19, 2013 at 1:45 PM, David Conrad <drc@virtualized.org> wrote:
On Mar 19, 2013, at 10:12 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
There's nothing inherent in BGP that would not work with an unconstrained growth of the routing table, right? You just need enough bandwidth and interrupts to deal with updates.
With enough thrust, pigs fly quite well. Landing can get messy though...
I was being serious... the current 'bgp unconstrained dies' problem isn't such a problem if you have (today): 4-8 cores 16 gb ram ssd gigabit ethernet
or as you'd call this, your desktop computer... trying to do this on a 600mhz mips with 512mb ram is, clearly, a problem. put modern hardware to work and it gets simpler. Yes, the above addresses getting/sending 'rib' data, it doesn't address programming a FIB, but rethinking the programming of the fib a bit could, I bet, even get us to a palatable point for a longer while, in a relatively short period of time.
Try telling this to a vendor that uses these common components (eg: Juniper)
We have had numerous performance issues that have been attributed to software defects they haven't observed on their 'modern' hardware, but is visible in the prior generation or few back of routing engines.
There's also the problem of fitting the data in the appropriate SRAM on a linecard which is very expensive on a per-bit basis to purchase and on a per-watt basis to operate. This is why some folks have TCAM based platforms, which have their own tradeoffs.
Has anyone implemented (properly) FIB compression? Will LISP get is an order of magnitude gain over proper FIB compression? (Honest question, don't know enough about LISP to say.)
We all can't just forward with N*10GE interfaces in a x86_64 platform. That may work if your scale is small, but when you're quite large the dynamics change considerably.
Also, a "modern router" might look like this:
cisco ASR9K Series (Intel 686 F6M14S4) processor with 12582912K bytes of memory. Intel 686 F6M14S4 processor at 2134MHz, Revision 2.174
vs
Cisco 7206VXR (NPE-G1) processor (revision B) with 983040K/65536K bytes of memory. Processor board ID 23671962 SB-1 CPU at 700MHz, Implementation 1025, Rev 0.2, 512KB L2 Cache
Those that are confusing the two need to be sent to reeducation camp :)
And do you have the slightest problem with BGP churn / convergence / filtering / etc. on the former? -- TTFN, patrick
On Mar 19, 2013, at 2:15 PM, "Patrick W. Gilmore" <patrick@ianai.net> wrote:
Has anyone implemented (properly) FIB compression?
Since you say FIB, yes. If all RIB prefixes and their sub-prefixes go the same next-hop, then you don't need to install additional entries in the FIB. Some platforms only can install ~8k/second. Reducing duplicate entries is always helpful. One can't do RIB compression due to the nature of the protocol. - Jared
On 2013-03-19, at 13:50, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Tue, Mar 19, 2013 at 1:45 PM, David Conrad <drc@virtualized.org> wrote:
On Mar 19, 2013, at 10:12 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
There's nothing inherent in BGP that would not work with an unconstrained growth of the routing table, right? You just need enough bandwidth and interrupts to deal with updates.
With enough thrust, pigs fly quite well. Landing can get messy though...
I was being serious... the current 'bgp unconstrained dies' problem isn't such a problem if you have (today):
We've been watching unconstrained growth of the routing table for quite some time, and the result is an Internet that still keeps the unwashed masses Harlem shaking. It doesn't *look* like a picture of dramatic, world-ending instability. There's no obvious indicator in the story so far to confirm the prediction "unconstrained growth of the routing table will kill the Internet". Which is not to say that the prediction is wrong, but at some point you've got to look at the guy wearing the sign with the crazed expression and wonder whether he's a couple of sandwiches short of a picnic. You could say that growth in the routing system to date *has* been constrained, by economics or regulation or RIR policies or uptake of 32-bit AS numbers or something else, and express concern that those constraints are changing and that change is dangerous. But after a while your hands get tired and you have to stop waving them. We've been saying "unconstrained growth bad" for BGP for years. Presumably we're not all insane. Where is the science? Joe
On Tue, Mar 19, 2013 at 2:12 PM, Joe Abley <jabley@hopcount.ca> wrote:
Which is not to say that the prediction is wrong, but at some point you've got to look at the guy wearing the sign with the crazed expression and wonder whether he's a couple of sandwiches short of a picnic.
i also brought wine to the picnic, want to come with me now?
On Mar 19, 2013, at 2:12 PM, Joe Abley <jabley@hopcount.ca> wrote:
We've been saying "unconstrained growth bad" for BGP for years. Presumably we're not all insane. Where is the science?
I think there is a lot of fear around this topic. I'm waiting to see the great meltdown at 512k fib entries in networks. We saw the same at 128k and 256k with some platforms. The impact on 512k will be just as great if not larger, but also very transient. I've observed a great deal of asymmetrical BGP participants in recent years. They send a set of routes, sometimes small for their own global good, but take only on-net or default routes from their providers. There is also the fact that many traffic-engineering techniques are quite coarse due to the protocol design. The days of using prepending and aggregation/deaggregation are still with us, even when more sophisticated methods (communities, etc..) exist. I'm starting to decide that the real issue is that most people just can't route (including some major networks). The system works because the broken part gets greased, but there are a lot of cosmetic and non-cosmetic defects that linger because people don't realize they are there or are a problem. If you want data on that, including my minimalistic "faux" science, there is plenty to be had. - Jared
[Thanx for changing subject - should have done it myself a couple posts ago.] Composed on a virtual keyboard, please forgive typos. On Mar 19, 2013, at 14:26, Jared Mauch <jared@puck.nether.net> wrote:
On Mar 19, 2013, at 2:12 PM, Joe Abley <jabley@hopcount.ca> wrote:
We've been saying "unconstrained growth bad" for BGP for years. Presumably we're not all insane. Where is the science?
I think there is a lot of fear around this topic. I'm waiting to see the great meltdown at 512k fib entries in networks. We saw the same at 128k and 256k with some platforms. The impact on 512k will be just as great if not larger, but also very transient.
No way we transition to LISP (or anything else) before hitting that wall. So sit back & enjoy the fireworks. My guess is they will be I impressive and short-lived. But I've been wrong before.
I've observed a great deal of asymmetrical BGP participants in recent years. They send a set of routes, sometimes small for their own global good, but take only on-net or default routes from their providers.
There is also the fact that many traffic-engineering techniques are quite coarse due to the protocol design. The days of using prepending and aggregation/deaggregation are still with us, even when more sophisticated methods (communities, etc..) exist. I'm starting to decide that the real issue is that most people just can't route (including some major networks). The system works because the broken part gets greased, but there are a lot of cosmetic and non-cosmetic defects that linger because people don't realize they are there or are a problem. If you want data on that, including my minimalistic "faux" science, there is plenty to be had.
I'm wondering why that will be any better if we swap out the underlying protocol. It's not like trying something new will -increase- the average clue level of the monkeys banging on keyboards trying to accidentally compose a routing sonnet. And up-ending the installed base is almost certainly going to decrease the d(clue)/dt, as well as the second derivative. "Never underestimate the power of human stupidity." Which is all just a fancy way of saying you can't fix people being idiots by changing a protocol, or hardware, or ... well, anything. -- TTFN, patrick
Patrick, On Mar 19, 2013, at 12:07 PM, Patrick W. Gilmore <patrick@ianai.net> wrote:
Which is all just a fancy way of saying you can't fix people being idiots by changing a protocol, or hardware, or ... well, anything.
One of the advantages I see in LISP(-like) solutions is that it allows multi-homing without having to do BGP... Regards, -drc
David Conrad wrote:
One of the advantages I see in LISP(-like) solutions is that it allows multi-homing without having to do BGP...
By having a lot larger table than BGP. http://datatracker.ietf.org/doc/draft-ietf-lisp-architecture/?include_text=1 It should be noted that the caching spoken of here is likely not classic caching, where there is a fixed/limited size cache, and entries have to be discarded to make room for newly needed entries. The economics of memory being what they are, there is no reason to discard mappings once they have been loaded (although of course implementations are free to chose to do so, if they wish to). Worse, the table is updated so frequently. http://datatracker.ietf.org/doc/draft-ietf-lisp-introduction/?include_text=1 A node may have more than one RLOC, or its RLOC may change over time (e.g. if the node is mobile), but it keeps the same EID. Assuming that there are 4G mobile devices in the world, the mapping table has more than 4G entries each updated every minute or second. The problem of LISP is that it breaks the end to end principle to introduce intelligent intermediate entities of ITR and ETR. Mobility can best be handled end to end by end systems of MN, HA and, optionally, CN. Masataka Ohta PS Considering that the Internet is connectionless because all the routers have routing tables covering all the IP addresses in realtime, LISP won't be operational unless most of routers in DFZ have full mapping table in realtime.
On Mar 20, 2013, at 6:23 AM, Masataka Ohta wrote:
The problem of LISP is that it breaks the end to end principle to introduce intelligent intermediate entities of ITR and ETR.
It is always amusing to see people allude to the end-to-end principle to support their arguments, when in fact the end-to-end principle is either inapplicable to the topic at hand, or actually lends support to the opposite of their arguments.
Considering that the Internet is connectionless because all the routers have routing tables covering all the IP addresses in realtime, LISP won't be operational unless most of routers in DFZ have full mapping table in realtime.
LISP gains value as more of the edge is incorporated into the system. This doesn't mean it has no value during intermediate stages of deployment. There are cogent arguments to be made against LISP and LISP-like systems. But none of those arguments have been raised in this thread, so far. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
Dobbins, Roland wrote:
It is always amusing to see people allude to the end-to-end principle to support their arguments, when in fact the end-to-end principle is either inapplicable to the topic at hand, or actually lends support to the opposite of their arguments.
Abstract nonsense.
There are cogent arguments to be made against LISP and LISP-like systems. But none of those arguments have been raised in this thread, so far.
Never ignore the following problem: Assuming that there are 4G mobile devices in the world, the mapping table has more than 4G entries each updated every minute or second. Masataka Ohta
On Mar 20, 2013, at 7:05 AM, Masataka Ohta wrote:
Abstract nonsense.
No, it isn't. The *actual* end-to-end principle states that whenever possible and whenever it makes sense, application-specific functionality ought to be incorporated into end-nodes rather than into intermediary systems. In the case of LISP, it can be noted that either a) the end-to-end principle has no relevance, or b) LISP is closer to adherence to the end-to-end principle than the current routing system. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
Dobbins, Roland wrote:
The *actual* end-to-end principle states that whenever possible and whenever it makes sense, application-specific functionality ought to be incorporated into end-nodes rather than into intermediary systems.
Wrong. See below how it is stated.
b) LISP is closer to adherence to the end-to-end principle than the current routing system.
W.r.t. multihoming, neither follows the end to end principle of: http://groups.csail.mit.edu/ana/Publications/PubPDFs/End-to-End%20Arguments%... The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the endpoints of the communication system. Therefore, providing that questioned function as a feature of the communication system itself is not possible. (Sometimes an incomplete version of the function provided by the communication system may be useful as a performance enhancement.) Masataka Ohta
On Mar 20, 2013, at 8:39 AM, Masataka Ohta wrote:
Wrong.
No, it isn't wrong. That's how it's interpreted: 'The principle, called the end-to-end argument, suggests that functions placed at low levels of a system may be redundant or of little value when compared with the cost of providing them at that low level.' and 'The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the endpoints of the communication system. Therefore, providing that questioned function as a feature of the communication system itself is not possible. (Sometimes an incomplete version of the function provided by the communication system may be useful as a performance enhancement.)'
W.r.t. multihoming, neither follows the end to end principle of:
Yes, which is why I said that it doesn't really apply in the first place. But if one insists on viewing it through the prism of the end-to-end principle, LISP adheres to it more than does the current routing system. Anyway, I'm done feeding this particular troll for good. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
Dobbins, Roland wrote:
But if one insists on viewing it through the prism of the end-to-end principle, LISP adheres to it more than does the current routing system.
It merely means you are using a wrong prism. Let's consider a very basic multihoming with two locators, one of which is blackholed. Then, how can an ITR, which initially choose a blackholed locator, know that the locator is not working and fall back to another locator? In general, we should assume protocol used is something unknown to the ITR (not TCP, of course) and/or not assume the ITR is on return path so that the ITR can't have any "knowledge" that an end system receiving any reply. In this case, The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the endpoints of the communication system. means only the higher stack of the end system have "the knowledge" that sent packets are not replied. But, the end system can't "help" the ITR, because it does not even know the ITR exist. Therefore, Therefore, providing that questioned function as a feature of the communication system itself is not possible. providing that questioned function of destination locator selection for multihoming as a feature of ITR is not possible.
Anyway, I'm done feeding this particular troll for good.
Use a looking glass to see a troll. Always check the original paper to avoid wrong interpretations of yours. Masataka Ohta PS Is it time to shutdown LISP WG?
On Wed, Mar 20, 2013 at 1:40 AM, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
Then, how can an ITR, which initially choose a blackholed locator, know that the locator is not working and fall back to another locator?
In general, we should assume protocol used is something unknown to the ITR (not TCP, of course) and/or not assume the ITR is on return path so that the ITR can't have any "knowledge" that an end system receiving any reply.
I can't speak for LISP per se, but the general solution for map-encap systems like LISP is that the ITR tags the first packet to the ETR and some percentage of subsequent packets to the ETR with an ack request. If it doesn't get an ack from the ETR (not the final destination), then the ETR or the path to it is depreferenced. The ack-request tagged tunnel packets and the acks sent in response are protocol-agnostic with respect to the inner packets between the source and destination. If the ETR is unavailable, those carried packets will be dropped. The ITR cares only about promptly discovering that the ETR is unavailable so that it can select an alternate ETR which works. The error correction mechanisms in the carried protocols then take over and resolve the lost packets. The path from the ETR to the destination, and by extension the full path from the ITR to the destination, is not the ITR's responsibility. Some local system is responsible for detecting connectivity between the ETR and destination and updating the destination-to-ETR map accordingly. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
William Herrin wrote:
I can't speak for LISP per se, but the general solution for map-encap systems like LISP is that the ITR tags the first packet to the ETR and some percentage of subsequent packets to the ETR with an ack request. If it doesn't get an ack from the ETR (not the final destination), then the ETR or the path to it is depreferenced.
As the ETR is not the final destination, it is subject to blackholing after ETR, which means: The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the endpoints of the communication system. Granted that it is no worse than multihoming by routing protocols. But, it merely means that neither BGP nor LISP works "completely and correctly".
The path from the ETR to the destination, and by extension the full path from the ITR to the destination, is not the ITR's responsibility.
It merely means that LISP is not end to end, because of not ITRs but ETRs. In addition, that the ITR, which may be located near the destination rather than source, can not receive ACKs does not mean the end system can not receive ACKs, which is ITR related lack of "completely and correctly" property.
Some local system is responsible for detecting connectivity between the ETR and destination and updating the destination-to-ETR map accordingly.
Some local system? Such a local system must, according to the end to end argument: The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the endpoints of the communication system. Therefore, providing that questioned function as a feature of the communication system itself is not possible. needs "knowledge and help of the application standing at the endpoints of the communication system" involving LISP++ specific protocol changes of the end systems. If you require all the end systems local to ETRs compliant to some new protocol, do it locally only on the end systems, from the beginning, without involving intelligent intermediate entities of *TRs only to make it work not "completely and correctly" Masataka Ohta
Hi,
As the ETR is not the final destination, it is subject to blackholing after ETR, which means:
The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the endpoints of the communication system.
Granted that it is no worse than multihoming by routing protocols.
But, it merely means that neither BGP nor LISP works "completely and correctly".
Well, yeah, if your internal routing (behind the ETR) breaks then your network is broken... Met vriendelijke groet, Sander Steffann
I appreciate everyones comments on this issue but I think you nay-sayers are going to lose. I think the future of the internet is distributed routing where the end points ultimately decide how their packets flow. I think joe 6-pack should in fact be able to be connected to as many providers as he wants, should be able to specify any mixture of connections from consumer dsl to carrier ethernet or beyond, and have the same level of service as multi-homed bgp speaking networks do today in terms of route diversity, fail-over and 'portability' (in terms of bringing your netblocks to another provider). Not a troll, just looking at the future here. Mike-
On 20 March 2013 17:30, Mike <mike-nanog@tiedyenetworks.com> wrote:
I appreciate everyones comments on this issue but I think you nay-sayers are going to lose. I think the future of the internet is distributed routing where the end points ultimately decide how their packets flow.
You have actually *heard* of BGP version 4, right? We've only been using it for 20 years, you'd have thought people would switch to it in their masses... M
On Mar 20, 2013, at 1:43 PM, Matthew Walster <matthew@walster.org> wrote:
On 20 March 2013 17:30, Mike <mike-nanog@tiedyenetworks.com> wrote:
I appreciate everyones comments on this issue but I think you nay-sayers are going to lose. I think the future of the internet is distributed routing where the end points ultimately decide how their packets flow.
You have actually *heard* of BGP version 4, right? We've only been using it for 20 years, you'd have thought people would switch to it in their masses...
What's interesting is I see more people (eg: datacenter operators) pushing for BGP in their devices, and scale in them because it is well fed and maintained vs trusting/using OSPF/CLNS/ISIS and getting the performance limits there fixed. They would rather use the TCP timeouts vs OSPF timeouts for link discovery and routing performance. This tells me there is perhaps a gap in capabilities or performance that isn't well documented and being worked around. - Jared
On Mar 20, 2013, at 1:30 PM, Mike <mike-nanog@tiedyenetworks.com> wrote:
I appreciate everyones comments on this issue but I think you nay-sayers are going to lose. I think the future of the internet is distributed routing where the end points ultimately decide how their packets flow. I think joe 6-pack should in fact be able to be connected to as many providers as he wants, should be able to specify any mixture of connections from consumer dsl to carrier ethernet or beyond, and have the same level of service as multi-homed bgp speaking networks do today in terms of route diversity, fail-over and 'portability' (in terms of bringing your netblocks to another provider). Not a troll, just looking at the future here.
I certainly think there's a lot that can be done at middle-layers, eg: tunnels to a few different providers. I can be on a Comcast CM and ATT DSL link and establish a link to a tunnel destination in Chicago that is low-latency for me and the bits will all flow that way. The last mile loop problem though? As of today, neither of those two providers reaches my home with their service. I doubt I'm going to see a lot of choice unless the market changes considerably. The last-mile costs are all in the contractors, permits and engineering studies for weight on the poles. (Or directional boring/conduit costs underground). Locally the permit is $200/road crossed, either above or under, this starts to contribute to the build cost in unexpected ways. If you looked at the google fiber proposal, they wanted space on both poles and otherwise to minimize permitting and other costs. Last time I talked to someone about outside plant construction the answer was the cost was functionally strand-agnostic. That tells me the cost isn't in the cabling. - Jared
I certainly think there's a lot that can be done at middle-layers, eg: tunnels to a few different providers. I can be on a Comcast CM and ATT DSL link and establish a link to a tunnel destination in Chicago that is low-latency for me and the bits will all flow that way.
The last mile loop problem though?
sweden and japan, among others, have some experiences (good and mediocre) in this area randy
On 3/20/13 11:30 AM, Mike wrote:
I appreciate everyones comments on this issue but I think you nay-sayers are going to lose. I think the future of the internet is distributed routing where the end points ultimately decide how their packets flow. I think joe 6-pack should in fact be able to be connected to as many providers as he wants, should be able to specify any mixture of connections from consumer dsl to carrier ethernet or beyond, and have the same level of service as multi-homed bgp speaking networks do today in terms of route diversity, fail-over and 'portability' (in terms of bringing your netblocks to another provider). Not a troll, just looking at the future here.
Why did I just suddenly have a flashback to this? https://www.nanog.org/mailinglist/mailarchives/old_archive/2003-10/msg01440.... -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
Sander Steffann wrote:
As the ETR is not the final destination, it is subject to blackholing after ETR, which means:
The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the endpoints of the communication system.
Granted that it is no worse than multihoming by routing protocols.
But, it merely means that neither BGP nor LISP works "completely and correctly".
Well, yeah, if your internal routing (behind the ETR) breaks then your network is broken...
No, what can break is internal routing of one of your ISP, which is why you, an end user, want multihoming. William Herrin wrote: Masataka Ohta
On Wed, Mar 20, 2013 at 11:42 AM, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
William Herrin wrote:
I can't speak for LISP per se, but the general solution for map-encap systems like LISP is that the ITR tags the first packet to the ETR and some percentage of subsequent packets to the ETR with an ack request. If it doesn't get an ack from the ETR (not the final destination), then the ETR or the path to it is depreferenced.
As the ETR is not the final destination, it is subject to blackholing after ETR, which means:
The path from the ETR to the destination, and by extension the full path from the ITR to the destination, is not the ITR's responsibility.
Already had the correct answer in there; you didn't need to restate it.
It merely means that LISP is not end to end
Yeah. LISP provides virtual packet-switched data circuits. Like a metro-ethernet from here to my ISP. The protocols transiting LISP are what's end to end. And no aspect of LISP that I'm aware of compels changed behavior by those protocols.
Some local system is responsible for detecting connectivity between the ETR and destination and updating the destination-to-ETR map accordingly.
Some local system?
Yeah, you know, like OSPF or EIGRP. Just like exporting a route from the IGP to the EGP except that you're exporting a route from the IGP to the LISP map instead. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
William Herrin wrote:
Some local system is responsible for detecting connectivity between the ETR and destination and updating the destination-to-ETR map accordingly.
Some local system?
Yeah, you know, like OSPF or EIGRP. Just like exporting a route from the IGP to the EGP except that you're exporting a route from the IGP to the LISP map instead.
You assume an end user's network exchange route with its ISP. Though it causes a lot of problems, OK. Even then, that a route from an ETR of an ISP to an end system in end user's network is blackholed means that a routing protocol tells the ETR that there is a route to the end system. Masataka Ohta
Hi, On 20 Mar. 2013, at 06:40 , Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote: [snip]
Then, how can an ITR, which initially choose a blackholed locator, know that the locator is not working and fall back to another locator?
In LISP you can check reachability with the echo-nonce function. [snip]
PS
Is it time to shutdown LISP WG?
I do not think this is the right mailing list to discuss something like this. ciao Luigi
On 20 Mar. 2013, at 01:05 , Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
Dobbins, Roland wrote:
It is always amusing to see people allude to the end-to-end principle to support their arguments, when in fact the end-to-end principle is either inapplicable to the topic at hand, or actually lends support to the opposite of their arguments.
Abstract nonsense.
There are cogent arguments to be made against LISP and LISP-like systems. But none of those arguments have been raised in this thread, so far.
Never ignore the following problem:
Assuming that there are 4G mobile devices in the world, the mapping table has more than 4G entries each updated every minute or second.
Masataka that's simply not true. See my proviso email. ciao Luigi
Masataka Ohta
Hi Masataka, On 20 Mar. 2013, at 00:23 , Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:
David Conrad wrote:
One of the advantages I see in LISP(-like) solutions is that it allows multi-homing without having to do BGP...
By having a lot larger table than BGP.
http://datatracker.ietf.org/doc/draft-ietf-lisp-architecture/?include_text=1
It should be noted that the caching spoken of here is likely not classic caching, where there is a fixed/limited size cache, and entries have to be discarded to make room for newly needed entries. The economics of memory being what they are, there is no reason to discard mappings once they have been loaded (although of course implementations are free to chose to do so, if they wish to).
LISP will not have huge caches: (1) http://www.ietf.org/proceedings/69/slides/RRG-4.pdf and more recently: (2) https://www.net.t-labs.tu-berlin.de/papers/KIF-ADDITLCAWISKAI-11.pdf
Worse, the table is updated so frequently.
FIrst of all the table a cache is filled on-demand, so you update only what you need, secondly (1) shows that this refresh traffic is in the same order of magnitude of DNS requests. If you are able to support DNS you are able to deal with LISP cache update traffic.
http://datatracker.ietf.org/doc/draft-ietf-lisp-introduction/?include_text=1
A node may have more than one RLOC, or its RLOC may change over time (e.g. if the node is mobile), but it keeps the same EID.
Assuming that there are 4G mobile devices in the world, the mapping table has more than 4G entries each updated every minute or second.
This is true in a push model like BGP not in LISP, which is a pull model (on-demand).
The problem of LISP is that it breaks the end to end principle to introduce intelligent intermediate entities of ITR and ETR.
true
Mobility can best be handled end to end by end systems of MN, HA and, optionally, CN.
which rely on intelligent anchor nodes spread in the network, where is the difference? Luigi
Masataka Ohta
PS
Considering that the Internet is connectionless because all the routers have routing tables covering all the IP addresses in realtime, LISP won't be operational unless most of routers in DFZ have full mapping table in realtime.
On Mar 19, 2013, at 4:48 PM, David Conrad <drc@virtualized.org> wrote:
Patrick,
On Mar 19, 2013, at 12:07 PM, Patrick W. Gilmore <patrick@ianai.net> wrote:
Which is all just a fancy way of saying you can't fix people being idiots by changing a protocol, or hardware, or ... well, anything.
One of the advantages I see in LISP(-like) solutions is that it allows multi-homing without having to do BGP...
What i've observed over the years is many of the reasons people use BGP and PI space is to make it easier to change internet providers. Much of this originally was due to everything being hardcoded, long dns caches and TTLs, etc.. With the exception of a few devices (eg: site-to-site VPN, etc..) these are a lot easier to handle than they were 15 years ago. I recall renumbering two different dns servers at one point, and we would always get something weird happening where the old domain/IP would come up with someones new registration. The process is mature, and I suspect many of the issues could be mitigated. Large datacenters now trust and are renumbered with DHCP. Installation of hosts happens quickly. moving of services happens quickly. The challenge is the people who are not there yet. - jared
With all due respect to the emperors involved, the complete "zomg, don't grow the routing table!" argument includes the seldom-stated tagline, "....faster than I want to replace my existing hardware." IOW, it's never been about the tech, or "the good of the network," it's about the capex. I'm not saying that isn't a noble, useful goal, especially when you consider the SMBs who are going to be entering the multihoming market soon as David pointed out, and also (and perhaps more importantly) the folks in the emerging economies that are still ramping up their Internet access. But focusing on the technological problems is a waste of time since they all have technological solutions. Doug
On Tue, Mar 19, 2013 at 4:16 PM, Doug Barton <dougb@dougbarton.us> wrote:
With all due respect to the emperors involved, the complete "zomg, don't grow the routing table!" argument includes the seldom-stated tagline, "....faster than I want to replace my existing hardware."
Hi Doug, Sure, it's about the bucks. Each additional route is a collective loss of around $8,000 per year, and all of it overhead for which we can't bill specific customers. We'd prefer less of that than more. http://bill.herrin.us/network/bgpcost.html
IOW, it's never been about the tech, or "the good of the network," it's about the capex.
Actually, that's not true. There was a point in the '90s where the route count very nearly outstripped our ability to build routers which could handle it. That's when we first hit the brakes. Since then our ability to build routers has improved faster than the routing table has grown, but a material change in the rate of growth could quickly wipe out that lead. Routine full-bore multihoming for SMBs and SOHOs will have to wait for a less consumptive technology than BGP. (And if you've got the cash, call me 'cause I have a plan.) Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
Chris, On Mar 19, 2013, at 10:50 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
With enough thrust, pigs fly quite well. Landing can get messy though... I was being serious...
As was I.
put modern hardware to work and it gets simpler.
Yes, applying more thrust makes things simpler: all you need is money, bandwidth capacity, rackspace, power, cooling, time (to replace old equipment), etc. Moore's Law will probably save us. Probably. But I know you know all of this. Where I think things get a bit more interesting is if you assume smaller and smaller businesses start seeing "always on" Internet sufficiently important as to justify multihoming with PI. In the US alone there are 6M SMEs with payrolls (21M without). Perhaps router vendors should adopt Doritos motto: "crunch all you want, we'll make more"... Regards, -drc
On Tue, Mar 19, 2013 at 2:12 PM, David Conrad <drc@virtualized.org> wrote:
Chris,
On Mar 19, 2013, at 10:50 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
With enough thrust, pigs fly quite well. Landing can get messy though... I was being serious...
As was I.
:)
put modern hardware to work and it gets simpler.
Yes, applying more thrust makes things simpler: all you need is money, bandwidth capacity, rackspace, power, cooling, time (to replace old equipment), etc. Moore's Law will probably save us. Probably.
But I know you know all of this.
tli says moore's law doesn't apply to routing gear (in the large-hardware world)... he's been wrong once or twice, but his slides seemed convincing (to me). I take your point though, there's a cost to getting out of the hole (if there is a hole,ie: @jabley) I also think we don't have to do this 'today', but getting the right plans in place to migrate in the right direction seems like an ok plan too.
Where I think things get a bit more interesting is if you assume smaller and smaller businesses start seeing "always on" Internet sufficiently important as to justify multihoming with PI. In the US alone there are 6M SMEs with payrolls (21M without). Perhaps router vendors should adopt Doritos motto: "crunch all you want, we'll make more"...
no doubt, this is marshall's numbers (or a form of them) from ~7+ yrs ago now? which ted and I used ~6 yrs ago to (un)successfully argue that we (the intertubes) need something more than multihoming as we have it today in ipv4/ipv6... sharing that state across the globe is expensive in today's hardware (or yesteryear's hardware). I totally think that as the intertubes become more and more 'critical' to people's business(es) we'll see more and more regulation that, as a side effect, leads to more reliable connectivity being demanded. That'll be nice, in a way :) anyway, we seem to mostly agree, which again makes me realize I'm not crazy... but I stil have wine and sandwiches, come along with jabley and I? -chris
Chris, On Mar 19, 2013, at 11:27 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
I also think we don't have to do this 'today', but getting the right plans in place to migrate in the right direction seems like an ok plan too.
+1 For that plan I personally like the idea of the layer of indirection separating identification from location as I see it having lots of benefits like scalability, site multi-homing without BGP-fu, provider independence, etc. albeit at a complexity/performance cost. The advantage I see in LISP(-like) approaches is that it pushes that cost out to the edge where all you need to do is give your hamsters vitamins, leaving the core untouched.
In the US alone there are 6M SMEs with payrolls (21M without). Perhaps router vendors should adopt Doritos motto: "crunch all you want, we'll make more"...
no doubt, this is marshall's numbers (or a form of them) from ~7+ yrs ago now?
I suppose from the same source (http://www.census.gov/econ/smallbus.html)
anyway, we seem to mostly agree, which again makes me realize I'm not crazy...
The more likely alternative is that we both are.
but I stil have wine and sandwiches, come along with jabley and I?
I'll bring the deep fried hamster. Regards, -drc
On Tue, Mar 19, 2013 at 2:44 PM, David Conrad <drc@virtualized.org> wrote:
anyway, we seem to mostly agree, which again makes me realize I'm not crazy...
The more likely alternative is that we both are.
doh! the unexpected third option!
but I stil have wine and sandwiches, come along with jabley and I?
I'll bring the deep fried hamster.
hurray! protein!
Composed on a virtual keyboard, please forgive typos. On Mar 19, 2013, at 13:45, David Conrad <drc@virtualized.org> wrote:
On Mar 19, 2013, at 10:12 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
There's nothing inherent in BGP that would not work with an unconstrained growth of the routing table, right? You just need enough bandwidth and interrupts to deal with updates.
With enough thrust, pigs fly quite well. Landing can get messy though...
The demise of BGP from unrestrained table growth has been predicted for decades. Part of this is because my million dollar router has a slower central proc and less RAM than my $2k laptop. So yeah, pigs can fly with sufficient thrust, but we are currently using hamsters on a wheel level thrust. There is a middle ground. Before we claim BGP is dead again, let's take a moment and ensure we didn't cripple it first. The protocol, as Chris said, has no inherent problems scaling for the a while at least. It may not be "optimal", but there is something to be said for a protocol with a 100% installed base that works, and works well. -- TTFN, patrick
On Mar 19, 2013, at 11:11 AM, Patrick W. Gilmore <patrick@ianai.net> wrote:
The demise of BGP from unrestrained table growth has been predicted for decades.
The exhaustion of IPv4 has been predicted for decades.
Part of this is because my million dollar router has a slower central proc and less RAM than my $2k laptop. So yeah, pigs can fly with sufficient thrust, but we are currently using hamsters on a wheel level thrust. There is a middle ground.
Sure. However, oddly, vendors haven't been rushing to do this. I suspect (but do not know for sure) this might be because the market for non-hamster driven routers is pretty small and the requirements that market has can be quite hard to meet ("you want packets switched how fast? with how many millions of knobs?"). Maybe the market will change or a new entrant will come along and upset the applecart (and not be acquired to be killed).
Before we claim BGP is dead again, let's take a moment and ensure we didn't cripple it first. The protocol, as Chris said, has no inherent problems scaling for the a while at least.
DId someone claim BGP was dead?
It may not be "optimal", but there is something to be said for a protocol with a 100% installed base that works, and works well.
LISP doesn't replace BGP. It merely adds a layer of indirection so you don't have to propagate identity information along with routing topology, allowing much greater aggregation. Regards, -drc
In a message written on Tue, Mar 19, 2013 at 11:33:33AM -0700, David Conrad wrote:
LISP doesn't replace BGP. It merely adds a layer of indirection so you don't have to propagate identity information along with routing topology, allowing much greater aggregation.
The problem with LISP is that when the complexity of the entire system is taken into account it is not signficantly more efficient than the current system. Even if it works perfectly, it makes no economic sense to spend the time and money to swap out the current system for something with approximately the same scaling properties and costs going forward. Any replacement would probably have to be an order of magnitude better at least to justify the pain of switching. LISP also has some potential downsides at Internet scale. Those who remember the 7500 platform when caching was the rage know what happens when you have to flush the cache for example. A LISP network is a similar model, with LISP nodes caching rather than linecards. There is potential for distributed uglyness. However, the LISP folks made a rather smart course correction in my opinion, and one I never would have thought to make. The LISP testbed network proved that LISP was a nice way to overlay an arbitrary topoligy on top of the existing Internet. Compared to many other "VPN" solutions it has a lot of nice properties. Some folks are now using LISP to network a collection of sites using commodity internet access making very resiliant topologies quickly and easily. I suspect LISP may find a very productive niche. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On Mar 20, 2013, at 1:57 AM, Leo Bicknell wrote:
However, the LISP folks made a rather smart course correction in my opinion, and one I never would have thought to make. The LISP testbed network proved that LISP was a nice way to overlay an arbitrary topoligy on top of the existing Internet.
FYI, this wasn't a 'course-correction'. It is the result of deliberate design choices and was identified as a key benefit of LISP during the initial discussions around what became known as LISP. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
Leo, On Mar 19, 2013, at 11:57 AM, Leo Bicknell <bicknell@ufp.org> wrote:
LISP doesn't replace BGP. It merely adds a layer of indirection so you don't have to propagate identity information along with routing topology, allowing much greater aggregation. The problem with LISP is that when the complexity of the entire system is taken into account it is not signficantly more efficient
In a message written on Tue, Mar 19, 2013 at 11:33:33AM -0700, David Conrad wrote: than the current system.
When was the last time you (as a network operator) cared about the efficiency of the entire system? LISP (and similar) system are inherently more complex because they're adding a new element to the network -- TANSTAAFL. The point is that the complexity is added at the edge where it is easy/cheap (per node or site). Yes, entire system complexity goes up. However from the perspective of the core where life is fast/expensive, complexity goes down since identity is separated from location.
A LISP network is a similar model, with LISP nodes caching rather than linecards.
You're comparing the equivalent of a DNS lookup with a FIB lookup. Yes, there is a performance hit when you do the mapping of identity to location (TANSTAAFL), however this is at the edge in the millisecond DRAM-stored connection initiation world, not in the core in the nanosecond SRAM-stored packet forwarding world. Regards, -drc
In a message written on Tue, Mar 19, 2013 at 02:11:54PM -0400, Patrick W. Gilmore wrote:
The demise of BGP from unrestrained table growth has been predicted for decades. Part of this is because my million dollar router has a slower central proc and less RAM than my $2k laptop. So yeah, pigs can fly with sufficient thrust, but we are currently using hamsters on a wheel level thrust. There is a middle ground.
Many of the people arguing over the demise of BGP fail to realize this is a plot over time, and that individual points may be interesting but also misleading. The computational complexity of BGP grows over time. The drivers are well known, more routes, and more paths to those routes. This can be calculated as a theoretical (like Big-O notation style), or plotted as a historical CPU-Ops/Route type metric. This plot has some interesting step functions up and down, like the introduction of CIDR, or IPv6. On the other side is the computational power of CPU's and the size of RAM. These also grows over time, although sometimes not in the ways predicted. For instance single CPU cores hit a bit of a wall so the world went to a multi-core solution. When a router vendor choses a box their goal is to use the cheapest CPU and least memory that will stay ahead of the complexity curve over the expected life of the box. Most of the posts about the death of BGP center around a popular box at the time (AGS+, Cisco 7500, Cisco 6500-Sup720) where the vendor "got it wrong". Perhaps they picked too small of a CPU. Perhaps one of the interesting step functions happened in the world, or, much of the time, perhaps ISP's are using devices well past their original intended service life! Sometimes it's not even the router vendor's direct fault, Intel may have had problems delivering CPU's on time forcing a compromise, or the RAM market may have fallen apart due to a earthquake. Backing up to a long-ball view of the world, there's no reason to expect BGP computational load to exceed the capabilities of the CPU's likely to be in routers in the next 10-15 years. Sure, a platform or two here or there will have issues, but life will move on as it always did in the past. What I find interesting is that there hasn't been a stronger move to decouple control-plane from forwarding plane. When you look at most Juniper hardware, or even modern Cisco designs like an ASR1k/9k, they are virtually 100% separated internally. All that is lacking is a standardized interface across platforms. Juniper is actually closer given their internal ethernet connection model. Basically the question is why is an RE/RP specific to a particular chassis, or even vendor? Why aren't they modular and swappable? If I'm using an ASR9k for 10GE for a supercomputer and need only statics, why can't I put in a $2k "slow" RP? If my MX80 is doing duty for route-views why can't I put in the $25k quad-quad-core RP with 64G of RAM? Indeed, in many cases, why aren't these things an external, separately rack mountable box with simply an interconnect to speak to the control plane? I'm guessing the answer is profits. :( -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On Tue, Mar 19, 2013 at 2:50 PM, Leo Bicknell <bicknell@ufp.org> wrote:
Juniper is actually closer given their internal ethernet connection model. Basically the question is why is an RE/RP specific to a particular chassis, or even vendor?
openflow/sdn/pce ... congrats! you predicted correctly!
On Mar 20, 2013, at 1:59 AM, Christopher Morrow wrote:
openflow/sdn/pce ... congrats! you predicted correctly!
The whole point of these things is to obviate the RP, et. al. - which is why the network infrastructure vendors loathe them. ;> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Mar 20, 2013, at 1:50 AM, Leo Bicknell wrote:
What I find interesting is that there hasn't been a stronger move to decouple control-plane from forwarding plane.
Given the design of TCP/IP and the routing protocols, we can't really achieve true separation at the protocol level. They simply aren't intended to work with fully de-coupled, separated signal and bearer, in old-style terminology. loud As for RP interchangeability in terms of hardware, there's no economic incentive for vendors to do this, as you say. And the designs of RP/backplane/linecard are highly interdependent. Here's some of the initial thinking which led to the promulgation of LISP: <http://www.nanog.org/meetings/nanog37/presentations/vince-fuller.pdf> As others have noted, there are a lot of other advantages to separating locator from EID; with a system like LISP, one gains potentially very useful mechanisms for protocol transitions (i.e., IPv4 to IPv6), network mobility, 'cloud'-type applications, etc. But the thoughts contained in that preso comprise a great deal of the original motivation. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
i am not saying bgp and forwarding can deal with growth forever, but o over my career, the death of spinning oxide has always been two years away. yet the hardwhere jocks have continued to pull the rabbit out of the hat. perhaps, many decades later, ssds have finally caught up and physical limits are finally approaching. o a dozen or so years ago, i shared an nsf grant with lixia, dan, and others called "better bgp," based on the assumption that bgp was not gonna scale. i took the contrary position, we actually had no clear measurement showing it was not going to scale. out of this came beacons, 'happy packets', etc. so i think we need some measurements of the sky before we can judge the rate of its descent. randy
The last presentations that I saw about it said that we are going to be fine: http://www.iepg.org/2011-11-ietf82/2011-11-13-bgp2011.pdf http://www.iepg.org/2011-11-ietf82/iepg-201111.pdf Regards, as On 20/03/2013 02:53, Randy Bush wrote:
i am not saying bgp and forwarding can deal with growth forever, but
o over my career, the death of spinning oxide has always been two years away. yet the hardwhere jocks have continued to pull the rabbit out of the hat. perhaps, many decades later, ssds have finally caught up and physical limits are finally approaching.
o a dozen or so years ago, i shared an nsf grant with lixia, dan, and others called "better bgp," based on the assumption that bgp was not gonna scale. i took the contrary position, we actually had no clear measurement showing it was not going to scale. out of this came beacons, 'happy packets', etc.
so i think we need some measurements of the sky before we can judge the rate of its descent.
randy
On 20 March 2013 11:44, Arturo Servin <arturo.servin@gmail.com> wrote:
The last presentations that I saw about it said that we are going to be fine:
http://www.iepg.org/2011-11-ietf82/2011-11-13-bgp2011.pdf http://www.iepg.org/2011-11-ietf82/iepg-201111.pdf
It isn't just about "imminient death of the net predicted" though - our reliance on the current BGP model for route adverisement is restricting the deployment of better connectivity paradigms. For example I know there are enterprises that would like to multihome but they find the current mechanism a barrier to this - for a start they can't justify the size of PI space that would guarantee them entry to the global routing table. ISPs differentiate between "regular" and "BGP-capable" connections - is this desirable for the evolution of the Internet? or is it the reason that BGP appears to be able to cope, because ISPs are throttling the potential growth? LISP is about seperating the role of the ISP (as routing provider) from the end user or content provider/consumer. Aled
On 20/03/2013 09:07, Aled Morris wrote:
On 20 March 2013 11:44, Arturo Servin <arturo.servin@gmail.com <mailto:arturo.servin@gmail.com>> wrote:
The last presentations that I saw about it said that we are going to be fine:
http://www.iepg.org/2011-11-ietf82/2011-11-13-bgp2011.pdf http://www.iepg.org/2011-11-ietf82/iepg-201111.pdf
It isn't just about "imminient death of the net predicted" though - our reliance on the current BGP model for route adverisement is restricting the deployment of better connectivity paradigms.
Agree with that. But as today I do not think LISP would be the solution.
For example I know there are enterprises that would like to multihome but they find the current mechanism a barrier to this - for a start they can't justify the size of PI space that would guarantee them entry to the global routing table.
Which is good. If they cannot justify PI space may be they should not get into the global routing table. It is a problem for them, yes. Do we have a solution? Not yet.
ISPs differentiate between "regular" and "BGP-capable" connections - is this desirable for the evolution of the Internet? or is it the reason that BGP appears to be able to cope, because ISPs are throttling the potential growth?
It is an operational practice. Maintaining BGP sessions have a cost. Also, at least in the cases that I know those connections also have different SLAs which is the real cost, not just the BGP.
LISP is about seperating the role of the ISP (as routing provider) from the end user or content provider/consumer.
Yes, but as mentioned before the cost is in the edge, the benefit in the core. The economic equation is all wrong. There is not economic incentive for the edge to deploy LISP. We are facing the same problem that we have with IPv6. Now, if with LISP as an edge site I can have multihome, high availability, not to renumber my network, or any other combination of benefits and it does cost less than PI+BGP or PA+<adyourflavorofNAThere> then it may work.
Aled
Regards, as
Arturo, On Mar 20, 2013, at 5:32 AM, Arturo Servin <arturo.servin@gmail.com> wrote:
For example I know there are enterprises that would like to multihome but they find the current mechanism a barrier to this - for a start they can't justify the size of PI space that would guarantee them entry to the global routing table.
Which is good. If they cannot justify PI space may be they should not get into the global routing table.
The implication of this statement is that if you cannot afford the RIR fees, the routers, the technical expertise to run those routers, the additional opex associated with "BGP-capable" Internet connectivity, etc., the services/content you provide don't deserve resiliency/redundancy/etc. I have trouble seeing how this can be called "good". A "necessary evil given broken technology" perhaps, but not "good".
LISP is about seperating the role of the ISP (as routing provider) from the end user or content provider/consumer.
Yes, but as mentioned before the cost is in the edge, the benefit in the core.
Being able to effectively multi-home without BGP, removing the need to ever renumber, etc., all sound like benefits to the edge to me.
The economic equation is all wrong.
People keep saying this. For core providers, the equation doesn't change. Well, OK, they may lose the additional fees they get for "BGP-capable" connections and they won't have the 'benefit' of the cost of renumbering to reduce customer thrash, however they continue to get paid for providing connectivity services. They might even save some money in the long run as they won't need to replace their hamsters with guinea pigs quite as frequently. For edges, the addition of a network element gives them freedom and resiliency at the cost of additional complexity and a bit of additional latency/reduced bandwidth. However, it is the edges that would pay the cost to get the benefit. I have trouble seeing how this economic equation is wrong.
There is not economic incentive for the edge to deploy LISP. We are facing the same problem that we have with IPv6.
Not really. For example, you (or somebody) have to edit/recompile code to use IPv6. You do not have to recompile code to use LISP. Regards, -drc
On 3/20/13 12:26 PM, David Conrad wrote:
Arturo,
On Mar 20, 2013, at 5:32 AM, Arturo Servin <arturo.servin@gmail.com> wrote:
For example I know there are enterprises that would like to multihome but they find the current mechanism a barrier to this - for a start they can't justify the size of PI space that would guarantee them entry to the global routing table.
Which is good. If they cannot justify PI space may be they should not get into the global routing table.
The implication of this statement is that if you cannot afford the RIR fees, the routers, the technical expertise to run those routers, the additional opex associated with "BGP-capable" Internet connectivity, etc., the services/content you provide don't deserve resiliency/redundancy/etc.
You deserve it, but can you afford it? (at least with the technology that we have today).
I have trouble seeing how this can be called "good". A "necessary evil given broken technology" perhaps, but not "good".
May be not my best choice of words. What I meant was that if you cannot justify PI, probably you do not have the means to run multihome today. It is not really good, in fact it sucks but this is the reality.
LISP is about seperating the role of the ISP (as routing provider) from the end user or content provider/consumer.
Yes, but as mentioned before the cost is in the edge, the benefit in the core.
Being able to effectively multi-home without BGP, removing the need to ever renumber, etc., all sound like benefits to the edge to me.
The economic equation is all wrong.
Is LISP able to provide all those benefits?
People keep saying this.
For core providers, the equation doesn't change. Well, OK, they may lose the additional fees they get for "BGP-capable" connections and they won't have the 'benefit' of the cost of renumbering to reduce customer thrash, however they continue to get paid for providing connectivity services. They might even save some money in the long run as they won't need to replace their hamsters with guinea pigs quite as frequently.
For edges, the addition of a network element gives them freedom and resiliency at the cost of additional complexity and a bit of additional latency/reduced bandwidth. However, it is the edges that would pay the cost to get the benefit. I have trouble seeing how this economic equation is wrong.
There is not economic incentive for the edge to deploy LISP. We are facing the same problem that we have with IPv6.
Not really.
Not in the details, but in the macro it is. A technology that has to be lead by somebody that may not have the incentive to do it.
For example, you (or somebody) have to edit/recompile code to use IPv6. You do not have to recompile code to use LISP.
But as edge site I have to have a capable router, have engineers to deploy LISP (or pay for it), etc. Without a clear benefit I do not seeing any one doing it. But I've already said it in my previous emal: "Now, if with LISP as an edge site I can have multihome, high availability, not to renumber my network, or any other combination of benefits and it does cost less than PI+BGP or PA+<adyourflavorofNAThere> then it may work."
Regards, -drc
Regards, as
Sent from my iPad On Mar 20, 2013, at 10:26 AM, David Conrad <drc@virtualized.org> wrote:
Arturo,
On Mar 20, 2013, at 5:32 AM, Arturo Servin <arturo.servin@gmail.com> wrote:
For example I know there are enterprises that would like to multihome but they find the current mechanism a barrier to this - for a start they can't justify the size of PI space that would guarantee them entry to the global routing table.
Which is good. If they cannot justify PI space may be they should not get into the global routing table.
Any organization can easily justify a /48 if they can show two LOIs or contracts for peering or transit from independent ASNs.
The implication of this statement is that if you cannot afford the RIR fees, the routers, the technical expertise to run those routers, the additional opex associated with "BGP-capable" Internet connectivity, etc., the services/content you provide don't deserve resiliency/redundancy/etc.
I have trouble seeing how this can be called "good". A "necessary evil given broken technology" perhaps, but not "good".
+1
LISP is about seperating the role of the ISP (as routing provider) from the end user or content provider/consumer.
Yes, but as mentioned before the cost is in the edge, the benefit in the core.
Being able to effectively multi-home without BGP, removing the need to ever renumber, etc., all sound like benefits to the edge to me.
What part of "without BGP" benefits the edge? Multihoming with BGP is much simpler at the edge than implementing LISP. Owen
Composed on a virtual keyboard, please forgive typos. On Mar 20, 2013, at 8:07, Aled Morris <aledm@qix.co.uk> wrote:
On 20 March 2013 11:44, Arturo Servin <arturo.servin@gmail.com> wrote:
The last presentations that I saw about it said that we are going to be fine:
http://www.iepg.org/2011-11-ietf82/2011-11-13-bgp2011.pdf http://www.iepg.org/2011-11-ietf82/iepg-201111.pdf
It isn't just about "imminient death of the net predicted" though - our reliance on the current BGP model for route adverisement is restricting the deployment of better connectivity paradigms.
For example I know there are enterprises that would like to multihome but they find the current mechanism a barrier to this - for a start they can't justify the size of PI space that would guarantee them entry to the global routing table.
Then they are not well educated. Which is not surprising, and unlikely to change of we change the underlying protocol. The barrier to entry for multihoming has never been lower. Moreover, not sure it is the protocol's job to lower it further.
ISPs differentiate between "regular" and "BGP-capable" connections - is this desirable for the evolution of the Internet? or is it the reason that BGP appears to be able to cope, because ISPs are throttling the potential growth?
I don't know a single ISP that wants to throttle growth by not accepting additional customers, BGP speaking or not. (I do know several that want to throttle growth through not upgrading their links because they have a captive audience they are trying to ransom. But that is neither relevant to this discussion, not controversial - unless you are paid by one of those ISPs....) And please don't reply with "then why can't I run BGP on my [cable|DSL|etc.] link?" Broadband providers are not trying to throttle growth by not allowing grandma to do BGP, and swapping to LISP or anything else won't change that.
LISP is about seperating the role of the ISP (as routing provider) from the end user or content provider/consumer.
I am unconvinced that is a good idea. At least using the definition of "end use" or "consumer" I frequently hear. -- TTFN, patrick
I don't know a single ISP that wants to throttle growth by not accepting additional customers, BGP speaking or not. (I do know several that want to throttle growth through not upgrading their links because they have a captive audience they are trying to ransom. But that is neither relevant to this discussion, not controversial - unless you are paid by one of those ISPs….)
Comcast Verizon AT&T Time Warner Cable Cox CenturyLink to name a few. Not one of them will run BGP with a residential subscriber.
And please don't reply with "then why can't I run BGP on my [cable|DSL|etc.] link?" Broadband providers are not trying to throttle growth by not allowing grandma to do BGP, and swapping to LISP or anything else won't change that.
Sure they are. If they weren't, it would be relatively straight forward to add the necessary options to DHCP for a minimal (accept default, advertise local) BGP configuration and it would be quite simple for CPE router manufacturers to incorporate those capabilities. The problem is BGP doesn't scale to that level and everyone knows it, so, we limit growth by not allowing it to be a possibility. You are right, however, LISP won't change that.
LISP is about seperating the role of the ISP (as routing provider) from the end user or content provider/consumer.
I am unconvinced that is a good idea. At least using the definition of "end use" or "consumer" I frequently hear.
+1 However, a locator/id separation without map/encap is a desirable thing that could allow the routing system to scale better. Unfortunately, we failed to address this issue when designing IPv6. It will not get correctly solved without a revision to the header design. There is no will to change the packet header in the near future. We're having too much "fun" rolling out the current one. Owen
On 3/20/2013 9:25 AM, Owen DeLong wrote:
I don't know a single ISP that wants to throttle growth by not accepting additional customers, BGP speaking or not. (I do know several that want to throttle growth through not upgrading their links because they have a captive audience they are trying to ransom. But that is neither relevant to this discussion, not controversial - unless you are paid by one of those ISPs….) Comcast Verizon AT&T Time Warner Cable Cox CenturyLink
to name a few.
Not one of them will run BGP with a residential subscriber.
And please don't reply with "then why can't I run BGP on my [cable|DSL|etc.] link?" Broadband providers are not trying to throttle growth by not allowing grandma to do BGP, and swapping to LISP or anything else won't change that. Sure they are. If they weren't, it would be relatively straight forward to add the necessary options to DHCP for a minimal (accept default, advertise local) BGP configuration and it would be quite simple for CPE router manufacturers to incorporate those capabilities.
The problem is BGP doesn't scale to that level and everyone knows it, so, we limit growth by not allowing it to be a possibility.
Is someone working on 8-byte ASNs yet?
On 3/20/13 6:25 AM, Owen DeLong wrote:
I don't know a single ISP that wants to throttle growth by not accepting additional customers, BGP speaking or not. (I do know several that want to throttle growth through not upgrading their links because they have a captive audience they are trying to ransom. But that is neither relevant to this discussion, not controversial - unless you are paid by one of those ISPs….)
Comcast Verizon AT&T Time Warner Cable Cox CenturyLink
to name a few.
Not one of them will run BGP with a residential subscriber.
Based on the average clue of your average residential subscriber (anyone here need not apply) I'd say that's a good thing. ~Seth
On 2013-03-20, at 10:55, Seth Mattinen <sethm@rollernet.us> wrote:
On 3/20/13 6:25 AM, Owen DeLong wrote:
I don't know a single ISP that wants to throttle growth by not accepting additional customers, BGP speaking or not. (I do know several that want to throttle growth through not upgrading their links because they have a captive audience they are trying to ransom. But that is neither relevant to this discussion, not controversial - unless you are paid by one of those ISPs….)
Comcast Verizon AT&T Time Warner Cable Cox CenturyLink
to name a few.
Not one of them will run BGP with a residential subscriber.
Based on the average clue of your average residential subscriber (anyone here need not apply) I'd say that's a good thing.
In practice, it seems to me that the way people multi-home these days for client-filled networks is: 1. Number everything internally using private-use addresses 2. Use one NAT per upstream 3. Send your outbound flows through whichever NAT seems appropriate There seem to be no shortage of SMB appliances that will take care of this for you without you having to understand anything. The phrase that seems to be used when describing these routers is "dual WAN". https://www.mushroomnetworks.com http://www.peplink.com/balance/ http://www.tp-link.com/ http://www.draytek.com/ It's trivial to configure this kind of thing on more general-purpose gear too, obviously, but that requires Actual Knowledge of How Things Work whereas these products aim to get things running without any of that. This style of multi-homing is invisible from the perspective of the routing system. Obviously this doesn't work nicely for inbound connections, but the fact that people do it anyway suggests that isn't a deal-killer (presumably every server that needs to accept an inbound connection these days lives elsewhere, in someone's cloud). I'm not suggesting this is good architecture, but it happens. Even if BGP on res-grade internet access products was trivially available, I can see this kind of NAT hack being more popular. I think it's incorrect to insist that the Network doesn't support pervasive end-site multi-homing when it's clear that people are doing it anyway. Joe
On Mar 20, 2013, at 11:39 AM, Joe Abley <jabley@hopcount.ca> wrote:
I think it's incorrect to insist that the Network doesn't support pervasive end-site multi-homing when it's clear that people are doing it anyway.
I know some small WISPs that balance multiple business DSL links using some of these things. Their boxes are interesting when it comes to the amount of state they hold and how they route those load balanced links. They auto-failover when a link stops working which is nice. If you have one of the static IPs you are obviously bound to that line. I'm surprised nobody here is posting how they use their 3G/LTE USB modem in a SRX or Cisco router and run BGP over that. (I'm sure someone on-list has done it). - Jared
Joe Abley wrote:
In practice, it seems to me that the way people multi-home these days for client-filled networks is:
1. Number everything internally using private-use addresses 2. Use one NAT per upstream 3. Send your outbound flows through whichever NAT seems appropriate
Very reasonable approach. But, how to choose the proper NAT is still a problem. If there were standard home/site IGP supported by NAT boxes, to which ISPs tell ASPATHLEN as metric, it could help a lot. However, because packets are hot potatoes, ISPs are motivated to tell something larger than ASPATHLEN as the metric, unless they are legally ("MUST" phrase in IETF standards could be) forced not to do so.
There seem to be no shortage of SMB appliances that will take care of this for you without you having to understand anything. The phrase that seems to be used when describing these routers is "dual WAN".
https://www.mushroomnetworks.com http://www.peplink.com/balance/ http://www.tp-link.com/ http://www.draytek.com/
It's trivial to configure this kind of thing on more general-purpose gear too, obviously, but that requires Actual Knowledge of How Things Work whereas these products aim to get things running without any of that.
This style of multi-homing is invisible from the perspective of the routing system.
That is the correct thing to do.
Obviously this doesn't work nicely for inbound connections, but the fact that people do it anyway suggests that isn't a deal-killer
That is a problem MUST be solved by transport (or, at least application) layer of end systems to be able to handle multiple public addresses. It's not very difficult to modify TCP to accept packets with multiple source addresses and send to multiple destination addresses, if the addresses are carried with TCP optional header of SYN and SYNACK packets. Moreover, you can run SMTP and DNS servers with multiple public addresses, because clients are, though at the application layer, implemented to try all the public addresses of servers, even though they are tried only after transport layer time outs.
(presumably every server that needs to accept an inbound connection these days lives elsewhere, in someone's cloud).
It partially because of the current difficulty to have multihoming. But, unless having an entry in DFZ is somehow appropriately taxed, the number of data centers to offer such cloud will increase. The fundamental economical problem of routing table bloat today is that one can multihome with its own DFZ routing table entry without paying anything to all the ISPs and other entities who must pay for enough resources to support the bloating DFZ routing table. Masataka Ohta
I'm not suggesting this is good architecture,
Yes, it is.
I think it's incorrect to insist that the Network doesn't support pervasive end-site multi-homing when it's clear that people are doing it anyway.
Fully agreed. Masataka Ohta
Sent from my iPad On Mar 20, 2013, at 9:55 AM, Seth Mattinen <sethm@rollernet.us> wrote:
On 3/20/13 6:25 AM, Owen DeLong wrote:
I don't know a single ISP that wants to throttle growth by not accepting additional customers, BGP speaking or not. (I do know several that want to throttle growth through not upgrading their links because they have a captive audience they are trying to ransom. But that is neither relevant to this discussion, not controversial - unless you are paid by one of those ISPs….)
Comcast Verizon AT&T Time Warner Cable Cox CenturyLink
to name a few.
Not one of them will run BGP with a residential subscriber.
Based on the average clue of your average residential subscriber (anyone here need not apply) I'd say that's a good thing.
~Seth
Why? If BGP were plug-and-play automated with settings specified by the provider, what would the user's clue level have to do with it? Owen
On Wed, 20 Mar 2013 15:16:57 -0500, Owen DeLong said:
On Mar 20, 2013, at 9:55 AM, Seth Mattinen <sethm@rollernet.us> wrote:
Based on the average clue of your average residential subscriber (anyone here need not apply) I'd say that's a good thing.
If BGP were plug-and-play automated with settings specified by the provider, what would the user's clue level have to do with it?
The hypothetical existence of such a box doesn't change the fact that providers have to make business decisions based on actual boxes and users. Yes, if a plug-n-play idiot-proof BGP box existed, then the profit calculus would be different. On the other hand, if there existed a reliable cost-effective means for faster-than-light signaling, if would drastically change intercontinental peering patterns. All the same, anybody who's planning their interconnects in 2013 reality and not looking at who has 40K km of underwater cable and who doesn't is in for a surprise....
On Mar 22, 2013, at 15:44 , <Valdis.Kletnieks@vt.edu> wrote:
On Wed, 20 Mar 2013 15:16:57 -0500, Owen DeLong said:
On Mar 20, 2013, at 9:55 AM, Seth Mattinen <sethm@rollernet.us> wrote:
Based on the average clue of your average residential subscriber (anyone here need not apply) I'd say that's a good thing.
If BGP were plug-and-play automated with settings specified by the provider, what would the user's clue level have to do with it?
The hypothetical existence of such a box doesn't change the fact that providers have to make business decisions based on actual boxes and users.
Yes, if a plug-n-play idiot-proof BGP box existed, then the profit calculus would be different. On the other hand, if there existed a reliable cost-effective means for faster-than-light signaling, if would drastically change intercontinental peering patterns. All the same, anybody who's planning their interconnects in 2013 reality and not looking at who has 40K km of underwater cable and who doesn't is in for a surprise....
There is a difference and you know it. A reliable cost-effective means for FTL signaling is a hard problem without a known solution. An idiot-proof simple BGP configuration is a well known solution. Automating it would be relatively simple if there were the will to do so. However, the reality is that ISPs don't want the solution for a number of reasons: 1. ISPs are actually motivated to prevent customer mobility, not enable it. 2. ISPs are motivated to reduce, not increase the number of multi-homed sites occupying slots in routing tables. In addition, most of the consumers that could benefit from such a solution do not have enough knowledge to know what they should be demanding from their vendors, so they don't demand it. This is a classic case of the "invisible hand" only works when all participants have equal access to information and relatively equal knowledge. The problem with technical products and especially with technical based services products is that the vendor consistently has much greater knowledge than the subscriber and is therefore in a position to optimize for the vendors best interests even when they are counter to the best interests of their customers. Owen
On 3/23/13, Owen DeLong <owen@delong.com> wrote:
A reliable cost-effective means for FTL signaling is a hard problem without a known solution.
Faster than light signalling is not merely a hard problem. Special relativity doesn't provide that information may travel faster than the maximum speed C. If you want to signal faster than light, then slow down the light.
An idiot-proof simple BGP configuration is a well known solution. Automating it would be relatively simple if there were the will to do so.
Logistical problems... if it's a multihomed connection, which of the two or three providers manages it, and gets to blame the other provider(s) when anything goes wrong: or are you gonna rely on the customer to manage it? Someone might be able to make a protocol that lets this happen, which would need to detect on a per-route basis any performance/connectivity issues, but I would say it's not any known implementation of BGP.
1. ISPs are actually motivated to prevent customer mobility, not enable it.
2. ISPs are motivated to reduce, not increase the number of multi-homed sites occupying slots in routing tables.
This is not some insignificant thing. The ISPs have to maintain routing tables as well; ultimately the ISP's customers are in bad shape, if too many slots are consumed. How about 3. Increased troubleshooting complexity when there are potential issues or complaints. The concept of a "fool proof" BGP configuration is clearly a new sort of myth. The idea that the protocol on its own, with a very basic config, does not ever require any additional attention, to achieve expected results; where expected results include isolation from any faults with the path from one of of the user's two, three, or four providers, and balancing for optimal throughput and best latency/loss to every destination. BGP multihoming doesn't prevent users from having issues because: o Connectivity issues that are a responsibility of one of their provider's That they might have expected multihoming to protect them against (latency, packet loss). o very Poor performance of one of their links; or poor performance of one of their links to their favorite destination o Asymmetric paths; which means that when latency or loss is poor, the customer doesn't necessarily know which provider to blame, or if both are at fault, and the providers can spend a lot of time blaming each other. These are all solvable problems, but at cost, and therefore not for massmarket lowest cost ISP service. It's not as if they can have "Hello, DSL technical support... did you try shutting off your other peers and retesting'?" The average end user won't have a clue -- they will need one of the providers, or someone else to be managing that for them, and understand how each provider is connected. I don't see large ISPs training up their support reps for DSL $60/month services, to handle BGP troubleshooting, and multihoming management/repair.
In addition, most of the consumers that could benefit from such a solution do not have enough knowledge to know what they should be demanding from their vendors, so they don't demand it.
Owen -- -JH
On Mar 23, 2013, at 12:12 , Jimmy Hess <mysidia@gmail.com> wrote:
On 3/23/13, Owen DeLong <owen@delong.com> wrote:
A reliable cost-effective means for FTL signaling is a hard problem without a known solution.
Faster than light signalling is not merely a hard problem. Special relativity doesn't provide that information may travel faster than the maximum speed C. If you want to signal faster than light, then slow down the light.
An idiot-proof simple BGP configuration is a well known solution. Automating it would be relatively simple if there were the will to do so.
Logistical problems... if it's a multihomed connection, which of the two or three providers manages it, and gets to blame the other provider(s) when anything goes wrong: or are you gonna rely on the customer to manage it?
The box could (pretty easily) be built with a "Primary" and "Secondary" port. The cable plugged into the primary port would go to the ISP that sets the configuration. The cable plugged into the other port would go to an ISP expected to accept the announcements of the prefix provided by the ISP on the primary port. BFD could be used to illuminate a tri-color LED on the box for each port, which would be green if BFD state is good and red if BFD state is bad. At that point, whichever one is red gets the blame. If they're both green, then traffic is going via the primary and the primary gets the blame. If you absolutely have to troubleshoot which provider is broken, then start by unplugging the secondary. If it doesn't start working in 5 minutes, then clearly there's a problem with the primary regardless of what else is happening. Lather, rinse, repeat for the secondary.
Someone might be able to make a protocol that lets this happen, which would need to detect on a per-route basis any performance/connectivity issues, but I would say it's not any known implementation of BGP.
A few additional options to DHCP could actually cover it from the primary perspective. For the secondary provider, it's a little more complicated, but could be mostly automated so long as the customer identifies the primary provider and/or provides an LOA for the authorized prefix from the primary to the secondary. The only complexity in the secondary case is properly filtering the announcement of the prefix assigned by the primary.
1. ISPs are actually motivated to prevent customer mobility, not enable it.
2. ISPs are motivated to reduce, not increase the number of multi-homed sites occupying slots in routing tables.
This is not some insignificant thing. The ISPs have to maintain routing tables as well; ultimately the ISP's customers are in bad shape, if too many slots are consumed.
I never said it was insignificant. I said that solving the multihoming problem in this manner was trivial if there was will to do so. I also said that the above were contributing factors in the lack of will to do so.
How about 3. Increased troubleshooting complexity when there are potential issues or complaints.
I do not buy that it is harder to troubleshoot a basic BGP configuration than a multi-carrier NAT-based solution that goes woefully awry. I'm sorry, I've done the troubleshooting on both scenarios and I have to say that if you think NAT makes this easier, you live in a different world than I do.
The concept of a "fool proof" BGP configuration is clearly a new sort of myth.
Not really. Customer router accepts default from primary and secondary providers. So long as default remains, primary is preferred. If primary default goes away, secondary is preferred. Customer box gets prefix (via DHCP-PD or static config or whatever either from primary or from RIR). Advertises prefix to both primary and secondary. All configuration of the BGP sessions is automated within the box other than static configuration of customer prefix (if static is desired). Primary/Secondary choice is made by plugging providers into the Primary or Secondary port on the box.
The idea that the protocol on its own, with a very basic config, does not ever require any additional attention, to achieve expected results; where expected results include isolation from any faults with the path from one of of the user's two, three, or four providers, and balancing for optimal throughput and best latency/loss to every destination.
I have installed these configurations at customer sites for several of my consulting clients that wanted to multihome their SMBs. Some of them have been running for more than 8 years without a single issue. For all of the above requirements, no. You can't do that with the most advanced manual BGP configurations today. However, if we reduce it to: 1. The internet connection stays up so long as one of the two providers is up. 2. Traffic prefers the primary provider so long as the primary provider is up. 3. My addressing remains stable so long as I remain connected to the primary provider (or if I use RIR based addressing, longer). Then what I have proposed actually is achievable, does work, and does actually meet the needs of 99+% of organizations that wish to multihome.
BGP multihoming doesn't prevent users from having issues because:
o Connectivity issues that are a responsibility of one of their provider's That they might have expected multihoming to protect them against (latency, packet loss).
Correct. However, this is true of ANY multihoming solution. The dual- provider NAT solution certainly does NOT improve this.
o very Poor performance of one of their links; or poor performance of one of their links to their favorite destination
See above.
o Asymmetric paths; which means that when latency or loss is poor, the customer doesn't necessarily know which provider to blame, or if both are at fault, and the providers can spend a lot of time blaming each other.
See above.
These are all solvable problems, but at cost, and therefore not for massmarket lowest cost ISP service.
My point is that the automated simple BGP solution I propose can provide a better customer experience than the currently popular NAT-based multihoming with simpler troubleshooting and lower costs.
It's not as if they can have "Hello, DSL technical support... did you try shutting off your other peers and retesting'?"
ROFL.
The average end user won't have a clue -- they will need one of the providers, or someone else to be managing that for them, and understand how each provider is connected.
Again, you're setting a much higher goal than I was. My goal was to do something better than what is currently being done. (Connect a router to two providers and use NAT to choose between them).
I don't see large ISPs training up their support reps for DSL $60/month services, to handle BGP troubleshooting, and multihoming management/repair.
But they already get stuck with this in the current NAT-based solution which is even harder to troubleshoot and creates even more problems. Owen
You do realize that there are quite a few people (home broadband subscribers?) who just "go do something else" when their internet goes down, right? There are people who don't understand the difference between "a site being slow" and packet-loss. For many of these people, losing internet service carries zero business impact, and relatively little life impact; they might even realize they have better things to do than watch cat videos or scroll through endless social media feeds. Will they really demand ubiquitous, unabridged connectivity? When? On Mar 23, 2013 12:58 PM, "Owen DeLong" <owen@delong.com> wrote:
On Mar 23, 2013, at 12:12 , Jimmy Hess <mysidia@gmail.com> wrote:
On 3/23/13, Owen DeLong <owen@delong.com> wrote:
A reliable cost-effective means for FTL signaling is a hard problem
a known solution.
Faster than light signalling is not merely a hard problem. Special relativity doesn't provide that information may travel faster than the maximum speed C. If you want to signal faster than light, then slow down the
An idiot-proof simple BGP configuration is a well known solution.
Automating
it would be relatively simple if there were the will to do so.
Logistical problems... if it's a multihomed connection, which of the two or three providers manages it, and gets to blame the other provider(s) when anything goes wrong: or are you gonna rely on the customer to manage it?
The box could (pretty easily) be built with a "Primary" and "Secondary"
without light. port.
The cable plugged into the primary port would go to the ISP that sets the configuration. The cable plugged into the other port would go to an ISP expected to accept the announcements of the prefix provided by the ISP on the primary port.
BFD could be used to illuminate a tri-color LED on the box for each port, which would be green if BFD state is good and red if BFD state is bad.
At that point, whichever one is red gets the blame. If they're both green, then traffic is going via the primary and the primary gets the blame.
If you absolutely have to troubleshoot which provider is broken, then start by unplugging the secondary. If it doesn't start working in 5
then clearly there's a problem with the primary regardless of what else is happening.
Lather, rinse, repeat for the secondary.
Someone might be able to make a protocol that lets this happen, which would need to detect on a per-route basis any performance/connectivity issues, but I would say it's not any known implementation of BGP.
A few additional options to DHCP could actually cover it from the primary perspective.
For the secondary provider, it's a little more complicated, but could be mostly automated so long as the customer identifies the primary provider and/or provides an LOA for the authorized prefix from the primary to the secondary.
The only complexity in the secondary case is properly filtering the announcement of the prefix assigned by the primary.
1. ISPs are actually motivated to prevent customer mobility, not enable it.
2. ISPs are motivated to reduce, not increase the number of multi-homed sites occupying slots in routing tables.
This is not some insignificant thing. The ISPs have to maintain routing tables as well; ultimately the ISP's customers are in bad shape, if too many slots are consumed.
I never said it was insignificant. I said that solving the multihoming
in this manner was trivial if there was will to do so. I also said that
were contributing factors in the lack of will to do so.
How about 3. Increased troubleshooting complexity when there are potential issues or complaints.
I do not buy that it is harder to troubleshoot a basic BGP configuration than a multi-carrier NAT-based solution that goes woefully awry.
I'm sorry, I've done the troubleshooting on both scenarios and I have to say that if you think NAT makes this easier, you live in a different world than I do.
The concept of a "fool proof" BGP configuration is clearly a new sort of myth.
Not really.
Customer router accepts default from primary and secondary providers. So long as default remains, primary is preferred. If primary default goes away, secondary is preferred.
Customer box gets prefix (via DHCP-PD or static config or whatever either from primary or from RIR). Advertises prefix to both primary and secondary.
All configuration of the BGP sessions is automated within the box other than static configuration of customer prefix (if static is desired).
Primary/Secondary choice is made by plugging providers into the Primary or Secondary port on the box.
The idea that the protocol on its own, with a very basic config, does not ever require any additional attention, to achieve expected results; where expected results include isolation from any faults with the path from one of of the user's two, three, or four providers, and balancing for optimal throughput and best latency/loss to every destination.
I have installed these configurations at customer sites for several of my consulting clients that wanted to multihome their SMBs.
Some of them have been running for more than 8 years without a single issue.
For all of the above requirements, no. You can't do that with the most advanced manual BGP configurations today.
However, if we reduce it to:
1. The internet connection stays up so long as one of the two providers is up.
2. Traffic prefers the primary provider so long as the primary
minutes, problem the above provider
is up.
3. My addressing remains stable so long as I remain connected to the primary provider (or if I use RIR based addressing, longer).
Then what I have proposed actually is achievable, does work, and does actually meet the needs of 99+% of organizations that wish to multihome.
BGP multihoming doesn't prevent users from having issues because:
o Connectivity issues that are a responsibility of one of their
provider's
That they might have expected multihoming to protect them
against
(latency, packet loss).
Correct. However, this is true of ANY multihoming solution. The dual- provider NAT solution certainly does NOT improve this.
o very Poor performance of one of their links; or poor performance of one of their links to their favorite destination
See above.
o Asymmetric paths; which means that when latency or loss is poor, the customer doesn't necessarily know which provider to blame, or if both are at fault, and the providers can spend a lot of
time
blaming each other.
See above.
These are all solvable problems, but at cost, and therefore not for massmarket lowest cost ISP service.
My point is that the automated simple BGP solution I propose can provide a better customer experience than the currently popular NAT-based multihoming with simpler troubleshooting and lower costs.
It's not as if they can have "Hello, DSL technical support... did you try shutting off your other peers and retesting'?"
ROFL.
The average end user won't have a clue -- they will need one of the providers, or someone else to be managing that for them, and understand how each provider is connected.
Again, you're setting a much higher goal than I was.
My goal was to do something better than what is currently being done. (Connect a router to two providers and use NAT to choose between them).
I don't see large ISPs training up their support reps for DSL $60/month services, to handle BGP troubleshooting, and multihoming management/repair.
But they already get stuck with this in the current NAT-based solution which is even harder to troubleshoot and creates even more problems.
Owen
On Sat, Mar 23, 2013 at 07:47:12PM -0700, Kyle Creyts wrote:
You do realize that there are quite a few people (home broadband subscribers?) who just "go do something else" when their internet goes down, right?
[...]
Will they really demand ubiquitous, unabridged connectivity?
When?
Probably around the time their phone, TV, books, shopping, and *life* are all delivered over that connectivity. Especially if they don't have any meaningful local storage or processing, as everything has been delegated to "the cloud". In practice, however, I suspect that we as providers will just get a whole lot better at providing connectivity, rather than have everyone work out how to do fully-diverse BGP from their homes. - Matt
On Sat, Mar 23, 2013 at 07:47:12PM -0700, Kyle Creyts wrote:
You do realize that there are quite a few people (home broadband subscribers?) who just "go do something else" when their internet goes down, right? [...]
Will they really demand ubiquitous, unabridged connectivity?
When? Probably around the time their phone, TV, books, shopping, and *life* are all delivered over that connectivity. Especially if they don't have any meaningful local storage or processing, as everything has been delegated to "the cloud". When the cable is down there's the verizon usb stick (which this point can be into the router and serve the whole house), when verizon is down
In practice, however, I suspect that we as providers will just get a whole lot better at providing connectivity, rather than have everyone work out how to do fully-diverse BGP from their homes. I'm going to be somewhat contrarian, connectivity/availability with cloud services is important, where you access them from not so much. I doubt very much that reliance on the cloud drives multihoming for end-sites/consumers, it drives a demand for connectivity diversity so
On 3/23/13 9:13 PM, Matt Palmer wrote: there's t-mobile handset. when t-mobile is down there's a workphone with at&t. When the cable/verizon/t-mobile/at&t are all down for any signifcant length of time, I expect to be digging my neighbors our of the sorts of natural disasters that befall California and listening to the radio and maybe 2-meter. that one failure mode doesn't leave you stranded.
- Matt
On Sat, Mar 23, 2013 at 10:47 PM, Kyle Creyts <kyle.creyts@gmail.com> wrote:
Will they really demand ubiquitous, unabridged connectivity?
When?
When the older generation that considers the Internet a side show dies off. When your grandparents' power went out, they broke out candles and kerosene lamps. When yours goes out, you pull out flashlights and generators. And when it stays out you book a motel room so your family can have air conditioning and television. For most folks under 30 and many who are older, Internet isn't a side show, it's a way of life. An outage is like a power failure or the car going kaput: a major disruption to life's flow. This need won't be ubiquitous for two to three decades, but every year between now and then the percentage of your customer base which demands unabridged connectivity will grow. What do you have in the pipeline to address that demand as it arrives? BGP multihoming won't get the job done for the hundred million households in North America, let alone the seven billion people in the world. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Mar 24, 2013, at 12:06 PM, William Herrin <bill@herrin.us> wrote:
... For most folks under 30 and many who are older, Internet isn't a side show, it's a way of life. An outage is like a power failure or the car going kaput: a major disruption to life's flow.
Yes, this is increasingly the case (and may not be as generational as you think)
This need won't be ubiquitous for two to three decades, but every year between now and then the percentage of your customer base which demands unabridged connectivity will grow.
I believe that the percentage which _expects_ unabridged connectivity today is quite high, but that does not necessarily mean actual _demand_ (i.e. folks who go out and make the necessary arrangements despite the added cost and hassle...) The power analogy might be apt here; I know many folks who have a home UPS, a few that have a manual generator, and just one or two who did the entire home automatic UPS/generator combo that's really necessary for 100% reliable power. This reflects a truism: while many people may expect 100% reliable today, the demand (in most areas) simply doesn't match.
What do you have in the pipeline to address that demand as it arrives?
See above: increasing expectations does not necessarily equate with demand. FYI, /John Disclaimer: My views alone. Sent via less than 100% reliable networks.
In a message written on Sun, Mar 24, 2013 at 12:54:18PM -0400, John Curran wrote:
I believe that the percentage which _expects_ unabridged connectivity today is quite high, but that does not necessarily mean actual _demand_ (i.e. folks who go out and make the necessary arrangements despite the added cost and hassle...)
Actually, I think most of the people who care have made alternate arrangements, but I have to back up first. Like most of the US, I live in a area with a pair of providers, BigCable and BigTelco. The reality of those two providers is that from my house they both run down the same backyard. The both go to pedistals next to each other at the edge of the neighborhood. They both ride the same poles down towards the center of town. At some point they finally diverge, to two different central offices. About 80% of the time when one goes out the other does as well. The backhoe digs up both wires. The pole taken out by a car accident takes them both down. Heck, when the power goes out to a storm neither has a generator for their pedistal. The other 20% of the time one has an equipment failure and the other does not. Even if I wanted to pay 2x the monthly cost to have both providers active (and could multi-home, etc), it really doesn't create a significanlty higher uptime, and thus is economically foolish. However, there is an alternative that shares none of this infrastructure. A cell card. Another option finally available due to higher speeds and better pricing is a satellite service. These provide true redundancy from all the physical infrastructure I described above. It could be aruged then, the interesting multi-homing case is between my Cable Modem and my Cell Card, however even that is not the case. Turns out my cell hard has bad latency compared to the cable modem, so I don't want to use it unless I have to, and it also turns out the cell provider charges me for usage, at a modestly high rate, so I don't want to use it unless I have to. The result is an active/passive backup configuration. A device like a cradlepoint can detect the cable modem being down and switch over to the cell card. Sure, incoming connections are not persisitent, but outbound it's hard to notice other than performance getting worse. TL;DR People paying for redundancy want physical redundancy including the last mile. In the US, that exists approximately nowhere for residential users. With no diverse paths to purchase, the discussion of higher level protocol issues is academic. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
As an under-30, working in the industry, I have to say, when the power goes out at home for a few days, we pull out the camping gear. When our cable-based internet goes out, our life changes hardly at all. We go for a walk, or hike, do the things we would normally. I can imagine that an outage of 1 week would be slightly different, but I'm pretty sure that the spans of most of the outages which would be resolved by multi-provider solutions like those outlined herein would probably only apply to situations where the outage would only last less than 48 hours. On Sun, Mar 24, 2013 at 9:06 AM, William Herrin <bill@herrin.us> wrote:
On Sat, Mar 23, 2013 at 10:47 PM, Kyle Creyts <kyle.creyts@gmail.com> wrote:
Will they really demand ubiquitous, unabridged connectivity?
When?
When the older generation that considers the Internet a side show dies off.
When your grandparents' power went out, they broke out candles and kerosene lamps.
When yours goes out, you pull out flashlights and generators. And when it stays out you book a motel room so your family can have air conditioning and television.
For most folks under 30 and many who are older, Internet isn't a side show, it's a way of life. An outage is like a power failure or the car going kaput: a major disruption to life's flow.
This need won't be ubiquitous for two to three decades, but every year between now and then the percentage of your customer base which demands unabridged connectivity will grow.
What do you have in the pipeline to address that demand as it arrives? BGP multihoming won't get the job done for the hundred million households in North America, let alone the seven billion people in the world.
Regards, Bill Herrin
-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
-- Kyle Creyts Information Assurance Professional BSidesDetroit Organizer
On Mar 23, 2013, at 7:47 PM, Kyle Creyts <kyle.creyts@gmail.com> wrote:
Will they really demand ubiquitous, unabridged connectivity?
Let's back up. End users do not as a rule* have persistent inbound connections. If they have DSL and a Cable Modem they can switch manually (or with a little effort automatically) if one goes down. * Servers-at-home-or-small-office is the use case for Owen's magic BGP box. Which is true for many of us and other core geeks but not an appreciable percent of the populace. I believe that full BGP to end user is less practical for this use case than a geographically dispersed BGP external facing intermediary whose connectivity to the "end user servers" is full-mesh multi-provider-multi-physical-link VPNs. It's a lot easier to manage and has less chance of a config goof blowing up bigger network neighbors. Every time I look at productizing this, though, the market's too small to support it. Which probably means it's way too small for home BGP... George William Herbert Sent from my iPhone
I assume those people will not bother with any attempt to multihome in any form. They are not, therefore, part of what is being discussed here. Owen On Mar 23, 2013, at 19:47 , Kyle Creyts <kyle.creyts@gmail.com> wrote:
You do realize that there are quite a few people (home broadband subscribers?) who just "go do something else" when their internet goes down, right?
There are people who don't understand the difference between "a site being slow" and packet-loss. For many of these people, losing internet service carries zero business impact, and relatively little life impact; they might even realize they have better things to do than watch cat videos or scroll through endless social media feeds.
Will they really demand ubiquitous, unabridged connectivity?
When?
On Mar 23, 2013 12:58 PM, "Owen DeLong" <owen@delong.com> wrote:
On Mar 23, 2013, at 12:12 , Jimmy Hess <mysidia@gmail.com> wrote:
On 3/23/13, Owen DeLong <owen@delong.com> wrote:
A reliable cost-effective means for FTL signaling is a hard problem without a known solution.
Faster than light signalling is not merely a hard problem. Special relativity doesn't provide that information may travel faster than the maximum speed C. If you want to signal faster than light, then slow down the light.
An idiot-proof simple BGP configuration is a well known solution. Automating it would be relatively simple if there were the will to do so.
Logistical problems... if it's a multihomed connection, which of the two or three providers manages it, and gets to blame the other provider(s) when anything goes wrong: or are you gonna rely on the customer to manage it?
The box could (pretty easily) be built with a "Primary" and "Secondary" port.
The cable plugged into the primary port would go to the ISP that sets the configuration. The cable plugged into the other port would go to an ISP expected to accept the announcements of the prefix provided by the ISP on the primary port.
BFD could be used to illuminate a tri-color LED on the box for each port, which would be green if BFD state is good and red if BFD state is bad.
At that point, whichever one is red gets the blame. If they're both green, then traffic is going via the primary and the primary gets the blame.
If you absolutely have to troubleshoot which provider is broken, then start by unplugging the secondary. If it doesn't start working in 5 minutes, then clearly there's a problem with the primary regardless of what else is happening.
Lather, rinse, repeat for the secondary.
Someone might be able to make a protocol that lets this happen, which would need to detect on a per-route basis any performance/connectivity issues, but I would say it's not any known implementation of BGP.
A few additional options to DHCP could actually cover it from the primary perspective.
For the secondary provider, it's a little more complicated, but could be mostly automated so long as the customer identifies the primary provider and/or provides an LOA for the authorized prefix from the primary to the secondary.
The only complexity in the secondary case is properly filtering the announcement of the prefix assigned by the primary.
1. ISPs are actually motivated to prevent customer mobility, not enable it.
2. ISPs are motivated to reduce, not increase the number of multi-homed sites occupying slots in routing tables.
This is not some insignificant thing. The ISPs have to maintain routing tables as well; ultimately the ISP's customers are in bad shape, if too many slots are consumed.
I never said it was insignificant. I said that solving the multihoming problem in this manner was trivial if there was will to do so. I also said that the above were contributing factors in the lack of will to do so.
How about 3. Increased troubleshooting complexity when there are potential issues or complaints.
I do not buy that it is harder to troubleshoot a basic BGP configuration than a multi-carrier NAT-based solution that goes woefully awry.
I'm sorry, I've done the troubleshooting on both scenarios and I have to say that if you think NAT makes this easier, you live in a different world than I do.
The concept of a "fool proof" BGP configuration is clearly a new sort of myth.
Not really.
Customer router accepts default from primary and secondary providers. So long as default remains, primary is preferred. If primary default goes away, secondary is preferred.
Customer box gets prefix (via DHCP-PD or static config or whatever either from primary or from RIR). Advertises prefix to both primary and secondary.
All configuration of the BGP sessions is automated within the box other than static configuration of customer prefix (if static is desired).
Primary/Secondary choice is made by plugging providers into the Primary or Secondary port on the box.
The idea that the protocol on its own, with a very basic config, does not ever require any additional attention, to achieve expected results; where expected results include isolation from any faults with the path from one of of the user's two, three, or four providers, and balancing for optimal throughput and best latency/loss to every destination.
I have installed these configurations at customer sites for several of my consulting clients that wanted to multihome their SMBs.
Some of them have been running for more than 8 years without a single issue.
For all of the above requirements, no. You can't do that with the most advanced manual BGP configurations today.
However, if we reduce it to:
1. The internet connection stays up so long as one of the two providers is up.
2. Traffic prefers the primary provider so long as the primary provider is up.
3. My addressing remains stable so long as I remain connected to the primary provider (or if I use RIR based addressing, longer).
Then what I have proposed actually is achievable, does work, and does actually meet the needs of 99+% of organizations that wish to multihome.
BGP multihoming doesn't prevent users from having issues because:
o Connectivity issues that are a responsibility of one of their provider's That they might have expected multihoming to protect them against (latency, packet loss).
Correct. However, this is true of ANY multihoming solution. The dual- provider NAT solution certainly does NOT improve this.
o very Poor performance of one of their links; or poor performance of one of their links to their favorite destination
See above.
o Asymmetric paths; which means that when latency or loss is poor, the customer doesn't necessarily know which provider to blame, or if both are at fault, and the providers can spend a lot of time blaming each other.
See above.
These are all solvable problems, but at cost, and therefore not for massmarket lowest cost ISP service.
My point is that the automated simple BGP solution I propose can provide a better customer experience than the currently popular NAT-based multihoming with simpler troubleshooting and lower costs.
It's not as if they can have "Hello, DSL technical support... did you try shutting off your other peers and retesting'?"
ROFL.
The average end user won't have a clue -- they will need one of the providers, or someone else to be managing that for them, and understand how each provider is connected.
Again, you're setting a much higher goal than I was.
My goal was to do something better than what is currently being done. (Connect a router to two providers and use NAT to choose between them).
I don't see large ISPs training up their support reps for DSL $60/month services, to handle BGP troubleshooting, and multihoming management/repair.
But they already get stuck with this in the current NAT-based solution which is even harder to troubleshoot and creates even more problems.
Owen
On Sat, 23 Mar 2013 11:28:07 -0700, Owen DeLong said:
A reliable cost-effective means for FTL signaling is a hard problem without a known solution.
Agreed.
An idiot-proof simple BGP configuration is a well known solution. Automating it would be relatively simple if there were the will to do so.
As others pointed out, a reliable cost effective way of automating the layer 8 problems is *also* a hard problem without a known solution.
On Fri, Mar 22, 2013 at 6:44 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Wed, 20 Mar 2013 15:16:57 -0500, Owen DeLong said:
On Mar 20, 2013, at 9:55 AM, Seth Mattinen <sethm@rollernet.us> wrote:
Based on the average clue of your average residential subscriber (anyone here need not apply) I'd say that's a good thing.
If BGP were plug-and-play automated with settings specified by the provider, what would the user's clue level have to do with it?
The hypothetical existence of such a box doesn't change the fact that providers have to make business decisions based on actual boxes and users.
Providers who don't wish to be leap-frogged have to make business decisions about unserved and underserved demand for which they don't already have an effective product.
Yes, if a plug-n-play idiot-proof BGP box existed, then the profit calculus would be different. On the other hand, if there existed a reliable cost-effective means for faster-than-light signaling, if would drastically change intercontinental peering patterns.
That's not a particularly compelling counterpoint. We have a mechanism for multihoming: BGP. We have a mechanism for flying to the moon: rocket ships. At a strictly technical level, either could be made suitable for use by John Q. Public. In both cases the cost attributable to John Q's desired activity, when using known techniques, greatly exceeds his budget. That having been said, I'd be very interested in your take on how FTL would change intercontinental peering patterns. How would dropping all links to a 0 ms latency change the ways in which we choose to interconnect and why? Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Mar 20, 2013, at 09:25 , Owen DeLong <owen@delong.com> wrote:
I don't know a single ISP that wants to throttle growth by not accepting additional customers, BGP speaking or not. (I do know several that want to throttle growth through not upgrading their links because they have a captive audience they are trying to ransom. But that is neither relevant to this discussion, not controversial - unless you are paid by one of those ISPs….)
Comcast Verizon AT&T Time Warner Cable Cox CenturyLink
to name a few.
Not one of them will run BGP with a residential subscriber.
Who cares? [See below.]
And please don't reply with "then why can't I run BGP on my [cable|DSL|etc.] link?" Broadband providers are not trying to throttle growth by not allowing grandma to do BGP, and swapping to LISP or anything else won't change that.
Sure they are. If they weren't, it would be relatively straight forward to add the necessary options to DHCP for a minimal (accept default, advertise local) BGP configuration and it would be quite simple for CPE router manufacturers to incorporate those capabilities.
The problem is BGP doesn't scale to that level and everyone knows it, so, we limit growth by not allowing it to be a possibility.
This is patently false. No network has a decision matrix that is "BGP doesn't scale, so let's refuse money from customers". Every single one of the companies you listed will run BGP with customers. You limited this to "residential subscriber". Companies do not have only "residential customers". Pay more, get more. Pay $40, get less. Shocker. "Not if you don't pay for it" is not a valid argument against "every $COMPANY has $FEATURE". I said the barrier to entry for multihoming was lower than it has ever been. I didn't say it was zero. You are a pretty smart guy, so I'm going to give you the benefit of the doubt and assume you just kinda-sortta forgot or did not consider the whole "money" thing, despite the fact the only reason nearly every Internet entity exists. (Now I wonder how many people are going to tell me about the N% which are non-profits, despite the fact I said "nearly"?) -- TTFN, patrick
You are right, however, LISP won't change that.
LISP is about seperating the role of the ISP (as routing provider) from the end user or content provider/consumer.
I am unconvinced that is a good idea. At least using the definition of "end use" or "consumer" I frequently hear.
+1
However, a locator/id separation without map/encap is a desirable thing that could allow the routing system to scale better. Unfortunately, we failed to address this issue when designing IPv6. It will not get correctly solved without a revision to the header design. There is no will to change the packet header in the near future. We're having too much "fun" rolling out the current one.
Owen
Sent from my iPad On Mar 20, 2013, at 10:18 AM, "Patrick W. Gilmore" <patrick@ianai.net> wrote:
On Mar 20, 2013, at 09:25 , Owen DeLong <owen@delong.com> wrote:
I don't know a single ISP that wants to throttle growth by not accepting additional customers, BGP speaking or not. (I do know several that want to throttle growth through not upgrading their links because they have a captive audience they are trying to ransom. But that is neither relevant to this discussion, not controversial - unless you are paid by one of those ISPs….)
Comcast Verizon AT&T Time Warner Cable Cox CenturyLink
to name a few.
Not one of them will run BGP with a residential subscriber.
Who cares? [See below.]
Not one of them will run BGP with a commercial subscriber using a cost-effective edge technology.
And please don't reply with "then why can't I run BGP on my [cable|DSL|etc.] link?" Broadband providers are not trying to throttle growth by not allowing grandma to do BGP, and swapping to LISP or anything else won't change that.
Sure they are. If they weren't, it would be relatively straight forward to add the necessary options to DHCP for a minimal (accept default, advertise local) BGP configuration and it would be quite simple for CPE router manufacturers to incorporate those capabilities.
The problem is BGP doesn't scale to that level and everyone knows it, so, we limit growth by not allowing it to be a possibility.
This is patently false. No network has a decision matrix that is "BGP doesn't scale, so let's refuse money from customers".
In so many words, no, but it is the net effect when you distill down the other contents of the matrix.
Every single one of the companies you listed will run BGP with customers. You limited this to "residential subscriber". Companies do not have only "residential customers". Pay more, get more. Pay $40, get less. Shocker.
I pay $99/month to Comcast and they won't even give me a static address. That's a "business class" service from them. OTOH, I have two ISPs that do BGP with me for free.
"Not if you don't pay for it" is not a valid argument against "every $COMPANY has $FEATURE".
I said the barrier to entry for multihoming was lower than it has ever been. I didn't say it was zero.
The barrier is lower, but it's still higher than it should be.
You are a pretty smart guy, so I'm going to give you the benefit of the doubt and assume you just kinda-sortta forgot or did not consider the whole "money" thing, despite the fact the only reason nearly every Internet entity exists. (Now I wonder how many people are going to tell me about the N% which are non-profits, despite the fact I said "nearly"?)
I'm paying way more per month to the providers that refuse to do BGP with/for me than I am paying to the providers that ARE doing BGP with/for me. Clearly money is not the issue. Owen
On Mar 20, 2013, at 16:20 , Owen DeLong <owen@delong.com> wrote:
On Mar 20, 2013, at 10:18 AM, "Patrick W. Gilmore" <patrick@ianai.net> wrote:
On Mar 20, 2013, at 09:25 , Owen DeLong <owen@delong.com> wrote:
Not one of them will run BGP with a residential subscriber.
Who cares? [See below.]
Not one of them will run BGP with a commercial subscriber using a cost-effective edge technology.
[snip] I'm literally at a loss how to respond. This whole post is either a contradiction ("ISPs do free BGP", "the barrier to entry for BGP is higher than it should be"), or non-sequitors ("Comcast charges me $99 and doesn't give me a static IP address", uh... so?), or simply wrong statements ("BGP doesn't scale so we limit growth", "no we don't", "not in so many words, but yes we do"). What exactly are you trying to say? Because I apparently am too stupid to understand.
You are a pretty smart guy, so I'm going to give you the benefit of the doubt and assume you just kinda-sortta forgot or did not consider the whole "money" thing, despite the fact the only reason nearly every Internet entity exists. (Now I wonder how many people are going to tell me about the N% which are non-profits, despite the fact I said "nearly"?)
I'm paying way more per month to the providers that refuse to do BGP with/for me than I am paying to the providers that ARE doing BGP with/for me. Clearly money is not the issue.
You are confused. Money is (to a first approximation) ALWAYS the problem. Just because two companies sell things at different prices does not mean money is not at the heart of each company's pricing strategy. OF COURSE it is. They.... Oh, never mind. I'm going to take away the benefit of the doubt I gave you. And I think I'm going to stop feeding the ridiculously obvious troll. -- TTFN, patrick
In message <59415DCC-2D4E-4DD9-87C9-0B56BF24FCCF@ianai.net>, "Patrick W. Gilmor e" writes:
On Mar 20, 2013, at 09:25 , Owen DeLong <owen@delong.com> wrote:
I don't know a single ISP that wants to throttle growth by not = accepting additional customers, BGP speaking or not. (I do know several = that want to throttle growth through not upgrading their links because = they have a captive audience they are trying to ransom. But that is = neither relevant to this discussion, not controversial - unless you are = paid by one of those ISPs=85.) =20 Comcast Verizon AT&T Time Warner Cable Cox CenturyLink =20 to name a few. =20 Not one of them will run BGP with a residential subscriber.
Who cares? [See below.]
And please don't reply with "then why can't I run BGP on my = [cable|DSL|etc.] link?" Broadband providers are not trying to throttle = growth by not allowing grandma to do BGP, and swapping to LISP or = anything else won't change that. =20 Sure they are. If they weren't, it would be relatively straight = forward to add the necessary options to DHCP for a minimal (accept = default, advertise local) BGP configuration and it would be quite simple = for CPE router manufacturers to incorporate those capabilities. =20 The problem is BGP doesn't scale to that level and everyone knows it, = so, we limit growth by not allowing it to be a possibility.
This is patently false. No network has a decision matrix that is "BGP = doesn't scale, so let's refuse money from customers".
Every single one of the companies you listed will run BGP with = customers. You limited this to "residential subscriber". Companies do = not have only "residential customers". Pay more, get more. Pay $40, get = less. Shocker.
"Not if you don't pay for it" is not a valid argument against "every = $COMPANY has $FEATURE".
I said the barrier to entry for multihoming was lower than it has ever = been. I didn't say it was zero.
You are a pretty smart guy, so I'm going to give you the benefit of the = doubt and assume you just kinda-sortta forgot or did not consider the = whole "money" thing, despite the fact the only reason nearly every = Internet entity exists. (Now I wonder how many people are going to tell = me about the N% which are non-profits, despite the fact I said = "nearly"?)
--=20 TTFN, patrick
And homenet at the IETF demonstrated multi-homed residential connections with IPv6 without NAT using multiple PA addresses. If a upsteam goes down the connections over that upstream break. New connection use the working upstream. It's not quite the same as using PI but it is a 99.9% solution. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Mar 20, 2013, at 7:25 AM, Owen DeLong <owen@delong.com> wrote:
And please don't reply with "then why can't I run BGP on my [cable|DSL|etc.] link?" Broadband providers are not trying to throttle growth by not allowing grandma to do BGP, and swapping to LISP or anything else won't change that.
Sure they are. If they weren't, it would be relatively straight forward to add the necessary options to DHCP for a minimal (accept default, advertise local) BGP configuration and it would be quite simple for CPE router manufacturers to incorporate those capabilities.
The problem is BGP doesn't scale to that level and everyone knows it, so, we limit growth by not allowing it to be a possibility.
I suspect it has nothing to do with the scaling properties of routing tables and everything to do with customer support costs. The metrics associated with broadband services are quite daunting; i.e. costs from a single technical customer support call can exceed the entire expected profit over the typical customer contract period... In such circumstances, you really don't want any quantity of residential customers running BGP, as it increases the probability of customer care calls. It's only at a different revenue point (i.e. "small-business service") that it becomes viable. FYI, /John
Sent from my iPad On Mar 20, 2013, at 10:25 AM, John Curran <jcurran@istaff.org> wrote:
On Mar 20, 2013, at 7:25 AM, Owen DeLong <owen@delong.com> wrote:
And please don't reply with "then why can't I run BGP on my [cable|DSL|etc.] link?" Broadband providers are not trying to throttle growth by not allowing grandma to do BGP, and swapping to LISP or anything else won't change that.
Sure they are. If they weren't, it would be relatively straight forward to add the necessary options to DHCP for a minimal (accept default, advertise local) BGP configuration and it would be quite simple for CPE router manufacturers to incorporate those capabilities.
The problem is BGP doesn't scale to that level and everyone knows it, so, we limit growth by not allowing it to be a possibility.
I suspect it has nothing to do with the scaling properties of routing tables and everything to do with customer support costs.
The metrics associated with broadband services are quite daunting; i.e. costs from a single technical customer support call can exceed the entire expected profit over the typical customer contract period...
An interesting idea. In my case, I average about 3 calls per month to Comcast. I suspect this more than consumes the $99/month I pay them for internet service. Further, I often get service credits out of those calls that further reduce their income. If they provided native dual-stack with BGP and their service didn't go down on a regular basis, it would result in fewer calls, at least from me.
In such circumstances, you really don't want any quantity of residential customers running BGP, as it increases the probability of customer care calls. It's only at a different revenue point (i.e. "small-business service") that it becomes viable.
I don't want the residential customers themselves running BGP at all. However, if there were motivation on the provider side, automated BGP configuration could enable consumers to attach to multiple providers and actually reduce support calls significantly. Owen
On Mar 20, 2013, at 2:25 PM, Owen DeLong <owen@delong.com> wrote:
I don't want the residential customers themselves running BGP at all. However, if there were motivation on the provider side, automated BGP configuration could enable consumers to attach to multiple providers and actually reduce support calls significantly.
Okay, I'll agree, but "if there were motivation" is a very large "if"... The only motivation would be money, as represented by customers leaving to competitors as a result of a service provider not offering your proposed service bundle. If you can figure out a way to persuade service providers of this belief, I would ask that you also persuade them that they have to offer dual-stack for all of their customers (which may have already resulted in them losing a small number of customers who expected IPv6 by now... :-) Thanks! /John
On Mar 20, 2013, at 8:11 PM, John Curran <jcurran@istaff.org> wrote:
On Mar 20, 2013, at 2:25 PM, Owen DeLong <owen@delong.com> wrote:
I don't want the residential customers themselves running BGP at all. However, if there were motivation on the provider side, automated BGP configuration could enable consumers to attach to multiple providers and actually reduce support calls significantly.
Okay, I'll agree, but "if there were motivation" is a very large "if"...
The only motivation would be money, as represented by customers leaving to competitors as a result of a service provider not offering your proposed service bundle.
If you can figure out a way to persuade service providers of this belief, I would ask that you also persuade them that they have to offer dual-stack for all of their customers (which may have already resulted in them losing a small number of customers who expected IPv6 by now... :-)
You, of all people, John, are very aware of my efforts on this basis. I agree it's a very large if. In fact, the very real probability that dissatisfied customers will use their ability to multi home and run BGP as a reduction of the pain point of changing subscribers is probably the largest reason that it is not available. The providers have exact opposite motivation. This is a fine example of how the efficiency of the invisible hand fails when it comes to technical products where the masses fail to actually realize that they are being shafted and artificially constrained by the limitations placed on them by their vendors. However, that's getting a bit far afield for NANOG, so I tried to stick to the technical aspects of the argument. Owen
On 3/20/13, John Curran <jcurran@istaff.org> wrote:
On Mar 20, 2013, at 2:25 PM, Owen DeLong <owen@delong.com> wrote:
However, if there were motivation on the provider side, automated BGP configuration could enable consumers to attach to multiple providers and actually reduce support calls significantly.
Do you really think making a large SP making theirt customers' configuration more complicated, using a protocol at a scale its implementations were never designed for, will amount to a reduction in net average support costs? See... I think there is an economic argument to made against massive multihoming; upward sloping supply curve situation, ultimately slots in the global routing table are a competitive market. Providing service to a network that wants to be multihomed could be expected to incur a greater marginal price on the provider (additional overhead to implement, maintain, and service the more complicated service). If that added price tag exceeds the amount that the customer values their marginal benefit from multihoming, then requiring multihoming hurts the provider, because a lesser quantity is purchased, and hurts the customer, because their increased payment in excess of the benefit is added cost. The more multihomed customers, the more routes, the greater the marginal cost of adding every BGP router, the greater cost of every route advertised, which you could speculate the tier1's will ultimately be passing onto service providers, and then the customers, in due time. The increased price tags reduce the quantity of services purchased.
If you can figure out a way to persuade service providers of this belief, I would ask that you also persuade them that they have to offer dual-stack for all of their customers (which may have already resulted in them losing a small number of customers who expected IPv6 by now... :-)
Until people are actually using dual-stack services, the current perceived benefit is $0, so it's really a tough argument to make. You have to rely on the prediction, that within a few years, dual-stack services will provide the added benefit of full internet reachability, and ipv4-only services will have significant impairments.
Thanks! /John -- -JH
On Wed, Mar 20, 2013 at 9:25 AM, Owen DeLong <owen@delong.com> wrote:
However, a locator/id separation without map/encap is a desirable thing that could allow the routing system to scale better. Unfortunately, we failed to address this issue when designing IPv6. It will not get correctly solved without a revision to the header design. There is no will to change the packet header in the near future. We're having too much "fun" rolling out the current one.
Hi Owen, Right problem, wrong part of the problem. As is, the IPv6 layer 3 headers have plenty of bits to do a dandy job in a loc/id separation scheme: merely strip the ID function from the IP address and push it up the stack to layer 4. The crux of the problem, then, is that ID should be maintained by the layer 4 protocol with a dynamic and changeable mapping to the layer 3 locator. We don't need a new IP. We need a new TCP. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On 3/20/13 6:40 AM, Patrick W. Gilmore wrote:
I don't know a single ISP that wants to throttle growth by not accepting additional customers, BGP speaking or not. (I do know several that want to throttle growth through not upgrading their links because they have a captive audience they are trying to ransom. But that is neither relevant to this discussion, not controversial - unless you are paid by one of those ISPs....)
Ran across more then one in the Idaho/Oregon/Washington State area that told me it was "too hard" for them to do BGP with customers, or my favorite, that it would cost over 1k USD for setup, and 500 USD a month to do a BGP session with them from our rack in their data center. Some are either just lazy or incompetent. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
On 3/20/2013 12:36 PM, Brielle Bruns wrote:
On 3/20/13 6:40 AM, Patrick W. Gilmore wrote:
I don't know a single ISP that wants to throttle growth by not accepting additional customers, BGP speaking or not. (I do know several that want to throttle growth through not upgrading their links because they have a captive audience they are trying to ransom. But that is neither relevant to this discussion, not controversial - unless you are paid by one of those ISPs....)
Ran across more then one in the Idaho/Oregon/Washington State area that told me it was "too hard" for them to do BGP with customers, or my favorite, that it would cost over 1k USD for setup, and 500 USD a month to do a BGP session with them from our rack in their data center.
Some are either just lazy or incompetent.
Some just want to justify HUGE blocks of IP space to the RIR too. Andrew
Randy, On Mar 19, 2013, at 10:53 PM, Randy Bush <randy@psg.com> wrote:
i am not saying bgp and forwarding can deal with growth forever,
As I said when I started tilting at this particular windmill, with enough thrust pigs can fly quite well. However, perhaps instead of attaching bigger/hotter/more expensive rocket engines to the pig, moving to a more suitable lifting body might both reduce the need for the engine upgrades as well as provide some benefits to the folks wanting to buy airfare. Regards, -drc
Hi,
First of all, multihomed sites with its own global routing table entries bloats the global routing table, which is the major cause of global routing table bloat and is not acceptable.
Sorry, but that is false. Looking at the CIDR report (http://www.cidr-report.org/as2.0/#Gains) the routing table could shrink from 449k to 258k just by aggregating announcements. That's a reduction of 42.5%. I can't see how multihomed end-site announcements can be worst than that... There would almost be no routing table left ;-) Anyway... Drifting off-topic for this thread. Sander
Sander Steffann wrote:
Sorry, but that is false. Looking at the CIDR report (http://www.cidr-report.org/as2.0/#Gains) the routing table could shrink from 449k to 258k just by aggregating announcements.
What if, NLIs are aggregated?
That's a reduction of 42.5%. I can't see how multihomed end-site announcements can be worst than that...
See "5.2. Limiting the Number of TLAs" of my draft.
Anyway... Drifting off-topic for this thread.
Current poor support for multihomed sites is a reason why BCP38 is not operational. Masataka Ohta
Masataka, Do you have data to support your claim? I would said that poor BCP38 deployment it is because a lack of economical incentive. I have only empirical data to support my claim though (which is private conversations with ISPs not doing it and saying that they do not see a cost effective advantage). Regards, as On 3/18/13 12:02 PM, Masataka Ohta wrote:
Current poor support for multihomed sites is a reason why BCP38 is not operational.
I think BCP it is a solution. Perhaps not complete but hardy any single solution would be suitable for a complex problem as this one. If you are the end-user organization with a multihomed topology you apply BCP38 within your own scope. This will help to have less spoofed traffic. Not solving all the problems but it would help not seeing your spoofed packets all over the Internet. And about the routing table size, it is not multihomed sites the offenders, it is large ISPs fragmenting because of traffic engineering or because lack of BGP knowledge. .as On 3/18/13 2:47 AM, Masataka Ohta wrote:
Mark Andrews wrote:
Yes, BCP38 is the solution.
It is not a solution at all, because it, instead, will promote multihomed sites bloats the global routing table.
How does enforcing that source address entering your net from customers sites match thoses that have been allocated to them bloat the routing table?
First of all, multihomed sites with its own global routing table entries bloats the global routing table, which is the major cause of global routing table bloat and is not acceptable.
Then, the only solution is to let the multihomed sites have multiple prefixes, each of which is aggregated by each provider.
But, then, all the end systems are required to choose the proper source addresses corresponding to destination addresses, which requires IGPs carry such information.
See draft-ohta-e2e-multihoming-05 for details.
Now if you only accept address you have allocated to them by you then that could bloat the routing table but BCP 38 does NOT say to do that. Simlarly URP checking is not BCP 38.
That BCP 38 is narrowly scoped is not my problem.
With SIDR each multi-homed customer could provide CERTs which proves they have been allocated a address range which could be feed into the acl generators as exceptions to the default rules. This is in theory automatible.
The problem is not in individual ISPs but in the global routing table size.
How does that solve the problem?
In the end to end fashion.
See draft-ohta-e2e-multihoming-05 for details.
Masataka Ohta
On 2013-03-18, at 08:53, Arturo Servin <arturo.servin@gmail.com> wrote:
And about the routing table size, it is not multihomed sites the offenders, it is large ISPs fragmenting because of traffic engineering or because lack of BGP knowledge.
The usual concern with multi-homed end sites is that end sites with IPv4 PA addresses assigned from provider X who wish to multi-home with provider Y wind up adding at least two entries to the global table, a more specific route to each of X and Y (which X will need to leak beneath the covering supernet if it wants to deliver the customer any traffic). I don't know of any recent analysis which differentiates between this multi-homing pressure on the global table vs. inter-domain traffic engineering or gratuitous deaggregation, but it's fair to say I have not been looking. Joe
Arturo Servin wrote:
If you are the end-user organization with a multihomed topology you apply BCP38 within your own scope. This will help to have less spoofed traffic. Not solving all the problems but it would help not seeing your spoofed packets all over the Internet.
It does not help not seeing a spoofed packets with source addresses of yours.
And about the routing table size, it is not multihomed sites the offenders, it is large ISPs fragmenting because of traffic engineering or because lack of BGP knowledge.
As the number of *LARGE* ISPs is limited, it is not a problem. Masataka Ohta
It does if I am not the sender. .as On 3/18/13 12:10 PM, Masataka Ohta wrote:
Arturo Servin wrote:
If you are the end-user organization with a multihomed topology you apply BCP38 within your own scope. This will help to have less spoofed traffic. Not solving all the problems but it would help not seeing your spoofed packets all over the Internet. It does not help not seeing a spoofed packets with source addresses of yours.
participants (34)
-
Adam Vitkovsky -
Aled Morris -
Andrew D Kirch -
Arturo Servin -
Brielle Bruns -
Christopher Morrow -
Damian Menscher -
David Conrad -
Dobbins, Roland -
Doug Barton -
George Herbert -
Jared Mauch -
Jimmy Hess -
Joe Abley -
joel jaeggli -
John Curran -
Jon Lewis -
Kyle Creyts -
Leo Bicknell -
Luigi Iannone -
Mark Andrews -
Masataka Ohta -
Matt Palmer -
Matthew Walster -
Mike -
ML -
Owen DeLong -
Patrick W. Gilmore -
Randy Bush -
Sander Steffann -
Seth Mattinen -
Steven Fischer -
Valdis.Kletnieks＠vt.edu -
William Herrin