I'm really interested to see what happens when we start filling those same routers with ipv6 routes. Randy Bush <randy@psg.com> Sent by: owner-nanog@merit.edu 10/08/2007 06:10 PM To "Justin M. Streiner" <streiner@cluebyfour.org> cc nanog <nanog@merit.edu> Subject Re: How Not to Multihome

It's not 'law' per se, but having the customer originate their own announcements is definitely the Right Way to go.

it is interesting, and worrysome, to consider this in light of likely growth in the routing table (ref ipv4 free pool run out discussion) and vendors' inability to handle large ribs and fibs on enterprise class routers. randy

please elaborate. My knowledge of IPv6 is admittedly lacking, but I always assumed that the routing tables would be much larger if the internet were to convert from IPv4 due to the sheer number of networks available. Joel Jaeggli <joelja@bogus.com> Sent by: owner-nanog@merit.edu 10/08/2007 06:49 PM To Keegan.Holley@sungard.com cc Randy Bush <randy@psg.com>, nanog <nanog@merit.edu>, owner-nanog@merit.edu, "Justin M. Streiner" <streiner@cluebyfour.org> Subject Re: How Not to Multihome Keegan.Holley@sungard.com wrote:

I'm really interested to see what happens when we start filling those same routers with ipv6 routes.

All 970 of them? joelja

*Randy Bush <randy@psg.com>* Sent by: owner-nanog@merit.edu

10/08/2007 06:10 PM

To "Justin M. Streiner" <streiner@cluebyfour.org> cc nanog <nanog@merit.edu> Subject Re: How Not to Multihome

...

It's not 'law' per se, but having the customer originate their own announcements is definitely the Right Way to go.

it is interesting, and worrysome, to consider this in light of likely growth in the routing table (ref ipv4 free pool run out discussion) and vendors' inability to handle large ribs and fibs on enterprise class routers.

randy

On Mon, 8 Oct 2007, Keegan.Holley@sungard.com wrote:

please elaborate. My knowledge of IPv6 is admittedly lacking, but I always assumed that the routing tables would be much larger if the internet were to convert from IPv4 due to the sheer number of networks available.

Not many networks are pushing IPv6 at this point, or more correctly, not many relative to the 230k+ routes that currently makes up a full IPv4 routing table. There likely won't be a mass flash-cut to IPv6, which is a subject that has been debated to death and back again here on NANOG over the last few weeks :) jms

Mike Leber <mleber@he.net> wrote on 10/08/2007 07:36:56 PM:

On Mon, 8 Oct 2007 Keegan.Holley@sungard.com wrote:

...

I'm really interested to see what happens when we start filling those

same

...

routers with ipv6 routes.

Well, IPv6 prefixes will eventually be some number between the total number of ASes in use (which represents the number of networks that can afford and desire to run BGP) and the number of IPv4 prefixes in use (which represents the number of customers that can afford, justify, and desire to get address space).

So today, if IPv6 was instantly ubiquitously deployed by every network on the planet that runs IPv4: you would would see between 26,249 and 235,174 IPv6 routes (data from http://bgp.potaroo.net/as6447/).

Wouldn't resources still be an issue. Since the address space is so much larger wouldn't the 235k v6 routes take up more than 4 times the router memory?

You bought or are planning to buy core routers that support IPv6 at wirespeed in hardware didn't you?

If you are (or plan to be) operating an IPv4 network for over 5 years

alone the folks here that can say 10 or 15+ years), you are planning core router purchases on a cycle like 3 to 5 years by estimating what you need at the end of that time and then specifying accordingly.

By looking at the graph at the top of the page http://bgp.potaroo.net/as6447/ for total route announcements you could make a wild guess that if you want a router that has a high probability of working without needing workarounds (or giving you unnecessary

(let headaches)

in 3 years it needs to handle 500,000 IPv4 routes and 500,000 IPv6 routes in hardware when you buy it. Arguably this is overkill for IPv6, and might last 5 to 7 years.

What about MPLS? The last time I poked around in one of the core routers there were about 1.2 million routes. This includes all the private space that is routed between different sites in the same vpn and customers that use overlapping IP space. I had always assumed the routing tables would be massive under IPv6.

*All* the core router vendors (Cisco, Juniper, Foundry, Force10, etc)

sell

routers that can do this today. If you are buying something that can't handle the 3 year IPv4 requirement, let alone the IPv6 requirement, why are doing that to yourself?

Can I interest you in some used M40's?

Contrary to what seems to be popular misconception, your refrigerator will not be multihomed under IPv6. There are dynamic economic pressures (such as consolidation, competition, effective regulatory monopoly, etc) that limit the number of networks in the global routing table.

Well I guess I can nix that rootkit for IP enabled cukoo clocks. : ) Keegan

That brings up an interesing point. My biggest fear was that one of my other customers could possible be closer to me that the ISP that provides the primary link and it would cause them to favor the backup link because of AS path. I think they are going to fight me on this and telling them to multihome to their original ISP would probably be frowned upon at this point. I was hoping that there was an RFC for multihoming that I could use to bail myself out. "Justin M. Streiner" <streiner@cluebyfour.org> Sent by: owner-nanog@merit.edu 10/08/2007 05:55 PM To Keegan.Holley@sungard.com cc nanog <nanog@merit.edu> Subject Re: How Not to Multihome On Mon, 8 Oct 2007, Keegan.Holley@sungard.com wrote:

I have a client that wants us to advertise an IP block assigned by another ISP. I know that the best practice is to have them request an AS number from ARIN and peer with us, etc. However, I cannot find any information that states as law. Does anyone know of a document or RFC that states this?

It's not 'law' per se, but having the customer originate their own announcements is definitely the Right Way to go. Some providers take a pretty dim view of seeing chunks of their address space show up in advertisements originating from someone who isn't one of their customers. It can make troubleshooting connectivity problems for that customer (from the provider's point of view) very painful, i.e. "Hey, this AS, who isn't one of our customers, is hijacking IP space assigned to one of our customers!" The provider could then contact your host's upstream(s) and ask them to drop said announcement under the impression they're stopping someone from doing something bad. Also, if some network out there aggregates prefixes in an aggessive/odd manner, the disjoint announcement, and the reachability info it contains could be washed out of their routing tables, causing connectivity problems. Standard caveats about the block being a /24 or larger also apply. jms

On Oct 8, 2007, at 6:19 PM, Justin M. Streiner wrote:

On Mon, 8 Oct 2007, Keegan.Holley@sungard.com wrote:

...

That brings up an interesing point. My biggest fear was that one of my other customers could possible be closer to me that the ISP that provides the primary link and it would cause them to favor the backup link because of AS path. I think they are going to fight me on this and telling them to multihome to their original ISP would probably be frowned upon at this point. I was hoping that there was an RFC for multihoming that I could use to bail myself out.

If you went ahead and did this, the more specific route being announced by you on behalf of your customer would be more likely to attract traffic back to you. Prefix length is checked in the BGP route selection process before AS path length. This would work in normal "everything works fine" situations, but when things break, troubleshooting the source of the customer's reachabilit woes will get very interesting.

You have made an assumption that the original upstream would not originate a prefix equivalent to the one you are originating. -- TTFN, patrick

On Oct 8, 2007, at 9:46 PM, Justin M. Streiner wrote:

On Mon, 8 Oct 2007, Patrick W. Gilmore wrote:

...

...

If you went ahead and did this, the more specific route being announced by you on behalf of your customer would be more likely to attract traffic back to you. Prefix length is checked in the BGP route selection process before AS path length. This would work in normal "everything works fine" situations, but when things break, troubleshooting the source of the customer's reachabilit woes will get very interesting.

You have made an assumption that the original upstream would not originate a prefix equivalent to the one you are originating.

Internally or externally? A /24 would exist in the provider's IGP to point traffic to that customer.

Well, "internally" is kinda useless to this discussion, wouldn't you think? I get the feeling that you are trying to ask a clever question there, but it didn't come across that way.

Off the top of my head, I don't see why the provider who holds the parent block would do this externally. If the provider has, say, a /18 and they assign a /24 of that to this customer, there would be no legitimate reason to originate that /24 and propagate it out to the rest of the Internet. Note that I don't consider breaking that /18 up into 64 /24s and announcing them all separately to accomplish some sort of poor-man's traffic engineering to be a legitimate reason :)

Interesting. Did you not read the first paragraph in this e-mail? In fact, I seem to recall that you wrote it (attribution is missing, so I can't be 100% certain). Personally, I'd call that a "legitimate reason". To be clear, I am not suggesting de-aggregating every CIDR down to / 24s. But the global table doesn't grow any more whether the customer announces the /24 from their own ASN, or if you muti-originate it from two upstreams - or just one upstream for that matter. So there is no "legitimate reason" to _not_ announce it, but there is a reason to announce it. -- TTFN, patrick

On 10/8/07, Keegan.Holley@sungard.com <Keegan.Holley@sungard.com> wrote:

That brings up an interesing point. My biggest fear was that one of my other customers could possible be closer to me that the ISP that provides the primary link and it would cause them to favor the backup link because of AS path. I think they are going to fight me on this and telling them to multihome to their original ISP would probably be frowned upon at this point. I was hoping that there was an RFC for multihoming that I could use to bail myself out.

What they're asking to do is really Just Fine. It may or may not accomplish what they want, depending on how they do it, e.g. whether traffic will go through their other ISP or yours. The main reason it's a Not Best Practice is that it often doesn't get what the user wants, e.g. the user only has a /29, so only the aggregate advertisement from their primary ISP works anyway, or it's PA space so they'll still have to give it up if they change ISPs. But if they've got a /24, that's big enough that most carriers will see it these days, and there's no need to clutter up the BGP ASN space if your user doesn't need the extra flexibility. There may even be a market for ISPs to do multihoming on a cabal basis, e.g. Carrier X and Carrier Y get a /20 that they assign routes from to customers that use both of them, but only advertise the aggregate to the rest of the net. They'd still need to handle the more specific routes internally and exchange them across their peering link, but wouldn't have to bother the rest of the net with that level of detail. -- ---- Thanks; Bill Note that this isn't my regular email account - It's still experimental so far. And Google probably logs and indexes everything you send it.

Also, if some network out there aggregates prefixes in an aggressive/ odd manner, the disjoint announcement, and the reachability info it contains could be washed out of their routing tables, causing connectivity problems.

How is this different than if the customers gets their own ASN and announces a sub-block from one of the providers? Or are you suggesting they should get PI space? ARIN will only hand out /22's or larger. If a client wants to multihome with a /23 or a /24 it has to be assigned by one of hte ISP's and removed from the aggregate. "Patrick W. Gilmore" <patrick@ianai.net> Sent by: owner-nanog@merit.edu 10/08/2007 06:16 PM To nanog <nanog@merit.edu> cc "Patrick W. Gilmore" <patrick@ianai.net> Subject Re: How Not to Multihome On Oct 8, 2007, at 5:55 PM, Justin M. Streiner wrote:

On Mon, 8 Oct 2007, Keegan.Holley@sungard.com wrote:

...

I have a client that wants us to advertise an IP block assigned by another ISP. I know that the best practice is to have them request an AS number from ARIN and peer with us, etc. However, I cannot find any information that states as law. Does anyone know of a document or RFC that states this?

It's not 'law' per se, but having the customer originate their own announcements is definitely the Right Way to go.

That is not at all guaranteed.

Some providers take a pretty dim view of seeing chunks of their address space show up in advertisements originating from someone who isn't one of their customers. It can make troubleshooting connectivity problems for that customer (from the provider's point of view) very painful, i.e. "Hey, this AS, who isn't one of our customers, is hijacking IP space assigned to one of our customers!" The provider could then contact your host's upstream (s) and ask them to drop said announcement under the impression they're stopping someone from doing something bad.

If you do you have permission from the owner of the block, you Should Not Announce it. If the owner gives you permission and can't figure out why their block is originated by another ASN as well, they need help. (Yes, I realize the latter part of the last sentence is probably true for the majority of providers, but whatever.) In either case, your hypothetical question should not hold.

Also, if some network out there aggregates prefixes in an aggessive/ odd manner, the disjoint announcement, and the reachability info it contains could be washed out of their routing tables, causing connectivity problems.

How is this different than if the customers gets their own ASN and announces a sub-block from one of the providers? Or are you suggesting they should get PI space? -- TTFN, patrick

On Oct 8, 2007, at 6:45 PM, Justin M. Streiner wrote:

On Mon, 8 Oct 2007, Patrick W. Gilmore wrote:

...

...

It's not 'law' per se, but having the customer originate their own announcements is definitely the Right Way to go.

That is not at all guaranteed.

I never said it was. My experience, both in my previous life as the operator of a regional ISP and since then in other capacities is that having disjoint origins for a chunk of some provider's address space is basically asking for trouble, and it's the kind of trouble that may ony pop up when something breaks.

I'm afraid your experience is not the same as many, many people. There are currently ~1500 prefixes with inconsistent origin AS. These are trivially identifiable: <http://www.cymru.com/BGP/incon_asn_list.html> Some of them are obvious mistakes (I doubt HKSuper is supposed to originate 4/8). But many of them are not, and the Internet works just fine.

My experience has also been that if a customer has a need to multihome and is willing to invest both in the equipment and the expertise to do so, then so be it.

That statement does not say anything.

...

If you do you have permission from the owner of the block, you Should Not Announce it.

Agreed.

...

If the owner gives you permission and can't figure out why their block is originated by another ASN as well, they need help. (Yes, I realize the latter part of the last sentence is probably true for the majority of providers, but whatever.)

...

In either case, your hypothetical question should not hold.

...

Also, if some network out there aggregates prefixes in an aggessive/odd manner, the disjoint announcement, and the reachability info it contains could be washed out of their routing tables, causing connectivity problems.

How is this different than if the customers gets their own ASN and announces a sub-block from one of the providers?

In the case you described, the provider who holds the parent address block should expect to see an advertisement for a chunk of that block come in as part of the BGP feeds they receive from their upstreams, and they need to accept traffic accordingly. The customer would need to tell the provider of their intentions to multihome. If the provider in question employs some type of ingress/egress filtering, that filter would need to be updated to recognize that traffic sourced from that sub-block as legitimate, even if it comes in over their normal transit pipes.

In the case I described, where the end user requested that a third party provide transit for their PA space, without that provider necessarily being aware of it is when things can break in strange and spectacular ways.

I stated above that you should not announce another provider's space without their permission, and you specifically agreed. Here you are talking about that space being announced without the owner's knowledge. Make up your mind. We both agree that announcing another provider's space without their consent is a Very Bad Thing. I doubt you will find people willing to post here to the contrary. If the owner _does_ know the space is being announced, the edge filters are no different whether it is originated in the second upstream's ASN or an ASN owned by the customer. So again, I ask, how is that different?

...

Or are you suggesting they should get PI space?

PI space, while nice, is not an option for many end users.

Which is why I was assuming you did not mean PI space, but wanted to ask in case you were going to suggest it. -- TTFN, patrick

On Oct 9, 2007, at 8:15 AM, Jamie Bowden wrote:

...

On Oct 8, 2007, at 6:45 PM, Justin M. Streiner wrote:

...

On Mon, 8 Oct 2007, Patrick W. Gilmore wrote:

...

...

If you do you have permission from the owner of the block, you Should Not Announce it.

Agreed.

I stated above that you should not announce another provider's space without their permission, and you specifically agreed. Here you are talking about that space being announced without the owner's knowledge. Make up your mind.

We both agree that announcing another provider's space without their consent is a Very Bad Thing. I doubt you will find people willing to post here to the contrary. If the owner _does_ know the space is being announced, the edge filters are no different whether it is originated in the second upstream's ASN or an ASN owned by the customer.

So again, I ask, how is that different?

You'll note that you left a word out above. He's agreeing that even if you do have permission, you shouldn't announce space from another provider.

Type-o's suck. Thanx for pointing it out. Sorry for the confusion. Justin, if Provider A _has_ permission from Provider B to announce a prefix, do you believe Provider A should be allowed to announce the prefix? -- TTFN, patrick

Justin M. Streiner wrote:

On Tue, 9 Oct 2007, Patrick W. Gilmore wrote:

...

Justin, if Provider A _has_ permission from Provider B to announce a prefix, do you believe Provider A should be allowed to announce the prefix?

As long as all of the relevant parties know about it and are OK with it, that's fine. It's just not my first choice for solving the customer's multihoming dilemma, that's all :)

jms

Back when I was a NOC monkey (that stopped a month ago), I had exactly that situation. I had MCI and SBC as upstreams. Before multihoming, my network was split in two segments, one for each substream. This made things like DNS interesting. When I got my ASN, I got agreement from both MCI and SBC to announce my /21 allocations from them over both upstream circuits. As a result, I was able to go back to a single inside network, a single pair of DNS servers, and no more cross-router traffic via the Internet cloud. I then got my ARIN allocation and went through the Fiscal Quarter From Hell renumbering everything into the new number block. I dropped MCI (long story) and lit up Idacomm, but kept SBC link and numbers. When I left the ISP, my routers had been announcing my suballocation of SBC space for more than a year. With their permission. Their only requirement is that I have proper routing objects in a routing registry so SBC could see that the route I was announcing was valid. (What was VERY interesting was that I was using the ARIN registry, and SBC was not. The resulting bru-ha-ha uncovered a synchronization problem that ARIN had, and that ARIN fixed.)

On Oct 9, 2007, at 1:53 PM, Valdis.Kletnieks@vt.edu wrote:

...

There are currently ~1500 prefixes with inconsistent origin AS. These are trivially identifiable:

<http://www.cymru.com/BGP/incon_asn_list.html>

Some of them are obvious mistakes (I doubt HKSuper is supposed to originate 4/8). But many of them are not, and the Internet works just fine.

And as Justin said, some sizable fraction of those 1500 prefixes are quite possibly *appearing* to Work Just Fine currently, but if something breaks, that will be 1500 NOC monkeys facing some difficult-to-debug routing issues....

Considering the number of inconsistently originated prefixes has been non-trivial for at least a decade, I have trouble believing this is a huge threat to the internet. Or even those 1500 NOC monkeys. (And wouldn't it be 3K - at least 2 ASNs per prefix? :) -- TTFN, patrick

On 9 Oct 2007, at 17:47, Andy Davidson wrote: [...]

However, if a different third-party network then sweeps up their routing table by looking to remove more specifics that seem 'spoofed' using IRR data, the routes you intend to push onto the internet may well start to disappear from their perspective.

I don't think this should be possible if the database implements RPSS (RFC 2725) properly. I believe that it should only be possible to create a more specific route object with the agreement using whatever PGP/X.509 security you like if you have used mnt-lower and mnt-routes attributes as appropriate. I'm not sure I'd want to publish my routing policy in a database that didn't have a reasonable implementation of RPSS. Leo