Nice lecture, thanks. But I saw always when thinking about multihoming some other problems also. 2] Multihomed to different ISP without BGP, supposing I need only one /24. It will be PA or PI addressing space ?? With respect to CIDR aggregation effort, make it sense to require /24 PI address block?? How will route the second ISP my PA from the other ISP (he will de-aggregate the block of the second provider to announce more specific prefix??) ?? 2] Multihomed to different ISP with BGP , supposing I need only one /24. Again, make it sense to have my own AS and /24 PI block ?? Supposing to have IBGP to one ISP (to avoid assigning of independent AS) and EBGP to the second ISP, again will my router announce the /24 inside of the first ISP address block to the EBGP peer ?? 2. Multi-homed. Enterprise connects to more than one ISP. May or may not use BGP. Wishes to protect against problems in the ISP routing system, and will accept additional complexity and router requirements to get this. May also have differing service agreements for Internet access for different divisions.
At 12:25 -0400 7/24/97, Jan Novak wrote:
Nice lecture, thanks. But I saw always when thinking about multihoming some other problems also.
In the taxonomy I proposed, I tried to stay at the level of the customer requirement, rather than the specific details of addressability. The cases you cite are legitimate, but are in my mind at the next level of detail -- how one actually implements multihoming. I'll make some general comments. I need to think a bit whether these would be logically at a more detailed level of the taxonomy, or are in an implementation taxonomy of their own. Even beyond BGP capabiities (of either the enterprise or the ISP), your examples are going to be affected by the particular ISPs' (and their upstreams) aggregation and prefix filtering policies.
2] Multihomed to different ISP without BGP, supposing I need only one /24. It will be PA or PI addressing space ??
From the taxonomy standpoint, it could be either. Current registry
policies from RFC 2050 would generally say PA.
With respect to CIDR aggregation effort, make it sense to require /24 PI address block?? How will route the second ISP my PA from the other ISP (he will de-aggregate the block of the second provider to announce more specific prefix??) ??
2] Multihomed to different ISP with BGP , supposing I need only one /24. Again, make it sense to have my own AS and /24 PI block ??
Supposing to have IBGP to one ISP (to avoid assigning of independent AS) and EBGP to the second ISP, again will my router announce the /24 inside of the first ISP address block to the EBGP peer ??
Interesting approach. In general, the ISPs I know would be reluctant to run iBGP with a customer, unless they had total control of all BGP speakers. If I understand you correctly, the enterprise would have to tag its advertisements to the second ISP with the ASN of the first, since the enterprise doesn't have its own. Again, I think most ISPs would be reluctant to give up this amount of control.
2. Multi-homed. Enterprise connects to more than one ISP. May or may not use BGP. Wishes to protect against problems in the ISP routing system, and will accept additional complexity and router requirements to get this. May also have differing service agreements for Internet access for different divisions.
On Thu, 24 Jul 1997, Howard C. Berkowitz wrote:
Interesting approach. In general, the ISPs I know would be reluctant to run iBGP with a customer, unless they had total control of all BGP speakers. If I understand you correctly, the enterprise would have to tag its advertisements to the second ISP with the ASN of the first, since the enterprise doesn't have its own. Again, I think most ISPs would be reluctant to give up this amount of control.
I think most of the companies running redundant links now have their own address space and ASN. We got our primary address blocks back when a company could still do that. I think there's going to have to be some way to address that with semi-portable AS' in the near future though, as more criticality transitions to the Net. That, or people will start buying up service providers to get address blocks, then they'll own the routers, and work out their iBGP issues "internally". Not that that works for smaller companies who want it, but if you're a multi-billion dollar corporation, it's an option (yes, it should scare you). I know at least one tier 1 has started offering seperate wireline into different NAPs in the DC area, which is about as good as you can get without going to two providers. They want a lot of money for it though, and the gains of a second provider are much more cost-effective from a strict redundancy standpoint. I don't know how we can get a combination of aggragate routing and multi-homing to scale correctly, but I think it's becomming more important that we do so. Paul ------------------------------------------------------------------------- Paul D. Robertson gatekeeper@gannett.com
What's interesting to me is the few brave souls who are geographically dispersed corporations using their own WAN for multihoming. If you already have leased lines between your offices and each office has its own internet access through a local provider, you can protect against failures of the local providers by rerouting over the WAN leased-lines to another office and popping out on *their* local provider's link. Very cost effective, since you already are paying for the "backup link" anyways as part of your enterprise network. -- gabriel m schuyler, outlaw schuyler@vorpal.net vorpal network - san francisco http://www.vorpal.net/
Supposing to have IBGP to one ISP (to avoid assigning of independent AS) and EBGP to the second ISP, again will my router announce the /24 inside of the first ISP address block to the EBGP peer ??
Interesting approach. In general, the ISPs I know would be reluctant to run iBGP with a customer, unless they had total control of all BGP speakers. If I understand you correctly, the enterprise would have to tag its advertisements to the second ISP with the ASN of the first, since the enterprise doesn't have its own. Again, I think most ISPs would be reluctant to give up this amount of control.
Without the ISP having total control over the customer router, a misconfiguration of filters on the customer side could easily cause the customer to be a valid (and 1 hop) path in the tables from ISP A to ISP B. Doesn't sound like a possibility I would be willing to have hanging over my head. --- -=<:gEm:>=- -<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-- Gordon Mercer -=<Dedicated>=- [digitalNATION] 703 642 2800 -=<Servers>=- gmercer@dn.net <::>=-=<::>=--=<::>=-=<::>=--=<::>=-=<::>=--=<::>=-=<::>
Without the ISP having total control over the customer router, a misconfiguration of filters on the customer side could easily cause the customer to be a valid (and 1 hop) path in the tables from ISP A to ISP B. Doesn't sound like a possibility I would be willing to have hanging over my head.
Well, since my bandwidth is necessary for my business, I think I'd be much more concerned about becomming the valid route than my upstreams, if they get better routing through me, it's not necessarily a bad thing for them unless they're concerned about me snarfing traffic. Plus, you can filter out what you send to me if you're my upstream. That means you'll need a misconfigured router on your side *and* one on mine. I don't know your competency, but I'm fairly certain of mine ;). I put a lot more time, effort and care into choosing a provider than you do into choosing a customer. I don't think it's as big of an issue, other than the obvious effects of router filtering performance, and the chance that the upstream could hose his filters when he goes to listen for routes to me from external sources if he's already got major paranoia filters. Hopefully, he's got that filtered to only happen from my other peering points though. It's not rocket science, but it does take some care in set-up. You have as much chance of getting control of my gateway routers as you have of turning into a purple poodle. I'd purchase Yet Another Service Provider and route a tier lower before I'd play that game. I've got a lot more to lose than my upstreams. Paul ------------------------------------------------------------------------- Paul D. Robertson gatekeeper@gannett.com
You wrote:
Without the ISP having total control over the customer router, a misconfiguration of filters on the customer side could easily cause the customer to be a valid (and 1 hop) path in the tables from ISP A to ISP B. Doesn't sound like a possibility I would be willing to have hanging over my head.
Well, since my bandwidth is necessary for my business, I think I'd be much more concerned about becomming the valid route than my upstreams, if they get better routing through me, it's not necessarily a bad thing for them unless they're concerned about me snarfing traffic.
They've also got to worry about your bandwidth, which could become a big issue depending on the size of the two providers involved.
Plus, you can filter out what you send to me if you're my upstream. That means you'll need a misconfigured router on your side *and* one on mine. I don't know your competency, but I'm fairly certain of mine ;). I put a lot more time, effort and care into choosing a provider than you do into choosing a customer. Paul
In the particular scenario being discussed, which routes would you want from your upstream? You might want full routes for the ability to actually choose best path, and then the upstream providers loose control over what you are sending where. A and B can both filter what the customer sends to them based on network, and then the problem is solved. Unfortunately, this does not always give customers the flexibility they are looking for. I'm sure you know exactly what you are doing, but not every Joe that a provider takes on does. My point is only that this is a situation that I would not want to bring upon myself. --- -=<:gEm:>=- -<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-- Gordon Mercer -=<Dedicated>=- [digitalNATION] 703 642 2800 -=<Servers>=- gmercer@dn.net <::>=-=<::>=--=<::>=-=<::>=--=<::>=-=<::>=--=<::>=-=<::>
On Thu, 24 Jul 1997, Gordon Mercer wrote:
You wrote:
Without the ISP having total control over the customer router, a misconfiguration of filters on the customer side could easily cause the customer to be a valid (and 1 hop) path in the tables from ISP A to ISP B. Doesn't sound like a possibility I would be willing to have hanging over my head.
Well, since my bandwidth is necessary for my business, I think I'd be much more concerned about becomming the valid route than my upstreams, if they get better routing through me, it's not necessarily a bad thing for them unless they're concerned about me snarfing traffic.
They've also got to worry about your bandwidth, which could become a big issue depending on the size of the two providers involved.
If they've oversold their provisioning, then yes, they would, but I can't see how other than that they would. Perhaps I'm missing something? In my particular case, my upstreams are UUNet and BBN, and I've been particularly happy with the current arrangement.
In the particular scenario being discussed, which routes would you want from your upstream? You might want full routes for the ability to actually choose best path, and then the upstream providers loose control over what you are sending where.
I get full routes from my peers. That doesn't mean they send me traffic based on destination addresses outside of those specifically linked to my AS. Why would they route traffic destined to someone else through my path if they were paranoid about me polluting things? I'd expect them to no do that as much as I expect them to not accept routes advertised by me that aren't in the address blocks I've specified. Maybe I'm missing something here, but it seems pretty cut and dried, and other than the filtering/CPU issues I don't see a major downside. Certainly my upstreams didn't seem to have a problem implementing it, and it's saved us bigtime a number of times since we started it.
I'm sure you know exactly what you are doing, but not every Joe that a provider takes on does. My point is only that this is a situation that I would not want to bring upon myself.
I can understand that. In my case, it was a couple of years in coming, but we'd planned for it at the start, and gotten agreements from the providers to do it during circuit upgrades. I'd have dropped a provider who wouldn't have agreed, since I had it as a critical need which it took a while to get funded, and to get management to buy in to. Long-term, I'm more concerned with the route aggragation problems once other folks start jumping on the bandwagon than I am with any particular application, mine included. Not just because I'm carrying full tables, but because CIDR was a necessary evil, and we're basically moving towards negating its advantages. Paul ------------------------------------------------------------------------- Paul D. Robertson gatekeeper@gannett.com
You wrote:
On Thu, 24 Jul 1997, Gordon Mercer wrote:
You wrote:
Without the ISP having total control over the customer router, a misconfiguration of filters on the customer side could easily cause the customer to be a valid (and 1 hop) path in the tables from ISP A to ISP B. Doesn't sound like a possibility I would be willing to have hanging over my head.
Well, since my bandwidth is necessary for my business, I think I'd be much more concerned about becomming the valid route than my upstreams, if they get better routing through me, it's not necessarily a bad thing for them unless they're concerned about me snarfing traffic.
They've also got to worry about your bandwidth, which could become a big issue depending on the size of the two providers involved.
If they've oversold their provisioning, then yes, they would, but I can't see how other than that they would. Perhaps I'm missing something? In my particular case, my upstreams are UUNet and BBN, and I've been particularly happy with the current arrangement.
I think I see where the miscommunication lies. We were discussing ISP's running IBGP sessions with multi-homed customers. giving you the ability to announce routes to another provider tagged with my AS is what makes me nervous. Are you announcing routes to BBN as AS 701? or to UUNet from AS 1? Besides that, becoming a valid shortest path between two providers that do more traffic between them than your link to either of them can handle IS dangerous for them, because it restricts their other customers' ability to talk to each other. If one of my customers had 'router bgp 7019' somehwere in their router configs, I wouldn't sleep well at night. Peace, Gordon --- -=<:gEm:>=- -<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-- Gordon Mercer -=<Dedicated>=- [digitalNATION] 703 642 2800 -=<Servers>=- gmercer@dn.net <::>=-=<::>=--=<::>=-=<::>=--=<::>=-=<::>=--=<::>=-=<::>
On Jul 24, Gordon Mercer <gmercer@dn.net> wrote:
I think I see where the miscommunication lies. We were discussing ISP's running IBGP sessions with multi-homed customers. giving you the ability to announce routes to another provider tagged with my AS is what makes me nervous.
Hmm. I could /maybe/ see allowing that as a seperate IBGP community, but even then it's dangerous. It's easy to get an ASN (even though it takes a while; I think the 'NIC still does those manually), so there's really no reason not to require multi-homed customers to do so if they want to talk BGP with you. But, seperate real ASN or seperate community, you'll still want to filter incoming announcements from the customer -- just as they'd probably want to filter announcements from you. ********************************************************* J.D. Falk voice: +1-415-482-2840 Supervisor, Network Operations fax: +1-415-482-2844 PRIORI NETWORKS, INC. http://www.priori.net "The People You Know. The People You Trust." *********************************************************
On Thu, Jul 24, 1997 at 04:49:29PM -0700, J.D. Falk wrote:
But, seperate real ASN or seperate community, you'll still want to filter incoming announcements from the customer -- just as they'd probably want to filter announcements from you.
I think you mean confederation, JD. Alec -- +------------------------------------+--------------------------------------+ |Alec Peterson - ahp@hilander.com | Erols Internet Services, INC. | |Network Engineer | Springfield, VA. | +------------------------------------+--------------------------------------+
You wrote:
On Thu, Jul 24, 1997 at 04:49:29PM -0700, J.D. Falk wrote:
But, seperate real ASN or seperate community, you'll still want to filter incoming announcements from the customer -- just as they'd probably want to filter announcements from you.
I think you mean confederation, JD.
Alec
--
Don't think he did, Alec. Using communities would make it much easier to filter the routes to the customer than using confederation. I don't think there's any need to implement confedrations here. Sounds like headaches I don't need. Communities would allow you to filter very specifically only routes coming from the router. The real problem here is that the ISP with the EBGP session still depends on the ISP with the IBGP session to do things correctly, unless customer routes are filtered at a network level -- Something I've never liked doing, but always felt was necessary. How can I have a setup that is flexible enough to satisfy my customer (and my workload) but safe for me? I've had customers running OSPF with one of my routers that was redistributing OSPF into BGP, and it was probably one of the stupidest mistakes I've ever made. Screwed me when some dumbass decided he could use whatever networks he wanted on the Sun they were running gated on.
+------------------------------------+--------------------------------------+
|Alec Peterson - ahp@hilander.com | Erols Internet Services, INC. | |Network Engineer | Springfield, VA. |
+------------------------------------+--------------------------------------+ --- -=<:gEm:>=- -<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-<sMp>-- Gordon Mercer -=<Dedicated>=- [digitalNATION] 703 642 2800 -=<Servers>=- gmercer@dn.net <::>=-=<::>=--=<::>=-=<::>=--=<::>=-=<::>=--=<::>=-=<::>
On Fri, Jul 25, 1997 at 09:01:13AM -0400, Gordon Mercer wrote:
Don't think he did, Alec. Using communities would make it much easier to filter the routes to the customer than using confederation. I don't think there's any need to implement confedrations here. Sounds like headaches I don't need. Communities would allow you to filter very specifically only routes coming from the router.
Well, comparing a 'real AS to a separate community' doesn't really sound right to me. Replacing community with confederation would make more sense, although I do see your point. However I believe JD's point is that it isn't _necessary_ to get a separate ASN if you've got a small downstream who doesn't care about having his AS visible to the outside world.
The real problem here is that the ISP with the EBGP session still depends on the ISP with the IBGP session to do things correctly, unless customer routes are filtered at a network level -- Something I've never liked doing, but always felt was necessary.
Unfortunately it is, as the AS7007 disaster illustrated all too clearly.
How can I have a setup that is flexible enough to satisfy my customer (and my workload) but safe for me?
MCI has a route registry that you send updates to just like the RADB (the RADB and MCI RR actually exchange data). I believe MCI then builds network-based access lists based on that database.
I've had customers running OSPF with one of my routers that was redistributing OSPF into BGP, and it was probably one of the stupidest mistakes I've ever made.
NONONONONO! Speaking IGP with customers bad!
Screwed me when some dumbass decided he could use whatever networks he wanted on the Sun they were running gated on.
Yep, there's the problem. BGP was designed to be an inter-domain routing protocol, and should be used as such. Unfortunately we need some sort of network-level control over what a customer sends upstream. Implementing some sort of automated scheme (like the MCI RR for example) is IMO the only scalable way of doing so. Alec -- +------------------------------------+--------------------------------------+ |Alec Peterson - ahp@hilander.com | Erols Internet Services, INC. | |Network Engineer | Springfield, VA. | +------------------------------------+--------------------------------------+
On Thu, 24 Jul 1997, Gordon Mercer wrote:
I think I see where the miscommunication lies. We were discussing ISP's running IBGP sessions with multi-homed customers. giving you the ability to announce routes to another provider tagged with my AS is what makes me nervous.
Ah, ok, that's the miscommunication.
Are you announcing routes to BBN as AS 701? or to UUNet from AS 1?
Nah, they wouldn't let me, I'd have to fix UUNet's TCO, and the first thing to go would be that damn ATM switch, *then* I'd have to fix BBN, and there ain't that much time in the world..... Paul ------------------------------------------------------------------------- Paul D. Robertson gatekeeper@gannett.com
At 12:01 -0400 7/24/97, root@gannett.com wrote:
Without the ISP having total control over the customer router, a misconfiguration of filters on the customer side could easily cause the customer to be a valid (and 1 hop) path in the tables from ISP A to ISP B. Doesn't sound like a possibility I would be willing to have hanging over my head.
Well, since my bandwidth is necessary for my business, I think I'd be much more concerned about becomming the valid route than my upstreams, if they get better routing through me, it's not necessarily a bad thing for them unless they're concerned about me snarfing traffic.
Plus, you can filter out what you send to me if you're my upstream. That means you'll need a misconfigured router on your side *and* one on mine. I don't know your competency, but I'm fairly certain of mine ;). I put a lot more time, effort and care into choosing a provider than you do into choosing a customer.
I don't think it's as big of an issue, other than the obvious effects of router filtering performance, and the chance that the upstream could hose his filters when he goes to listen for routes to me from external sources if he's already got major paranoia filters. Hopefully, he's got that filtered to only happen from my other peering points though.
It's not rocket science, but it does take some care in set-up. You have as much chance of getting control of my gateway routers as you have of turning into a purple poodle. I'd purchase Yet Another Service Provider and route a tier lower before I'd play that game. I've got a lot more to lose than my upstreams.
Paul, you clearly know what you are doing. But it's amazing how many organizations don't understand fundamental global routing concepts, and believe waving money at ISPs will make them do what they want even if that makes no sense. I've been doing design seminars for the pre-/post-sales tech support organizations of several national-level carriers. In a recent class, the students brought up a problem with one of their accounts, which I shall call Major Clueless Bank (MCB). Said bank wanted to offer consumer banking over the Internet. All their direct connectivity came from my client, National Service Provider (NSP-1), at several geographically dispersed points. By my taxonomy, single-homed, multi-linked. MCB desired to level the load over their various server farms and links to NSP-1. They had fixated on BGP as the way to do what they thought they wanted to do, which was to affect the MED passed to peers of NSP-1 based on loading of their servers. They also wanted to affect NSP-1's interior routing so they could advertise more specific routes to each of their server farms, again based on _their_ load. Several million a year in revenues were involved. IMHO, on looking at what they were trying to do, it wasn't even a routing problem. What they wanted was probably best done with DNS load control. They simply did not realize that what they wanted in routing would have marginal effect on the direct peers of NSP-1, and none on non-adjacent AS. Their fundamental mental model was an enterprise network where they were in control. And their next level of detail assumed everything could be controlled with IP routing. The concept that other traffic flowed in NSP-1, and that they could not control the routing of other AS with whom they had no business relationship, simply didn't penetrate. So if the ISP has to set general policies,they need to protect themselves against the NCBs of the world. Paranoid filtering isn't enough if the customer is demanding something not possible. A part of making multihoming practical is managing customer expectations and educating enterprise network designers (or encouraging them to _have_ designers). Howard I see this again and again.
On Thu, 24 Jul 1997, Howard C. Berkowitz wrote:
MCB desired to level the load over their various server farms and links to NSP-1. They had fixated on BGP as the way to do what they thought they wanted to do, which was to affect the MED passed to peers of NSP-1 based on loading of their servers. They also wanted to affect NSP-1's interior routing so they could advertise more specific routes to each of their server farms, again based on _their_ load. Several million a year in revenues were involved.
We went through that phase with senior management in one or two of our divisions. It's surprising how some folks interpret "It doesn't do that" to mean "Ask me again".
IMHO, on looking at what they were trying to do, it wasn't even a routing problem. What they wanted was probably best done with DNS load control.
Distributed Director is looking better all the time, if only they'd drop the price down to semi-managable. *sigh*
They simply did not realize that what they wanted in routing would have marginal effect on the direct peers of NSP-1, and none on non-adjacent AS. Their fundamental mental model was an enterprise network where they were in control. And their next level of detail assumed everything could be controlled with IP routing.
Bwaahahahah but I *do* control the Internet ;)
The concept that other traffic flowed in NSP-1, and that they could not control the routing of other AS with whom they had no business relationship, simply didn't penetrate.
So if the ISP has to set general policies,they need to protect themselves against the NCBs of the world. Paranoid filtering isn't enough if the customer is demanding something not possible. A part of making multihoming practical is managing customer expectations and educating enterprise network designers (or encouraging them to _have_ designers).
Good point. I really just wanted to get a combination of things across, firstly that it's doable, and I think we probably are doing it in the most logical way, second of all, the routing infrastructure needs to change or routing aggragation will break, and lastly that even though it isn't always true, it is possible that the ISP is the least "victimized" in an incorrect set-up. But then, I think you've all got the easy jobs, since I have to deal with most of the same issues (over 130 business units will do that), as well as Appletalk, IPX and all the st00pid MS network garbage ;) [1] Paul "Arcserve backup is killing one of my internal 7513s" Robertson [1] Yes, it's a troll, save the list follow-ups and flame directly ------------------------------------------------------------------------- Paul D. Robertson gatekeeper@gannett.com
participants (7)
-
Alec H. Peterson -
Gabriel M. Schuyler -
Gordon Mercer -
Howard C. Berkowitz -
J.D. Falk -
jan.novak＠aliatel.cz -
root＠gannett.com