Failover how much complexity will it add?
HI, I was recently brought onto a project where some failover is desired, but I think that the number of connections provisioned is excessive. Also hoping to get some guidance with regards to how well I can get the failover to actually work. So currently 4 X 100Mb/s Internet connections have been provisioned. One is to be used for general Internet, out of the organisation, it also terminates VPNs from remote sites belonging to the organisation and some publicly accessible servers -routed DMZ and translated IPs. Second Internet connection to be used for a separate system which has a site-to-site VPN to a third party support vendor. Internet connections 3 and 4 are currently thought of as providing backups for one and two. Both connections firewalled by a Juniper SSG of some description. Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are: - Should I make sure that the backup Internet connection is from a separate provider? - How can I acheive a failover which doesn't require me to change all the remote VPN endpoints in case of a failover? Its possible to configure failover VPNs on the Junipers, which should take care of this, but how do I take care of the DMZ hosts and external translation? - In fact I think I'm asking what are my options with regard to failover between one Internet connection and the other? I'm hoping to figure out whether adding an extra Internet connection actually gives us that much, in fact whether it justifies the complexity and spend. Many Thanks for your comments. Adel
-----Original Message----- From: adel@baklawasecrets.com [mailto:adel@baklawasecrets.com] Sent: Sunday, November 08, 2009 4:52 AM To: nanog@nanog.org Subject: Failover how much complexity will it add?
HI,
I was recently brought onto a project where some failover is desired, but I think that the number of connections provisioned is excessive. Also hoping to get some guidance with regards to how well I can get the failover to actually work. So currently 4 X 100Mb/s Internet connections have been provisioned. One is to be used for general Internet, out of the organisation, it also terminates VPNs from remote sites belonging to the organisation and some publicly accessible servers -routed DMZ and translated IPs. Second Internet connection to be used for a separate system which has a site-to-site VPN to a third party support vendor. Internet connections 3 and 4 are currently thought of as providing backups for one and two. Both connections firewalled by a Juniper SSG of some description.
Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are:
- Should I make sure that the backup Internet connection is from a separate provider?
Yes yes yes yes a thousand times yes. Depending on the criticality of internet connectivity you should also aim to have your redundant connections coming from a complete separate direction. Example, fiber from Level 3 come from the north in a dedicated conduit and fiber from Verizon coming in a dedicated conduit from the south of the building. Why? Put simply we had construction ignore the painted lines and dig up our conduit a few years back. At that point we have 4 bonded T1's from a single carrier. That was a long couple of days... Carrier diversity is not a bad thing, spend some time shopping an additional provider. Make sure they operate their own network for last mile, and also make sure they don’t piggyback off the same network your main carrier does anywhere locally. Comcast Ethernet, Verizon and Cogent make great secondary connections when you need high availability. You don’t need your secondary to have 99.999% uptime. 97% is usually good enough if it's on a separate network. I wouldn't sway from the big names for your primary connections either.
- How can I acheive a failover which doesn't require me to change all the remote VPN endpoints in case of a failover? Its possible to configure failover VPNs on the Junipers, which should take care of this, but how do I take care of the DMZ hosts and external translation?
With recent experience with the Juniper SSG VPN functions put nicely they suck. VPN failover is in there, but we had issues with the tunnel staying active for extended periods of time. Also depending on if you do a route based or a policy based VPN, it becomes so much of a headache. We used 2 SSG550 devices as a proof of concept and the one thing which annoyed me to no end was the complete and total crap options within then VPN configuration. When I typically set up a VPN, I use a SonicWall NSA or E-class device (yes I know hiss boo) or an ASA. Saying that the Juniper was lacking was a complete understatement. I personally would completely avoid even attempting VPN failover within a Juniper device. I will say they are rock solid though for generic firewall functionality, just try to keep the config simple or they turn into giant slow dogs.
- In fact I think I'm asking what are my options with regard to failover between one Internet connection and the other?
Considering you have 4x 100mbit lines, have you looked at BGP? Even if you drop line 2 and its associated backup, you have 2x 100mbit lines. Or even if you have 3 unique carriers with a 100mbit from each of them it makes BGP very appealing. I think this would be an ideal situation for a BGP setup using a couple of small routers. You could probably get away with something as small as a Cisco 3825 for each connection (purely redundancy). If the Cisco name scares you Juniper routers are great as well. Don’t forget Vyatta! If you do BGP, you have 1 VPN to configure, you have 1 tunnel to configure, there is no VPN failover configuration and hopefully you are not pushing more than 1 subnet across the VPN otherwise you end up doing a route based VPN instead of a policy based VPN and you will be significantly happier. That’s a Juniper headache for another day however.
I'm hoping to figure out whether adding an extra Internet connection actually gives us that much, in fact whether it justifies the complexity and spend.
What you really need to know to ask this is how much is Mr. Customer going to yell and scream if someone cuts only internet connection and it's going to be down for about 3 days while they patch the conduit and fiber.
Many Thanks for your comments.
Adel
On 2009-11-08-10:23:41, Blake Pfankuch <bpfankuch@cpgreeley.com> wrote:
Make sure they operate their own network for last mile [...] I wouldn't sway from the big names for your primary connections either.
Because ownership of the provider/subsidiary delivering the last mile means one hand is talking to the other, and you're going to get good service and reliability as a result? And "big names" never have any peering-related spats and always deliver the best possible end-user experience, right? :-) (Some good points further on, though important we don't lead the OP down the wrong path or with a false sense of security there...) -a
On Sun, 08 Nov 2009 08:23:41 MST, Blake Pfankuch said:
I wouldn't sway from the big names for your primary connections either.
This is, of course, dependent on the OP's location and budget. I know when we were getting our NLR connection set up, there was a fair amount of "You want 40G worth of DWDM *where*?" involved, and the resulting topology was... complicated. At least at one time, there were places where our provider was running our link across lambdas of a subsidiary of ours, which are going across physical fiber owned by the provider... turtles all the way down. ;)
adel@baklawasecrets.com wrote:
HI,
Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are:
I wouldnt jump to any conclusions that everything will work properly if you are terminating multiple connections directly on the SSG, what with egress likely being different than the ingress, even if you are using the same IP range (BGP) on all the links. You could really be asking for trouble if you are planning on using a different ISP provided IP range on each connection for each purpose. Front it all with routers that can policy route, whether or not you also use BGP. Joe
adel@baklawasecrets.com wrote:
HI,
I was recently brought onto a project where some failover is desired, but I think that the number of connections provisioned is excessive. Also hoping to get some guidance with regards to how well I can get the failover to actually work. So currently 4 X 100Mb/s Internet connections have been provisioned. One is to be used for general Internet, out of the organisation, it also terminates VPNs from remote sites belonging to the organisation and some publicly accessible servers -routed DMZ and translated IPs. Second Internet connection to be used for a separate system which has a site-to-site VPN to a third party support vendor. Internet connections 3 and 4 are currently thought of as providing backups for one and two. Both connections firewalled by a Juniper SSG of some description.
Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are:
- Should I make sure that the backup Internet connection is from a separate provider?
- How can I acheive a failover which doesn't require me to change all the remote VPN endpoints in case of a failover? Its possible to configure failover VPNs on the Junipers, which should take care of this, but how do I take care of the DMZ hosts and external translation?
- In fact I think I'm asking what are my options with regard to failover between one Internet connection and the other?
Forget all of that and just multihome to two separate providers with BGP. Also make sure that of the providers you choose that one is not a customer of the other. Instant, painless redundancy. Having multiple circuits to one provider *will not* back anything up if that provider has an outage as they are %99.999 likely to be part of the same larger circuit and certainly share the same infrastructure at the provider.
I'm hoping to figure out whether adding an extra Internet connection actually gives us that much, in fact whether it justifies the complexity and spend.
Only if you calculate the cost (money, time, angry customers, etc.) of an outage to be greater than the cost of additional connectivity.
Many Thanks for your comments.
~Seth
Seth Mattinen [sethm@rollernet.us] said:
Forget all of that and just multihome to two separate providers with BGP --Assuming that you're advertising PI space or can work around that appropriately with your providers, I agree, that's the ideal situation.
Having multiple circuits to one provider *will not* back anything up if that provider has an outage as they are %99.999 likely to be part of the same larger circuit --True - if you don't specify otherwise when you're ordering, then why would they make the effort? Comments made in some of the other responses in this thread are also valid even with a single service provider - diverse entry points into your facility, diverse upstream circuit routing, and homing to different POPs - which may mean backhauling your secondary circuit away from your local POP and taking a hit for the higher latency on that second link. The moral of this is that whether you're using one provider or more than one, state your diversity requirements clearly up front, and then stay involved and make sure that what's presented to you is _actually_ diverse (oldsflash: even the best intentioned people sometimes make mistakes, especially when there's a handoff to a different last mile provider who may not have been clear on the requirement ). Of course, all of this is potentially wasted effort if the data center you're providing connectivity for does not also maintain the same kind of diversity itself in terms of power, connectivity, architecture, etc.
and certainly share the same infrastructure at the provider. --If you enter a single provider's network at diverse points, then that local infrastructure isn't the same at least. But by the same measure, if that provider has a major BGP issue for example, then yeah - they're both screwed... in which case we loop back to the dual provider scenario you mentioned in the first place :)
Ultimately choosing the appropriate solution will boil down to the what level of service unavailability one can tolerate in the first place, and put a business value on that impact. From that one can derive technical options, then go cap in hand with a business case to the poor soul paying the bill ;-) j.
participants (7)
-
Adam Rothschild -
adel＠baklawasecrets.com -
Blake Pfankuch -
Joe Maimon -
John.Herbert＠ins.com -
Seth Mattinen -
Valdis.Kletnieks＠vt.edu