I'm looking for some advice on IGP tricks such that we can give out the same IPs for a specific service (say...DNS) to all of our customers despite geography and have my IGP route those packets to a regional server, thus accomplishing some inherent level of load balancing, and even eventually returning data based on client geography. I understand that nanog may not be the appropriate forum for this discussion, but I thought perhaps that someone could point me to a better place to discuss this? Or perhaps those of you with experience in the area could email me off-list? -- nicholas harteau nrh@ikami.com
"nicholas" == nicholas harteau <nrh@ikami.com> writes:
nicholas> I'm looking for some advice on IGP tricks such that we nicholas> can give out the same IPs for a specific service nicholas> (say...DNS) to all of our customers despite geography nicholas> and have my IGP route those packets to a regional nicholas> server, thus accomplishing some inherent level of load nicholas> balancing, and even eventually returning data based on nicholas> client geography. Assign a unique (or several uniqe...) address that the customers will use, say 1.1.1.1. Configure this address onto the loopback interface of each of the servers as a secondary address. Have the host run a routing protocol (OSPF lends itself to this nicely since the area construct us usefull) and have it inject 1.1.1.1/32 into the IGP. Then arrange so that the IGP will direct all traffic to the server within the the same area as the source of the traffic. There are several ways to do this: - filter routing updates for 1.1.1.1/32 at the area border routers. This is not so good as it would probably be good to redirect traffic to a server in another area should the local one go down for some reason. - with OSPF, configure the cost parameter on inter area links to a value larger than that associated with any intra area path -- this will make sure that a routes from one area are always preferred over inter area routes. In fact I'm pretty sure OSPF does this automatially anyways, but it's nice to have things explicitly configured like this -- your "serving" area needn't necessarily correspond with an OSPF area in this case (a single OSPF area may be divided into two "serving areas" , for example). You can achieve load balancing in a more fault tolerant manner in this way too -- say you want to load balance across two web servers, but you want one to take over all of the load should the other fail. Configure the dns to round robin on 1.1.1.2 and 1.1.1.3, say and configure both of these addresses as aliases on the loopback interfaces of both servers. The first server advertises the first address with a lower cost than the second and the second does the reverse. The difference in cost should be greater than the cost associated with the best path between the servers. What is the general feeling about running routing protocols on web/dns/mail servers? Cheers, -w -- Will Waites \________ ww@shadowfax.styx.org\____________________________ Idiosyntactix Ministry of Research and Development\
ww@shadowfax.styx.org: Tuesday, May 16, 2000 10:34 PM
What is the general feeling about running routing protocols on web/dns/mail servers?
Technically, not a problem. However, there is a school of thought that thinks that to be a bad policy. That routing functions should be on appliance-level systems, like routers. There is also some merit in that appliances are more reliable, mainly because nothing *else* can cause an operational interrupt. Unix systems are *real* good about process control. but, there are still some things that makes it advisable to reboot a system, at times. If that system is ALSO a critical router then the entire net is down until the reboot is complete. It is generally not considered a stable state of affairs. Having been in exactly that situation, I can tell you that it isn't fun. Eventually, the pain aversion therapy works and your behavior is modified accordingly.
On Tue, 16 May 2000 23:22:09 PDT, "Roeland M.J. Meyer" said:
What is the general feeling about running routing protocols on web/dns/mail servers?
Technically, not a problem. However, there is a school of thought that thinks that to be a bad policy. That routing functions should be on appliance-level systems, like routers. There is also some merit in that appliances are more reliable, mainly because nothing *else* can cause an operational interrupt. Unix systems are *real* good about process control. but, there are still some things that makes it advisable to reboot a system, at times. If that system is ALSO a critical router then the entire net is down until the reboot is complete. It is generally not
How about the case of a system with several network interfaces on different subnets, and using a routing protocol to better pick which interface to send a connection out on? This is probably more applicable to mail servers - web and dns servers don't have as much latitude as they sort of have to answer on the interface they were contacted on... Valdis Kletnieks Operating Systems Analyst Virginia Tech
The scenario the original poster described, wanting to have regionally based caching DNS servers for clients is a fine example of when in fact it is a good and plausible idea to run a routing protocol on a Unix machine. I've run ospf on the DNS servers to redistribute the same /32 loopback address at different pops on local machines. I know of at least one large provider who uses BGP to achieve the same thing...It works well because bind tends to be far more stable and robust than the routing protocol program. I would recommend BGP since you can filter everything to the (DNS) server and only announce the /32... I can imagine doing the same thing for smtp relay boxes, never tried it though. Adi In message <00e201bfbfc8$3b7597f0$eaaf6cc7@PEREGRIN>, "Roeland M.J. Meyer" writes:
ww@shadowfax.styx.org: Tuesday, May 16, 2000 10:34 PM
What is the general feeling about running routing protocols on web/dns/mail servers?
Roeland M.J. Meyer wrote:
ww@shadowfax.styx.org: Tuesday, May 16, 2000 10:34 PM
What is the general feeling about running routing protocols on web/dns/mail servers?
Technically, not a problem. However, there is a school of thought that thinks that to be a bad policy. That routing functions should be on appliance-level systems, like routers.
Has anyone given any thought to using some mechanism like 'redistribute connected' in conjunction with IPIP tunnels to hosts? When host goes poof, tunnel interface goes down, route goes away. No complicated routing processes are involved. -- nicholas harteau nrh@ikami.com
On Tue, 16 May 2000, Roeland M.J. Meyer wrote:
ww@shadowfax.styx.org: Tuesday, May 16, 2000 10:34 PM
What is the general feeling about running routing protocols on web/dns/mail servers?
Technically, not a problem. However, there is a school of thought that thinks that to be a bad policy. That routing functions should be on appliance-level systems, like routers. There is also some merit in that appliances are more reliable, mainly because nothing *else* can cause an ... reboot a system, at times. If that system is ALSO a critical router then the entire net is down until the reboot is complete. It is generally not
Running a routing protocol on a unix box doesn't mean you're using it as a router. Perhaps he just wants OSPF on a few servers so they can send their packets more efficiently. Consider a case where you have a few access servers and unix servers on the same switch and a router connecting that POP to your backbone. Having a routing protocol on those unix boxes means they can send packets directly to the appropriate access server (or the router) rather than everything to the router, just to have it spit the packets back out headed for an access server on that segment. ---------------------------------------------------------------------- Jon Lewis *jlewis@lewis.org*| I route System Administrator | therefore you are Atlantic Net | _________http://www.lewis.org/~jlewis/pgp for PGP public key__________
On Wed, May 17, 2000 at 10:14:58PM -0400, jlewis@lewis.org mailed:
Running a routing protocol on a unix box doesn't mean you're using it as a router. Perhaps he just wants OSPF on a few servers so they can send their packets more efficiently. Consider a case where you have a few access servers and unix servers on the same switch and a router connecting that POP to your backbone. Having a routing protocol on those unix boxes means they can send packets directly to the appropriate access server (or the router) rather than everything to the router, just to have it spit the packets back out headed for an access server on that segment.
Pardon my ignorance here, but wont ICMP redirects take care of this situation already? -- Bryan C. Andregg * <bandregg@redhat.com> * Red Hat, Inc. gpg 1024D/19893A19 A8DA 869A 037A C6B5 BF07 AB61 E406 414B 1989 3A19 pgp2 1024/625FA2C5 F5 F3 DC 2E 8E AF 26 B0 2C 31 78 C2 6C FB 02 77 pgp5 1024/0x46E7A8A2 46EB 61B1 71BD 2960 723C 38B6 21E4 23CC 46E7 A8A2
On Thu, 18 May 2000, Bryan C. Andregg wrote:
their packets more efficiently. Consider a case where you have a few access servers and unix servers on the same switch and a router connecting that POP to your backbone. Having a routing protocol on those unix boxes means they can send packets directly to the appropriate access server (or the router) rather than everything to the router, just to have it spit the packets back out headed for an access server on that segment.
Pardon my ignorance here, but wont ICMP redirects take care of this situation already?
Some platforms don't deal well relying on redirects. The first time they try to reach a destination, a redirect causes them to insert a host route in their routing table. If that destination moves (say a static IP connecting to whatever access server they happen to hit), some OS's will refuse to accept further redirects pointing the destination toward a different gateway. ---------------------------------------------------------------------- Jon Lewis *jlewis@lewis.org*| I route System Administrator | therefore you are Atlantic Net | _________http://www.lewis.org/~jlewis/pgp for PGP public key__________
On Thu, 18 May 2000 10:57:31 EDT, Jon Lewis said:
On Thu, 18 May 2000, Bryan C. Andregg wrote:
Pardon my ignorance here, but wont ICMP redirects take care of this situation already?
Some platforms don't deal well relying on redirects. The first time they try to reach a destination, a redirect causes them to insert a host route in their routing table. If that destination moves (say a static IP connecting to whatever access server they happen to hit), some OS's will refuse to accept further redirects pointing the destination toward a different gateway.
In addition, there's the routing table size issue - I had an NTP server that erroneously got Path MTU Discovery turned on. Debugging routing table problems is.. um... interesting... when you have 4,000+ static host routes (nothing like watching the DNS burp because you said 'netstat -r' rather than '-r -n' ;) At least the PMTU discovery support I've seen expires those routes after a while - often ICMP redirects live forever, resulting in a long list of host routes all pointing at the default router.... There's also the issue that most routing protocols can be configured to only accept updates from a given access list (which should probably be peer routers) - ICMP redirects can come from anybody, exposing you to a man-in-the-middle attack. (Yes, I know it's *NOT* complete protection, but disabling acceptance of ICMP redirects closes at least SOME issues). -- Valdis Kletnieks Operating Systems Analyst Virginia Tech
On Thu, 18 May 2000 Valdis.Kletnieks@vt.edu wrote:
In addition, there's the routing table size issue - I had an NTP server that erroneously got Path MTU Discovery turned on. Debugging routing table problems is.. um... interesting... when you have 4,000+ static host routes (nothing like watching the DNS burp because you said 'netstat -r' rather than '-r -n' ;)
What platform adds static routes in response to PMTU discovery? I've never seen that before. What purpose does it serve to add statics? Some kind of caching of MTU per destination? Brandon Ross 404-522-5400 VP Engineering, NetRail http://www.netrail.net AIM: BrandonNR ICQ: 2269442 Read RFC 2644! Stop Smurf attacks! Configure your router interfaces to block directed broadcasts. See http://www.quadrunner.com/~chuegen/smurf.cgi for details.
On Thu, 18 May 2000 20:39:46 EDT, Brandon Ross <bross@netrail.net> said:
What platform adds static routes in response to PMTU discovery? I've never seen that before. What purpose does it serve to add statics? Some kind of caching of MTU per destination?
Exactly. AIX 4.3 uses a host route to cache the discovered PMTU. I got surprised in 4.3.3 when PMTU discovery became the default... What systems DONT use a host route to cache the PMTU, and what do they use instead to store the values? -- Valdis Kletnieks Operating Systems Analyst Virginia Tech
Sez "Brandon Ross" <bross@netrail.net>
What platform adds static routes in response to PMTU discovery? I've never seen that before. What purpose does it serve to add statics? Some kind of caching of MTU per destination?
This is actually very common in "off-brand" (ie. anything besides Linux, *BSD, and Solaris) IP stacks. The current shipping versions of many OSes do some very odd things: cache a static route upon performing PMTUD, cache static routes to one of multiple default gateways, refuse to honor gratuitous ARPs, refuse to time out ARP entries, cache ARP entries with mcast MACs, cache ICMP redirects classfully, etc. What's worse is these "caches" tend to be permanent, usually not even allowing manual overrides (other than a reboot). S | | Stephen Sprunk, K5SSS, CCIE #3723 :|: :|: Network Consulting Engineer, NSA :|||: :|||: 14875 Landmark Blvd #400; Dallas, TX .:|||||||:..:|||||||:. Email: ssprunk@cisco.com
This is actually very common in "off-brand" (ie. anything besides Linux, *BSD, and Solaris) IP stacks. The current shipping versions of many OSes do some very odd things: cache a static route upon performing PMTUD, cache static routes to one of multiple default gateways, refuse to honor gratuitous ARPs, refuse to time out ARP entries, cache ARP entries with mcast MACs, cache ICMP redirects classfully, etc.
What's worse is these "caches" tend to be permanent, usually not even allowing manual overrides (other than a reboot).
Yah, BSDI added this to version 3 and I bought the source license just so I could hack it out, it really broke things on a box running any kind of routeing protocol. Regards, Neil.
From: Bryan C. Andregg [mailto:bandregg@redhat.com] Sent: Thursday, May 18, 2000 8:15 AM
Running a routing protocol on a unix box doesn't mean you're using it as a router. Perhaps he just wants OSPF on a few servers so
their packets more efficiently. Consider a case where you have a few access servers and unix servers on the same switch and a router connecting that POP to your backbone. Having a routing protocol on
On Wed, May 17, 2000 at 10:14:58PM -0400, jlewis@lewis.org mailed: they can send those unix boxes
means they can send packets directly to the appropriate access server (or the router) rather than everything to the router, just to have it spit the packets back out headed for an access server on that segment.
Pardon my ignorance here, but wont ICMP redirects take care of this situation already?
ICMP redirects create a potential security vulnerability, for man-in-the-middle attacks. MHSC.NET doesn't allow them. Not host, at MHSC.NET, will respond to them (in theory <g>).
-----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160
"jlewis" == jlewis <jlewis@lewis.org> writes:
jlewis> On Tue, 16 May 2000, Roeland M.J. Meyer wrote: >> > ww@shadowfax.styx.org: Tuesday, May 16, 2000 10:34 PM >> >> > What is the general feeling about running routing protocols >> on > web/dns/mail servers? [...] jlewis> Running a routing protocol on a unix box doesn't mean jlewis> you're using it as a router. Perhaps he just wants OSPF jlewis> on a few servers so they can send their packets more jlewis> efficiently. Consider a case where you have a few access jlewis> servers and unix servers on the same switch and a router jlewis> connecting that POP to your backbone. Having a routing jlewis> protocol on those unix boxes means they can send packets jlewis> directly to the appropriate access server (or the router) jlewis> rather than everything to the router, just to have it spit jlewis> the packets back out headed for an access server on that jlewis> segment. Correct. It also means that they can inject some routing information - -- i.e. that the ip address configured onto their loopback interface is reachable through the ip address on their ethernet interface subject to a certain cost. The intent is to provide an efficient failover mechanism (one host crashes, a backup host is automatically used since they both have the same address on their loopbacks, but the backup is advertising it with a higher cost), as well as the ability to easily relocate services to different machines and load balance in a way that doesn't break if one host goes away (à la DNS round-robin) or cost a fortune (à la local-director). - -w - -- Will Waites \________ ww@shadowfax.styx.org\____________________________ Idiosyntactix Ministry of Research and Development\ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.1 (OpenBSD) Comment: Processed by Mailcrypt 3.5.5 and Gnu Privacy Guard <http://www.gnupg.org/> iQEXAwUBOSQl5g4cK24IcAwYFAOpywP/SjUKo09WCeBW0Wjq+6znsYJ6Eh7+Tirh Ik7To9fFie00kmpc4VSYitKFpQEBNX7qN9y0vtdq0B5hzzUbUeDqip+xM3gdPO5G VV2wSdob84S59eruRl6zqU7+WrcCqW5kq5OG9e3U1sG/5b38jcJbFB8KgvXwthgO I8aZhsEK6YkD/RRZNEtkLbwHvBdMXI4SpxVIY5rjWCIbmgriGHzenJ9NVOXufgqv KeJ8617eFcgSOCZUMtopoT2Q+Iorzm1nVGLHNdI8F/vTutV0RDgSGKjhOqx9yK69 aoVLsuqzr8qVQolzNh3ZPOZh2a2YkvpgtMDVK5fsSxrqNc37/4C4aYBl =6Zzr -----END PGP SIGNATURE-----
participants (11)
-
Brandon Ross -
Bryan C. Andregg -
jlewis＠lewis.org -
Neil J. McRae -
nicholas harteau -
R.P. Aditya -
Roeland M.J. Meyer -
Roeland Meyer (E-mail)
-
Stephen Sprunk -
Valdis.Kletnieks＠vt.edu -
ww＠shadowfax.styx.org