On 04 Jul 2002 11:48:47 -0700 Paul Vixie <vixie@vix.com> wrote:

mike@sentex.net (Mike Tancsa) writes:

...

... Still, I think the softest targets are the root name servers. I was glad to hear at the Toronto NANOG meeting that this was being looked into from a routing perspective. Not sure what is being done from a DoS perspective.

Now that we've seen enough years of experience from Genuity.orig, UltraDNS, Nominum, AS112, and {F,K}.root-servers.net, we're seriously talking about using anycast for the root server system. This is because a DDoS isn't just against the servers, but against the networks leading to them. Even if we provision for a trillion packets per second per root server, there is no way to get the whole Internet, which is full of Other People's Networks, provisioned at that level. Wide area anycast, dangerous though it can be, works around that.

Is this the anycast based on MSDP ? Regards Marshall Eubanks

See www.as112.net for an example of how this might work. "More later." -- Paul Vixie

On Thu, 4 Jul 2002, Marshall Eubanks wrote: > Is this the anycast based on MSDP ? Anycast, not multicast. -Bill

> But the only IPv4 anycast > that I know of does use MSDP : > http://www.ietf.org/internet-drafts/draft-ietf-mboned-anycast-rp-08.txt > Is there a different proposal ? What's the RFC / I-D name ? You seem to be confusing anycast with something complicated. It's not a protocol, it's a method of assigning and routing addresses. -Bill

Doesnt announcing the same routing prefix into BGP from multiple locations do the same thing without needing a new range or enhancement in IGMP etc ? We do this in IGP currently.. Steve On Fri, 5 Jul 2002, Barry Raveendran Greene wrote:

FYI - for those scratching their heads on "anycast" .....

I just pushed out a paper on anycast by Chris Metz. Good foundation material.

http://www.cisco.com/public/cons/isp/essentials/ip-anycast-cmetz-03.pdf

...

-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Bill Woodcock Sent: Friday, July 05, 2002 4:56 AM To: Marshall Eubanks Cc: nanog@merit.edu Subject: Re: Internet vulnerabilities

> But the only IPv4 anycast > that I know of does use MSDP : > http://www.ietf.org/internet-drafts/draft-ietf-mboned-anycast-rp-08.txt > Is there a different proposal ? What's the RFC / I-D name ?

You seem to be confusing anycast with something complicated. It's not a protocol, it's a method of assigning and routing addresses.

-Bill

Hi Marshall If a route isnt withdrawn when the end network/device fails then no system will fix that. Presumably anycast wouldnt enable load balancing anyway as BGP only installs a single route? Or are you thinking both of these would be solved with a BGP enhancement? Dont understand the multiple anycast comment, do you mean as it stands now? If so it works fine if you inject the same route into an IGP providing you ensure theres no IGP load balancing if you intend on doing TCP (altho most applications for this appear to be UDP single request-responses) Steve On Fri, 5 Jul 2002, Marshall Eubanks wrote:

On Fri, 5 Jul 2002 13:36:49 +0100 (BST) "Stephen J. Wilcox" <steve@opaltelecom.co.uk> wrote:

...

Doesnt announcing the same routing prefix into BGP from multiple locations do the same thing without needing a new range or enhancement in IGMP etc ?

We do this in IGP currently..

Steve

As I see it, the problems with doing this in BGP are

- it's static - no failover. If AS 701 and AS 1239 are both announcing a route to foo, and your preferred route is "through" AS701, and the AS701 foo goes down, then you do not automatically switch over to the AS1239 foo, even if you could reach it.

- there is no way to have multiple anycast addresses within an AS

- load balancing is tough

These may all be solved, though... it's hard to tell without a protocol description.

Regards Marshall Eubanks

...

On Fri, 5 Jul 2002, Barry Raveendran Greene wrote:

...

FYI - for those scratching their heads on "anycast" .....

I just pushed out a paper on anycast by Chris Metz. Good foundation material.

http://www.cisco.com/public/cons/isp/essentials/ip-anycast-cmetz-03.pdf

...

-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Bill Woodcock Sent: Friday, July 05, 2002 4:56 AM To: Marshall Eubanks Cc: nanog@merit.edu Subject: Re: Internet vulnerabilities

> But the only IPv4 anycast > that I know of does use MSDP : > http://www.ietf.org/internet-drafts/draft-ietf-mboned-anycast-rp-08.txt > Is there a different proposal ? What's the RFC / I-D name ?

You seem to be confusing anycast with something complicated. It's not a protocol, it's a method of assigning and routing addresses.

-Bill

ME> Date: Fri, 05 Jul 2002 09:05:44 -0400 ME> From: Marshall Eubanks ME> - it's static - no failover. If AS 701 and AS 1239 are both ME> announcing a route to foo, and your preferred route is ME> "through" AS701, and the AS701 foo goes down, then you do not ME> automatically switch over to the AS1239 foo, even if you ME> could reach it. ??? ME> - there is no way to have multiple anycast addresses within ME> an AS ??? ME> - load balancing is tough Just as tough as load-balancing over different upstreams in a multihomed network. That's all anycast really is: multihoming with the added twist of using multiple, separate systems instead of one. Each system has a unique, non-anycast IP address bound as the primary IP, allowing communication between the disjoint parts. Secondary IP(s) live(s) in the anycast range, and is/are routed appropriately. You can bind the appropriate 192.175.48/24 addresses to your NSen and run an authoritative copy of the root TLD. IIRC, Paul even mentioned doing this a few weeks ago... I believe the thread was on dynamic DNS updates and Win2000's broken implementation. Think of anycast as DDoS in reverse: Instead of distributed traffic sources, one has distributed traffic sinks. Hence the attractiveness in surviving DDos attacks. Eddy -- Brotsman & Dreger, Inc. - EverQuick Internet Division Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 (785) 865-5885 Lawrence and [inter]national Phone: +1 (316) 794-8922 Wichita ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.

Dear Rodney; Thanks for the info. Rodney Joffe wrote:

Marshall,

First, I hope you don't mind that I cut all the additional cc's. I don't think any of the folks really needed extra copies ;-)

Now...

Marshall Eubanks wrote:

...

On Fri, 5 Jul 2002 13:36:49 +0100 (BST) "Stephen J. Wilcox" <steve@opaltelecom.co.uk> wrote:

...

Doesnt announcing the same routing prefix into BGP from multiple locations do the same thing without needing a new range or enhancement in IGMP etc ?

We do this in IGP currently..

Well, this doesn't need anything to change with normal BGP. It really has very little to do with IGMP per se. The anycast routing prefix is announced into many different networks, and as the end user, you will see many paths, hopefully. If you only see one because of your IBGP, then that's the path you'll take. If you see many, you'll take the one that *your* ospf or isis setup prefers.

...

As I see it, the problems with doing this in BGP are

- it's static - no failover. If AS 701 and AS 1239 are both announcing a route to foo, and your preferred route is "through" AS701, and the AS701 foo goes down, then you do not automatically switch over to the AS1239 foo, even if you could reach it.

No. Its not static. You may have misunderstood. Anycast is not just multiple routes. It is also multiple machines in different places. So

That's the point :)

there is really no single "foo". There are many "foos". Each one may have more than one connection to the net. The announcements appear behind many ASs. When your system sees many paths to "foo", it does not know that in fact, each path goes to a different machine entirely, on a different network even, in a different physical location. There's another part that goes with anycast use, and dns; when any particular foo goes down, or fails in any way, not just by physically failing, it stops announcing itself (the router or routing software it uses withdraws the route) and it is no longer one of the paths your network will see. So if you were seeing it from 701, and 1239, and if anycast is

Let's go through this a little. Let's say that you and I are running the foo service in anycast. You announce the foo IP address (say in a /24) behind your AS, I announce the same /24 behind my AS. Now, if my foo server goes down, how do my routers know to withdraw the announcements ? If they don't, why wouldn't people "closer" to me still try and get the foo service from me, alas, without success. That's what I meant. Or, are you saying that an anycast host has to be a router running BGP ? So if it goes down, so would the service and the announcements? This works for DNS, but not for the things I would like to anycast.

truly being used, you'll actually see the route being withdrawn from the network(s) that has the foo that went bad. Unless, of course, there are multiple foos in that network. In which case you will see no change and you will still get to foo via the original route you preferred, just not the foo you had used previously. And it makes no difference to you, because in almost all of the cases, the query is answered in a single packet, so persistence is irrelevant.

...

- there is no way to have multiple anycast addresses within an AS

Huh? What in the world do you mean here?

Sorry, too early in the AM. Withdrawn.

...

- load balancing is tough

Yes, which is why the load balancing services in the world are sold at a premium. And it is not all that tough. ;-) With anycast, it is not tough, at all, until you have to deal with the subject that brought this thread up, ddos attacks. In which case it need real engineering.

...

These may all be solved, though... it's hard to tell without a protocol description.

If you're talking about anycast and the way we're all using is in the dns, there is no protocol as such. It uses existing mechanisms. All the same protocols. You're currently making use of dns that uses anycast, but you didn't have to modify anything, or download any new software, or make any changes, did you?

Nope. Thanks for the info. Marshall

...

...

...

...

...

But the only IPv4 anycast that I know of does use MSDP : You seem to be confusing anycast with something complicated. It's not a protocol, it's a method of assigning and routing addresses.

-Bill

You really do seem to be fixated on multicast still. anycast /= multicast.

HTH

-- Regards Marshall Eubanks This e-mail may contain confidential and proprietary information of Multicast Technologies, Inc, subject to Non-Disclosure Agreements T.M. Eubanks Multicast Technologies, Inc 10301 Democracy Lane, Suite 410 Fairfax, Virginia 22030 Phone : 703-293-9624 Fax : 703-293-9609 e-mail : tme@multicasttech.com http://www.multicasttech.com Test your network for multicast : http://www.multicasttech.com/mt/ Status of Multicast on the Web : http://www.multicasttech.com/status/index.html

> Or, are you saying that an anycast host has to be a router running BGP ? No, typically they run OSPF. > So if it goes down, so would the service and the announcements? Correct. If a device wants to witdraw itself from a service pool, it withdraws the host route associated with that service. > This works for DNS, but not for the things I would like to anycast. Mmm, like what? This is all ancient history at this point... It seems unlikely that anyone would discover something that didn't work at this late date. -Bill

In the referenced message, Rodney Joffe said:

Just as a guess, Marshall is probably thinking of using anycast for something other than DNS, like http, or ftp, or telnet. And he's wondering about state ;-)

If you don't use per-packet type load-sharing, but something like per-flow, or per-src/dst-hash, then you can use anycast for protocols which require state (including tcp or other connection-oriented protocols, for that matter). Worst-case when the server you are communicating with fails, the connection is broken, much as it would had anycast not been in use. A resilient protocol/application could then reestablish the connection somewhat quickly. Actually, I guess the worst-case would be when the original machine came back online, the connection would once again be broken, but not due to a failure. This would probably require people to set things up (if they are smart) not to _automatically_ return to service. That way, the previously failed box could be re-inserted during a maintenance, for least detrimental effect. This type of thing would be most useful for read operations, and becomes somewhat of a problem for write operations, due to synchronization. Therefore, it might be useful for a web server, but not so for ftp (unless, you had a separate master machine for folks/customers to do put operations, which is then synchonized and pushed to the anycast read boxes). Of course, this would only be relevant/useful within your own enterprise, since others may be using per-packet load-sharing (which comes with a host of problems as it is, like the high potential for out of order packets.) Whether doing this is practical is an exercise for the reader, but it certainly is possible. Presently, I think most folks doing anycast take the easy stance and just use it for non-stateful connectionless protocols (esp. dns).

On Sun, 7 Jul 2002, Gerardo A. Gregory wrote:

Can someone from WorldComm please verify a fiber cut that happened today at around 11:30 am (Central). I have bveen informed that a fiber cut in Illinois (or Indiana) has been in effect (until just a few minutes) for all of the afternoon and most of the evening.

Worldcom is reporting a problems near Chicago. Earthlink is reporting problems affecting its customers in Indiana, Illinois, Iowa, Michigan, Wisconsin and Ohio. http://help.mindspring.com/netstatus/ http://www.noc.uu.net/ Cable & Wireless is showing delays out of Cleveland, Ohio http://sla.cw.net/ AT&T and Sprint aren't reporting any problems. http://ipnetwork.bgtmo.ip.att.net/index.html http://www.sprint.net/ MFN's and PSI's network status pages have stopped working for me, so I don't know if they are having problems. http://www.above.net/html/techlog.txt http://www.psi.net/cgi-bin/netstatus.pl5

MFNs status page is: http://www.mfn.com/network/ip_networkstatus.shtm#sjc Jane Sean Donelan wrote:

On Sun, 7 Jul 2002, Gerardo A. Gregory wrote:

...

Can someone from WorldComm please verify a fiber cut that happened today at around 11:30 am (Central). I have bveen informed that a fiber cut in Illinois (or Indiana) has been in effect (until just a few minutes) for all of the afternoon and most of the evening.

Worldcom is reporting a problems near Chicago. Earthlink is reporting problems affecting its customers in Indiana, Illinois, Iowa, Michigan, Wisconsin and Ohio.

http://help.mindspring.com/netstatus/ http://www.noc.uu.net/

Cable & Wireless is showing delays out of Cleveland, Ohio

http://sla.cw.net/

AT&T and Sprint aren't reporting any problems.

http://ipnetwork.bgtmo.ip.att.net/index.html http://www.sprint.net/

MFN's and PSI's network status pages have stopped working for me, so I don't know if they are having problems.

http://www.above.net/html/techlog.txt http://www.psi.net/cgi-bin/netstatus.pl5

On Sun, 7 Jul 2002, Richard A Steenbergen wrote: > I think the problem they are refering to is what happens if your routing > topology changes (or worse, flaps). A stateful connection (like TCP) which > would have stayed up during a routing change could potentially be shifted > to a different server which obviously wouldn't know the other one's state. Yes. As I said in a previous message in this thread, that's a common objection brought up by people who've never run an anycast network and are trying to think of reasons why it might be problematic. But since in the real world that appears to happen two orders of magnitude less frequently than connection failures due to loss of connectivity, _when you take no steps to prevent it_, and the prevention is both trivial and necessary with HTTP, which is the protocol most commonly anycasted, it's not an issue at all. -Bill

I don't understand many of the cyber-scare articles. If I was cynical, and I thought we had a clever government, I would say it was all a diversionary tactic to distract attackers from the more vulnerable infrastructures. Disrupting the Internet is a matter of scale and time. It is fairly trivial to disrupt large portions of the Internet for short periods of time. You don't need to be a hacker to do that. Most of the senior network engineers on this list have done it by accident or unplanned maintenance. Just look at the Internet during major maintenance windows to see what can be done. With BGP dampening, its possible to DOS yourself. On the other hand, disrupting a large portion of the Internet for more than a few (e.g. 6) hours is slightly more difficult. Most of that time is consumed by response team activation. Nevertheless there are a few attacks which could take longer than 24 hours to recover. The loyal order of disgruntled, unemployed network engineers met at a bar at a previous NANOG and come up with several interesting, yet practical attacks. I'm not talking about permanent events, such as a massive solar flare ending all life on earth. What's nice about the Internet is it is a relatively loosly-coupled. Which means many different people can work on fixing their part of the Internet without needing too much coordination. The Internet doesn't have the equivalent of a LERG, so you can connect your piece of the net back into whatever other pieces of the Net still working without centralized coordination. Highly visible things like root name servers are under attack a lot, but for the most part the net stumbles throught it. Highly visibile things tend to also be highly protected. But why bother? There are other infrastructures which are more vulnerable to attack than the Internet, and more likely to get significantly more news coverage than any attack on the Internet could achieve.

On Fri, 5 Jul 2002, Barry Raveendran Greene wrote: > http://www.cisco.com/public/cons/isp/essentials/ip-anycast-cmetz-03.pdf Yes, this document correctly described IPv4 anycast. It somewhat overstates the severity of the issue with TCP and the dynamicism of the underlying network topology... Although that's often brought up by people who've never used anycast before and think they're being clever, in actual deployed networks it's accounted for less than 0.001% of total traffic volume, or far less than is generally lost across the network anyway. It also describes the group-membership-announcement issue, which is basically a non-issue now that all the host vendors can support OSPF, which neatly fills the need. -Bill