...

Did Youtube not pay their domain bill? ^^

% dig @a.gtld-servers.net. ns yotube.com ^ Still early, Steinar?

You're right, clearly insufficient amounts of coffee here... Steinar Haug, Nethelp consulting, sthaug@nethelp.no

Still down either way... =================================================== ; <<>> DiG 9.2.4 <<>> dns1.sjl.youtube.com @a.gtld-servers.net ; (2 servers found) ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22563 ;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2 ;; QUESTION SECTION: ;dns1.sjl.youtube.com. IN A ;; ADDITIONAL SECTION: dns1.sjl.youtube.com. 172800 IN A 208.65.152.201 dns2.sjl.youtube.com. 172800 IN A 208.65.152.137 =================================================== =================================================== ; <<>> DiG 9.2.4 <<>> www.youtube.com @208.65.152.201 ; (1 server found) ;; global options: printcmd ;; connection timed out; no servers could be reached ; <<>> DiG 9.2.4 <<>> www.youtube.com @208.65.152.137 ; (1 server found) ;; global options: printcmd ;; connection timed out; no servers could be reached =================================================== Kipkemoi Kibiego wrote:

yotube.com != youtube.com

sthaug@nethelp.no wrote:

...

Did Youtube not pay their domain bill?

% dig @a.gtld-servers.net. ns yotube.com

yotube.com. 2D IN NS ns1.parked.com. yotube.com. 2D IN NS ns2.parked.com.

Steinar Haug, Nethelp consulting, sthaug@nethelp.no

_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog

---------------------------------------------- This message has been scanned for viruses and dangerous content by Jambo MailScanner, and is believed to be clean. --------------------------------------------- "easy access to the world"

_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog

Maybe that block is anycasted? On 5/3/08 9:45 AM, "Randy Bush" <randy@psg.com> wrote:

...

dns1.sjl.youtube.com. 172800 IN A 208.65.152.201 dns2.sjl.youtube.com. 172800 IN A 208.65.152.137

2182 lesson again, probably. after all, microsoft/hotmail/... being borked for a day can't happen to me!

randy

_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog

If they were anycasted, shouldn't they be reachable from _somewhere_ ? Those servers are dead from the 4 corners of the US that I have resources to use for testing. Brant I. Stevens wrote:

Maybe that block is anycasted?

I received a report from a user at 9:46 EDT that they couldn't access youtube, so at least some users were affected. Regards Marshall On May 3, 2008, at 10:25 AM, David Coulson wrote:

Depends - It doesn't help if the DNS server is dead, but the front-end is still advertising the routes.

It came back to life for me a few moments ago (via Cogent) and it looks like the routing did not change (there is a bunch of 10/8 stuff in the traceroute).

Eric Spaeth wrote:

...

If they were anycasted, shouldn't they be reachable from _somewhere_ ? Those servers are dead from the 4 corners of the US that I have resources to use for testing.

_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog

David Coulson wrote:

Depends - It doesn't help if the DNS server is dead, but the front-end is still advertising the routes.

Possibly a good argument for allowing the DNS servers to originate the routes for them...? I've seen configuration where the routes were injected based on link state via crossover cable, so at least if the whole machine pukes the route is dropped. But if the resolver or OS itself is just hung then yeah... Mike

On Sat, 3 May 2008, Mike Lewinski wrote:

David Coulson wrote:

...

Depends - It doesn't help if the DNS server is dead, but the front-end is still advertising the routes.

Possibly a good argument for allowing the DNS servers to originate the routes for them...? I've seen configuration where the routes were

Running Quagga or something similar on the anycasted server to announce the routes is the standard way of setting up anycast. That way, if the server fails completely, the route goes away. A common improvement on that is to run a script on the server that checks to make sure the name server process is running and responding correctly, and kills BGP if it isn't. That covers cases where named has problems that don't take down the whole server. The first one depends heavily on the server, if it's going to fail, failing the right way. If it loses power or stops all processes, the route goes away and traffic gets redirected elsewhere. If named dies or stops responding, you're stuck. The second method solves a lot of that sort of problems, but there are still conceivable situations where the server would get into a state of partial failure and be unable to withdraw the route. Still, that's probably the best option. Another approach would be to run the monitoring script and BGP process on a separate host that would presumably be healthy even when the name server host is in trouble. That approach has issues too, in that you lose the guarantee that the route will go away if the name server box is turned off. The right solution is to design the anycast servers to be as sure as possible that the route will go away when you want it gone, but to have multiple non-interdependent anycast clouds in the NS records for each zone. If the local node in one cloud does fail improperly, something will still be responding on the other cloud's IP address. Note that any of these failure scenarios is still preferable to what you get with unicast servers. With unicast, if the server has trouble, the route always stays up, and the the traffic always ends up in a black hole. -Steve

note that joe's example brings up the interface before starting the name server program, and bringing it down if the name server program exits. this presumes that the name server will start very quickly, and that while running, it is healthy. since i've seen name server programs be unhealthy while running, and/or take a long time to start, i'm now considering an outboard shell script that runs some kind of DNS query and decides, based on the result, whether to bring the dedicated loopback interface up or down.

All deference to this model, we've all seen these kinds of problems with name servers. We *can* be certain that bringing a loopback interface up or down takes almost no time (with the implied effect to a speaker like Quagga). There is *no* reason with a sufficiently deep name server depth (depends on your load) that your monitoring script should *need* to hurry to test this condition. Every 5-10 or even 15 minutes to see if its eligible to bring up, more frequently to see if its eligible to take down. This also reduces oscillation. This means, bring up/kill off your name server in one cronjob (automatically taking the interface down at the end or after a kill), and monitor/talk to the interface in another (up function and sometimes the down). You'll be much happier. Deepak Jain AiNET

scg@gibbard.org (Steve Gibbard) writes:

...

... if each anycast cluster is really several servers, each using OSPF ECMP, then you can lose a server and still have that cluster advertising the route upstream, and only when you lose all servers in a cluster will that route be withdrawn.

This is getting into minutia, but using multipath BGP will also accomplish this without having to get the route from OSPF to BGP. This simplifies things a bit, and makes it safer to have the servers and routers under independent control.

i think the minutia is good, especially after a long weekend of layer 9 threads. my limited understanding of multipath bgp is that it's a global config knob for routers, not a per peer knob, and that it has disasterous consequences if the router is also carrying a full table and has many peers. also, in OSPF, ECMP is not optional, even though most BSD-based software routers don't implement it yet (since multipath routing is very new.) so, we have been using OSPF for this, it just works out better. i dearly do wish that something like a "service advertisement protocol" existed, that did what OSPF ECMP did, without a router operator effectively giving every customer the ability to inject other customer routes, or default routes. in that sense, i agree with your "safer... independent control" assertion.

But yes, Joe's ISC TechNote is an excellent document, and was a big help in figuring out how to set this up a few years ago.

and now for something completely different -- where in the interpipes could a document like that have been published, vs. ISC's web site? the amount of red tape and delay involved in Usenix or IETF or IEEE or ACM are vastly more than most smart ops people are willing to put in. where is the light / middle weight class, or is every organization or person who wants to publish this kind of thing going to continue to have the exclusive and bad choice of "blog it, or write an article for ;login:/ACM-Queue/Circle-ID, or write an academic paper and wait ten months"? isn't this a job for... NANOG? -- Paul Vixie

On May 5, 2008, at 12:07 PM, Paul Vixie wrote:

...

But yes, Joe's ISC TechNote is an excellent document, and was a big help in figuring out how to set this up a few years ago.

and now for something completely different -- where in the interpipes could a document like that have been published, vs. ISC's web site? the amount of red tape and delay involved in Usenix or IETF or IEEE or ACM are vastly more than most smart ops people are willing to put in. where is the light / middle weight class, or is every organization or person who wants to publish this kind of thing going to continue to have the exclusive and bad choice of "blog it, or write an article for ;login:/ACM-Queue/Circle-ID, or write an academic paper and wait ten months"? isn't this a job for... NANOG?

If you're asking seriously: arXiv.org is a pretty reasonable candidate for less-formal but more-public publication of things like Joe's TechNote. It's taken off seriously in physics, but I don't know anyone who uses it seriously for computer science stuff. Probably because our conferences have much faster turnaround than most discipline's journals do. But arXiv exists, it'll probably be around for a while, and it provides a reasonable starting point for hosting and citing the documents... -Dave

On Mon, May 5, 2008 at 10:07 AM, Paul Vixie <vixie@isc.org> wrote:

scg@gibbard.org (Steve Gibbard) writes:

...

...

... if each anycast cluster is really several servers, each using OSPF ECMP, then you can lose a server and still have that cluster advertising the route upstream, and only when you lose all servers in a cluster will that route be withdrawn.

This is getting into minutia, but using multipath BGP will also accomplish this without having to get the route from OSPF to BGP. This simplifies things a bit, and makes it safer to have the servers and routers under independent control.

i think the minutia is good, especially after a long weekend of layer 9 threads. my limited understanding of multipath bgp is that it's a global config knob for routers, not a per peer knob, and that it has disasterous consequences if the router is also carrying a full table and has many peers.

I am not sure what routers specifically are being discussed here, but in JunOS you can enable multipath on a global, group or single neighbor level, possibly eliminating your concern...

also, in OSPF, ECMP is not optional, even though most BSD-based software routers don't implement it yet (since multipath routing is very new.) so, we have been using OSPF for this, it just works out better. i dearly do wish that something like a "service advertisement protocol" existed, that did what OSPF ECMP did, without a router operator effectively giving every customer the ability to inject other customer routes, or default routes. in that sense, i agree with your "safer... independent control" assertion.

...

But yes, Joe's ISC TechNote is an excellent document, and was a big help in figuring out how to set this up a few years ago.

and now for something completely different -- where in the interpipes could a document like that have been published, vs. ISC's web site? the amount of red tape and delay involved in Usenix or IETF or IEEE or ACM are vastly more than most smart ops people are willing to put in. where is the light / middle weight class, or is every organization or person who wants to publish this kind of thing going to continue to have the exclusive and bad choice of "blog it, or write an article for ;login:/ACM-Queue/Circle-ID, or write an academic paper and wait ten months"? isn't this a job for... NANOG? -- Paul Vixie

_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog

-- Chris Grundemann www.linkedin.com/in/cgrundemann

On May 6, 2008, at 12:59 AM, Steven M. Bellovin wrote:

If not, what should the criteria be for an "official" note of the paper?

Perhaps it's an oversimplification, but can't those who wish to publish such information simply deliver their papers at a NANOG meeting (after acceptance by the Program Committee, which acts as a gate), and then the NANOG folks post the documents along with any slides and the VoDs of their presentations, in the usual fashion? ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // +66.83.266.6344 mobile History is a great teacher, but it also lies with impunity. -- John Robb

smb@cs.columbia.edu ("Steven M. Bellovin") writes:

...

...

If not, what should the criteria be for an "official" note of the paper?

Perhaps it's an oversimplification, but can't those who wish to publish such information simply deliver their papers at a NANOG meeting (after acceptance by the Program Committee, which acts as a gate), and then the NANOG folks post the documents along with any slides and the VoDs of their presentations, in the usual fashion?

That's certainly one very good answer. Are there others?

--Steve Bellovin, http://www.cs.columbia.edu/~smb

i think that's a good first tier, but there's still delay and congestion in that path. delay, because nanog meetings only happen N times per year, so an idea may have to wait months before it's widely circulated. congestion, because nanog meetings are of fixed duration and there is, and has to be, competition for the slots, to make the meeting interesting, keep quality high. as a second tier, if a technote draft could be sent to nanog-pc at any time, and the readable ones sent to nanog@ (at a maximum of one per week, so there would still be some quality-control related congestion, and rate limiting), and if the nanog-pc could then use mailing list feedback to judge whether the technote deserved to be given a number and put on www.nanog.org somewhere, we could be doing something really interesting with the expertise assembled here. -- Paul Vixie

... A web site like arxiv is good for some stuff. But -- should there be a link from nanog.org to operational content? Should nanog.org have its own archive? Should there be a peer review process? If not, what should the criteria be for an "official" note of the paper?

--Steve Bellovin, http://www.cs.columbia.edu/~smb

i wouldn't want to see a full academia-style peer review process, since that problem is pretty well solved elsewhere, and we're not having that problem. but a nanog-style peer review process, where the nanog-pc acts as the judge of how a technote was received by the mailing list, might work. such that if nanog-pc puts their stamp of approval on it, the connotation would be "more than one set of eyes has been laid on this, and it's not totally worthless." i say nanog-like because it's a new trail to blaze based on nanog's culture which, while often hard to cope with, has some innovative, genuine strengths.

On 6/05/2008, at 4:07 AM, Paul Vixie wrote:

i dearly do wish that something like a "service advertisement protocol" existed, that did what OSPF ECMP did, without a router operator effectively giving every customer the ability to inject other customer routes, or default routes.

This stuff about customers and things sounds too hard. Steve, have you actually had to do anycast without having control of the routing hop in front of your service providing hosts, or is this getting unnecessarily complicated? I'd imagine that the ability to install routing equipment would be a pre-requisite for any anycast service deployment.. Perhaps what would make more sense here is Foundry (F5, etc.) building an anycast feature - anycast prefixes are withdrawn when a cluster relying on that anycast prefix goes below a threshold. These load balancing switches already do all this service health check stuff and have done for years, so why are we re-inventing the wheel? -- Nathan Ward ps. I'm amused that your message that started with "i think the minutia is good, especially after a long weekend of layer 9 threads." ended with a paragraph of L9 :-)

On 6/05/2008, at 1:21 PM, Joe Abley wrote:

On 5 May 2008, at 20:50, Nathan Ward wrote:

...

Perhaps what would make more sense here is Foundry (F5, etc.) building an anycast feature - anycast prefixes are withdrawn when a cluster relying on that anycast prefix goes below a threshold.

I'm not sure exactly what feature is required, here. f5s of my acquaintance are already very capable of making OSPF LSAs based on virtual servers' pools being non-empty. Do it on more than one f5 in the same area, and you're anycasting service availability with the current feature set.

Can they do it with BGP for Internet anycast?

The general reason why people prefer to find alternative solutions rather than use dedicated load-balancers are that the dedicated load- balancers are hellishly more expensive than the $5 gigabit switch you probably already have in your garage.

The dedicated load balancers also talk BGP (well, ones I've played with), so that does away with the need for a BGP speaking router. -- Nathan Ward

On Tue, 6 May 2008, Nathan Ward wrote:

This stuff about customers and things sounds too hard.

Steve, have you actually had to do anycast without having control of the routing hop in front of your service providing hosts, or is this getting unnecessarily complicated? I'd imagine that the ability to install routing equipment would be a pre-requisite for any anycast service deployment..

Yes I have. Or rather, I've done the network infrastructure for anycast services without having administrative control of the anycasted servers. PCH's anycast platform hosts some blade servers for some other DNS infrastructure operators (in addition to the name servers PCH operates itself). Those operators operate their own servers. PCH operates the routing infrastructure. There is filtering in place to limit the routing announcements from the servers. But also, most of the larger organizations I've worked for have had separate systems and network engineering groups. In general, the network groups haven't wanted to let the systems engineers configure the routers, and the systems groups haven't wanted to let network engineers configure the servers (with good reason). Filtering of routing announcements from anycast servers would be useful in that environment too. To address Paul's point about multipath BGP, I never saw Cisco's implementation of it causing a problem even with full routing tables. I haven't used any other implementations. In the Cisco version (and at least for EBGP; I haven't looked at this with IBGP), it only applies to otherwise identical AS paths. Multiple directly-connected DNS servers sourcing the same announcement with the same AS path and other BGP attributes get load balanced between. Paths learned from different peers had different AS paths and do not get balanced between. I suppose there probably is load balancing in cases where there are multiple sessions with the same peer at the same exchange. That's a relatively rare case in this implementation, and using hash based rather than per-packet load balancing makes it not really matter. -Steve