- NANOG - lists.nanog.org

Re: Sanity worm defaces websites using php bug
by sgorman1＠gmu.edu 21 Dec '04

21 Dec '04

Produces something along the lines of this: http://www.noobforces.net/forum/viewtopic.php?p=1445&sid=d2260869a73fb5aca2 ----- Original Message ----- From: "Fergie (Paul Ferguson)" <fergdawg(a)netzero.net> Date: Tuesday, December 21, 2004 1:11 pm Subject: Sanity worm defaces websites using php bug > > > These people don't waste much time when a new exploit > found, do they? Geez. > > http://isc.sans.org/diary.php?date=2004-12-21 > > - ferg > > -- > "Fergie", a.k.a. Paul Ferguson > Engineering Architecture for the Internet > fergdawg(a)netzero.net or > fergdawg(a)sbcglobal.net >

1 0

Re: Anycast 101
by Christopher L. Morrow 21 Dec '04

21 Dec '04

On Tue, 21 Dec 2004, Iljitsch van Beijnum wrote: > > On 20-dec-04, at 17:32, Paul Vixie wrote: > > >>> 1. There should always be non-anycast alternatives > > >> I believe there is a strong consensus about that. And therefore a > >> strong agreement that ".org" is seriously wrong. > > > i believe that icann/afilias/ultradns would be very receptive to input > > from the ietf-dnsop wg on this topic. but it's not cut and dried -- if > > you have two widely anycast'd servers plus one non-anycast server "just > > in case something bad happens to anycast" you're doing two questionable > > things: (1) treating anycast as new/unstable/experimental which it's > > not, > > and (2) limiting your domain's availability to the strength of that one > > non-anycast server. > > ??? How are things worse with two anycasted addresses and one > non-anycasted address vs two anycasted addresses? There seem to have been a few (and paul/joe/isc/rodney have FAR more experience with this than I) different and 'interesting' failure modes for anycast DNS services to date: 1) complete dns outage (software caused problem on all devices) 2) route damping due to flaps (maybe software, maybe hardware, maybe link, maybe DoS) 3) inconsistency of dns data (perhaps not a problem at all) The trouble as I've seen it has always related to 'view' more than anything else. So, aside from a complete software basd outage, normally .ORG (to use as an example) is up for larger portions of the Internet. Often folks will report '.ORG is Down!!!' when they really mean: "The pod of .ORG tld1 nearest to my location is down." So, for most DNS applications you really don't care if the /32 is anycasted in BGP or is unicasted in BGP, you just care about reachability. Anycast provides geographic diversity, robustness through that diversity, resilience to single-network-problems (uunet can die while the anycast pod on qwest is still 'ok') and 'quick response' (close response really) to the end-users of the service. Making a mix of uni/anycast seems to not really solve the problems above, it may solve other problems, but I suppose that you'd have to scope and risk-analyze them to see how relevant they are to the entire userbase. I'd venture that the original 'out of sequence packets' problem falls down fairly low on the totem of risk/reward, but that's a guess. > Anycast can increase the number of physical servers, but this is also > possible with more traditional clustering techniques or anycasting that > stays outside BGP. look at what anycast in today BGP can get you in terms of geographic diversity, closeness to the eyeball and stability of the overall 'service'. Then think about how to provision Paul's 10,000 10gb link server farm... There is more than just 'clustering' to be done for DNS services at the TLD/Root level (and the roots do all manner of oddball versions of these things depending on their individual fancy as bill and paul point out regularly) > > What I find surprising is that every IP address gets to query the > roots, despite the fact that most addresses don't have any need to do > this or know how to do it properly. It would make perfect sense to me > that people would have to sign up for "root service" before they get to > talk to the root servers. This way, all unknown addresses can be > filtered out. (Or more practical, rate limited.) Obviously something > like this would face deployment issues, but if we're serious about DDoS > issues these kinds of options are the ones we should consider. > also keep in mind that not only the dns 'server' is under attack, as paul pointed out, and others before him, link filler attacks are also effective. So, block my /32 cable modem from quering a-m.root-servers.net, but my packet still got to the upstream link(s) for a-m.root-servers.net... the DoS was still effective in many cases :( Also, the IAB might consider 'root query service' some form of 'end to end filtering' and might not feel warm and fuzzy about that form of extortion. > Another way to approach this would be for larger ISPs to connect to one > or more roots using private peering. This gives those operators both > the means and an incentive to keep such links free of clutter. This also exists today... for many larger networks. Anycast extends that capability and provides some more close 'partnership' with the dns-service-provider and network-operator, or could in theory do that. I'm leary of the 'anycast is bad' or 'anycast is good' discussions in general. There are good applications of anycast, there are certainly horrid ones (true pplb of all links inside a IGP with IGP anycast of a TCP service could lead to some 'interesting' results) doing the risk analysis and proper engineering will help get away from the 'bad' and to more of the 'good' I hope. On the PPLB front though, that is almost always done via the IGP since BGP mostly holds a single next-hop (unless you non-default install it and deal with the multipath 'features') So, unless you have 2 instances inside your AS for an Anycast destination you really only go 1 path to the destination. Or more clearly, if you are on UUNET's network and want to get to 'tld1.ultradns.net' you have to go through Verio (on the east coast atleast) to get there and pplb really doesn't seem to matter for 'anycast', atleast no more than getting to 'www.sun.com' which is 'unicast' by sun... -Chris

1 0

Re: Anycast 101
by Paul Vixie 21 Dec '04

21 Dec '04

> bearing in mind there have been issues with org but not . i have > thought in the past there probably should be mroe than two ns records > in ns .. all i can say is: > > i believe that icann/afilias/ultradns would be very receptive to > > input from the ietf-dnsop wg on this topic. but it's not cut and > > dried ... ...by which i mean, even if it's the same people as would comment here on nanog, icann/afilias/ultradns would probably pay a lot more attention if the ietf's dnsop working group made such recommendations than if nanog made them. > its also not clear if there are two sets of nameservers in service or > one set serving both addresses? i think it's an even/odd thing. so, tld1.ultradns.net and tld2.ultradns.net each refer to a specific subset of .ORG nameservers. > i'm straying from the anycast theory a bit, but thought i'd mention my > 2c on past issues with org (altho i dont believe they were caused by > anycast itself) anycast can be a source of dns incoherency. sometimes this is deliberate as in the case of akamai; sometimes it's an accident like one server being lame or broken; sometimes it's a protocol limitation like not all servers have AXFR'd a new zone-content yet. i've been thinking that to correctly diagnose unintentional dns incoherency would require that one of the many distributed networks of "probe boxes" run a looking-glass-like cgi page that let you launch a dns query from 100 or 1000 different places simultaneously, and compare the results. perhaps dfk and ripe's dnsmon are listening today. i've also been thinking that AXFR's known incoherency could be reduced by using some kind of in-band embargo that would bring a new zone version online synchronously on servers supporting this feature and configured to enable it for a particular zone. those kinds of thoughts are appropriate for nanog. whether .org should use anycast, or how many NS RRs it should have, are not.

1 0

Cingular server outage?
by Fergie (Paul Ferguson) 21 Dec '04

21 Dec '04

According to KHOU 11 News in Dallas/Ft. Worth: [snip] "Millions of cell phone customers in Texas and 12 other states were without cell phone service Monday. The outage involves Cingular Wireless and AT&T. According to Cingular Wireless, service is out completely in Texas, Oklahoma, Kansas, Arkansas, Missouri, Illinois, Wisconsin, Ohio and Indiana. A company spokesperson said some customers can make calls, but no one can receive them. It's partially out in Georgia, Kentucky and Tennessee. The outage was caused by damaged servers. It's not clear when the problem will be repaired." [snip] - ferg (a Cingular customer in Austin) Ref: http://www.wfaa.com/sharedcontent/dws/news/texassouthwest/stories/122004txc… (registration required) -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(a)netzero.net or fergdawg(a)sbcglobal.net

1 0

Re: Anycast 101
by Paul Vixie 21 Dec '04

21 Dec '04

> [Warning: I've never actually deployed an anycast DNS setup so you are > free to ignore my message.] i'm not ignoring you because you raised two important issues. > > 1. There should always be non-anycast alternatives > > I believe there is a strong consensus about that. And therefore a > strong agreement that ".org" is seriously wrong. i believe that icann/afilias/ultradns would be very receptive to input from the ietf-dnsop wg on this topic. but it's not cut and dried -- if you have two widely anycast'd servers plus one non-anycast server "just in case something bad happens to anycast" you're doing two questionable things: (1) treating anycast as new/unstable/experimental which it's not, and (2) limiting your domain's availability to the strength of that one non-anycast server. in the root server system we're about half anycast and half not, at the maximum practical NS RRset size, which as you certainly know, is 13. if .ORG's NS RRset were to be changed to include non-anycast nodes, i'd hope for 11 of them, or however many underlying servers there actually are. but at that point, the only thing anycast would buy you is ddos resistance and the ability to have more than 13 physical servers... which is all the root server system wants from anycast, but maybe not all that afilias and ultradns and icann want from anycast in .ORG. > This is after all a good engineering practice: when you deploy > something new, do it carefully and not everywhere at the same time. this is the second important point you raise. anycast isn't new. rodney pioneered it commercially in 1997 or so. it had been in campus-area use for at least six years by that time and perhaps 10 years depending on how you count. akamai has been using it since 1999 or so. if there were any stability problems like the pplb assertions made elsewhere in this thread, we'd've all been seeing them for a long time by now.

2 1

Re: Dampening considered harmful? (Was: Re: verizon.net and other email grief)
by Jerry Pasker 21 Dec '04

21 Dec '04

>Well, a particular router doesn't get to set its dampening according >to its 'view' today, and that view is going to vary depending on >prefix. > >I would like to argue that how we define flapping today is simply a >broken concept. We count up/down/path change transitions, but such >transitions can exist due to connectivity or implementation >differences and may have nothing to do with not complying with >ettiquette. If people were damped for a single up/down, then yes, that's a problem. Dampening someone because they have a single path change is not the idea. That would sort of nullify the spirit of being multihomed with BGP, or even using BGP in any production or transit network for that matter. IF there's a connection problem, or implementation difference that makes a lot of up/down, then dampening could occur close to the "problem" but it will be contained close, and won't spread to the rest of the internet. >I submit that a better way of measuring flap is to look at the >period over which a particular prefix is flapping. Anti-social >behavior is many changes occurring over long periods of time. >However, many changes over a short period may well not be a problem. > >Regards, >Tony > So configure your route dampening that way. No one's twisting your arm to use the defaults. Want many flaps in the short term to not be damped? Simply increase your suppress limit. Way way up. Increase or decrease your half-life as you see fit. Just don't expect the rest of the world to do the same. But do expect to see others damp the flapping that you leak to their network. <gut feeling> If no one on the net ran dampening, routers wouldn't get anything done except processing constant churn. </gut feeling> I'm grateful that routers out there in the world are doing route-damp so my router's CPU doesn't have to deal with needless route-churn, even if it does impact my connectivity (to poorly connected) end sites somewhat. The stability is worth it. Another item to consider, is this is a sentence from RFC2439 " By damping their own routing information, providers can reduce their own need to make requests of other providers to clear damping state after correcting a problem." Using that logic, I'm also grateful for my transit providers dampening the advertisements to their peers, of my flapping transit links to them. That way, when they flap, (hey, it happens to everyone eventually) it doesn't get me damped by the rest of the internet, and my connections to different transit providers can do their job of carrying the traffic. Dampening, in it's current state, really is a good idea. -Jerry

2 1

Botnet pointer
by Hannigan, Martin 21 Dec '04

21 Dec '04

I've received a number of emails that indicates a lot of people are in the dark about BotNets. John Kristoff made an excellent technical presentation regarding BotNets at NANOG 32 in Reston. http://www.nanog.org/mtg-0410/pdf/kristoff.pdf I have an executive level summary I'm willing to share if it can be useful. Off-list please. Thanks. -M< -- Martin Hannigan (c) 617-388-2663 VeriSign, Inc. (w) 703-948-7018 Network Engineer IV Operations & Infrastructure hannigan(a)verisign.com

5 5

Re: Dampening considered harmful? (Was: Re: verizon.net and other email grief)
by Jerry Pasker 20 Dec '04

20 Dec '04

>An even more important consideration is whether our current paradigm >of flap dampening actually is the behavior that we want to penalize. > >If a single link bounces just once, then thanks to our mesh, >confederations, differing MRAI's etc., we can see many many changes >to the AS path, resulting in dampening. Do we really want to >inflict pain over such an incident? > >Tony > Of course not. Dampening should be set on a router according to that router's views. The more chances a router can see a single event multiplied, the more lax dampening should be on that router. I think Rob Thomas's saying of "know your network" really applies here. In my book, the threat of dampening to anyone not playing nice is the true value of route dampening. Automatic enforcement of etiquette. -Jerry .

1 0

Re: Botnet pointer
by Fergie (Paul Ferguson) 20 Dec '04

20 Dec '04

Here's a decent pointer: http://en.wikipedia.org/wiki/Botnet - ferg -- "william(at)elan.net" <william(a)elan.net> wrote: Can somebody also share good definition of "BOT" and "BOTNET" for glossary and description of 2-4 lines? -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(a)netzero.net or fergdawg(a)sbcglobal.net

2 1

Spammers ordered to pay $1 billion
by Hank Nussbacher 20 Dec '04

20 Dec '04

http://www.cnn.com/2004/LAW/12/18/spam.lawsuit.ap/index.html What a nice present for the holiday season :-) -Hank

4 3