- Test - lists.nanog.org

Re: NANOG36-NOTES 2006.02.13 talk 7 QoS in MPLS environments
by Fergie 13 Feb '06

13 Feb '06

Complete red herring (tm). - ferg ps. Thanks for the notes, though. ;-) -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(a)netzero.net or fergdawg(a)sbcglobal.net ferg's tech blog: http://fergdawg.blogspot.com/

1 0

NANOG36-NOTES 2006.02.13 talk 7 QoS in MPLS environments
by Matthew Petach 13 Feb '06

13 Feb '06

Here's my notes from the MPLS QoS tutorial; wish I could have been in two places at once to catch the ISPSec BOF as well. I won't be taking notes at Eddie Deens, though, so it'll be up to Ren's camera to capture the details for those following along at home. < http://nanog.multiply.com/ > Matt 2006.02.13 QoS in MPLS networks tutorial notes. See notes for Agenda, outline, etc. at http://www.nanog.org/mtg-0602/sathiamurthi.html Traffic characterizations go beyond simple DiffServ bit distinctions Understand traffic types and sources and nature of traffic before Latency, Jitter, Loss three traffic parameters to be tracked that influence choices made when applying QoS It's all about managing finite resources rate control, queing, scheduling, etc. congestion management, admission control routing control traffic protection The QoS Triangle (no, not bermuda triangle) Identify Traffic Type Determine QoS parameters Apply QoS settings 2 approaches to QoS fine-grained approach or combination of flows to same traffic type, to same source. Needs to have same characteristics so you can consider them as an aggregated flow. Best Effort is simplest QoS Integrated services (Hard QoS) Differentiated Services (soft QoS) Best Effort is simple, traditional internet Integrated services model, RFC 1633, guarantees per flow QoS strict bandwidth reservations. RSVP, RFC 2055, PATH/RESV messages Admission controls must be configured on every router along path Works well on small scale. Scaling challenge with large numbers of flows. What about aggregating flows into integrated services? DiffServ arch; RFC 2475 scales well with large flows through aggregation creates a means for traffic conditioning (TC) defines per-hop behaviour (PHB) edge nodes perform TC keeps core doing forwarding tough to predict end to end behaviour esp with multiple domains how do you handle capacity planning? Diff services arch slide with pictures of traffic flow. TCA prepares core for the traffic flow that will be coming in; allows core to do per-hops behaviour at the core. IETF diffserv model redefine ToS byte in IP header to differentiated services code point (DSCP) uses 6 bits to define behaviour into behaviour aggregates. Class Selector (CS0 through CS 7) classifier; selects packets based on headers. Classification and Marking flows have 5 parameters; IP src, dest, prececedence, DSCP bits, You can handle traffic metering via adjusting the three flows. 3 parameters used by the token bucket; committed information rate conformed and extended burst size Policing vs shaping. policing drops excess traffic; it accomodates bursts; anything beyond that gets dropped; or, can be re-marked. Shaping smooths traffic but increases latency. buffers packets. policing uses the token bucket scheme tokens added to the bucket at the committed rate depth of the bucket determines the burst size packets arriving when there's enough tokens in the bucket are conforming packets arriving when the bucket is out of tokens are non-conforming; either coloured, dropping, etc. diagram of token bucket, very nice. shaping--use the token bucket scheme as well smooths through buffering queued packets transmitted as tokens are available. 1 aspect is traffic conditioning at edge 2 aspect is per hop behaviour PHB relates to resource allocation for a flow resource allocation is typically bandwidth queing / scheduling mechanisms FIFO/WFQ/MWRR(weighted)/MDRR (deficit) congestion avoidence RED (random early detection / Weighted random early drop Queing/scheduling needs some data mining to decide how to prioritize certain classes of traffic. de-queues depends on weights assigned to different flows. Congestion avoidance technique when there is congestion what should happen? tail drop (hit max queue length) drop selectively but based on IP Prec/DSCP bit Congestion control for TcP adaptive dominant transport protocol Slide showing problem of congestion; without technique, have uncontrolled congestion, big performance impact due to retransmissions. TCP traffic and congestion congestion vs slow-start sender/recieever negotiate on it. source throttles back traffic. (control leverages this behaviour) Global synchroniztion happens when many flows pass through a congested link; each flow going through starts following the same backoff and ramp up, leads to sawtooth curves. RED a congestion avoidance mechanism works with TCP uses packet drop probability and avg queue size avoids global synchronization of many flows. minimizes packet delay jitter by managing queue size RED has minimum and maximum threshold; average queue size is used to avoid dealing with transient bursts. WRED combines RED with IP precedence or DSCP to implement multiple service classes each service class has its own min and max threshold and drop rate. nice slides of lower and higher thresholds for different traffic types. When is WRED used? only when TCP is bulk of traffic. Won't help UDP or other IP MPLS and QoS, into DiffServ avoid vendor CLI as much as possible for the talk. stick with techniques only. do classification and marking at edge, then do per hop behaviour on when to queue or drop packets within the core. Within the MPLS domain, do you lose all the nice classification information? No, you tunnel information from IP DiffServ into MPLS DiffServ. MPLS DiffServ doesn't introduce new QoS architecture uses diffserv defined for IP QoS (RFC 2745) MPLS DiffServ is defined in RFC3270 uses MPLS shim header show slide of diffserv scalability via aggregation traffic enters at PE router, goes through P core, comes out PE at other side. MPLS scalability comes from aggregation of traffic on the edge processing of aggregate only in the core deal with buckets only, thus can scale well. the PE router has to put 2 labels on; next router What's unchanged in MPLS diffserv? traffic conditioning agreements same classification, marking, shaping, policing still happen at the edge buffer management adn packet scheduling mechanisms used to implement PHB PHB definitions EF: low delay/jitter/loss AF: low loss BE: no guarantees (best effort) what's NEW in MPLS diffserv? Prec/DSCP field not visible to MPLS LSRs info on diffserv must be made visible to LSR in MPLS header using EXP field/label how is DSCP mapped into EXP--some interation between them. EXP is 3 bits, S is 1 bit. Typical mapping Expedidted forwarding: EF DSCP 6 bits to 3 bits of EXP bits. 101000 maps to 101 but then you lose bits of informatin. IP DSCP 6 bits whle MPLS EXP = 3bits (RFC 3270) if 8 or less PHBs are used, map DSCP to EXP directly, with E-LSPs with preconfigured mappings If more than 8 PHBs, needed to be mapped in label and EXP; L-LSPs are needed Both E-LSP and L-LSP can use LDP or RSVP for label distribution. MPLS: flows associated with FEC mapped to one label DS: flows associated with class, mappable to EXP MPLS diffserv tunneling modes Based on RFC 3270 Modes uniform short-pipe pipe how do you implement the modes? depends on your engineering decisions. uniform mode assume the entire admin domain of the SP is under single diffserv domain then like a requirement to keep colouring info the same (uniform) when going from IP to IP, to MPLS, back again, etc. in both MPLS to MPLS and to IP cases, tehe PHB of the topmost popped label is copied into the new top label or the IP DSCP if no label remains. Short pipe mode assume an ISP network implmementing a diffserv model assumes customers implement a different policy. note that the policy applied outbound on egress interface is basd on DSCP of the customer, hence the short-pipe naming. Pipe-mode same as short-pipe however, SP wants to drive the outbound PHBs of the topmost popped label is copied to the new top label classification is based on mpls-exp field (EXP=0) of the topmost received MPLS frame MPLS TE and DiffServ is diffserv good enough to determine end to end quality of service? nope. what happens if there's no congestion, but a link fails? when link fails, and reroute happens across a new link; the new link gets congested due to combined traffic. You may need to engineer your traffic on non-optimal path to assure enough bandwidth will be ready for it. So you have BW optimization and congestion management in parallel TE + DiffSErve spread traffic around with more flexibility than IGP supports. MPLS labels can be used to engineer explicit paths tunnels are uni-directional How does MPLS TE work? Explicit routing constraint-based routing admission control protection capabilities RSVP-TE to establish LSPs ISIS and OSPF extensions to advertise link attributes Diffserv aware TE per-class constraint based routing per class admission control so best effort can go on one link, while low-latency can be shifted along a different link. Link BW distributed in pools of BW constraints (BC) up to 8 BW pools different BW pool models Maximum Allocation Model (MAM) Maximum Reservable Bandwidth (MRB) BC0: 20% Best Effort (admission class 1) BC1: 50% Premium (admission class 2) BC2: 30% Voice (admission class 3) Per class traffic engineering concept; all 3 sum to MRB If for any reason the part of traffic hard reserved isn't being used, it's wasted; nobody gets to burst into it. No sharing of unused capacity. But simple, independent. DS-TE BW Pools--Russian Dolls Model (RDM) BW pool applies to one or more classes Global BW pool (BC0) equals MRB BC0...BCn used for computing unreserved BW for class n so BC0: MRB (best effort + premium + voice) BC1: 50% premium + voice BC2: 30% Voice Downside is higher bandwidth class may push out some lower traffic that was flowing already. Aggregate TE in diffserv network DS TE and QoS Diffserve-TE doesn't preclude the necessity of configuring PHB QoS in the TE path; DiffServ TE operates in conjunction with QoS mechanisms. Traffic engineering is a huge field; so it's hard to cover in a short period of time. Summary: QoS techniques effective allocation of network resources IP DiffServ Service Differentiation good starting point, bu doesn't scale that well MPLS and DiffServ Builds scalable networks for service providers DiffServ Tunnelling modes Scalable and flexible QoS options Supports Draft Tunneling Mode RFC DiffServ TE provides strict point-to-point guarantees pipe models are your choice, how do you want to architect your network? What are _your_ traffic needs? When you need to drop traffic, determine how you'll drop traffic based on DSCP bits so you can set watermarks on the traffic; some traffic more lenient about drops, other traffic not so lenient about drops. Question: Fred W. from Bechtel. With IPv6, there's a 20 byte flow label, rather than the 8 bit agony of mapping the v4 DSCP bits; does that give more flexibility, more choices, are there fewer headaches associated with v6 QoS handling? Short answer--the presenters aren't as focused on v6 development, so they don't have a concrete answer to give there, sorry. That wraps up the presentation/tutorial at 1715 hours pacific time.

1 0

NANOG36-NOTES 2006.02.12 talk 5 IPv6 --fear and GOSIP in Dallas
by Matthew Petach 13 Feb '06

13 Feb '06

Apparently the video feed is of very good quality this time around--many thanks to Brokaw for the good bandwidth to the hotel! Last set of notes before lunch. Matt 2006.02.12 NANOG IPv6 transition panel panel member briefs at http://www.nanog.org/mtg-0602/golding.html IPv6: time for transition, or just more GOSIP? GOSIP was initiative to use OSI networking throughout the government. 5 participants Joe Houle ATT Jared Mauch NTT America Wes George, Sprint Jason Schiller, UUNet/Verizon Fred Wettling, Bechtel Tried to get government people, since they went v6, but they're not forthcoming with details; you know how government people are. :D Daniel Golding, The burton group Joe Houle, ATT is up first. Emerging service for ATT for L2/L3, IP private networking, v6, etc. fall under his baliwick. He'd count himself as pragmaticlly pro; IPv6, why now? He does believe we're running out of IPv4 addresses. NATs and non-unique addresses make offering quality services difficult. Convergence doesn't work well over NAT'd addresses. why governments? US government doesn't want the have-have-not split to continue; the v6'ers may be the "have" side and we don't want to be on the have-not side. NTT America (AKA 2914) Native dual-stack IPv4/IPv6 since fall 2003 Cisco 7200, 7500, "76k" Juniper M series, T-series Wes George, Rob Rackell hat, couldn't be here due to weather. Pro v6, looking at it with skepticism. Sprint close to center of v6 world. 200pps on v6 network. Internet doesn't use v6 for real yet this is not the movie as the ISO fun; this time the government is paying! IPv6 is something that US carriers can make money on in the VPN space It is not valuable as an internet transport yet spend less time marketing about how cool it is, and go fix the issues!! multihoming, micromobility, SHIMv6 is a host solution. This time around, the government is paying. They don't know exactly what they want, but they know they want it. hoping carriers will figure it out and tell them. Jason Schiller, UUnet/Verizon. public v6 roadmap. AS284 US/AS12702 EMEA/AS18061 AsPAC) for v6 only Over network utilizeing GRE Phase 2 6PE solution in AS701 dual stack v4/v6 on edge mail, DNS support later phase 2a upgrade exising non-6PE capable edge routers 2007, phase 2b native v6 in the core (maybe) Problem is, no money yet in v6, so can't roll out aggressively at all. But if no money, why put it in the core? Well, to be ready in case it DOES take off in the future. Fred Wettling, Bechtel--large enterprise, also with v6 business council. Bechtel Telecoms (A & C for big carriers like Sprint, ATT, etc). Interested in non-traditional transport of IP services. shift in plant automation networks from proprietary to IP; so want to be ahead of the curve on it. Bechtel's internal test started last year, will be deployed out to 40,000 by this year; a bit of the chicken and egg issue, go back to 1995, IE v1 vs today; things will progress, things will take off, the goal is to be ahead of the curve. Daniel Golding, host for the panel. Question 1: Why IPv6, why now? Why are you implementing v6, other than it's cool? Is it address exhaustion, new capability, Gov't RFP requirements, vendors pushing new hardware? Jared notes they rolled it out in 2003 due to global pressures; they wanted to keep a unified network model worldwide, and as a subsidiary of a japanese company, and the largest player in that space, combined with government mandates, really pushed them in that space early. It _is_ a technical cool thing, it's good to be a market leader. Jared notes that they've been running dual stack v4/v6, it just works. ATT VoIP has been a driver, just doesn't work over NAT, so what other solutions are there? Really, address exhaustion, non-unique addresses propagating throughout space is just putting roadblock after roadblock in front of convergence. Dan asks why do we need NAT--we're not OUT of v4 addresses yet; Joe notes that people are really using NAT as a security mechanism right now, more so than really worrying about conserving address space. Yes, it's bogus, but it's what people have been sold on right now, so it gets widely used. Jared pitches in and notes that the push for encapsulation of everything encapsulated over port 80 is getting more and more widespread. People are attempting to use "firewalls" and "NATS" to give themselves the notion of security, even though most infection rates now are coming from other vectors (spyware, infected email, etc), rather than outside probing. Dan notes we don't need to do NAT, they can go to their upstream, to ARIN. But ARIN frowns on using public space for private use? Bechtel notes they're running into more and more problems as they try to get companies to do joint ventures, as every company uses 10.x space, and they have to do NAT over NAT, it's evil. He's also an IMOD (infrastructure modernization) player, it's a 4 billion dollar upgrade for the military, and it has a specific subsection about using v6 to avoid that problem. Also, intercorporate and collaborative efforts are exposing more and more info to the outside, with DMZs, etc. We need to make sure security is being correctly approached, NOT from using IP addressing tricks, but rather correctly. Jason notes v6 is also a "young", "fresh" stack which is nice. But Dan isn't sure that's a benefit per se. Joe would like to be able to use the flow label header, get beyond diff-serv level to get application aware network. Right now, just a bunch of bits, but since it's build into the header, it could really provide some real value add. Fred notes it's like people asking 12 years ago "what's the value/benefit of the Internet?" There will be things that will come out that we can't imagine yet once we start deploying it. Asking "like what" is asking us to imagine what we haven't conceived of yet. Fred notes that v6 will actually be simpler for many of their company-to-company connectivity. Toshiba, I300 TV, 3 ethernet jacks on it, it speaks v6, now carriers that speak v6 don't need redundant services, it can all be handled through the one link. Can't do that easily on v4. Dan notes multihoming for non-LIRs isn't doable under v6 at the moment, and that seems to be one of the holdups. People at last NANOG didn't like the shim6 solution. But enterprise multihoming is a customer requirement, people aren't willing to give that up. If only the big guys can get v6 space, we won't get widespread adoption. Joe notes shim6 is the only thing in the works to support multihoming. But Dan notes that the big content players like Yahoo have rejected it, since their servers don't have resources to handle 1800+ addresses on each server. Jared notes that even though we don't want to keep upgrading our routers, there's no other sane path that we can see. But Joe notes there's potential for explosion that scares him. It *could* happen. Someone could deaggregate the v4 space, though, and Jared notes it could blow up our current routers already today. So what about measuring fragmentation in v4 world today, vs in the v6 world, and can we accept that? Jared notes we'll come to a consensus on it, the way we did with CIDR and Sprint's filters for v4 space for a long period of time. Verio was one of the players holding that line for a long time. Fred notes business council will chime in on PPML soon. Bechtel has 40 different ISPs around the world; their network changes on a regular basis; trying to add and subtract address space each time a carrier comes or goes will be a major headache. Bechtel would like to work with carriers to advertise their netblocks, and namespace, NOT add a headache on re-addressing every time they switch providers. Bechtel got their IPv6 space via ARIN. They build turnkey solutions for owners, they operate as an LIR from that aspect since they turn over address space with the plants they build to their customers. Jason speaks up about number of 24s in table, vs number of 48s possible. About 16 million /24s in v4 space. 137 billion /48 blocks in v6 space, so about 2 million times more /48s than /24s. customers might go shorter than that to do traffic engineering. So, what is Verizon's answer? Couple of answers: Tony Hain--multihoming in v6 just works, do it, let the routers keep up. Jason's personal view is that we need another solution; we can't risk networks blowing up, but shim6 does seem pretty broken. Wes notes memory is pretty cheap, in general. But specific memory, like TCAMs, are more expensive, as are custom ASICs. It's like building a network based on classful fast path forwarding; the problem there is starting from bad assumptions. Sprint is adhering carefully to the RFC that Rob wrote a while ago, about announcing the top aggregation, nothing more. That limits even PA multihoming, really. Dan rephrases, do you think most providers will only announce their block, or will they leak more specifics? Jared encourages people to filter on RIR boundries; if ARIN says they only allocate /48s, then filter on that, and only accept up to that size. Customers have come pressuring them to allow announcing their /64, but that's just silly. Verizon--taking queue from IETF/ARIN, aggregation is the holy grail, so they only advertise their aggregate. To the global internet, only send the aggreate. That's the policy thus far. If people start putting /48's into the global routing table, will show that aggregation isn't the holy grail, will probably mean we do the v4 model. Joe is on the edge; he agrees, but he's worried about the slipperly slope. Fred notes if we have a brick wall, we won't go anywhere, at least on a slipperly slope, we'll get some movement; he can't be locked into IP space locked into providers, he works with hundreds of companies around the world. Wes--VPN advances will probably move v6 forward, as will government, more than in "normal" IP v4 model. Less FUD driven. IS it NANOG's problem? It's a plumbing problem, and we're a bunch of carpenters. It's an IETF problem, more than a NANOG problem; but they haven't given us the right tools to work with. But trouble to see ROI for operators to go to IETF, for vendors much easier to see the ROI. The operators need to get involved in listing requirements for protocols, before the work even gets started. Bunch of people run for the mikes: Joel Yagli, UofO; for all: looks in routing tables, doesn't see that many routes, even if people do some deaggregation, probably won't see that many routes. In context of shim6, carries more state in hosts; much more problem to put pain in thousands of hosts; upgrade smaller number of hosts, rather than larger number of hosts. What about I2 v6 view? They get block from their upstream, they get no multihoming option. it's important in the v4 space, but not in v6 yet. No plans to upgrade to shim6 on their hosts, no. 170K internet routes. Add 150K internal v4 routes, you're into 300K routes for v4 table. Add to that the v6 space, assuming one aggregate and deaggration based on v4 model. 24K aggregates, 35K deaggs, would add another 70K. There's about 59K intentional deaggregates happening in the v4 space today. If you add all that up, and do 2547 entries in FIB, the v6 contribution is very part of the problem. Jason notes that vendors currently claim their boxes Bill Woodcock--Bechtel has IPv6 space, some other large enterprises that have their own v6 space. In ARIN region, only way to get v6 space is to assert that you will have a large number of v6 customers imminently. If you would like ARIN policies to be updated to reflect reality, would be good to get more involved with ARIN policy--please!! Fred notes the only sustainable model for them was to get their own allocation, given that they work with so many end-customers. Ted Seeley, Sprint. Aimed at Jason. Scale of infrastructure. Entire v6 space fits in 256K of TCAM right now. Jared, have you done homework on how often they will need to upgrade their linecards? They do upgrades to handle v4 traffic, and the v6 upgrades come along for the ride. But for 200pps, don't need to worry today about v6 traffic. When v6 takes off, it *will* deaggregate, there's no stopping it. How will it be handled? The deaggregation in v6 space is unlikely to be as bad as in the v4 space. May see a boom where people need more space, may need to deal with it then. But people are finding the same challenges with 2547 customers with thousands of RFC1918 addresses and route targets in VRFs to keep track of? It's a problem overall, why is v6 being singled out? You'll have the same growth potential in the v6 space, and if something has to be tossed out, it might as well be v6, because everything else has revenue associated with it. Steve Feldman, CNET, medium sized content company. If the internet goes away, they get no revenue. management mitigates against internet going away by having many upstreams. They need to multihome, they don't care "how" as much, they have to do it. If there isn't a multihoming solution in place for v6, companies just won't move to it, period. Their upstreams that do provide v6 space won't allow multihoming. John Curran, IP directorate, 1993/1994, it's been going through evolutions for years. ARIN chair, they pay attention in board meetings, they verify applications in both v4 and v6 space. As we go through v4 address depletion in coming years, the stringency for new allocations will increase. What should ARIN do to their peers to ask for more IPv4 addresses, vs IPv6 allocations. Randy Bush, IIJ; considering that number of v6 allocations has more hosts behind it than the number of bits moving across it. Rob Seastrom, speaking for/and about himself. He runs v6 in his AS. The router is subject to being cannabalized for v4 router until there is content on v6 and there is multihoming for v6. Rough consensus and working code has been replaced by analysis paralysis. Tony Hain--part of reason for analysis paralysis is initial requirements were conflicting; network and endsites had conflicting requirements, so is there any wonder the compromise that resulted is not to anyone's liking? The IETF is spiralling out of control due to lack of feedback from operators in the IETF process. Fear is mentioned over and over again. Fear of what could happen if we allow PI space. He's had a draft of a different way of looking at problem in a different way. Allow for people you "care" about to have a routing slot; how do you distinguish big guy from little guy you can aggregate in a sane and fair way? Is mythical global routing table really real? route-views would say it's not. If we're not willing to step up to IETF and make our voices heard, we'll be stuck with a broken protocol. Ted Seely notes the vendors monopolize the IETF and want to sell boxes, not make working code!! Joel Yagli, timeframe for reachitecting the global routing table for v4/v6 isn't relevant; if we keep talking about this in 10 years, we're screwed. Randy notes we've been saying that for 10 years. There are people who have been working for 15 years to get paradigm changed, and they're tired of wading back into the cesspool yet again!! They've shed blood in that process, and the food isn't even that good! OK, that wraps up the panel, many thanks to everyone, including Dan for chairing it--very good, very active panel! Dead wireless unit at front of room,, will be fixed over lunch. Be back at 2pm for tutorials!

2 1

NANOG36-NOTES 2006.02.13 talk 3 NTT labs AAAA query explosion worries
by Matthew Petach 13 Feb '06

13 Feb '06

(Huge apologies in advance for any and all names I completely mangle! check http://nanog.multiply.com/ to see names/faces correctly handled by Ren. ^_^; ) Matt 2006.02.13, talk 3 NTT labs, (Steve Feldman apologizes for mangling the pronnounciation of their names). NTT information sharing platform labs (didn't get names/info from opening slide) Outline Expect increase in number of DNS queries this year Discussion effect on cache server load and user response time how can we decrease number of unnecessary queries? Today's topic we focus on increase in number of queries between users and cache servers caused by IPv6 support number of 4A queries same as that of A queries domain name completion function DN completion by OS DN completion by application IPv6 enabled OS increases 4A queries Vista will be v6 enbled by default IPv6 and OS resolver IPv6 enabled OS sends 4A queries for every name resolution BSD/Windows Sends both A and 4A queries for every name resolution currently no way to disable one or the other Domain Name Completion when a name resolution fails, both OS and APP automatically try different prefix/suffix completions. OS using these domains to complete: FreeBSD: specified by "search" in /etc/resolv.conf, distributed by DHCP Windows: configured in control panel, distributed by DHCP Applications: Mozilla: retries with www domain prefix IE searches domain using MSN search and then retries name resolutions for domains by adding .com, .org, .net, .edu. Convenient for user, perhaps, hard on nameservers. Combination in FreeBSD completions are different depending on OS FreeBSD tried domain completions for A and 4A for each case. Windows tries all 4A records first, THEN tries all A records. So IPv6 queries in Windows means even if there's an A record in v4 space, it exhausts ALL 4A possibilities FIRST, before going back to get A record. Longhorn/Vista IPv6 default enabled ALWAYS tries 4A queries first! IE7 plus Vista results in 12 DNS queries per user click, best case. Worst case, one user click results in 40 DNS queries!! Slide showing projected impact based on historical data plus projected Vista deployment. Right now, 4A queries only about 5% of queries. After Vista, size of increase could dwarf rest of DNS queries. Release of Windows Vista (IPv6 by default) doubles at least the number of user queries causes more queries in domain name completions and domain search sequences Operators cache servers should be prepared for those increases stop domain distribution to users by DHCP or PPPoE Developers of OS is current search order of resolvers appropriate? eg should "A" record be resolved before domain completion. Ed from Neustar, at microphone: before we consider this a problem, consider from point of application provider; when you need a name, you don't know what transport you may have underneath; if you wait for NXDomain, you increase latency, so app developers generally send all queries at once. What about changing DNS to allow asking for multiple questions at once? Changing application behaviour isn't likely to happen, and changing protocols isn't easy; so why not just beef up the infrastructure to handle it? Joel Yagli, UofOregon; do you know how many of those queries will need to fail over from UDP to TCP due to responses being too large to fit into a single UDP response? Most of the responses coming back don't have data, so they don't need to go to TCP. Tony Bates--what happens when v6 record is returned as valid; does the chain stop there? Also, if you flip to return A record first, we'll never to move to v6. We NEED to start resolving v6 records first, to help move the 'Net off IPv4. Applause, on to next talk.

2 1

NANOG36-NOTES 2006.02.13 talk4 DNS infrastructure distribution
by Matthew Petach 13 Feb '06

13 Feb '06

2006.02.13 Steve Gibbard DNS infrastructure Distribution Steve Gibbard Packet Clearing House http://www.pch.net/ scg at pch.net Introduction Previous talk on importance of keeping criticical infrastructure local Without local infastructure, local communications are subject to far away outages, costs, and performance Critical infrastructure includes DNS If a domain is critical, so is everything above it in the hierarchy Sri Lanka a case in point. Previous talk was in Seattle last spring, highlighted undersea cable being cut; even local DNS queries failed since TLD servers couldn't be reached, even though local connectivity still worked. The ship dragging anchor in harbor cut only undersea path out of the country; international calling was down, and all of the Internet. But unlike local telephone system, even local networks failed to work. Root server placement Currently 110 root servers(?) Number is a moving target Operated by 12 organizations 13 IP addresses at most 13 servers visible from any one place at any one time six are anycast four are anycasted in large numbers All remaining unicast roots are in the bay area, LA, or washington DC Distribution by continet 34 in NA 8 each in BA/DC/ 5 in LA Only non-coastal roots in US are Chicago and Atlanta canada, monterrey, mexico some others 34 in Europe clusters of 4 each in London, and amsterdam, Europe's biggest exchanges even throughout rest of europe for rest. Distribution by continent 26 in Asia (excluding middle east) 5 in japan (4 tok, 1 kyoto) 3 in india, korea, singapore 2 in hongkong, jakarta, and beijing south asia an area of rapid expansion 6 in australia/new zealand 2 in brisbane 1 each in auckland, perth, sydney, and wellington 5 in middle east 1 each ankara, tel aviv, doha, dubai, abu dhabi 3 in africa 2 in johannesburg 1 in nairobi, 1 more being shipped very little intercity onr intercountry connectivity 2 in SA sao palo santiago de chile other parts of world not really served at all. world map with blobs showing coverage. Huge areas not covered. overlaid fiber maps with dots to get ideas of coverage (redundant); everyone else is one fiber or satellite cut from being isolated and dark. Pretty much follows the areas with money. Root server expansion 4 of 12 root servers actively installing new roots 110 root servers big improvement over 13 from 3 years ago two operators (autonomica, ISC) (I and F) are installing wherever they can get funding funding sources typically RIRs, local governments, or ISP associations Limitations in currently unserved areas are generally due to lack of money Fs and Is In large portions of world, several closest roots are Is and Fs At most 2 root IP addresses visible; others far way Does this matter? gives poorly connected regions less ability to use BINDs failure and closest server detection mechanisms non-BIND implementations may default to far-away roots Should all 13 roots be anycasted evenly? CAIDA study from 2003 assumed a maximum of 13 locations; not really relevant anymore Big Clusters Lots of complaints about uneven distribution Only really a concern if resources are finite Large numbers in some places donesn't prevent growth in others Bay Area and DC clusters seem a bit much, but sort of match topology Western Europe's dense but relatively even distribution exactly right Two per city perhaps a good goal for everywhere TLD distribution Like the root, locally used TLDs need to be served locally Locally used TLDs: local ccTLD; any other TLDs commonly in use Regions don't need ALL TLDs. gTLD distribution: .com/.net .com/.net well connected to the "internet core" servers in the big cities of US, Europe, Asia non-core location: sydney. Map of world with .com/.net overlaid with fiber maps shows "well-served areas" again following the money, with even less coverage outside NA/Europe/Asia. gTLD dist: .org/.info/.coop share same servers considered confidential. data may be incomplete significantly fewer publically visible servers, almost all in internet core. only one public locatino in each of asia and europe Even worse coverage worldwide, though they do have south africa. Do have some caching boxes next to caching resolvers at the big ISPs; not sure if it increases coverage or not. Few other gTLDs, didn't map them. .gov is us-centric .edu is US, some eu, some asia .int is california, netherlands, UK (not very international!!) Where should gTLDs be? presumably depend on their markets if it's ok for large portiions of the world to not use those gTLDs, then it's OK for them to not be hosted there. ccTLD dist: answers to where ccTLDs should be more straightforward working in their own regions a must working in the "core" could be a plus just over 2/3 of ccTLDs are hosted in their own countries (but a lot of those aren't ... Green map shows those countries that host their own ccTLDs locally. Most islands are red, in danger of being cut off from their ccTLDs. ccTLDs not slaved in the core 18 ccTLDs aren't slaved in the global core if their regions are cut off, those ccTLDs won't be visible to the rest of the world is that really an issue, if you can't get to the end site anyhow? violation of RFC2182, unclear data results not so much matters if nobody from out .bb .bd .bh .cn .ec .gf .jm .kg .kw .mp .mq Example countries Kenya exchange point, root server, ccTLD server, all external connectivity by satellite Pakistan root server, no exchange point, no TLDs locally (so how much use is the local root server?) Kenya: local exchange in nairobi root server ccTLD server so even if external link goes down, country can stay mostly functional. Pakistan: local root server (for at least one ISP) no TLDs .pk hosted entirely in US no local exchange to share local root server single fiber connection; when it breaks, nothing works. Local peering caveat local traffic has to be kept local before keeping DNS local is of much benefit. Requires either strict monopoly, or local exchange points Examples here highlight that. Methodology Get name server addresses for TLDs Assume everything in a /24 is same place or set of places. (really down and dirty shell scripts) bad assumption for UUnet nameservers; didn't find others. 625 /24s contain name servers for TLDs 135 host multiple TLDs; over 60 in RIPEs case Figure out where those subnets are traceroutes/ask questions Subnets with 10+ TLDs--read it from the slides. :D 193.0.12/24 192.36.125/24 Other sources www.root-servers.org had root server data; assumed accurate. ultraDNS locations considers its locations confidential Got info from Afilias's .Net application. Told missed some. In general, most other TLD operators were very helpful. Thanks! http://www.pch.net/resources/papers/infrastructure-distribution/ Mark Kosters, Verisign; notes there's two other root server groups also going anycast wherever people will pay to host them. K with RIPE is now going outside region, and Verisign (J?) is also talking about serving in multiple regions. Dealing with local customs getting in country tends to be the biggest challenge; PCH has seen similar challenges getting into countries. OK, break time now.

1 0

NANOG36-NOTES 2006.02.13 talk 2 Duane Wessels, DNS cache poisoning
by Matthew Petach 13 Feb '06

13 Feb '06

2006.02.13 talk 2 DNS cache poisoners Lazy, Stupid, or Evil Duane Wessels Motivation During March/April 2005, SANS internet storm center reported a number of DNS cache poisoning "attacks" were occurring Poisoned nameservers have bogus NS records for the com zone SANS ISC theorizes it may have been a vector for spyware propagation Microsoft windows (most versions) and symantic firewall products are affected. Slides are on the website, BTW. The poisoning attack: an auth nameserver (where queries normally go) is configured to return bogus and out of baliwick NS auth records. caching resolver receives and trusts those bogus referrals future queries for names in poisoned zone go to the bogus NS dig +trace longislandauction.com will show the poisoned NS auth responses NS auth1.ns.sargasso.net. which has NS com. auth1.ns.sargasso.net. so any caching resolvers may consider auth1.ns.sargasso.net authoritative for any unknowns in com zone. Vulnerable implementations: Windows NT (by default, SP4, can tweak it via reg) Windows 2K, (by default, later fixed) Windows 2003 (not by default, but easy to unfix) Symantec gateway firewalls SYM04-010 and SYM05-010 to Yahoo search and find more. How to find poisoners? start with a large list of DNS names or zones discover set of auth servers for the zone by following referrals on down from root query each auth nameserver compare the NS RR set in each reply to the previously-learned referrals for parent zones this technique only finds parent-zone poisoning. February 2006 Survey input list is about 6 million names from nameservers they have access to. Found 284 "poisoning" nameservers; returns bogus NS entries for root or TLD. . has 217 com 49 net 29 org 24 au 3 cc 2 cn 1 to 1 default 1 some nameservers poison more than one zone. List of some poisoners on slide 12. dns.internic.ca ns1.afternic.com ns0.directnic.com ns1.domainsarefree.com etc. Never attribute to malice what can be adequately to be explained by stupidity Many of the nameservers that return bad referrals appear to be companies in the DNS business registrars resellers speculators typo profiteers others appear to be legitimate companies they should know better many of the names leading to poisoners are either expired or parked Is the sky falling? with so many poisoners out there, why don't we hear more about the problem? Most implementations don't allow root to be poisoned If you were surfing the web with poisoned DNS cache, would you know it? let's simulate it... for every bad referral found, we put the nameserver's IP address go to www.google.com go to www.microsoft.com see what you get. bbns01.secureserver.net, for example, happily pretends to be google.com. dns.domainsatcost.ca is amusing, because their ads are from google, even as they hijack it. a.ns.nameflux.com at least does an HTTP redirect dns2.nai.com doesn't return any A record, so you at least know *something*'s wrong. More examples follow... ns1.frakes.net ns.pairnic.com "smart people use pairnic for DNS"... Duane would beg to differ. 65.75.128.178.com returns an amusing message that is clearly wrong, blaming the clients for the traffic the DNS server itself is causing. Lazy, Stupid, or Evil Laziness: ns1.hi2000.com The admin is too lazy to put each domain delegated to them into separate zone files. Instead, they create a com zone and list A records for each delegation. Laziness such as this is probably the source of most of the poison out there. (includes guess at what their zone file looks like) Stupidity: ns1.frakes.net Typos, combined with laziness, create an interesting situation. Looks like Frakes.net is using the com zone technique, but forgot to make the nameservers fully qualified. Note that ns1.com etc are legitimate DNS names and have A records different than those returned by ns1.frakes.net. Just forgot the dot after the trailing name on the NS record. Evilness: Our definition of an evil poisoning nameserver is one where it answers queries with the wrong address, and maybe proxies web traffic sent there so you get what you (mostly) expect. To help find them, give each source of poison an evilness ranking from 1-5, with one point for each issue below: Returning bad referral poisoning a TLD Answering an A query for "important names" Answering query incorrectly Answering the query such that the web browser looks like it *might* be correct DNS A few fours, no fives. Miscellany Some of the poison sources that we find are actually vulnerable implementations that hve been previously poisoned by someone else. Remember: authoritative nameservers should NEVER accept recursive queries!! Some NS records have non-FQDN names. The name "ns" is a popular example. It's a good thing even the vulnerable implementations don't let the root zone become poisoned. Bottom Line: Several hundred misconfigured nameserves out there that return bad referrals that can poison DNS caches About 75% try to poison root zone, which usually has no effect Probably 90% of nameservers out there today are not vulnerable to this type of poisoning Most attempted poisonings http://dns.measurement-factory.com/surveys/poisoners.html wessels at measurement-factory.com Question from microphone: what do you do when you find these poisoned nameservers? And how often do you run the scripts to avoid false positives? Right now, he doesn't do anything other than put it in the database seen above. He tried contacting one, but never got any answer back. Contacting people is very hit or miss; hard to get responses from people. For second question, he's run the survey a few times; it takes about 10 days to go through the 6 million names, so he might try running it once a month or so. He doesn't think of any cases of "false" positives; they may fix their config, but if it was positive, it's not "false". Applause, and onto the next talk.

1 0

NANOG36-NOTES talk 1--steve feldman
by Matthew Petach 13 Feb '06

13 Feb '06

Based on generally positive feedback from many people, I'll be posting my notes from the conference. I'll preface the subject line with NANOG36-NOTES, so if you want to mass-skip the thread, it should be easy to do so. 2006.02.13 NANOG36 day 1 Opening/welcome to Dallas Steve Feldman starts off--many people are still trapped out east unfortunately. Steve Feldman, Program Chair, CNET networks. Texas Compact Car == SUV Our Host: Brokaw Brian Mike Todd Parker Raj Patel Brad Parker Thanks to NANOG program commmitted List of them went too quickly Agenda Changes--Tuesday 12:10--12:25...went too fast to see. Agenda Changes Wednesday 9:30-10 Hurricane Katrina: Telecom infrastructre, impacts, solutions, and opportunities... (more on slide) Reminders Network Security con't use cleartext passwordss do use end-to-end encryption (ssh, VPN) PGP Key signing see link off wwww.nanog.org for details Yahoo Reception Bear and Gear reception Interpreting Badges. Blue -- steering committee Yellow -- program committee green badge -- yellow plus blue, both committees green dot: peering black dot: security red dot: PGP signer RED badges will be for mailing list panel members Lighting Talks six 10-minute slots available Criterion: on-topic for mailing list Signups start now! http://www.nanogpc.org/lightning Random acceptance of submissions made before 2pm Monday Submission order after that (if slots remain) That's it for Steve Feldman, next up is Brokaw Price from Yahoo. Welcome to Dallas, on behalf of Yahoo. Not many hotels in Dallas that can host a group of this size with two ballrooms (one for general session, one for beer and gear; and the Hyatt was the only other one with space, and they're booked since Katrina wiped out other conference spaces in the south.). There's a trolley one block over that will take you to the downtown restaurant areas, it's free, feel free to take it and explore the area, note it stops running around 9:30pm. It does quiet down after dark in downtown, unfortunately. One thing really needed for a NANOG is a really good sized Internet link. The hotel had a pair of T1s to start with, and they were getting a second pair when Yahoo began pulling the fiber in for NANOG. Terminal room is now virtual; feel free to use the laptops and printers should you need to print documents, boarding passes, etc. The laptops are cabled down, but if you need to borrow one, just track Brokaw down, and he'll take care of you. He's been feeling like he's part of the garment industry in doing all the gear (two sets of tee shirts, and the fleeces for people who peer with Yahoo--if you don't already peer with Yahoo, jump in and send us email--peering(a)yahoo-inc.com) There will be an awesome party tonight at Eddie Deens, everyone should make sure to attend--it'll be fun, Texas style, and Texas sized. :) Many thanks to Mike Gallagher for doing the NANOG36 specific website with details on the local area and local options for attendees. Betty Burke asked him to say a few words about what it takes to put on a conference like this; in many ways, it's been like being a huge wedding planner, only weddings don't need large internet connectivity. Start planning early!! Brokaw thanks the Merit people for being so supportive; they've been complete animals, biting into the details with gusto; they're like true roadies, getting gear packaged and shipped, audio gear, video gear, cables, power, everything. Many thanks to Larry, Betty, Chris, Dave, Susan, Laurie, Dwayne, Steve, Tony, Tom, Greg, SC, PC, everyone else, it's all been completely worth it! It's definitely exciting times--we're building something huge; traffic levels are growing at near-exponential levels, datacenters are rolling out faster and faster. It's great being part of this community, and we all need to help keep it alive, to nurture it and helping it grow. If you haven't hosted a NANOG yet, definitely consider it; it's an interesting process. It starts off with "What's NANOG?" "What's the ROI on a NANOG?" Dan Golding joking about having a PBS-style thermometer graph showing how much new peering we get each day, to demonstrate the ROI for hosting a NANOG. But really, it's about sharing the support for the community--it's about stepping up to the plate, and saying "it's our turn to pitch in." There's 26 inches of snow in central park, which is keeping many, many of our colleagues away; and if you DO host, try to avoid Valentine's Day!! If you have any questions about hosting, feel free to call us. The good people at ATT have been great working with us. The fiber coming into the hotel terminates in the parking garage, but that was about 60 feet from where it needed to be. December 23rd, discovered the shortfall, and discovered how to pull innerduct in through a hotel on short notice. No matter what else happens for NANOG Huge thanks to Yahoo crew, especially Mike Gallagher, Brian Lacroix, Todd Parker, Raj Patel, Brad Parker, The whole ATT crew for their OptiMAN handoff Plan ahead! :) And make sure your brand people Jeannie Conduit(?), Jean Modie(?) are on board with your drawings and designs. There are bar rooms on either side of the bar, the Brassiere, and the Pyramid rooms can be used for hanging out if the bar gets too full, they have good connectivity. So, let's get it underway!! First up is Duane Wessels.

1 0

Lighting talks at NANOG 36
by Steve Feldman 13 Feb '06

13 Feb '06

NANOG 36 attendees may now submit lightning talk proposals at http://www.nanogpc.org/lightining See that page for details. Steve

1 1

Any interest in notes from the talks at NANOG?
by Matthew Petach 13 Feb '06

13 Feb '06

Since there are several attendees that are snowed in and won't be able to make it to Dallas for NANOG, I was thinking of posting my notes from each presentation to the nanog list, so those who are stranded can follow along from home. Would that be of interest to the list, or would it be just so much more useless spam-like fodder to be deleted? So far, the response to my notes from last night's community meeting has been positive--but that's only 2 people, so I don't know if I've helped 2 people, but alienated 8,036 more in the process. ^_^; Let me know if you'd be interested in having me post the notes. Thanks! Matt

1 0

Gmail multiple messages on mailing lists - resolved
by Mark Foster 13 Feb '06

13 Feb '06

>From the Sysadmin of the box: "Resolved, in theory - we were being hit by a bug in SpamAssassin 3.1.0." I am not sure what exactly about Gmail made SpamAssassin go nuts... but it is resolved for now. Thanks to everyone who assisted, including Google, who were pretty quick to at least acknowledge my report. Now, if only their details were on puck.nether.net :) Mark.

1 0