- NANOG - lists.nanog.org

Re: Day tickets
by Brandon Butterworth 12 Jun '06

12 Jun '06

> For the RIPE meeting, this has been solved by introducing day tickets. RIPE is a whole week at Butlins[1] to Nanogs' day by the sea brandon [1] http://en.wikipedia.org/wiki/Butlins

1 0

Working on a long-term ipv6 multihoming solution
by Vince Fuller 12 Jun '06

12 Jun '06

This is a follow-up to my and Jason's presentation from Wednesday. Several people mentioned in the hallways that they were interested in following this issue and possibly helping work on the solution. If you are one of them and haven't already seen a message subscribing you to the mailing list, please send email to: ipmh-interest(a)external.cisco.com It would be helpful to indicate whether you are interested in following the work or in actively participating in it. Note that "ipmh-interest" is not the actual mailing list (there will be two separate lists established: "ipmh-discuss", which is a open mailing list to follow and track the work and "ipmh-design" which will be a closed list for a small, focused design team. If anybody has issues with this list being hosted at Cisco (it is most definitely not a Cisco-only effort and shouldn't be perceived as such; I'm only hosting it there for convenience), please provide that comment too; if there is strong consensus, I'll be happy to move the list to a more "neutral" place. --Vince P.S. Jeff Burgan: can you please contact me offline? It looks like your employer's email system is misconfigured and is bouncing all mail that I attempt to send to you.

1 0

to DLV or not
by Edward Lewis 12 Jun '06

12 Jun '06

My background and position on this is best summed up as one of the early implementers of DNSSEC and now working for a gTLD/ccTLD registry. In between I spent a lot of time developing, redeveloping DNSSEC in the face of operational realities. (To those who are critics of DNSSEC, I ask forgiveness for my wasted middle-age.) DLV is a concept that someone somewhere in the past few years came up with to put needed DNSSEC data in a special location. Although DLV per se is novel in the development of DNSSEC, it is well in-line with the earliest intentions of the protocol dreamers. The original DNSSEC design was to allow any resolver (client) to decide how it would collect the needed records to substantiate an answer. In the mid to late 90's we tried to figure out how to first express a policy and then come up with something that could take the policy and direct the operation of a DNS validator (the thing that gives a thumbs up or down to an answer after checking the DNSSEC stuff). We punted, resulting in RFC 3008, which said that the only "common" policy was to follow the tree, i.e., root signs tlds, tlds, sign down, etc. A few years later, a project called FMESHD to reopen the policy to be more general. Again, the problem proved too big to solve. Why DLV is different from these two failures is that we had been trying to solve the general case without a validator in hand. (We did have validators, but nothing that was integrated with a real name server.) DLV is starting from an implementation and is a pragmatic attack on the problem. DLV is not as general as the original idea, but wider than RFC 3008. The main concern with DLV is that is it not scaleable. That's inherent in the problem so I am not surprised. The tradeoff is that you can go "off the tree" but at the cost of "knowing where you are going off the tree." Some folks feel that DLV will compete with TLD registries and delay their interest. Or maybe, for the same reason, spur their interest. My opinion is neither, DLV is orthogonal to the TLDs. DLV may be a good measure of interest in DNSSEC though. Either there will be no interest, a quick spike in interest, or a sustained growth. My guess is the middle option - a lot of early registrations and then slow growth. If that's the case, scaling isn't the concern and it won't spur the registries to add DNSSEC. So, ISC's DLV operation? The developers of BIND are free to distribute code with a validation policy that looks up data in their DLV. If all works, then all is good. If it's a disaster, ISC's DLV will cease and the code will cease to have the feature. "Let the market decide." I don't think that what the pro-s and anti-s *think* matters as much as having tangible data on what *happened*. If the techies still rule the Internet, something like DLV ought to be given a try. Show off a technical solution and use that as an example of the way forward. We've been stalling long enough trying to move policy makers to sign the root zone. -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Edward Lewis +1-571-434-5468 NeuStar Nothin' more exciting than going to the printer to watch the toner drain...

1 0

Re: 2006.06.07 NANOG-NOTES TCP Anycast--don't spread the FUD!
by Matthew Petach 12 Jun '06

12 Jun '06

On 6/12/06, Rodrick Brown <rodrick.brown(a)gmail.com> wrote: > Looks like this document maybe have been removed? the link appears to > be dead any mirrors? > The slide deck hadn't been put online when I sent my notes; I took a guess at what the location might end up being, but guessed wrong. The actual location ended up being http://www.nanog.org/mtg-0606/pdf/levine.pdf Matt > -- > Rodrick R. Brown > Senior Systems Engineer > http://www.rodrickbrown.com > http://groups.yahoo.com/group/wallstandtech >

1 0

Zebra/linux device production networking?
by Nick Burke 12 Jun '06

12 Jun '06

Greetings fellow nanogers, Long time lurker, first time poster (please, be gentle!). After looking at the archives, I didn't see this particular discussion, so here we go. First, a little background.. My CTO made my stomach curdle today when he announced that he wanted to do away with all our cisco [routers] and instead use Linux/zebra boxen. We are a small company, so naturally penny pinching is the primary motivation. That, and the sheer joy of watching me squirm. He has informed me that he has found "many people" who do this for their "core devices". I'm not so certain about this whole situation, so I humbly ask: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? And, if you care to spend this much time, what pitfalls/benefits did you find out about after implementation? Has there been any discussion (or musings) of moving towards such a solution? I've seen a lot of articles talking about it, but I've not actually seen many network operators chiming in. Here's the article that started it all (this was featured on /., so likely you've read it already). http://www.businessweek.com/technology/content/nov2004/tc20041129_5206_tc02… and another: http://www.networkworld.com/community/?q=node/5693 Feel free to respond off list. If anyone else is interested, I will of course summarize to list or to individuals. (ps, particulars are deliberately not included.. I'm not looking for advice, just if anyone has any solid experience with this..)

17 19

Extreme Networks BD 6808 errors -- help to interpret.
by Mattias Ahnberg 12 Jun '06

12 Jun '06

I've recently stumbled over an error in the logs of one of my Black Diamond 6808's. Due to redundant MSMs this hasn't had any practical effect yet, but I have just initiated a ticket on the matter. Parallell to that I thought I'd take the oportunity to ask if anyone here has any more insight on what this means, what is happening and possibly what could cause it. Is it simply a matter of a broken MSM, or what could trigger something like this? Replies onlist or offlist appreciated, whatever you prefer. Thank you in advance, SYST: The signature of NMC0 is disappered. Reset the NMC10 SYST: Removed MSM-B base module SYST: slave MSM communications pipe is closed SYST: MSM-B is inserted. SYST: Start initializing MSM-B base module SYST: MSM-B SN = 701021-17 0235F-70778 SYST: slave MSM communications pipe is open SYST: Done initializing MSM-B base module SYST: MSM-B HW_AN=0 SW_AN=0 DECODE=0 INTSTAT=0 ANRCVCFG=0 CTRL=10000000 SYST: slot 3 HW_AN=a1 SW_AN=13 DECODE=1b INTSTAT=bf8 ANRCVCFG=41a0 CTRL=ff3ffc00 SYST: MSM-B=[701021-17 0235F-70778 ] SYST: slot 3=[701026-15 0315F-80358 ] SYST: backplane=[701000-08 0230h00123 ] SYST: [2] Connection between MSM-B mother board port 2 and I/O module 3 port 1 broke, fix immediately SYST: MSM-B HW_AN=0 SW_AN=0 DECODE=0 INTSTAT=0 ANRCVCFG=0 CTRL=10000000 SYST: slot 4 HW_AN=a1 SW_AN=13 DECODE=84 INTSTAT=bf8 ANRCVCFG=41a0 CTRL=ff3ffc00 SYST: MSM-B=[701021-17 0235F-70778 ] SYST: slot 4=[701024-21 0238F-70372 ] SYST: backplane=[701000-08 0230h00123 ] SYST: [2] Connection between MSM-B mother board port 3 and I/O module 4 port 1 broke, fix immediately SYST: MSM-B HW_AN=0 SW_AN=0 DECODE=0 INTSTAT=0 ANRCVCFG=0 CTRL=10000000 SYST: slot 6 HW_AN=a1 SW_AN=13 DECODE=1b INTSTAT=bf8 ANRCVCFG=41a0 CTRL=ff3ffc00 SYST: MSM-B=[701021-17 0235F-70778 ] SYST: slot 6=[701026-15 0315F-80383 ] SYST: backplane=[701000-08 0230h00123 ] SYST: [2] Connection between MSM-B mother board port 5 and I/O module 6 port 1 broke, fix immediately KERN: Secondary Mgmt port initialized KERN: Secondary Mgmt port enabled SYST: The signature of NMC0 is disappered. Reset the NMC10 SYST: Removed MSM-B base module SYST: slave MSM communications pipe is closed SYST: MSM-B is inserted. SYST: Start initializing MSM-B base module SYST: MSM-B SN = 701021-17 0235F-70778 SYST: slave MSM communications pipe is open SYST: Done initializing MSM-B base module KERN: Secondary Mgmt port initialized KERN: Secondary Mgmt port enabled SYST: The signature of NMC0 is disappered. Reset the NMC10 SYST: Removed MSM-B base module SYST: slave MSM communications pipe is closed SYST: MSM-B is inserted. SYST: Start initializing MSM-B base module SYST: MSM-B SN = 701021-17 0235F-70778 SYST: slave MSM communications pipe is open SYST: Done initializing MSM-B base module KERN: Secondary Mgmt port initialized KERN: Secondary Mgmt port enabled -- /mattias ahnberg (AS20514).

3 3

a fun hijack: 1/8, 2/8, 3/8, 4/8, 5/8, 7/8, 8/8, 12/8 briefly announced by AS 23520 (today)
by Josh Karlin 10 Jun '06

10 Jun '06

Check out the IAR for "Potential Prefix Hijacks" and if you're coming to this more than 24 hours after the post, do a search on AS 23520 as the hijacking AS. I don't know how long the routes were announced, but they seem to be gone now. Or maybe the IAR is horribly broken, in which case I will be lynched :) IAR: http://cs.unm.edu/~karlinjf/IAR/ Josh

9 11

2006.06.05 NANOG-NOTES BGP tools BOF notes
by Matthew Petach 10 Jun '06

10 Jun '06

(ok, last set of notes for tonight, and then it's off to bed for 90 minutes of sleep before heading back to the convention center. ^_^; --MNP) 2006.06.05 Welcome to the 4th BGP Tools BOF! [slides are at http://www.nanog.org/mtg-0606/pdf/lixia-zhang.pdf Nick Feamster GeorgeTech Dan Massey CUS Mohit Lad and Lixia Zhang, UCLA The Goal sharing some tools develop from our research efforts. hopefully will be useful for operations community. Also to collect input on new tools we would like to see so they can develop them. Routing Configuration Checker Nick Feamster O-BGP data organization tool Dan Massey [slides are at http://www.nanog.org/mtg-0606/pdf/dan-massey.pdf The Datapository by Nick Feamster [I'm sorry, that just sounds *far* too much like something you do *NOT* want your bedside nurse administering...--MNP] Visualizing BGP dynamics using Link-Rank by Mohit Lad Open discussions and demos Nick Feamster Network Troubleshooting: rcc and beyond rcc: router configuration checker proactive routing configuration analysis idea: analyze configs before deployment many faults can be detected with static analysis. rcc implementation. http://nms.csail.mit.edu/rcc/ preprocessor -> parser -> relational database (mySQL), constraints <-> verifier <-> faults verifier is a template checker and set of constraints your configs are checked against. He's looking for GUI developers. very bare-bones command line right now. Parsing configurations--shows some output. He shows examples of the abilene configs, which are non anonymized. show all routers peering with a given AS, can look at route maps in each direction, etc. After running rcc on it, you get a web output which shows relationships--oh, pictures don't matter, with some more grease could be a reasonable representation of your network. Q: Randy Bush asks if it could show which peering sessions are missing? A: Not yet, but it could be added, thank you! Shows processing and errors; you get a page that summarizes the things RCC thinks are errors. Signalling partition? that's a missing iBGP session; he needs some better lingo in places. Also shows anomalous imports, could be intended for traffic engineering; that's "inconsistent policy" in ISP speak. Some of the names will get fixed to make Randy Bush happy. Yes, but surprises happen! link failures node failures traffic volumes shift network devices "wedged" ... two problems detection localization Need to marry static config analysis with dynamic information (route is configured but isn't in the dynamic table) he skips a closer look, just some jargon. Detection: analyze routing dynamics; drill down on interesting operational issues. idea: routers exhibit correlated behaviour blips across signals may be more operationally interesting than any spike in one signalling system. How do you spot things in the churn? Detection three types of events single-router bursts correlated bursts multi-router bursts <---common; and commonly missed using simple thresholds Localization: joint dynamic/static which routers are "border routers" for that burst topological properties of routers in the burst. proactive analysis -> deployment -> dynamic -> reactive detection -> diagnosis/correction -> static -> By going back to the configs, lets you see if it's something happening inside the network, or on the edge. Specific Focus: firewall configuration difficult to understand and audit configs subject to continual modifications roughly 1-2 touches per day federated policy, distributed dependencies each department has independent policies local changes may affect global behaviour (These are pulled from Georgia Tech; 130 firewall configs. Builds static connectivity matrix.) Reactive monitoring...use probes from subnets to verify reachability/connectivity. (immediate) open issues reachability and reliability of controller service-level probes diagnostic tools != service-level happiness policy conformance. Q: can it give suggested remediation, or provide config templates for new routers being added? A: Good idea! OK, over to next presenter. Helps with understanding BGP data. BGP data collection and organization (OBGP) Tool Colorado state university/university of Arizona/UCLA BGP data collection takes lots of BGP data, from RIPE RIS, etc. ISP BGP peer router -> update oreg -> rib+update -> feeds into gigabytes of data, different formats, potential errors enter in, and severe lack of metadata. Other tools can use it, LinkRank, BGP-Inspect, and a bunch of people cite it in reports and research. OBGP motivation Large Volume of Data data from many sources (RIPE, RV, private data) Long time scales and very recent (real-time?) data Slightly different formats RIPE/RV use different naming conventions different dump intervals different timezones for older data Lack of MetaData would like to only see desired peers and desired update types Possible errors in the data are updates missing due to log errors? what is lost due to session failures? So, OBGP is the "thing" in the middle. A simple perl script called oBGP that simplifies data. Features: Uniform data organization consistent and easy to use for scripts consistent view of multiple monitoring points annonatations/labels can be stripped, help locate useful data easily table transfer detection distinguish updates from data collection peering Data inconsistency detection and correction understand and fix possible data errors Uniform data organization Uniform naming and organization conventions for all monitoring points RIB and update data split by peer One rib and update file per peer per day, dumped at beginning of the day. Labels and Annotations are more interesting Existing format labels update as announce (A) or Withdraw (W) also includes some STATE messages OBGP enhances the labels Adds a status message Adds an update type More STATE messages route table dump table transfers A:INC:DPATH (shows it's an announcement, it's incremental, and it's updating the destination path OBGP Added labels |<original update type:<status info>:<OBGP udate type>| <orginal update type> add E for error correction <status infor> INC incremental update TT table transfer update RIB: correction update <OBGP update type> new announcement duplicate announcement change in AS path (DPATH) change in other attribute (not ASpath) withdraw duplicate withdraw If you don't need this, it's just a few extra characters in your log; but could be useful. Using Labels to filter data example: find suballocation hijacks Only need new announcements and withdraws so 83% of the update data can be ignored. Is the collected data accurate? May lose updates due to data collection errors start with an accurate RIB apply updates in log should match the next RIB dumped by the router modulo some race conditions near dump time does this clearly work with RouteViews? 85 of 111 peers from RV suffered inconsistencies in 2006 May About 25 were rock solid right on. One peer had 378,998 inconsistencies in one day. Is this evenly distributed? Not really. Inconsistencies and session failures session down: RIB-IN drops to empty session up: table transfer (failure to recognize a session dropping) look for table transfer, can estimate where sessions went down and came back up. How long does an error persist? Lifetime of correction updates can last 43 days! If you miss an update, you can have bad data for a long, LONG time!! Correction updates added by OBGP E:RIB updates; figure a change in RIB had to happen due to a routing update that was missed. Summary: consistent format adds label to easily sort and limit adds additional state messages identifies and corrects update error messages http://netsec.cs.colostate.edu/tools.html [NOTE URL at end of slide deck is WRONG --MNP] If you're using RouteViews or RIPE RIS, consider using this tool, and give feedback! Randy is using it to check propagation of his prefixes, and for research. RIPE NCC--performance of these tools? With multiple collectors, perl didn't scale. Perl is mainly demonstration. He pulls data from RIPE and has it stored, hopes to make it public some day. he has stacks of disks with text format data for easy search; considering binary format for it. Randy--on that subject: Matt Rowan, he's spent half his life getting the data out of the system; make it in funny format and sticking it back in, Disk is cheap! Look at raw data. With binary data, what tools are there? Hard enough to look at router configs. One tool to look at binary data, lots of tools to look at text. Q: Matt asks how much space it takes to store data A: Takes about 1TB to store all the RIS data. Q: Are they planning to make it available to the public? A: Well, he'd like to host it at route-views or ripe, rather than create a new site. How long does it take to process the RIPE data? Need a fast CPU, will take a couple of days to process the data. Q: can it deal with live updates? It can keep up with route-views and RIPE, but that's not live; there is a lag; route-views is every 15 minutes. The update files sometimes take 8 hour lags to show up on the site. The Datapository Nick Feamster and David Anderson? Architecture: raw data -> compute engines-> storage and DB plus archival storage ->analysis. Very alpha right now datapository.net NOT realtime! inserting data in greedy approach; when he needs it, he inserts it, and starts running queries. You can see a list of feeds, he has abilene but not route-views yet. Can restrict it, look at neighbor ASes, etc. see it in graphical form, or list form can diagnose issues, has an XML query engine and output for programatically accessing it. If you use matlab, could be interesting to throw this into a multidimensional time series. Randy Bush notes all his tools take MRT output. Oh, he can spit out sparse matrices He could spit out MRT format; he has python that speaks MRT format. he'll look at adding that. Do spammers hijack BGP routes? Theory: 1 announce BGP route for mail server 2 send lots of spam 3 withdraw route, becoming invisible reality? let's check! export formats Web interface XML/RPC text-based output programmatic interface output to matlab and per Randy Bush, MRT format would be good too! BGP-Inspect vs this tool? this has additional datasets beside BGP, like active probes, traffic, etc. This has a better collection setup as well; unified formats. Mohit will do last one, Link-Rank show the dynamics Visualizing BGP dynamics with Link-Rank constructing rank-change graphs closest to BGPlay. weight is number of prefixes reached across that link. weight changes are on specific links, can do easy root-cause analysis. Activity bar--routing activity across time. http://linkrank.cs.ucla.edu/ green shows gains, red shows losses, sums all gains, sums all losses. visualization graph of where prefixes gained and lost. again, green are links that gain, red are links that lost. other observation points highlighted in orange May 23rd, instability 293 flapping from 1239 to 3356 for multiple observation points, dashes are lost, solids are gains. highlight sources and sinks cutting one link explains most of the errors 3561 to 4134 link issue case II, one node that sucks in all the flows, no single link, 3356 Ongoing work, automated root cause identification min-cut scheme characterizing Can look at destination link-rank graphs, see how the rest of the internet change going to you. connectivity issues to 7018 should show your prefix hijackings moving from one link to another. Could simplify this to BGPlay if you wanted. open source http://linkrank.cs.ucla.edu/ current version download client, configure... Future version work with any BGP data Q: Matt asks when we'll be able to use our own BGP data; A: about 4-5 months, hopefully! Haven't looked at netflow yet. Q: Randy Bush. Common problem we all face. I'm at 42 peering points; my neighbors are X. I have route views dumps, I have my BGP dumps. I have my netflow data. Want a whatifatron that shows what happens to my traffic if depeer someone, or add someone, or peer with SingTel in singapore, or stop peering with Joe in SF. That's a question many operators ask every day. A: Matt notes that if they can solve that question/write something that does all that, they'll have Arbor and others beating on their door. ^_^ Panel wraps up at 1728 hours Pacific time.

3 2

2006.06.07 NANOG-NOTES TCP Anycast--don't spread the FUD!
by Matthew Petach 10 Jun '06

10 Jun '06

(this was one of the coolest talks from the three days, actually, and has gotten me *really* jazzed about some cool stuff we can do internally. Huge props to Matt, Barrett, and Todd for putting this together!! --Matt) 2006.06.07 TCP anycast, Matt Levine, Barrett Lyon with thanks to Todd Underwood TCP anycast, don't believe the FUD Todd Underwood is in Chicago Barrett Lyon starts off. [slides may eventually be at: http://www.nanog.org/mtg-0606/pdf/tcp-anycast.pdf IPv4 anycast from a network perspective, nothing special just another route with multiple next-hops services exist on each next-hop, and respond from the anycast IP address. It's the packets, stupid perceived problem: TCP and anycast don't play together for long-lived flows. eg, high-def porn downloads [do porn streams need to last more than 2 minutes?] some claim it exists, and works... yes, been in production for years now. Anycast at CacheFly deployed in 2002 prefix announced on 3 continents 3 POPs in US 5 common carriers (transit) + peering be sensible to who you peer with Effective BGP communities from upstreams is key keep traffic where you want it. Proxy Anycast proxy traffic is easy to anycast! move HTTP traffic through proxy servers. customers are isolated on a VIP/virtual address, which happens to exist in every datacenter. Virtual address lives over common carriers allowing even distribution of traffic state is accomplished with custom hardware to keep state information synchronized across proxies. Node geography anycast nodes that do not keep state must be geographically separated Coasts and countries work really well for keeping route instability largely isolated. Nodes that are near by could possibly require state between them if local routes are unstable. IP utilization "Anycast is wasteful" people use /24's as their service blocks; use 1 /32 out of a whole /24. Really? How much IP space do you need to advertise from 4 sites via unicast? Carriers and Peering for content players, having even peering and carriers is key. you may cause EU eyeballs to go to CA if you're not careful with where you peer with people. having an EU centric transit provider in the US without having the same routes in EU could cause EU traffic to home in the US Use quality global providers to keep traffic balanced. When peering... keep in mind a peer may isolate traffic to a specific anycast node Try to peer with networks where it makes sense; don't advertise your anycast to them where they don't have eyeballs! Try to make sure your peers and transit providers know your communities and what you're trying to do, and make sure you understand their communities well! Benefits of Anycast. for content players moving traffic without major impact or DNS lag provides buffers for major failures allows for simplistic traffic management, with a major (potential) performance upside. it's BGP you don't control, though, so not much you can do to adjust inbound wins. HTTP has significant cost to using DNS to try to shift traffic around; six or more DNS lookups to acquire content; anycast trims those DNS lookups down significantly! Ability to interface tools to traffic management. No TTL issues! Data, May 9, 2006 Renesys: monitored changes in atomic-aggregator for a CacheFly anycast prefix AS path changes and pop changes Keynote: monitored availability/performance of 30k file Revision3: monitored behavour of "longlived" downloads of DiggNation videocast--over 7TB transferred. Renesys data: 130BGP updates for may 9th; low volume day stable prefixes 34 distinct POP changes based on atomic aggregator property on prefixes 130 updates is considered a stable prefix. SJC issue: thirty-five minute window, 0700 to 0735 UTC, saw: 98 updates, 20 actual pop changes based on atomic aggregator changes, all from one san jose provider, fail from SJC to CHI back to SJC unable to correlate these shifts with any traffic changes; mostly likely we don't have a big enough sample size. possibly just not a lot of people using those routes. BGP seems stable--what about TCP flows? AVG time between SJC and CHI and back again was about 20 seconds; very quick on the trigger to go back to SJC; would break all TCP sessions happening at the time. For the most part, TCP seems stable. Keynote: 30k download from 31 locations every 5 minutes, or average of 1 poll per 9.6 seconds compared against 'Keynote Business 40' data collected on May 9, 2006 represents short-lived TCP flows, though. Orange line is Keynote business 40 pegged 100% availability load time was lower than the business 40. (0.2s vs 0.7s for business 40) Revsion 3 data monitored IPTV downloads for 24 hours (thanks, jay!) span port; analyzed packet captures look for new TCP sessions not beginning with SYN compare that against global active connection table. looked for sessions that appeared out of nowhere. Long-lived data 683,204 TCP sessions. anything less than 10 minutes thrown out 23,795 sessions lasted longer than 10 minutes. average file size for the day was 300MB 4 TCP sessions moved between POPs. 0.0006% total pop switch 'failure' rate 0.017% for long-lived (more than 10 minutes) sessions. only looking for sessions that starts in A, moves to B; they dropped 0 of them, due to their state preservation mechanism. you would have dropped 4 out of 23,795 connections with this without state preservation mechanism. Anycast gotchas. large scale changes in provider policies can impact your traffic, up to you to figure out what happened. "Things that are bad" become much, much worse, notably per-packet load balancing across provider or topologicl boundries. eg customer with 2 T1s between Anaheim and Dallas Sprint POPs. per-packet load balancing across the two, before the anycast nodes shared state information between nodes was probably still seeing performance issues. conclusion: stateful anycast is not inherently unstable, and failure/disconnect rates are inline with offering unicast services this is counter-intuitive to some published conclusions from previously published data. some other company did work in this; is there other failure rate data available? Verisign said to not do TCP anycast, but they didn't publish failure rates for TCP. Need to see where TCP really, really breaks. "Trust us, it works" widespread failures cause havok; however, internet doesn't go crazy *that* often. Transitioning to IPv6: as for the move to IPv6, there *is* a plan for it; plan is to hope they're dead by the time customers actually demand v6. what you can do stop telling people TCP doesn't work if you haven't tried it yourself! Just makes them mad to hear it. if your application doesn't handle TCP/IP failures gracefully, don't run anycast, in fact don't run it on the internet at all. SMTP and HTTP work well; browsers support reset and retries, for example Experiment with other applications share your experiences--want to know if their results are anomalous, if they're crazy, or if this really does just work. Q: Mark Kosters, Verisign: did one presentation. It depends on client base; their client base was very far flung, many problems in far reaches of internet and per-packet load balancing showed up in their data. How big a clientele spread do you want to reach, do you hit core, or far edge? A: they try to reach customers where they get money for the material being served; not necessarily geared for global connectivity. Yes, unicast in outlying cases will be more robust, but doesn't scale as well. Q: Randy Bush, IIJ, 10th anniversary of much of the Anycast UDP deployment. Response of "it's good if you engineer it" applies to most things. They wanted to narrow peering and topology; Randy says there are exactly cases where that is exactly NOT true. In 1996, they needed to shed traffic off their backbone; had to support streaming, they did TCP from all anycast to all peers, exact opposite of what they were calling for. A: This talk is specific to content they deal with directly, was engineered for their customer's content and needs. Main point is that it's not as bad as people claim, and benefits can be substantial. And yes, you will need to engineer it to your particular case. Q: Bill Woodcock: over past 10-12 years, many have seen long-lived TCP connections with 0.01% failure rates; Mark's results with j-root were suprisingly different. Methodologies were sound, so why might we be seeing these bimodal results? A: different results trying to be achieved. Streaming video vs universal reachability for DNS. Perhaps worth looking at the specific needs being aimed for? Dalnet runs with very long lived TCP connections, some last for months, they're not picky about who they get transit from, and they're anycasting their IRC servers. Q: Danny McPherson, 10 years ago people were using this to try to pass stuff faster to try to make Keynote numbers look better. Origin AS, announcing prefixes from multiple places, may trigger security, prefix hijacking alerts; the state sharing they do may have helped considerably. A: Not as scary as people thinks it is. some open source state synchronization tools would be nice. If someone can do some work in that area, they'd love to support it. Q: Michael ?, UCB: thanks, good to see studies like this! Not all applications/protocols can handle this; hard to necessarily then generalize this to "TCP" in general like that. A: they meant application more than protocol. Q: TCP has methods of dealing with out-of-order packets. not many people use per-packet load balancing; you don't know what might change in future, though. Q: Matt Peterson, can you give description of size of files being moved around? Also, state mechanisms--you mentioned open source--is it something you might consider releasing? A: content, don't remember exactly, look at Revision3, see their show. Average file size was 350Mb, range from 200-650MB. For their state, they hacked stuff together themselves; they'd be happier to support something community based, rather than release their code. any more Q ask them offline, we HAVE to keep moving to get out of the convention center ontime!!

2 1

2006.06.07 NANOG-NOTES Smart Network Data Services
by Matthew Petach 10 Jun '06

10 Jun '06

(I'm starting to guess I'd finish sending these out faster if I stopped falling asleep on my keyboard so often... --Matt) 2006.06.07 Welcome to Wednesday morning http://www.nanog.org/ click on Evaluation Form Let us know how the M-W vs S-Tu format; next time will be S-Tu due to ARIN joint meeting, but need more feedback! Bill Woodcock, been on program committee And lightning talk people need to send their slides to Steve Feldman!! Elliot Gilliam, ISP community, notifications to Smart Network Data Services [slides are at http://www.nanog.org/mtg-0606/pdf/eliot-gillum.pdf AGENDA postmaster services SNDS problem goal today tomorrow motivation feedback/dialog questions/discussion Postmaster--starting point for any issues you have sending mail into Hotmail/MSN Live. It's like AOL skunkfeed, you can do junk mail reporting. Lets you see what bad stuff is coming from your domain. SenderID Site is at: http://postmaster.msn.com/snds/ Problem: bad stuff on the internet (spam, phishing, zombies, ID theft, DDoS) makes customers unhappy. Solution #1 -- try to stop it before it hits customers doesn't really *solve* the problem Solution #2 -- take what we learn, apply it upstream, get more bang for buck #2: #1 is too low ISP-centric efficiency solution #1, n ISPs have n-1 problems, total is O(n^2) n ISPs have 1 problem (themselves), total is O(n) reduces work of the overall system. Crux today people and ISPs are measured by how much BAD stuff they *receive* Not judged by what they send out. similar to healthcare industry no tight feedback loop to ISP behaviour nice quotes on slides http://www.circleid.com/posts/how_to_stop_spam 7 step program (like 12 step, but shorter) 1: recognize the problem: SNDS 2: believe that someone can help you : Me 3: Decide to do something : You 8: Make an inventory of those harmed : SNDS 9: Make amends to them : Tools 10: Continue to inventory : SNDS 12: Tell others about the program : You What is SNDS Website that offers free, instant access to MSN data on activity coming from your IP space data that correlates with "internet evils" informs ISP to enable local policy decisions Automated authorization mechanism uses WHOIS and rDNS users are people not companies A force multiplier attempt. You can do it on your own, no need to sign up your company officially as long as you're an rWHOIS/WHOIS contact. SNDS goal: provide info which allows ISPs to detect and fix any undesired activity. qualitative and quantitative data "No ISP left behind" stop problems upstream of the destination Bring total cost of remediation to absolute minimum keep service free Make internet a better place. We have data! Windows Live Mail/MSN Hotmail is a spam and spoofing target. 4 billion inbound mails/day 90/10 spam/ham by filtering technologies User reports on spam, fraud, etc. Inbound mail system slide--ugly to read, too dark. SNDS website slide shown. You can see daily aggregated traffic from your network; activity periods, IPs, commands and messages seen on port 25, samples of exchanges. Filter results on your mail rate at which users press "this is junk" on your mail. Trap counts for when IPs hit their junk filters. comments column is catch-all for anything else they might put in; like open proxies, when tested positive. "export to CSV" button, so you can feed the data in to your own systems if you want. Today's Scenario Illustrate magnitude and evidence of a problem. additional resources monitoring infrastructure SNDS Stats 2500 users mostly senders 67 million IPs 10-20% of inbound mail and complaints Output drops by 57% on /24+ when monitored by SNDS SNDS tomorrow Usability signup by ASN better support for upstream providers access transfer Utility programmatic access Data virus-infected emails phishing honeymonkey sample messages Expand the the coverage, try to hit more of the problems on the net. Provide sample messages, compelling evidence when facing customers This hasn't shipped yet, it's what he's hoping to have in a month or two. Tomorrow's Scenarios Lowered barrier to entry recurring "cost" ISP types end-user tier 1/2 monitoring, tier 2/3 directly attack more than just spam virus emails -> infected PCs, outbound virus filters phishing/malware hosting -> takedowns. Is asymmetric routing a sign of people trying to launch hidden abuses of the net? Looking to hit more issues, like spotting virus-laden messages; either infected, or an open relay. Hoping that automation speeds response. Safety Tools Stinger: http://vil.nai.com/vil/stinger Nessus: http://www.nessus.org/ [oy, read the list from his slide, it's long.] green items on the list are free, others are pay-for products. Pay-for isn't necessarily a bad thing if you get benefit! Safety tool breakdown from MSN on next slide. Motivation: Hypothesis: everyone benefits Customers: infected uses get fixed safer, cheaper, better internet experience ISPs solution #1 isn't solving the problem altruistic is the "new" selfish Microsoft only benefits if everyone else does make business case why they're doing this. They need to stop paying costs of trying to deal with spam. Wants to get benefit of being one of the people seeing a cleaner internet ISP Motivation Customers they're unhappy, unsafe they like people who fix that be the hero retain customers win new ones fixing has more benefits than bandaging [bandaging is just sticking fingers in the dike, it doesn't scale, eventually we run out of fingers to stick in the holes] cost reductions bandwidth--slow growth demands support--fewer complaints to your help desk. Community NANOG Motivation alternatives Industry scorecard public recognition public shame Logo ISP program--how clean are you? Business case Some nice quotes from different people around the business case needed here. appeal to cost reduction and revenue generation this is starting to happen. let your sales and marketing people know about this. Boston university business case, students arriving with computers presented danger/load to their help desk. Qwest provides windows/one software to their users. Feedback: usability--how easily can you work with it? utility--what can you do? what's missing tools to aid customer remediation need IPv6 support at some point how do ISPs see cost vs benfits costs, benefits, NANOG aggregation how do we get critical mass? msn-snds(a)microsoft.com Discussion: How does SNDS fit into the larger ecosystem relationship to senderBase.org SCOMP/JMRP REACT adam, rick at support intelligence Yahoo is working on a system like this, Irene Lai is here to work on that, email her if you're interested. Should/how do other ISPs provide this? common schema, authorization, authentication federation, delegation, aggregation Forum bof/track? NANOG/MAAWG? Mailing list: upstream(a)mipassoc.org Conclusion: http://postmaster.msn.com/ http://postmaster.msn.com/snds/ Try it! tell people about it! Q: Matt asks whether Microsoft will point their own systems at it, since Nick Feamster's presentation showed on slide 12 that Microsoft was #10 on the list of spam *sources* that his honeypots saw? A: Yes, he is connecting the systems that track mail sending from Hotmail to this as well, so that they can start making sure they're cleaning their own house as well. on to next talk.

2 2