- NANOG - lists.nanog.org

Re: IPv4 Resource Distribution After IANA Free Pool Exhaustion -- ARIN AC BoF
by Owen DeLong 21 Feb '08

21 Feb '08

The proposal was posted to PPML, but, since the AC has not yet moved it forward to formal proposal status, it doesn't have a number and isn't on the ARIN web site just yet. The thread on PPML is available here: [ppml] Policy Proposal: IPv4 Transfer Policy Proposal Owen

2 1

survey for remote attendees, too pls
by Todd Underwood 21 Feb '08

21 Feb '08

hi, as we wrap up nanog42 in san jose, we are bugging everyone in the meeting room to submit their surveys. i'd like to also explicitly request that remote attendees via the webcast (even if you only saw some of hte conference) please fill out the survey at: http://www.surveymonkey.com/s.aspx?sm=150FgSnFemffuGMOYbcLBQ_3d_3d your feedback really does tell those of us who put this conference together whether we are doing a good job and what you might like to see in the future. thanks! t. -- _____________________________________________________________________ todd underwood +1 603 643 9300 x101 renesys corporation general manager babbledog todd(a)renesys.com http://www.renesys.com/blog

3 2

2008.02.20 NANOG 42 IPv4 PTR queries for unallocated space
by Matthew Petach 21 Feb '08

21 Feb '08

Notes from day 4 of NANOG 42, we're on the home stretch now, so there's only a bit more of my spewage to wade through before we can resume our normal discussion of the imminent death of the internet as we know it. :) Again, apologies for typos, misspellings of names, etc. Matt 2008.02.20 opening remarks and analysis of PTR queries for IPv4 reserved addresses Trying to measure use of unallocated IPv4 address space Starting with Leo Vigoda, working at IANA now. leo.vigoda(a)icann.org At last NANOG, did lightning talk about how people were using unallocated space, but didn't show lots of data. Now, has a way to hopefully measure IPv4 addresses that people are trying to use. Not all the addresses, but at least one view of it. Problem? All unallocated unicast space will eventually be allocated; no more free /8's in the IANA pool at some point, as v6 won't get rolled out in time. Some people are using the same addres space that's currently in the 'unallocated' pool. Tried to measure it by looking at DNS queries at l.root for addresses in the unallocated blocks. root servers provide in-addr.arpa delegations, if a /8 isn't allocated, there's no delegation, so the root servers see the queries. Can't measure well-maintained networks with split-horizon DNS, or with good egress filters. Doesn't know how to measure truly private internal use; if you have ideas, let him know. Results slide; long blue line at the far left, lots and lots of queries, then shallows off to the right side; some old rubbish names in in-addr people are querying for. Left side is the fun stuff. 222/8, allocated to APNIC; not sure why it gets a lot more queries than the other /8's. Could be mail server lookups, since there's a lot of mail that comes out of 222/8; could be one rogue server. If you assume it's bogus, and chuck it out, data 'looks' better. 10/8 is near the top, as expected. 0/8 RFC3330 2/8 is unallocated; in the top 15 range. Odd there's unallocated space so close to the top of the queries range. If you take out all the unallocated stuff, there's still a lot of entries; looking at top 10 list: 2/8, 1/8, 23/8, 5/8, 100/8 is there at #5, which is odd. the next /8's they *were* going to give ARIN were 100/101, they decided to NOT give it out to figure out why there's so many hits on it in this round. The top 10 list is the 30 day period ending sunday; he also compared to statistics he used at the esNOG meeting in madrid a few months ago. Not very static; people leave top 10, new come on. 28dec-26jan, vs 19jan to 17feb 2/8 and 1/8 stay in #1 and #2 other /8s shift; 5/8 rises, 23/8 rises, 100/8 drops, 27/8 drops. It's not static, there's movement, people are shifting and shuffling blocks. Not measuring the query source address and AS # that could be gathered to inform people they're trying to talk to unallocated space. Only gathering from l.root, would be good to get data from other root servers. Leo's not a proper researcher, so would be good to get skillset from a real researcher. Would like to get data from nodes all over the world to get more global information; can see if there's more of a local component, or if this is a global phenomenon. This would be a very big data analysis challenge, combining l, k, and other root server operators, and get a more complete view of the data. Use it to either warn people, or help get things fixed if at all possible. Q: Louis Lee, Equinix, want to find out if they've looked to see if the top10 prefixes are in the global routing table when the queries come in? A: No, that hasn't been looked at, but a proper researcher would probably have considered and tied that data in as well. Q: Leo Bicknell, ISC: did they look at the source of queries? Are they all coming from a common resolver? A: no, not yet; would be the sort of thing that a proper researcher would be able to dig into. There are privacy considerations with looking at sources of queries, etc; needs to be thought through very carefully to make sure no per Q: Keith Mitchell, Interesting work, thanks Leo; there's lots of information for gathering, sharing, and protecting privacy in OIRC; if he would like to work with OIRC, they can give pointers on how to handle that data with necessary Q: further instrumenting to see if there's noise in the allocated space to see if this may be just part of overall background noise? A: this is the raw, basic data Q: Duane Wessels, measurement factory; DNS queries behave differently when a result exists vs doesn't exist; may be worthwhile to temporarily make them exist on a set of servers, and collect the data from there, and see if the resolution process follows the delegation. Q: Troy Jessup, Utah edu network; could 1, 2, 100, etc. network queries be due to malware that starts at one end of a block and just starts scanning indiscriminantly. 100 seems to be a common starting point for malware, for example. Haven't looked into it in that much detail--is it just 1.1.1.1/32, or 1.1.1.1/16, how much coverage within the block is there? is it just hot spots? These are really good ideas for the Real Researcher(tm) to delve into, once one can be tracked down. If people want to send suggestions, email him at leo.vigoda(a)icann.org

7 7

Re: 2008.02.20 NANOG 42 IPv4 PTR queries for unallocated space
by Iljitsch van Beijnum 21 Feb '08

21 Feb '08

On 21 feb 2008, at 11:07, Joseph Jackson wrote: > How come RFC 1918 addresses aren't real enough? Because they look like RFC 1918 addresses. On 21 feb 2008, at 11:30, Andy Davidson wrote: >> For instance, I do a training course where people need to configure >> routers, and I use addresses out of 96.0.0.0/8 for that, because it >> has to be clear that we're talking about real addresses and not RFC >> 1918 stuff. > Do you think you could consider some of the rfc3330 ranges like TEST- > NET - 192.0.2.0/24 - or if you need more than one network, > 191.255.0.0/16 ? Nothing is impossible, but shoehorning stuff into a small range doesn't make for a realistic training setup. >> Would it be useful for IANA to publish the order in which they're >> going to allocate /8s? That way, it's easier for people to plan >> getting out of the way of real deployment in time. > Well it's a pretty safe bet that most of today's unused /8s will be > allocated within the next couple of years ! That's true. (Well, it's more like three years: there are currently 41 /8s available and last year, 14 were allocated to the RIRs.) I guess that means that people who want to use "off-label" address space should probably use the legacy /8s that are assigned but don't show up in the routing table.

1 0

2008.02.20 NANOG 42 IPv6 hour feedback and closing notes
by Matthew Petach 21 Feb '08

21 Feb '08

All done--thanks to CRG West for a kick-ass NANOG 42, and we'll see you all in Brooklyn in June! Matt 2008.02.20 Randy Bush, results of IPv6 hour from yesterday. Bill Fenner, Duane Wessels, and ???? Larry made it work to the extent that it did. Grabbed packet headers for the hour; did a simple distribution of IPv4, IPv6, and ARP. Should have subtracted out the NAT-PT traffic from v4, as it gets counted as both. most of the v6 traffic was NAT-PT; that's why v4 tracks v6 identically for the hour. When experiment started, amount of ARP traffic doubled. Duane was pinging the v4 addresses on the subnet; there were ARPs for offnet addresses. Some implementation seem to be making on-net assumption; with no default route, just try arping for it? Keep in mind the v6-xp had v4 but no default. MAC addresses associated to given SSID blue and white were V6 SSIDs, you can see that people *did* shift from the v4 to v6 SSIDs. nanog and nanog-a got double-counted, as they went to the same back-end; but it's still very clear most people tried switching to v6 SSIDs. Duane shows the pinging of v4 addresses during the hour. People definitely stopped trying to use v4 while it was turned off, and started again when they got back from lunch. Linux NAT-PT one known existing implementation http://tomicki.net/naptd.php Userpace based daemon had planned on using Cisco NAT-PT...but unable to make it work. Moved NANOG stats collector Linux box to trunk port configured 3 vlans for nanog-v6, nanog-xp-v6, It puts interface in promiscuous mode, listens for all packets, does translation in userspace. Uses DNS lookups, it synthesizes quad-A responses when there's no A response forthcoming. Daemon repeatedly appeared to hang Had to restart it numerous times Seemed to work better during v6 only hour. There's also v6 on regular networks, could be hanging due to nat-pt box seeing v6 on the main nanog/nanog-a networks (outside). cacti graphs, total traffic through CRG-West 20m to noon, drops to 2-3M, 12:20 NAT-PT started working better, hit 10Mb, drops at 1300 for lunch. total number of associations on APs; peaked at 314 early, at noon drops down to about 100 clients at noon; went up to about 175 during the hour. that's just layer 2 association; but the previous traffic was indeed showing there was some v6 happening. Collected *lots* of "these need to be fixed" data as well for vendors. Jay Hennigan, Impulse Internet--used nanog-v6 and it's getting much better performance than it was during the v6 hour. competition for resource, or need for it to listen to all three networks, could be a bit of a reconfiguration that helped get things going. Closing comments from Todd Very impressive turnout for a four day NANOG! Thanks to CRG-West, wonderful hotel, party last night, many thanks to break sponsors, they help keep the conference price reasonable for us. Statistics! 575 registered attendees (483 paid) 20% ISP 21% network operator 18% equipment vendor 12% content provider 6% research 2% government bunch of ISP folks moving to content houses top countries 488 US 19 Japan 13 UK 11 Canada 20 others orgs: 41 Cisco 16 juniper 14 equinix 13 CRG West Much related to close travel for them. ;) FILL OUT YOUR SURVEY! www.nanog.org! Need to know what was good, what was bad. NANOG43 in Brooklyn, June, TelX is sponsoring it, will be a fantastic, impressive logistical feat given hotel prices in NY; getting price under $200/night is very, very impressive. Program Committee--huge kudos for getting program out soon enough for people to make plans around it. If you have ideas for presentations, come talk to the yellow badged people! If you can help, talk to the yellow badge people! And FILL OUT YOUR SURVEY!

1 0

2008.02.20 NANOG 42 Morpheus, flexible interdomain routing
by Matthew Petach 21 Feb '08

21 Feb '08

Two more sets of notes to go. :) Matt 2008.02.20 Morpheus, enabling flexible interdomain routing policies Yi Wang, Princeton, along with Ioannis and Jennifer. Goal is to get feedback from operators on how useful this would be for getting more usage out of links. Large ISPs have rich path diversity top 2% ASes have 10 or more AS paths for many destinations 5-10 router-level paths for many destinations Different paths have different security levels prefix/subprefix hijacking is a concern avoid an undesirable AS on a path for better performance. today, customers may want different paths; finance wants security, content wants high bandwidth, voip wants low latency/jitter. No "one-size-fits-all" but today, we just have 'one route fits all' Morpheus routing control platform that enables a single ISP to flexibly pick paths for different customers. intra-AS routing architecture it's backward compatible no changes to the routers Use separate morpheus servers to pick the best routes for the routers. Not all routers have to have full set of information. Use MPLS tunnels or IP-in-IP to carry information. Support for multipath already available virtual routing and forwarding on Cisco virtual router on Juniper Limitations of current BGP attributes 1) overloading of BGP attributes localpref used for both business relationships and traffic engineering 2) difficult to incorporate side information (measurement data, business relationships, registry information, history of prefix/origin, route stability) Inside morpheus server: policy objectives as independent modules. Each module tags routes in separate spaces (solves limitation 1) easy to add side information (solves limitation 2) different modules can be implemented independently (eg by third parties) -- evolvability This way, you can take best from multiple vendors or third parties and wrap them together. 3) BGP strictly ranks attributes one over the next; no way to make tradeoffs between policy objectives Use weighted sum instead of strict ranking After a set of classifiers applied, you specify a weight to see which set of heuristics get used. Allow for multiple decision processes running in parallel each with a different set of requirements, and a different set of weights, resulting in different path selections. Prototype implementation in XORP pipeline of classifier modules Classifier works very efficiently total decisiontime graph Morpheus is faster than the standard BGP decision process, when there are multiple alternative routes for a prefix. Evaluation environment 40 pops, 1 morpheus server in each pop each morphous server with 240 peers Morpheus can support a large number of decision processes in parallel; tested up to 40 DP in parallel. no threat to stability only announce non-default routes to stub customers a significant portion of customers are stubs. provides a simple, extensible, flexible way for ISPs to practically control traffic. He has questions for operators on how they set weights, and adjust policies for customers, and how much control they might be willing to give to customers; send feedback to yiwang(a)cs.princeton.edu Q: Todd asks how you have upstream routers making decisions based on differences in source information? A: Goal is to use different tunnels for different customers; so you tunnel to outgoing interfaces to direct your traffic to where you want it. Q: Randy Bush, IIJ, after years of trying to get resiliency over the networks; what happens when the central server fails? Does it revert back to normal routing? A: It's a typical distributed system; every server has a complete view of your routing information. Use multiple servers for a given edge router; so every edge router peers with two different Morpheus servers. But what happens when both fail? The engineers will work hard to bring them back up. And probability of both failing will be much lower than if just one fails. Q: He mentions connections between Morpheus servers; how is that handled--is it proprietary? A: No, not proprietary, more like using BGP AddPath extension, and use BGP as the framework for it.

1 0

2008.02.20 NANOG 42 Graceful Restart, NSF and NSR
by Matthew Petach 21 Feb '08

21 Feb '08

Last set of three notes...was going to send them from the ballroom, but they started tearing down as soon as the closing finished, even though it was 30 minutes early. ^_^;; Matt 2008.02.20 Graceful restart, non-stop routing, spiteful switchover, and non-stop forwarding. Ken Weissner, kweissne(a)cisco.com systems and technology architect introduction to HA technologies Lots of moving parts, we start with some definitions: HA -- high availability general term SSO -- Stateful Switchover two processors/processes, transfers information from one processor so second can pick up from first one. NSF -- NonStop Forwarding forwarding is key portion; router continues moving packets as control plane recovers/restarts GR -- Graceful Restart IETF specificed mechanism, allows peers to give time for peer to come back before routes are flushed; both sides have to agree on it first. NSR -- NonStop Routing maintain session completely without losing state as it shifts from one processor to the other, without having to alert peer via graceful restart Thos four all work together to allow unplanned switchover to occur with minimal interruption of service (not quite "none"). ISSU -- In Service Software Upgrade use above items to allow upgrading of software while packets continue to move. Why do people care about SSO/NSF? Concerns about single point of failure; customer aggregation or customer connect point which would otherwise impact many, many customers when issues arise. They can also do it on non-distributed platforms; very little packet loss on them. Diagram showing what impacts happen with adjacency failures prior to SSO/NSF. Another slide showing nonstop forwarding due to graceful restart mechanisms. supported for LDP, OSPF, BGP, ISIS Graceful restart for EIGRP, and two different draft mechanisms for OSPF. Two modes; "aware" and "capable". If you're a device that is "capable", your peers need to be "aware" that you are capable of doing it. the capable device tells its 'aware' peers that it will come back within a timeout interval. "aware" and "helper" are generally synonymous. configuration for capability needs to be turned on; not on by default. awareness is turned on by default in Cisco code. TCP based protocols have to have it configured on both sides for graceful restart to be able to function. graceful restart concerns voiced at nanog 40 If I'm an aware peer, how do I tell if my neighbor really went away, and I should reroute quickly, or if it's going to be back shortly, and I should continue to move packets towards it. Need to know if NSF is active or not; but NSF isn't something you configure. so, graceful restart concerns addressed: For BGP, there's a restart timer which limits amount of time before peer comes back, which limits the amount of blackhole time; default is 120 seconds, but can be set shorter to limit the duration of blackhole events. Other conditions also apply; if link is POS point to point, linkdown will abort GR, and will tear down the session and flush routes. Once open message with restart bit set comes across, routes are put into stale bucket; still used, but stale counter begins to count down until they get flushed. Also, added "end of RIB" update message; at that point, it can build new table based on updates, and can act upon changes in the stale table; clear truly stale entries, stop stale timer, and process is done. For OSPF, two ways to handle it: RFC3623, vs draft-nguyen-ospf-restart-06 slight differences; does a new route update cause an abort out of graceful restart or not? Cisco supports Nonstop routing for ISIS and BGP in IOS inbox solution, no other communication needed. BGP configured on neighbor basis; ISIS is box-wide. Hybrid BGP NSR run GR on route reflectors Per peer GSR/NSR config for BGP currently BGP GR is globally enabled for all peers in the routing process Where BGP NSR is available, configured on a per peer basis. GR gets used over NSR if peer supports GR Need mechansism to protect interfaces/L2 state and forwarding; protect both data plane and control plane. Features work together to provide protection and redundancy; should really use them all together as designed. eg, enabling SSO also enables NSF (FIB checkpointing). Each routing protocol on the box needs to be aware and configured to get full benefit of NSR. routing protocol timers often cranked down for faster detection of failures. When switchover happens, takes a little while for new processor to catch up, if dead timer is set too low, graceful restart may not kick in before the sessions are torn down. First packet can be pushed in 10 seconds for BGP; but setting timers down to 1/5 or anything below 10 seconds can result in oscillation or other problematic interactions.

1 0

A couple or advanced references...
by Paul Ferguson 20 Feb '08

20 Feb '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Apologies for the noise, but I'd like to go ahead and provide references for a couple of data points which I plan to mention tomorrow during my brief presentation -- they are not referenced in my presentation slides, but they do highlight the issues I'm trying to address. Each are very recently announced studies, papers, or announced statistics. The first one is a study conducted by the fine folks at Google, wherein they "...investigated billions of URLs and found more than three million unique URLs on over 180,000 web sites automatically installing malware". The paper is located here: "All Your iFrame Are Point to Us" http://research.google.com/archive/provos-2008a.pdf ...and associated blog entry here: http://googleonlinesecurity.blogspot.com/2008/02/all-your-iframe-are-point- to-us.html This study reinforces what we are seeing -- literally hundreds of thousands of compromises on the web and server -side. Second, is a paper recently jointly released/presented by Ga. Tech and Google on the the rampant escalation of rogue/malicious DNS resolution paths: http://www.citi.umich.edu/u/provos/papers/ndss08_dns.pdf The numbers are somewhat... staggering. The two issues above contribute directly, and overlap, more than most people are aware. And thirdly is a figure that some folks may already be aware of; the fact that identity theft was the number one source of consumer fraud complaints submitted to the U.S. Federal Trade Commission in 2007. According to the agency's yearly report on fraud complaints for 2007, of 813,899 total complaints received in 2007, 258,427, or 32 percent, were related to identity theft: http://www.ftc.gov/opa/2008/02/fraud.pdf According to the FTC, total consumer fraud losses totaled $1.2 billion, with the average monetary loss for an individual at $349. Credit card fraud was the most common form of reported identity theft at 23 percent, followed by utilities fraud at 18 percent, employment fraud at 14 percent, and bank fraud at 13 percent. Now, there is a certain "fudge factor" in these numbers, of course, but I only mention these issues as a preface for the topics that I plan to solicit the NANOG community's assistance in addressing. Thanks, and see you tomorrow! :-) - - ferg -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFHunbjq1pz9mNUZTMRAri9AKD8wY2qH07AMhpDc2dZpJkdFAHVFQCdEa+t uI1Cwhy1TlHjI6DlQHy5SCM= =V9Dm -----END PGP SIGNATURE----- -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/

5 4

2008.02.20 NANOG 42 Lightning talk notes, round 3
by Matthew Petach 20 Feb '08

20 Feb '08

I went ahead and grouped the notes from all three lightning talks together into one set of notes, as the speakers were powering through so quickly there was no time to even save one file and start a new one. ^_^; Again, apologies in advance for the horrific typos and misspellings, I think we're getting a higher and higher prepondeance of fast talkers presenting these days. :) Matt 2008.02.20 Lightning talks Three lightning talks, 10 minutes each. (notes from all 3 in same file for me.) Martin Hannigan from start to transport: submarine cable deployment hannigan(a)gmail.com, Keflavik Iceland deployed multiple submarine cables, has a background as a captain, done submarine cable deployment. iceland surrounded by water lowcap cable <5G hicap cable 720G lowcap cable is to EOL soon hicap subject to backhoe issue Farice1 Drop a berrilium steel splitter in middle of atlantic to peel off fibers to islands. Security benefits--communication to other countries, rights to access, freedom of information. Performance concerns (speed of light, etc.) .75ms per 150km ROI--will you get return on your investment parameters wet/dry plant manufacturer (tyco) capacity desktop study (DTS) tends to be vendor locked in for full system. biggest cable is 8pair cable. undersea routing--where will you drop it Risks length--too long may be too muc delay repeater distance and sparing; closer, the more you can pump out and handle upgrades to future tech, but costs more. shipping lanes, fisheries, geological considerations fisherman have associations; you've got to talk to the commercial fishers anchors, backhoes--more than 50% cause of outages. permitting to cross international boundries, and pay taxes. sign intent to proceed; gets materials going commision desktop study realeastate for landing spots complete engineering ocean survey, map topography A to Z manufacturer cable in segments, then join it at repeaters. A/Z ends dropped to the beach, cable rolled out Suprise, Greenland connect! 3 routes out of country now 41 active underwater US east and west coast 50 active cable ships that lay/repair ships are shared out, contracted to each other if another ship is closer to where a cut is; they all use a common jointing technique. Cables powered from landing stations, 48vDC powered into the cable. More denser, the cheaper. east coast to london, cables cross all over the place. Telegeography and Cable Operators are your friends. Use armor on cables, bury offshore; costs more, but worth it. prefix mapping to sub cables might be interesting. Be wary of FUD. Fishing conflicts analysis--reroute around fisheries. all nautical charts are in meters; 2150 meters down. Bathymetry charts showing currents, can cause friction landing station approach, very shallow. Credits. wow. ======================================================== Duane Wessels, the measaurement factory day in the life of the internet measurement project. will have a measurement day in March. our data will help researchers You don't have to send data to them; just need it to be collected, indexed, into database so researchers can get to it. motivated by 2001 report challenging reseachers to collect a solid day's worth of data. CAIDA and OARC did one in 2002, 48 hour DNS root server collection expanded in 2007 inspired to further expand for 2008 2007 5 root servers (CEFKM) NaMeX internet exchange, AS 112 Open root servers, etc. what did they learn? 5% of clients responsible for 95% of load at root servers 1-2% of queries are legitimate; rest is junk. query rates doubled from 2006 anycast very effective shows where clients are vs where servers are longitude chart vs bar colours showing where clients connect to servers. what is all the pollution at the roots? lots of repeated queries, identical queries invalid TLDs seem to hit F.root what learned? you always need more disk space than you think collecting is easy; indexing is hard. questions to pursue with DITL data: DNS source of garbage trends in v6, DNSsec, etc. traffic/performance questions botnets? voice/video? is v6 different than v4? RND vs commercial traffic patterns? need to accomodate privacy/legal concerns Make data selectively available, anonymize it. index it into CAIDA's DatCat Please participate! http://www.caida.org/projects/ditl/ http://imdc.datcat.org/ ===================================================== Peter Loesher slide decks are all on the website, btw, so feel free to read through thm on your own. Joao_Damas(a)isc.org BIND, AAAA and the root servers finally, AAAA records for the root servers in the root-servers.net zone initially 6 out of 13 are providing v6 root service about 80-100qps of v6 right now. F root uses v6 in same way as v4; all local nodes are advertising /48, covering /47 advertised on the global nodes to avoid blackholding 13 of 43 nodes worldwide serving v6. Most traffic going to paris and amsterdam, rest goes to global nodes in bay area. very little in japan, possibly due to peering culture in japan. peering with f root over v6 we are ready almost everywhere, let them know, they'll peer with you via v6 BIND changes BIND needs no code changes on its own. ISC will provide updated hints file in 9.5 in the mean time you can fetch your own copy from: ftp://rs.internic.net/domain *whew* PGP signing party in Garden room, quick break. Final talks will start at 11.

1 0

2008.02.20 NANOG 42 current state of 10gig pluggable technologies
by Matthew Petach 20 Feb '08

20 Feb '08

Continuing on with the notes from this morning's talks, pushing out the notes while we take a bit of a break. :) Matt 2008.02.20 10 gig pluggable technology Richard A Steenbergen from nlayer ras(a)nlayer.net Many thanks to Todd for donation of the slideset! Title is 10gig, but it applies to other technologies as well; fiberchannel, etc. What are pluggables? transcievers you plug into your router or switch, they can drive multiple media types (copper, fiber, short reach, long reach, etc) There's standardization among the parts, so that vendors can use common parts. technical benefits, you can mix-and-match media types on the same router/switch card; easier replacement in the event of optical failure. financial benefit--pay as you populate. reusable, easily re-deployable elsewhere cards are reusable as technology evolves; can migrate from fiber to copper easily. standardization lowers costs. standardized, allows for interoperability standardized via vendor MSA (multiple source agreement) specs published under SFF, which defines the strict specs to be followed. First gen pluggable technology; started in ethernet land with GBIC, first came from the fiberchannel world; commonly in use from 1998 to 2002. >From 2002 onward, started shifting to sfp form factor; multirate, so you can use same pluggable for 1g, 2g, 4g fiberchannel, can use for gig or OC48 SONET First introduced digital optical monitoring (DOM) in 2004. better density. 1g to 2.5g, had GBIC and SFP. For 10G, many, many choices (7) 300-pin XENPAK XPAK X2 XFP ... Problem in operator community now is that with so many options, sparing is hard. slide with overall size comparisons. First 10G was 300-pin MSA; not pluggable, but a snap-on connector; 8-14W, 16x622Mbps interface. Same thing used for 40G XENPAK MSA, march 2001, second most popular at this point. power 6-10W XAUI (4x3.125G) PHY framer onboard 80km+ ZR, DWDM tuned, other exotic optics X2 MSA, July 2002 same electrical as Xenpak 4-5W, rest is same as xenpak, just smaller. Cisco and HP are only users of it; lukewarm support for it. very few exotic optics choices for it. XPAK MSA similar to X2, XAUI interface uses LC connector, different form factor, for use on PCI cards no exotic optics not used in network world. XFP MSA, march 2002 XFI (9.995-11.1G serial interface) variable speed serial interface PHY framer is offloaded; can be 10G, OC192, transport, etc. One of most commonly deployed optics exotic: ZR/DWDM, limited CX4, no LX4 very, very popular eliminating serdes for 10GBASE-RW is big power saver power 1.5-3.5W SFP+ MSA draft dec 2007 interface SFI (8.5 - 11.1G) same dimensions as SFP, same connector extend down to do 8G fiberchannel PHY framer offloaded no exotic optics smaller than XFP, offloads CDR function SFI similar to XFI, adds 8G FC speed limited power use power 1-1.5W 10gig pluggable component technology PHY, physical layer component PCS --physical coding sublayer PMA physical medium attachment sublayer serdes CDR clocking data recovery chip 10gig PHY comes in 3 variants 10GBASE-R LAN 10GBASE-W WAN similar to R, but wrapped in OC192 SONET compatible frame 10GBASE-X Lan phy 4x2.5G parallel signal LX4 is 4 optical channels, mux built in CX4 is 4 copper channels same choices for interconnect; XAUI is 4x3.125G parallel channels (8B/10B encoding) XFI/SFI, single 10gig serial channel uses 64B/66B, much more efficient encoding each time you convert, it increases heat, power, consumes port space, etc. Ultimate goal is to match technologies; if you have 4 parallel lanes on each side better XAUI to 10GBASE-X XFI/SFI to 10GBASE-R/W So XAUI to 10GBASE-R, inefficient XENPAK has room for serdes stages, fortunately Future directions for technology? new technogies replace old 10GBASE-LRM long reach multimode replaces LX4 10GBASE-T to replace old CX4 for copper uses standard cat6/cat7, 8p8c (RJ45) connectors Advantages of offloading PHY framer can use same pluggable for multiple protocols allows better component reuse, lowers costs can use in transport gear with forward error correction 10GE WAN PHY much bette when done on host can use any PMD without having to buy special pluggables improves sparing, lowers costs, expands PMD options improves WAN layer control signalling and alarms provides access to SONET alarms, path trace, etc. vastly improves troubleshooting with OC192 carriers Advantages of offloading the CDR make pluggable smaller, uses less power disadvantages mostly zero sum gain; no components saved, just moved to board not all CDRs are created equal may want higher quality on longer runs can't upgrade pluggable to get better EDC (electronic disperson compensation) technology Evolution of pluggable power use (watts) power consumption going down over time; different classes of power use. Not only dropping power, but removing entire power classes. comparison chart of optics slide. Software side of pluggables talk to host via low speed control bus; power down, alarm signalling, etc. Digital Optical Monitoring lets you see power levels on the link, speeds up troubleshooting. TDR functionality for copper hosts can also read EEPROM data allows vendors to lock users into particular pluggable. Vendors don't make their own pluggables, they're made by JDS Uniphase, finisar, hitachi, etc. pluggable has EEPROM, set vendor string $1.4B in profit at Cisco from reselling pluggable optics. how keep customers locked in: FUD (won't support it, might blow up, support contract is void, etc.) financial component; many sellers, few buyers; if Cisco controls 70% of market, they can set the prices, and prevent them from being resold outside their channel. Vendor locking--if it's not ours, router won't support it. Big customers are pushing back on that. counterfiting cisco is now the new way to make money, as you make profit by charging for name. new model is doing feature impairment; certain features like DOM are set so that it's only usable with their own pluggable. What format is right for you? XENPAK has best selection, well stocked, most exotic optics options downside; very large, draws a lot of power, not protocol agnostic, not friendly with WAN PHY X2--smaller than xenpak, lower power allows for higher density full support for LX4/CX4, pushed by cisco very little deployed base, few sources, few exotic optics same as XENPAK, not protocol agnostic XFP--advantages--large deployed base, cheap, easy to find, technologies still being upgraded on it. power/density pretty good for most users. protocol agnostic, easy spare, good WAN PHY cheapest option for long reach/DWDM SFP+ -- very high density (48 per blade) physically same as SFP may allow for extremely low cost device (GoogleSwitch) downside; hard cutoff on power budget, hard to do longer reach optics; no DWDM, no LR, no ZR; won't ever be able to be full replacement for XFP because of that.

1 0