(This time around I opted to go to the peering BOF and take some notes. It's the one downside to parallel tracks--wish I could be in two places at once. ^_^;; --MNP) 2006.06.05 Peering BOF Bill Norton introduces the Agenda; unfortunately, my laptop took so long to boot, I missed the Agenda slide. Doug Toy?, Transit Cost Survey, data collected at NANOG 36; he's just here to present the collected info, not really representing anyone. Recap: At NANOG 36, people indicated their cost per Mb and commit level. length of contract was usually 1-2 years. 42 data samples collected avg $25/Mb $95/$10 were the extremes. Avg commit level 1440 Mbps Other observations as expected, cost per Mbs tends to decrease as the commit level increases. Tier1's are more expensive Cost tends to vary more with Tier 1 providers than with others. between 0-500Mb commit level, prices are all over; at higher commits, prices level out at the bottom. Question: Mbps, is that the cap, the usage, inbound plus outbound? A: That's the general 95th percentile higher of the two inbound and outbound. Committed amount. Graphs tend to approach a hard bottom; as commit increases, doesn't change all that much. Bottom is around $10/Mb, even though commit levels increase. Of samples collected, 2/3 were from Tier1 providers. 90% of contracts are 1-2 year in length, so didn't cause much variance. Tier 1 definition is based on Wikipedia definition. Questions from audience? Q: Data looked pretty clean; were there samples pulled out to make it look cleaner? A: No, other than people who left fields blank on the survey. Q: was there a timestamp of when contract started? A: much of it wasn't complete. Mostly within the 1-2 year range for length as well as start date, so nothing really ancient in there. BillNorton; people had some concerns about violating NDA or contract details when filling out the survey. Where do we draw the line in doing these types of surveys? SteveGibbard; NDA is agreement between transit provider and customer, and this was anonymized and voluntary. Data is interesting, both for purchasers and sellers of transit. Q: 42 samples graphed, there were 80-100 people in the room at the time; so the real comments from the rest of people weren't counted? A: No, there were less than 50 submissions total, of which 42 were complete. Q: Patrick; many people put more than one transit provider on their form; how were those other transit providers handled? A: no clue, he just got a spreadsheet with data. Back to Bill Norton Peering Lists Issue -- make available to customer prospects? 15 mins. Peering disclosure dilemma: customers often asked for peering list, sometimes peerings restricted under NDAs Metric for determining connectedness, capacity, resiliancy. Is there a better metric for customers? IX capacity in/out Peering pipe size? ISPs are getting commonly asked about this, based on hands raised in the room. How many people lose business because the customer doesn't get an answer? Sylvie, VSNL notes they provide the info when they're under an RFP; they won't give capacity, they give an aggregate, they won't go peer by peer, that would be a violation of NDA. BillN: are the NDAs written to allow total numbers like that? Sylvie: you should not disclose capacity per location or per circuit, but they don't forbid aggregate total numbers. BillN: is there something else that could be given to the customer that would satisfy their question without revealing what Chuck: A lot of ISPs lie about their peerings; he runs AMSIX, people claim to have multiple gig to the peering exchange, he knows they don't really have that much. Patrick: but he can look at the peering stats on AMSIX--Chuck notes only members can. Patrick: customers ask how many gigs they can send to a provider; it's available headroom, so they ask their upstreams how much available headroom is left. Most providers are having a lot of trouble getting the right capacities to the right networks. The reason many don't answer is they don't like the answer they have to give. Ted Seely, Sprint, how do you solve the problem? There's lots of traffic that needs to be exchanged, how do you fix it? Patrick: how about everyone upgrade to 10gigE in many places? If you can't afford it, stop selling bandwidth. But most people can't go to all the different providers, they have to buy from a small subset of providers. RAS: No technology problem with doing it; it's the money. Not charging enough to cover the costs of the technology you need to install to cover the bandwidth. Ted notes you can't just link at one spot, you have to connect at six places, and you need to have links in and out of the site to support the volume, etc. Patrick: can you tell us how much they exchange? 40Gb times 6 providers in six locations is probably more traffic that Sprint has in total. Ted Seeley; it's a time scale issue--yes, it can be solved, but in what timeframe? BillN feels it's reasonable information for a customer to request. Is there a proxy that customers could use that would give similar information? Tristan Horn, Q: has anyone use skitter to try to figure out the answer on their own? Ted: last skitter pass was March, 2005--data isn't current enough to be useful. Michael Abrams, tcp test, consumers can run to see how much traffic they can pass along; it becomes very quickly apparent who has problems, and consumers tend to reach quickly to that, go elsewhere. Shows packet loss and latency at different throughputs. Sweden, size of CA, they have 4 exchange points, and want more. Still gamers are most vocal about any problematic interconnects. Video Peering Video Stats and distribution models -- all YouTube Peering Personal -- Colin They're getting into the peering fray, and only a year old. This is gigs and gigs, has potential to dwarf current peering traffic. Current issues could be tiny compared to the flood of potential issues when hundreds of Gbs comes flooding towards customers. Voxley? on the web; 320x240 pixel image, want to do a one hour episode, will be about 210MB file. currently, show is watched by 10,000,000 households. So, that's 2.1PB to transfer all those episodes. If you try to stream it, via say iTunes, over 3 days, would be about 64Gbps solid for 3 days. That's a pretty big number. That's one episode, from one show. American Idol is even bigger. He claims a better way is using peer to peer. 2.5% of the agents could be propagation nodes, or 256,000 PCs as transfer agents; that would fit on existing broadband connections. What about multicast? problem at core, end distribution; pushing to 256,000 is much more doable. Ted Seeley notes you still have to pass the unicast streams eventually, so you still haven't solved the issue. If you don't do multicast or caching at the edge, you've just shuffled the problem. Swedish police, 100Gb of peer-to-peer traffic at peak, AMSIX lost 10gig, LINX lost 5gigs, probably lost about 40Gb weds/thurs last week when the swedish police shut it down. Which site? Pirate Bay torrent tracker. Peer to peer has similar replication pattern to multicast? But it's not localized properly, unlike multicast. RickW, the pirate bay site, that site was only distributing meta information. peer to peer traffic can be 12-20% higher because of the duplicated traffic. Peer to peer networks aren't as efficient as unicast streams. Matt notes 64Gb, 256,000 subscribers isn't that big a stretch anymore. Peer to peer is unlike multicast, because it doesn't have any localization to the best of his understanding. Do peer to peer systems use latency to control who they share with? Somewhat; faster peers are selected over slower peers. Comment from audience is that live events are still going to be the challenge; HDTV is getting gb/sec from cameras, needs to feed it out, no chance to cache, so multicast ends up being the only viable option for it. Question about is the assumption that getting it over the internet is the right model, vs using settop boxes? How do you get a whole country to change their viewing habits? Korea Telecom uses peer to peer to pass content around; it is limited to a certain DSLAM or POP, which answers Matt's localization/topology issue. Ted Seeley; yes, people are watching material online vs set top boxes, the paradigm is shifting already. RAS, works through numbers; a DVD of MPEG2 will be around 4GB, if you drop out the special bits, audio tracks, etc; heavily compressed, 600MB, to around 1GB mpeg4 for DVD quality, 2-4GB compressed for HD quality. if we take 1000MB filesize, cost per movie to deliver this. Assume $10/Mb, you get 1Mbit/sec/month for $1, Assume normal traffic curve, 150GB/month for your per Mb $10 point. Assume it's mpeg4 compressed to 1GB, that's 150 movies per $10/month, that's less than the cost of postage. How does it best make sense to distribute these files? Compression reduces file sizes about 15% per year. interstreamtv.com, has article about it. 1Mb/sec will do 320GB/month. So number of movies would vary. Peer to peer want immediately, and want quality, so the model might be problematic. Caching at the pop edge would help. And live events, multicast would help, use unicast to fix lost packets. How do you compensate for lossy networks? Forward error correction, which means you lose the benefit of compression you were hoping to get. Larger buffer won't help for retransmits unless it's TCP. Curtis points out live streaming vs Richard's downloaded movies. RAS was replicating Netflix model, you order in advance, get it later that day. If you want to stream live stuff, it needs to be multicast, provider needs to do the replication. How much of it is unavailable in the country where the provider is located, that's why you can't cache or work with the provider to distribute it. Brokaw notes more streaming is coming; more video stuff will be coming in the months and years to come. For the access networks, how many are multicast enabled, and are ready for helping this scale? 3 hands. Does the community agree we have a bit of a challenge here? Matt Peterson, encryption to make sure the file is different for each user, to make sure it's legal. Bittorrent with RSS to get metadata, weather channel model, ultraDNS, Akamai model multicast model What about skycache? One-to-many distribution? Colin from YouTube will start off the peering personals. They've recieved an AS, they push about 20Gb sustained, their first datacenter went live in the middle of March. It is definitely growing. Not multicast enabled, no demand for it. The average show they have is 4 minutes, hundreds of millions of different videos, not a good multicast target. 20% growth month to month, 50 million videos viewed per day. 20gbit/sec outbound? "Ask a ninja net neutrality" video. Product launched in December, that's when they saw real traffic; been moving more and more in house in March and April. They're pushing 20gig in house, not from CDN network. This is is all short form video content, hopefully no licensed content sneaking in. San Jose Lundy on equinix fabric PAIX on paix fabric Los Angeles on equinix fabric Linking all 3 locations together. Metro, long-haul providers take a long time. Lundy is poorly connected. Getting connectivity at 10 gig (Fiber swaps, metro links, cross datacenter crossconnects) Other items Starting to see traffic from v6-v4 gateways most hardware providers do v6 support in software Seeing v6 traffic from v6-to-v4 gateways from Asia. Are people doing much successful v6 on modern devices? Force10 s50, 4500 from cisco, all seem to be software based v6. Should they have a native v6 presence, go dual stack, what hardware would be good to use? sup720 will be the cheapest hardware to do hardware v6 at line rate 10 gig. Can't do it on the lower end. Person from puerto rico notes that hardware switches can do 200Gb/sec of IPv6 switching, why can't routers do it? Frustrated audience reactions from all over about differences between switching and routing. Robert Seastrom--should you do v6 at all? Should you be a pioneer, and make the v6 people happy, sure, do it; if you want to make money, no. Q: how do they make money, and will that model support their bandwidth? it's advertising supported, hopefully it will support their needs. Q: how do you compete with Google? No idea, he's not a business person. Q: Content providers often don't get peers at public sites, much harder to get peers than access providers; are they running into that? A: Right now, peers are still coming to him; he won't know about resistance until he goes out asking for it. Summary: Open Peering Policy peering@youtube.com colin@youtube.com AS# 36561 How much of the 20gig is going across peering links vs transit links? It's all transit right now--he's just now asking about peering. Jeffrey notes that if there are people who pay transit to get your content, they will peer regardless of ratios. DanGolding notes that the tier1's are stuck on a treadmill; they can't peer with you even if they desire it, for fear of messing with their own ratios. Josh asks if they will go to other locations than just the west coast of US. Colin notes that they're doing a datacenter month. Chicago is next. Dan Golding, network neutrality on the peering front. One year old company, video content, already doing 20gig/sec. This is a buttload of traffic, and they're already getting into the peering mode; will this traffic ultimately dwarf the rest of our traffic? Network Neutrality, an open discussion. Tier1research. What it means to peer. network neutrality: an introduction what is network neutrality simply put, it's the idea that internet service providers should treat all legitimate traffic the same way regardless of origin and destination. Why should I care? a significant portion of the internet is based off mutual... The Bell perspective An unfair load the bell companies-att and verizon-aren't big fans of neutrality. they feels they're carrying traffic for others and are uncompensated or not compensated sufficiently Google, Yahoo, making big profits from the sweat of their brows. The Bell Answer $$$ needs to flow from content carriers to the Bells. How would Bells make this work again? everything old is new again. in the PSTN world, the bells charge reciprocal compensation for voice calls terminating on their networks. Talk of prioritization... [he's really whizzing through his slides!] Content perspective The internet works because of neutrality non-neutrality would stifle innovation The bells have a bad business model, they're asking content to help prop it up. Prioritization is a protection model; The bells are in a power position their control of the internet is at an all time high. the conglomerate controls a huge amount of access, they are tier 1 internet peers, SFI peering with each others. Others are tempted to jump on. MSOs see non-neutrality as a way to keep other's VoIP off their networks On the other hand, the Bells will squeeze them; the cable companies lack peering. His take; leans towards content. Would be fair, except their networks were built under monopoly auspices. Consumers have little choice in many locations some sympathy for bells; it's their network, after all. How will this end? possibility #1; regulation of internet to enforce neutrality. regulation makes the internet less flexible probably vague law with lots of FCC regulations Regulatory Affairs officer has to vet ACLs Difficult to do filtering quickly ability to provide tiered services will be more difficult. Possibility #2 bells get their way VZ and ATT become tier 0 internet carriers depeer everyone else as soon as current restrictions expire charging reciprocal compensation to all others where are the MSOs? other access providers? why buy transit from anyone other than a bell? potentially disastrous scenario internet growth is stifled forces you to always buy transit from the bell companies. Possiblity #3 threat of regulation causes a continuation of status quo. best solution loss of neutrality could destroy the internet internet regulation could stifle it; and the bells are VERY good at regulation. perhaps the evolution of tiered consumer services could come about by volume, not quality consumers warming to this model content providers sponsor some access models. Patrick Gilmore: Tier 0, how does that work when VZ and ATT don't make up the bulk of the internet anymore. What about the rest of the world? The bells get regulations changed when the world doesn't move in the direction they like; reciprocal compensation got removed when the ISPs became CLECs and got lots of compensation from the Bells. Now, regulations modified. Sean Donelan; it's net neutrality, not internet neutrality, so what about other players? broadcasters, radio, etc. For voice calls, not such a big issue. broadcast networks don't have much of an idea of what this is about; not sure they see the applicability of this yet. When they do, they have huge lobbying power as well. Bells have very sophisticated lobbying efforts, make them very smart. The newbies like Google and Yahoo aren't that savvy in washington yet. 60% liklihood that the end result is regulation; is *not* a happy outcome. Is there some pivot point that may make people change their models? Patrick Gilmore asks for non-US peering coordinators; who would care if ATT/VZ depeered you? Not many respond Curtis--not just a US problem; similar discussions in European countries. At what layer does the competition come into play? In European, there's no text on the table yet, but there's discussions taking place at least. Somewhat of a reverse problem, it's the new players looking at tax monies. AMSIX, much better competition on the last mile and the infrastructure, the big players can't play the game the same way. Dan Golding notes that if we had many different ways of getting local loop to your house, it would be less of an issue. Incent development of alternate methods; wifi, 3G/4G/5G networks, etc. Mikhail Abramson, with high speed cellular, mobile is making network neutrality less of an issue, since you do have more options. The GSM providers are happy with the internet bandwidth usage on mobile data, it's making them really good money. Matt Peterson? Last comment about cell phones; but people are used to packet munging; most people are used to a clampdown at this point; they may not even notice anymore. Until Grandma using YouTube understands network neutrality, it'll be this community only fighting/caring. If we all went to a common $10 provider, we could create a new tier0 and bypass the bellheads. A few peering personals. Jeffrey Papen, MySpace, AS 33739, well over 20gigs, in LA and by august, SJ, CHI, ASH, all Equinix locations. Needs to be in peeringdb.org!! People are now talking about putting peeringdb.org in their peering contracts. Stafford Rau, CLEC in Portland, ops, in WA, northwest, currently acquiring ELI from frontier, will be about twice as big. SIX westin, PAIX, AS12003, 7845, will roll all of it including ELI into 7385, they don't get 5650 unfortunately. 2-3gig transit now. Philip, MSO in Canada 5769, 13571, about 15gigs, videotron.ca Matt Peterson, SIX seattle, copy it in SF, neutral, non-profit, 365 Main, mostly content providers, CNET, craigslist, others. load up your blogs, SFMIX, 365 Main, he or Tim Pozar from UnitedLayer. MetaInterfaces, video, 5gigs, makes money, tells you what kind of video it is. ^_^; Vish from Netflix, looking for potential peers in SJC, CHI, ASH, vish@netflix.com really aiming for eyeballs. Ann claiborne, prolexic technologies, also in peeringDB, NOTA, LINX, peering@prolexic.com hoping to come to west coast. Dash to restrooms and then to the next BOF.