October 2012 - NANOG - lists.nanog.org

End User Experience
by Dan Snyder 25 Oct '12

25 Oct '12

Hello All- I am a currently researching how network operators monitor and/or detect degradation of the "end user's" experience. I would really appreciate it if everyone could take a few minutes to fill out my survey. It's at * http://questionpro.com/t/AJVauZOb2F* * * Thank you all again. Dan

3 5

need help 40 G
by Deric Kwok 25 Oct '12

25 Oct '12

Hi all I would like to ask any free program/method can test 40G I tested it by ip erf program but it is not over 20G Thank you

1 0

XO ethernet over copper (EoC) - MTU
by Scooby 25 Oct '12

25 Oct '12

We recently had an XO EoC install turned up and it seems to have an odd MTU. XO Support had no idea what I was talking about when I called to ask. Does anyone know what the proper MTU should be? Thanks in advance, Scooby

1 0

hotmail.com live.com admin needed
by Carlos M. Perez 24 Oct '12

24 Oct '12

Hi, We're trying to resolve some delivery issues reported by hotmail users. Started happening a few weeks ago. Getting immediate NDRs, and the server that is supposed to receive the email has no records of attempts. The messages also don't match what the receiving server should be sending. The server(s) listed in the MX should receive all email without authentication, since it's a mail filtering service (Maxmail) === Reporting-MTA: dns;snt0-omc3-s27.snt0.hotmail.com Received-From-MTA: dns;SNT133-W53 Arrival-Date: Mon, 22 Oct 2012 14:09:49 -0700 Final-Recipient: rfc822;administrator(a)xxxx.com Action: failed Status: 5.5.0 Diagnostic-Code: smtp;550 authentication required === Kindly contact me off-list. Thanks, -- Carlos M. Perez Runcentral, LLC

5 8

One big cluster: How CloudFlare launched 10 data centers in 30 days
by Eugen Leitl 24 Oct '12

24 Oct '12

http://arstechnica.com/information-technology/2012/10/one-big-cluster-how-c… One big cluster: How CloudFlare launched 10 data centers in 30 days With high-performance computing, "pixie-booting" servers a half-world away. by Sean Gallagher - Oct 19 2012, 0:13am -200 Big Data Cloud Open Source Supercomputing The Web The inside of Equinix's co-location facility in San Jose—the home of CloudFlare's primary data center. Photo: Peter McCollough/Wired.com On August 22, CloudFlare, a content delivery network, turned on a brand new data center in Seoul, Korea—the last of ten new facilities started across four continents in a span of thirty days. The Seoul data center brought CloudFlare's number of data centers up to 23, nearly doubling the company's global reach—a significant feat in itself for a company of just 32 employees. But there was something else relatively significant about the Seoul data center and the other 9 facilities set up this summer: despite the fact that the company owned every router and every server in their racks, and each had been configured with great care to handle the demands of CloudFlare's CDN and security services, no one from CloudFlare had ever set foot in them. All that came from CloudFlare directly was a six-page manual instructing facility managers and local suppliers on how to rack and plug in the boxes shipped to them. "We have nobody stationed in Stockholm or Seoul or Sydney, or a lot of the places that we put these new data centers," CloudFlare CEO Matthew Prince told Ars. "In fact, no CloudFlare employees have stepped foot in half of the facilities where we've launched." The totally remote-controlled data center approach used by the company is one of the reasons that CloudFlare can afford to provide its services for free to most of its customers—and still make a 75 percent profit margin. In the two years since its launch, the content delivery network and denial-of-service protection company has helped keep all sorts of sites online during global attacks, both famous and infamous—including recognition from both Davos and LulzSec. And all that attention has amounted to Yahoo-sized traffic—the CloudFlare service has handled over 581 billion pageviews since its launch. Yet CloudFlare does all this without the sort of Domain Name Service "black magic" that Akamai and other content delivery networks use to forward-position content—and with only 32 employees. To reach that level of efficiency, CloudFlare has done some black magic of a different sort, relying on open-source software from the realm of high-performance computing, storage tricks from the world of "big data," a bit of network peering arbitrage and clever use of a core Internet routing technology. In the process, it has created an ever-expanding army of remote-controlled service points around the globe that can eat 60-gigabit-per-second distributed denial of service attacks for breakfast. Routing with Anycast CloudFlare's CDN is based on Anycast, a standard defined in the Border Gateway Protocol—the routing protocol that's at the center of how the Internet directs traffic. Anycast is part of how BGP supports the multi-homing of IP addresses, in which multiple routers connect a network to the Internet; through the broadcasts of IP addresses available through a router, other routers determine the shortest path for network traffic to take to reach that destination. Using Anycast means that CloudFlare makes the servers it fronts appear to be in many places, while only using one IP address. "If you do a traceroute to Metallica.com (a CloudFlare customer), depending on where you are in the world, you would hit a different data center," Prince said. "But you're getting back the same IP address." That means that as CloudFlare adds more data centers, and those data centers advertise the IP addresses of the websites that are fronted by the service, the Internet's core routers automatically re-map the routes to the IP addresses of the sites. There's no need to do anything special with the Domain Name Service to handle load-balancing of network traffic to sites other than point the hostname for a site at CloudFlare's IP address. It also means that when a specific data center needs to be taken down for an upgrade or maintenance (or gets knocked offline for some other reason), the routes can be adjusted on the fly. That makes it much harder for distributed denial of service attacks to go after servers behind CloudFlare's CDN network; if they're geographically widespread, the traffic they generate gets spread across all of CloudFlare's data centers—as long as the network connections at each site aren't overcome. In September, Prince said, "there was a brand new botnet out there launching big attacks, and it targeted one of our customers. It generated 65 gigabits per second of traffic hitting our network. But none of that traffic was focused in one place—it was split fairly evenly across our 23 data centers, so each of those facilities only had to deal with about 3 gigs of traffic. That's much more manageable." Net-rich, power-poor Making CloudFlare's approach work requires that it put its networks as close as possible to the core routers of the Internet—at least in terms of network hops. While companies like Google, Facebook, Microsoft, and Yahoo have gone to great lengths to build their own custom data centers in places where power is cheap and where they can take advantage of the economies of scale, CloudFlare looks to use existing facilities that "your network traffic would be passing through even if you weren't using our service," Prince said. As a result, the company's "data centers" are usually at most a few racks of hardware, installed at co-location facilities that are major network exchange points. Prince said that most of his company's data centers are set up at Equinix IBX co-location facilities in the US, including CloudFlare's primary facility in San Jose—a facility also used by Google and other major cloud players as an on-ramp to the Internet. CloudFlare looks for co-location facilities with the same sort of capabilities wherever it can. But these sorts of facilities tend to be older, without the kind of power distribution density that a custom-built data center would have. "That means that to get as much compute power as possible into any given rack, we're spending a lot of time paying attention to what power decisions we make," Prince said. The other factor driving what goes into those racks is the need to maximize the utilization of CloudFlare's outbound Internet connections. CloudFlare buys its bandwidth wholesale from network transit providers, committing to a certain level of service. "We're paying for that no matter what," Prince said, "so it's optimal to fill that pipe up." That means that the computing power of CloudFlare's servers is less of a priority than networking and cache input/output and power consumption. And since CloudFlare depends heavily on the facility providers overseas or other partners to do hardware installations and swap-outs, the company needed to make its servers as simple as possible to install—bringing it down to that six-page manual. To make that possible, CloudFlare's engineering team drew on experience and technology from the high-performance computing world. The magical pixie-booted data center "A lot of our team comes from the HPC space," Prince said. "They include people who built HPC networks for the Department of Energy, where they have an 80 thousand node cluster, and had to figure out how to get 80,000 computers, fit them into one space, cable them in a really reliable way, and make sure that you can manage them from a single location." One of the things that CloudFlare brought over from the team's DoE experience was the Perceus Provisioning System, an open-source provisioning system for Linux used by DoE for its HPC environments. All of CloudFlare's servers are "pixie-booted" (using a Preboot eXecution Environment, or PXE) across a virtual private network between data centers; servers are delivered with no operating system or configuration whatsoever, other than a bootloader that calls back to Perceus for provisioning. "The servers come from whatever equipment vendor we buy them from completely bare," Prince said. "All we get from them is the MAC address." CloudFlare's servers run on a custom-built Linux distribution based on Debian. For security purposes, the servers are "statelessly" provisioned with Perceus—that is, the operating system is loaded completely in RAM. The mass storage on CloudFlare servers (which is universally based on SSD drives) is used exclusively for caching data from clients' sites. The gear deployed to data centers that gets significant pre-installation attention from CloudFlare's engineers is the routers—primarily supplied by Juniper Networks, which works with CloudFlare to preconfigure them before being shipped to new data centers. Part of the configuration is to create virtual network connections over the Internet to the other CloudFlare data centers, which allows each data center to use its nearest peer to pull software from during provisioning and updating. "When we booted up Vienna, for example," said Prince, "the closest data center was Frankfurt, so we used the Frankfurt facility to boot the new Vienna facility." One server in Vienna was booted first as the "master node," with provisioning instructions for each of the other machines. Once the servers are all provisioned and loaded, "they call back to our central facility (in San Jose) and say, 'Here are our MAC addresses, what do you need us to do?'" Once the machines have passed a final set of tests, each gets designated with an operational responsibility: acting as a proxy for Web requests to clients' servers, managing the cache of content to speed responses, DNS and logging services. Each of those services can be run on any server in the stack and step up to take over a service if one of its comrades fails. Caching and hashing Caching is part of the job for every server in each CloudFlare facility, and being able to scale up the size of the cache is another reason for the modular nature of how the company thinks of servers. Rather than storing cached webpage objects in a traditional database or file system, CloudFlare uses a hash-based database that works in a fashion similar to "NoSQL" databases like 10gen's MongoDB and Amazon's Dynamo storage system. When a request for a webpage comes in for the first time, CloudFlare retrieves the site contents. A consistent hashing algorithm in CloudFlare's caching engine then converts the URL used to call each element into a value, which is used as the key under which the content is stored locally at each data center. Each server in the stack is assigned a range of hashes to store content for, and subsequent requests for the content are routed to the appropriate server for that cache. Unlike most database applications, the cache stored at each CloudFlare facility has an undefined expiration date—and because of the nature of those facilities, it isn't a simple matter to add more storage. To keep the utilization level of installed storage high, the cache system simply purges older cache data when it needs to store new content. The downside of the hash-based cache's simplicity is that it has no built-in logging system to track content. CloudFlare can't tell customers which data centers have copies of which content they've posted. "A customer will ask me, 'Tell me all of the files you have in cache,'" Prince said. "For us, all we know is there are a whole bunch of hashes sitting on a disk somewhere—we don't keep track of which object belongs to what site." The upside, however, is that the system has a very low overhead as a result and can retrieve site content quickly and keep those outbound pipes full. And when you're scaling a 32-person company to fight the speed of light worldwide, it helps to keep things as simple as possible.

2 1

Inter-domain OTN, does it happen in the real world?
by Will Orton 24 Oct '12

24 Oct '12

Reading about OTN networks, I see that "IrDI" is specified to handle the case where one OTN network needs to connect to another natively with OTN signals. Is this done in the real world? Does OTN network operator A ever go to OTN network operator B and say, "I'd like to buy a OTU2 from city X to city Y on your long haul network (at buildings J and K where we can connect simply with short-distance SMF/1310 signals), and what TCM levels can you give me?" I understand this in the case of lit 10GbE-WANPHY, LAN, and OC-192, but are OTU "lit" signals bought and sold wholesale this way too? Is there generally a price premium over the more normal client signals? -Will Orton

3 2

SCTP Firewall with multi-homing support
by Lee, Steven (Network Consulting Malaysia) 24 Oct '12

24 Oct '12

Hi all, has anyone have the experience on SCTP firewall to protect the SIGTRAN communication between local mobile operator and roaming partner? If yes, please contact me offline. Thanks and regards, kinwei

1 0

Windstream outage in WI and MI?
by bernie listsub 24 Oct '12

24 Oct '12

We're seeing multiple sites in our MPLS that are down, and got word from our Windstream sales rep that there was a fiber cut affecting WI and MI. The NOC is either ringing busy or giving a high call volume greeting. Anyone else affected or know more? -Bernie ================================ Nick "Bernie" Bernhardt nick.bernhardt(a)landmark.coop ================================ "We are a COOPERATIVE business dedicated to providing rural and urban customers with the HIGHEST QUALITY products and services. We will enhance our producers' profitability, EXCEED customer expectations and keep our cooperative financially STRONG."

2 1

Coded TCP
by Rodrick Brown 24 Oct '12

24 Oct '12

"With coded TCP, blocks of packets are clumped together and then transformed into algebraic equations that describe the packets. If part of the message is lost, the receiver can solve the equation to derive the missing data. The process of solving the equations is simple and linear, meaning it doesn't require much processing on behalf of the router/smartphone/laptop. In testing, the coded TCP resulted in some dramatic improvements. MIT found that campus WiFi (2% packet loss) jumped from 1Mbps to 16Mbps. On a fast-moving train (5% packet loss), the connection speed jumped from 0.5Mbps to 13.5Mbps." http://www.mit.edu/~medard/papers2011/Modeling%20Network%20Coded%20TCP.pdf --RB

6 6

Issues encountered with assigning .0 and .255 as usable addresses?
by Paul Zugnoni 24 Oct '12

24 Oct '12

Curious whether it's commonplace to find systems that automatically regard .0 and .255 IP addresses (ipv4) as src/dst in packets as traffic that should be considered invalid. When you have a pool of assignable addresses, you should expect to see x.x.x.0 and x.x.x.255 in passing traffic (ie. VIP or NAT pool, or subnets larger than /24). Yet I've run into a commercial IP mgmt product and getting reports of M$ ISA proxy that is specifically blocking traffic for an IP ending in .0 or .255. Any experience or recommendations? Besides replace the ISA proxy…. Since it's not mine to replace. Also curious whether there's an RFC recommending against the use of .0 or .255 addresses for this reason.

18 20