- NANOG - lists.nanog.org

OpenFlow @ GOOG
by Eugen Leitl 17 Apr '12

17 Apr '12

http://www.wired.com/wiredenterprise/2012/04/going-with-the-flow-google/all… Going With The Flow: Google’s Secret Switch To The Next Wave Of Networking By Steven Levy April 17, 2012 | 11:45 am | Categories: Data Centers, Networking In early 1999, an associate computer science professor at UC Santa Barbara climbed the steps to the second floor headquarters of a small startup in Palo Alto, and wound up surprising himself by accepting a job offer. Even so, Urs Hölzle hedged his bet by not resigning from his university post, but taking a year-long leave. He would never return. Hölzle became a fixture in the company — called Google. As its czar of infrastructure, Hölzle oversaw the growth of its network operations from a few cages in a San Jose co-location center to a massive internet power; a 2010 study by Arbor Networks concluded that if Google was an ISP it would be the second largest in the world (the largest is Tier 3, which services over 2,700 major corporations in 450 markets over 100,000 fiber miles.) ‘You have all those multiple devices on a network but you’re not really interested in the devices — you’re interested in the fabric, and the functions the network performs for you,’ Hölzle says. Google treats its infrastructure like a state secret, so Hölzle rarely speaks about it in public. Today is one of those rare days: at the Open Networking Summit in Santa Clara, California, Hölzle is announcing that Google essentially has remade a major part of its massive internal network, providing the company a bonanza in savings and efficiency. Google has done this by brashly adopting a new and radical open-source technology called OpenFlow. Hölzle says that the idea behind this advance is the most significant change in networking in the entire lifetime of Google. In the course of his presentation Hölzle will also confirm for the first time that Google — already famous for making its own servers — has been designing and manufacturing much of its own networking equipment as well. “It’s not hard to build networking hardware,” says Hölzle, in an advance briefing provided exclusively to Wired. “What’s hard is to build the software itself as well.” In this case, Google has used its software expertise to overturn the current networking paradigm. If any company has potential to change the networking game, it is Google. The company has essentially two huge networks: the one that connects users to Google services (Search, Gmail, YouTube, etc.) and another that connects Google data centers to each other. It makes sense to bifurcate the information that way because the data flow in each case has different characteristics and demand. The user network has a smooth flow, generally adopting a diurnal pattern as users in a geographic region work and sleep. The performance of the user network also has higher standards, as users will get impatient (or leave!) if services are slow. In the user-facing network you also need every packet to arrive intact — customers would be pretty unhappy if a key sentence in a document or e-mail was dropped. The internal backbone, in contrast, has wild swings in demand — it is “bursty” rather than steady. Google is in control of scheduling internal traffic, but it faces difficulties in traffic engineering. Often Google has to move many petabytes of data (indexes of the entire web, millions of backup copies of user Gmail) from one place to another. When Google updates or creates a new service, it wants it available worldwide in a timely fashion — and it wants to be able to predict accurately how quickly the process will take. “There’s a lot of data center to data center traffic that has different business priorities,” says Stephen Stuart, a Google distinguished engineer who specializes in infrastructure. “Figuring out the right thing to move out of the way so that more important traffic could go through was a challenge.” But Google found an answer in OpenFlow, an open source system jointly devised by scientists at Stanford and the University of California at Berkeley. Adopting an approach known as Software Defined Networking (SDN), OpenFlow gives network operators a dramatically increased level of control by separating the two functions of networking equipment: packet switching and management. OpenFlow moves the control functions to servers, allowing for more complexity, efficiency and flexibility. “We were already going down that path, working on an inferior way of doing software-defined networking,” says Hölzle. “But once we looked at OpenFlow, it was clear that this was the way to go. Why invent your own if you don’t have to?” Google became one of several organizations to sign on to the Open Networking Foundation, which is devoted to promoting OpenFlow. (Other members include Yahoo, Microsoft, Facebook, Verizon and Deutsche Telekom, and an innovative startup called Nicira.) But none of the partners so far have announced any implementation as extensive as Google’s. Why is OpenFlow so advantageous to a company like Google? In the traditional model you can think of routers as akin to taxicabs getting passengers from one place to another. If a street is blocked, the taxi driver takes another route — but the detour may be time-consuming. If the weather is lousy, the taxi driver has to go slower. In short, the taxi driver will get you there, but you don’t want to bet the house on your exact arrival time. With the software-defined network Google has implemented, the taxi situation no longer resembles the decentralized model of drivers making their own decisions. Instead you have a system like the one envisioned when all cars are autonomous, and can report their whereabouts and plans to some central repository which also knows of weather conditions and aggregate traffic information. Such a system doesn’t need independent taxi drivers, because the system knows where the quickest routes are and what streets are blocked, and can set an ideal route from the outset. The system knows all the conditions and can institute a more sophisticated set of rules that determines how the taxis proceed, and even figure whether some taxis should stay in their garages while fire trucks pass. Therefore, operators can slate trips with confidence that everyone will get to their destinations in the shortest times, and precisely on schedule. Continue reading ‘Going With The Flow: Google’s Secret Switch To The Next Wave Of Networking‘ … Making Google’s entire internal network work with SDN thus provides all sorts of advantages. In planning big data moves, Google can simulate everything offline with pinpoint accuracy, without having to access a single networking switch. Products can be rolled out more quickly. And since “the control plane” is the element in routers that most often needs updating, networking equipment is simpler and enduring, requiring less labor to service. Most important, the move makes network management much easier. By early this year, all of Google’s internal network was running on OpenFlow. ‘Soon we will able to get very close to 100 percent utilization of our network,’ Hölzle says. “You have all those multiple devices on a network but you’re not really interested in the devices — you’re interested in the fabric, and the functions the network performs for you,” says Hölzle. “Now we don’t have to worry about those devices — we manage the network as an overall thing. The network just sort of understands.” The routers Google built to accommodate OpenFlow on what it is calling “the G-Scale Network” probably did not mark not the company’s first effort in making such devices. (One former Google employee has told Wired’s Cade Metz that the company was designing its own equipment as early as 2005. Google hasn’t confirmed this, but its job postings in the field over the past few years have provided plenty of evidence of such activities.) With SDN, though, Google absolutely had to go its own way in that regard. “In 2010, when we were seriously starting the project, you could not buy any piece of equipment that was even remotely suitable for this task,” says Hotzle. “It was not an option.” The process was conducted, naturally, with stealth — even the academics who were Google’s closest collaborators in hammering out the OpenFlow standards weren’t briefed on the extent of the implementation. In early 2010, Google established its first SDN links, among its triangle of data centers in North Carolina, South Carolina and Georgia. Then it began replacing the old internal network with G-Scale machines and software — a tricky process since everything had to be done without disrupting normal business operations. As Hölzle explains in his speech, the method was to pre-deploy the equipment at a site, take down half the site’s networking machines, and hook them up to the new system. After testing to see if the upgrade worked, Google’s engineers would then repeat the process for the remaining 50 percent of the networking in the site. The process went briskly in Google’s data centers around the world. By early this year, all of Google’s internal network was running on OpenFlow. Though Google says it’s too soon to get a measurement of the benefits, Hölzle does confirm that they are considerable. “Soon we will able to get very close to 100 percent utilization of our network,” he says. In other words, all the lanes in Google’s humongous internal data highway can be occupied, with information moving at top speed. The industry considers thirty or forty percent utilization a reasonable payload — so this implementation is like boosting network capacity two or three times. (This doesn’t apply to the user-facing network, of course.) Though Google has made a considerable investment in the transformation — hundreds of engineers were involved, and the equipment itself (when design and engineering expenses are considered) may cost more than buying vendor equipment — Hölzle clearly thinks it’s worth it. Hölzle doesn’t want people to make too big a deal of the confirmation that Google is making its own networking switches — and he emphatically says that it would be wrong to conclude that because of this announcement Google intends to compete with Cisco and Juniper. “Our general philosophy is that we’ll only build something ourselves if there’s an advantage to do it — which means that we’re getting something we can’t get elsewhere.” To Hölzle, this news is all about the new paradigm. He does acknowledge that challenges still remain in the shift to SDN, but thinks they are all surmountable. If SDN is widely adopted across the industry, that’s great for Google, because virtually anything that happens to make the internet run more efficiently is a boon for the company. As for Cisco and Juniper, he hopes that as more big operations seek to adopt OpenFlow, those networking manufacturers will design equipment that supports it. If so, Hölzle says, Google will probably be a customer. “That’s actually part of the reason for giving the talk and being open,” he says. “To encourage the industry — hardware, software and ISP’s — to look down this path and say, ‘I can benefit from this.’” For proof, big players in networking can now look to Google. The search giant claims that it’s already reaping benefits from its bet on the new revolution in networking. Big time. Steven Levy Steven Levy's deep dive into Google, In The Plex: How Google Thinks, Works And Shapes Our Lives, was published in April, 2011. Steven also blogs at StevenLevy.com. Check out Steve's Google+ Profile . Read more by Steven Levy Follow @StevenLevy on Twitter.

2 1

Re: trouble with Paradyne Bitstorm 2600 DSLAM
by A.T. 17 Apr '12

17 Apr '12

Thanks. I have already tried interrupting boot process. Booting up with parameter 0x00020 (disable login security) don't seem affect outcome, both serial and management ethernet still asks login. On Tue, 2012-04-17 at 09:09 -0500, Chris Boyd wrote: > Try pressing enter several times after powering the system up. > > --Chris > > On Apr 17, 2012, at 8:19 AM, A.T. wrote: > > > Hello, all! > > > > > > I have Paradyne Bitstorm 2600 DSLAM, but no password. Is it possible to > > reset this device to factory configuration? Manuals don't say much, only > > specified way to restore factory settings assume logged in > > administrator. > > > > Thanks! > > > > > > > > > > > > > > !DSPAM:1001,4f8d6e61937107248971901! >

1 0

trouble with Paradyne Bitstorm 2600 DSLAM
by A.T. 17 Apr '12

17 Apr '12

Hello, all! I have Paradyne Bitstorm 2600 DSLAM, but no password. Is it possible to reset this device to factory configuration? Manuals don't say much, only specified way to restore factory settings assume logged in administrator. Thanks!

1 0

Network Storage
by Maverick 16 Apr '12

16 Apr '12

Hello Everyone, Can you please comment on what is best solution for storing network traffic. We have been graciously granted access by our network administrator to capture traffic but the one Tera byte disk space is no match with the data that we are seeing, so it fills up quickly. We can't get additional space on the server itself so I am looking for some external solutions. Can you please suggest something that would be best for Gbps speeds . Best, Ali

18 20

Communal Dining
by Ronald Bonica 16 Apr '12

16 Apr '12

Folks, You are all invited to an extremely informal dinner at our house at 6PM on Saturday, April 21. Spouses and children are all invited. I will bake bread and put on a huge pot of soup. If your kids are picky eaters, feel free to bring whatever they will eat. Our house is located at: 241 West Meadowland Lane Sterling, Virgina 20164 703 430 8379 -------------------------- Ron and Nancy Bonica vcard: www.bonica.org/ron/ronbonica.vcf

3 2

FW: Communal Dining
by Ronald Bonica 16 Apr '12

16 Apr '12

Folks, Sorry, you are not all invited to dinner. I apologize for the spam. MS mail address completion helped me a little more than I wanted. Ron > -----Original Message----- > From: Ronald Bonica > Sent: Monday, April 16, 2012 10:05 AM > To: 'FRBILLS(a)aol.com'; 'Nicholas Hinko'; 'Susan Hinko'; jay cuasay; > 'William Richey'; Will Ress; 'maria torres'; 'landreyko(a)gmail.com'; > nanog(a)nanog.org > Subject: Communal Dining > > Folks, > > You are all invited to an extremely informal dinner at our house at 6PM > on Saturday, April 21. Spouses and children are all invited. I will > bake bread and put on a huge pot of soup. If your kids are picky > eaters, feel free to bring whatever they will eat. > > Our house is located at: > > 241 West Meadowland Lane > Sterling, Virgina 20164 > 703 430 8379 > > -------------------------- > Ron and Nancy Bonica > vcard: www.bonica.org/ron/ronbonica.vcf >

2 1

facebook ipv6 is down?
by Ido Szargel 16 Apr '12

16 Apr '12

Hi, It seems that on the last 3 hours facebook isn't available via ipv6, when tracing from HE I don't even get to FB network, only as far as Ashburn on their network, When tracing from nl-ix I get to facebook network but the trace stops traceroute6 -I www.v6.facebook.com traceroute to www.v6.facebook.com (2620:0:1cfe:face:b00c:0:3:0), 30 hops max, 40 byte packets 1 2a01:1b0:705::f (2a01:1b0:705::f) 3.858 ms 3.744 ms 3.504 ms 2 bit.telecity2.nlsix.net (2001:7f8:13::a501:2859:2) 2.133 ms 2.061 ms 1.923 ms 3 br01.ams1.tfbnw.net (2001:7f8:1::a503:2934:1) 3.276 ms 3.159 ms 3.000 ms 4 ae28.bb02.iad1.tfbnw.net (2620:0:1cff:dead:beef::485) 90.835 ms 91.009 ms 90.953 ms 5 ae12.bb02.sjc1.tfbnw.net (2620:0:1cff:dead:beef::85) 160.883 ms 160.820 ms 160.897 ms 6 ae2.pr01.sjc1.tfbnw.net (2620:0:1cff:dead:beef::10) 152.688 ms 152.638 ms 152.890 ms 7 * * * 8 * * * 9 * * * 10 * * * Is anyone else having the same issue? Thanks, Ido

8 9

Re: Most energy efficient (home) setup
by Mike 16 Apr '12

16 Apr '12

I think the simple test for this problem is to take a non-ECC machine, boot from a CD/USB Key/etc with memtest or memtest86+ on it, and see if you get errors over the course of a few days. Getting errors will certainly prove that this problem exists (or that you have bad ram).

2 1

SORBS?!
by Chris Conn 14 Apr '12

14 Apr '12

Hello, Is anyone from SORBS still listening? We have a few IP addresses here and there that are listed, one in particular that has been for a spam incident from over a year ago. The "last spam" date is 03/05/2011 according to their lookup tools. We don't have access to their Net Manager even if our ARIN POC corresponds to the account on their system we opened a while ago. We use their ISP feedback form and never get any responses back. Is SORBS still relevant and functional? Sincerely, Chris Conn B2B2C.ca

39 65

IPv6 Launch day preparation
by Pierre-Yves Maunier 14 Apr '12

14 Apr '12

Hello all, we've had an interesting day today in Paris discussing about how to prepare the IPv6 implementation in various networks. With a couple of vendors and software companies (Juniper, Cisco, A10 Networks, Fortinet, BreakingPoint, StoneSoft, Infoblox, PaloAlto) we decided to make a very technical day about IPv6 transition techniques and various ways to implement it in different type of networks. We've had the morning for some mechanisms presentations and all the afternoon with a lab interconnecting all the vendors and having various scenarios to test all the mechanisms we discussed in the morning. The lab description can be found here : http://g6.asso.fr/wp-content/uploads/2012/03/ipv6-launch-lab.pdf (it's in "frenglish" but really understable for english speaking only people, personnaly I'm dual stack french/english :-) ) Actually we had : - a Juniper MX480 with a service card to act as a eyeball network providing IPv6 only connectivity to their customers and a NAT64 (Juniper) + DNS64 (Infoblox) to allow them to reach IPv4 only contents - Stonesoft running a dual stack entreprise network (FW+IPS) that can reach all IPv4/IPv6 contents - A10 networks configured as a IPv4 load balancer and IPv6 NAT64 gateway for contents servers using private IPv4 only address space : The IPv4 only servers were reachable from IPv6 only machines - PaloAlto and Fortinet as IPv4/IPv6 Firewalls - an IPv4 only web server (to be accessible from the IPv6 only eyeball network thanks to the NAT64+DNS64 setup) - an IPv6 only web server And in order to stress all the vendors hardware/software, we had breaking point which was able to generate IPv4/IPv6 applicative traffic from several kpps/mbps to loads of Mpps/Gbps It was very interesting to have this setup on which anybody in the audience was able to ask for specific test/configuration to understand all the mechanisms. Unfortunately we did not have a link to the DFZ to run more interesting tests but we will probably have another Lab day with a better setup for this (and having all the people in the audience using their laptops to simulate eyeball customers). Our goal was mainly to make people aware about what can be done to move forward with IPv6 and I think we achieved this goal. If you have any comments, feel free to reply here. -- Pierre-Yves Maunier

2 1