Re: [Paper] B4: Experience with a Globally-Deployed Software Defined
On Sat, Aug 17, 2013 at 2:32 PM, Avi Freedman <freedman@freedman.net> wrote:
No, people never use *flow controllers* for anything.
People have been doing SDN since before Google was around. OK, so it was horrible expect scripts but it worked.
Not really.
Note I am talking about flow controllers in my first point. (And I was trying to be funny to match Todd's tone, though I guess it's dangerous to try to copy the master) Re: flow controllers - The idea of centralized decision makers doing something (typically per flow) has been proposed, in my experience, by those with little operational experience or those with extraordinarily constrained topoligies, types of traffic, and usually external filtering to constrain the types of traffic one could face. Because... There have been no proposals that I have seen (or that those who are at the Major Vendors who follow it more closely tell me about when I ask a few times/year) to actually deal with the every-packet-is-a-flow problem we saw first with 7206VXRs and that remain a real possibility for Internet- connected networks. Distributing flow controllers and making them hierarchical doesn't seem to help in the architectures that I've seen proposed. So it seems to be of use only for very tiny networks or for very constrained and filtered or non Internet-connected topologies. I'd be interested to be shown otherwise.
Automatic reconfiguration of routers is not what a software-defined network is.
It is one of the things (but not all of the things) that SDN provides.
A software defined network is one where the forwarding behavior can be completely defined in software running outside of the devices that perform the forwarding.
That said, I wince every time someone starts talking about (not suggesting you are here but many do) making the routing engineer or designer in a box that sits on the bottom or besides the network. Those who have experience and/or run larger infrastructure usually say words like "of course we have to worry about feedback loops" but many don't. I think innovation is great but I don't think there are that many shops that are better off writing their own control pane (centralized, distribtued, whatever) right now. It's worth remembering that Google is a software company. They are far ahead in software defined everything.
You can write expect scripts all day; but you cannot turn your basic switch into a Load balancer or stateful firewall with one. or decide in real time exactly which destination Ethernet ports a packet coming in a certain port is going to touch, without having structured VLANs and static MAC tables on the switches ahead of time.
Changing device configurations with expect scripts is a limited tiny subset of what SDN is.
True, but the number of production environments that are going to be more stable or scalable by having people build their own control logic is pretty small in my experience. And being able to debug and reach out to a community of operators with a common set of experience of what to do, not to do, and how to debug is extraordinarily valuable for production networks. When I look at most of the non-Google big guys, SDN means pushing the vendors for better control plane instrumentation and ability to program (but more on the instrumentation side as where the gaps have been), and potentially to get some cross-provider way of doing the above. + having merchant silicon one can get/use for cheap, typically for more constrained topologies, doing pretty dumb switching and/or routing stuff.
-JH
Where I see the delta a lot given the customer conversations I have is in the magic provisioning of cloud network infrastructures. New school SDN is that everything is a tunnel, magic software maps things, commercial providers doing this uniformly have to aggressively rate- limit their clients, and performance for content delivery is limited because the hypervisors must be briding and can't do PCI passthrough or SR/IOV. Old school SDN (not really that old school) is API-based provisioning of network devices with vendor support (let's say Juniper) to do filtering, VLANs, and shaping and tunnels where needed. It'll definitely be interesting to see where things go over the next few years. I know tens of companies who have run away from cloud providers with new(er) school SDN-ish infrastructures for the simplicity of just having some high performance dedicated machines/hypervisors with dead simple switching infrastructure. Anyway, innovation is great but I just see few companies with the understanding to go build their own control plane software to connect to the Internet with. And those vendors who do build it will get borged by one of the routing/switching vendors and things will become product features, differentiated by providers, most likely. (Though I hope not) Avi
On Sat, Aug 17, 2013 at 8:08 PM, Avi Freedman <freedman@freedman.net> wrote:
On Sat, Aug 17, 2013 at 2:32 PM, Avi Freedman <freedman@freedman.net> wrote:
OK, so it was horrible expect scripts but it worked. Not really.
The idea of centralized decision makers doing something (typically
per flow) has been proposed, in my experience, by those with little operational experience or those with extraordinarily constrained topoligies, types of traffic, and usually external filtering to constrain the types of traffic one could face.
Flow controllers are for constrained topologies. You have some interesting problems, if you decide to try to bring flow controllers into a carrier network, and have them processing flows coming in from an unprotected network. How do you feel about a few billion flows per second, thanks to the attacker's randomized source IPs and port numbers? You might very well have to "constrain the topology" by having a capability to provide a flow entry for the destination IP address, and associating all the traffic with just one "flow".
closely tell me about when I ask a few times/year) to actually deal with the every-packet-is-a-flow problem we saw first with 7206VXRs and that remain a real possibility for Internet- connected networks. Distributing flow controllers and making
The centralized controller doesn't work well in topologies where you cannot take the first packet -- setup the flow tables on the devices, and keep further control traffic limited in number of messages and limited in number of bits. You need a network path for control traffic, you need CPU capacity for your flow processing -- and there will always be RTT latency and various other penalties incurred by extra overhead to reach a remote controller that is geographically farther away from the route processor since the speed of light is finite, and any centralized controller will have finite capacity... Having all the flow processing on central controllers is the opposite of load-balancing --- it is concentration of a distributed load; which is a recipe for creating a capacity bottleneck on the controller and on all the devices that have to have uncongested paths to the controllers... It's a similar idea to storing part of your router's RIB on a hard disk or magnetic tape, because RAM is so expensive ---- you're going to increase the forwarding delays of some traffic, or some connection initializations by doing so. Therefore; the only flow controller that really makes sense in all situations is eventually going to be one that can download its entire state into every router --- that is: A route controller program built into every router and switch (even if defined by software). Such a product for SDN doesn't exist yet? -- other than traditional networking devices which don't allow you to implement an API and download arbitrary code to them in a vendor-independent way. It seems like there is still a lot of work and standardization for vendors to be doing, before a majority of ISP networks could consider using SDN anywhere.
So it seems to be of use only for very tiny networks or for very constrained and filtered or non Internet-connected topologies.
Yes. That seems to be where the greatest benefits of current SDN products could realistically be experienced at the present time; small filtered and private topologies, where the controller as a bottleneck risk is minimal.
in software running outside of the devices that perform the forwarding.
That said, I wince every time someone starts talking about (not suggesting you are here but many do) making the routing engineer or designer in a box that sits on the bottom or besides the network.
I don't know -- if you make a box that can help configure your network; you'll still need a routing engineer to setup and maintain that box, make sure it continues to work properly, spend all day on hold waiting for support to help get your network back online, and diagnose things when an inevitable bug crops up and starts costing more damage than money saved by deploying the magic box. I think innovation is great but I don't think there are that many shops
that are better off writing their own control pane (centralized, distribtued, whatever) right now.
Yes. Most would have no business trying to write their own control plane. Not too long ago someone wrote about "Tempering your MacGuyver" streak. Which is exactly what should apply here; SDN is something to learn about developments in, and one of those new flow controller schemes could be used in a pinch as a way around certain problems ---- new tech doesn't mean it's somehow better, or somehow magically defeats inherent physical or computational problems. Otherwise Rule #1 of building reliable infrastructure -- is use time-tested technology in well-understood vendor supported configurations in every instance, except in the situations where there exists compelling justification to vary. Flow controllers won't belong on everyone's network just because they're new tech -- until they have more to offer, it would be the exception to the rule that you should want one.
It's worth remembering that Google is a software company. They are far ahead in software defined everything.
It could be interesting if Google released some of their sw and sw/hw solutions I suppose. Of course there are likely to be people copying them in such case, because of a perception (true or false), that using SDN tech early on is part of their success story.
When I look at most of the non-Google big guys, SDN means pushing the vendors for better control plane instrumentation and ability to program (but more on the instrumentation side as where the gaps have been), and potentially to get some cross-provider way of doing the above.
That bit is the most useful. Network devices have notoriously poor control plane instrumentation and programmability; SNMP and Netflow have their limitations. Improvements in these areas are valuable in their own right. -- -JH
participants (2)
-
freedman＠freedman.net -
Jimmy Hess