Open Souce Network Operating Systems
If one were to deploy whitebox switches, X86 servers, low cost ARM and MIBPS CPE devices, and basically anything that can run linux today, what network operating system would you recommend? The goal would be to have a universal network operating system that runs across a variety of devices.
From low cost residential CPE's with wifi to switches to BGP speaking routers. Is there anything that can do it all today?
I will use something like OpenWRT as an example. I don't consider this anywhere near carrier grade, but it runs on X86 and low cost routers. I don't think it will run on whitebox switches though. Mikrotik RouterOS would be another example as it can run on low cost Routerboards, and X86 servers. But it is not opensouce. Is there any up and coming projects to look into?
Hey, Have a look at a similar thread from recently: http://seclists.org/nanog/2018/Jan/180 /Ruairi On 17 January 2018 at 14:28, Colton Conor <colton.conor@gmail.com> wrote:
If one were to deploy whitebox switches, X86 servers, low cost ARM and MIBPS CPE devices, and basically anything that can run linux today, what network operating system would you recommend? The goal would be to have a universal network operating system that runs across a variety of devices. From low cost residential CPE's with wifi to switches to BGP speaking routers. Is there anything that can do it all today?
I will use something like OpenWRT as an example. I don't consider this anywhere near carrier grade, but it runs on X86 and low cost routers. I don't think it will run on whitebox switches though.
Mikrotik RouterOS would be another example as it can run on low cost Routerboards, and X86 servers. But it is not opensouce.
Is there any up and coming projects to look into?
There's AT&T's dNOS effort[1], though I think that wasn't really targeting CPE so much as DC and carrier type WAN gear. A single platform for DC, aggregation, and other SP roles is already pretty ambitious. Adding CPE into the mix as well is another big stretch even beyond that. It's also more at the "initiative" stage than anything fully fledged, afaict. -- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal [1] https://about.att.com/content/dam/innovationblogdocs/att-routing-nos-open-ar... On Wed 2018-Jan-17 14:35:09 +0000, Ruairi Carroll <ruairi.carroll@gmail.com> wrote:
Hey,
Have a look at a similar thread from recently: http://seclists.org/nanog/2018/Jan/180
/Ruairi
On 17 January 2018 at 14:28, Colton Conor <colton.conor@gmail.com> wrote:
If one were to deploy whitebox switches, X86 servers, low cost ARM and MIBPS CPE devices, and basically anything that can run linux today, what network operating system would you recommend? The goal would be to have a universal network operating system that runs across a variety of devices. From low cost residential CPE's with wifi to switches to BGP speaking routers. Is there anything that can do it all today?
I will use something like OpenWRT as an example. I don't consider this anywhere near carrier grade, but it runs on X86 and low cost routers. I don't think it will run on whitebox switches though.
Mikrotik RouterOS would be another example as it can run on low cost Routerboards, and X86 servers. But it is not opensouce.
Is there any up and coming projects to look into?
On Wed, Jan 17, 2018 at 08:28:13AM -0600, Colton Conor wrote:
The goal would be to have a universal network operating system that runs across a variety of devices.
And for certain uses, that would be handy. Of course it would also be handy to an attacker who found or purchased or was given an exploitable hole in that OS, because then they could pwn the entire intrastructure simultaneously. One-stop shopping, as it were. ---rsk
Yeah, it'd be silly for organisations to try and standardise their environments for services or infrastructure. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rich Kulawiec Sent: Thursday, 18 January 2018 4:03 AM To: nanog@nanog.org Subject: Re: Open Souce Network Operating Systems On Wed, Jan 17, 2018 at 08:28:13AM -0600, Colton Conor wrote:
The goal would be to have a universal network operating system that runs across a variety of devices.
And for certain uses, that would be handy. Of course it would also be handy to an attacker who found or purchased or was given an exploitable hole in that OS, because then they could pwn the entire intrastructure simultaneously. One-stop shopping, as it were. ---rsk
On Wed 2018-Jan-17 23:11:14 +0000, Matthew Smee <matthew.smee@sydney.edu.au> wrote:
Yeah, it'd be silly for organisations to try and standardise their environments for services or infrastructure.
I'm somewhat in two minds there. Options to tackle operational complexity/expense: Option 1: Require a homogeneous environment or minimize vendors/platforms as much as possible. Option 2: Accept vendor/platform diversity as inevitable and build systems/abstractions around that. Is #1 achievable? If you're expending time/effort/resources achieving #1 and fall short, don't you have to do #2 anyway? Much has also been said on monocultures in infrastructure: having a single bug impact all of your gear sucks. If I can manage a pair of border routers, for instance, from two different vendors in an abstracted/consistent enough manner that I don't deal with their idiosyncrasies on a daily basis, am I not better off than running a single platform / code train in that function? -- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal
On 01/17/2018 07:48 PM, Hugo Slabbert wrote:
On Wed 2018-Jan-17 23:11:14 +0000, Matthew Smee <matthew.smee@sydney.edu.au> wrote:
Yeah, it'd be silly for organisations to try and standardise their environments for services or infrastructure.
Was this spoken tongue-in-check, or in all seriousness? I would say there is power in standardization. And, yes, there is risk depending upon your attack surface. When we concern ourselves with the attack service, how much time do we spend on mitigating issues on the edge (the attacked surface), vs internal infrastructure where we can apply labour and effort saving automation and orchestration?
I'm somewhat in two minds there. Options to tackle operational complexity/expense:
Option 1: Require a homogeneous environment or minimize vendors/platforms as much as possible.
Maybe deal with on a case by case basis? Or implement solutions which are 'easy' to mitigate?
Option 2: Accept vendor/platform diversity as inevitable and build systems/abstractions around that.
And do this in an intelligent manner, depending upon management and risk profiles. Infrastructure tends to be wide ranging. When one thinks about the big picture, where do you _really_ need the diversity, and where you can you gain the most by standardization? Standard engineering response: it depends. And I hope that readers are not trying to draw a line in the sand. I'm hoping that we are open to optimization and orchestration based upon the infrastructure at hand.
Is #1 achievable? If you're expending time/effort/resources achieving #1 and fall short, don't you have to do #2 anyway?
Doesn't this go the other way? ... that if you are spending so much effort that building infrastructure, and you can't get a maintainable/upgradeable/orchestratable solution in place, is #2 relevant?
Much has also been said on monocultures in infrastructure: having a single bug impact all of your gear sucks. If I can manage a pair of border routers, for instance, from two different vendors in an
but when I think about this, I'm not thinking about just border routers, I'm thinking about core routing, virtualization infrastructure, carrying customer private circuits, delivering traffic to individual customers, gear for telemetry, implementing security, .... When you think about all the devices involved in various levels and styles of service delivery, there are ways to make that homogenous, and much of it has various attack surfaces, and, well, vendors have their strengths and their weaknesses, so ... #1 a homogenous network makes it easier to intimately understand the possible weaknesses, and attempt protection mechansims, but for #2 with multiple vendors, the effort on platform education increases, depending upon the size of your shop, ...
abstracted/consistent enough manner that I don't deal with their idiosyncrasies on a daily basis, am I not better off than running a single platform / code train in that function?
or across many functions?
-- Raymond Burkholder ray@oneunified.net https://blog.raymond.burkholder.net -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
#Devil'sAdvocate On 01/17/2018 07:28 AM, Colton Conor wrote:
If one were to deploy whitebox switches, X86 servers, low cost ARM and MIBPS CPE devices, and basically anything that can run linux today, what network operating system would you recommend?
Linux. I fail to see the need for various packages that act like something else (or their own entity) to translate to underlying Linux configuration. Why not just put things in the various configuration files that Linux uses? Why have something that translates? If you are going to have a translation layer, why not do it with standard automation layers that you're likely already using. I.e. Ansible / Puppet / Chef / Salt et al.
The goal would be to have a universal network operating system that runs across a variety of devices. >From low cost residential CPE's with wifi to switches to BGP speaking routers. Is there anything that can do it all today?
Aside from the concern that others have mentioned about the same vuln on many pieces of equipment, I don't see a problem. Use a standard (or at least consistent in your org) kernel configuration (as necessary & appropriate for the hardware) and standard configuration of application & configuration. Compile as necessary for the architecture that you're on.
I will use something like OpenWRT as an example. I don't consider this anywhere near carrier grade, but it runs on X86 and low cost routers. I don't think it will run on whitebox switches though.
What is different about OpenWRT vs VyOS? It's my understanding that they are both running Linux and have some sort of (proprietary) configuration layer abstracting away the kernel.
Mikrotik RouterOS would be another example as it can run on low cost Routerboards, and X86 servers. But it is not opensouce.
Is open source a requirement? I consider it a strong desire, but not a requirement.
Is there any up and coming projects to look into?
Linux. -- Grant. . . . unix || die
I would start with following the Free Range Routing project, and related but independent (and more tangible) projects like pfSense (esp. the upcoming 3.0 release) and Cumulus Linux. Going deeper, perhaps Carrier Grade Linux, DPDK, and ONOS (all Linux Foundation projects). I think scaling vertically from CPEs to core stack is a stretch, especially if you mean a DIY approach, however. ----- Original Message ----- From: "Colton Conor" <colton.conor@gmail.com> To: "nanog" <nanog@nanog.org> Sent: Wednesday, January 17, 2018 9:28:13 AM Subject: Open Souce Network Operating Systems If one were to deploy whitebox switches, X86 servers, low cost ARM and MIBPS CPE devices, and basically anything that can run linux today, what network operating system would you recommend? The goal would be to have a universal network operating system that runs across a variety of devices.
From low cost residential CPE's with wifi to switches to BGP speaking routers. Is there anything that can do it all today?
I will use something like OpenWRT as an example. I don't consider this anywhere near carrier grade, but it runs on X86 and low cost routers. I don't think it will run on whitebox switches though. Mikrotik RouterOS would be another example as it can run on low cost Routerboards, and X86 servers. But it is not opensouce. Is there any up and coming projects to look into? -- Eron Lloyd Information Technology Director 717-344-5958 eron@mawcom.com MAW Communications, Inc.
Eron, Thanks for the advice. My understanding if Free Range Routing is a package of software that runs in linux, but not a full and true NOS right? Is pfSense 3.0 going to be dramatically different that the current version? I never considered this a NOS but more of a firewall platform with some routing capabilities. I looked into Cumulus Linux, but it seems to only run on the supported hardware which is while box switches. Can you run Cumulus Linux on a X86 server with intel NICs? Can you run Cumulus on a raspberry pi? Ideally I think I am looking to a Linux operating system that can run on multiple CPU architectures, has device support for Broadcom and other Merchant silicon switching and wifi adapters. On Thu, Jan 18, 2018 at 9:25 PM, Eron Lloyd <eron@mawcom.com> wrote:
I would start with following the Free Range Routing project, and related but independent (and more tangible) projects like pfSense (esp. the upcoming 3.0 release) and Cumulus Linux. Going deeper, perhaps Carrier Grade Linux, DPDK, and ONOS (all Linux Foundation projects). I think scaling vertically from CPEs to core stack is a stretch, especially if you mean a DIY approach, however.
----- Original Message ----- From: "Colton Conor" <colton.conor@gmail.com> To: "nanog" <nanog@nanog.org> Sent: Wednesday, January 17, 2018 9:28:13 AM Subject: Open Souce Network Operating Systems
If one were to deploy whitebox switches, X86 servers, low cost ARM and MIBPS CPE devices, and basically anything that can run linux today, what network operating system would you recommend? The goal would be to have a universal network operating system that runs across a variety of devices. From low cost residential CPE's with wifi to switches to BGP speaking routers. Is there anything that can do it all today?
I will use something like OpenWRT as an example. I don't consider this anywhere near carrier grade, but it runs on X86 and low cost routers. I don't think it will run on whitebox switches though.
Mikrotik RouterOS would be another example as it can run on low cost Routerboards, and X86 servers. But it is not opensouce.
Is there any up and coming projects to look into? -- Eron Lloyd Information Technology Director 717-344-5958 eron@mawcom.com MAW Communications, Inc.
On Sat, Jan 20, 2018 at 9:32 AM, Colton Conor <colton.conor@gmail.com> wrote:
My understanding if Free Range Routing is a package of software that runs in linux, but not a full and true NOS right?
Why not consider Linux a NOS? Installing Free Range Routing adds control plane protocols: BGP, OSPF, ISIS, etc.
I looked into Cumulus Linux, but it seems to only run on the supported hardware which is while box switches. Can you run Cumulus Linux on a X86 server with intel NICs? Can you run Cumulus on a raspberry pi?
Cumulus Linux is basically Ubuntu with Free Range Routing pre-installed along with a daemon that offloads forwarding from the Linux kernel to an ASIC. CumulusVX is a free Cumulus Linux virtual machine that is useful for staging / testing configurations since it has the same behavior as the hardware switch. On X86 servers with Intel NICs, just run Linux. Cumulus Host Pack can be installed to add Free Range Routing and other Cumulus tools on the server. Alternatively, you can choose any Linux control plane, automation, or monitoring tools and install them on the hosts and Cumulus Linux switches to unify management and control, e.g. Bird, collectd, telegraf, Puppet, Chef, Ansible, etc. Linux distros (including Ubuntu) are available for non-X86 hardware like Raspberry Pi etc.
Ideally I think I am looking to a Linux operating system that can run on multiple CPU architectures, has device support for Broadcom and other Merchant silicon switching and wifi adapters.
If you consider Linux as the NOS then it already meets these requirements.
Peter, Thanks for the information. Do you have a recommendation of which distribution of Linux to use for this? Is there one that is more network centric than another? On Sat, Jan 20, 2018 at 1:11 PM, Peter Phaal <peter.phaal@gmail.com> wrote:
On Sat, Jan 20, 2018 at 9:32 AM, Colton Conor <colton.conor@gmail.com> wrote:
My understanding if Free Range Routing is a package of software that runs in linux, but not a full and true NOS right?
Why not consider Linux a NOS? Installing Free Range Routing adds control plane protocols: BGP, OSPF, ISIS, etc.
I looked into Cumulus Linux, but it seems to only run on the supported hardware which is while box switches. Can you run Cumulus Linux on a X86 server with intel NICs? Can you run Cumulus on a raspberry pi?
Cumulus Linux is basically Ubuntu with Free Range Routing pre-installed along with a daemon that offloads forwarding from the Linux kernel to an ASIC. CumulusVX is a free Cumulus Linux virtual machine that is useful for staging / testing configurations since it has the same behavior as the hardware switch.
On X86 servers with Intel NICs, just run Linux. Cumulus Host Pack can be installed to add Free Range Routing and other Cumulus tools on the server. Alternatively, you can choose any Linux control plane, automation, or monitoring tools and install them on the hosts and Cumulus Linux switches to unify management and control, e.g. Bird, collectd, telegraf, Puppet, Chef, Ansible, etc.
Linux distros (including Ubuntu) are available for non-X86 hardware like Raspberry Pi etc.
Ideally I think I am looking to a Linux operating system that can run on multiple CPU architectures, has device support for Broadcom and other Merchant silicon switching and wifi adapters.
If you consider Linux as the NOS then it already meets these requirements.
I would second Peter's advise. Colton, for you I would recommended you visit Cumulus' web site and follow their tutorials. That should provide you with enough insights for your next step. On Saturday, January 20, 2018, 11:27:38 AM PST, Colton Conor <colton.conor@gmail.com> wrote: Peter, Thanks for the information. Do you have a recommendation of which distribution of Linux to use for this? Is there one that is more network centric than another? On Sat, Jan 20, 2018 at 1:11 PM, Peter Phaal <peter.phaal@gmail.com> wrote:
On Sat, Jan 20, 2018 at 9:32 AM, Colton Conor <colton.conor@gmail.com> wrote:
My understanding if Free Range Routing is a package of software that runs in linux, but not a full and true NOS right?
Why not consider Linux a NOS? Installing Free Range Routing adds control plane protocols: BGP, OSPF, ISIS, etc.
I looked into Cumulus Linux, but it seems to only run on the supported hardware which is while box switches. Can you run Cumulus Linux on a X86 server with intel NICs? Can you run Cumulus on a raspberry pi?
Cumulus Linux is basically Ubuntu with Free Range Routing pre-installed along with a daemon that offloads forwarding from the Linux kernel to an ASIC. CumulusVX is a free Cumulus Linux virtual machine that is useful for staging / testing configurations since it has the same behavior as the hardware switch.
On X86 servers with Intel NICs, just run Linux. Cumulus Host Pack can be installed to add Free Range Routing and other Cumulus tools on the server. Alternatively, you can choose any Linux control plane, automation, or monitoring tools and install them on the hosts and Cumulus Linux switches to unify management and control, e.g. Bird, collectd, telegraf, Puppet, Chef, Ansible, etc.
Linux distros (including Ubuntu) are available for non-X86 hardware like Raspberry Pi etc.
Ideally I think I am looking to a Linux operating system that can run on multiple CPU architectures, has device support for Broadcom and other Merchant silicon switching and wifi adapters.
If you consider Linux as the NOS then it already meets these requirements.
On Sat, Jan 20, 2018 at 11:26 AM, Colton Conor <colton.conor@gmail.com> wrote:
Thanks for the information. Do you have a recommendation of which distribution of Linux to use for this? Is there one that is more network centric than another?
Cumulus Linux, OpenSwitch, and Open Network Linux are all Debian based so there is probably a greater ecosystem of developers and network centric tools being built for Debian.
Feedback about Cumulus has been positive : https://www.mail-archive.com/cisco-nsp@puck.nether.net/msg66192.html if i am not mistaken, they have added lots of networking enhancements to the OS, they have videos on youtube that will paint the picture. On Sat, Jan 20, 2018 at 11:26 AM, Colton Conor <colton.conor@gmail.com> wrote:
Peter,
Thanks for the information. Do you have a recommendation of which distribution of Linux to use for this? Is there one that is more network centric than another?
On Sat, Jan 20, 2018 at 1:11 PM, Peter Phaal <peter.phaal@gmail.com> wrote:
On Sat, Jan 20, 2018 at 9:32 AM, Colton Conor <colton.conor@gmail.com> wrote:
My understanding if Free Range Routing is a package of software that
runs
in linux, but not a full and true NOS right?
Why not consider Linux a NOS? Installing Free Range Routing adds control plane protocols: BGP, OSPF, ISIS, etc.
I looked into Cumulus Linux, but it seems to only run on the supported hardware which is while box switches. Can you run Cumulus Linux on a X86 server with intel NICs? Can you run Cumulus on a raspberry pi?
Cumulus Linux is basically Ubuntu with Free Range Routing pre-installed along with a daemon that offloads forwarding from the Linux kernel to an ASIC. CumulusVX is a free Cumulus Linux virtual machine that is useful for staging / testing configurations since it has the same behavior as the hardware switch.
On X86 servers with Intel NICs, just run Linux. Cumulus Host Pack can be installed to add Free Range Routing and other Cumulus tools on the server. Alternatively, you can choose any Linux control plane, automation, or monitoring tools and install them on the hosts and Cumulus Linux switches to unify management and control, e.g. Bird, collectd, telegraf, Puppet, Chef, Ansible, etc.
Linux distros (including Ubuntu) are available for non-X86 hardware like Raspberry Pi etc.
Ideally I think I am looking to a Linux operating system that can run on multiple CPU architectures, has device support for Broadcom and other Merchant silicon switching and wifi adapters.
If you consider Linux as the NOS then it already meets these requirements.
Colton, Maybe it is obvious to some, but I just want to point out that the reason Cumulus Linux publishes list of supported hardware is kind of two fold: 1st is Linux inherently doesn't program the hardware. So if you install Ubuntu on some Quanta switch, you still need a way to program the ASIC. Cumulus Linux is open source with the exception of switchd, which is what they use to take network state from the kernel and program the silicone with it. switchd can only program "supported" silicon. 2nd (and I think that applies to any OS, not just a NOS) is that supported hardware is the one they know how to (or taught linux to) control the LEDs, the power supplies, the fans, etc. Technically you can run Cumulus Linux on some unsupported switches, but I would not put that in production. So if you accept that linux is an OS and it needs drivers to drive the hardware, you can, hopefully, see that you can't just take any linux distro and run it on just any switch. You will need drivers that this linux OS can use to drive the hardware (including the ASIC) that you chose to run this linux on. Cumulus Networks does just that. They provide you with the drivers for the ASIC and environmentals. And since they do it, they list the hardware they support (aka provided the drivers for). Hope this helps. -Andrey. --Andrey On Thu, May 3, 2018 at 12:29 AM, Kasper Adel <karim.adel@gmail.com> wrote:
Feedback about Cumulus has been positive :
https://www.mail-archive.com/cisco-nsp@puck.nether.net/msg66192.html
if i am not mistaken, they have added lots of networking enhancements to the OS, they have videos on youtube that will paint the picture.
On Sat, Jan 20, 2018 at 11:26 AM, Colton Conor <colton.conor@gmail.com> wrote:
Peter,
Thanks for the information. Do you have a recommendation of which distribution of Linux to use for this? Is there one that is more network centric than another?
On Sat, Jan 20, 2018 at 1:11 PM, Peter Phaal <peter.phaal@gmail.com> wrote:
On Sat, Jan 20, 2018 at 9:32 AM, Colton Conor <colton.conor@gmail.com> wrote:
My understanding if Free Range Routing is a package of software that
runs
in linux, but not a full and true NOS right?
Why not consider Linux a NOS? Installing Free Range Routing adds control plane protocols: BGP, OSPF, ISIS, etc.
I looked into Cumulus Linux, but it seems to only run on the supported hardware which is while box switches. Can you run Cumulus Linux on a X86 server with intel NICs? Can you run Cumulus on a raspberry pi?
Cumulus Linux is basically Ubuntu with Free Range Routing pre-installed along with a daemon that offloads forwarding from the Linux kernel to an ASIC. CumulusVX is a free Cumulus Linux virtual machine that is useful for staging / testing configurations since it has the same behavior as the hardware switch.
On X86 servers with Intel NICs, just run Linux. Cumulus Host Pack can be installed to add Free Range Routing and other Cumulus tools on the server. Alternatively, you can choose any Linux control plane, automation, or monitoring tools and install them on the hosts and Cumulus Linux switches to unify management and control, e.g. Bird, collectd, telegraf, Puppet, Chef, Ansible, etc.
Linux distros (including Ubuntu) are available for non-X86 hardware like Raspberry Pi etc.
Ideally I think I am looking to a Linux operating system that can run
on
multiple CPU architectures, has device support for Broadcom and other Merchant silicon switching and wifi adapters.
If you consider Linux as the NOS then it already meets these requirements.
❦ 3 mai 2018 13:39 -0700, Andrey Khomyakov <khomyakov.andrey@gmail.com> :
1st is Linux inherently doesn't program the hardware. So if you install Ubuntu on some Quanta switch, you still need a way to program the ASIC. Cumulus Linux is open source with the exception of switchd, which is what they use to take network state from the kernel and program the silicone with it. switchd can only program "supported" silicon.
Since a few years, Linux has an offload framework for L2/L3 (switchdev). There is a toy driver (Rocker, supported by QEMU) and several silicons supported (at least Mellanox Spectrum, but it seems there are a few others). -- The mind is its own place, and in itself Can make a Heav'n of Hell, a Hell of Heav'n. -- John Milton
participants (14)
-
Andrey Khomyakov -
Colton Conor -
Edwin Pers -
Eron Lloyd -
Grant Taylor -
Hugo Slabbert -
i mawsog -
Kasper Adel -
Matthew Smee -
Peter Phaal -
Raymond Burkholder -
Rich Kulawiec -
Ruairi Carroll -
Vincent Bernat