Last-resort tunnel recommendations on WAN network ...
Greetings, Dear Community! Consider the following scenario: major colo with a pair of transits, peering, and a single transport back to another colo on our backbone. Transport carries public but also several overlays (VRFs) for management and whatnot. If the transport fails, we're good on transit/peering, but we can't get back to the mothership for mgmt. We're looking at solutions (secure tunnels over transit) to bring the severed colo back to "HQ" ... looking at a hub/spoke topology with the intent of possibly doing this more than once. Requirements: * Multiple VRFs across the tunnel * OSPF - each VRF should have its own instance, so we need something that supports interface-based tunneling since IPsec doesn't handle multicast well. Open to other tunneling strategies. Wireguard? OpenVPN? * v6 a plus (OSPFv3) * 10G should suffice across the board, but it should have interfaces that are LAGable. The appliances we have considered so far do most if not all of these things, but they come with a lot of features (and cost) we simply don't need (e.g., UTM, DPI) Also open to something server (VM) based since our traffic requirements aren't that significant. Easy to support is obviously a plus. Curious if others have had similar needs and how they solved this problem. Recommendations (good or bad) greatly appreciated. Thank you! - bryan
Hi Bryan! Just wondering if you could be over complicating things. One option is to continue to manage your devices in-band with your current strategy and if that fails, switch to an out-of-band solution. This way, the box is still reachable no matter what happens short of a complete power outage or worse. One option to do this is with Opengear hardware (https://opengear.com/) and using their Lighthouse software (https://opengear.com/products/lighthouse/) to manage it back at HQ. There are other options out there as well but this could be cheaper than buying a new server for wireguard (also a viable option) or a full-on networking device. Of course, there might be other considerations given your statement "management and whatnot" but if all you are concerned about is contacting the box when a line goes down, this is might be a good option for your use case. Regards, Michael The views and opinions included in this email belong to the author and are not representative of the views and opinions of the company which employs me. If you find a spelling or grammatical error, you may keep it.
On 12/03/2026 20:25, Bryan Holloway via NANOG wrote:
* OSPF - each VRF should have its own instance, so we need something that supports interface-based tunneling since IPsec doesn't handle multicast well. Open to other tunneling strategies. Wireguard? OpenVPN?
We've built a DCN network for our optical backbone based on pfSense and FreeBSD with WireGuard, OSPF and BGP, across diverse DIA links in each data centre. Works pretty good. WireGuard is awesome! Can't imagine how we made IPSec work :-)... Mark.
Linux + WireGuard does most of what you need easily and all of what you described with some effort. I’d use a separate egg tunnel for each VRF rather than trying your mix them, but you do you. Owen
On Mar 12, 2026, at 19:58, Mark Tinka via NANOG <nanog@lists.nanog.org> wrote:
On 12/03/2026 20:25, Bryan Holloway via NANOG wrote:
* OSPF - each VRF should have its own instance, so we need something that supports interface-based tunneling since IPsec doesn't handle multicast well. Open to other tunneling strategies. Wireguard? OpenVPN?
We've built a DCN network for our optical backbone based on pfSense and FreeBSD with WireGuard, OSPF and BGP, across diverse DIA links in each data centre.
Works pretty good.
WireGuard is awesome! Can't imagine how we made IPSec work :-)...
Mark. _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/5QSGH5FP...
Mikrotik can do all of that. At 10gig speeds without issues. Contact linktechs.net if you need suggestions, etc. -----Original Message----- From: Bryan Holloway via NANOG <nanog@lists.nanog.org> Sent: Thursday, March 12, 2026 2:25 PM To: North American Network Operators Group <nanog@lists.nanog.org> Cc: Bryan Holloway <bryan@shout.net> Subject: Last-resort tunnel recommendations on WAN network ... Greetings, Dear Community! Consider the following scenario: major colo with a pair of transits, peering, and a single transport back to another colo on our backbone. Transport carries public but also several overlays (VRFs) for management and whatnot. If the transport fails, we're good on transit/peering, but we can't get back to the mothership for mgmt. We're looking at solutions (secure tunnels over transit) to bring the severed colo back to "HQ" ... looking at a hub/spoke topology with the intent of possibly doing this more than once. Requirements: * Multiple VRFs across the tunnel * OSPF - each VRF should have its own instance, so we need something that supports interface-based tunneling since IPsec doesn't handle multicast well. Open to other tunneling strategies. Wireguard? OpenVPN? * v6 a plus (OSPFv3) * 10G should suffice across the board, but it should have interfaces that are LAGable. The appliances we have considered so far do most if not all of these things, but they come with a lot of features (and cost) we simply don't need (e.g., UTM, DPI) Also open to something server (VM) based since our traffic requirements aren't that significant. Easy to support is obviously a plus. Curious if others have had similar needs and how they solved this problem. Recommendations (good or bad) greatly appreciated. Thank you! - bryan _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/77KTXLHW...
Thank you to everyone who offered up suggestions. To summarize, I agree that a UNIX VM is the most flexible solution, but we have concerns about supporting it. Router-jocks won't know how to troubleshoot if the guy who put it together gets hit by a bus. Yes, Wireguard is hands-down easier to implement than IPsec!! I love it. I use it on my home networks, and it was trivial to set up. If only there were more appliances that used it out of the box. This would be my ideal solution. And yes -- MikroTik supports Wireguard, but in our experience, Mikrotik's VRF implementation isn't ready for prime-time. Thanks again to everyone that chimed in. Very much appreciated! - bryan On 3/12/26 19:25, Bryan Holloway via NANOG wrote:
Greetings, Dear Community!
Consider the following scenario: major colo with a pair of transits, peering, and a single transport back to another colo on our backbone.
Transport carries public but also several overlays (VRFs) for management and whatnot.
If the transport fails, we're good on transit/peering, but we can't get back to the mothership for mgmt.
We're looking at solutions (secure tunnels over transit) to bring the severed colo back to "HQ" ... looking at a hub/spoke topology with the intent of possibly doing this more than once.
Requirements: * Multiple VRFs across the tunnel
* OSPF - each VRF should have its own instance, so we need something that supports interface-based tunneling since IPsec doesn't handle multicast well. Open to other tunneling strategies. Wireguard? OpenVPN?
* v6 a plus (OSPFv3)
* 10G should suffice across the board, but it should have interfaces that are LAGable.
The appliances we have considered so far do most if not all of these things, but they come with a lot of features (and cost) we simply don't need (e.g., UTM, DPI)
Also open to something server (VM) based since our traffic requirements aren't that significant.
Easy to support is obviously a plus.
Curious if others have had similar needs and how they solved this problem.
Recommendations (good or bad) greatly appreciated.
Thank you!
- bryan
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/ nanog@lists.nanog.org/message/77KTXLHWHFQTPIGG7EQCWQVNZVTJP3TS/
[offlist] If your router jocks can’t handle networking and routing on Linux with FRR, WireGuard, and systemd-networks, then you’ve fired the wrong router jocks. Yes, there are syntactic differences and a little bit of implementation detail to learn, but the process of learning route and forwarding packets is still fundamentally the same and the rest is no more difficult than switching between Cisco, juniper, artists, etc. FWIW, I just went through the process of the Southern California Linux Expo migrating our routers from Juniper to NixOS, so I do understand the process well. Router jocks can be trained. I am available to train and/or implement for $250/hr. Let me know if that would be helpful. Owen (Router jock and Linux admin)
On Mar 17, 2026, at 07:43, Bryan Holloway via NANOG <nanog@lists.nanog.org> wrote:
Thank you to everyone who offered up suggestions.
To summarize, I agree that a UNIX VM is the most flexible solution, but we have concerns about supporting it. Router-jocks won't know how to troubleshoot if the guy who put it together gets hit by a bus.
Yes, Wireguard is hands-down easier to implement than IPsec!! I love it. I use it on my home networks, and it was trivial to set up. If only there were more appliances that used it out of the box. This would be my ideal solution.
And yes -- MikroTik supports Wireguard, but in our experience, Mikrotik's VRF implementation isn't ready for prime-time.
Thanks again to everyone that chimed in. Very much appreciated!
- bryan
On 3/12/26 19:25, Bryan Holloway via NANOG wrote: Greetings, Dear Community! Consider the following scenario: major colo with a pair of transits, peering, and a single transport back to another colo on our backbone. Transport carries public but also several overlays (VRFs) for management and whatnot. If the transport fails, we're good on transit/peering, but we can't get back to the mothership for mgmt. We're looking at solutions (secure tunnels over transit) to bring the severed colo back to "HQ" ... looking at a hub/spoke topology with the intent of possibly doing this more than once. Requirements: * Multiple VRFs across the tunnel * OSPF - each VRF should have its own instance, so we need something that supports interface-based tunneling since IPsec doesn't handle multicast well. Open to other tunneling strategies. Wireguard? OpenVPN? * v6 a plus (OSPFv3) * 10G should suffice across the board, but it should have interfaces that are LAGable. The appliances we have considered so far do most if not all of these things, but they come with a lot of features (and cost) we simply don't need (e.g., UTM, DPI) Also open to something server (VM) based since our traffic requirements aren't that significant. Easy to support is obviously a plus. Curious if others have had similar needs and how they solved this problem. Recommendations (good or bad) greatly appreciated. Thank you! - bryan _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/ nanog@lists.nanog.org/message/77KTXLHWHFQTPIGG7EQCWQVNZVTJP3TS/
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/X6D6QINE...
Plus in these days when you can always phone your friend Claude for help remembering obscure details, there's just no excuse :) On Tue, Mar 17, 2026 at 12:00 PM Owen DeLong via NANOG < nanog@lists.nanog.org> wrote:
[offlist]
If your router jocks can’t handle networking and routing on Linux with FRR, WireGuard, and systemd-networks, then you’ve fired the wrong router jocks.
Yes, there are syntactic differences and a little bit of implementation detail to learn, but the process of learning route and forwarding packets is still fundamentally the same and the rest is no more difficult than switching between Cisco, juniper, artists, etc.
FWIW, I just went through the process of the Southern California Linux Expo migrating our routers from Juniper to NixOS, so I do understand the process well. Router jocks can be trained.
I am available to train and/or implement for $250/hr. Let me know if that would be helpful.
Owen (Router jock and Linux admin)
On Mar 17, 2026, at 07:43, Bryan Holloway via NANOG < nanog@lists.nanog.org> wrote:
Thank you to everyone who offered up suggestions.
To summarize, I agree that a UNIX VM is the most flexible solution, but we have concerns about supporting it. Router-jocks won't know how to troubleshoot if the guy who put it together gets hit by a bus.
Yes, Wireguard is hands-down easier to implement than IPsec!! I love it. I use it on my home networks, and it was trivial to set up. If only there were more appliances that used it out of the box. This would be my ideal solution.
And yes -- MikroTik supports Wireguard, but in our experience, Mikrotik's VRF implementation isn't ready for prime-time.
Thanks again to everyone that chimed in. Very much appreciated!
- bryan
On 3/12/26 19:25, Bryan Holloway via NANOG wrote: Greetings, Dear Community! Consider the following scenario: major colo with a pair of transits, peering, and a single transport back to another colo on our backbone. Transport carries public but also several overlays (VRFs) for management and whatnot. If the transport fails, we're good on transit/peering, but we can't get back to the mothership for mgmt. We're looking at solutions (secure tunnels over transit) to bring the severed colo back to "HQ" ... looking at a hub/spoke topology with the intent of possibly doing this more than once. Requirements: * Multiple VRFs across the tunnel * OSPF - each VRF should have its own instance, so we need something that supports interface-based tunneling since IPsec doesn't handle multicast well. Open to other tunneling strategies. Wireguard? OpenVPN? * v6 a plus (OSPFv3) * 10G should suffice across the board, but it should have interfaces that are LAGable. The appliances we have considered so far do most if not all of these things, but they come with a lot of features (and cost) we simply don't need (e.g., UTM, DPI) Also open to something server (VM) based since our traffic requirements aren't that significant. Easy to support is obviously a plus. Curious if others have had similar needs and how they solved this problem. Recommendations (good or bad) greatly appreciated. Thank you! - bryan _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/ nanog@lists.nanog.org/message/77KTXLHWHFQTPIGG7EQCWQVNZVTJP3TS/
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/X6D6QINE...
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/CAO6I2HO...
On 2026-03-17 08:42, Bryan Holloway via NANOG wrote:
Thank you to everyone who offered up suggestions.
To summarize, I agree that a UNIX VM is the most flexible solution, but we have concerns about supporting it. Router-jocks won't know how to troubleshoot if the guy who put it together gets hit by a bus.
Automating the deployment and keeping configs in version control is one way to surface all the configuration aspects of a linux server. Commits with comments adds colour as changes progress. After having stuff like this in use for a while, the Director actually made a public statement that the org was so much more efficient and effective in service delivery. He turned it into a selling point.
Yes, Wireguard is hands-down easier to implement than IPsec!! I love it. I use it on my home networks, and it was trivial to set up. If only there were more appliances that used it out of the box. This would be my ideal solution.
And yes -- MikroTik supports Wireguard, but in our experience, Mikrotik's VRF implementation isn't ready for prime-time.
Linux VRF is. Plus EBGP and VxLAN and MPBGP and EVPN.
Thanks again to everyone that chimed in. Very much appreciated!
On 3/17/26 23:08, Raymond Burkholder via NANOG wrote:
Linux VRF is. Plus EBGP and VxLAN and MPBGP and EVPN.
Linux also has a lot of options aside from VRF that can be useful. It has robust policy routing as well as full-on network namespaces, so you have both a lighter option and a heavyweight but incredibly flexible option and all without resorting to full virtualization and the performance and administration hassles that can come with it. Since none are tied directly to signaling, you can also signal any of those three however you like assuming you can find a way to tie it into your routing daemon. That's usually impractical with policy routing, but VRFs and especially network namespaces (where you're just going to be running multiple routing daemon instances by design) give lots of options. I'm not aware of any "routing" platforms that provide that level of flexibility and especially without getting absurdly complicated to configure since they, by design, try to centralize everything into a single configuration database (though JunOS's config format is very nice, here). -- Brandon Martin
Agreed… However… Linux has a bunch of different possible ways to administer all of this stuff. The most comprehensive CLI mechanism is iproute2 (the “ip” command and some related constructs). The most comprehensive and capable persistent configuration database mechanism is systemd-networkd. Other persistent mechanisms include: Netplan (YAML based configurations that now days mostly get parsed into systemd-networkd files and then executed). Debian Traditional (the /etc/network/interfaces file and/or interfaces.d directory, ifup/ifdown/etc.) Lacks many features, but most can be worked around with iproute2 shell commands added to triggers in the file. NetworkManager (semi-capable, but any capabilities it lacks are just hard to cope with). My strong recommendation is take the time to learn systemd-networkd and use it. It’s a bit of a pain and some of the syntax can be arcane and frustrating. It’s also annoying the way it dithers the configuration for a given interface across a multitude of files in some cases. However, when I think the obvious corner cases through and consider the alternatives, I usually find myself realizing that they’ve probably made as good a choice as any for what needs to be done. Overall, it’s a pretty comprehensive interface and provides good logs for troubleshooting in most circumstances. Owen
On Mar 18, 2026, at 11:50, Brandon Martin via NANOG <nanog@lists.nanog.org> wrote:
On 3/17/26 23:08, Raymond Burkholder via NANOG wrote:
Linux VRF is. Plus EBGP and VxLAN and MPBGP and EVPN.
Linux also has a lot of options aside from VRF that can be useful. It has robust policy routing as well as full-on network namespaces, so you have both a lighter option and a heavyweight but incredibly flexible option and all without resorting to full virtualization and the performance and administration hassles that can come with it.
Since none are tied directly to signaling, you can also signal any of those three however you like assuming you can find a way to tie it into your routing daemon. That's usually impractical with policy routing, but VRFs and especially network namespaces (where you're just going to be running multiple routing daemon instances by design) give lots of options.
I'm not aware of any "routing" platforms that provide that level of flexibility and especially without getting absurdly complicated to configure since they, by design, try to centralize everything into a single configuration database (though JunOS's config format is very nice, here).
-- Brandon Martin _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/LX4HIWKV...
On 3/19/26 02:22, Owen DeLong wrote:
Linux has a bunch of different possible ways to administer all of this stuff.
Indeed, and that's pretty much the downside of Linux-as-a-router and a lot of open source options in general. Too much choice can be at minimum overwhelming and even a problem. If you can find a distribution that integrates everything with your favorite method of administration, you're on the right track.
My strong recommendation is take the time to learn systemd-networkd and use it. It’s a bit of a pain and some of the syntax can be arcane and frustrating. It’s also annoying the way it dithers the configuration for a given interface across a multitude of files in some cases. However, when I think the obvious corner cases through and consider the alternatives, I usually find myself realizing that they’ve probably made as good a choice as any for what needs to be done.
Yes, systemd-networkd is very capable (if arcane and unfriendly at times) for static configuration persistence. NetworkManager is more geared toward and indeed seems better for systems that are using lots of runtime dynamic configuration (DHCP, 802.1x, etc.) and also provides better desktop environment integration. I mostly use systemd-networkd on my generic-Linux-distro-as-a-router and adjacent (local DHCP/DNS, for example) systems.
On 3/19/26 7:41 AM, Brandon Martin via NANOG wrote:
On 3/19/26 02:22, Owen DeLong wrote:
Linux has a bunch of different possible ways to administer all of this stuff.
Indeed, and that's pretty much the downside of Linux-as-a-router and a lot of open source options in general. Too much choice can be at minimum overwhelming and even a problem. I've used OpenBSD now for over 20 years as a router at home and in datacenters. CARP and PF and the rest of it are absolutely first class!
-- Chris Paul | Rex Consulting, Inc. | https://www.rexconsulting.net
Yes, systemd-networkd is very capable (if arcane and unfriendly at times) for static configuration persistence. NetworkManager is more geared toward and indeed seems better for systems that are using lots of runtime dynamic configuration (DHCP, 802.1x, etc.) and also provides better desktop environment integration.
Agree to disagree here. The ability to control DHCP and 802.1x in systemd-networkd is far more mature and capable than NetworkManager. The one and only place where network manager shines is if you are traversing many different wireless networks with different but basic configurations and limited IPv6 concerns. (It’s great for a normal laptop user who can manage it entirely within the GUI). I would never use it on anything doing any sort of routing or bridging. Not even like a mobile hotspot.
I mostly use systemd-networkd on my generic-Linux-distro-as-a-router and adjacent (local DHCP/DNS, for example) systems.
I’ve decided that it’s the best Swiss Army knife for my needs and rather than try to stay up on the idiosyncrasies of multiple packages, I try to just use it wherever I can. YMMV. Owen
Disagree strongly here. CARP is an abomination that should never have been inflicted on the world. CARP is an act of arrogance wherein the BSD developers went to the IETF and demanded a port allocation for a service that was largely duplicative of existing functionality provided by VRRP. They refused to go through the internet-draft process or work with the relevant working group(s) and instead ended up just overloading the port assignments for VRRP such that CARP becomes a huge PITA for any network engineer who encounters it on a network running or deploying VRRP. The BSD team lost a lot of my previous respect when they did this. Yes, BSD routing is also a viable candidate, but CARP just shouldn’t be. Owen
On Mar 19, 2026, at 11:07, Owen DeLong <owen@delong.com> wrote:
Yes, systemd-networkd is very capable (if arcane and unfriendly at times) for static configuration persistence. NetworkManager is more geared toward and indeed seems better for systems that are using lots of runtime dynamic configuration (DHCP, 802.1x, etc.) and also provides better desktop environment integration.
Agree to disagree here. The ability to control DHCP and 802.1x in systemd-networkd is far more mature and capable than NetworkManager. The one and only place where network manager shines is if you are traversing many different wireless networks with different but basic configurations and limited IPv6 concerns. (It’s great for a normal laptop user who can manage it entirely within the GUI).
I would never use it on anything doing any sort of routing or bridging. Not even like a mobile hotspot.
I mostly use systemd-networkd on my generic-Linux-distro-as-a-router and adjacent (local DHCP/DNS, for example) systems.
I’ve decided that it’s the best Swiss Army knife for my needs and rather than try to stay up on the idiosyncrasies of multiple packages, I try to just use it wherever I can.
YMMV.
Owen
https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol --- Cisco informed the OpenBSD <https://en.wikipedia.org/wiki/OpenBSD> developers that it would enforce its patent on HSRP. Cisco's position may have been due to their lawsuit with Alcatel. As Cisco's licensing terms prevented an open-source VRRP implementation, the OpenBSD developers began developing CARP instead. OpenBSD focuses on security. They designed CARP to use cryptography <https://en.wikipedia.org/wiki/Cryptography>. This made CARP fundamentally different from VRRP and ensured that CARP did not infringe on Cisco's patent. CARP became available in October 2003.[3] <https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol#cite_note-3> Later, it was integrated into FreeBSD <https://en.wikipedia.org/wiki/FreeBSD> (first released in May 2005 with FreeBSD 5.4),[4] <https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol#cite_note-4> NetBSD <https://en.wikipedia.org/wiki/NetBSD> and Linux <https://en.wikipedia.org/wiki/Linux> (ucarp).[1] <https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol#cite_note-ucarp_manpage-1> While Cisco's US patent expired in 2014, the two incompatible protocols continue to coexist. daniel On Thu, Mar 19, 2026 at 8:12 PM Owen DeLong via NANOG <nanog@lists.nanog.org> wrote:
Disagree strongly here.
CARP is an abomination that should never have been inflicted on the world. CARP is an act of arrogance wherein the BSD developers went to the IETF and demanded a port allocation for a service that was largely duplicative of existing functionality provided by VRRP. They refused to go through the internet-draft process or work with the relevant working group(s) and instead ended up just overloading the port assignments for VRRP such that CARP becomes a huge PITA for any network engineer who encounters it on a network running or deploying VRRP.
The BSD team lost a lot of my previous respect when they did this. Yes, BSD routing is also a viable candidate, but CARP just shouldn’t be.
Owen
On Mar 19, 2026, at 11:07, Owen DeLong <owen@delong.com> wrote:
Yes, systemd-networkd is very capable (if arcane and unfriendly at times) for static configuration persistence. NetworkManager is more geared toward and indeed seems better for systems that are using lots of runtime dynamic configuration (DHCP, 802.1x, etc.) and also provides better desktop environment integration.
Agree to disagree here. The ability to control DHCP and 802.1x in systemd-networkd is far more mature and capable than NetworkManager. The one and only place where network manager shines is if you are traversing many different wireless networks with different but basic configurations and limited IPv6 concerns. (It’s great for a normal laptop user who can manage it entirely within the GUI).
I would never use it on anything doing any sort of routing or bridging. Not even like a mobile hotspot.
I mostly use systemd-networkd on my generic-Linux-distro-as-a-router and adjacent (local DHCP/DNS, for example) systems.
I’ve decided that it’s the best Swiss Army knife for my needs and rather than try to stay up on the idiosyncrasies of multiple packages, I try to just use it wherever I can.
YMMV.
Owen
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/FHSFXQ44...
participants (10)
-
Brandon Martin -
Bryan Holloway -
Chris Paul -
Daniel Schroder -
Dennis - LTI Support -
Dorn Hetzel -
Mark Tinka -
Michael Greenup -
Owen DeLong -
Raymond Burkholder