While I have found some information on a project called linux-mpls I am having a hard time finding any solid VRF framework for Linux. I have a monitoring system that needs check devices that sit in overlapping private ip space, and I was wondering if there is anyway I could use some kind or VRF type solution that would allow me to label the "site" the traffic is intended for. The upstream router supports VRF/MPLS, but I need to know how I can get the server to label the traffic. I would appreciate any input. --- Brian Raaen Network Architect braaen@zcorum.com
On Aug 23, 2011, at 9:45 AM, nanog@rhemasound.org wrote:
While I have found some information on a project called linux-mpls I am having a hard time finding any solid VRF framework for Linux. I have a monitoring system that needs check devices that sit in overlapping private ip space, and I was wondering if there is anyway I could use some kind or VRF type solution that would allow me to label the "site" the traffic is intended for. The upstream router supports VRF/MPLS, but I need to know how I can get the server to label the traffic. I would appreciate any input.
In linux, you can manage the different routing tables. You can do this with the iptables + iproute2 series of commands. The tables 254/255 are the main and local tables. You shouldn't have too much trouble finding information via google on how to manage your needs. - Jared
On Tue, 23 Aug 2011 09:50:30 -0400, Jared Mauch wrote:
On Aug 23, 2011, at 9:45 AM, nanog@rhemasound.org wrote:
While I have found some information on a project called linux-mpls I am having a hard time finding any solid VRF framework for Linux. I have a monitoring system that needs check devices that sit in overlapping private ip space, and I was wondering if there is anyway I could use some kind or VRF type solution that would allow me to label the "site" the traffic is intended for. The upstream router supports VRF/MPLS, but I need to know how I can get the server to label the traffic. I would appreciate any input.
In linux, you can manage the different routing tables.
You can do this with the iptables + iproute2 series of commands. The tables 254/255 are the main and local tables.
You shouldn't have too much trouble finding information via google on how to manage your needs.
- Jared I guess VRF more close to Linux containers.
--- System administrator Denys Fedoryshchenko Virtual ISP S.A.L.
Jared, Thank you for your reply. The one issue I have is how can I label traffic to match a given table (i.e. ping VRF or snmp VRF). I don't see any way this can be done with normal BSD sockets, finding a way to get my application to 'color' the traffic has been a little evasive. The developers I am working with are using Mule for their data collection. I would really prefer to add an MPLS tag to mark the traffic, but I will investigate what I can do using the Linux routing features and 802.1q tags. --- Brian Raaen Network Architect braaen@zcorum.com On Tue, Aug 23, 2011 at 09:50:30AM -0400, Jared Mauch wrote:
On Aug 23, 2011, at 9:45 AM, nanog@rhemasound.org wrote:
While I have found some information on a project called linux-mpls I am having a hard time finding any solid VRF framework for Linux. I have a monitoring system that needs check devices that sit in overlapping private ip space, and I was wondering if there is anyway I could use some kind or VRF type solution that would allow me to label the "site" the traffic is intended for. The upstream router supports VRF/MPLS, but I need to know how I can get the server to label the traffic. I would appreciate any input.
In linux, you can manage the different routing tables.
You can do this with the iptables + iproute2 series of commands. The tables 254/255 are the main and local tables.
You shouldn't have too much trouble finding information via google on how to manage your needs.
- Jared
Jared, Thank you for your reply. The one issue I have is how can I label traffic to match a given table (i.e. ping VRF or snmp VRF). I don't see any way this can be done with normal BSD sockets, finding a way to get my application to 'color' the traffic has been a little evasive. The developers I am working with are using Mule for their data collection. I would really prefer to add an MPLS tag to mark the traffic, but I will investigate what I can do using the Linux routing features and 802.1q tags.
I don't know about Mule, but Zabbix has the concept of premise-based proxy servers which work around this issue, and it works quite well. Perhaps this issue can be solved at the application layer with some similar proxying methodology, rather than making this a very complicated routing issue?
Hello, I implemented it via dot1q vlans+iproute2+iptables. Description can be found at http://forum.nag.ru/forum/index.php?showtopic=57082&st=0&p=501082&#entry501082 . Please use the Google translator to translate from Russian to English. 23.08.2011, 17:45, nanog@rhemasound.org:
While I have found some information on a project called linux-mpls I am having a hard time finding any solid VRF framework for Linux. I have a monitoring system that needs check devices that sit in overlapping private ip space, and I was wondering if there is anyway I could use some kind or VRF type solution that would allow me to label the "site" the traffic is intended for. The upstream router supports VRF/MPLS, but I need to know how I can get the server to label the traffic. I would appreciate any input.
--- Brian Raaen Network Architect braaen@zcorum.com
-- wbr, Sergey V. Lobanov E-mail: sergey@lobanov.in
On 23/08/11 13:45 +0000, nanog@rhemasound.org wrote:
While I have found some information on a project called linux-mpls I am having a hard time finding any solid VRF framework for Linux. I have a monitoring system that needs check devices that sit in overlapping private ip space, and I was wondering if there is anyway I could use some kind or VRF type solution that would allow me to label the "site" the traffic is intended for. The upstream router supports VRF/MPLS, but I need to know how I can get the server to label the traffic. I would appreciate any input.
Although I can't vouch for it, quagga seems to have the command set to function as an MPLS PE router (possibly in conjunction with linux-mpls) to pass vpnv4 routes and tags. That doesn't address how you're going to mux socket connections to the overloaded IP addresses in different VRFs, which would seem to require MPLS knowledge within your monitoring application to support (unless you're running multiple instances). You might consider a more straight forward approach, such as running a separate instance of your monitoring application within a VM, bridged to a separate VLAN towards your MPLS PE, or just running two hosts. -- Dan White
On 23 August 2011 14:45, <nanog@rhemasound.org> wrote:
While I have found some information on a project called linux-mpls I am having a hard time finding any solid VRF framework for Linux. I have a monitoring system that needs check devices that sit in overlapping private ip space, and I was wondering if there is anyway I could use some kind or VRF type solution that would allow me to label the "site" the traffic is intended for. The upstream router supports VRF/MPLS, but I need to know how I can get the server to label the traffic. I would appreciate any input.
I would probably go for the suggestion of (ab)using QoS tags for the routing table selection, but just to throw this alternate idea out there: 1.0.0.0/8 1:1 NATed to 10.0.0.0/8 marked to use routing table 1, which routes to network 1 2.0.0.0/8 1:1 NATed to 10.0.0.0/8 marked to use routing table 2, which routes to network 2 etc That way your application layer won't need any additional logic and can just deal with them as separate non-overlapping IP spaces, this won't work if you have too many overlapping networks (but then linux only supports 252 additional routing tables anyway afaik) or if you need external connectivity that can't be proxied. In a similar manner if your tools support IPv6 you could have a /96 that is NAT64'ed on to each different network, i'm not sure about this for a production setup although it would have the added benefit that you can expose these routes to your management network to provide easier access from your other machines if you wanted to. - Mike
On Tue, 2011-08-23 at 13:45 +0000, nanog@rhemasound.org wrote:
While I have found some information on a project called linux-mpls I am having a hard time finding any solid VRF framework for Linux.
The Linux kernel as shipped by Linus supports multiple routing tables and allows you to forward traffic from interfaces to differing tables -- that is, can implement VRF. The abstraction is better than on most routers, with policy routing allowing the selection of the routing table (to implement a VRF the policy is a simple "if received on interface X then use realm N"). Searching "realms" or running "man ip" will get you started. The Linus kernel does not have support for MPLS. You could patch the kernel, and then use Quagga as the router to populate the MPLS forwarding table. But personally, if you have a MPLS-speaking router upstream I'd simply bridge each MPLS tunnel into a VLAN to the Linux computer. Then you can use a stock vendor kernel, with its lack of maintenance hassles. -- Glen Turner <http://www.gdt.id.au/~gdt/>
The only issue with this is that the Linux box is not acting as a router, but as the egress devices. I'm trying to figure out how to properly get my application to 'color' the traffic. standard BSD sockets appear to have no concept of 'Labels'. Still seeing what I can do to match the traffic. I am probably going to see if I can work out a hack with the development team to use DSCP values to tag the traffic and then act accordingly on the ingress router. I appreciate all the ideas presented so far. --- Brian Raaen Network Architect braaen@zcorum.com On Wed, Aug 24, 2011 at 12:46:54PM +0930, Glen Turner wrote:
On Tue, 2011-08-23 at 13:45 +0000, nanog@rhemasound.org wrote:
While I have found some information on a project called linux-mpls I am having a hard time finding any solid VRF framework for Linux.
The Linux kernel as shipped by Linus supports multiple routing tables and allows you to forward traffic from interfaces to differing tables -- that is, can implement VRF. The abstraction is better than on most routers, with policy routing allowing the selection of the routing table (to implement a VRF the policy is a simple "if received on interface X then use realm N"). Searching "realms" or running "man ip" will get you started.
The Linus kernel does not have support for MPLS. You could patch the kernel, and then use Quagga as the router to populate the MPLS forwarding table. But personally, if you have a MPLS-speaking router upstream I'd simply bridge each MPLS tunnel into a VLAN to the Linux computer. Then you can use a stock vendor kernel, with its lack of maintenance hassles.
-- Glen Turner <http://www.gdt.id.au/~gdt/>
On Wed, Aug 24, 2011 at 10:06 AM, Brian Raaen <nanog@rhemasound.org> wrote:
The only issue with this is that the Linux box is not acting as a router, but as the egress devices. I'm trying to figure out how to properly get my application to 'color' the traffic. standard BSD sockets appear to have no concept of 'Labels'. Still seeing what I can do to match the traffic. I am probably going to see if I can work out a hack with the development team to use DSCP values to tag the traffic and then act accordingly on the ingress router. I appreciate all the ideas presented so far.
You could also have a look at linux namespaces if you want to manage routing tables per process. Especially the new setns syscall could be handy: <https://lwn.net/Articles/407495/> Greetings, Hannes
On 2011-08-24 06:06, Brian Raaen wrote:
The only issue with this is that the Linux box is not acting as a router, but as the egress devices. I'm trying to figure out how to properly get my application to 'color' the traffic. standard BSD sockets appear to have no concept of 'Labels'.
Just FYI: on OpenBSD you can set the VRF (aka "routing table" or "routing domain") per socket with code like this: int s, table; s = socket(...); table = 123; setsockopt(s, IPPROTO_IP, SO_RTABLE, &table, sizeof(table)); Simon -- DTN made easy, lean, and smart --> http://postellation.viagenie.ca NAT64/DNS64 open-source --> http://ecdysis.viagenie.ca STUN/TURN server --> http://numb.viagenie.ca
On Wed, Aug 24, 2011 at 09:18:29AM -0400, Simon Perreault wrote:
On 2011-08-24 06:06, Brian Raaen wrote:
The only issue with this is that the Linux box is not acting as a router, but as the egress devices. I'm trying to figure out how to properly get my application to 'color' the traffic. standard BSD sockets appear to have no concept of 'Labels'.
Just FYI: on OpenBSD you can set the VRF (aka "routing table" or "routing domain") per socket with code like this:
int s, table; s = socket(...); table = 123; setsockopt(s, IPPROTO_IP, SO_RTABLE, &table, sizeof(table));
Or exec your commands wrapped in route -T$TABLE exec $* Caveat: ipv6 vrf's did not work the last time I tried, and I think they still don't. OpenBSD should also do MPLS VPNs with the VRF's, but it's also pretty much experimental. It worked fine in a quick lab test at my last try, I should dig my lab notes and document it... Some things, like /etc/resolv.conf, still need some attention with VRFs.
On 2011-08-24 13:37, Jussi Peltola wrote:
Just FYI: on OpenBSD you can set the VRF (aka "routing table" or "routing domain") per socket with code like this:
int s, table; s = socket(...); table = 123; setsockopt(s, IPPROTO_IP, SO_RTABLE, &table, sizeof(table));
Or exec your commands wrapped in route -T$TABLE exec $*
Caveat: ipv6 vrf's did not work the last time I tried, and I think they still don't.
The fix for that was committed to HEAD recently. I think it's going to be part of 5.0 or 5.1. Effectively it means s/IPPROTO_IP/SOL_SOCKET/ in the example code above. Simon -- DTN made easy, lean, and smart --> http://postellation.viagenie.ca NAT64/DNS64 open-source --> http://ecdysis.viagenie.ca STUN/TURN server --> http://numb.viagenie.ca
I want to thank everyone for their input and I have gleened many useful ideas from this discussion. Hopefully some standard like BSD sockets will be written for routing realms/vrfs, then let the fun begin. It appears that the Java based framework our developers used can not be extended to allow direct packet/socket manupulation, so we will be looking are using different vm's to get around our issue. Again I really enjoyed this discussion with everyone and am excited about the progress that is being made in bringing this concept dirrectly to the host. --- Brian Raaen Network Architect braaen@zcorum.com On Wed, Aug 24, 2011 at 07:58:55PM +0200, Hannes Frederic Sowa wrote:
On Wed, Aug 24, 2011 at 7:37 PM, Jussi Peltola <pelzi@pelzi.net> wrote:
Or exec your commands wrapped in route -T$TABLE exec $*
FYI, on linux you can use 'ip netns exec'. The subcommand is rather new and you will only find it in the git repository.
Greetings,
Hannes
On Aug 24, 2011, at 6:06 AM, Brian Raaen wrote:
The only issue with this is that the Linux box is not acting as a router, but as the egress devices. I'm trying to figure out how to properly get my application to 'color' the traffic. standard BSD sockets appear to have no concept of 'Labels'. Still seeing what I can do to match the traffic. I am probably going to see if I can work out a hack with the development team to use DSCP values to tag the traffic and then act accordingly on the ingress router. I appreciate all the ideas presented so far.
You can classify this in the OUTPUT or POSTROUTING table with ipchains. Take a look at the man page for it. There's lots of information online about how to do this. I recall a sysadmin who I worked with 15 years ago that thought of routers as the black boxes that got their packets around, but a little bit of understanding of these lower levels of the kernel/networks will go a long way. Some help: INPUT (for packets destined to local sockets) FORWARD (for packets being routed through the box) OUTPUT (for locally-generated packets; for altering locally-generated packets before routing) PREROUTING (for altering packets as soon as they come in) POSTROUTING (for altering packets as they are about to go out) http://linux-ip.net/html/adv-multi-internet.html should also prove useful in your research. You likely are going to end up using the localhost fwmark/mark. Some tools show this number in hex, others decimal, so keep this in mind during your debug process. - Jared
Em 24/08/2011, às 11:28, Jared Mauch <jared@puck.nether.net> escreveu:
On Aug 24, 2011, at 6:06 AM, Brian Raaen wrote:
The only issue with this is that the Linux box is not acting as a router, but as the egress devices. I'm trying to figure out how to properly get my application to 'color' the traffic. standard BSD sockets appear to have no concept of 'Labels'. Still seeing what I can do to match the traffic. I am probably going to see if I can work out a hack with the development team to use DSCP values to tag the traffic and then act accordingly on the ingress router. I appreciate all the ideas presented so far.
You can classify this in the OUTPUT or POSTROUTING table with ipchains. Take a look at the man page for it. There's lots of information online about how to do this. I recall a sysadmin who I worked with 15 years ago that thought of routers as the black boxes that got their packets around, but a little bit of understanding of these lower levels of the kernel/networks will go a long way.
Some help:
INPUT (for packets destined to local sockets) FORWARD (for packets being routed through the box) OUTPUT (for locally-generated packets; for altering locally-generated packets before routing) PREROUTING (for altering packets as soon as they come in) POSTROUTING (for altering packets as they are about to go out)
http://linux-ip.net/html/adv-multi-internet.html should also prove useful in your research. You likely are going to end up using the localhost fwmark/mark. Some tools show this number in hex, others decimal, so keep this in mind during your debug process.
More VRF info: http://lartc.org/lartc.html#LARTC.RPDB.SIMPLE -- Eduardo Schoedler
participants (13)
-
Brian Raaen -
Dan White -
Denys Fedoryshchenko -
Eduardo Schoedler -
Glen Turner -
Hannes Frederic Sowa -
Jared Mauch -
Jussi Peltola -
Mike Jones -
nanog＠rhemasound.org
-
Nathan Eisenberg -
Sergey V. Lobanov -
Simon Perreault