IPoIB looks more like an application than a network protocol to Infiniband. The IB fabric doesn't have a concept of broadcast, so ARP works much differently than it does in IPv4/ethernet world - basically an all-nodes multicast group handles the distribution of ARP messages. That said, the ib drivers that come with redhat/centos are pretty good, and you can always download the official OFED drivers from the OFA at https://www.openfabrics.org/linux-sources.html if the stuff in your linux distribution is missing something. I've set up IPoIB routers running 10G NICs on the ethernet side and QDR HCAs on the IB side, using quagga to plug in to the rest of my OSPF network, and it works fine. Basically you just need to set up quagga like you would if you were going to turn a linux box into an ethernet router and don't worry about the fact that it's actually IB on one side of the router - your network statements, etc., in OSPF in quagga won't change at all. You'll find that some things in IB have no equivalent to ethernet. For example, if you want to have gateway redundancy for traffic exiting the IB fabric, your first instinct will be to look for VRRP for IB, but you won't find it, because of the ARP differences I talked about above. To get around this you can set up linux-ha or some other type of heartbeat arrangement and bring up a virtual IP on the active gateway, which can be shifted over to the standby gateway when the ha scripts detect a problem. Some vendors also have proprietary solutions to this problem but they tend to be expensive. So, I'd say, read up on quagga and give that a try, and I think you'll find that as long as the IB drivers are up to snuff (the sminfo command returns valid results, etc.) it'll pretty much just work for you. I'm also happy to discuss more offline if you prefer. Tom Tom On Tue, Feb 19, 2013 at 5:55 PM, Jon Lewis <jlewis@lewis.org> wrote:
On Tue, 19 Feb 2013, Landon Stewart wrote:
Oh by vendor I mean VAR I guess. Mostly I'm also wondering how an IB
network handles IPoIB and how one uses IB with a gateway to layer 3 Ethernet switches or edge routers. If anyone has any resources that provide details on how this works and how ethernet VLANs are handled I'd appreciate it.
My limited IB experience has been that the IB switch acts much like a dumb ethernet switch, caring only about which IB hardware addresses are reachable via which port. Routing between IPoIB and IP over ethernet can be done by any host with interfaces on both networks and IP forwarding enabled. In our setups, we've used IPoIB, but with 1918 addresses and not routed beyond the IB network.
------------------------------**------------------------------**---------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/**pgp<http://www.lewis.org/~jlewis/pgp>for PGP public key_________
-- ----------------------------------------------------------------------------- Tom Ammon Network Engineer M: (801) 674-9273 tom@tomsbox.net -----------------------------------------------------------------------------