Anyone know of a good InfiniBand vendor in the US?
Hello NANOG, We are thinking of utilizing some InfiniBand stuff for some specific application in our data centres. We are new to InfiniBand however so we want to get some equipment and see if it does what we need. Does anyone know of a good vendor in the US? East or West coast, doesn't matter. If anyone has any good advice or information about InfiniBand that would be nice to hear too as we are totally new to it at present. -- Landon Stewart <LStewart@Superb.Net> Sr. Administrator Systems Engineering Superb Internet Corp - 888-354-6128 x 4199 Web hosting and more "Ahead of the Rest": http://www.superbhosting.net
Hi Landon: We deliver Infiniband based servers and switches. We have been working with Infiniband for many years already. What are you looking for? Alex www.pssclabs.com On 2/19/2013 2:11 PM, Landon Stewart wrote:
Hello NANOG,
We are thinking of utilizing some InfiniBand stuff for some specific application in our data centres. We are new to InfiniBand however so we want to get some equipment and see if it does what we need. Does anyone know of a good vendor in the US? East or West coast, doesn't matter. If anyone has any good advice or information about InfiniBand that would be nice to hear too as we are totally new to it at present.
VAR or Manufacturer? Mellanox are essentially the defacto standard for IB switches and HCAs. Sent from my mobile device, so please excuse any horrible misspellings. On Feb 19, 2013, at 14:12, Landon Stewart <lstewart@superb.net> wrote:
Hello NANOG,
We are thinking of utilizing some InfiniBand stuff for some specific application in our data centres. We are new to InfiniBand however so we want to get some equipment and see if it does what we need. Does anyone know of a good vendor in the US? East or West coast, doesn't matter. If anyone has any good advice or information about InfiniBand that would be nice to hear too as we are totally new to it at present.
-- Landon Stewart <LStewart@Superb.Net> Sr. Administrator Systems Engineering Superb Internet Corp - 888-354-6128 x 4199 Web hosting and more "Ahead of the Rest": http://www.superbhosting.net
Oh by vendor I mean VAR I guess. Mostly I'm also wondering how an IB network handles IPoIB and how one uses IB with a gateway to layer 3 Ethernet switches or edge routers. If anyone has any resources that provide details on how this works and how ethernet VLANs are handled I'd appreciate it. On 19 February 2013 14:37, Matt Addison <matt.addison@lists.evilgeni.us>wrote:
VAR or Manufacturer? Mellanox are essentially the defacto standard for IB switches and HCAs.
Sent from my mobile device, so please excuse any horrible misspellings.
On Feb 19, 2013, at 14:12, Landon Stewart <lstewart@superb.net> wrote:
Hello NANOG,
We are thinking of utilizing some InfiniBand stuff for some specific application in our data centres. We are new to InfiniBand however so we want to get some equipment and see if it does what we need. Does anyone know of a good vendor in the US? East or West coast, doesn't matter. If anyone has any good advice or information about InfiniBand that would be nice to hear too as we are totally new to it at present.
-- Landon Stewart <LStewart@Superb.Net> Sr. Administrator Systems Engineering Superb Internet Corp - 888-354-6128 x 4199 Web hosting and more "Ahead of the Rest": http://www.superbhosting.net
-- Landon Stewart <LStewart@Superb.Net> Sr. Administrator Systems Engineering Superb Internet Corp - 888-354-6128 x 4199 Web hosting and more "Ahead of the Rest": http://www.superbhosting.net
On Tue, 19 Feb 2013, Landon Stewart wrote:
Oh by vendor I mean VAR I guess. Mostly I'm also wondering how an IB network handles IPoIB and how one uses IB with a gateway to layer 3 Ethernet switches or edge routers. If anyone has any resources that provide details on how this works and how ethernet VLANs are handled I'd appreciate it.
My limited IB experience has been that the IB switch acts much like a dumb ethernet switch, caring only about which IB hardware addresses are reachable via which port. Routing between IPoIB and IP over ethernet can be done by any host with interfaces on both networks and IP forwarding enabled. In our setups, we've used IPoIB, but with 1918 addresses and not routed beyond the IB network. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
IPoIB looks more like an application than a network protocol to Infiniband. The IB fabric doesn't have a concept of broadcast, so ARP works much differently than it does in IPv4/ethernet world - basically an all-nodes multicast group handles the distribution of ARP messages. That said, the ib drivers that come with redhat/centos are pretty good, and you can always download the official OFED drivers from the OFA at https://www.openfabrics.org/linux-sources.html if the stuff in your linux distribution is missing something. I've set up IPoIB routers running 10G NICs on the ethernet side and QDR HCAs on the IB side, using quagga to plug in to the rest of my OSPF network, and it works fine. Basically you just need to set up quagga like you would if you were going to turn a linux box into an ethernet router and don't worry about the fact that it's actually IB on one side of the router - your network statements, etc., in OSPF in quagga won't change at all. You'll find that some things in IB have no equivalent to ethernet. For example, if you want to have gateway redundancy for traffic exiting the IB fabric, your first instinct will be to look for VRRP for IB, but you won't find it, because of the ARP differences I talked about above. To get around this you can set up linux-ha or some other type of heartbeat arrangement and bring up a virtual IP on the active gateway, which can be shifted over to the standby gateway when the ha scripts detect a problem. Some vendors also have proprietary solutions to this problem but they tend to be expensive. So, I'd say, read up on quagga and give that a try, and I think you'll find that as long as the IB drivers are up to snuff (the sminfo command returns valid results, etc.) it'll pretty much just work for you. I'm also happy to discuss more offline if you prefer. Tom Tom On Tue, Feb 19, 2013 at 5:55 PM, Jon Lewis <jlewis@lewis.org> wrote:
On Tue, 19 Feb 2013, Landon Stewart wrote:
Oh by vendor I mean VAR I guess. Mostly I'm also wondering how an IB
network handles IPoIB and how one uses IB with a gateway to layer 3 Ethernet switches or edge routers. If anyone has any resources that provide details on how this works and how ethernet VLANs are handled I'd appreciate it.
My limited IB experience has been that the IB switch acts much like a dumb ethernet switch, caring only about which IB hardware addresses are reachable via which port. Routing between IPoIB and IP over ethernet can be done by any host with interfaces on both networks and IP forwarding enabled. In our setups, we've used IPoIB, but with 1918 addresses and not routed beyond the IB network.
------------------------------**------------------------------**---------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/**pgp<http://www.lewis.org/~jlewis/pgp>for PGP public key_________
-- ----------------------------------------------------------------------------- Tom Ammon Network Engineer M: (801) 674-9273 tom@tomsbox.net -----------------------------------------------------------------------------
I wanted to bring attention to the following draft proposal from Mellanox to export traffic information from InfiniBand switches: http://sflow.org/draft_sflow_infiniband.txt If you are an InfiniBand user, this is a great opportunity to think about the types of metrics that you woud want from your switches in order to better understand performance. The operational sensibility that the NANOG audience brings is particularly valuable. Comments on the proposal are welcome on the sFlow discussion group: http://groups.google.com/group/sflow On Wed, Feb 20, 2013 at 2:25 PM, Tom Ammon <thomasammon@gmail.com> wrote:
IPoIB looks more like an application than a network protocol to Infiniband. The IB fabric doesn't have a concept of broadcast, so ARP works much differently than it does in IPv4/ethernet world - basically an all-nodes multicast group handles the distribution of ARP messages. That said, the ib drivers that come with redhat/centos are pretty good, and you can always download the official OFED drivers from the OFA at https://www.openfabrics.org/linux-sources.html if the stuff in your linux distribution is missing something.
I've set up IPoIB routers running 10G NICs on the ethernet side and QDR HCAs on the IB side, using quagga to plug in to the rest of my OSPF network, and it works fine. Basically you just need to set up quagga like you would if you were going to turn a linux box into an ethernet router and don't worry about the fact that it's actually IB on one side of the router - your network statements, etc., in OSPF in quagga won't change at all.
You'll find that some things in IB have no equivalent to ethernet. For example, if you want to have gateway redundancy for traffic exiting the IB fabric, your first instinct will be to look for VRRP for IB, but you won't find it, because of the ARP differences I talked about above. To get around this you can set up linux-ha or some other type of heartbeat arrangement and bring up a virtual IP on the active gateway, which can be shifted over to the standby gateway when the ha scripts detect a problem. Some vendors also have proprietary solutions to this problem but they tend to be expensive.
So, I'd say, read up on quagga and give that a try, and I think you'll find that as long as the IB drivers are up to snuff (the sminfo command returns valid results, etc.) it'll pretty much just work for you. I'm also happy to discuss more offline if you prefer.
Tom
Tom
On Tue, Feb 19, 2013 at 5:55 PM, Jon Lewis <jlewis@lewis.org> wrote:
On Tue, 19 Feb 2013, Landon Stewart wrote:
Oh by vendor I mean VAR I guess. Mostly I'm also wondering how an IB
network handles IPoIB and how one uses IB with a gateway to layer 3 Ethernet switches or edge routers. If anyone has any resources that provide details on how this works and how ethernet VLANs are handled I'd appreciate it.
My limited IB experience has been that the IB switch acts much like a dumb ethernet switch, caring only about which IB hardware addresses are reachable via which port. Routing between IPoIB and IP over ethernet can be done by any host with interfaces on both networks and IP forwarding enabled. In our setups, we've used IPoIB, but with 1918 addresses and not routed beyond the IB network.
------------------------------**------------------------------**---------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/**pgp<http://www.lewis.org/~jlewis/pgp>for PGP public key_________
-- ----------------------------------------------------------------------------- Tom Ammon Network Engineer M: (801) 674-9273 tom@tomsbox.net -----------------------------------------------------------------------------
participants (6)
-
Alex Lesser -
Jon Lewis -
Landon Stewart -
Matt Addison -
Peter Phaal -
Tom Ammon