Hi, This is a pretty moronic question, but I've been searching RFC's on- and-off for a couple of weeks and can't find an answer. So I'm hoping someone here will know it offhand. I've been looking through RFC's trying to find a clear statement that having two interfaces in the same subnet does not work, but can't find it that statement anywhere. The OS in this case is Linux. I know it can be done with clever routing and prioritization and such, but this has to do with vanilla config, just setting up two interfaces in one network. I would be grateful for a pointer to such an RFC statement, assuming it exists. Thanks! Chris
On May 11, 2009, at 4:29 PM, Chris Meidinger wrote:
This is a pretty moronic question, but I've been searching RFC's on- and-off for a couple of weeks and can't find an answer. So I'm hoping someone here will know it offhand.
I've been looking through RFC's trying to find a clear statement that having two interfaces in the same subnet does not work, but can't find it that statement anywhere.
The OS in this case is Linux. I know it can be done with clever routing and prioritization and such, but this has to do with vanilla config, just setting up two interfaces in one network.
I would be grateful for a pointer to such an RFC statement, assuming it exists.
Why would an RFC prohibit this? Most _implementations_ do, but as far as network "rules" in general it is a valid configuration. -- TTFN, patrick
On 11.05.2009, at 22:34, Patrick W. Gilmore wrote:
On May 11, 2009, at 4:29 PM, Chris Meidinger wrote:
I would be grateful for a pointer to such an RFC statement, assuming it exists.
Why would an RFC prohibit this?
Most _implementations_ do, but as far as network "rules" in general it is a valid configuration.
That was essentially my conclusion as well: logically it can't work, but I wasn't certain where it might be forbidden. Thusly did I come to NANOG with the question, thinking smarter people than I might know. If it's completely down to implementation, or really to the interaction between TCP and underlying IP, then so be it. I was hoping that I might just not have thought of the right place to look. On 11.05.2009, at 22:39, Mikael Abrahamsson wrote:
On Mon, 11 May 2009, Chris Meidinger wrote:
I've been looking through RFC's trying to find a clear statement that having two interfaces in the same subnet does not work, but can't find it that statement anywhere.
I don't know if it still works, but it did in Linux little over 10 years back. Proxy-arp:ed all the IPs in the /27 in the /24 and everything was fine (legacy reasons plus radiolink which I didn't want to run a lot of broadcasts over). There are "legitimate" cases where you might want to do this.
Yes, I've gotten it to work as well as little as 10 days ago, but it's not something that $random_customer should be doing as a matter of practice. Thus, again, my hope that I just wasn't thinking of the right place to look to find an IETF recommendation against doing so. Thanks for the input! Chris
On May 11, 2009, at 4:45 PM, Chris Meidinger wrote:
On 11.05.2009, at 22:34, Patrick W. Gilmore wrote:
On May 11, 2009, at 4:29 PM, Chris Meidinger wrote:
I would be grateful for a pointer to such an RFC statement, assuming it exists.
Why would an RFC prohibit this?
Most _implementations_ do, but as far as network "rules" in general it is a valid configuration.
That was essentially my conclusion as well: logically it can't work, but I wasn't certain where it might be forbidden.
How did you read what I wrote and say what you said? I've read it several times and I can't get from point A to point B. Did you do a major typo or something? I said it is valid. After saying you came to the same conclusion, you then said "logically it can't work". Your statement not only contradicts what I said, but contradicts the fact is has and does work. Let me be clear: IT IS NOT FORBIDDEN. IT WORKS FINE. -- TTFN, patrick
On Mon, 11 May 2009, Chris Meidinger wrote:
I've been looking through RFC's trying to find a clear statement that having two interfaces in the same subnet does not work, but can't find it that statement anywhere.
I don't know if it still works, but it did in Linux little over 10 years back. Proxy-arp:ed all the IPs in the /27 in the /24 and everything was fine (legacy reasons plus radiolink which I didn't want to run a lot of broadcasts over). There are "legitimate" cases where you might want to do this. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Mon, May 11, 2009 at 3:39 PM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
... There are "legitimate" cases where you might want to do this.
NetApp filers consider this to be a legitimate configuration, even a "supported and recommended one". If I understand the documentation correctly, NetApp will (somehow) remember the physical interface a request arrived on, and make sure to its send replies out same. It sounds like a slight breakage of the expected behavior in order to gain load-sharing for their multiple NICs attached to the same subnet. IIRC, you can turn the feature off if it makes an issue. --D
On May 11, 2009, at 4:48 PM, Duane Waddle wrote:
On Mon, May 11, 2009 at 3:39 PM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
IIRC, you can turn the feature off WHEN it makes an issue.
Fixed that for you. S550 attached to a 6509, Dell blade in a blade chassis with Power connect switches cross-connected to the 6509 in trunking mode. Got nothing but errors, discards, and slow performance across the link (only to the NAS....everything else was fine). Changed the S550 into redundancy mode (one NIC in standby rather than active/active) and everything is just peachy. I don't have the time or inclination to figure out why.....I just expect these things to work when the manufacturer says they will. Of course, I already have a huge issue with NetApp considering the EOLed a 3 month old piece of gear two months after I bought it.
What does two interfaces in one subnet mean? Two NICs? Or virtual interfaces? Mikael Abrahamsson wrote:
On Mon, 11 May 2009, Chris Meidinger wrote:
I've been looking through RFC's trying to find a clear statement that having two interfaces in the same subnet does not work, but can't find it that statement anywhere.
I don't know if it still works, but it did in Linux little over 10 years back. Proxy-arp:ed all the IPs in the /27 in the /24 and everything was fine (legacy reasons plus radiolink which I didn't want to run a lot of broadcasts over). There are "legitimate" cases where you might want to do this.
On 12/05/2009, at 9:00 AM, Charles Wyble wrote:
What does two interfaces in one subnet mean?
Two NICs? Or virtual interfaces?
Also, what does one subnet mean? A. Using the same IP prefix on two different networks (ie. ethernet broadcast domains) with an interface in to each, or B. running two interfaces in to the same network (ie. ethernet broadcast domain). In the case of A, are you re-using numbers on each side? In the case of B, are you wanting both interfaces to have the same number(s)? -- Nathan Ward
Unless you configure Layer 2 for two interfaces, it's not going to work. It is invalid from networking principle. If you have to send the traffic for host in same subnet you configured, which interface it should send out ? Basically it may create broadcast storm loop by putting two ip addresses in same subnet in different interface. It may be allowed from host-level, but from router equipment, I don't think it was allowed at all. Alex Chris Meidinger wrote:
Hi,
This is a pretty moronic question, but I've been searching RFC's on-and-off for a couple of weeks and can't find an answer. So I'm hoping someone here will know it offhand.
I've been looking through RFC's trying to find a clear statement that having two interfaces in the same subnet does not work, but can't find it that statement anywhere.
The OS in this case is Linux. I know it can be done with clever routing and prioritization and such, but this has to do with vanilla config, just setting up two interfaces in one network.
I would be grateful for a pointer to such an RFC statement, assuming it exists.
Thanks!
Chris
On 11.05.2009, at 23:19, Alex H. Ryu wrote:
Unless you configure Layer 2 for two interfaces, it's not going to work. It is invalid from networking principle. If you have to send the traffic for host in same subnet you configured, which interface it should send out ? Basically it may create broadcast storm loop by putting two ip addresses in same subnet in different interface. It may be allowed from host-level, but from router equipment, I don't think it was allowed at all.
Alex, I _personally_ know that it's a problem. I was hoping for an RFC- reference, or similar standards document, to show to customers to convince them to stop trying to hack things to make it work. Chris
On 5/11/09 3:23 PM, Chris Meidinger wrote:
On 11.05.2009, at 23:19, Alex H. Ryu wrote:
Unless you configure Layer 2 for two interfaces, it's not going to work. It is invalid from networking principle. If you have to send the traffic for host in same subnet you configured, which interface it should send out ? Basically it may create broadcast storm loop by putting two ip addresses in same subnet in different interface. It may be allowed from host-level, but from router equipment, I don't think it was allowed at all.
Alex, I _personally_ know that it's a problem. I was hoping for an RFC-reference, or similar standards document, to show to customers to convince them to stop trying to hack things to make it work.
Chris
In Linux, I ran into the exact situation talked about in the link: http://lwn.net/Articles/45373/ Basically, recent versions of Linux will respond to arp requests for IPs on another interface on the receiving interface. Basically, you end up with traffic going in/out of unexpected interfaces. I discovered my iptables rules weren't quite working right and I couldn't get into one of my boxen because the allow was set to eth0, and the packets were coming in/out of eth1 even though the IP was bound to eth0. One of the more interesting gotchas that had me stumped for hours before I found out what was really going on. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
Date: Mon, 11 May 2009 16:19:56 -0500 From: "Alex H. Ryu" <r.hyunseog@ieee.org>
Unless you configure Layer 2 for two interfaces, it's not going to work. It is invalid from networking principle. If you have to send the traffic for host in same subnet you configured, which interface it should send out ? Basically it may create broadcast storm loop by putting two ip addresses in same subnet in different interface. It may be allowed from host-level, but from router equipment, I don't think it was allowed at all.
Alex
I am a bit baffled as to why people think: 1. It won't work 2. It is a bad thing to do if it would work Neither is true. If it is two separate interfaces with two MAC addresses, it will work fine IF one of the interfaces is configured with a netmask of 255.255.255.255 (/32). Of course, you will have to add routes for the second interface if you expect to source traffic from it, but it really in not rare. Many network devices are intended to be configured this way. NetApp was mentioned, but it is not unique. Doing this is expressly covered in the documentation for FreeBSD. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
Chris Meidinger wrote:
Hi,
This is a pretty moronic question, but I've been searching RFC's on-and-off for a couple of weeks and can't find an answer. So I'm hoping someone here will know it offhand.
I've been looking through RFC's trying to find a clear statement that having two interfaces in the same subnet does not work, but can't find it that statement anywhere.
The OS in this case is Linux. I know it can be done with clever routing and prioritization and such, but this has to do with vanilla config, just setting up two interfaces in one network.
I would be grateful for a pointer to such an RFC statement, assuming it exists.
Thanks!
Chris
On 11.05.2009, at 23:42, Kevin Oberman wrote:
Date: Mon, 11 May 2009 16:19:56 -0500 From: "Alex H. Ryu" <r.hyunseog@ieee.org>
Unless you configure Layer 2 for two interfaces, it's not going to work. It is invalid from networking principle. If you have to send the traffic for host in same subnet you configured, which interface it should send out ? Basically it may create broadcast storm loop by putting two ip addresses in same subnet in different interface. It may be allowed from host-level, but from router equipment, I don't think it was allowed at all.
Alex
I am a bit baffled as to why people think: 1. It won't work 2. It is a bad thing to do if it would work
Neither is true. If it is two separate interfaces with two MAC addresses, it will work fine IF one of the interfaces is configured with a netmask of 255.255.255.255 (/32). Of course, you will have to add routes for the second interface if you expect to source traffic from it, but it really in not rare.
This is, of course, how I've done it at times in the past. Routing management can, however, become quite a pain over time. The customer expectation is, naturally, that any traffic related to a connection that comes in to the first interface should go back out that interface, and anything related to a connection that came into the second interface should go back out there. (All this without any specific routing etc.) I think we both know that that's not going to happen automagically. Chris
On May 11, 2009, at 5:19 PM, Alex H. Ryu wrote:
Unless you configure Layer 2 for two interfaces, it's not going to work.
It can work. Of course it _may_ not, depending upon your implementation, but then some implementations can't get a single interface to work properly per subnet.
It is invalid from networking principle.
You are confused, there is nothing invalid about the configuration.
If you have to send the traffic for host in same subnet you configured, which interface it should send out ?
Pick an interface and send the packet. It's not rocket science. I can come up with half a dozen algorithms off the top of my head while typing the last sentence.
Basically it may create broadcast storm loop by putting two ip addresses in same subnet in different interface.
That is an interesting statement. Could you explain how this can happen without crafting an idiotic implementation spec (e.g. every packet goes out both interfaces)?
It may be allowed from host-level, but from router equipment, I don't think it was allowed at all.
Ever used HSRP / VRRP? Two interfaces in the same subnet. Works fine. In fact, most people think it works _better_ than one interface in the same subnet. -- TTFN, patrick
On 11.05.2009 23:47 Patrick W. Gilmore wrote
On May 11, 2009, at 5:19 PM, Alex H. Ryu wrote:
It may be allowed from host-level, but from router equipment, I don't think it was allowed at all.
Ever used HSRP / VRRP? Two interfaces in the same subnet. Works fine. In fact, most people think it works _better_ than one interface in the same subnet.
I guess you are mixing interfaces with IPs now. Don't you? Arnold -- Arnold Nipper / nIPper consulting, Sandhausen, Germany email: arnold@nipper.de phone: +49 6224 9259 299 mobile: +49 172 2650958 fax: +49 6224 9259 333
On May 11, 2009, at 6:13 PM, Arnold Nipper wrote:
On 11.05.2009 23:47 Patrick W. Gilmore wrote
On May 11, 2009, at 5:19 PM, Alex H. Ryu wrote:
It may be allowed from host-level, but from router equipment, I don't think it was allowed at all.
Ever used HSRP / VRRP? Two interfaces in the same subnet. Works fine. In fact, most people think it works _better_ than one interface in the same subnet.
I guess you are mixing interfaces with IPs now. Don't you?
Each interface has its own IP address. The two Interfaces _also_ share a virtual IP address. IOW: No. Are you? -- TTFN, patrick
On 12.05.2009 00:25 Patrick W. Gilmore wrote
On May 11, 2009, at 6:13 PM, Arnold Nipper wrote:
On 11.05.2009 23:47 Patrick W. Gilmore wrote
On May 11, 2009, at 5:19 PM, Alex H. Ryu wrote:
It may be allowed from host-level, but from router equipment, I don't think it was allowed at all.
Ever used HSRP / VRRP? Two interfaces in the same subnet. Works fine. In fact, most people think it works _better_ than one interface in the same subnet.
I guess you are mixing interfaces with IPs now. Don't you?
Each interface has its own IP address. The two Interfaces _also_ share a virtual IP address.
IOW: No. Are you?
But still each device only has _one_ interface in the same subnet. Though with two IP addresses sometimes. Arnold -- Arnold Nipper / nIPper consulting, Sandhausen, Germany email: arnold@nipper.de phone: +49 6224 9259 299 mobile: +49 172 2650958 fax: +49 6224 9259 333
On May 11, 2009, at 6:35 PM, Arnold Nipper wrote:
On 12.05.2009 00:25 Patrick W. Gilmore wrote
On May 11, 2009, at 6:13 PM, Arnold Nipper wrote:
On 11.05.2009 23:47 Patrick W. Gilmore wrote
On May 11, 2009, at 5:19 PM, Alex H. Ryu wrote:
It may be allowed from host-level, but from router equipment, I don't think it was allowed at all.
Ever used HSRP / VRRP? Two interfaces in the same subnet. Works fine. In fact, most people think it works _better_ than one interface in the same subnet.
I guess you are mixing interfaces with IPs now. Don't you?
Each interface has its own IP address. The two Interfaces _also_ share a virtual IP address.
IOW: No. Are you?
But still each device only has _one_ interface in the same subnet. Though with two IP addresses sometimes.
Of course, was thinking about using it on the same router. But I guess that doesn't work so well, does it? :) -- TTFN, patrick
Chris, I work with iHDTV <http://ihdtv.org>, a project that sends uncompressed high definition television (1.5 Gbps) as UDP over two 1 Gbps interfaces. If both interfaces are on the same subnet, the OS sees the same router (gateway) address on both interfaces, and the results are sub-optimal ... around 50% packet loss. Dave On Mon, May 11, 2009 at 3:29 PM, Chris Meidinger <cmeidinger@sendmail.com>wrote:
Hi,
This is a pretty moronic question, but I've been searching RFC's on-and-off for a couple of weeks and can't find an answer. So I'm hoping someone here will know it offhand.
I've been looking through RFC's trying to find a clear statement that having two interfaces in the same subnet does not work, but can't find it that statement anywhere.
The OS in this case is Linux. I know it can be done with clever routing and prioritization and such, but this has to do with vanilla config, just setting up two interfaces in one network.
I would be grateful for a pointer to such an RFC statement, assuming it exists.
Thanks!
Chris
On Mon, May 11, 2009 at 2:22 PM, David Devereaux-Weber <ddevereauxweber@gmail.com> wrote:
Chris,
I work with iHDTV <http://ihdtv.org>, a project that sends uncompressed high definition television (1.5 Gbps) as UDP over two 1 Gbps interfaces. If both interfaces are on the same subnet, the OS sees the same router (gateway) address on both interfaces, and the results are sub-optimal ... around 50% packet loss.
packet loss is probably due to the network switch having to re-learn the location of the MAC address constantly as it sees packets on two or more ports with the same MAC address (think STP loops). If your network stack and network device (switch) supports LACP, then you can have multiple connections between a host and a network device. That is a very easy way to increase capacity and add redundancy. That is how all of our VMWare ESX 3.5i servers are connected. Hector
On Mon, May 11, 2009 at 5:28 PM, Hector Herrera <hectorherrera@gmail.com> wrote:
On Mon, May 11, 2009 at 2:22 PM, David Devereaux-Weber <ddevereauxweber@gmail.com> wrote:
... both> interfaces are on the same subnet, the OS sees the same router (gateway) address on both interfaces, and the results are sub-optimal ... around 50% packet loss.
packet loss is probably due to the network switch having to re-learn the location of the MAC address constantly as it sees packets on two or more ports with the same MAC address (think STP loops).
My understand of the scenario is: Two physical interfaces, each with a unique IP address, in the same Ethernet broadcast domain, on the same IP (sub)network. If that's the case, the MAC address won't change. The cards stay put. So a layer two switch will be none the wiser. The reason this doesn't work (for most implementations) is that most IP routers look only at the destination IP address, and keep no state. (Here, I'm using "router" to include the routing engine built-in to any full IP implementation, not just dedicated equipment from Cisco, et. al.) So we have a host with IP addresses A and B on the same subnet. A packet comes in from some other host X. The application software does whatever it does, and sends a response. The router looks at the destination IP address X, and sees that it has two routes, A and B. Depending on implementation, the router may send everything out the first interface it finds in the routing table (e.g., use A and ignore B), or round-robin between the two, or who-knows-what. Either way, if the packet *from* X was addressed *to* B but the response comes back from *A*, then host X is going to drop the packet as invalid/irrelevant/etc. With Linux, at least, you *can* use the routing policy database to configure the kernel router to pay attention to more than just the destination IP address. For example, you can have it look at the source IP address (A or B), and route out the appropriate interface. However, IIRC, this only works if the application software binds to individual network interfaces. If the app software just listens for anything (0.0.0.0), then the kernel gets to pick the source IP address for any response. I can post examples with gory details from our firewall, if anyone needs them. -- Ben
On May 11, 2009, at 5:40 PM, Ben Scott wrote:
On Mon, May 11, 2009 at 5:28 PM, Hector Herrera <hectorherrera@gmail.com
wrote: On Mon, May 11, 2009 at 2:22 PM, David Devereaux-Weber <ddevereauxweber@gmail.com> wrote:
... both> interfaces are on the same subnet, the OS sees the same router (gateway) address on both interfaces, and the results are sub-optimal ... around 50% packet loss.
packet loss is probably due to the network switch having to re-learn the location of the MAC address constantly as it sees packets on two or more ports with the same MAC address (think STP loops).
My understand of the scenario is: Two physical interfaces, each with a unique IP address, in the same Ethernet broadcast domain, on the same IP (sub)network.
If that's the case, the MAC address won't change. The cards stay put. So a layer two switch will be none the wiser.
The reason this doesn't work (for most implementations) is that most IP routers look only at the destination IP address, and keep no state. (Here, I'm using "router" to include the routing engine built-in to any full IP implementation, not just dedicated equipment from Cisco, et. al.)
So we have a host with IP addresses A and B on the same subnet. A packet comes in from some other host X. The application software does whatever it does, and sends a response. The router looks at the destination IP address X, and sees that it has two routes, A and B.
Depending on implementation, the router may send everything out the first interface it finds in the routing table (e.g., use A and ignore B), or round-robin between the two, or who-knows-what. Either way, if the packet *from* X was addressed *to* B but the response comes back from *A*, then host X is going to drop the packet as invalid/irrelevant/etc.
You are assuming facts not in evidence. It doesn't matter which physical interface transmits the packet. For instance, if I ping a router's loopback interface, there is nothing stopping the router from making the loopback the source IP address of the return packet even though the (virtual) loopback interface _obviously_ did not physically transmit the packet. Another example: Imagine a web server with two uplinks in _different_ subnets running Quagga. Now assume the web server gets an HTTP request and the route back to the requesting host changes before all the packets are returned. Does the download break? Sure, if you use an implementation too broken for words. If not, things work just fine. Could everyone please stop coming up with "if people are stupid and break things, things don't work" examples. We all agree on that. Back in reality land, things that broken tend not to be used. (And please no jokes about cisco or microsoft or whatever.) -- TTFN, patrick
With Linux, at least, you *can* use the routing policy database to configure the kernel router to pay attention to more than just the destination IP address. For example, you can have it look at the source IP address (A or B), and route out the appropriate interface. However, IIRC, this only works if the application software binds to individual network interfaces. If the app software just listens for anything (0.0.0.0), then the kernel gets to pick the source IP address for any response.
I can post examples with gory details from our firewall, if anyone needs them.
-- Ben
On Mon, May 11, 2009 at 6:01 PM, Patrick W. Gilmore <patrick@ianai.net> wrote:
You are assuming facts not in evidence.
I *have* actually done this before, so I'd like to think, for my own purposes at least, my experiences are factual. :)
It doesn't matter which physical interface transmits the packet.
Well, in the general sense, I suppose not. The computer can put whatever it wants in an Ethernet frame, and as long as it's valid for the receiving system, it will work. But in the Linux IP stack, at least, and by default, the physical interface used to send a datagram is determined by the route selected, and that also determines the source IP address put on the datagram. At the same time, the only thing which influences route selection is the destination IP address. In particular, there's no concept of "session" or "connection" in that. So client X attempts to open a TCP connection to IP address B on my example server. When the server sends its SYN-ACK response, it doesn't pay attention to the fact that the connection "came in on" B. It just looks at destination X. If it decides A is the best route, then the SYN-ACK datagram will have source IP address A. But X is looking for a datagram from A. The datagram from B will get to X, but X will promptly drop it, as it's not expecting anything from B. Again, this is all by default. If you configure policy routing properly, many things can be made to work.
Another example: Imagine a web server with two uplinks in _different_ subnets running Quagga.
That's a different scenario entirely. Diverse routes work fine because all the intermediate routers work the same way I describe above: They don't care where the packet came from, they don't know about "connections", they just forward packets to the destination. If the actual interface went down, you can bet that the HTTP request in progress will be killed, because the TCP session is dependent on an IP address that just evaporated. :) -- Ben
On Mon, May 11, 2009 at 7:04 PM, Ben Scott <mailvortex@gmail.com> wrote:
On Mon, May 11, 2009 at 6:01 PM, Patrick W. Gilmore <patrick@ianai.net> wrote: [snip]
Many OSes should handle it correctly, in principle, there's nothing wrong with hosts homed twice to the same network and addressed inside the same subnet, but for Linux hosts, no.... Unless you have a _very_ special need: just use IP aliasing and bonding (with supported NICs); I say you'll probably be a lot happier. But go ahead... try testing multiple interfaces on the same subnet for a while on a Linux host, see how well that works out, that's how you can truly see..... Possible dangers to keep in mind (bad defaults of Linux), with multiple interfaces on same subnet (without bonding): * ARP Flux: Linux treats IP addresses as owned by the "host" not the "interface", the first interface to receive an ARP request replies. So if Linux receives an ARP request for interface B's IP address on interface A, it will reply on interface A, thus associating interface B's IP with interface A's hardware address. Oh, but next time an ARP request comes in, interface B might get it instead. (*even if the IP is outside interface A's subnet range, Linux just always responds to ARP for any IP it thinks it owns in global scope, no matter what interface received the ARP.... this can enable connectivity you never intended, i.e. If you have a Linux-based firewall, the inside interface's IP can be detected from outside by ARP requesting for random private IPs. ....) * Some popular distros set net.ipv4.rp_filter (Reverse Path Forwarding) on by default. This means Linux can reject the very traffic that said promiscuous ARP activity brought in, unless outgoing traffic for that destination would be routed using the interface it was received on. Get ready to tune arp_ignore, arp_filter, turn off rp_filter, and possibly have to apply some kernel patches (depending on OS version).. -- -J
Remember, Linux has no concept of downing an interface when the link goes away, so even if you have two interfaces with the same subnet config, the kernel will continue to send traffic to a disconnected interface. Utilizing the bonding kernel module is the only really effective way to handle layer-2 redundancy for a Linux system. I use it for a large number of production Linux systems and it works pretty much flawlessly. James Hess wrote:
On Mon, May 11, 2009 at 7:04 PM, Ben Scott <mailvortex@gmail.com> wrote:
On Mon, May 11, 2009 at 6:01 PM, Patrick W. Gilmore <patrick@ianai.net> wrote: [snip]
Many OSes should handle it correctly, in principle, there's nothing wrong with hosts homed twice to the same network and addressed inside the same subnet, but for Linux hosts, no....
Unless you have a _very_ special need: just use IP aliasing and bonding (with supported NICs); I say you'll probably be a lot happier.
But go ahead... try testing multiple interfaces on the same subnet for a while on a Linux host, see how well that works out, that's how you can truly see.....
Possible dangers to keep in mind (bad defaults of Linux), with multiple interfaces on same subnet (without bonding):
* ARP Flux: Linux treats IP addresses as owned by the "host" not the "interface", the first interface to receive an ARP request replies. So if Linux receives an ARP request for interface B's IP address on interface A, it will reply on interface A, thus associating interface B's IP with interface A's hardware address.
Oh, but next time an ARP request comes in, interface B might get it instead.
(*even if the IP is outside interface A's subnet range, Linux just always responds to ARP for any IP it thinks it owns in global scope, no matter what interface received the ARP.... this can enable connectivity you never intended, i.e. If you have a Linux-based firewall, the inside interface's IP can be detected from outside by ARP requesting for random private IPs. ....)
* Some popular distros set net.ipv4.rp_filter (Reverse Path Forwarding) on by default. This means Linux can reject the very traffic that said promiscuous ARP activity brought in, unless outgoing traffic for that destination would be routed using the interface it was received on. Get ready to tune arp_ignore, arp_filter, turn off rp_filter, and possibly have to apply some kernel patches (depending on OS version)..
-- -J
Once upon a time, David Coulson <david@davidcoulson.net> said:
Remember, Linux has no concept of downing an interface when the link goes away
Not true in several different ways. You can run netplugd or Network Manager to control it. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
On May 11, 2009, at 8:04 PM, Ben Scott wrote:
On Mon, May 11, 2009 at 6:01 PM, Patrick W. Gilmore <patrick@ianai.net> wrote:
It doesn't matter which physical interface transmits the packet.
Well, in the general sense, I suppose not. The computer can put whatever it wants in an Ethernet frame, and as long as it's valid for the receiving system, it will work.
The OP asked for an RFC showing why this is forbidden. It is not. It works fine. I stated many times that several implementations deny your ability to do this, but the "rules" permit it just fine.
Another example: Imagine a web server with two uplinks in _different_ subnets running Quagga.
That's a different scenario entirely. Diverse routes work fine because all the intermediate routers work the same way I describe above: They don't care where the packet came from, they don't know about "connections", they just forward packets to the destination.
Do you even read your own posts? Specifically: On May 11, 2009, at 5:40 PM, Ben Scott wrote:
Either way, if the packet *from* X was addressed *to* B but the response comes back from *A*, then host X is going to drop the packet as invalid/irrelevant/etc.
The receiving host X does not care (or even know) if A and B are in the same prefix. Intermediate systems have nothing to do with it as they do not touch the source IP address in the packet. (We are obviously ignoring NAT/PAT, etc.) So if it works with "diverse routes", it works without diverse routes. In other words, you contradicted yourself. Don't worry, you are in good company in this thread. The OP alone did that 4 times by my count, and I stopped reading his posts because he did it so often. To summarize: Two physical interfaces on one machine in the same prefix is allowed. There is no RFC against it - just the opposite. So quit arguing over "but my $THING doesn't support it properly" or "but it will break $SOMETHING" or whatever your favorite hang-up is. -- TTFN, patrick
On Mon, May 11, 2009 at 11:01 PM, Patrick W. Gilmore <patrick@ianai.net> wrote:
It doesn't matter which physical interface transmits the packet.
Well, in the general sense, I suppose not. The computer can put whatever it wants in an Ethernet frame, and as long as it's valid for the receiving system, it will work.
The OP asked for an RFC showing why this is forbidden.
Um, yah, I know. You replied to me telling me I was "assuming facts not in evidence". So I was responding to that. Neither my message which you replied to, nor your reply, nor my reply to *your reply*, had anything to do with the OP's request for an RFC citation. Notice the distinct lack of the string "RFC" in my post, your reply, or my reply to your reply. To borrow your phrase, "Do you read your own posts?"
Do you even read your own posts? Specifically:
On May 11, 2009, at 5:40 PM, Ben Scott wrote:
Either way, if the packet *from* X was addressed *to* B but the response comes back from *A*, then host X is going to drop the packet as invalid/irrelevant/etc.
The receiving host X does not care (or even know) if A and B are in the same prefix.
Look again. A and B are *IP addresses*, not hosts. If I'm standing on my home PC and try to connect to Google at 64.233.161.104, my home PC is going to open a TCP connection to that IP address. If Google then tries to send me a response back from 64.233.161.105, then my PC is going to ignore that packet, because it's expecting a response from 64.233.161.104. They're both coming from Google, and maybe they're evening coming from the same node at Google, but since *my* computer doesn't get the response it expects, it doesn't work. Get it? Just to be clear, I'm not talking about what the RFC's say in the above three paragraphs. I'm talking about the specifics of a failure mode which occur under certain conditions in the implementation the OP is using.
So if it works with "diverse routes", it works without diverse routes.
Not when the problem is malfunction at one of the end points!
Two physical interfaces on one machine in the same prefix is allowed.
Indeed, and unlike yourself, I was offering practical, operational advice on how one might configure the given implementation to support just that. -- Ben
On May 11, 2009, at 11:43 PM, Ben Scott wrote:
On Mon, May 11, 2009 at 11:01 PM, Patrick W. Gilmore <patrick@ianai.net
wrote:
Do you even read your own posts? Specifically:
On May 11, 2009, at 5:40 PM, Ben Scott wrote:
Either way, if the packet *from* X was addressed *to* B but the response comes back from *A*, then host X is going to drop the packet as invalid/irrelevant/etc.
The receiving host X does not care (or even know) if A and B are in the same prefix.
Look again. A and B are *IP addresses*, not hosts.
Sometimes people agree with me, sometimes they don't, and sometimes I agree with them. But I've yet to have someone claim to be arguing against me while proving my point. Anyone else want to unconfused Ben? I obviously cannot. -- TTFN, patrick
On Tue, May 12, 2009 at 12:12 AM, Patrick W. Gilmore <patrick@ianai.net>wrote:
On May 11, 2009, at 11:43 PM, Ben Scott wrote:
On Mon, May 11, 2009 at 11:01 PM, Patrick W. Gilmore <patrick@ianai.net> wrote:
Do you even read your own posts? Specifically:
On May 11, 2009, at 5:40 PM, Ben Scott wrote:
Either way, if
the packet *from* X was addressed *to* B but the response comes back from *A*, then host X is going to drop the packet as invalid/irrelevant/etc.
The receiving host X does not care (or even know) if A and B are in the same prefix.
Look again. A and B are *IP addresses*, not hosts.
Sometimes people agree with me, sometimes they don't, and sometimes I agree with them. But I've yet to have someone claim to be arguing against me while proving my point.
Anyone else want to unconfused Ben? I obviously cannot.
Really nothing clever about this at all in application or practicality. host> cat /etc/services | grep nntp nntp 119/tcp readnews untp # USENET News Transfer Protocol host1> cat /etc/hosts | grep news 127.0.0.1 localhost 192.168.0.100 clue-store 192.168.0.100 news-out 192.168.0.101 news-in If this didn't work, you'd suppose that virtual machines and IP aliasing wouldn't work either. Unless routes facing the world on the device are "tweaked", this should work fine and be reliable (if implemented cluefully). Am I not getting it? Best, -M< -- Martin Hannigan martin@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants
Anyone else want to unconfused Ben? I obviously cannot.
The numerous misconceptions propagated in this thread prompted me to go through the relevant sections of RFC 1122 and write a short article on multihomed IP host issues. http://wiki.nil.com/Multihomed_IP_hosts Your contributions, either as an e-mail reply or direct wiki edit are most welcome. After the kinks have been ironed out, I'll add figures illustrating the points in the text. ** I would prefer wiki edits although I have to admit the registration is a bit of a pain and the anonymous edits are not allowed. Thanks in advance to anyone who decides to contribute :) Ivan http://www.ioshints.info/about http://blog.ioshints.info/
Try this: http://www.linuxfoundation.org/en/Net:Bridge --Curtis Patrick W. Gilmore wrote:
On May 11, 2009, at 5:40 PM, Ben Scott wrote:
On Mon, May 11, 2009 at 5:28 PM, Hector Herrera <hectorherrera@gmail.com> wrote:
On Mon, May 11, 2009 at 2:22 PM, David Devereaux-Weber <ddevereauxweber@gmail.com> wrote:
... both> interfaces are on the same subnet, the OS sees the same router (gateway) address on both interfaces, and the results are sub-optimal ... around 50% packet loss.
packet loss is probably due to the network switch having to re-learn the location of the MAC address constantly as it sees packets on two or more ports with the same MAC address (think STP loops).
My understand of the scenario is: Two physical interfaces, each with a unique IP address, in the same Ethernet broadcast domain, on the same IP (sub)network.
If that's the case, the MAC address won't change. The cards stay put. So a layer two switch will be none the wiser.
The reason this doesn't work (for most implementations) is that most IP routers look only at the destination IP address, and keep no state. (Here, I'm using "router" to include the routing engine built-in to any full IP implementation, not just dedicated equipment from Cisco, et. al.)
So we have a host with IP addresses A and B on the same subnet. A packet comes in from some other host X. The application software does whatever it does, and sends a response. The router looks at the destination IP address X, and sees that it has two routes, A and B.
Depending on implementation, the router may send everything out the first interface it finds in the routing table (e.g., use A and ignore B), or round-robin between the two, or who-knows-what. Either way, if the packet *from* X was addressed *to* B but the response comes back from *A*, then host X is going to drop the packet as invalid/irrelevant/etc.
You are assuming facts not in evidence. It doesn't matter which physical interface transmits the packet. For instance, if I ping a router's loopback interface, there is nothing stopping the router from making the loopback the source IP address of the return packet even though the (virtual) loopback interface _obviously_ did not physically transmit the packet.
Another example: Imagine a web server with two uplinks in _different_ subnets running Quagga. Now assume the web server gets an HTTP request and the route back to the requesting host changes before all the packets are returned. Does the download break? Sure, if you use an implementation too broken for words. If not, things work just fine.
Could everyone please stop coming up with "if people are stupid and break things, things don't work" examples. We all agree on that.
Back in reality land, things that broken tend not to be used. (And please no jokes about cisco or microsoft or whatever.)
On 12.05.2009, at 19:37, Curtis Maurand wrote:
Try this:
Wow. It's really hard to convince people that you're not trying to solve a problem today but to avoid one tomorrow. Stil, I want to thank everyone that responded - both publicly and privately - for the valuable input. (Even the posts that tore me up were useful.) I was surprised and pleased about both the quality of input that I received, and the speed with which that input came. Thanks NANOG! Chris
In my case, each Ethernet interface has its own unique MAC address. Dave On Mon, May 11, 2009 at 4:28 PM, Hector Herrera <hectorherrera@gmail.com>wrote:
On Mon, May 11, 2009 at 2:22 PM, David Devereaux-Weber <ddevereauxweber@gmail.com> wrote:
Chris,
I work with iHDTV <http://ihdtv.org>, a project that sends uncompressed high definition television (1.5 Gbps) as UDP over two 1 Gbps interfaces. If both interfaces are on the same subnet, the OS sees the same router (gateway) address on both interfaces, and the results are sub-optimal ... around 50% packet loss.
packet loss is probably due to the network switch having to re-learn the location of the MAC address constantly as it sees packets on two or more ports with the same MAC address (think STP loops).
If your network stack and network device (switch) supports LACP, then you can have multiple connections between a host and a network device. That is a very easy way to increase capacity and add redundancy.
That is how all of our VMWare ESX 3.5i servers are connected.
Hector
David Devereaux-Weber wrote:
I work with iHDTV <http://ihdtv.org>, a project that sends uncompressed high definition television (1.5 Gbps) as UDP over two 1 Gbps interfaces. If both interfaces are on the same subnet, the OS sees the same router (gateway) address on both interfaces, and the results are sub-optimal ... around 50% packet loss.
Check closes and you may discover that the loss isn't on the path host->router, but on the reverse. Some hosts/interfaces are optimised for L2 link-aggregation, in which case they by default present the same MAC (host MAC-address) on multiple interfaces. Those will fail when configured with separate IP-addresses on the same VLAN/subnet. L2 switches that support link-aggregation will try to spread the traffic across the available ports on one VLAN on which it sees the same MAC. 2 ports gives 50% loss, 3 ports 67% and so on. Most host interfaces will not pick up IP-packets which destination doesn't match it's assigned address. //per
Chris Meidinger wrote:
Hi,
This is a pretty moronic question, but I've been searching RFC's on-and-off for a couple of weeks and can't find an answer. So I'm hoping someone here will know it offhand. I've been looking through RFC's trying to find a clear statement that having two interfaces in the same subnet does not work, but can't find it that statement anywhere. The OS in this case is Linux. I know it can be done with clever routing and prioritization and such, but this has to do with vanilla config, just setting up two interfaces in one network. I would be grateful for a pointer to such an RFC statement, assuming it exists.
If your goal is to achieve redundancy or to increase bandwidth, you can bond the interfaces together - assuming that you have a switch / switch stack that supports 802.3ad. Then you could assign multiple IPs to the bonded interface without any layer 3 messyness. - Dan
On 11.05.2009, at 23:31, Dan White wrote:
Chris Meidinger wrote:
Hi,
This is a pretty moronic question, but I've been searching RFC's on- and-off for a couple of weeks and can't find an answer. So I'm hoping someone here will know it offhand. I've been looking through RFC's trying to find a clear statement that having two interfaces in the same subnet does not work, but can't find it that statement anywhere. The OS in this case is Linux. I know it can be done with clever routing and prioritization and such, but this has to do with vanilla config, just setting up two interfaces in one network. I would be grateful for a pointer to such an RFC statement, assuming it exists.
If your goal is to achieve redundancy or to increase bandwidth, you can bond the interfaces together - assuming that you have a switch / switch stack that supports 802.3ad.
Then you could assign multiple IPs to the bonded interface without any layer 3 messyness.
I should have been clearer. The case in point is having two physical interfaces, each with a unique IP, in the same subnet. For example, eth0 is 10.0.0.1/24 and eth1 is 10.0.0.2/24, nothing like bonding going on. The customers usually have the idea of running one interface for administration and another for production (which is a _good_ idea) but they want to do it in the same subnet (not such a good idea...) Chris
On Mon, May 11, 2009 at 5:38 PM, Chris Meidinger <cmeidinger@sendmail.com> wrote:
For example, eth0 is 10.0.0.1/24 and eth1 is 10.0.0.2/24, nothing like bonding going on. The customers usually have the idea of running one interface for administration and another for production (which is a _good_ idea) but they want to do it in the same subnet (not such a good idea...)
I just posted on this, but I didn't really address your original question, so: I'm not aware of anything in the RFCs or other standards which prohibits this. But then, I haven't gone looking, because... It *can* be made to work in practice, for certain scenarios. For example, if you're talking a web server, and you bind the "production" site to 10.0.0.2 and the "administration" site to 10.0.0.1, and configure policy routing (you said Linux, right?) to route appropriately, it should work. It works because Apache can bind sites to individual interfaces. -- Ben
On 11.05.2009, at 23:48, Ben Scott wrote:
On Mon, May 11, 2009 at 5:38 PM, Chris Meidinger <cmeidinger@sendmail.com> wrote:
For example, eth0 is 10.0.0.1/24 and eth1 is 10.0.0.2/24, nothing like bonding going on. The customers usually have the idea of running one interface for administration and another for production (which is a _good_ idea) but they want to do it in the same subnet (not such a good idea...)
I just posted on this, but I didn't really address your original question, so: I'm not aware of anything in the RFCs or other standards which prohibits this. But then, I haven't gone looking, because...
It *can* be made to work in practice, for certain scenarios. For example, if you're talking a web server, and you bind the "production" site to 10.0.0.2 and the "administration" site to 10.0.0.1, and configure policy routing (you said Linux, right?) to route appropriately, it should work. It works because Apache can bind sites to individual interfaces.
Just to restate here, for people who have been responding both publicly and privately: I know that *I* can make it work, and I know that *you* can make it work. But I also know that it's not likely to stay working. One day, down the road, something will break. Then, my poor support team will spend days trying to diagnose the problem. So I want to stop the customer from trying to force a round peg into a square hole, and just use separate subnets for the different interfaces. As someone said before, it's not rocket science. Still though, thanks for all the input; it's really useful. Chris
On May 11, 2009, at 5:59 PM, Chris Meidinger wrote:
Just to restate here, for people who have been responding both publicly and privately:
I know that *I* can make it work, and I know that *you* can make it work. But I also know that it's not likely to stay working.
One day, down the road, something will break. Then, my poor support team will spend days trying to diagnose the problem.
Could you show me a network configuration that does not qualify for that last sentence? Or for that matter, _anything_ related to ... well, anything? -- TTFN, patrick
On Mon, May 11, 2009 at 2:59 PM, Chris Meidinger <cmeidinger@sendmail.com>wrote:
Just to restate here, for people who have been responding both publicly and privately:
I know that *I* can make it work, and I know that *you* can make it work. But I also know that it's not likely to stay working.
You might like to go and tell Sun, and most people who run Solaris. This is not only a valid configuration in Solaris, but it's Sun's recommended redundancy method, via Solaris "IP Multipathing" (IPMP) I have personally configured hundreds of Solaris systems with IPMP, and every single one of them has stayed working, for years in many cases. Yes, you will need a different MAC address for each interface (which isn't the default on many systems, especially Suns), and you won't necessarily be able to control which interface traffic will be sent over if both links are up - but if you're happy with those requirements it's it's a perfectly valid deployment. Scott.
use separate subnets for the different interfaces. As someone said before, it's not rocket science.
It can be a barrier to selling gear if you don't have multiple subnets easily available to you - which is a very big deal for vendors doing evals or for development teams doing staging. Almost never an issue in production, I agree. I worked on a product that had this "1 address for the service", "1 address for administration" concept as its recommended deployment. When the product was in testing or in pre-production the potential customer would sensibly want to set it up the same way it would be in production - with 2 interfaces, each with a different address. But then the prospect would tell our Sales Engineer that they had only one subnet available for testing and it would take weeks or months to remedy that. Half the time that subnet would be DHCP only. As a vendor, our motivation was to lubricate the eval and pre production stages so we could quickly move onto the next trial with a satisfied customer in our wake. We, eventually, supported it all quite smoothly taking into consideration the arp and src address interface selection methods noted elsewhere in this thread. It never posed complications interacting with anything external to our gear. As such, I don't think it is fair to characterize it as a square peg. related link how to configure Linux to do do src address based routing: http://www.linuxjournal.com/article/7291 .. though I agree bonding is a better answer to the motivation laid out in the article. final semi related thought - I have seen devices with an admin/production NIC split where the production (insecure) interface will packets accept all the way up the stack that are (IP.dst == adminIP) as long as you put the production mac as the destination on the packet. That kinda leads to a false sense of security just because they are on different subnets. Gear that doesn't have physically separate processors for control/admin and data/production has to work a lot harder to make sure those things stay separated. -- PenBay Networks VOIPRecorder - Record Calls Made with Vonage(tm) on Your Computer! www.penbaynetworks.com
From: Chris Meidinger <cmeidinger@sendmail.com> Date: Mon, 11 May 2009 23:38:30 +0200
On 11.05.2009, at 23:31, Dan White wrote:
Chris Meidinger wrote:
Hi,
This is a pretty moronic question, but I've been searching RFC's on- and-off for a couple of weeks and can't find an answer. So I'm hoping someone here will know it offhand. I've been looking through RFC's trying to find a clear statement that having two interfaces in the same subnet does not work, but can't find it that statement anywhere. The OS in this case is Linux. I know it can be done with clever routing and prioritization and such, but this has to do with vanilla config, just setting up two interfaces in one network. I would be grateful for a pointer to such an RFC statement, assuming it exists.
If your goal is to achieve redundancy or to increase bandwidth, you can bond the interfaces together - assuming that you have a switch / switch stack that supports 802.3ad.
Then you could assign multiple IPs to the bonded interface without any layer 3 messyness.
I should have been clearer. The case in point is having two physical interfaces, each with a unique IP, in the same subnet.
For example, eth0 is 10.0.0.1/24 and eth1 is 10.0.0.2/24, nothing like bonding going on. The customers usually have the idea of running one interface for administration and another for production (which is a _good_ idea) but they want to do it in the same subnet (not such a good idea...)
This will not work right. One interface can be 10.0.0.1/24, but any added interfaces would need to be /32 (10.0.0.2/32). What your customer wants can probably be done, but it is a really bad idea. Put them in different subnets. If you need to, break off a /30 from the /24. (That is a bit messy as you meed to break the /24 into a /25, a /26, a /27..., but it should work fine. Since the main interface has to talk to ALL of the subnets, you will need to use one address from each and that is pretty wasteful, but it should work.) Just really UGLY! If only a part of the address space need be used, it gets easier and less ugly. If a /25 will work, it's pretty much normal configuration on both interfaces. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
Once upon a time, Kevin Oberman <oberman@es.net> said:
From: Chris Meidinger <cmeidinger@sendmail.com> For example, eth0 is 10.0.0.1/24 and eth1 is 10.0.0.2/24, nothing like bonding going on. The customers usually have the idea of running one interface for administration and another for production (which is a _good_ idea) but they want to do it in the same subnet (not such a good idea...)
This will not work right. One interface can be 10.0.0.1/24, but any added interfaces would need to be /32 (10.0.0.2/32).
I don't know which OS(es) you are using, but that's not true in Linux. I see this all the time at home; if I plug my notebook into the wired LAN and still have the wireless enabled, both will get an IP (in the same subnet) from DHCP. The wired link is the preferred default route by default, but you can easily set up routes for some networks via the wireless link. You can also set up multipath routing to send packets out both links. I think you can also use IP policy routing to control the choice of outbound interface by rule (e.g. based on source address). -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Date: Mon, 11 May 2009 18:29:08 -0500 From: Chris Adams <cmadams@hiwaay.net>
Once upon a time, Kevin Oberman <oberman@es.net> said:
From: Chris Meidinger <cmeidinger@sendmail.com> For example, eth0 is 10.0.0.1/24 and eth1 is 10.0.0.2/24, nothing like bonding going on. The customers usually have the idea of running one interface for administration and another for production (which is a _good_ idea) but they want to do it in the same subnet (not such a good idea...)
This will not work right. One interface can be 10.0.0.1/24, but any added interfaces would need to be /32 (10.0.0.2/32).
I don't know which OS(es) you are using, but that's not true in Linux. I see this all the time at home; if I plug my notebook into the wired LAN and still have the wireless enabled, both will get an IP (in the same subnet) from DHCP. The wired link is the preferred default route by default, but you can easily set up routes for some networks via the wireless link.
You can also set up multipath routing to send packets out both links. I think you can also use IP policy routing to control the choice of outbound interface by rule (e.g. based on source address).
This is true if you are using the WPA supplicant. It does a bit of magic. (You can do the magic by hand without the supplicant, but it is a pain or was the last time I tried.) -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
If it were me and had the requirement of having both NICs in the same L2 segment, but unique IP addresses, I'd assign a secondary IP address to the Layer3 SVI on the upstream device, and give the 2nd NIC on the server an IP on that secondary IP block. Ken Matlock Network Analyst Exempla Healthcare (303) 467-4671 matlockk@exempla.org -----Original Message----- From: Chris Meidinger [mailto:cmeidinger@sendmail.com] Sent: Monday, May 11, 2009 3:39 PM To: Dan White Cc: nanog@nanog.org Subject: Re: two interfaces one subnet On 11.05.2009, at 23:31, Dan White wrote:
Chris Meidinger wrote:
Hi,
This is a pretty moronic question, but I've been searching RFC's on- and-off for a couple of weeks and can't find an answer. So I'm hoping someone here will know it offhand. I've been looking through RFC's trying to find a clear statement that having two interfaces in the same subnet does not work, but can't find it that statement anywhere. The OS in this case is Linux. I know it can be done with clever routing and prioritization and such, but this has to do with vanilla config, just setting up two interfaces in one network. I would be grateful for a pointer to such an RFC statement, assuming it exists.
If your goal is to achieve redundancy or to increase bandwidth, you can bond the interfaces together - assuming that you have a switch / switch stack that supports 802.3ad.
Then you could assign multiple IPs to the bonded interface without any layer 3 messyness.
I should have been clearer. The case in point is having two physical interfaces, each with a unique IP, in the same subnet. For example, eth0 is 10.0.0.1/24 and eth1 is 10.0.0.2/24, nothing like bonding going on. The customers usually have the idea of running one interface for administration and another for production (which is a _good_ idea) but they want to do it in the same subnet (not such a good idea...) Chris
:Hi, : :This is a pretty moronic question, but I've been searching RFC's on- :and-off for a couple of weeks and can't find an answer. So I'm hoping :someone here will know it offhand. : :I've been looking through RFC's trying to find a clear statement that :having two interfaces in the same subnet does not work, but can't find :it that statement anywhere. : :The OS in this case is Linux. I know it can be done with clever :routing and prioritization and such, but this has to do with vanilla :config, just setting up two interfaces in one network. : :I would be grateful for a pointer to such an RFC statement, assuming :it exists. RFC1122, Section 3.3.4.1 explicitly says this IS a legal config from an IP perspective: 3.3.4 Local Multihoming 3.3.4.1 Introduction A multihomed host has multiple IP addresses, which we may think of as "logical interfaces". These logical interfaces may be associated with one or more physical interfaces, and these physical interfaces may be connected to the same or different networks. There are other considerations here -- OS, link-layer, etc. Obviously, you want to do such things with care. But simply from a "standards" perspective, it's ok. There are a lot of hosts that historically didn't have enough RFC1122 compliance to make such configurations problematic (e.g. section 3.3.1.2 and multiple default route support vs. old BSD IP stacks) but that doesn't invalidate the standards. -- Michael J. O'Connor mjo@dojo.mi.org =--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--= "Pain has an element of blank." -Emily Dickinson
On 11.05.2009, at 23:39, Mike O'Connor wrote:
:Hi, : :This is a pretty moronic question, but I've been searching RFC's on- :and-off for a couple of weeks and can't find an answer. So I'm hoping :someone here will know it offhand. : :I've been looking through RFC's trying to find a clear statement that :having two interfaces in the same subnet does not work, but can't find :it that statement anywhere. : :The OS in this case is Linux. I know it can be done with clever :routing and prioritization and such, but this has to do with vanilla :config, just setting up two interfaces in one network. : :I would be grateful for a pointer to such an RFC statement, assuming :it exists.
RFC1122, Section 3.3.4.1 explicitly says this IS a legal config from an IP perspective:
3.3.4 Local Multihoming
3.3.4.1 Introduction
A multihomed host has multiple IP addresses, which we may think of as "logical interfaces". These logical interfaces may be associated with one or more physical interfaces, and these physical interfaces may be connected to the same or different networks.
There are other considerations here -- OS, link-layer, etc. Obviously, you want to do such things with care. But simply from a "standards" perspective, it's ok. There are a lot of hosts that historically didn't have enough RFC1122 compliance to make such configurations problematic (e.g. section 3.3.1.2 and multiple default route support vs. old BSD IP stacks) but that doesn't invalidate the standards.
Thank you! This is what I (wasn't) looking for, but was destined to find. I'll look for other arguments against the practice. And again, because I didn't say it before: Thanks for the pointer!! This is just what I was looking for to stop looking. Best, Chris
Hi Chris, I remember this. I remember it in an early IP RFC, but couldn't find it in 10 minutes of searching. It had to do with intefaces cannot have overlapping address space. One of the IETF greybeards ought to know. It's been a while since I was writing code with marked up rfc's in front of me... Chris
I think the idea of one interface per subnet originates in the early RFCs, such as RFC 1009 "Requirements for Internet Gateways": "Section 1.1.2 Networks and Gateways ... A gateway is connected to two or more networks, appearing to each of these networks as a connected host. Thus, it has a physical interface and an IP address on each of the connected networks ... " So by using singular terminology ( "a connected host", "a physical interface", "an IP address") instead of plural, a single interface per subnet for gateways (read routers) is implied. This is not to say that it will not work, at least on servers. Standards aside, a good reason why this is not a best practice is the concept of asynchronous routing where a packet arrives on one interface, and the reply leaves on the other interface with a different source IP on the reply. Most firewalls will reject packets such as this. -----Original Message----- From: Chris Meidinger [mailto:cmeidinger@sendmail.com] Sent: Monday, May 11, 2009 1:29 PM To: nanog@nanog.org Subject: two interfaces one subnet Hi, This is a pretty moronic question, but I've been searching RFC's on- and-off for a couple of weeks and can't find an answer. So I'm hoping someone here will know it offhand. I've been looking through RFC's trying to find a clear statement that having two interfaces in the same subnet does not work, but can't find it that statement anywhere. The OS in this case is Linux. I know it can be done with clever routing and prioritization and such, but this has to do with vanilla config, just setting up two interfaces in one network. I would be grateful for a pointer to such an RFC statement, assuming it exists. Thanks! Chris
participants (28)
-
Alex H. Ryu -
Arnold Nipper -
Ben Scott -
Brielle Bruns -
Charles Wyble -
Chris Adams -
Chris Meidinger -
chris.ranch＠nokia.com -
Curtis Maurand -
Dan White -
Daryl G. Jurbala -
David Coulson -
David Devereaux-Weber -
Duane Waddle -
Hector Herrera -
Holmes,David A -
Ivan Pepelnjak -
James Hess -
Kevin Oberman -
Martin Hannigan -
Matlock, Kenneth L -
Mikael Abrahamsson -
Mike O'Connor -
Nathan Ward -
Patrick McManus -
Patrick W. Gilmore -
Per Heldal -
Scott Howard