Best practice for ptp/loopback numbering for "small" enterprise multihome setup
Hello, tl;dr - If I only have a /24 PI - is there any way to use this and not “chop it up / deagg” to use for ptp/loopbacks ? Hopefully I can explain this in a manner that makes sense. Say I have a vanilla dual router/dual upstream setup (think enterprise internet edge). It’s basically an “H” shape: - Two ISPs - Two routers (“crosslink” is the middle of the H - iBGP) - Each router has at least a link downstream into my public “outside” segment. I run an FHRP here. This is where my DMZ firewalls, VPN endpoints etc. have their outside interfaces. Let’s also say I only have a /24 of PI. I need to number the crosslink and the loopbacks. The upstreams will use their own /30 / /31 let’s say for the top of the H. My downstream interfaces will have my /24 (or parts of it) on the bottom of the H. My understanding and instinct is that at least the crosslink should be numbered with public addresses. One scenario where this might matter is if router2 loses his inside interface (ex: switch failure) - he still needs to be able to deliver traffic via crosslink via router1 and onward downstream. When this traffic traverses here, if there is any kind of error (i.e. router is sourcing an ICMP packet of some kind) - this will be sourced from router1’s crosslink yes (assuming router1 is generating the error of course) ? Loopback’s may be negotiable, as only router1/2 are using these to pin up iBGP. Nevertheless, my instinct would be to also use public addresses here. As I said in the tl;dr - my main point of contention here is breaking up my /24 I.e. use the very top /30s / /31s for ptp/loop. I would then have at most the bottom /25 to use contig. on my “lan” - and I would need to use the next /26, /27 and so on in some manner for the space to be useable... Here are some other options, and my understanding of the pros/cons: - Use RFC1918 Makes my eye twitch out of the gate. Not to mention packets sourced from here *should* get blocked by my upstreams by way of uRPF. Likely to be filtered at other points and directions as well… - Get a /29 from one of the ISPs for ptp/loops Better than RFC1918. Kind of weird. If this is from isp1 - should have no issues sourcing toward them. Might have issues (uRPF) toward isp2. Announce this via BGP / no-export to both ISPs ? Now it’s getting even weirder… - Use my own PI space Should have no filtering issues at all. Now I have to deagg my /24 and use the pieces (largest /25). Am I making too big of a deal of this ? If you’ve read this far - I do appreciate it. Anxious to hear feedback on this. PS: I would likely ACL on my upstream interfaces to block direct packets to my routers themselves as well.
tl;dr - If I only have a /24 PI - is there any way to use this and not “chop it up / deagg” to use for ptp/loopbacks ?
i take real addresses out of the /24 for p2p i take 1918 addresses for ibgp loopbacks randy --- randy@psg.com `gpg --locate-external-keys --auto-key-locate wkd randy@psg.com` signatures are back, thanks to dmarc header butchery
i take real addresses out of the /24 for p2p i take 1918 addresses for ibgp loopbacks
This. On Fri, Mar 26, 2021 at 2:22 PM Randy Bush <randy@psg.com> wrote:
tl;dr - If I only have a /24 PI - is there any way to use this and not “chop it up / deagg” to use for ptp/loopbacks ?
i take real addresses out of the /24 for p2p i take 1918 addresses for ibgp loopbacks
randy
--- randy@psg.com `gpg --locate-external-keys --auto-key-locate wkd randy@psg.com` signatures are back, thanks to dmarc header butchery
On 3/26/2021 12:01 PM, vom513 wrote:
Hello,
tl;dr - If I only have a /24 PI - is there any way to use this and not “chop it up / deagg” to use for ptp/loopbacks ?
Hopefully I can explain this in a manner that makes sense.
Say I have a vanilla dual router/dual upstream setup (think enterprise internet edge).
It’s basically an “H” shape:
- Two ISPs - Two routers (“crosslink” is the middle of the H - iBGP) - Each router has at least a link downstream into my public “outside” segment. I run an FHRP here. This is where my DMZ firewalls, VPN endpoints etc. have their outside interfaces.
Let’s also say I only have a /24 of PI.
I need to number the crosslink and the loopbacks. The upstreams will use their own /30 / /31 let’s say for the top of the H. My downstream interfaces will have my /24 (or parts of it) on the bottom of the H.
Couple things come to mind that might be a more efficient use of address space: First, you don't need two routers in order to have dual upstreams. Have you considered multi-homing using a single router? If you need redundancy, it could be built into a single chassis. Another option is that some routers can perform active/standby failover without the need for extra public addresses. For example, two Cisco ASAs would have a cross-link, but this link is limited to keeping state and HA heartbeat between the two units and can be numbered with either an IPv6 link local or RFC-1918 address. Other platforms may have the option for Virtual Chassis, VSS, stacking, or similar technology that can conserve address space compared to two independent and traditionally addressed routers.
On Fri, Mar 26, 2021 at 11:07 AM vom513 <vom513@gmail.com> wrote:
As I said in the tl;dr - my main point of contention here is breaking up my /24 I.e. use the very top /30s / /31s for ptp/loop. I would then have at most the bottom /25 to use contig. on my “lan” - and I would need to use the next /26, /27 and so on in some manner for the space to be useable...
If you feel like getting fancy... Use /32 routes to reclaim the unused base and broadcast address in any /30s Pick the next largest size block that has your /24 neither at the start or end and assign that to your lan. Use proxy arp and more specific routes to grab traffic that doesn't fall in the /24 or is part of the loopback and point to point numbering and move it off the lan and towards its destination (including upstream). This way, every address in the /24 that you don't specifically use elsewhere is usable as a unicast address on your LAN segment, including the .0 and .255 addresses. e.g. You have 10.0.1.0/24 Put 10.0.0.0/22 on your LAN Add proxy arp and route 10.0.0.0/24 upstream Add proxy arp and route 10.0.2.0/23 upstream Add proxy arp and route 10.0.1.254/32 to your first router loopback etc. If you're really clever you can convince the stations that 10.0.0.1 is the default gateway but convince the router that 10.0.0.1 is upstream so that the router doesn't even need a dedicated IP address facing the LAN. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
On 3/26/2021 2:01 PM, William Herrin wrote:
If you feel like getting fancy...
Use /32 routes to reclaim the unused base and broadcast address in any /30s
Pick the next largest size block that has your /24 neither at the start or end and assign that to your lan.
Use proxy arp and more specific routes to grab traffic that doesn't fall in the /24 or is part of the loopback and point to point numbering and move it off the lan and towards its destination (including upstream).
..
You have 10.0.1.0/24
Put 10.0.0.0/22 on your LAN Add proxy arp and route 10.0.0.0/24 upstream Add proxy arp and route 10.0.2.0/23 upstream Add proxy arp and route 10.0.1.254/32 to your first router loopback etc.
If you're really clever you can convince the stations that 10.0.0.1 is the default gateway but convince the router that 10.0.0.1 is upstream so that the router doesn't even need a dedicated IP address facing the LAN.
Regards, Bill Herrin
And here I almost went as far as to suggest unnumbered IPs.... you're plan is... well... diabolical in comparison.
On Fri, Mar 26, 2021 at 12:14 PM Blake Hudson <blake@ispn.net> wrote:
And here I almost went as far as to suggest unnumbered IPs.... you're plan is... well... diabolical in comparison.
Hi Blake, I aim to please. I really wish the router vendors supported a statically configured "ICMP error from" address overriding source address selection for ICMP error messages so that you could just put RFC1918 on the router to router links instead of wasting global IP addresses on them. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
On Fri, 26 Mar 2021 at 20:21, William Herrin <bill@herrin.us> wrote:
On Fri, Mar 26, 2021 at 12:14 PM Blake Hudson <blake@ispn.net> wrote:
And here I almost went as far as to suggest unnumbered IPs.... you're plan is... well... diabolical in comparison.
Hi Blake,
I aim to please. I really wish the router vendors supported a statically configured "ICMP error from" address overriding source address selection for ICMP error messages so that you could just put RFC1918 on the router to router links instead of wasting global IP addresses on them.
Yes, in such a case, it would be extremely useful. However for bigger transit networks, it is not nice if you only see loopbacks in traceroutes, as opposed to interfaces. In this particular case: The only P2P interface that needs addressing is the link between the two routers. For an enterprise stub-AS, I would certainly compromise and use RFC1918 here as well (and use the entire /24 on the user facing interface). Other than traceroutes, this does not realistically cause an issue, *IN THIS PARTICULAR* scenario: you will not have different MTU's requiring proper PMTUD handling in this direction and some broken traceroutes won't be a problem for this business case. We can't stop huge eyeball networks from using RFC1918 addressing on P2P links, I guess we can live with a stub-AS doing it. Another alternative is to use the actual user interface to get your iBGP across, which is publicly addressed. Of course think about different failure scenarios (partitioning), but it can work too. Maybe use a different RFC1918 addressed P2P link with higher IGP costs, so the 1918 dedicated link is used for in case of LAN partitioning only, and usually you see public IP's in the traceroute. I would avoid single control plane redundancy with proprietary solutions like VSS and the likes. At some point they will tell you: for this OS upgrade, you need to take BOTH devices out, because you can't upgrade the cluster from major release X to Y, without completely bringing it down. Lukas
On 3/26/2021 3:27 PM, Lukas Tribus wrote:
Another alternative is to use the actual user interface to get your iBGP across, which is publicly addressed.
I think this may actually be the best solution proposed so far. iBGP (as spec'd by the op) can be setup between the WAN, LAN, loopback, or tunnel interfaces just as easily as using a dedicated p2p link. No reason to add complexity by throwing in an unnecessary p2p link in the mix.
On Fri, 26 Mar 2021 at 20:01, William Herrin <bill@herrin.us> wrote:
On Fri, Mar 26, 2021 at 11:07 AM vom513 <vom513@gmail.com> wrote:
As I said in the tl;dr - my main point of contention here is breaking up my /24 I.e. use the very top /30s / /31s for ptp/loop. I would then have at most the bottom /25 to use contig. on my “lan” - and I would need to use the next /26, /27 and so on in some manner for the space to be useable...
If you feel like getting fancy...
Use /32 routes to reclaim the unused base and broadcast address in any /30s
Pick the next largest size block that has your /24 neither at the start or end and assign that to your lan.
Use proxy arp and more specific routes to grab traffic that doesn't fall in the /24 or is part of the loopback and point to point numbering and move it off the lan and towards its destination (including upstream).
In a lab, sure. In production, you may be able to troubleshoot this a few months from now, but how will the on-duty junior engineer handle this at 03 AM?
This way, every address in the /24 that you don't specifically use elsewhere is usable as a unicast address on your LAN segment, including the .0 and .255 addresses.
e.g.
You have 10.0.1.0/24
Put 10.0.0.0/22 on your LAN Add proxy arp and route 10.0.0.0/24 upstream Add proxy arp and route 10.0.2.0/23 upstream Add proxy arp and route 10.0.1.254/32 to your first router loopback etc.
If you're really clever you can convince the stations that 10.0.0.1 is the default gateway but convince the router that 10.0.0.1 is upstream so that the router doesn't even need a dedicated IP address facing the LAN.
What you are suggesting is to configure public IP address space that isn't yours, this should be a big nono. At the very least you can't reach the public IP addresses 10.0.0.0 and 10.0.3.255 from the hosts, because they won't be sending ARP requests for subnet and broadcast addresses. So you break real public IP's of other people. In the worst case scenario, some of this goes wrong and you cause entire public subnets to be unreachable from this network. We are talking about how to avoid 1918 addressing on P2P links. I'd say NOT configuring other people's public IP addresses on your network is more important than NOT configuring 1918 addressing on your network. Lukas
On Fri, Mar 26, 2021 at 1:42 PM Lukas Tribus <lukas@ltri.eu> wrote:
In production, you may be able to troubleshoot this a few months from now, but how will the on-duty junior engineer handle this at 03 AM?
Hi Lukas, In the network Vom describes, he is surely the only network engineer. I would agree that it is not something anyone should try at scale -- the configuration complexity is higher than any ordinary network architecture. Vom's question was how to carve off some addresses without being stuck at 1/2 the allocation as his maximum subnet size. At the sacrifice of some complexity, it can be done. As described, you can even recapture 3 addresses that would normally be lost to you were you not attempting to carve off addresses.
What you are suggesting is to configure public IP address space that isn't yours, this should be a big nono.
That's one way of looking at it. Here's a different one: It is an entirely legitimate network configuration to give your LAN a 0.0.0.0 netmask and rely on proxy arp to route off of it for non-local addresses. Nobody does it this way, it's inefficient and gets very complex when there's more than one router, but it in no way implies configuring yourself address space which is not yours.
At the very least you can't reach the public IP addresses 10.0.0.0 and 10.0.3.255 from the hosts, because they won't be sending ARP requests for subnet and broadcast addresses.
In the described configuration, those addresses are almost guaranteed to be base addresses or broadcast addresses of someone else's network which you wouldn't be able to reach or access anyway. There is a tiny chance that someone else did the same thing you did or decided to use a /32 route to capture and use those two addresses as unicast, but you've a better chance of winning the lottery or being hit by lightning than finding those two addresses in use. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
Hi Bill, On Fri, 26 Mar 2021 at 22:16, William Herrin <bill@herrin.us> wrote:
On Fri, Mar 26, 2021 at 1:42 PM Lukas Tribus <lukas@ltri.eu> wrote:
In production, you may be able to troubleshoot this a few months from now, but how will the on-duty junior engineer handle this at 03 AM?
Hi Lukas,
In the network Vom describes, he is surely the only network engineer.
Actually I think it's more likely that he's a contractor/consultant, but either way, contractor/consultant or employee ... all of them change over time. This falls into the kind of duct-tape "solutions" that inevitably cause issues down the line, which then have to be diagnosed by engineers at other networks. I'm that engineer at the other network, diagnosing the issue because quote "it must be your fault because we only have the issue with that single IP address of yours. Also Youtube works just fine here."
Vom's question was how to carve off some addresses without being stuck at 1/2 the allocation as his maximum subnet size. At the sacrifice of some complexity, it can be done. As described, you can even recapture 3 addresses that would normally be lost to you were you not attempting to carve off addresses.
Almost anything can be done by increasing complexity. But here the cure is worse than the disease.
What you are suggesting is to configure public IP address space that isn't yours, this should be a big nono.
That's one way of looking at it. Here's a different one: It is an entirely legitimate network configuration to give your LAN a 0.0.0.0 netmask and rely on proxy arp to route off of it for non-local addresses. Nobody does it this way, it's inefficient and gets very complex when there's more than one router, but it in no way implies configuring yourself address space which is not yours.
You are configuring a prefix that is not assigned to you and not specifically reserved for local connectivity like 1918 (instead it is almost certainly partially assigned to another AS for IPv4 unicast use on the public Internet). That is the very definition of configuring address space which is not yours, whether you're successful or not at reducing the impact with proxy-arp. Sure, in this specific case 0.0.0.0 and 255.255.255.255 truly will never be used by anyone on the Internet (and also your hosts/router will almost certainly crash, because ARP tables are not designed for hundreds of thousands of entries), but that's not the point. It's not legitimate in my book, when we are talking about hosts on the public Internet which are required to connect to the rest of the Internet, maybe even host services. If the bar is "can I be sued for this?" than you are certainly right.
At the very least you can't reach the public IP addresses 10.0.0.0 and 10.0.3.255 from the hosts, because they won't be sending ARP requests for subnet and broadcast addresses.
In the described configuration, those addresses are almost guaranteed to be base addresses or broadcast addresses of someone else's network which you wouldn't be able to reach or access anyway. There is a tiny chance that someone else did the same thing you did or decided to use a /32 route to capture and use those two addresses as unicast, but you've a better chance of winning the lottery or being hit by lightning than finding those two addresses in use.
Eyeball networks assign /32 to end users including .0 and .255. Of course the likelihood that those two addresses are actually requiring end-to-end connectivity to this AS is not huge. But the fact of the matter is that you are knowingly breaking a valid configuration in other people's networks with a non-null likelihood of it causing problems, and for what? To avoid 1918 addressing on a single P2P link of an enterprise stub-AS? This cure truly is worse than the disease and it would certainly be unacceptable in my book. Now let's talk about the likelihood of the OP leaking the /22 to the transits (which hopefully filter strictly) in the process of setting this up. I won't need lottery or lightning analogies for that. Lukas Lukas
participants (6)
-
Blake Hudson -
Lukas Tribus -
Randy Bush -
Tom Beecher -
vom513 -
William Herrin