Increase bandwidth usage in partial-mesh network?
Looking for recommendtions or suggestions... I've got a downstream customer asking for help; they have a private internal network that I've taken to calling the "partial-mesh network from hell": it's got two partially-overlapping radio networks, mixed with islands of isolated fiber connectivity. Dynamic routing protocols (IS-IS, OSPF, EIGRP, etc.) generally will only select the _best_ path, they won't spread the load unless all paths are equal - and they are very unequal in this network, ECMP would likely fail horribly. The network is becoming bandwidth-limited, so they're wanting to make use of all available paths, not just the single "best" path. It's also remote and spread out, so adding new links or upgrading existing links is difficult and expensive. Oh, and their routers are overdue for a refresh, so acquiring replacement h/w is now possible. Has anyone come across any product or technology that can handle the multi-path-ness and the private-network-ness like a regular router, but also provides the intelligent per-flow path steering based on e.g. latency, like an SD-WAN device (and/or some firewalls)? Here's hoping, -Adam Adam Thompson Consultant, Infrastructure Services [1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.ca<mailto:athompson@merlin.mb.ca> www.merlin.mb.ca<http://www.merlin.mb.ca/>
Hey! From the description it must be one of our clients! Just beware if you go this route, a network that is probably already unstable and unreliable will become at least an order of magnitude worse. You can't fix ten lbs of stuff into a 4 lb stuff bag. The internet protocols do not tolerate congestion well. On Wed, Oct 13, 2021 at 1:31 PM Adam Thompson <athompson@merlin.mb.ca> wrote:
Looking for recommendtions or suggestions...
I've got a downstream customer asking for help; they have a private internal network that I've taken to calling the "partial-mesh network from hell": it's got two partially-overlapping radio networks, mixed with islands of isolated fiber connectivity. Dynamic routing protocols (IS-IS, OSPF, EIGRP, etc.) generally will only select the _best_ path, they won't spread the load unless all paths are equal - and they are very unequal in this network, ECMP would likely fail horribly. The network is becoming bandwidth-limited, so they're wanting to make use of all available paths, not just the single "best" path. It's also remote and spread out, so adding new links or upgrading existing links is difficult and expensive. Oh, and their routers are overdue for a refresh, so acquiring replacement h/w is now possible.
Has anyone come across any product or technology that can handle the multi-path-ness and the private-network-ness like a regular router, but also provides the intelligent per-flow path steering based on e.g. latency, like an SD-WAN device (and/or some firewalls)?
Here's hoping, -Adam
*Adam Thompson* Consultant, Infrastructure Services [image: 1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.ca www.merlin.mb.ca
-- Fletcher Kittredge GWI 207-602-1134 www.gwi.net
On 10/13/21 19:59, Fletcher Kittredge wrote:
Hey! From the description it must be one of our clients!
Just beware if you go this route, a network that is probably already unstable and unreliable will become at least an order of magnitude worse. You can't fix ten lbs of stuff into a 4 lb stuff bag. The internet protocols do not tolerate congestion well.
It sounds like they need to get back to the basics first. Simplification, in lieu of added complexity, seems to be the appealing approach. Mark.
Hah, no not your client 🙂. Their existing network is actually surprisingly stable, but it is bandwidth-constrained. As well as the various other replies I've seen here and off-list (THANKS!), the only commercial product I've found so far that might have a hope of handling this is HPE/Aruba's Silverpeak line. We'll see what else comes out of the woodwork, though - if nothing else, it's a very interesting exercise! Adam Thompson Consultant, Infrastructure Services [1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.ca<mailto:athompson@merlin.mb.ca> www.merlin.mb.ca<http://www.merlin.mb.ca/> ________________________________ From: Fletcher Kittredge <fkittred@gwi.net> Sent: October 13, 2021 12:59 To: Adam Thompson <athompson@merlin.mb.ca> Cc: nanog <nanog@nanog.org> Subject: Re: Increase bandwidth usage in partial-mesh network? Hey! From the description it must be one of our clients! Just beware if you go this route, a network that is probably already unstable and unreliable will become at least an order of magnitude worse. You can't fix ten lbs of stuff into a 4 lb stuff bag. The internet protocols do not tolerate congestion well. On Wed, Oct 13, 2021 at 1:31 PM Adam Thompson <athompson@merlin.mb.ca<mailto:athompson@merlin.mb.ca>> wrote: Looking for recommendtions or suggestions... I've got a downstream customer asking for help; they have a private internal network that I've taken to calling the "partial-mesh network from hell": it's got two partially-overlapping radio networks, mixed with islands of isolated fiber connectivity. Dynamic routing protocols (IS-IS, OSPF, EIGRP, etc.) generally will only select the _best_ path, they won't spread the load unless all paths are equal - and they are very unequal in this network, ECMP would likely fail horribly. The network is becoming bandwidth-limited, so they're wanting to make use of all available paths, not just the single "best" path. It's also remote and spread out, so adding new links or upgrading existing links is difficult and expensive. Oh, and their routers are overdue for a refresh, so acquiring replacement h/w is now possible. Has anyone come across any product or technology that can handle the multi-path-ness and the private-network-ness like a regular router, but also provides the intelligent per-flow path steering based on e.g. latency, like an SD-WAN device (and/or some firewalls)? Here's hoping, -Adam Adam Thompson Consultant, Infrastructure Services [1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.ca<mailto:athompson@merlin.mb.ca> www.merlin.mb.ca<http://www.merlin.mb.ca/> -- Fletcher Kittredge GWI 207-602-1134 www.gwi.net<http://www.gwi.net>
On Wed, Oct 13, 2021 at 10:30 AM Adam Thompson <athompson@merlin.mb.ca> wrote:
Has anyone come across any product or technology that can handle the multi-path-ness and the private-network-ness like a regular router, but also provides the intelligent per-flow path steering based on e.g. latency, like an SD-WAN device (and/or some firewalls)?
The babel protocol does some of this. https://datatracker.ietf.org/doc/html/rfc6126 Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
Assuming that the reasons for the low bandwidth and use of radio is due to physical constraints - distances, inhospitable terrain between nodes, etc. In this case, some good 'ol MPLS traffic engineering using LSP's with bandwidth reservations may be the way to influence how traffic is routed. Then, they may need some platform to provide observability and potentially dynamic re-routing of LSP's based on actual or predicted congestion situations. If traffic patterns and utilization are not ideally deterministic, then skip the bandwidth reservation and ensure that the automation is in place to reroute traffic when necessary. I know, adding complexity, but if you just can't build the links you would want, this may be a way to work with what you've got. Best Regards, Mauricio Rodriguez Founder / Owner Fletnet Network Engineering (www.fletnet.com) *Follow us* on LinkedIn <https://www.linkedin.com/company/fletnetnetworks> Mauricio.Rodriguez@fletnet.com Office: +1 786-309-1082 Direct: +1 786-309-5493 On Wed, Oct 13, 2021 at 1:33 PM Adam Thompson <athompson@merlin.mb.ca> wrote:
Looking for recommendtions or suggestions...
I've got a downstream customer asking for help; they have a private internal network that I've taken to calling the "partial-mesh network from hell": it's got two partially-overlapping radio networks, mixed with islands of isolated fiber connectivity. Dynamic routing protocols (IS-IS, OSPF, EIGRP, etc.) generally will only select the _best_ path, they won't spread the load unless all paths are equal - and they are very unequal in this network, ECMP would likely fail horribly. The network is becoming bandwidth-limited, so they're wanting to make use of all available paths, not just the single "best" path. It's also remote and spread out, so adding new links or upgrading existing links is difficult and expensive. Oh, and their routers are overdue for a refresh, so acquiring replacement h/w is now possible.
Has anyone come across any product or technology that can handle the multi-path-ness and the private-network-ness like a regular router, but also provides the intelligent per-flow path steering based on e.g. latency, like an SD-WAN device (and/or some firewalls)?
Here's hoping, -Adam
*Adam Thompson* Consultant, Infrastructure Services [image: 1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.ca www.merlin.mb.ca
-- This message (and any associated files) may contain confidential and/or privileged information. If you are not the intended recipient or authorized to receive this for the intended recipient, you must not use, copy, disclose or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by sending a reply e-mail and delete this message. Thank you for your cooperation.
Maybe something like this (if you can break it into different bgp ASNs by network area): "draft-mohanty-bess-ebgp-dmz-03" https://datatracker.ietf.org/doc/html/draft-mohanty-bess-ebgp-dmz-03 On Wed, Oct 13, 2021, 10:30 Adam Thompson <athompson@merlin.mb.ca> wrote:
Looking for recommendtions or suggestions...
I've got a downstream customer asking for help; they have a private internal network that I've taken to calling the "partial-mesh network from hell": it's got two partially-overlapping radio networks, mixed with islands of isolated fiber connectivity. Dynamic routing protocols (IS-IS, OSPF, EIGRP, etc.) generally will only select the _best_ path, they won't spread the load unless all paths are equal - and they are very unequal in this network, ECMP would likely fail horribly. The network is becoming bandwidth-limited, so they're wanting to make use of all available paths, not just the single "best" path. It's also remote and spread out, so adding new links or upgrading existing links is difficult and expensive. Oh, and their routers are overdue for a refresh, so acquiring replacement h/w is now possible.
Has anyone come across any product or technology that can handle the multi-path-ness and the private-network-ness like a regular router, but also provides the intelligent per-flow path steering based on e.g. latency, like an SD-WAN device (and/or some firewalls)?
Here's hoping, -Adam
*Adam Thompson* Consultant, Infrastructure Services [image: 1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.ca www.merlin.mb.ca
On 10/13/21 11:29 AM, Adam Thompson wrote:
I've got a downstream customer asking for help; they have a private internal network that I've taken to calling the "partial-mesh network from hell": it's got two partially-overlapping radio networks, mixed with islands of isolated fiber connectivity. Dynamic routing protocols (IS-IS, OSPF, EIGRP, etc.) generally will only select the _best_ path, they won't spread the load unless all paths are equal - and they are very unequal in this network, ECMP would likely fail horribly. The network is becoming bandwidth-limited, so they're wanting to make use of all available paths, not just the single "best" path. It's also remote and spread out, so adding new links or upgrading existing links is difficult and expensive. Oh, and their routers are overdue for a refresh, so acquiring replacement h/w is now possible.
Has anyone come across any product or technology that can handle the multi-path-ness and the private-network-ness like a regular router, but also provides the intelligent per-flow path steering based on e.g. latency, like an SD-WAN device (and/or some firewalls)?
Maybe add a little bit of linear optimization on top of faucet/openvswitch/openflow to calculate best paths based upon bandwidth, paths, and fill-factors. There is a presentation where Google uses that technique to obtain high utilization on their links (not necessarily those tools though). Raymond Burkholder
Has anyone come across any product or technology that can handle the multi-path-ness and the private-network-ness like a regular router, but also provides the intelligent per-flow path steering based on e.g. latency, like an SD-WAN device (and/or some firewalls)? Maybe add a little bit of linear optimization on top of faucet/openvswitch/openflow to calculate best paths based upon bandwidth, paths, and fill-factors. There is a presentation where Google uses that technique to obtain high utilization on their links (not necessarily those tools though). Raymond Burkholder This is what a large Italian wisp has done, here are a couple of presentations made at our ITNOG sessions. I’m not sure if they have open sourced anything though. https://www.itnog.it/itnog4/files/14-Traffic%20Engineering%20-%20the%20EOLO%... https://www.itnog.it/itnog3/files/ITNOG3-EOLO.pdf Brian
participants (9)
-
Adam Thompson -
Arie Vayner -
Brian Turnbow -
Fletcher Kittredge -
Karsten Thomann -
Mark Tinka -
Mauricio Rodriguez -
Raymond Burkholder -
William Herrin