I haven't looked that deeply yet. I was assuming you could just start with a single pizza box and add more on as requirements matured. It certainly gets more complicated quickly if you can't do that.
Fabric, fabric, fabric. For a modular chassis, the fabric capacity has to be sized so it's fully non-blocking for a fully populated set of linecards. Even if you're only using 2/10 slots, still needs to be sized for all 10. For distributed chassis , you can get away with sizing your fabric stage for exactly what you need, but you still have to be aware of expansion, port allocations, etc. You normally would pre-plan out the max size , since you can't easily just shuffle those 'internal links' around. You also tend to want to buy switches for the full fabric in one shot anyways ; mixing buffer sizes in these at the same stage/level is death. When the vendor builds it in, you don't have to think about all of these things. But if you build it yourself you really have to understand it. On Sat, Dec 21, 2024 at 5:25 PM Mike Hammett <nanog@ics-il.net> wrote:
True. Small networks would just have a single pizza box and call it a day.
I haven't looked that deeply yet. I was assuming you could just start with a single pizza box and add more on as requirements matured. It certainly gets more complicated quickly if you can't do that.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> ------------------------------ *From: *"Yan Filyurin" <yanf787@gmail.com> *To: *"Mike Hammett" <nanog@ics-il.net> *Cc: *"Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org> *Sent: *Saturday, December 21, 2024 3:52:27 PM *Subject: *Re: Distributed Router Fabrics
Maybe more like medium, but if you know that you won't grow beyond a certain size and growth trajectory, chassis would make life easier. If you are dealing with some compute and you know how many racks you have, same thing. In fact with small networks, you are actually starting out with more than what you need, since you have to install "line card" and "backplane" boxes. Plus Route Processors. So you are thinking of going beyond the capacity of a single pizza box (or half of device), you are starting with a chassis.
If you are going down the road of pizza boxes, it could be easier to standardize deployments to a single type of device. And not think about which chassis to buy.
On Sat, Dec 21, 2024 at 2:54 PM Mike Hammett <nanog@ics-il.net> wrote:
Oh, so you're saying that small networks benefit more from a traditional chassis than a distributed fabric? I would have thought it the other way around in that you could start with a single pizza box, then add another appropriate to the need, then another appropriate to the need as opposed to trying to figure out if you needed a 4, 8, 13, 16, or 20 slot chassis and then end up either over (or under) buying.
I guess it also depends on one's definition of small.
I guess it also depends on what tooling is available. So often, I see platforms offer a bunch of programability, but then no one commercially (or open source) provides that tooling - they expect you to build it yourself. Most anyone can sit down at XYZ chassis and figure it out, but if it's obscure distributed system without centralized tooling, that could be tricky. Well, if you have more than a handful of boxes.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> ------------------------------ *From: *"Tom Beecher" <beecher@beecher.cc> *To: *"Yan Filyurin" <yanf787@gmail.com> *Cc: *"Mike Hammett" <nanog@ics-il.net>, "NANOG" <nanog@nanog.org> *Sent: *Saturday, December 21, 2024 12:33:54 PM *Subject: *Re: Distributed Router Fabrics
It's just tradeoffs.
Many of the benefits ( smaller failure domains, power savings , incremental expandability ) can be counterbalanced by increased operational complexity. From my experiences, if you don't have proper automation/tooling for management/configuration and fault detection, it's a nightmare. If you do have those things, then the benefits can be substantial.
I think every network will have a tipping point in which such a model starts to make more sense, but at smaller scales I think fat chassis are still likely a better place to be.
On Sat, Dec 21, 2024 at 9:51 AM Yan Filyurin <yanf787@gmail.com> wrote:
When you say distributed router fabrics, are you thinking OCP concept with interconnect switch with ATM-like cell relay (after flowery speeches about "not betting against Ethernet", or course)?
https://www.youtube.com/watch?v=l_hyZwf6-Y0
https://www.ufispace.com/company/blog/what-is-a-distributed-disaggregated-ch...
mostly advocated by Drivenets. It has been a while, but from what I remember, the argument, and it has a lot of merit, is you can scale to a lot bigger "chassis" than you could with any bigiron device. If you look at Broadcom latest interconnect specs https://www.broadcom.com/products/ethernet-connectivity/switching/stratadnx/..., you can build a pretty big Pops, and while they are trying to appeal mostly to AI cluster crowd, one could build aggregation services with that, or something smaller and you get incremental scaling and possibly higher availability, since everything is separated and you could even get enough RPs for proper consensus. I admit, I have never seen it outside of lab environment, but AT&T appears to like it. Plus all the mechanics of getting through your fabric are still handled by the vendor and you manage it like a single node.
One could argue that with chassis systems, you can still scale incrementally, use different line card ports for access and aggregation and your leaf/interconnect is purely electrical, so you are not spending money on optics, so it does not exactly invalidate chassis setup and that is why every big vendor will sell you both, especially if you are not of AT&T scale.
There is of course the other design with normal Ethernet fabrics based on Fat Tree or some other topology with all the normal protocols between the devices, but then you are in charge of setting up, traffic engineering and scaling those protocols. IETF has done interesting things with these scaling ideas and some vendors may have even implemented them to the point that they work. :) But "too many devices" argument starts creeping in.
Yan
On Fri, Dec 20, 2024 at 5:43 PM Mike Hammett <nanog@ics-il.net> wrote:
I've noticed that the whitebox hardware vendors are pushing distributed router fabrics, where you can keep buying pizza boxes and hooking them into a larger and larger fabric. Obviously, at some point, buying a big chassis makes more sense. Does it make sense building up to that point? What are your thoughts on that direction?
----- Mike Hammett [ http://www.ics-il.com/ | Intelligent Computing Solutions ] [ https://www.facebook.com/ICSIL ] [ https://plus.google.com/+IntelligentComputingSolutionsDeKalb ] [ https://www.linkedin.com/company/intelligent-computing-solutions ] [ https://twitter.com/ICSIL ] [ http://www.midwest-ix.com/ | Midwest Internet Exchange ] [ https://www.facebook.com/mdwestix ] [ https://www.linkedin.com/company/midwest-internet-exchange ] [ https://twitter.com/mdwestix ] [ http://www.thebrotherswisp.com/ | The Brothers WISP ] [ https://www.facebook.com/thebrotherswisp ] [ https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg ]