On Wed, 25 Dec 2024 at 05:34, Matthew Petach <mpetach@netflight.com> wrote:
Power is a *huge* part of the equation that I think many people overlook. When you look at what a really big chassis takes in terms of power feeds, it's not uncommon to need relatively specialized 3-phase 240V power feeds for the very-high-end chassis box that give you the same type of high speed port densities that a pizza-box fabric folded Clos model can yield. (not to pick on any vendor, but here's an example of the types of power feeds a large chassis can require:)
This and most of the differences are implementation details, not fundamental. If we assume some fantastical world, where you can make anything appear out of thin air with no cost, no one will use the same interfaces to connect customers and to build interconnect between chips. Because the serdes fabric interfaces are lower power, pin, thermal, cost than the real customer facing port and you can have higher density of them. On very high and impractical level, that's the difference, how do you interconnect chips, do you use some specialised solution that understands that both ends are going to be the chips in the same rack or do you use generic solution that makes no assumption on what is going to be connected on the other end. In practice people do these stack of switches, because they want rightsized platforms and there isn't quite the right size available from anyone that they care to deploy. So the rightsized stack ends up being commercially more viable and more energy efficient, because the right box for the application has no commercial availability. Luckily for most of us, these problems do not matter, as chip densities are front-running even most hyperscalers, when Amazon presented their stack-of-switches solutions couple years ago, giving the densities and how many front-facing ports are 'wasted' on internal interconnect, vendors were already shipping single chip solutions matching the stack-of-switches non-interconnect port densities, i.e. already this hyperscaler solution could have been single chip device, not stack-of-switches, not fabric box. And this is true for almost every buyer in the market, you need just a single chip box today, densities are absurd for almost everyone. -- ++ytti