I'm having a hard time seeing how this reduces cable costs or increases network durability. Each individual server is well connected to 3-4 other servers in the rack, but the rack still only has two uplinks. For many servers in the rack you're adding 3-4 routing hops between an end node and the rack uplink. Additionally, with only 2 external links tied to 2 specific nodes, you introduce more risks. If one of the uplink nodes fails, you've got to re-route all of the nodes that were using it as the shortest path to now exit through the other uplink node -- the worst case in the example then increases from the original 4-hops-to-exit to now 7-hops-to-exit. As far as cable costs go, you might have slightly shorter cables, but far more complex wiring pattern -- so in essence you're trading off a small amount of cable cost for a higher amount of installation and troubleshooting cost. Also, using this layout, you dramatically reduce the effective bandwidth available between devices, since per-device links now have to be used for backhaul/transport in addition to device-specific traffic. Finally, you have to manage per-server routing service configurations to make this work -- more points of failure and increased setup/troubleshooting cost. In a ToR switch scenario, you do one config on one switch, plug in the cables, and you're done -- problems happen, you go to the one switch, not chasing a needle through a haystack of interconnected servers. If your RU count is worth more than the combination of increased installation, server configuration, troubleshooting, latency, and capacity costs, then this is a good solution. Either way, it's a neat idea and a fun thought experiment to work through. Thanks! Dan -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of NAOTO MATSUMOTO Sent: Wednesday, February 11, 2015 11:32 PM To: nanog@nanog.org Subject: FYI: An Easy way to build a server cluster without top of rack switches (MEMO) Hi all! We wrote up TIPS memo "an easy way to build a server cluster without top of rack switches" concept. This model have a reduce switches and cables costs and high network durability by lightweight and simple configuration. if you interest in, please try to do yourself this concept ;-) An Easy way to build a server cluster without top of rack switches (MEMO) http://slidesha.re/1EduYXM Best regards, -- Naoto MATSUMOTO