Hi Dan and ken. I respect your great works. Certainly, our scenario was network classics and it just does not "one size fits all" network architecture. Many people tried to built centralized and decentralized networks many years ago, some guys output implementation like this. Interconnect of K computer (torus fusion) https://www.fujitsu.com/global/Images/fujitsu-hpc-roadmap-beyond-petascale-c... I agree with you point. Our approach have to do more simple way on physical and logical network engineering, and change the mindset, I think. (e.g. network cabling procedure and troubleshooting and handling) But, some guys need more cost effective server cluster environment and they does't care network latency like Low-End Web Hosting. e.g. Intel Diversity of Server workloads http://bit.ly/1BgFH65 [JPG]. Now, Many people do not use Dijkstra and automaton theory on the server side, but it is great mechanism for network durability if they controlled. The Ethernet NIC's bandwidth is increasing day by day, the cost is decreasing too. I say again, our scenario is not "one size fits all" network architecture, but we believe that something will happen for some guys works. ;-) best regards, -- Naoto MATSUMOTO On Sat, Feb 14, 2015 at 7:08 AM, Dan Eckert <daniel.eckert@microsoft.com> wrote:
I'm having a hard time seeing how this reduces cable costs or increases network durability. Each individual server is well connected to 3-4 other servers in the rack, but the rack still only has two uplinks. For many servers in the rack you're adding 3-4 routing hops between an end node and the rack uplink.
Additionally, with only 2 external links tied to 2 specific nodes, you introduce more risks. If one of the uplink nodes fails, you've got to re-route all of the nodes that were using it as the shortest path to now exit through the other uplink node -- the worst case in the example then increases from the original 4-hops-to-exit to now 7-hops-to-exit.
As far as cable costs go, you might have slightly shorter cables, but far more complex wiring pattern -- so in essence you're trading off a small amount of cable cost for a higher amount of installation and troubleshooting cost.
Also, using this layout, you dramatically reduce the effective bandwidth available between devices, since per-device links now have to be used for backhaul/transport in addition to device-specific traffic.
Finally, you have to manage per-server routing service configurations to make this work -- more points of failure and increased setup/troubleshooting cost. In a ToR switch scenario, you do one config on one switch, plug in the cables, and you're done -- problems happen, you go to the one switch, not chasing a needle through a haystack of interconnected servers.
If your RU count is worth more than the combination of increased installation, server configuration, troubleshooting, latency, and capacity costs, then this is a good solution. Either way, it's a neat idea and a fun thought experiment to work through.
Thanks! Dan
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of NAOTO MATSUMOTO Sent: Wednesday, February 11, 2015 11:32 PM To: nanog@nanog.org Subject: FYI: An Easy way to build a server cluster without top of rack switches (MEMO)
Hi all!
We wrote up TIPS memo "an easy way to build a server cluster without top of rack switches" concept.
This model have a reduce switches and cables costs and high network durability by lightweight and simple configuration.
if you interest in, please try to do yourself this concept ;-)
An Easy way to build a server cluster without top of rack switches (MEMO) http://slidesha.re/1EduYXM
Best regards, -- Naoto MATSUMOTO