Joe Schmoe, At 01:19 PM 26/04/2004, Joe Schmoe wrote: [..]

So the question is, if you need to build a fiber SAN across several datacenters, do you have the ability to have a real fiber connection between all of them, or is it a virtual connection, where your fiber goes to their routers, which goes to fiber between their routers, etc., making several hops between each node of your SAN ?

firstly, if you're talking "SAN" as in "Fibre Channel", note that it won't go via "routers"; you'll find that FC needs a native transport that isn't encapsulated into IP (i.e. WDM or SONET transport). note that this may or may not be a good idea: Fibre Channel has lots of inherent limitations as distances increases. at the most basic level, there is something called Buffer Credits, which is analogous to Admission Control. once you've exhausted the BB_Credits, your maximum throughput will drop significantly. rule-of-thumb based on average-frame-size-distribution on FC is ~1 BB_Credit per kilometre @ 2G FC and ~1 BB_Credit per 2km @ 1G FC. if you DO wish to transport FC inside Fibre Channel, there are two paths to choose from: FCIP (Fibre Channel over IP) or iFCP. i'd recommend the former, but note that i will be biased in saying that -- i 'do' storage at Cisco/Andiamo as my day job. note that BB_Credits will likely not even end up being your bottleneck, but rather the fact that there are large chunks of both applications, filesystems and even SCSI that are simply synchronous in nature and slow down dramatically as latency increases. there are some modern techniques that can be used to counteract those (e.g. spoofing the SCSI R2T, reducing a SCSI Write from 2 round-trips to one), and even spoofing an entire SCSI Write locally (effectively turning sync into async), and you may want to add both compression and/or encryption to your storage traffic also...

This seems "bad" to me, because a router problem between the two nodes would essentially "break" the SAN - much like breaking a hardware RAID array by disconnecting a drive.

note that with FCIP, it is possible to engineer traffic in a way that failure of router/nodes won't cause a fabric disruption. FCIP is FC encapsulated over TCP, so one can engineer parallel FCIP links (in a port-channel for load-balancing if you wish) to go via separate physical paths & across diverse carriers if you wish. note that it is important to divorce the concept of what is something like 'mirroring' from 'replication'. it would be unlikely that you'd ever want to do 'RAID' across a geographically-dispersed SAN but its likely that you'd want to do 'replication'.

Comments ? How are people, from the point of view of the network, building fiber SANs, and what providers (L3 ?) are they using ?

Sprint did some testing. some details of that are at <http://www.cisco.com/warp/public/cc/pd/ps4159/ps4358/prodlit/sstcs_wp.pdf> i'm happy to talk more off-list, lest this violate nanog's terms of conditions. cheers, lincoln.