Sorry for any confusion. Fiber channel is old tech for sure and there was no implied intent to evaluate the tech itself. Roxane has apparently observed that on devices like the core directors the number of fiber channel ports being ordered is going up dramatically.... apparently about the only thing it makes sense to do with such ports is to use them for SANs. The remark was not intended to be definitive or in great depth and was added almost as an after thought at the end of a short (35 minute) interview at the san jose airport. Certainly there is major movement in large fiber based in metropolitan areas that have been built and are being built by banks, and fortune 500 kinds of enterprises as they move a LOT of data and voice traffic off the PSTN and on to new cheap equipment that they can get a much better return for on the dollars they invest. All a part of the commoditization and decentralization of telecom. I am not an expert in the application of the technology being discussed. However at some time in the next six month i probably will want to explore this in much more depth.

...

Although these new real time applica tions will clearly send more data over the network, the real killer application is going to be remote storage and synchronous storage. Synchronous storage means that you have two large servers doing the exact same thing at the exact same time in two different locations.

COOK Report: Like a decentralized disk array?

Googin: Yes. The backbone has to be incredibly fast because you cannot complete a transaction until you have acknowledgments from both disk drives. This will happen. Probably this year. What they are already doing is taking fiber channel and putting that on a Cienna Core Director optical switch port. Half of the ports being sold on the Core Director now are fiber channel. They aren't even Ethernet. And this is used for storage area nets (SANs). These are corporate MANs and will have nothing to do with sales to service providers. They are bypass business services where the storage arrays may not be more than a kilometer or two apart. These SANs are backing up continuously terabytes of data. We are talking huge applications that will use every bit of access to every bit of capacity they can get.

Erm - perhaps I'm misunderstanding what Googin is trying to say here, but if he's talking about synchronous remote replication over fibre channel, this exists today. In fact, it has existed for years, and before it was over fibre channel, it was over ESCON. Today, still, if you want to go father than a certain distance (70km maybe?) its generally recommended to switch over to ESCON rather than fibre channel.

In practicality, complete remote replication is often inadvisable for high-performance, heavy-write applications. The rule of thumb is that every KM adds another millisecond worth of I/O transaction time, so a few kilometers distance can add a significant overhead to writes.

Not to mention Fibre Channel is very unkind when it comes to recovery from fabric segmentation. A poorly designed (or just very unlucky) SAN can be completely downed on both sides of the split, somewhat ruining the disaster recovery strategy if the production and DR storage networks are both taken down.

Ugh - I'll be very happy when fibre channel is dead and buried.

Thanks, Matt

-- Matthew Zito GridApp Systems Email: mzito@gridapp.com Cell: 646-220-3551 Phone: 212-358-8211 x 359

-- ============================================================= The COOK Report on Internet, 431 Greenway Ave, Ewing, NJ 08618 USA 609 882-2572 (PSTN) 703 738-6031 (Vonage) cook@cookreport.com Subscription info & prices at http://cookreport.com/subscriptions.shtml Fiber & Wireless as First Mile Technology - Fiber Business Models & Architecture, July- September 2003, 130 pages available at http://cookreport.com/12.04-06.shtml =============================================================

Non-flawless software (i.e., 100% of it) is usually dealt with storage snapshots; if they are kept local or remote, that's a design choice that a metro SAN can impact. Doing backup/restores thru a metro SAN also has some advantages on facilities with tight-controlled physical access; the bandwagon that moves tapes to off-site storage still has more bandwidth than the fiber, but it can also be used to smuggle <insert-your-favorite-paranoia-here>. Rubens ----- Original Message ----- From: "Petri Helenius" <pete@he.iki.fi> To: "Gordon Cook" <cook@cookreport.com> Cc: <Nanog@merit.edu> Sent: Monday, May 19, 2003 7:33 AM Subject: Re: An area for operations growth - Storage Area Nets in MANS | | | Although I agree that storage networking as a backup/tandem resiliency | operation makes a lot of sense, | it does not come anywhere nearly "free" because the assumptions made | there require flawless software. | If you always synchronize your systems you'll also blow them both out | when you hit a software issue | and as most of the audience is painfully aware, they are not infrequent | enough to count out of the equation. | | So you need to keep track how to re-synchornize after an | upgrade/failure/etc. With a large stream of | updated data on the system keeping the state difference becomes | expensive quickly. (unless you are going | to re-sync everything after a failure, and then you'll be vulnerable | until that's completed) | | Pointers to perfect SAN solutions appreciated. | | Pete | | | Gordon Cook wrote: | | > | > Roxane Googin is usually ahead of the competition in spotting trends | > in data net use. here is what has has to say about enterprise use of | > MANs. | > | > | > COOK Report: Where are the metro area networks going? | > | > Googin: Probably the killer application for the MAN is grid computing | > and storage area networks. A lot of people think the driver for | > broadband access to the home is going to be multi-media. People, when | > they think about next generation networks, think home use. But this | > is never where the money has been. And no phone company is ever going | > to build the infrastructure. | > | > Although these new real time applica tions will clearly send more data | > over the network, the real killer application is going to be remote | > storage and synchronous storage. Synchronous storage means that you | > have two large servers doing the exact same thing at the exact same | > time in two different locations. | > | > COOK Report: Like a decentralized disk array? | > | > Googin: Yes. The backbone has to be incredibly fast because you | > cannot complete a transaction until you have acknowledgments from both | > disk drives. This will happen. Probably this year. What they are | > already doing is taking fiber channel and putting that on a Cienna | > Core Director optical switch port. Half of the ports being sold on | > the Core Director now are fiber channel. They aren't even Ethernet. | > And this is used for storage area nets (SANs). These are corporate | > MANs and will have nothing to do with sales to service providers. | > They are bypass business services where the storage arrays may not be | > more than a kilometer or two apart. These SANs are backing up | > continuously terabytes of data. We are talking huge applications that | > will use every bit of access to every bit of capacity they can get. | > | > COOK Report: Is 9/11 a motivation for this? | > | > Googin: Partly. Not only that but the whole paradigm of the real time | > organization will drive it. It used to be that your server had its | > own storage. It was a "stove pipe" connected to the CPU. Now as soon | > as you decouple that "stove pipe," you can put it anywhere. What they | > are finding is that if they have two of them that are mirrored in real | > time and place remotely that they do not have to do as much management | > of resources. | > | > COOK Report: Then forget the disaster back up and recovery | > operations? Those are the next to be marked for extinction? | > | > Googin: Oh yes. It is so elegant. The new architecture cannot be | > supported on direct attached storage. It must move off the server and | > it is doing so. If you have to go through a server to get to the | > storage, it simply doesn't work. Storage can't become just a utility | > until you move it off the server. The MAN will be backing up | > terabytes of data regularly and in real time. Disaster recovery, in | > real time, comes for "free". | > | > For more detail please see | > | > Fiber & Wireless as First Mile Technology - Fiber Business Models & | > Architecture http://cookreport.com/12.04-06.shtml | | |