On 25/01/2012, at 20:51, nanog-request(a)nanog.org wrote:
> Message: 4
> Date: Wed, 25 Jan 2012 15:11:42 -0500 (EST)
> From: Jay Ashworth <jra(a)baylink.com>
> To: NANOG <nanog(a)nanog.org>
> Subject: Equinix Miami 1 condemnation
> Message-ID:
> <22786388.6606.1327522302701.JavaMail.root(a)benjamin.baylink.com>
> Content-Type: text/plain; charset=utf-8
>
> Last week, we saw some traffic about the Lightfiber problems because EqM1
> is apparently in a building that's been condemned by the city or county
> of Miami.
>
> That struck me curious, so I wanted to look into it further.
>
> Amazingly, there doesn't seem to be any coverage of the incident, even in
> technical circles. Does anyone know anything they're permitted to tell
> about how a building which contained a datacenter managed to get itself
> condemned?
> This was the Metro Mall facility, operated by Switch and Data until
> Equinix's acquisition of S&D a few years ago. It is now known as
> Equinix-Miami 1. 1 NE 1st Street.
>
And...
> The issue here has nothing to do with any hurricanes in the past. My
> opinion is that they lacked proper documentation and planning. Having a
> bit of insider knowledge here and knowing where this fiber came from
> originally (ACSI) and how FiberLight came to acquire it, I'm fairly sure
> I'm on the money.
I have no NDA on this so I can talk better in details.
Equinix Miami 1 (formerly Switch & Data, formerly another company name I don't recall now) was the temporary building that the NAP was built on, before moving to the current location that Terremark is (if not wrong, they called it the "T-Rex project" and the NOTA exchange started there in that place). Thus, (nearly) every major fiber strands that come from the submarine cable landing stations in SFl come into EQX MIA1 first, physically, before going into Terremark at 9th Street.
The building itself (Metromall) is not a datacenter by design, it's a normal commercial building. Some handfuls of jewelry stores, coffee shop, travel agency, a hair dresser, in the lobby and ground and basement levels. The loading dock of the building is on the same alley as Equinix Miami 2 (36 NE 2nd Street/Telesource building/Telx/Colohouse/Fibermedia/Savvis) loading dock entrance. Global Crossing's datacenter also was in the Metromall at another floor before going into Terremark's 6th floor. Equinix/S&D was on the 5th floor (whole floor). They were the biggest lease there, but they had no word on the building administration or maintenance, only on their own floor, that was pretty empty (no more than two or three dozen of customers). S&D considered Miami more of a strategic network PoP than a real customer datacenter (as does Savvis, that has a full floor in the 36 NE 2nd St building, but no customers, and doesn't announce it as a datacenter/colocation PoP even having all that square footage and huge generators), they had only a small number of customers and the two facilities (MIA1 and MIA2) were acquisitions of former small/broken telco companies. They were not built by S&D, less yet by Equinix.
The building failed the 40-year inspection by the city of Miami. The landlord/owner of the building decided not to fix the numerous issues that were presented by the inspectors, and then decided to just let it go, close everything, expel all stores (mostly were on monthly leases) with little to no notice, and maybe sell the building or the ruins to someone who wanted it to fix or to demolish.
Equinix (or better, Switch and Data) was in negotiations with a company I worked for as an outsourced partner, to build a pipe between both buildings, when they faced a long delay on approval by the landlord, When they pressed, they got the real reason why he was holding it. Last June Equinix started to evacuate the building preemptively, moving all customers to Miami 2 (it's in the same block, just two corners or a backdoor away) and paying their moving costs (materials, manpower, telco circuit movings) up to a ceiling of $10k (in service credits, not cash, power install was covered by Equinix). They did some preparation work on Miami 2 for receiving these customers - not many - and started that, with an ETA to move everybody out by October 2011.
The "secret" ingredient was more of a "let's not annouce it so no one tries to signup to Miami 1 and then take advantage of service credits", until the unavoidable was unavoidable and until Miami 2 was ready for deployment of these customers. Only the customers, partners and the Digital Realty Trust management (Miami 2 is a DRL building)/security were aware of that while the move was not completed or at least mass moved. Some competitors got to the knowledge after the customers were notified (in June 2011) and some customers leaked that information by accident or intentionally at that time.
AFAIK, Fiberlight spliced all Fiber on the manhole during 3 nights in July/August 2011, to reroute them out of MIA1. What happened these couple days ago was something completely different and probably the result of something they forgot to do on that manhole, or pure stupidity of doing maintenance/upgrades on both sides of the ring at the same time.
-- rc
