A few acquisitions ago, between 2006-2010, I worked at a Wireless ISP in Northern Indiana. Our CEO decided to sell Internet service to school systems because the e-rate funding was too much to resist. He had the idea to install towers on the schools and sell service off that while paying the school for roof rights. About two years into the endeavor, I wake up one morning and walk to my car. Two FBI agents get out of an unmarked towncar. About an hour later, they let me go to the office where I found an entire barrage of FBI agents. It was a full raid and not the kind you want to see. Hard drives were involved and being made redundant, but the redundant copies were labeled and placed into boxes that were carried out to SUVs that were as dark as the morning coffee these guys drank. There were a lot of drives, all of our servers were in our server room at the office. There were roughly five or six racks of varying amounts of equipment in each.
After some questioning and assisting them in their cataloging adventure, the agents left us with a ton of questions and just enough equipment to keep the customers connected. CEO became extremely paranoid at this point. He told us to prepare to move servers to a different building. He went into a tailspin trying to figure out where he could hide the servers to keep things going without the bank or FBI seizing the assets. He was extremely worried the bank would close the office down. We started moving all network routing around to avoid using the office as our primary DIA.
One morning I get into the office and we hear the words we've been dreading: "We're moving the servers". The plan was to move them to a tower site that had a decent-sized shack on site. Connectivity was decent, we had a licensed 11GHz microwave backhaul capable of about 155mbps. The site was part of the old MCI microwave long-distance network in the 80s and 90s. It had redundant air conditioners, a large propane tank, and a generator capable of keeping the site alive for about three days. We were told not to notify any customers, which became problematic because two customers had servers colocated in our building. We consolidated the servers into three racks and managed to get things prepared with a decent UPS in each rack. CEO decided to move the servers at nightfall to "avoid suspicion". Our office was in an unsavory part of town, moving anything at night was suspicious. So, under the cover of half-ass darkness, we loaded the racks onto a flatbed truck and drove them 20 minutes to the tower. While we unloaded the racks, an electrician we knew was wiring up the L5-20 outlets for the UPS in each rack. We got the racks plugged in, servers powered up, and then the two customers came that had colocated equipment. They got their equipment powered up and all seemed ok.
Back at the office the next day we were told to gather our workstations and start working from home. I've been working from home ever since and quite enjoy it, but that's beside the point.
Summer starts and I tell the CEO we need to repair the AC units because they are failing. He ignores it, claiming he doesn't want to lose money the bank could take at any minute. About a month later, a nice hot summer day rolls in and the AC units both die. I stumble upon an old portable AC unit and put that at the site. Temperatures rise to 140F ambient. Server overheat alarms start going off, things start failing. Our colocation customers are extremely upset. They pull their servers and drop service. The heat subsides, CEO finally pays to repair one of the AC units.
Eventually, the company declares bankruptcy and goes into liquidation. Luckily another WISP catches wind of it, buys the customers and assets, and hires me. My happiest day that year was moving all the servers into a better-suited home, a real data center. I don't know what happened to the CEO, but I know that I'll never trust anything he has his hands in ever again.
 |
Adam Kennedy
Systems Engineer
Watch Communications | www.watchcomm.net
3225 W Elm St, Suite A
Lima, OH 45805    |