On Thu, Feb 18, 2021 at 8:31 AM Jared Mauch <jared@puck.nether.net> wrote:
On Thu, Feb 18, 2021 at 01:07:01AM -0800, Eric Kuhnke wrote:
> On that note, I'd be very interested in hearing stories of actual incidents
> that are the cause of why cardboard boxes are banned in many facilities,
> due to loose particulate matter getting into the air and setting off very
> sensitive fire detection systems.
>
> Or maybe it's more mundane and 99% of the reason is people unpack stuff and
> don't always clean up properly after themselves.

        We had a plastic bag sucked into the intake of a router in a
datacenter once that caused it to overheat and take the site down.  We
had cameras in our cage and I remember seeing the photo from the site of
the colo (I'll protect their name just because) taken as the tech was on
the phone and pulled the bag out of the router.

        The time from the thermal warning syslog that it's getting warm
to overheat and shutdown is short enough you can't really get a tech to
the cage in time to prevent it.


1: A previous employer was a large customer of a (now defunct) L3 switch vendor. The AC power inputs were along the bottom of the power supply, and the big aluminium heatsinks in the power supplies were just above the AC socket. 
Anyway, the subcontractor who made the power supplies for the vendor realized that they could save a few cents by not installing the little metal clip that held the heatsink to the MOSFET, and instead relying on the thermal adhesive to hold it...
This worked fine, until a certain number of hours had passed, at which point the goop would dry out and the heatsink would fall down, directly across the AC socket.... This would A: trip the circuit that this was on, but, more excitingly, set the aluminum on fire, which would then ignite the other heatsinks in the PSU, leading to much fire...

2: A somewhat similar thing would happen with the Ascend TNT Max, which had side-to-side airflow. These were dial termination boxes, and so people would install racks and racks of them. The first one would draw in cool air on the left, heat it up and ship it out the right. The next one over would draw in warm air on the left, heat it up further, and ship it out the right... Somewhere there is a fairly famous photo of a rack of TNT Maxes, with the final one literally on fire, and still passing packets. 
There is a related (and probably apocryphal) regarding the launch of the TNT. It was being shipped for a major trade-show, but got stuck in customs. After many bizarre calls with the customs folk, someone goes to the customs office to try and sort it out, and get greeted by custom agents with guns. They all walk into the warehouse, and discover that there is a large empty area around the crate, which is a wooden cube, with "TNT" stencilled in big red letters... 

3: I used to work for a small ISP in Yonkers, NY. We had a customer in Florida, and on a Friday morning their site goes down. We (of course) have not paid for Cisco 4 hour support (or, honestly, any support) and they have a strict SLA, so we are a little stuck.
We end up driving to JFK, and lugging a fully loaded Cisco 7507 to the check in counter. It was just before the last flight of the day, so we shrugged and said it was my checked bag. The excess baggage charges were eye-watering,  but it rode the conveyor belt with the rest of the luggage onto the plane. It arrived with just a bent  ejector handle, and the rest was fine.

4: Not too long after I started doing networking (and for the same small ISP in Yonkers), I'm flying off to install a new customer. I (of course) think that I'm hot stuff because I'm going to do the install, configure the router, whee, look at me! Anyway, I don't want to check a bag, and so I stuff the Cisco 2501 in a carryon bag, along with tools, etc (this was all pre-9/11!). I'm going through security and the TSA[0] person opens my bag and pulls the router out. "What's this?!" he asks. I politely tell him that it's a router. He says it's not. I'm still thinking that I'm the new hotness, and so I tell him in a somewhat condescending way that it is, and I know what I'm talking about. He tells me that it's not a router, and is starting to get annoyed. I explain using my "talking to a 5 year old" voice that it most certainly is a router. He tells me that lying to airport security is a federal offense, and starts looming at me. I adjust my attitude and start explaining that it's like a computer and makes the Internet work. He gruffly hands me back the router, I put it in my bag and scurry away. As I do so, I hear him telling his colleague that it wasn't a router, and that he certainly knows what a router is, because he does woodwork... 

5: Another one. In the early 2000s I was working for a dot-com boom company. We are building out our first datacenter, and I'm installing a pair of Cisco 7206s in 811 10th Ave. These will run basically the entire company, we have some transit, we have some peering to configure, we have an AS, etc. I'm going to be configuring all of this; clearly I'm a router-god...
Anyway, while I'm getting things configured, this janitor comes past, wheeling a garbage bin. He stops outside the cage and says "Whatcha doin'?". I go into this long explanation of how these "routers" <point> will connect to "the Internet" <wave hands in a big circle> to allow my "servers" <gesture at big black boxes with blinking lights> to talk to other "computers" <typing motion> on "the Internet" <again with the waving of the hands>. He pauses for a second, and says "'K. So, you doing a full iBGP mesh, or confeds?". I really hadn't intended to be a condescending ass, but I think of that every time I realize I might be assuming something about someone based on thier attire/job/etc.





W
[0]: Well, technically pre-TSA, but I cannot remember what we used to call airport security pre-TSA...

 

        I assume also the latter above, which is people have varying
definitons of clean.

        - Jared

--
Jared Mauch  | pgp key available via finger from jared@puck.nether.net
clue++;      | http://puck.nether.net/~jared/  My statements are only mine.


--
The computing scientist’s main challenge is not to get confused by the
complexities of his own making.
  -- E. W. Dijkstra