Certainly thankful no serious injuries or fatalities in this clusterblank. It seems the AT&T building at 185 2nd Ave N may have been a target, which would explain the timing (holiday morning when no one is out, as opposed to a holiday evening when there would be mass casualties). A little curious what that building has. Is it just a big co-lo place? Regional CLEC/ILEC? No earth-shattering revelations here. Admittedly just bored on a slow Christmas Day when my wife is at work (nurse) and kid is playing with a new tablet and I’m just watching the news trying to understand/figure out a little what and why. ---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com “Better even die free, than to live slaves.” - Frederick Douglas, 1863
The internet is buzzing with speculation about this. According to CNN the RV was at 2nd and Commerce st, which puts it 1-block away from the ATT building. If it were the target, I'd imagine they would have parked it closer. https://www.google.com/maps/search/2nd+and+congress+nashville/@36.1631367,-8... On Fri, Dec 25, 2020 at 11:20 AM Andy Ringsmuth <andy@andyring.com> wrote:
Certainly thankful no serious injuries or fatalities in this clusterblank.
It seems the AT&T building at 185 2nd Ave N may have been a target, which would explain the timing (holiday morning when no one is out, as opposed to a holiday evening when there would be mass casualties). A little curious what that building has. Is it just a big co-lo place? Regional CLEC/ILEC?
No earth-shattering revelations here. Admittedly just bored on a slow Christmas Day when my wife is at work (nurse) and kid is playing with a new tablet and I’m just watching the news trying to understand/figure out a little what and why.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
Definitely was not at that intersection. https://www.youtube.com/watch?v=5nygTJeu9fU That’s security camera footage from about 154 2nd Ave. The AT&T building is across the street to the right. Commerce Street is a block to the left. ---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com “Better even die free, than to live slaves.” - Frederick Douglas, 1863
On Dec 25, 2020, at 1:26 PM, cosmo <clinton.mielke@gmail.com> wrote:
The internet is buzzing with speculation about this. According to CNN the RV was at 2nd and Commerce st, which puts it 1-block away from the ATT building. If it were the target, I'd imagine they would have parked it closer.
https://www.google.com/maps/search/2nd+and+congress+nashville/@36.1631367,-8...
On Fri, Dec 25, 2020 at 11:20 AM Andy Ringsmuth <andy@andyring.com> wrote: Certainly thankful no serious injuries or fatalities in this clusterblank.
It seems the AT&T building at 185 2nd Ave N may have been a target, which would explain the timing (holiday morning when no one is out, as opposed to a holiday evening when there would be mass casualties). A little curious what that building has. Is it just a big co-lo place? Regional CLEC/ILEC?
No earth-shattering revelations here. Admittedly just bored on a slow Christmas Day when my wife is at work (nurse) and kid is playing with a new tablet and I’m just watching the news trying to understand/figure out a little what and why.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
I stand corrected! Maybe there is merit to this theory then. On Fri, Dec 25, 2020, 11:37 AM Andy Ringsmuth <andy@andyring.com> wrote:
Definitely was not at that intersection.
https://www.youtube.com/watch?v=5nygTJeu9fU
That’s security camera footage from about 154 2nd Ave. The AT&T building is across the street to the right.
Commerce Street is a block to the left.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
On Dec 25, 2020, at 1:26 PM, cosmo <clinton.mielke@gmail.com> wrote:
The internet is buzzing with speculation about this. According to CNN the RV was at 2nd and Commerce st, which puts it 1-block away from the ATT building. If it were the target, I'd imagine they would have parked it closer.
https://www.google.com/maps/search/2nd+and+congress+nashville/@36.1631367,-8...
On Fri, Dec 25, 2020 at 11:20 AM Andy Ringsmuth <andy@andyring.com>
Certainly thankful no serious injuries or fatalities in this clusterblank.
It seems the AT&T building at 185 2nd Ave N may have been a target, which would explain the timing (holiday morning when no one is out, as opposed to a holiday evening when there would be mass casualties). A little curious what that building has. Is it just a big co-lo place? Regional CLEC/ILEC?
No earth-shattering revelations here. Admittedly just bored on a slow Christmas Day when my wife is at work (nurse) and kid is playing with a new
wrote: tablet and I’m just watching the news trying to understand/figure out a little what and why.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
It seems to be here: https://www.google.com/maps/@36.1645601,-86.7768622,3a,60y,145.84h,88.19t/da... Here’s a link to a photo on Politico that matches: Note: Hooters sign on left.
On Dec 25, 2020, at 2:33 PM, Andy Ringsmuth <andy@andyring.com> wrote:
Definitely was not at that intersection.
https://www.youtube.com/watch?v=5nygTJeu9fU
That’s security camera footage from about 154 2nd Ave. The AT&T building is across the street to the right.
Commerce Street is a block to the left.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
On Dec 25, 2020, at 1:26 PM, cosmo <clinton.mielke@gmail.com> wrote:
The internet is buzzing with speculation about this. According to CNN the RV was at 2nd and Commerce st, which puts it 1-block away from the ATT building. If it were the target, I'd imagine they would have parked it closer.
https://www.google.com/maps/search/2nd+and+congress+nashville/@36.1631367,-8...
On Fri, Dec 25, 2020 at 11:20 AM Andy Ringsmuth <andy@andyring.com> wrote: Certainly thankful no serious injuries or fatalities in this clusterblank.
It seems the AT&T building at 185 2nd Ave N may have been a target, which would explain the timing (holiday morning when no one is out, as opposed to a holiday evening when there would be mass casualties). A little curious what that building has. Is it just a big co-lo place? Regional CLEC/ILEC?
No earth-shattering revelations here. Admittedly just bored on a slow Christmas Day when my wife is at work (nurse) and kid is playing with a new tablet and I’m just watching the news trying to understand/figure out a little what and why.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
Politico photo seems to have been filtered or dropped. 2nd Attempt: https://www.politico.com/news/2020/12/25/explosion-downtown-nashville-450448
On Dec 25, 2020, at 2:42 PM, Rodney Joffe <rjoffe@centergate.com> wrote:
It seems to be here:
https://www.google.com/maps/@36.1645601,-86.7768622,3a,60y,145.84h,88.19t/da...
Here’s a link to a photo on Politico that matches:
Note: Hooters sign on left.
On Dec 25, 2020, at 2:33 PM, Andy Ringsmuth <andy@andyring.com> wrote:
Definitely was not at that intersection.
https://www.youtube.com/watch?v=5nygTJeu9fU
That’s security camera footage from about 154 2nd Ave. The AT&T building is across the street to the right.
Commerce Street is a block to the left.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
On Dec 25, 2020, at 1:26 PM, cosmo <clinton.mielke@gmail.com> wrote:
The internet is buzzing with speculation about this. According to CNN the RV was at 2nd and Commerce st, which puts it 1-block away from the ATT building. If it were the target, I'd imagine they would have parked it closer.
https://www.google.com/maps/search/2nd+and+congress+nashville/@36.1631367,-8...
On Fri, Dec 25, 2020 at 11:20 AM Andy Ringsmuth <andy@andyring.com> wrote: Certainly thankful no serious injuries or fatalities in this clusterblank.
It seems the AT&T building at 185 2nd Ave N may have been a target, which would explain the timing (holiday morning when no one is out, as opposed to a holiday evening when there would be mass casualties). A little curious what that building has. Is it just a big co-lo place? Regional CLEC/ILEC?
No earth-shattering revelations here. Admittedly just bored on a slow Christmas Day when my wife is at work (nurse) and kid is playing with a new tablet and I’m just watching the news trying to understand/figure out a little what and why.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
During their press conference, the Nashville Metro PD put the RV at 166 2nd Ave N, which is across the street from the 185 2nd Ave N location. It's halfway up the block from 2nd & Commerce. On Fri, Dec 25, 2020 at 2:36 PM Andy Ringsmuth <andy@andyring.com> wrote:
Definitely was not at that intersection.
https://www.youtube.com/watch?v=5nygTJeu9fU
That’s security camera footage from about 154 2nd Ave. The AT&T building is across the street to the right.
Commerce Street is a block to the left.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
On Dec 25, 2020, at 1:26 PM, cosmo <clinton.mielke@gmail.com> wrote:
The internet is buzzing with speculation about this. According to CNN the RV was at 2nd and Commerce st, which puts it 1-block away from the ATT building. If it were the target, I'd imagine they would have parked it closer.
https://www.google.com/maps/search/2nd+and+congress+nashville/@36.1631367,-8...
On Fri, Dec 25, 2020 at 11:20 AM Andy Ringsmuth <andy@andyring.com>
Certainly thankful no serious injuries or fatalities in this clusterblank.
It seems the AT&T building at 185 2nd Ave N may have been a target, which would explain the timing (holiday morning when no one is out, as opposed to a holiday evening when there would be mass casualties). A little curious what that building has. Is it just a big co-lo place? Regional CLEC/ILEC?
No earth-shattering revelations here. Admittedly just bored on a slow Christmas Day when my wife is at work (nurse) and kid is playing with a new
wrote: tablet and I’m just watching the news trying to understand/figure out a little what and why.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
I see the logo now : https://www.google.com/maps/@36.1640386,-86.7765438,3a,34.8y,283.85h,92.88t/... Most amazingly, in some, *but not all*, Google street view shots, the ATT logo is completely blurred out! https://www.google.com/maps/@36.1641167,-86.7765941,3a,75y,243.72h,94.77t/da... On Fri, Dec 25, 2020 at 11:58 AM Matt Brennan <brennanma@gmail.com> wrote:
During their press conference, the Nashville Metro PD put the RV at 166 2nd Ave N, which is across the street from the 185 2nd Ave N location.
It's halfway up the block from 2nd & Commerce.
On Fri, Dec 25, 2020 at 2:36 PM Andy Ringsmuth <andy@andyring.com> wrote:
Definitely was not at that intersection.
https://www.youtube.com/watch?v=5nygTJeu9fU
That’s security camera footage from about 154 2nd Ave. The AT&T building is across the street to the right.
Commerce Street is a block to the left.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
On Dec 25, 2020, at 1:26 PM, cosmo <clinton.mielke@gmail.com> wrote:
The internet is buzzing with speculation about this. According to CNN the RV was at 2nd and Commerce st, which puts it 1-block away from the ATT building. If it were the target, I'd imagine they would have parked it closer.
https://www.google.com/maps/search/2nd+and+congress+nashville/@36.1631367,-8...
On Fri, Dec 25, 2020 at 11:20 AM Andy Ringsmuth <andy@andyring.com>
Certainly thankful no serious injuries or fatalities in this clusterblank.
It seems the AT&T building at 185 2nd Ave N may have been a target, which would explain the timing (holiday morning when no one is out, as opposed to a holiday evening when there would be mass casualties). A little curious what that building has. Is it just a big co-lo place? Regional CLEC/ILEC?
No earth-shattering revelations here. Admittedly just bored on a slow Christmas Day when my wife is at work (nurse) and kid is playing with a new
wrote: tablet and I’m just watching the news trying to understand/figure out a little what and why.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
That's not exceptionally uncommon. I have seen the same thing before in other sets of street view images. On Fri, Dec 25, 2020 at 3:15 PM cosmo <clinton.mielke@gmail.com> wrote:
I see the logo now :
https://www.google.com/maps/@36.1640386,-86.7765438,3a,34.8y,283.85h,92.88t/...
Most amazingly, in some, *but not all*, Google street view shots, the ATT logo is completely blurred out!
https://www.google.com/maps/@36.1641167,-86.7765941,3a,75y,243.72h,94.77t/da...
On Fri, Dec 25, 2020 at 11:58 AM Matt Brennan <brennanma@gmail.com> wrote:
During their press conference, the Nashville Metro PD put the RV at 166 2nd Ave N, which is across the street from the 185 2nd Ave N location.
It's halfway up the block from 2nd & Commerce.
On Fri, Dec 25, 2020 at 2:36 PM Andy Ringsmuth <andy@andyring.com> wrote:
Definitely was not at that intersection.
https://www.youtube.com/watch?v=5nygTJeu9fU
That’s security camera footage from about 154 2nd Ave. The AT&T building is across the street to the right.
Commerce Street is a block to the left.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
On Dec 25, 2020, at 1:26 PM, cosmo <clinton.mielke@gmail.com> wrote:
The internet is buzzing with speculation about this. According to CNN the RV was at 2nd and Commerce st, which puts it 1-block away from the ATT building. If it were the target, I'd imagine they would have parked it closer.
https://www.google.com/maps/search/2nd+and+congress+nashville/@36.1631367,-8...
On Fri, Dec 25, 2020 at 11:20 AM Andy Ringsmuth <andy@andyring.com>
Certainly thankful no serious injuries or fatalities in this clusterblank.
It seems the AT&T building at 185 2nd Ave N may have been a target, which would explain the timing (holiday morning when no one is out, as opposed to a holiday evening when there would be mass casualties). A little curious what that building has. Is it just a big co-lo place? Regional CLEC/ILEC?
No earth-shattering revelations here. Admittedly just bored on a slow Christmas Day when my wife is at work (nurse) and kid is playing with a new
wrote: tablet and I’m just watching the news trying to understand/figure out a little what and why.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
On Fri, 25 Dec 2020, cosmo wrote:
The internet is buzzing with speculation about this. According to CNN the RV was at 2nd and Commerce st, which puts it 1-block away from the ATT building. If it were the target, I'd imagine they would have parked it closer.
Folks are confusing two different AT&T buildings in Nashville. The AT&T office tower is the showpiece bulding (i.e. "Batman tower") at 333 Commerce St. The AT&T tandem office (i.e. typical windowless CO) is on Second Avenue, adjacent to where the explosion occurred. It is also called "Nashville Main" and the tandem office for central Tennessee. Additional outages are expected because the generators are not working, and the batteries are running down.
Ah-ha! Thanks for the correction. So the tandem office is the big red windowless building? On Fri, Dec 25, 2020 at 11:51 AM Sean Donelan <sean@donelan.com> wrote:
The internet is buzzing with speculation about this. According to CNN
On Fri, 25 Dec 2020, cosmo wrote: the RV
was at 2nd and Commerce st, which puts it 1-block away from the ATT building. If it were the target, I'd imagine they would have parked it closer.
Folks are confusing two different AT&T buildings in Nashville. The AT&T office tower is the showpiece bulding (i.e. "Batman tower") at 333 Commerce St.
The AT&T tandem office (i.e. typical windowless CO) is on Second Avenue, adjacent to where the explosion occurred. It is also called "Nashville Main" and the tandem office for central Tennessee.
Additional outages are expected because the generators are not working, and the batteries are running down.
AT&T statement regarding the intentional explosion in Nashville TN "Service for some customers in Nashville and the surrounding areas may be affected by damage to our facilities from the explosion this morning. We are in contact with law enforcement and working as quickly and safely as possible to restore service."
From local news reporting:
A widespread internet outage was reported in Nashville hours after a massive explosion downtown. AT&T internet and phone service was disrupted in the area about 12 p.m. Friday. A handful of local police departments reported the outage was disrupting 911 access, including some non-emergency lines, in their jurisdictions.
I think the outage is a bit more widespread than "Nashville and surrounding areas." Most (all?) of Kentucky is without AT&T cellular service right now. I can't say for sure of how many of AT&T's residential internet customers are affected, but reports on Twitter indicate it's a pretty significant chunk. I have AT&T ASE/metro services here in Kentucky that do not appear to be affected at this time. On Fri, Dec 25, 2020 at 2:33 PM Sean Donelan <sean@donelan.com> wrote:
AT&T statement regarding the intentional explosion in Nashville TN
"Service for some customers in Nashville and the surrounding areas may be affected by damage to our facilities from the explosion this morning. We are in contact with law enforcement and working as quickly and safely as possible to restore service."
From local news reporting:
A widespread internet outage was reported in Nashville hours after a massive explosion downtown. AT&T internet and phone service was disrupted in the area about 12 p.m. Friday.
A handful of local police departments reported the outage was disrupting 911 access, including some non-emergency lines, in their jurisdictions.
Can confirm internet service in Kentucky is being affected.
On Dec 25, 2020, at 3:33 PM, Josh Baird <joshbaird@gmail.com> wrote:
I think the outage is a bit more widespread than "Nashville and surrounding areas." Most (all?) of Kentucky is without AT&T cellular service right now.
I can't say for sure of how many of AT&T's residential internet customers are affected, but reports on Twitter indicate it's a pretty significant chunk. I have AT&T ASE/metro services here in Kentucky that do not appear to be affected at this time.
On Fri, Dec 25, 2020 at 2:33 PM Sean Donelan <sean@donelan.com> wrote:
AT&T statement regarding the intentional explosion in Nashville TN
"Service for some customers in Nashville and the surrounding areas may be affected by damage to our facilities from the explosion this morning. We are in contact with law enforcement and working as quickly and safely as possible to restore service."
From local news reporting:
A widespread internet outage was reported in Nashville hours after a massive explosion downtown. AT&T internet and phone service was disrupted in the area about 12 p.m. Friday.
A handful of local police departments reported the outage was disrupting 911 access, including some non-emergency lines, in their jurisdictions.
I can confirm people in Alabama have outages as well. Reports from boots on the ground in the Nashville area. Comcast / Xfinity seem to be ok. Verizon seems to be ok also. Not sure the logistics. On Fri, Dec 25, 2020 at 3:38 PM Matt Hoppes < mattlists@rivervalleyinternet.net> wrote:
Can confirm internet service in Kentucky is being affected.
On Dec 25, 2020, at 3:33 PM, Josh Baird <joshbaird@gmail.com> wrote:
I think the outage is a bit more widespread than "Nashville and surrounding areas." Most (all?) of Kentucky is without AT&T cellular service right now.
I can't say for sure of how many of AT&T's residential internet customers are affected, but reports on Twitter indicate it's a pretty significant chunk. I have AT&T ASE/metro services here in Kentucky that do not appear to be affected at this time.
On Fri, Dec 25, 2020 at 2:33 PM Sean Donelan <sean@donelan.com> wrote:
AT&T statement regarding the intentional explosion in Nashville TN
"Service for some customers in Nashville and the surrounding areas may be affected by damage to our facilities from the explosion this morning. We are in contact with law enforcement and working as quickly and safely as possible to restore service."
From local news reporting:
A widespread internet outage was reported in Nashville hours after a massive explosion downtown. AT&T internet and phone service was disrupted in the area about 12 p.m. Friday.
A handful of local police departments reported the outage was disrupting 911 access, including some non-emergency lines, in their jurisdictions.
Additional AT&T statement (5 pm CST) https://about.att.com/pages/disaster_relief/nashville.html We’re putting the full-force of our disaster recovery efforts into responding to this morning’s explosion in Nashville, including bringing in regional resources and our National Disaster Recovery teams. Power is essential to restoring wireless and wireline communications and we are working with law enforcement to get access to our equipment and make needed repairs. Given the damage to our facility it will take time to restore service. We have already rerouted significant traffic from this facility and are bringing in other equipment, including numerous portable cell sites to the area. There are serious logistical challenges to working in a disaster area and we will make measurable progress in the hours and days ahead. We're grateful for the work of law enforcement as they investigate this event while enabling us to restore service for our customers. We'll continue to provide updates here as our recovery progresses.
AT&T statement says nearly all services have been restore in Nashville as of Monday, 5pm CST They are working on permanent repairs. https://about.att.com/pages/disaster_relief/nashville.html AT&T's Network Disaster Recovery group faces management questions nearly every year to justifying their budget. While no one wants disasters, business continuity has to be part of the business. There are also mutual aid agreements between companies, but I don't know how many were invoked for this incident. https://about.att.com/ecms/dam/pages/disaster_relief/NDR_edited_04.22.19.pdf
From a few days ago. Obviously centralizing lots of ss7/pstn stuff all in one place has a long recovery time when it's physically damaged. Something to think about for entities that own and operate traditional telco COs and their plans for disaster recovery.
Nv1 Here is the latest update: 6:46AM 12/27: Work continues restoring service to the CRS routers in the Nashville Central Office. One router remains out of service and the other is in service with some links remaining out of service. The working bridge will reconvene at 08:00 CT with the following action plan: Additional cabling added to the first portable generator to enable full load capabilities (08:00 CT) Pigtails with camlocks installed for easy swap; investigate possibility to land generator on the emergency service board to give the site N+1 with a manual ability to choose anyone. (08:00 CT) check small power plants on floors 4 and 6 (08:00 CT) Investigate water damage on 1st floor and energize if safe (08:00 CT Air handlers for floors 4,5 and 6 (09:00 CT) complete all transport work Turn up SS7 Turn up 911 service - Approximately noon or after) Turn up switching service. TDM Switching team will reconvene at 09:00 CT and the Signaling team will reconvene at 11:00 CT on 12/27/2020. DMS equipment on the 1st floor will be assessed for water damage. Switching teams will monitor power and HVAC restoration and will begin switch restoration as soon as the go ahead is provided by the power team. Recovery Priorities: 1. 4th & 5th floors (Specify transport equipment needed to clear MTSO SS7 isolation & Datakit needed for Local Switch restoration). Transport SMEs currently working to turn up transport equipment 2. 6th floor (ESINET Groomers) 3. 10th and 8th floors (N4E) – Trunks 4. 1st floor (DMS: DS1, 5E: DS3) - Local POTS 5. 1st floor (DMS: DS0, DS2 | 5E: DS6) – Trunks 6. 11th floor (DMS: 01T) – Trunks 7. 4th floor (STP and SCP with mates up in Donelson) The next update will be issued at approximately 09:00 CT on December 27. Nv2 As of 09:00 CT: Teams worked through the night to restore service and improve conditions at the Nashville 2nd Ave Central Office. Since the initial service impact, over 75% of the Out of Service Mobility Sites have been restored. Certain call flows may be limited and should improve as additional restoration activities complete. The generator that is currently powering equipment on the 2nd and 3rd floor, was refueled and ran with no issues through the night. Overnight, the batteries connected to it, continued to charge. Teams have placed additional power cables, which once connected, will allow the working generator, to better handle the load in the building. In order to accomplish this, the generator will need to be shut down for 15-30 minutes this morning, so teams can connect the new cables to the system. The power team reports they are still on target to restore power and cooling to the 5th and 6th floor by approximately 12:00 CT. Also, a portable chiller will be delivered this morning and strategically placed, in case it is needed to assist in cooling the office. There is a Call Center at 333 Commerce, in Nashville that does not have network or phone services available. Corporate Real Estate (CRE) reports there is some damage to that office, but the extent of the damage will not be known until they can gain access to the site. Because of this, the impacted Call Center ceased operations until further notice. DMS switching equipment on the 1st floor will be assessed for water damage. Switching teams will monitor power and HVAC restoration. Equipment power ups will begin, as soon as the go ahead is provided by the power team. Two SatCOLTs remain positioned on the East and West sides of the NSVLTNMT Central Office providing critical communication for teams working restoration efforts. There are 17 assets deployed in the field- 15 are on air (the 2 at the CO and 13 supporting FN Customer Requests) and 2 are in hot-standby for FN Customers where macro service recently recovered. There is 1 asset staged at a deployment site in KY where macro service restored, and 8 additional assets are on route to Nashville today to fulfill pending FN Customer requests. Incoming requests continue to be triaged. The ones in areas where service looks to have been restored, are being held, while the others are being prioritized to be dispatched upon. The next update will be issued at approximately 14:00 CT, unless there is a significant change in status. Nv3 AT&T Nashville update below, received at 3:35PM 12/27. Since the initial service impact, over 95% of the Out of Service Mobility Sites have been restored. Certain call flows may be limited and should improve as additional restoration activities complete. Electricians have installed the additional power cables from the generator, to the emergency bus. These new cables will allow the generator to support more of the load, of the building. The portable chiller requested, has arrived on-site, and is available to assist in cooling, if needed. Generally speaking, there are four (4) phases of restoration per floor (Air Handler restoral, Power restoral, Transport Equipment restoral, and Switch/Application Equipment restoral). Teams report that Air Handlers are up and running, and all power plants are on floors 2 through 7 are online. Given significant progress made, floors 2 through 7, are ready for technology turn up. Relative to Priority Transport related equipment, approximately 90% of the elements have been turned up on floors 2 through 7. The Power team is currently working on Floors 8 through 11 (N4E). The first floor is not accessible, at this time. Once access is granted by federal and local authorities, further assessment and restoration efforts will begin. The generator is currently supporting approximately 50% of its capacity, and alternative plans are being considered to handle the full load of the building. Teams continue to work proactively in effort to identify potential issues and are actively engaged working to restore services and repair infrastructure. AT&T Network Disaster Recovery (NDR) has eleven (11) SatCOLTs in service (TN, AL, GA). Two (2) of the eleven (11) are deployed at the Nashville, TN Central Office to provide coverage for the AT&T response teams as well as FirstNet (FN) customers. One (1) COLT is in hot-standby (TN). Six (6) COLTs are en-route to deployment sites in TN and AL. Three (3) COLTs are being demobilized in Alabama and coming back to Nashville for new assignments and five (5) additional COLTs are en-route to the Nashville area to support additional requests. The next update will be issued at approximately 19:00 CT, unless there is a significant change in status. On Mon, Dec 28, 2020, 5:59 PM Sean Donelan <sean@donelan.com> wrote:
AT&T statement says nearly all services have been restore in Nashville as of Monday, 5pm CST
They are working on permanent repairs.
https://about.att.com/pages/disaster_relief/nashville.html
AT&T's Network Disaster Recovery group faces management questions nearly every year to justifying their budget. While no one wants disasters, business continuity has to be part of the business. There are also mutual aid agreements between companies, but I don't know how many were invoked for this incident.
https://about.att.com/ecms/dam/pages/disaster_relief/NDR_edited_04.22.19.pdf
AT&T Disaster Recovery Team is probably the best in the business. The resources they can bring to the table are unmatched. This would have been 100x worse if it hit a carrier neutral datacenter. They don’t have nearly the same resources to restore something like this. They usually do a road show (pre Covid). If you get a chance it’s definitely something you should go check out. Very impressive. Robert DeVita Founder & CEO Mejeticks c. 469-441-8864 e. radevita@mejeticks.com ________________________________ From: NANOG <nanog-bounces+radevita=mejeticks.com@nanog.org> on behalf of Eric Kuhnke <eric.kuhnke@gmail.com> Sent: Tuesday, December 29, 2020 5:06:00 PM To: Sean Donelan <sean@donelan.com> Cc: NANOG <nanog@nanog.org> Subject: Re: Nashville
From a few days ago. Obviously centralizing lots of ss7/pstn stuff all in one place has a long recovery time when it's physically damaged. Something to think about for entities that own and operate traditional telco COs and their plans for disaster recovery.
Nv1 Here is the latest update: 6:46AM 12/27: Work continues restoring service to the CRS routers in the Nashville Central Office. One router remains out of service and the other is in service with some links remaining out of service. The working bridge will reconvene at 08:00 CT with the following action plan: Additional cabling added to the first portable generator to enable full load capabilities (08:00 CT) Pigtails with camlocks installed for easy swap; investigate possibility to land generator on the emergency service board to give the site N+1 with a manual ability to choose anyone. (08:00 CT) check small power plants on floors 4 and 6 (08:00 CT) Investigate water damage on 1st floor and energize if safe (08:00 CT Air handlers for floors 4,5 and 6 (09:00 CT) complete all transport work Turn up SS7 Turn up 911 service - Approximately noon or after) Turn up switching service. TDM Switching team will reconvene at 09:00 CT and the Signaling team will reconvene at 11:00 CT on 12/27/2020. DMS equipment on the 1st floor will be assessed for water damage. Switching teams will monitor power and HVAC restoration and will begin switch restoration as soon as the go ahead is provided by the power team. Recovery Priorities: 1. 4th & 5th floors (Specify transport equipment needed to clear MTSO SS7 isolation & Datakit needed for Local Switch restoration). Transport SMEs currently working to turn up transport equipment 2. 6th floor (ESINET Groomers) 3. 10th and 8th floors (N4E) – Trunks 4. 1st floor (DMS: DS1, 5E: DS3) - Local POTS 5. 1st floor (DMS: DS0, DS2 | 5E: DS6) – Trunks 6. 11th floor (DMS: 01T) – Trunks 7. 4th floor (STP and SCP with mates up in Donelson) The next update will be issued at approximately 09:00 CT on December 27. Nv2 As of 09:00 CT: Teams worked through the night to restore service and improve conditions at the Nashville 2nd Ave Central Office. Since the initial service impact, over 75% of the Out of Service Mobility Sites have been restored. Certain call flows may be limited and should improve as additional restoration activities complete. The generator that is currently powering equipment on the 2nd and 3rd floor, was refueled and ran with no issues through the night. Overnight, the batteries connected to it, continued to charge. Teams have placed additional power cables, which once connected, will allow the working generator, to better handle the load in the building. In order to accomplish this, the generator will need to be shut down for 15-30 minutes this morning, so teams can connect the new cables to the system. The power team reports they are still on target to restore power and cooling to the 5th and 6th floor by approximately 12:00 CT. Also, a portable chiller will be delivered this morning and strategically placed, in case it is needed to assist in cooling the office. There is a Call Center at 333 Commerce, in Nashville that does not have network or phone services available. Corporate Real Estate (CRE) reports there is some damage to that office, but the extent of the damage will not be known until they can gain access to the site. Because of this, the impacted Call Center ceased operations until further notice. DMS switching equipment on the 1st floor will be assessed for water damage. Switching teams will monitor power and HVAC restoration. Equipment power ups will begin, as soon as the go ahead is provided by the power team. Two SatCOLTs remain positioned on the East and West sides of the NSVLTNMT Central Office providing critical communication for teams working restoration efforts. There are 17 assets deployed in the field- 15 are on air (the 2 at the CO and 13 supporting FN Customer Requests) and 2 are in hot-standby for FN Customers where macro service recently recovered. There is 1 asset staged at a deployment site in KY where macro service restored, and 8 additional assets are on route to Nashville today to fulfill pending FN Customer requests. Incoming requests continue to be triaged. The ones in areas where service looks to have been restored, are being held, while the others are being prioritized to be dispatched upon. The next update will be issued at approximately 14:00 CT, unless there is a significant change in status. Nv3 AT&T Nashville update below, received at 3:35PM 12/27. Since the initial service impact, over 95% of the Out of Service Mobility Sites have been restored. Certain call flows may be limited and should improve as additional restoration activities complete. Electricians have installed the additional power cables from the generator, to the emergency bus. These new cables will allow the generator to support more of the load, of the building. The portable chiller requested, has arrived on-site, and is available to assist in cooling, if needed. Generally speaking, there are four (4) phases of restoration per floor (Air Handler restoral, Power restoral, Transport Equipment restoral, and Switch/Application Equipment restoral). Teams report that Air Handlers are up and running, and all power plants are on floors 2 through 7 are online. Given significant progress made, floors 2 through 7, are ready for technology turn up. Relative to Priority Transport related equipment, approximately 90% of the elements have been turned up on floors 2 through 7. The Power team is currently working on Floors 8 through 11 (N4E). The first floor is not accessible, at this time. Once access is granted by federal and local authorities, further assessment and restoration efforts will begin. The generator is currently supporting approximately 50% of its capacity, and alternative plans are being considered to handle the full load of the building. Teams continue to work proactively in effort to identify potential issues and are actively engaged working to restore services and repair infrastructure. AT&T Network Disaster Recovery (NDR) has eleven (11) SatCOLTs in service (TN, AL, GA). Two (2) of the eleven (11) are deployed at the Nashville, TN Central Office to provide coverage for the AT&T response teams as well as FirstNet (FN) customers. One (1) COLT is in hot-standby (TN). Six (6) COLTs are en-route to deployment sites in TN and AL. Three (3) COLTs are being demobilized in Alabama and coming back to Nashville for new assignments and five (5) additional COLTs are en-route to the Nashville area to support additional requests. The next update will be issued at approximately 19:00 CT, unless there is a significant change in status. On Mon, Dec 28, 2020, 5:59 PM Sean Donelan < sean@donelan.com <mailto:sean@donelan.com> > wrote: AT&T statement says nearly all services have been restore in Nashville as of Monday, 5pm CST They are working on permanent repairs. https://about.att.com/pages/disaster_relief/nashville.html <https://mila.bitdam.com/api/v1.0/links/rewrite_click/?rewrite_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyZXdyaXRlX2lkIjoiNWZlYmI3MGZiOGZhNWRhYjg5ZjE0NzJiIiwidXJsIjoiIiwib3JnYW5pemF0aW9uX2lkIjoxNzgxfQ.fQSGLDVfXTrVFYepbx0AR_uG_qKj-UG337RhC567WBI&url=https%3A//about.att.com/pages/disaster_relief/nashville.html> AT&T's Network Disaster Recovery group faces management questions nearly every year to justifying their budget. While no one wants disasters, business continuity has to be part of the business. There are also mutual aid agreements between companies, but I don't know how many were invoked for this incident. https://about.att.com/ecms/dam/pages/disaster_relief/NDR_edited_04.22.19.pdf <https://mila.bitdam.com/api/v1.0/links/rewrite_click/?rewrite_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyZXdyaXRlX2lkIjoiNWZlYmI3MGZlZTIwYTJmMzM5ZGM5ZTgwIiwidXJsIjoiIiwib3JnYW5pemF0aW9uX2lkIjoxNzgxfQ.xIE6w200hxqs5MLmTyAw3TKHl50TG9NaxjGW32JTEr8&url=https%3A//about.att.com/ecms/dam/pages/disaster_relief/NDR_edited_04.22.19.pdf>
Is there a video of this? I would also love to see pictures of what the damage was inside the building and repairs. Not sure if that was documented anywhere. I would assume they are still doing repairs and upgrades to the facility. On Tue, Dec 29, 2020 at 8:18 PM Robert DeVita <radevita@mejeticks.com> wrote:
AT&T Disaster Recovery Team is probably the best in the business. The resources they can bring to the table are unmatched. This would have been 100x worse if it hit a carrier neutral datacenter. They don’t have nearly the same resources to restore something like this. They usually do a road show (pre Covid). If you get a chance it’s definitely something you should go check out. Very impressive.
Robert DeVita Founder & CEO Mejeticks c. 469-441-8864 e. radevita@mejeticks.com ------------------------------ *From:* NANOG <nanog-bounces+radevita=mejeticks.com@nanog.org> on behalf of Eric Kuhnke <eric.kuhnke@gmail.com> *Sent:* Tuesday, December 29, 2020 5:06:00 PM *To:* Sean Donelan <sean@donelan.com> *Cc:* NANOG <nanog@nanog.org> *Subject:* Re: Nashville
From a few days ago. Obviously centralizing lots of ss7/pstn stuff all in one place has a long recovery time when it's physically damaged. Something to think about for entities that own and operate traditional telco COs and their plans for disaster recovery.
Nv1
Here is the latest update: 6:46AM 12/27:
Work continues restoring service to the CRS routers in the Nashville Central Office. One router remains out of service and the other is in service with some links remaining out of service.
The working bridge will reconvene at 08:00 CT with the following action plan: Additional cabling added to the first portable generator to enable full load capabilities (08:00 CT) Pigtails with camlocks installed for easy swap; investigate possibility to land generator on the emergency service board to give the site N+1 with a manual ability to choose anyone. (08:00 CT) check small power plants on floors 4 and 6 (08:00 CT) Investigate water damage on 1st floor and energize if safe (08:00 CT Air handlers for floors 4,5 and 6 (09:00 CT) complete all transport work Turn up SS7 Turn up 911 service - Approximately noon or after) Turn up switching service. TDM Switching team will reconvene at 09:00 CT and the Signaling team will reconvene at 11:00 CT on 12/27/2020. DMS equipment on the 1st floor will be assessed for water damage. Switching teams will monitor power and HVAC restoration and will begin switch restoration as soon as the go ahead is provided by the power team.
Recovery Priorities: 1. 4th & 5th floors (Specify transport equipment needed to clear MTSO SS7 isolation & Datakit needed for Local Switch restoration). Transport SMEs currently working to turn up transport equipment 2. 6th floor (ESINET Groomers) 3. 10th and 8th floors (N4E) – Trunks 4. 1st floor (DMS: DS1, 5E: DS3) - Local POTS 5. 1st floor (DMS: DS0, DS2 | 5E: DS6) – Trunks 6. 11th floor (DMS: 01T) – Trunks 7. 4th floor (STP and SCP with mates up in Donelson)
The next update will be issued at approximately 09:00 CT on December 27.
Nv2
As of 09:00 CT: Teams worked through the night to restore service and improve conditions at the Nashville 2nd Ave Central Office. Since the initial service impact, over 75% of the Out of Service Mobility Sites have been restored. Certain call flows may be limited and should improve as additional restoration activities complete. The generator that is currently powering equipment on the 2nd and 3rd floor, was refueled and ran with no issues through the night. Overnight, the batteries connected to it, continued to charge. Teams have placed additional power cables, which once connected, will allow the working generator, to better handle the load in the building. In order to accomplish this, the generator will need to be shut down for 15-30 minutes this morning, so teams can connect the new cables to the system. The power team reports they are still on target to restore power and cooling to the 5th and 6th floor by approximately 12:00 CT. Also, a portable chiller will be delivered this morning and strategically placed, in case it is needed to assist in cooling the office. There is a Call Center at 333 Commerce, in Nashville that does not have network or phone services available. Corporate Real Estate (CRE) reports there is some damage to that office, but the extent of the damage will not be known until they can gain access to the site. Because of this, the impacted Call Center ceased operations until further notice. DMS switching equipment on the 1st floor will be assessed for water damage. Switching teams will monitor power and HVAC restoration. Equipment power ups will begin, as soon as the go ahead is provided by the power team. Two SatCOLTs remain positioned on the East and West sides of the NSVLTNMT Central Office providing critical communication for teams working restoration efforts. There are 17 assets deployed in the field- 15 are on air (the 2 at the CO and 13 supporting FN Customer Requests) and 2 are in hot-standby for FN Customers where macro service recently recovered. There is 1 asset staged at a deployment site in KY where macro service restored, and 8 additional assets are on route to Nashville today to fulfill pending FN Customer requests. Incoming requests continue to be triaged. The ones in areas where service looks to have been restored, are being held, while the others are being prioritized to be dispatched upon.
The next update will be issued at approximately 14:00 CT, unless there is a significant change in status.
Nv3
AT&T Nashville update below, received at 3:35PM 12/27.
Since the initial service impact, over 95% of the Out of Service Mobility Sites have been restored. Certain call flows may be limited and should improve as additional restoration activities complete.
Electricians have installed the additional power cables from the generator, to the emergency bus. These new cables will allow the generator to support more of the load, of the building. The portable chiller requested, has arrived on-site, and is available to assist in cooling, if needed. Generally speaking, there are four (4) phases of restoration per floor (Air Handler restoral, Power restoral, Transport Equipment restoral, and Switch/Application Equipment restoral). Teams report that Air Handlers are up and running, and all power plants are on floors 2 through 7 are online. Given significant progress made, floors 2 through 7, are ready for technology turn up. Relative to Priority Transport related equipment, approximately 90% of the elements have been turned up on floors 2 through 7. The Power team is currently working on Floors 8 through 11 (N4E). The first floor is not accessible, at this time. Once access is granted by federal and local authorities, further assessment and restoration efforts will begin.
The generator is currently supporting approximately 50% of its capacity, and alternative plans are being considered to handle the full load of the building. Teams continue to work proactively in effort to identify potential issues and are actively engaged working to restore services and repair infrastructure.
AT&T Network Disaster Recovery (NDR) has eleven (11) SatCOLTs in service (TN, AL, GA). Two (2) of the eleven (11) are deployed at the Nashville, TN Central Office to provide coverage for the AT&T response teams as well as FirstNet (FN) customers. One (1) COLT is in hot-standby (TN). Six (6) COLTs are en-route to deployment sites in TN and AL. Three (3) COLTs are being demobilized in Alabama and coming back to Nashville for new assignments and five (5) additional COLTs are en-route to the Nashville area to support additional requests.
The next update will be issued at approximately 19:00 CT, unless there is a significant change in status.
On Mon, Dec 28, 2020, 5:59 PM Sean Donelan < sean@donelan.com > wrote:
AT&T statement says nearly all services have been restore in Nashville as of Monday, 5pm CST
They are working on permanent repairs.
https://about.att.com/pages/disaster_relief/nashville.html <https://mila.bitdam.com/api/v1.0/links/rewrite_click/?rewrite_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyZXdyaXRlX2lkIjoiNWZlYmI3MGZiOGZhNWRhYjg5ZjE0NzJiIiwidXJsIjoiIiwib3JnYW5pemF0aW9uX2lkIjoxNzgxfQ.fQSGLDVfXTrVFYepbx0AR_uG_qKj-UG337RhC567WBI&url=https%3A//about.att.com/pages/disaster_relief/nashville.html>
AT&T's Network Disaster Recovery group faces management questions nearly every year to justifying their budget. While no one wants disasters, business continuity has to be part of the business. There are also mutual aid agreements between companies, but I don't know how many were invoked for this incident.
https://about.att.com/ecms/dam/pages/disaster_relief/NDR_edited_04.22.19.pdf
No doubt they're good, but the best support can't overcome bad design. From: NANOG <nanog-bounces+david.hiers=cdk.com@nanog.org> On Behalf Of Robert DeVita Sent: Tuesday, December 29, 2020 5:17 PM To: Eric Kuhnke <eric.kuhnke@gmail.com>; Sean Donelan <sean@donelan.com> Cc: NANOG <nanog@nanog.org> Subject: Re: Nashville AT&T Disaster Recovery Team is probably the best in the business. The resources they can bring to the table are unmatched. This would have been 100x worse if it hit a carrier neutral datacenter. They don't have nearly the same resources to restore something like this. They usually do a road show (pre Covid). If you get a chance it's definitely something you should go check out. Very impressive. Robert DeVita Founder & CEO Mejeticks c. 469-441-8864 e. radevita@mejeticks.com<mailto:radevita@mejeticks.com> ________________________________ From: NANOG <nanog-bounces+radevita=mejeticks.com@nanog.org<mailto:nanog-bounces+radevita=mejeticks.com@nanog.org>> on behalf of Eric Kuhnke <eric.kuhnke@gmail.com<mailto:eric.kuhnke@gmail.com>> Sent: Tuesday, December 29, 2020 5:06:00 PM To: Sean Donelan <sean@donelan.com<mailto:sean@donelan.com>> Cc: NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Nashville
From a few days ago. Obviously centralizing lots of ss7/pstn stuff all in one place has a long recovery time when it's physically damaged. Something to think about for entities that own and operate traditional telco COs and their plans for disaster recovery.
Nv1 Here is the latest update: 6:46AM 12/27: Work continues restoring service to the CRS routers in the Nashville Central Office. One router remains out of service and the other is in service with some links remaining out of service. The working bridge will reconvene at 08:00 CT with the following action plan: Additional cabling added to the first portable generator to enable full load capabilities (08:00 CT) Pigtails with camlocks installed for easy swap; investigate possibility to land generator on the emergency service board to give the site N+1 with a manual ability to choose anyone. (08:00 CT) check small power plants on floors 4 and 6 (08:00 CT) Investigate water damage on 1st floor and energize if safe (08:00 CT Air handlers for floors 4,5 and 6 (09:00 CT) complete all transport work Turn up SS7 Turn up 911 service - Approximately noon or after) Turn up switching service. TDM Switching team will reconvene at 09:00 CT and the Signaling team will reconvene at 11:00 CT on 12/27/2020. DMS equipment on the 1st floor will be assessed for water damage. Switching teams will monitor power and HVAC restoration and will begin switch restoration as soon as the go ahead is provided by the power team. Recovery Priorities: 1. 4th & 5th floors (Specify transport equipment needed to clear MTSO SS7 isolation & Datakit needed for Local Switch restoration). Transport SMEs currently working to turn up transport equipment 2. 6th floor (ESINET Groomers) 3. 10th and 8th floors (N4E) - Trunks 4. 1st floor (DMS: DS1, 5E: DS3) - Local POTS 5. 1st floor (DMS: DS0, DS2 | 5E: DS6) - Trunks 6. 11th floor (DMS: 01T) - Trunks 7. 4th floor (STP and SCP with mates up in Donelson) The next update will be issued at approximately 09:00 CT on December 27. Nv2 As of 09:00 CT: Teams worked through the night to restore service and improve conditions at the Nashville 2nd Ave Central Office. Since the initial service impact, over 75% of the Out of Service Mobility Sites have been restored. Certain call flows may be limited and should improve as additional restoration activities complete. The generator that is currently powering equipment on the 2nd and 3rd floor, was refueled and ran with no issues through the night. Overnight, the batteries connected to it, continued to charge. Teams have placed additional power cables, which once connected, will allow the working generator, to better handle the load in the building. In order to accomplish this, the generator will need to be shut down for 15-30 minutes this morning, so teams can connect the new cables to the system. The power team reports they are still on target to restore power and cooling to the 5th and 6th floor by approximately 12:00 CT. Also, a portable chiller will be delivered this morning and strategically placed, in case it is needed to assist in cooling the office. There is a Call Center at 333 Commerce, in Nashville that does not have network or phone services available. Corporate Real Estate (CRE) reports there is some damage to that office, but the extent of the damage will not be known until they can gain access to the site. Because of this, the impacted Call Center ceased operations until further notice. DMS switching equipment on the 1st floor will be assessed for water damage. Switching teams will monitor power and HVAC restoration. Equipment power ups will begin, as soon as the go ahead is provided by the power team. Two SatCOLTs remain positioned on the East and West sides of the NSVLTNMT Central Office providing critical communication for teams working restoration efforts. There are 17 assets deployed in the field- 15 are on air (the 2 at the CO and 13 supporting FN Customer Requests) and 2 are in hot-standby for FN Customers where macro service recently recovered. There is 1 asset staged at a deployment site in KY where macro service restored, and 8 additional assets are on route to Nashville today to fulfill pending FN Customer requests. Incoming requests continue to be triaged. The ones in areas where service looks to have been restored, are being held, while the others are being prioritized to be dispatched upon. The next update will be issued at approximately 14:00 CT, unless there is a significant change in status. Nv3 AT&T Nashville update below, received at 3:35PM 12/27. Since the initial service impact, over 95% of the Out of Service Mobility Sites have been restored. Certain call flows may be limited and should improve as additional restoration activities complete. Electricians have installed the additional power cables from the generator, to the emergency bus. These new cables will allow the generator to support more of the load, of the building. The portable chiller requested, has arrived on-site, and is available to assist in cooling, if needed. Generally speaking, there are four (4) phases of restoration per floor (Air Handler restoral, Power restoral, Transport Equipment restoral, and Switch/Application Equipment restoral). Teams report that Air Handlers are up and running, and all power plants are on floors 2 through 7 are online. Given significant progress made, floors 2 through 7, are ready for technology turn up. Relative to Priority Transport related equipment, approximately 90% of the elements have been turned up on floors 2 through 7. The Power team is currently working on Floors 8 through 11 (N4E). The first floor is not accessible, at this time. Once access is granted by federal and local authorities, further assessment and restoration efforts will begin. The generator is currently supporting approximately 50% of its capacity, and alternative plans are being considered to handle the full load of the building. Teams continue to work proactively in effort to identify potential issues and are actively engaged working to restore services and repair infrastructure. AT&T Network Disaster Recovery (NDR) has eleven (11) SatCOLTs in service (TN, AL, GA). Two (2) of the eleven (11) are deployed at the Nashville, TN Central Office to provide coverage for the AT&T response teams as well as FirstNet (FN) customers. One (1) COLT is in hot-standby (TN). Six (6) COLTs are en-route to deployment sites in TN and AL. Three (3) COLTs are being demobilized in Alabama and coming back to Nashville for new assignments and five (5) additional COLTs are en-route to the Nashville area to support additional requests. The next update will be issued at approximately 19:00 CT, unless there is a significant change in status. On Mon, Dec 28, 2020, 5:59 PM Sean Donelan < sean@donelan.com <mailto:sean@donelan.com> > wrote: AT&T statement says nearly all services have been restore in Nashville as of Monday, 5pm CST They are working on permanent repairs. https://about.att.com/pages/disaster_relief/nashville.html <https://urldefense.proofpoint.com/v2/url?u=https-3A__mila.bitdam.com_api_v1.0_links_rewrite-5Fclick_-3Frewrite-5Ftoken-3DeyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyZXdyaXRlX2lkIjoiNWZlYmI3MGZiOGZhNWRhYjg5ZjE0NzJiIiwidXJsIjoiIiwib3JnYW5pemF0aW9uX2lkIjoxNzgxfQ.fQSGLDVfXTrVFYepbx0AR-5FuG-5FqKj-2DUG337RhC567WBI-26url-3Dhttps-253A__about.att.com_pages_disaster-5Frelief_nashville.html&d=DwMF-g&c=N13-TaG7c-EYAiUNohBk74oLRjUiBTwVm-KSnr4bPSc&r=-GzOCp0ppLaBQPFaZ7lZ4bUUBQxpFBukitRP75oaRdQ&m=QP--yQuEX4jG-74vYtzEjEKuDLrRan890jWR_pl9hjU&s=kRIGqeCVD61v63ETdC8ooxHNNAP0FfZyzYnYzgnpLPQ&e=> AT&T's Network Disaster Recovery group faces management questions nearly every year to justifying their budget. While no one wants disasters, business continuity has to be part of the business. There are also mutual aid agreements between companies, but I don't know how many were invoked for this incident. https://about.att.com/ecms/dam/pages/disaster_relief/NDR_edited_04.22.19.pdf <https://urldefense.proofpoint.com/v2/url?u=https-3A__mila.bitdam.com_api_v1.0_links_rewrite-5Fclick_-3Frewrite-5Ftoken-3DeyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyZXdyaXRlX2lkIjoiNWZlYmI3MGZlZTIwYTJmMzM5ZGM5ZTgwIiwidXJsIjoiIiwib3JnYW5pemF0aW9uX2lkIjoxNzgxfQ.xIE6w200hxqs5MLmTyAw3TKHl50TG9NaxjGW32JTEr8-26url-3Dhttps-253A__about.att.com_ecms_dam_pages_disaster-5Frelief_NDR-5Fedited-5F04.22.19.pdf&d=DwMF-g&c=N13-TaG7c-EYAiUNohBk74oLRjUiBTwVm-KSnr4bPSc&r=-GzOCp0ppLaBQPFaZ7lZ4bUUBQxpFBukitRP75oaRdQ&m=QP--yQuEX4jG-74vYtzEjEKuDLrRan890jWR_pl9hjU&s=oSnRWvbMLFwj9B8SlCbqsU06WNkyqf2fALfQ2mlcMJg&e=> ________________________________ This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.
I wouldn't say bad design, I would say outdated design. How do you take a single fiber optic cable or a copper cable bundle going to a business/building or a house and terminate it at 2 different Central Offices? It inherently has a single points of failure. (why I added extra straps to my fiber outside at home because stupid wind blows the damn cable around that goes to the pole, I'm in NJ everything on poles here) I don't think a bomb going off was part of the redundancy design process when telephone central offices were first starting to be built. I heard the rumour is (perhaps fact) the generators were keeping everything up till about 12-1 PM eastern when they shut the gas supply off to the neighborhood earlier safety. Once the batteries drained, that was it. Family I have in Nashville were on the phone when it cut out and then everything went dark. I do think they could have done a better job with the wireless infra. No reason you can't have some cell sites that can feed to a different central office. The fact is, distributed systems could work better, many smaller switching and distribution buildings scattered about a metro area with redundant links. A mesh. That was the single point of failure is just the link from your house to the next hop. Obviously not feasible with copper pairs. - Javier On Thu, Jan 14, 2021 at 10:05 AM Hiers, David <David.Hiers@cdk.com> wrote:
No doubt they’re good, but the best support can’t overcome bad design.
*From:* NANOG <nanog-bounces+david.hiers=cdk.com@nanog.org> *On Behalf Of *Robert DeVita *Sent:* Tuesday, December 29, 2020 5:17 PM *To:* Eric Kuhnke <eric.kuhnke@gmail.com>; Sean Donelan <sean@donelan.com> *Cc:* NANOG <nanog@nanog.org> *Subject:* Re: Nashville
AT&T Disaster Recovery Team is probably the best in the business. The resources they can bring to the table are unmatched. This would have been 100x worse if it hit a carrier neutral datacenter. They don’t have nearly the same resources to restore something like this. They usually do a road show (pre Covid). If you get a chance it’s definitely something you should go check out. Very impressive.
Robert DeVita
Founder & CEO
Mejeticks
c. 469-441-8864
e. radevita@mejeticks.com ------------------------------
*From:* NANOG <nanog-bounces+radevita=mejeticks.com@nanog.org> on behalf of Eric Kuhnke <eric.kuhnke@gmail.com> *Sent:* Tuesday, December 29, 2020 5:06:00 PM *To:* Sean Donelan <sean@donelan.com> *Cc:* NANOG <nanog@nanog.org> *Subject:* Re: Nashville
From a few days ago. Obviously centralizing lots of ss7/pstn stuff all in one place has a long recovery time when it's physically damaged. Something to think about for entities that own and operate traditional telco COs and their plans for disaster recovery.
Nv1
Here is the latest update: 6:46AM 12/27:
Work continues restoring service to the CRS routers in the Nashville Central Office. One router remains out of service and the other is in service with some links remaining out of service.
The working bridge will reconvene at 08:00 CT with the following action plan:
Additional cabling added to the first portable generator to enable full load capabilities (08:00 CT)
Pigtails with camlocks installed for easy swap; investigate possibility to land generator on the emergency service board to give the site N+1 with a manual ability to choose anyone. (08:00 CT)
check small power plants on floors 4 and 6 (08:00 CT)
Investigate water damage on 1st floor and energize if safe (08:00 CT
Air handlers for floors 4,5 and 6 (09:00 CT)
complete all transport work
Turn up SS7
Turn up 911 service - Approximately noon or after)
Turn up switching service.
TDM Switching team will reconvene at 09:00 CT and the Signaling team will reconvene at 11:00 CT on 12/27/2020.
DMS equipment on the 1st floor will be assessed for water damage. Switching teams will monitor power and HVAC restoration and will begin switch restoration as soon as the go ahead is provided by the power team.
Recovery Priorities:
1. 4th & 5th floors (Specify transport equipment needed to clear MTSO SS7 isolation & Datakit needed for Local Switch restoration). Transport SMEs currently working to turn up transport equipment
2. 6th floor (ESINET Groomers)
3. 10th and 8th floors (N4E) – Trunks
4. 1st floor (DMS: DS1, 5E: DS3) - Local POTS
5. 1st floor (DMS: DS0, DS2 | 5E: DS6) – Trunks
6. 11th floor (DMS: 01T) – Trunks
7. 4th floor (STP and SCP with mates up in Donelson)
The next update will be issued at approximately 09:00 CT on December 27.
Nv2
As of 09:00 CT: Teams worked through the night to restore service and improve conditions at the Nashville 2nd Ave Central Office. Since the initial service impact, over 75% of the Out of Service Mobility Sites have been restored. Certain call flows may be limited and should improve as additional restoration activities complete.
The generator that is currently powering equipment on the 2nd and 3rd floor, was refueled and ran with no issues through the night. Overnight, the batteries connected to it, continued to charge. Teams have placed additional power cables, which once connected, will allow the working generator, to better handle the load in the building. In order to accomplish this, the generator will need to be shut down for 15-30 minutes this morning, so teams can connect the new cables to the system. The power team reports they are still on target to restore power and cooling to the 5th and 6th floor by approximately 12:00 CT. Also, a portable chiller will be delivered this morning and strategically placed, in case it is needed to assist in cooling the office.
There is a Call Center at 333 Commerce, in Nashville that does not have network or phone services available. Corporate Real Estate (CRE) reports there is some damage to that office, but the extent of the damage will not be known until they can gain access to the site. Because of this, the impacted Call Center ceased operations until further notice.
DMS switching equipment on the 1st floor will be assessed for water damage. Switching teams will monitor power and HVAC restoration. Equipment power ups will begin, as soon as the go ahead is provided by the power team.
Two SatCOLTs remain positioned on the East and West sides of the NSVLTNMT Central Office providing critical communication for teams working restoration efforts. There are 17 assets deployed in the field- 15 are on air (the 2 at the CO and 13 supporting FN Customer Requests) and 2 are in hot-standby for FN Customers where macro service recently recovered. There is 1 asset staged at a deployment site in KY where macro service restored, and 8 additional assets are on route to Nashville today to fulfill pending FN Customer requests. Incoming requests continue to be triaged. The ones in areas where service looks to have been restored, are being held, while the others are being prioritized to be dispatched upon.
The next update will be issued at approximately 14:00 CT, unless there is a significant change in status.
Nv3
AT&T Nashville update below, received at 3:35PM 12/27.
Since the initial service impact, over 95% of the Out of Service Mobility Sites have been restored. Certain call flows may be limited and should improve as additional restoration activities complete.
Electricians have installed the additional power cables from the generator, to the emergency bus. These new cables will allow the generator to support more of the load, of the building. The portable chiller requested, has arrived on-site, and is available to assist in cooling, if needed. Generally speaking, there are four (4) phases of restoration per floor (Air Handler restoral, Power restoral, Transport Equipment restoral, and Switch/Application Equipment restoral). Teams report that Air Handlers are up and running, and all power plants are on floors 2 through 7 are online. Given significant progress made, floors 2 through 7, are ready for technology turn up. Relative to Priority Transport related equipment, approximately 90% of the elements have been turned up on floors 2 through 7. The Power team is currently working on Floors 8 through 11 (N4E). The first floor is not accessible, at this time. Once access is granted by federal and local authorities, further assessment and restoration efforts will begin.
The generator is currently supporting approximately 50% of its capacity, and alternative plans are being considered to handle the full load of the building. Teams continue to work proactively in effort to identify potential issues and are actively engaged working to restore services and repair infrastructure.
AT&T Network Disaster Recovery (NDR) has eleven (11) SatCOLTs in service (TN, AL, GA). Two (2) of the eleven (11) are deployed at the Nashville, TN Central Office to provide coverage for the AT&T response teams as well as FirstNet (FN) customers. One (1) COLT is in hot-standby (TN). Six (6) COLTs are en-route to deployment sites in TN and AL. Three (3) COLTs are being demobilized in Alabama and coming back to Nashville for new assignments and five (5) additional COLTs are en-route to the Nashville area to support additional requests.
The next update will be issued at approximately 19:00 CT, unless there is a significant change in status.
On Mon, Dec 28, 2020, 5:59 PM Sean Donelan < sean@donelan.com > wrote:
AT&T statement says nearly all services have been restore in Nashville as of Monday, 5pm CST
They are working on permanent repairs.
https://about.att.com/pages/disaster_relief/nashville.html <https://urldefense.proofpoint.com/v2/url?u=https-3A__mila.bitdam.com_api_v1.0_links_rewrite-5Fclick_-3Frewrite-5Ftoken-3DeyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyZXdyaXRlX2lkIjoiNWZlYmI3MGZiOGZhNWRhYjg5ZjE0NzJiIiwidXJsIjoiIiwib3JnYW5pemF0aW9uX2lkIjoxNzgxfQ.fQSGLDVfXTrVFYepbx0AR-5FuG-5FqKj-2DUG337RhC567WBI-26url-3Dhttps-253A__about.att.com_pages_disaster-5Frelief_nashville.html&d=DwMF-g&c=N13-TaG7c-EYAiUNohBk74oLRjUiBTwVm-KSnr4bPSc&r=-GzOCp0ppLaBQPFaZ7lZ4bUUBQxpFBukitRP75oaRdQ&m=QP--yQuEX4jG-74vYtzEjEKuDLrRan890jWR_pl9hjU&s=kRIGqeCVD61v63ETdC8ooxHNNAP0FfZyzYnYzgnpLPQ&e=>
AT&T's Network Disaster Recovery group faces management questions nearly every year to justifying their budget. While no one wants disasters, business continuity has to be part of the business. There are also mutual aid agreements between companies, but I don't know how many were invoked for this incident.
https://about.att.com/ecms/dam/pages/disaster_relief/NDR_edited_04.22.19.pdf
------------------------------
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.
The building is an ILEC CO and wire centre. It is the main local transiting interconnect point for voice services in the 470 LATA, and also one of the toll tandems for the region. At 02:18 PM 25/12/2020, Andy Ringsmuth wrote:
Certainly thankful no serious injuries or fatalities in this clusterblank.
It seems the AT&T building at 185 2nd Ave N may have been a target, which would explain the timing (holiday morning when no one is out, as opposed to a holiday evening when there would be mass casualties). A little curious what that building has. Is it just a big co-lo place? Regional CLEC/ILEC?
No earth-shattering revelations here. Admittedly just bored on a slow Christmas Day when my wife is at work (nurse) and kid is playing with a new tablet and Iâm just watching the news trying to understand/figure out a little what and why.
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
âBetter even die free, than to live slaves.â - Frederick Douglas, 1863
-- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409
participants (13)
-
Andy Ringsmuth -
Clayton Zekelman -
cosmo -
Eric Kuhnke -
Hiers, David -
Javier J -
Josh Baird -
Matt Brennan -
Matt Hoppes -
Robert DeVita -
Rodney Joffe -
Sean Donelan -
Tom Beecher