Today Level3/CenturyLink is having a major outage. This is a great example of how much diversity matter. How important to plan your network right. These type of outages can happen to anyone. this is time to understand better how important it is to plan your network resiliency and diversity. This is also time to unite as telecom industry to learn from this big outage that impacted many. I hope technical team will share their learnings in NANOG or blog so others will learn from it. Good luck Engineering/Ops teams who are working on restoration. Thank you On Thu, Dec 27, 2018 at 23:24 Naslund, Steve <SNaslund@medline.com> wrote:

We see slow recovery. Dallas data service came back up, Dubuque voice service still down.

Steven Naslund

Chicago IL

...

...

Seems like things have stabilized as of about an hour ago for us.

...

-- Mehmet +1-424-298-1903

Maybe light at the end of the tunnel... latest we got from Centurylink was at 10:45pm CST. Our engineers and technicians have identified the network element that has affected our customer services. Services are restoring, and the current estimated time for full recovery is four hours. CenturyLink will be conducting an extensive post-incident investigation and root cause analysis to provide follow-up information to our customers impacted by this event. We will provide one more Sales Alert from this address once services are fully restored. -----Original Message----- From: NANOG <nanog-bounces+esundberg=nitelusa.com@nanog.org> On Behalf Of Brielle Bruns Sent: Friday, December 28, 2018 12:16 AM To: nanog@nanog.org Subject: Re: CenturyLink We have interwebs in Boise! At least, on the biz fiber w/ BGP. 720892 ipv4 routes, 62273 ipv6 routes. On 12/27/2018 1:16 PM, Coker, Steve - DEN wrote:

Seems like things have stabilized as of about an hour ago for us.

DIA Logo

*STEVE COKER***

/Senior Manager of Network Infrastructure - Business Technologies/

Denver International Airport

Technologies | Concourse A

*From:* NANOG <nanog-bounces@nanog.org> *On Behalf Of *Erik Sundberg *Sent:* Thursday, December 27, 2018 1:13 PM *To:* NANOG <nanog@nanog.org> *Subject:* RE: CenturyLink

The outage list already has a long thread on this.

https://puck.nether.net/mailman/listinfo/outages

**

**

**

*From:* NANOG <nanog-bounces@nanog.org <mailto:nanog-bounces@nanog.org>> *On Behalf Of *Dennis Burgess via NANOG *Sent:* Thursday, December 27, 2018 2:00 PM *To:* Naslund, Steve <SNaslund@medline.com <mailto:SNaslund@medline.com>>; nanog@nanog.org <mailto:nanog@nanog.org> *Subject:* RE: CenturyLink

National outage since 4:33 am this morning..

**

*LTI-Full_175px*

*Dennis Burgess, Mikrotik Certified Trainer *

Author of "Learn RouterOS- Second Edition"

*Link Technologies, Inc*-- Mikrotik & WISP Support Services

*Office*: 314-735-0270 Website: http://www.linktechs.net <http://www.linktechs.net/>

Create Wireless Coverage's with www.towercoverage.com

*From:* NANOG <nanog-bounces@nanog.org <mailto:nanog-bounces@nanog.org>> *On Behalf Of *Naslund, Steve *Sent:* Thursday, December 27, 2018 12:46 PM *To:* nanog@nanog.org <mailto:nanog@nanog.org> *Subject:* CenturyLink

Anyone have any insight to the nationwide CenturyLink issues/outages today? Just wondering. Know for sure that our connections to them from Florida, Iowa, and Washington State are all affected. Voice and data.

Steven Naslund

Chicago IL

---------------------------------------------------------------------- --

CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.

-- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.

Yes, there were 911 services affected. The latest word from C-link as of 1:46PM mountain is that all 911 services are restored where they are the provider. I'm not 100% sure if that's system-wide, or just my area in the northwest, however. Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Friday, December 28, 2018 1:03 AM, Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote:

On Fri, Dec 28, 2018 at 07:07:42AM +0000, Erik Sundberg ESundberg@nitelusa.com wrote a message of 131 lines which said:

...

CenturyLink will be conducting an extensive post-incident investigation and root cause analysis to provide follow-up information to our customers

Is this problem also responsible for the 911 outage? If so, the post-mortem analysis is not useful only for CenturyLink customers but for everyone on the west coast.

Ouch. Feel bad for the guys on the ground at C-link. Not a fun 24 hours. Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Friday, December 28, 2018 3:17 PM, Anne P. Mitchell, Esq. <amitchell@isipp.com> wrote:

And the other latest news is that the FCC is investigating the CenturyLink outage:

https://www.theinternetpatrol.com/fcc-investigating-centurylink-outage-says-...

...

On Dec 28, 2018, at 3:11 PM, Patrick Boyle via NANOG nanog@nanog.org wrote: Yes, there were 911 services affected. The latest word from C-link as of 1:46PM mountain is that all 911 services are restored where they are the provider. I'm not 100% sure if that's system-wide, or just my area in the northwest, however. Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Friday, December 28, 2018 1:03 AM, Stephane Bortzmeyer bortzmeyer@nic.fr wrote:

...

On Fri, Dec 28, 2018 at 07:07:42AM +0000, Erik Sundberg ESundberg@nitelusa.com wrote a message of 131 lines which said:

...

CenturyLink will be conducting an extensive post-incident investigation and root cause analysis to provide follow-up information to our customers

Is this problem also responsible for the 911 outage? If so, the post-mortem analysis is not useful only for CenturyLink customers but for everyone on the west coast.

On Fri, Dec 28, 2018 at 12:05 AM Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote:

Is this problem also responsible for the 911 outage? If so, the post-mortem analysis is not useful only for CenturyLink customers but for everyone on the west coast.

Looks like most time.nist.gov servers (3 x NIST sites on AS49) are single homed on CenturyLink, anyone noticed NTP issues yesterday? https://tf.nist.gov/tf-cgi/servers.cgi

Yep. We are required by FINRA to verify that our clocks on our trading systems are within a certain tolerance of NIST time. We are still seeing issues with 3 of the NIST servers this morning. Since NIST is on a skeleton crew anyway due to the government shutdown, I don't expect any resolution shortly. ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Yang Yu Sent: Friday, December 28, 2018 6:23 PM To: Stephane Bortzmeyer <bortzmeyer@nic.fr> Cc: nanog@nanog.org Subject: Re: CenturyLink On Fri, Dec 28, 2018 at 12:05 AM Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote:

Is this problem also responsible for the 911 outage? If so, the post-mortem analysis is not useful only for CenturyLink customers but for everyone on the west coast.

Looks like most time.nist.gov servers (3 x NIST sites on AS49) are single homed on CenturyLink, anyone noticed NTP issues yesterday? https://tf.nist.gov/tf-cgi/servers.cgi

We have two stratum-1 servers synced with GPS and a PTP feed from a provider that also provides PTP to market data systems, but we still have to monitor drift between system time and NIST time. Don't ask for the logic behind it, it's a regulation, not a technical requirement. ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Stephen Satchell Sent: Saturday, December 29, 2018 9:34 AM To: nanog@nanog.org Subject: Re: CenturyLink On 12/28/18 3:23 PM, Yang Yu wrote:

On Fri, Dec 28, 2018 at 12:05 AM Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote:

...

Is this problem also responsible for the 911 outage? If so, the post-mortem analysis is not useful only for CenturyLink customers but for everyone on the west coast.

Looks like most time.nist.gov servers (3 x NIST sites on AS49) are single homed on CenturyLink, anyone noticed NTP issues yesterday?

https://tf.nist.gov/tf-cgi/servers.cgi

I have GPS-based Stratum 1 NTP appliances in my network, so I wouldn't see any issues. I suspect many other operators are in the same situation.

Actually, on all our trading systems, our times are synced via PTP instead of ntpd for at least 50 microsecond accuracy. The stratum 1 clocks as well as NIST time are only used as comparison to verify compliance and reality. We use ntpq to determine the offset from NIST for reporting. ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Stephen Satchell Sent: Saturday, December 29, 2018 10:01 AM To: nanog@nanog.org Subject: Re: CenturyLink On 12/29/18 6:51 AM, Matthew Huff wrote:

We have two stratum-1 servers synced with GPS and a PTP feed from a provider that also provides PTP to market data systems, but we still have to monitor drift between system time and NIST time. Don't ask for the logic behind it, it's a regulation, not a technical requirement.

Having been a participant on Standards Working Groups, I understand completely. Regulations, like Standards, need to be written to apply to as wide a population as possible. Do those regulations dictate how you monitor drift? For example, can you have a system (I would use CentOS) that syncs to the appliance as well as NIST and your inside NTP sources, and use ntpq(8) to read the drift directly?

Speaking of GPS-enabled NTP appliances, etc. wondering what hardware people are using for this. thanks -----Original Message----- From: "Raymond Burkholder" <ray@oneunified.net> Sent: Saturday, December 29, 2018 12:01pm To: "Matthew Huff" <mhuff@ox.com>, "list@satchell.net" <list@satchell.net>, "nanog@nanog.org" <nanog@nanog.org> Subject: Re: CenturyLink On 2018-12-29 7:51 a.m., Matthew Huff wrote:

We have two stratum-1 servers synced with GPS and a PTP feed from a provider that also provides PTP to market data systems, but we still have to monitor drift between system time and NIST time. Don't ask for the logic behind it, it's a regulation, not a technical requirement.

On one occasion, due to bad firmware or a configuration issue, I have seen GPS stratum 1 diverge from NTP. It was somewhat eye brow raising to the company. My NTP monitored servers were shown to be diverging their GPS/NTP, but after looking at twice or thrice, it was the other way around.

We use an older model of https://www.microsemi.com/product-directory/enterprise-network-time-servers/... with rubidium oscillator. Not cheap, but hardened and extremely accurate. ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 From: Shawn L [mailto:shawnl@up.net] Sent: Sunday, December 30, 2018 9:40 AM To: nanog@nanog.org Cc: Matthew Huff <mhuff@ox.com>; list@satchell.net Subject: Re: CenturyLink Speaking of GPS-enabled NTP appliances, etc. wondering what hardware people are using for this. thanks -----Original Message----- From: "Raymond Burkholder" <ray@oneunified.net<mailto:ray@oneunified.net>> Sent: Saturday, December 29, 2018 12:01pm To: "Matthew Huff" <mhuff@ox.com<mailto:mhuff@ox.com>>, "list@satchell.net<mailto:list@satchell.net>" <list@satchell.net<mailto:list@satchell.net>>, "nanog@nanog.org<mailto:nanog@nanog.org>" <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: CenturyLink On 2018-12-29 7:51 a.m., Matthew Huff wrote:

We have two stratum-1 servers synced with GPS and a PTP feed from a provider that also provides PTP to market data systems, but we still have to monitor drift between system time and NIST time. Don't ask for the logic behind it, it's a regulation, not a technical requirement.

On one occasion, due to bad firmware or a configuration issue, I have seen GPS stratum 1 diverge from NTP. It was somewhat eye brow raising to the company. My NTP monitored servers were shown to be diverging their GPS/NTP, but after looking at twice or thrice, it was the other way around.

Regulatory. If we were to lose the GPS signal (antenna failure, etc...) then our stratum 1 time sources wouldn't drift as much and as quickly. For telco and general usage, the cost may not be worthwhile, but when you have auditors looking over your shoulder.... ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 -----Original Message----- From: Saku Ytti [mailto:saku@ytti.fi] Sent: Sunday, December 30, 2018 12:30 PM To: Matthew Huff <mhuff@ox.com> Cc: Shawn L <shawnl@up.net>; nanog@nanog.org Subject: Re: CenturyLink Hey Matthew,

We use an older model of https://www.microsemi.com/product-directory/enterprise-network-time-servers/... with rubidium oscillator. Not cheap, but hardened and extremely accurate.

Out of interest why rubidium? For short term stability oscillator choice isn't much of a matter for free running even rubidium will be off good +-1us per day or +-30us per week. I've always wondered what niche rubidium covers that isn't handled by much cheaper crystal ovens or actually precise oscillators. -- ++ytti

Yo Saku! On Sun, 30 Dec 2018 19:29:37 +0200 Saku Ytti <saku@ytti.fi> wrote:

I've always wondered what niche rubidium covers that isn't handled by much cheaper crystal ovens or actually precise oscillators.

The Rb frequency reference will be two or three orders of magnitude more stable than an expensive ovenized crystal. RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can’t measure it, you can’t improve it." - Lord Kelvin

There isn't a specific regulation on free-running GPS, just "due diligence". I work at a algorithmic program trading company (and have been for 20 years). We have a high ROI, the cost differential for the rubidium OC versus having to drop everything to conform to regulatory requirements due to a short GPS outage, makes this a no-brainer. ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Saku Ytti Sent: Monday, December 31, 2018 3:28 AM To: Gary E. Miller <gem@rellim.com> Cc: nanog@nanog.org Subject: Re: CenturyLink Hey Gary, On Mon, 31 Dec 2018 at 05:02, Gary E. Miller <gem@rellim.com> wrote:

The Rb frequency reference will be two or three orders of magnitude more stable than an expensive ovenized crystal.

Perhaps, but not supported by this: https://www.meinbergglobal.com/english/specs/gpsopt.htm For the tl;dr folk, crystal drifts +-4.5us per day, Rb +-1.1us (both seem like unsatisfactorily high numbers to me, i.e. you don't want to be free-running 24h with Rb). Luckily today we have GPS, Glonass, BeiDou, Galileo and couple smaller ones, so there should be somewhat reasonable amount of redundancy. Unsure which commercially available NTP or PPP master clocks support all four. But I of course readily accept Rb is objectively more accurate than crystal, I'm just curious where it matters and I'm curious which regulation applies, who fall under the regulation and what specifically does the regulation require about free-running accuracy. -- ++ytti

On Mon, 31 Dec 2018 10:28:25 +0200, Saku Ytti said:

For the tl;dr folk, crystal drifts +-4.5us per day, Rb +-1.1us (both seem like unsatisfactorily high numbers to me, i.e. you don't want to be free-running 24h with Rb).

There's another number that's missing - the stability of the drift. I'd rather be dealing with a crystal that's +2.788+/-0.003 than one that's +0.5+/-0.25

We have a few s600’s deployed as well, rock solid and van handle 10k+ queries a second. ns Sent from my iPhone On Dec 30, 2018, at 11:22 AM, Matthew Huff <mhuff@ox.com<mailto:mhuff@ox.com>> wrote: We use an older model of https://www.microsemi.com/product-directory/enterprise-network-time-servers/... with rubidium oscillator. Not cheap, but hardened and extremely accurate. ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 From: Shawn L [mailto:shawnl@up.net] Sent: Sunday, December 30, 2018 9:40 AM To: nanog@nanog.org<mailto:nanog@nanog.org> Cc: Matthew Huff <mhuff@ox.com<mailto:mhuff@ox.com>>; list@satchell.net<mailto:list@satchell.net> Subject: Re: CenturyLink Speaking of GPS-enabled NTP appliances, etc. wondering what hardware people are using for this. thanks -----Original Message----- From: "Raymond Burkholder" <ray@oneunified.net<mailto:ray@oneunified.net>> Sent: Saturday, December 29, 2018 12:01pm To: "Matthew Huff" <mhuff@ox.com<mailto:mhuff@ox.com>>, "list@satchell.net<mailto:list@satchell.net>" <list@satchell.net<mailto:list@satchell.net>>, "nanog@nanog.org<mailto:nanog@nanog.org>" <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: CenturyLink On 2018-12-29 7:51 a.m., Matthew Huff wrote:

We have two stratum-1 servers synced with GPS and a PTP feed from a provider that also provides PTP to market data systems, but we still have to monitor drift between system time and NIST time. Don't ask for the logic behind it, it's a regulation, not a technical requirement.

On one occasion, due to bad firmware or a configuration issue, I have seen GPS stratum 1 diverge from NTP. It was somewhat eye brow raising to the company. My NTP monitored servers were shown to be diverging their GPS/NTP, but after looking at twice or thrice, it was the other way around.