RE: Centurylink having a bad morning?
That site seems to be just for their cloud products, is there one of these for their actual network? From: R. Leigh Hennig <leigh@regula.one> Sent: Sunday, August 30, 2020 8:54 AM To: Drew Weaver <drew.weaver@thenap.com>; nanog@nanog.org Subject: Re: Centurylink having a bad morning? Global impact with issues reported by Fastly, Cloudflare, OpenDNS. https://status.ctl.io/ STARTED Sun Aug 30 2020 08:13 (EDT) Sun Aug 30 2020 12:13 (UTC) AFFECTED SERVICES External Cloud Network (CA3) DATE LATEST UPDATE Sun Aug 30 2020 08:13 (EDT) Sun Aug 30 2020 12:13 (UTC) Our technical teams are investigating an issue affecting some services in the CA3 data center. Ensuring the reliability of our services is our top priority. We will continue to provide status updates as this incident progresses. If you need further support, please contact us at help@ctl.io<mailto:help@ctl.io>. Sent from ProtonMail Mobile On Sun, Aug 30, 2020 at 8:14 AM, Drew Weaver via NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> wrote: Hello, Woke up this morning to a bunch of reports of issues with connectivity had to shut down some Level3/CTL connections to get it to return to normal. As of right now their support portal won’t load: https://www.centurylink.com/business/login/ Just wondering what others are seeing.
People are rebooting ghosting now. [image: image.png] https://twitter.com/ir_kujoe/status/1300066569645707265 Seeing other reports of this too. On Sun, Aug 30, 2020 at 9:45 AM Drew Weaver <drew.weaver@thenap.com> wrote:
That site seems to be just for their cloud products, is there one of these for their actual network?
*From:* R. Leigh Hennig <leigh@regula.one> *Sent:* Sunday, August 30, 2020 8:54 AM *To:* Drew Weaver <drew.weaver@thenap.com>; nanog@nanog.org *Subject:* Re: Centurylink having a bad morning?
Global impact with issues reported by Fastly, Cloudflare, OpenDNS.
STARTED
Sun Aug 30 2020 08:13 (EDT) Sun Aug 30 2020 12:13 (UTC) AFFECTED SERVICES
External Cloud Network (CA3) DATE LATEST UPDATE
Sun Aug 30 2020 08:13 (EDT) Sun Aug 30 2020 12:13 (UTC) Our technical teams are investigating an issue affecting some services in the CA3 data center. Ensuring the reliability of our services is our top priority. We will continue to provide status updates as this incident progresses. If you need further support, please contact us at help@ctl.io.
Sent from ProtonMail Mobile
On Sun, Aug 30, 2020 at 8:14 AM, Drew Weaver via NANOG <nanog@nanog.org> wrote:
Hello,
Woke up this morning to a bunch of reports of issues with connectivity had to shut down some Level3/CTL connections to get it to return to normal.
As of right now their support portal won’t load: https://www.centurylink.com/business/login/
Just wondering what others are seeing.
-- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com
Something just now changed in this situation and now it seems to have gotten worse. From: Jason Kuehl <jason.w.kuehl@gmail.com> Sent: Sunday, August 30, 2020 10:00 AM To: Drew Weaver <drew.weaver@thenap.com> Cc: R. Leigh Hennig <leigh@regula.one>; nanog@nanog.org Subject: Re: Centurylink having a bad morning? People are rebooting ghosting now. [cid:image001.png@01D67EB4.7EBAE940] https://twitter.com/ir_kujoe/status/1300066569645707265 Seeing other reports of this too. On Sun, Aug 30, 2020 at 9:45 AM Drew Weaver <drew.weaver@thenap.com<mailto:drew.weaver@thenap.com>> wrote: That site seems to be just for their cloud products, is there one of these for their actual network? From: R. Leigh Hennig <leigh@regula.one<mailto:leigh@regula.one>> Sent: Sunday, August 30, 2020 8:54 AM To: Drew Weaver <drew.weaver@thenap.com<mailto:drew.weaver@thenap.com>>; nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Centurylink having a bad morning? Global impact with issues reported by Fastly, Cloudflare, OpenDNS. https://status.ctl.io/ STARTED Sun Aug 30 2020 08:13 (EDT) Sun Aug 30 2020 12:13 (UTC) AFFECTED SERVICES External Cloud Network (CA3) DATE LATEST UPDATE Sun Aug 30 2020 08:13 (EDT) Sun Aug 30 2020 12:13 (UTC) Our technical teams are investigating an issue affecting some services in the CA3 data center. Ensuring the reliability of our services is our top priority. We will continue to provide status updates as this incident progresses. If you need further support, please contact us at help@ctl.io<mailto:help@ctl.io>. Sent from ProtonMail Mobile On Sun, Aug 30, 2020 at 8:14 AM, Drew Weaver via NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> wrote: Hello, Woke up this morning to a bunch of reports of issues with connectivity had to shut down some Level3/CTL connections to get it to return to normal. As of right now their support portal won’t load: https://www.centurylink.com/business/login/ Just wondering what others are seeing. -- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com<mailto:jason.w.kuehl@gmail.com>
That might be because of this: The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance On Sun, Aug 30, 2020 at 7:42 AM Drew Weaver <drew.weaver@thenap.com> wrote:
Something just now changed in this situation and now it seems to have gotten worse.
*From:* Jason Kuehl <jason.w.kuehl@gmail.com> *Sent:* Sunday, August 30, 2020 10:00 AM *To:* Drew Weaver <drew.weaver@thenap.com> *Cc:* R. Leigh Hennig <leigh@regula.one>; nanog@nanog.org *Subject:* Re: Centurylink having a bad morning?
People are rebooting ghosting now.
https://twitter.com/ir_kujoe/status/1300066569645707265
Seeing other reports of this too.
On Sun, Aug 30, 2020 at 9:45 AM Drew Weaver <drew.weaver@thenap.com> wrote:
That site seems to be just for their cloud products, is there one of these for their actual network?
*From:* R. Leigh Hennig <leigh@regula.one> *Sent:* Sunday, August 30, 2020 8:54 AM *To:* Drew Weaver <drew.weaver@thenap.com>; nanog@nanog.org *Subject:* Re: Centurylink having a bad morning?
Global impact with issues reported by Fastly, Cloudflare, OpenDNS.
STARTED
Sun Aug 30 2020 08:13 (EDT) Sun Aug 30 2020 12:13 (UTC) AFFECTED SERVICES
External Cloud Network (CA3) DATE LATEST UPDATE
Sun Aug 30 2020 08:13 (EDT) Sun Aug 30 2020 12:13 (UTC) Our technical teams are investigating an issue affecting some services in the CA3 data center. Ensuring the reliability of our services is our top priority. We will continue to provide status updates as this incident progresses. If you need further support, please contact us at help@ctl.io.
Sent from ProtonMail Mobile
On Sun, Aug 30, 2020 at 8:14 AM, Drew Weaver via NANOG <nanog@nanog.org> wrote:
Hello,
Woke up this morning to a bunch of reports of issues with connectivity had to shut down some Level3/CTL connections to get it to return to normal.
As of right now their support portal won’t load: https://www.centurylink.com/business/login/
Just wondering what others are seeing.
--
Sincerely,
Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com
participants (3)
-
Drew Weaver
-
Jason Kuehl
-
Joseph Jenkins