Outages list is going bananas right now: Same thing happened in Monroe LA that took down all of north Louisiana. It was an update that went bad and the switch had to be manually checked line by line. No backup was done before the maintenance. 16 hours of downtime about 45 days ago. On Tuesday, May 7, 2013, Frank Bulk wrote: Sounds like it was scheduled maintenance gone bad: "Perrine said he spoke with CenturyLink at 6 a.m.<x-apple-data-detectors://1> where they advised him the company was doing scheduled maintenance. At 7:35 a.m.<x-apple-data-detectors://2> they told him something had gone wrong during the maintenance and it was affecting customers in 13 states." http://www.tallahassee.com/article/20130507/NEWS/130507010/Centurylink-outag e-affecting-local-internet-provider?nclick_check=1 I bite my tongue in regards to Network Tallahasee's singlehomedness and Perrine's comment on his questioning of the timing of the update. I encourage all my competitors to stay single homed and to do their maintenance Sunday morning<x-apple-data-detectors://4>, while their vendors are at their lowest staffing levels. Frank -----Original Message----- From: Outages [mailto:outages-bounces@outages.org<x-apple-msg-load://2CF0901A-5EB4-4331-8B21-8DE35C0C12D0/>] On Behalf Of Frank Bulk Sent: Tuesday, May 07, 2013 8:29 AM To: 'Marco Prechel'; outages@outages.org<x-apple-msg-load://2CF0901A-5EB4-4331-8B21-8DE35C0C12D0/> Subject: Re: [outages] Centurylink nationwide outage I first saw timeouts with www.qwest.com<http://www.qwest.com/> over IPv6 at 3:08 am Central<x-apple-data-detectors://9> -- I was wondering why, now I know. =) Frank -----Original Message----- From: Outages [mailto:outages-bounces@outages.org<x-apple-msg-load://2CF0901A-5EB4-4331-8B21-8DE35C0C12D0/>] On Behalf Of Marco Prechel Sent: Tuesday, May 07, 2013 7:33 AM To: outages@outages.org<x-apple-msg-load://2CF0901A-5EB4-4331-8B21-8DE35C0C12D0/> Subject: [outages] Centurylink nationwide outage Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around 0400 EDT. ANS/ABS states it's a nationwide routing issue. No ETR. _______________________________________________ Outages mailing list Outages@outages.org<x-apple-msg-load://2CF0901A-5EB4-4331-8B21-8DE35C0C12D0/> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<x-apple-msg-load://2CF0901A-5EB4-4331-8B21-8DE35C0C12D0/> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<x-apple-msg-load://2CF0901A-5EB4-4331-8B21-8DE35C0C12D0/> https://puck.nether.net/mailman/listinfo/outages -- Jeremie Chism Triton Communications _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages Sent from my iPad On May 7, 2013, at 10:16 AM, "Jason Lester" <jlester@wcs.k12.va.us<mailto:jlester@wcs.k12.va.us>> wrote: Does anyone know what is going on with the nationwide CenturyLink outage? Their NOC recording says it is a BGP routing issue with their upstream peers affecting Internet traffic and traffic between regions. Our outside connectivity with them has basically been down since about 4:00AM (EDT) this morning. The prefixes we were receiving from them were fluctuating between a few hundred and a few thousand all morning. We are getting the full BGP table from them now (for about the last hour), but still not seeing any incoming traffic. Seems like a major issue since it has been almost 9 hours now. Thanks, Jason -- Jason Lester Administrator for Instructional Technology Washington County Public Schools Tel: 276-739-3060 Fax: 276-628-1893 http://www.wcs.k12.va.us

Widely discussed on outages@outages.org list (hint!) but for those not yet list members over there, 13 or more states in southeast US affected, reportedly routing / layer 3 issue, possibly BGP to outside but not clear. Some service restorations discussed. George William Herbert Sent from my iPhone On May 7, 2013, at 9:54 AM, Jason Lester <jlester@wcs.k12.va.us> wrote:

Does anyone know what is going on with the nationwide CenturyLink outage? Their NOC recording says it is a BGP routing issue with their upstream peers affecting Internet traffic and traffic between regions. Our outside connectivity with them has basically been down since about 4:00AM (EDT) this morning. The prefixes we were receiving from them were fluctuating between a few hundred and a few thousand all morning. We are getting the full BGP table from them now (for about the last hour), but still not seeing any incoming traffic. Seems like a major issue since it has been almost 9 hours now.

Thanks, Jason --

Jason Lester Administrator for Instructional Technology Washington County Public Schools Tel: 276-739-3060 Fax: 276-628-1893 http://www.wcs.k12.va.us