Partial Outage with TW Telecom and CenturyLink
Morning Everyone, Yesterday between about 1900 and 2230 UTC, we had a partial drop with reaching various sites through TW Telecom from our circuit in Orlando, FL. The unavailable sites included Facebook, Newegg, and Godaddy. The outage did not affect our Atlanta TW Telecom. I confered with a colleague who manages a large customer in Apopka who said that they appeared not to be affected. His circuit and ours loop to the same TW Telecom POP. But even more Murphy than that, our Centurylink secondary circuit was having a routing loop issue at the same time, so while our BGP routes were being advertised to world through Centurylink, the circuit was useless. Centurylink aknowledged the existence of a bigger transport issue and said that we weren't the only customer affected. Anybody else notice these issues or have any other insight? Thanks! Eric Miller
CenturyLink is reporting core routing issues at this time. Seeing the same issues with our DS-3, BGP stayed up, but traffic was not passing over the line. Had to manually shutdown the interface to get traffic flowing over our other providers link. What a mess. On 4/24/2012 8:22 AM, Eric C. Miller wrote:
Morning Everyone,
Yesterday between about 1900 and 2230 UTC, we had a partial drop with reaching various sites through TW Telecom from our circuit in Orlando, FL. The unavailable sites included Facebook, Newegg, and Godaddy. The outage did not affect our Atlanta TW Telecom. I confered with a colleague who manages a large customer in Apopka who said that they appeared not to be affected. His circuit and ours loop to the same TW Telecom POP.
But even more Murphy than that, our Centurylink secondary circuit was having a routing loop issue at the same time, so while our BGP routes were being advertised to world through Centurylink, the circuit was useless. Centurylink aknowledged the existence of a bigger transport issue and said that we weren't the only customer affected.
Anybody else notice these issues or have any other insight?
Thanks!
Eric Miller
-- ---- ---- ---- ---- Chris Gotstein, Network Engineer, U.P. Logon/Computer Connection U.P. http://uplogon.com | +1 906 774 4847 | chris@uplogon.com
this belongs on outages@ no? On Tue, Apr 24, 2012 at 12:14 PM, Chris Gotstein <chris@uplogon.com> wrote:
CenturyLink is reporting core routing issues at this time. Seeing the same issues with our DS-3, BGP stayed up, but traffic was not passing over the line. Had to manually shutdown the interface to get traffic flowing over our other providers link. What a mess.
On 4/24/2012 8:22 AM, Eric C. Miller wrote:
Morning Everyone,
Yesterday between about 1900 and 2230 UTC, we had a partial drop with reaching various sites through TW Telecom from our circuit in Orlando, FL. The unavailable sites included Facebook, Newegg, and Godaddy. The outage did not affect our Atlanta TW Telecom. I confered with a colleague who manages a large customer in Apopka who said that they appeared not to be affected. His circuit and ours loop to the same TW Telecom POP.
But even more Murphy than that, our Centurylink secondary circuit was having a routing loop issue at the same time, so while our BGP routes were being advertised to world through Centurylink, the circuit was useless. Centurylink aknowledged the existence of a bigger transport issue and said that we weren't the only customer affected.
Anybody else notice these issues or have any other insight?
Thanks!
Eric Miller
-- ---- ---- ---- ---- Chris Gotstein, Network Engineer, U.P. Logon/Computer Connection U.P. http://uplogon.com | +1 906 774 4847 | chris@uplogon.com
Already on outages. On 4/24/2012 11:28 AM, Christopher Morrow wrote:
this belongs on outages@ no?
On Tue, Apr 24, 2012 at 12:14 PM, Chris Gotstein<chris@uplogon.com> wrote:
CenturyLink is reporting core routing issues at this time. Seeing the same issues with our DS-3, BGP stayed up, but traffic was not passing over the line. Had to manually shutdown the interface to get traffic flowing over our other providers link. What a mess.
On 4/24/2012 8:22 AM, Eric C. Miller wrote:
Morning Everyone,
Yesterday between about 1900 and 2230 UTC, we had a partial drop with reaching various sites through TW Telecom from our circuit in Orlando, FL. The unavailable sites included Facebook, Newegg, and Godaddy. The outage did not affect our Atlanta TW Telecom. I confered with a colleague who manages a large customer in Apopka who said that they appeared not to be affected. His circuit and ours loop to the same TW Telecom POP.
But even more Murphy than that, our Centurylink secondary circuit was having a routing loop issue at the same time, so while our BGP routes were being advertised to world through Centurylink, the circuit was useless. Centurylink aknowledged the existence of a bigger transport issue and said that we weren't the only customer affected.
Anybody else notice these issues or have any other insight?
Thanks!
Eric Miller
-- ---- ---- ---- ---- Chris Gotstein, Network Engineer, U.P. Logon/Computer Connection U.P. http://uplogon.com | +1 906 774 4847 | chris@uplogon.com
-- ---- ---- ---- ---- Chris Gotstein, Network Engineer, U.P. Logon/Computer Connection U.P. http://uplogon.com | +1 906 774 4847 | chris@uplogon.com
Where is this outages list? On Apr 24, 2012, at 10:36 AM, Chris Gotstein wrote:
Already on outages.
On 4/24/2012 11:28 AM, Christopher Morrow wrote:
this belongs on outages@ no?
On Tue, Apr 24, 2012 at 12:14 PM, Chris Gotstein<chris@uplogon.com> wrote:
CenturyLink is reporting core routing issues at this time. Seeing the same issues with our DS-3, BGP stayed up, but traffic was not passing over the line. Had to manually shutdown the interface to get traffic flowing over our other providers link. What a mess.
On 4/24/2012 8:22 AM, Eric C. Miller wrote:
Morning Everyone,
Yesterday between about 1900 and 2230 UTC, we had a partial drop with reaching various sites through TW Telecom from our circuit in Orlando, FL. The unavailable sites included Facebook, Newegg, and Godaddy. The outage did not affect our Atlanta TW Telecom. I confered with a colleague who manages a large customer in Apopka who said that they appeared not to be affected. His circuit and ours loop to the same TW Telecom POP.
But even more Murphy than that, our Centurylink secondary circuit was having a routing loop issue at the same time, so while our BGP routes were being advertised to world through Centurylink, the circuit was useless. Centurylink aknowledged the existence of a bigger transport issue and said that we weren't the only customer affected.
Anybody else notice these issues or have any other insight?
Thanks!
Eric Miller
-- ---- ---- ---- ---- Chris Gotstein, Network Engineer, U.P. Logon/Computer Connection U.P. http://uplogon.com | +1 906 774 4847 | chris@uplogon.com
-- ---- ---- ---- ---- Chris Gotstein, Network Engineer, U.P. Logon/Computer Connection U.P. http://uplogon.com | +1 906 774 4847 | chris@uplogon.com
On 4/24/12 9:37 AM, Bret Palsson wrote:
Where is this outages list?
Yesterday at about 3 pm PDT DNS resolution problems were experienced through Centurylink. Apparently their Phoenix DNS servers were unreachable for some time. These types of incidents never happened with Qwest. Anyone else report a service degradation since Centurylink took over? On Tue, Apr 24, 2012 at 6:22 AM, Eric C. Miller <eric@ericheather.com>wrote:
Morning Everyone,
Yesterday between about 1900 and 2230 UTC, we had a partial drop with reaching various sites through TW Telecom from our circuit in Orlando, FL. The unavailable sites included Facebook, Newegg, and Godaddy. The outage did not affect our Atlanta TW Telecom. I confered with a colleague who manages a large customer in Apopka who said that they appeared not to be affected. His circuit and ours loop to the same TW Telecom POP.
But even more Murphy than that, our Centurylink secondary circuit was having a routing loop issue at the same time, so while our BGP routes were being advertised to world through Centurylink, the circuit was useless. Centurylink aknowledged the existence of a bigger transport issue and said that we weren't the only customer affected.
Anybody else notice these issues or have any other insight?
Thanks!
Eric Miller
We have a ticket open with Level3. Our customers on the the west coast using CenturyLink are not receiving traffic. -Bret On Apr 24, 2012, at 10:35 AM, david peahi wrote:
Yesterday at about 3 pm PDT DNS resolution problems were experienced through Centurylink. Apparently their Phoenix DNS servers were unreachable for some time. These types of incidents never happened with Qwest. Anyone else report a service degradation since Centurylink took over?
On Tue, Apr 24, 2012 at 6:22 AM, Eric C. Miller <eric@ericheather.com>wrote:
Morning Everyone,
Yesterday between about 1900 and 2230 UTC, we had a partial drop with reaching various sites through TW Telecom from our circuit in Orlando, FL. The unavailable sites included Facebook, Newegg, and Godaddy. The outage did not affect our Atlanta TW Telecom. I confered with a colleague who manages a large customer in Apopka who said that they appeared not to be affected. His circuit and ours loop to the same TW Telecom POP.
But even more Murphy than that, our Centurylink secondary circuit was having a routing loop issue at the same time, so while our BGP routes were being advertised to world through Centurylink, the circuit was useless. Centurylink aknowledged the existence of a bigger transport issue and said that we weren't the only customer affected.
Anybody else notice these issues or have any other insight?
Thanks!
Eric Miller
participants (6)
-
Bret Palsson -
Chris Gotstein -
Christopher Morrow -
david peahi -
Eric C. Miller -
Seth Mattinen