Hey all, anyone else having issues with Time Warner residential or business connections? One of our offices is down and the route is not currently in bgp. http://downdetector.com/status/time-warner-cable shows thousands of reports of outages on the consumer side starting an hour or so ago so I figure it's a larger issue than just my one office; couldn't reach anyone by phone. Thanks, David
David, I have a branch office in Syracuse,NY that appears to be down at the moment that uses a time warner business connection for internet access. -rob On Aug 27, 2014 5:20 AM, "David Hubbard" <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone else having issues with Time Warner residential or business connections? One of our offices is down and the route is not currently in bgp. http://downdetector.com/status/time-warner-cable shows thousands of reports of outages on the consumer side starting an hour or so ago so I figure it's a larger issue than just my one office; couldn't reach anyone by phone.
Thanks,
David
My whole campus (~10000 users) is down... Since roughly 6am. TWC is our upstream. -- Sent from my iPhone
On Aug 27, 2014, at 6:28 AM, Rob Barbeau <rob.barbeau@gmail.com> wrote:
David,
I have a branch office in Syracuse,NY that appears to be down at the moment that uses a time warner business connection for internet access.
-rob On Aug 27, 2014 5:20 AM, "David Hubbard" <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone else having issues with Time Warner residential or business connections? One of our offices is down and the route is not currently in bgp. http://downdetector.com/status/time-warner-cable shows thousands of reports of outages on the consumer side starting an hour or so ago so I figure it's a larger issue than just my one office; couldn't reach anyone by phone.
Thanks,
David
Same here. Seems like no traffic is exiting TWC: Tracing route to ns03.savvis.net [204.70.25.234] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms 10049.webjogger.net [204.8.80.49] 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com [24.29.112.25] 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com [69.193.225.74] 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com [69.193.225.137] 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com [69.193.225.222] 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rick Coloccia Sent: Wednesday, August 27, 2014 6:43 AM To: nanog@nanog.org Subject: Re: Time Warner outage? My whole campus (~10000 users) is down... Since roughly 6am. TWC is our upstream. -- Sent from my iPhone
On Aug 27, 2014, at 6:28 AM, Rob Barbeau <rob.barbeau@gmail.com> wrote:
David,
I have a branch office in Syracuse,NY that appears to be down at the moment that uses a time warner business connection for internet access.
-rob On Aug 27, 2014 5:20 AM, "David Hubbard" <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone else having issues with Time Warner residential or business connections? One of our offices is down and the route is not currently in bgp. http://downdetector.com/status/time-warner-cable shows thousands of reports of outages on the consumer side starting an hour or so ago so I figure it's a larger issue than just my one office; couldn't reach anyone by phone.
Thanks,
David
BGPMON shows my routes falling off the net at around 5:49am. We now sit at their mercy.... -- Sent from my iPhone
On Aug 27, 2014, at 6:46 AM, "Adam Greene" <maillist@webjogger.net> wrote:
Same here. Seems like no traffic is exiting TWC:
Tracing route to ns03.savvis.net [204.70.25.234] over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 10049.webjogger.net [204.8.80.49] 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com [24.29.112.25] 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com [69.193.225.74] 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com [69.193.225.137] 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com [69.193.225.222] 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out.
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rick Coloccia Sent: Wednesday, August 27, 2014 6:43 AM To: nanog@nanog.org Subject: Re: Time Warner outage?
My whole campus (~10000 users) is down... Since roughly 6am. TWC is our upstream.
-- Sent from my iPhone
On Aug 27, 2014, at 6:28 AM, Rob Barbeau <rob.barbeau@gmail.com> wrote:
David,
I have a branch office in Syracuse,NY that appears to be down at the moment that uses a time warner business connection for internet access.
-rob On Aug 27, 2014 5:20 AM, "David Hubbard" <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone else having issues with Time Warner residential or business connections? One of our offices is down and the route is not currently in bgp. http://downdetector.com/status/time-warner-cable shows thousands of reports of outages on the consumer side starting an hour or so ago so I figure it's a larger issue than just my one office; couldn't reach anyone by phone.
Thanks,
David
Just came back up for me. Sent from my iPad
On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia@geneseo.edu> wrote:
BGPMON shows my routes falling off the net at around 5:49am.
We now sit at their mercy....
-- Sent from my iPhone
On Aug 27, 2014, at 6:46 AM, "Adam Greene" <maillist@webjogger.net> wrote:
Same here. Seems like no traffic is exiting TWC:
Tracing route to ns03.savvis.net [204.70.25.234] over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 10049.webjogger.net [204.8.80.49] 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com [24.29.112.25] 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com [69.193.225.74] 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com [69.193.225.137] 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com [69.193.225.222] 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out.
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rick Coloccia Sent: Wednesday, August 27, 2014 6:43 AM To: nanog@nanog.org Subject: Re: Time Warner outage?
My whole campus (~10000 users) is down... Since roughly 6am. TWC is our upstream.
-- Sent from my iPhone
On Aug 27, 2014, at 6:28 AM, Rob Barbeau <rob.barbeau@gmail.com> wrote:
David,
I have a branch office in Syracuse,NY that appears to be down at the moment that uses a time warner business connection for internet access.
-rob On Aug 27, 2014 5:20 AM, "David Hubbard" <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone else having issues with Time Warner residential or business connections? One of our offices is down and the route is not currently in bgp. http://downdetector.com/status/time-warner-cable shows thousands of reports of outages on the consumer side starting an hour or so ago so I figure it's a larger issue than just my one office; couldn't reach anyone by phone.
Thanks,
David
Came back for us, too, at about 7am. Outage was a biggie this time: https://news.yahoo.com/time-warner-cable-suffering-massive-nationwide-outage -110549606.html -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of David Coulson Sent: Wednesday, August 27, 2014 6:59 AM To: Rick Coloccia Cc: <nanog@nanog.org> Subject: Re: Time Warner outage? Just came back up for me. Sent from my iPad
On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia@geneseo.edu> wrote:
BGPMON shows my routes falling off the net at around 5:49am.
We now sit at their mercy....
-- Sent from my iPhone
On Aug 27, 2014, at 6:46 AM, "Adam Greene" <maillist@webjogger.net> wrote:
Same here. Seems like no traffic is exiting TWC:
Tracing route to ns03.savvis.net [204.70.25.234] over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 10049.webjogger.net [204.8.80.49] 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com [24.29.112.25] 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com [69.193.225.74] 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com [69.193.225.137] 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com [69.193.225.222] 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out.
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rick Coloccia Sent: Wednesday, August 27, 2014 6:43 AM To: nanog@nanog.org Subject: Re: Time Warner outage?
My whole campus (~10000 users) is down... Since roughly 6am. TWC is our upstream.
-- Sent from my iPhone
On Aug 27, 2014, at 6:28 AM, Rob Barbeau <rob.barbeau@gmail.com> wrote:
David,
I have a branch office in Syracuse,NY that appears to be down at the moment that uses a time warner business connection for internet access.
-rob On Aug 27, 2014 5:20 AM, "David Hubbard" <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone else having issues with Time Warner residential or business connections? One of our offices is down and the route is not currently in bgp. http://downdetector.com/status/time-warner-cable shows thousands of reports of outages on the consumer side starting an hour or so ago so I figure it's a larger issue than just my one office; couldn't reach anyone by phone.
Thanks,
David
Can't wait to see the postmortem report on this one. On 8/27/2014 3:05 PM, Adam Greene wrote:
Came back for us, too, at about 7am.
Outage was a biggie this time:
https://news.yahoo.com/time-warner-cable-suffering-massive-nationwide-outage -110549606.html
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of David Coulson Sent: Wednesday, August 27, 2014 6:59 AM To: Rick Coloccia Cc: <nanog@nanog.org> Subject: Re: Time Warner outage?
Just came back up for me.
Sent from my iPad
On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia@geneseo.edu> wrote:
BGPMON shows my routes falling off the net at around 5:49am.
We now sit at their mercy....
-- Sent from my iPhone
On Aug 27, 2014, at 6:46 AM, "Adam Greene" <maillist@webjogger.net> wrote: Same here. Seems like no traffic is exiting TWC:
Tracing route to ns03.savvis.net [204.70.25.234] over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 10049.webjogger.net [204.8.80.49] 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com [24.29.112.25] 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com [69.193.225.74] 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com [69.193.225.137] 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com [69.193.225.222] 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out.
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rick Coloccia Sent: Wednesday, August 27, 2014 6:43 AM To: nanog@nanog.org Subject: Re: Time Warner outage?
My whole campus (~10000 users) is down... Since roughly 6am. TWC is our upstream.
-- Sent from my iPhone
On Aug 27, 2014, at 6:28 AM, Rob Barbeau <rob.barbeau@gmail.com> wrote:
David,
I have a branch office in Syracuse,NY that appears to be down at the moment that uses a time warner business connection for internet access.
-rob On Aug 27, 2014 5:20 AM, "David Hubbard" <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone else having issues with Time Warner residential or business connections? One of our offices is down and the route is not currently in bgp. http://downdetector.com/status/time-warner-cable shows thousands of reports of outages on the consumer side starting an hour or so ago so I figure it's a larger issue than just my one office; couldn't reach anyone by phone.
Thanks,
David
I believe this is what is commonly referred to in our industry as a “resume generating event”. On Aug 27, 2014, at 8:24 AM, Alexandru Suciu <alex.suciu.sum@gmail.com> wrote:
Can't wait to see the postmortem report on this one.
On 8/27/2014 3:05 PM, Adam Greene wrote:
Came back for us, too, at about 7am.
Outage was a biggie this time:
https://news.yahoo.com/time-warner-cable-suffering-massive-nationwide-outage -110549606.html
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of David Coulson Sent: Wednesday, August 27, 2014 6:59 AM To: Rick Coloccia Cc: <nanog@nanog.org> Subject: Re: Time Warner outage?
Just came back up for me.
Sent from my iPad
On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia@geneseo.edu> wrote:
BGPMON shows my routes falling off the net at around 5:49am.
We now sit at their mercy....
-- Sent from my iPhone
On Aug 27, 2014, at 6:46 AM, "Adam Greene" <maillist@webjogger.net> wrote: Same here. Seems like no traffic is exiting TWC:
Tracing route to ns03.savvis.net [204.70.25.234] over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 10049.webjogger.net [204.8.80.49] 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com [24.29.112.25] 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com [69.193.225.74] 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com [69.193.225.137] 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com [69.193.225.222] 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out.
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rick Coloccia Sent: Wednesday, August 27, 2014 6:43 AM To: nanog@nanog.org Subject: Re: Time Warner outage?
My whole campus (~10000 users) is down... Since roughly 6am. TWC is our upstream.
-- Sent from my iPhone
On Aug 27, 2014, at 6:28 AM, Rob Barbeau <rob.barbeau@gmail.com> wrote:
David,
I have a branch office in Syracuse,NY that appears to be down at the moment that uses a time warner business connection for internet access.
-rob On Aug 27, 2014 5:20 AM, "David Hubbard" <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone else having issues with Time Warner residential or business connections? One of our offices is down and the route is not currently in bgp. http://downdetector.com/status/time-warner-cable shows thousands of reports of outages on the consumer side starting an hour or so ago so I figure it's a larger issue than just my one office; couldn't reach anyone by phone.
Thanks,
David
This just keeps getting better and better: "Yahoo Logo Will be right back... Thank you for your patience. Our engineers are working quickly to resolve the issue." My upstream is Charter Business... On 08/28/2014 04:05 AM, Chris Garrett wrote:
I believe this is what is commonly referred to in our industry as a “resume generating event”.
On Aug 27, 2014, at 8:24 AM, Alexandru Suciu <alex.suciu.sum@gmail.com> wrote:
Can't wait to see the postmortem report on this one.
On 8/27/2014 3:05 PM, Adam Greene wrote:
Came back for us, too, at about 7am.
Outage was a biggie this time:
https://news.yahoo.com/time-warner-cable-suffering-massive-nationwide-outage -110549606.html
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of David Coulson Sent: Wednesday, August 27, 2014 6:59 AM To: Rick Coloccia Cc: <nanog@nanog.org> Subject: Re: Time Warner outage?
Just came back up for me.
Sent from my iPad
On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia@geneseo.edu> wrote:
BGPMON shows my routes falling off the net at around 5:49am.
We now sit at their mercy....
-- Sent from my iPhone
On Aug 27, 2014, at 6:46 AM, "Adam Greene" <maillist@webjogger.net> wrote: Same here. Seems like no traffic is exiting TWC:
Tracing route to ns03.savvis.net [204.70.25.234] over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 10049.webjogger.net [204.8.80.49] 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com [24.29.112.25] 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com [69.193.225.74] 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com [69.193.225.137] 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com [69.193.225.222] 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out.
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rick Coloccia Sent: Wednesday, August 27, 2014 6:43 AM To: nanog@nanog.org Subject: Re: Time Warner outage?
My whole campus (~10000 users) is down... Since roughly 6am. TWC is our upstream.
-- Sent from my iPhone
On Aug 27, 2014, at 6:28 AM, Rob Barbeau <rob.barbeau@gmail.com> wrote:
David,
I have a branch office in Syracuse,NY that appears to be down at the moment that uses a time warner business connection for internet access.
-rob On Aug 27, 2014 5:20 AM, "David Hubbard" <dhubbard@dino.hostasaurus.com> wrote:
> Hey all, anyone else having issues with Time Warner residential or > business connections? One of our offices is down and the route is > not currently in bgp. > http://downdetector.com/status/time-warner-cable > shows thousands of reports of outages on the consumer side starting > an hour or so ago so I figure it's a larger issue than just my one > office; couldn't reach anyone by phone. > > Thanks, > > David
I heard the following, It was actually an engineer doing maintenance on the dhpc server that stopped customers from getting an IP address when the connected to the network between 5:30 and 7:00. The funny part is upper Managment heard about it on the today show On Thu, Aug 28, 2014 at 8:46 AM, Stephen Satchell <list@satchell.net> wrote:
This just keeps getting better and better:
"Yahoo Logo Will be right back...
Thank you for your patience.
Our engineers are working quickly to resolve the issue."
My upstream is Charter Business...
On 08/28/2014 04:05 AM, Chris Garrett wrote:
I believe this is what is commonly referred to in our industry as a “resume generating event”.
On Aug 27, 2014, at 8:24 AM, Alexandru Suciu <alex.suciu.sum@gmail.com> wrote:
Can't wait to see the postmortem report on this one.
On 8/27/2014 3:05 PM, Adam Greene wrote:
Came back for us, too, at about 7am.
Outage was a biggie this time:
https://news.yahoo.com/time-warner-cable-suffering-massive-nationwide-outage
-110549606.html
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of David Coulson Sent: Wednesday, August 27, 2014 6:59 AM To: Rick Coloccia Cc: <nanog@nanog.org> Subject: Re: Time Warner outage?
Just came back up for me.
Sent from my iPad
On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia@geneseo.edu> wrote:
BGPMON shows my routes falling off the net at around 5:49am.
We now sit at their mercy....
-- Sent from my iPhone
On Aug 27, 2014, at 6:46 AM, "Adam Greene" <maillist@webjogger.net> wrote: Same here. Seems like no traffic is exiting TWC:
Tracing route to ns03.savvis.net [204.70.25.234] over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 10049.webjogger.net [204.8.80.49] 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com [24.29.112.25] 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com [69.193.225.74] 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com [69.193.225.137] 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com [69.193.225.222] 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out.
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rick Coloccia Sent: Wednesday, August 27, 2014 6:43 AM To: nanog@nanog.org Subject: Re: Time Warner outage?
My whole campus (~10000 users) is down... Since roughly 6am. TWC is our upstream.
-- Sent from my iPhone
> On Aug 27, 2014, at 6:28 AM, Rob Barbeau <rob.barbeau@gmail.com> wrote: > > David, > > I have a branch office in Syracuse,NY that appears to be down at the > moment that uses a time warner business connection for internet access. > > -rob > On Aug 27, 2014 5:20 AM, "David Hubbard" > <dhubbard@dino.hostasaurus.com> > wrote: > >> Hey all, anyone else having issues with Time Warner residential or >> business connections? One of our offices is down and the route is >> not currently in bgp. >> http://downdetector.com/status/time-warner-cable >> shows thousands of reports of outages on the consumer side starting >> an hour or so ago so I figure it's a larger issue than just my one >> office; couldn't reach anyone by phone. >> >> Thanks, >> >> David
-- //CL
Something sounds really unlikely about that. Lack of DHCP would not cause reachability problems except for the clients. The trace below looks like a transit connection that should be unaffected by DHCP. Looks more like a routing issue. Also sounds unlikely that one DHCP server would be covering this large a scope. Steven Naslund Chicago IL -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Chris Lane Sent: Thursday, August 28, 2014 8:23 AM To: Stephen Satchell Cc: nanog@nanog.org Subject: Re: Time Warner outage? I heard the following, It was actually an engineer doing maintenance on the dhpc server that stopped customers from getting an IP address when the connected to the network between 5:30 and 7:00. The funny part is upper Managment heard about it on the today show On Thu, Aug 28, 2014 at 8:46 AM, Stephen Satchell <list@satchell.net> wrote:
This just keeps getting better and better:
"Yahoo Logo Will be right back...
Thank you for your patience.
Our engineers are working quickly to resolve the issue."
My upstream is Charter Business...
On 08/28/2014 04:05 AM, Chris Garrett wrote:
I believe this is what is commonly referred to in our industry as a “resume generating event”.
On Aug 27, 2014, at 8:24 AM, Alexandru Suciu <alex.suciu.sum@gmail.com> wrote:
Can't wait to see the postmortem report on this one.
On 8/27/2014 3:05 PM, Adam Greene wrote:
Came back for us, too, at about 7am.
Outage was a biggie this time:
https://news.yahoo.com/time-warner-cable-suffering-massive-nationwide- outage
-110549606.html
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of David Coulson Sent: Wednesday, August 27, 2014 6:59 AM To: Rick Coloccia Cc: <nanog@nanog.org> Subject: Re: Time Warner outage?
Just came back up for me.
Sent from my iPad
On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia@geneseo.edu> wrote:
BGPMON shows my routes falling off the net at around 5:49am.
We now sit at their mercy....
-- Sent from my iPhone
On Aug 27, 2014, at 6:46 AM, "Adam Greene" <maillist@webjogger.net> wrote: Same here. Seems like no traffic is exiting TWC:
Tracing route to ns03.savvis.net [204.70.25.234] over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 10049.webjogger.net [204.8.80.49] 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com [24.29.112.25] 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com [69.193.225.74] 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com [69.193.225.137] 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com [69.193.225.222] 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out.
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rick Coloccia Sent: Wednesday, August 27, 2014 6:43 AM To: nanog@nanog.org Subject: Re: Time Warner outage?
My whole campus (~10000 users) is down... Since roughly 6am. TWC is our upstream.
-- Sent from my iPhone
> On Aug 27, 2014, at 6:28 AM, Rob Barbeau > <rob.barbeau@gmail.com> wrote: > > David, > > I have a branch office in Syracuse,NY that appears to be down > at the moment that uses a time warner business connection for > internet access. > > -rob > On Aug 27, 2014 5:20 AM, "David Hubbard" > <dhubbard@dino.hostasaurus.com> > wrote: > >> Hey all, anyone else having issues with Time Warner >> residential or business connections? One of our offices is >> down and the route is not currently in bgp. >> http://downdetector.com/status/time-warner-cable >> shows thousands of reports of outages on the consumer side >> starting an hour or so ago so I figure it's a larger issue >> than just my one office; couldn't reach anyone by phone. >> >> Thanks, >> >> David
-- //CL
I have never seen BGP drop peers because of a DHCP failure. I would love to see the breakdown on that. TW NOC had said it was a Level 3 fiber maintenance gone bad that took down their peering connections when we called in at the time of the event. Can anyone confirm one way or the other? On Aug 28, 2014, at 8:29 AM, Naslund, Steve <SNaslund@medline.com> wrote:
Something sounds really unlikely about that. Lack of DHCP would not cause reachability problems except for the clients. The trace below looks like a transit connection that should be unaffected by DHCP. Looks more like a routing issue. Also sounds unlikely that one DHCP server would be covering this large a scope.
Steven Naslund Chicago IL
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Chris Lane Sent: Thursday, August 28, 2014 8:23 AM To: Stephen Satchell Cc: nanog@nanog.org Subject: Re: Time Warner outage?
I heard the following,
It was actually an engineer doing maintenance on the dhpc server that stopped customers from getting an IP address when the connected to the network between 5:30 and 7:00. The funny part is upper Managment heard about it on the today show
On Thu, Aug 28, 2014 at 8:46 AM, Stephen Satchell <list@satchell.net> wrote:
This just keeps getting better and better:
"Yahoo Logo Will be right back...
Thank you for your patience.
Our engineers are working quickly to resolve the issue."
My upstream is Charter Business...
On 08/28/2014 04:05 AM, Chris Garrett wrote:
I believe this is what is commonly referred to in our industry as a “resume generating event”.
On Aug 27, 2014, at 8:24 AM, Alexandru Suciu <alex.suciu.sum@gmail.com> wrote:
Can't wait to see the postmortem report on this one.
On 8/27/2014 3:05 PM, Adam Greene wrote:
Came back for us, too, at about 7am.
Outage was a biggie this time:
https://news.yahoo.com/time-warner-cable-suffering-massive-nationwide- outage
-110549606.html
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of David Coulson Sent: Wednesday, August 27, 2014 6:59 AM To: Rick Coloccia Cc: <nanog@nanog.org> Subject: Re: Time Warner outage?
Just came back up for me.
Sent from my iPad
On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia@geneseo.edu> wrote:
BGPMON shows my routes falling off the net at around 5:49am.
We now sit at their mercy....
-- Sent from my iPhone
> On Aug 27, 2014, at 6:46 AM, "Adam Greene" > <maillist@webjogger.net> wrote: > Same here. Seems like no traffic is exiting TWC: > > Tracing route to ns03.savvis.net [204.70.25.234] over a maximum > of 30 > hops: > > 1 1 ms 1 ms 1 ms 10049.webjogger.net [204.8.80.49] > 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com > [24.29.112.25] > 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com > [69.193.225.74] > 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com > [69.193.225.137] > 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com > [69.193.225.222] > 6 * * * Request timed out. > 7 * * * Request timed out. > 8 * * * Request timed out. > 9 * * * Request timed out. > > -----Original Message----- > From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rick > Coloccia > Sent: Wednesday, August 27, 2014 6:43 AM > To: nanog@nanog.org > Subject: Re: Time Warner outage? > > My whole campus (~10000 users) is down... Since roughly 6am. > TWC is our upstream. > > -- > Sent from my iPhone > >> On Aug 27, 2014, at 6:28 AM, Rob Barbeau >> <rob.barbeau@gmail.com> wrote: >> >> David, >> >> I have a branch office in Syracuse,NY that appears to be down >> at the moment that uses a time warner business connection for >> internet access. >> >> -rob >> On Aug 27, 2014 5:20 AM, "David Hubbard" >> <dhubbard@dino.hostasaurus.com> >> wrote: >> >>> Hey all, anyone else having issues with Time Warner >>> residential or business connections? One of our offices is >>> down and the route is not currently in bgp. >>> http://downdetector.com/status/time-warner-cable >>> shows thousands of reports of outages on the consumer side >>> starting an hour or so ago so I figure it's a larger issue >>> than just my one office; couldn't reach anyone by phone. >>> >>> Thanks, >>> >>> David
-- //CL
Sounds more likely than "we broke the dhcp server". Sent from my T-Mobile 4G LTE Device -------- Original message -------- From: Chris Garrett <chris@aperturefiber.com> Date: 08/28/2014 6:39 AM (GMT-08:00) To: "Naslund, Steve" <SNaslund@medline.com> Cc: nanog@nanog.org Subject: Re: Time Warner outage? I have never seen BGP drop peers because of a DHCP failure. I would love to see the breakdown on that. TW NOC had said it was a Level 3 fiber maintenance gone bad that took down their peering connections when we called in at the time of the event. Can anyone confirm one way or the other? On Aug 28, 2014, at 8:29 AM, Naslund, Steve <SNaslund@medline.com> wrote:
Something sounds really unlikely about that. Lack of DHCP would not cause reachability problems except for the clients. The trace below looks like a transit connection that should be unaffected by DHCP. Looks more like a routing issue. Also sounds unlikely that one DHCP server would be covering this large a scope.
Steven Naslund Chicago IL
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Chris Lane Sent: Thursday, August 28, 2014 8:23 AM To: Stephen Satchell Cc: nanog@nanog.org Subject: Re: Time Warner outage?
I heard the following,
It was actually an engineer doing maintenance on the dhpc server that stopped customers from getting an IP address when the connected to the network between 5:30 and 7:00. The funny part is upper Managment heard about it on the today show
On Thu, Aug 28, 2014 at 8:46 AM, Stephen Satchell <list@satchell.net> wrote:
This just keeps getting better and better:
"Yahoo Logo Will be right back...
Thank you for your patience.
Our engineers are working quickly to resolve the issue."
My upstream is Charter Business...
On 08/28/2014 04:05 AM, Chris Garrett wrote:
I believe this is what is commonly referred to in our industry as a “resume generating event”.
On Aug 27, 2014, at 8:24 AM, Alexandru Suciu <alex.suciu.sum@gmail.com> wrote:
Can't wait to see the postmortem report on this one.
On 8/27/2014 3:05 PM, Adam Greene wrote:
Came back for us, too, at about 7am.
Outage was a biggie this time:
https://news.yahoo.com/time-warner-cable-suffering-massive-nationwide- outage
-110549606.html
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of David Coulson Sent: Wednesday, August 27, 2014 6:59 AM To: Rick Coloccia Cc: <nanog@nanog.org> Subject: Re: Time Warner outage?
Just came back up for me.
Sent from my iPad
On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia@geneseo.edu> wrote:
BGPMON shows my routes falling off the net at around 5:49am.
We now sit at their mercy....
-- Sent from my iPhone
> On Aug 27, 2014, at 6:46 AM, "Adam Greene" > <maillist@webjogger.net> wrote: > Same here. Seems like no traffic is exiting TWC: > > Tracing route to ns03.savvis.net [204.70.25.234] over a maximum > of 30 > hops: > > 1 1 ms 1 ms 1 ms 10049.webjogger.net [204.8.80.49] > 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com > [24.29.112.25] > 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com > [69.193.225.74] > 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com > [69.193.225.137] > 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com > [69.193.225.222] > 6 * * * Request timed out. > 7 * * * Request timed out. > 8 * * * Request timed out. > 9 * * * Request timed out. > > -----Original Message----- > From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rick > Coloccia > Sent: Wednesday, August 27, 2014 6:43 AM > To: nanog@nanog.org > Subject: Re: Time Warner outage? > > My whole campus (~10000 users) is down... Since roughly 6am. > TWC is our upstream. > > -- > Sent from my iPhone > >> On Aug 27, 2014, at 6:28 AM, Rob Barbeau >> <rob.barbeau@gmail.com> wrote: >> >> David, >> >> I have a branch office in Syracuse,NY that appears to be down >> at the moment that uses a time warner business connection for >> internet access. >> >> -rob >> On Aug 27, 2014 5:20 AM, "David Hubbard" >> <dhubbard@dino.hostasaurus.com> >> wrote: >> >>> Hey all, anyone else having issues with Time Warner >>> residential or business connections? One of our offices is >>> down and the route is not currently in bgp. >>> http://downdetector.com/status/time-warner-cable >>> shows thousands of reports of outages on the consumer side >>> starting an hour or so ago so I figure it's a larger issue >>> than just my one office; couldn't reach anyone by phone. >>> >>> Thanks, >>> >>> David
-- //CL
Agreed on DHCP, just passing along something i had heard about..... With that said, why wouldn't the TW guys just post something oh right they did, blame the other guy "Level3" On Thu, Aug 28, 2014 at 9:42 AM, Warren Bailey < wbailey@satelliteintelligencegroup.com> wrote:
Sounds more likely than "we broke the dhcp server".
Sent from my T-Mobile 4G LTE Device
-------- Original message -------- From: Chris Garrett <chris@aperturefiber.com> Date: 08/28/2014 6:39 AM (GMT-08:00) To: "Naslund, Steve" <SNaslund@medline.com> Cc: nanog@nanog.org Subject: Re: Time Warner outage?
I have never seen BGP drop peers because of a DHCP failure.
I would love to see the breakdown on that.
TW NOC had said it was a Level 3 fiber maintenance gone bad that took down their peering connections when we called in at the time of the event.
Can anyone confirm one way or the other?
On Aug 28, 2014, at 8:29 AM, Naslund, Steve <SNaslund@medline.com> wrote:
Something sounds really unlikely about that. Lack of DHCP would not cause reachability problems except for the clients. The trace below looks like a transit connection that should be unaffected by DHCP. Looks more like a routing issue. Also sounds unlikely that one DHCP server would be covering this large a scope.
Steven Naslund Chicago IL
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Chris Lane Sent: Thursday, August 28, 2014 8:23 AM To: Stephen Satchell Cc: nanog@nanog.org Subject: Re: Time Warner outage?
I heard the following,
It was actually an engineer doing maintenance on the dhpc server that stopped customers from getting an IP address when the connected to the network between 5:30 and 7:00. The funny part is upper Managment heard about it on the today show
On Thu, Aug 28, 2014 at 8:46 AM, Stephen Satchell <list@satchell.net> wrote:
This just keeps getting better and better:
"Yahoo Logo Will be right back...
Thank you for your patience.
Our engineers are working quickly to resolve the issue."
My upstream is Charter Business...
On 08/28/2014 04:05 AM, Chris Garrett wrote:
I believe this is what is commonly referred to in our industry as a “resume generating event”.
On Aug 27, 2014, at 8:24 AM, Alexandru Suciu <alex.suciu.sum@gmail.com> wrote:
Can't wait to see the postmortem report on this one.
On 8/27/2014 3:05 PM, Adam Greene wrote:
Came back for us, too, at about 7am.
Outage was a biggie this time:
https://news.yahoo.com/time-warner-cable-suffering-massive-nationwide- outage
-110549606.html
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of David Coulson Sent: Wednesday, August 27, 2014 6:59 AM To: Rick Coloccia Cc: <nanog@nanog.org> Subject: Re: Time Warner outage?
Just came back up for me.
Sent from my iPad
> On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia@geneseo.edu> wrote: > > BGPMON shows my routes falling off the net at around 5:49am. > > We now sit at their mercy.... > > > -- > Sent from my iPhone > >> On Aug 27, 2014, at 6:46 AM, "Adam Greene" >> <maillist@webjogger.net> wrote: >> Same here. Seems like no traffic is exiting TWC: >> >> Tracing route to ns03.savvis.net [204.70.25.234] over a maximum >> of 30 >> hops: >> >> 1 1 ms 1 ms 1 ms 10049.webjogger.net [204.8.80.49] >> 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com >> [24.29.112.25] >> 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com >> [69.193.225.74] >> 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com >> [69.193.225.137] >> 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com >> [69.193.225.222] >> 6 * * * Request timed out. >> 7 * * * Request timed out. >> 8 * * * Request timed out. >> 9 * * * Request timed out. >> >> -----Original Message----- >> From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rick >> Coloccia >> Sent: Wednesday, August 27, 2014 6:43 AM >> To: nanog@nanog.org >> Subject: Re: Time Warner outage? >> >> My whole campus (~10000 users) is down... Since roughly 6am. >> TWC is our upstream. >> >> -- >> Sent from my iPhone >> >>> On Aug 27, 2014, at 6:28 AM, Rob Barbeau >>> <rob.barbeau@gmail.com> wrote: >>> >>> David, >>> >>> I have a branch office in Syracuse,NY that appears to be down >>> at the moment that uses a time warner business connection for >>> internet access. >>> >>> -rob >>> On Aug 27, 2014 5:20 AM, "David Hubbard" >>> <dhubbard@dino.hostasaurus.com> >>> wrote: >>> >>>> Hey all, anyone else having issues with Time Warner >>>> residential or business connections? One of our offices is >>>> down and the route is not currently in bgp. >>>> http://downdetector.com/status/time-warner-cable >>>> shows thousands of reports of outages on the consumer side >>>> starting an hour or so ago so I figure it's a larger issue >>>> than just my one office; couldn't reach anyone by phone. >>>> >>>> Thanks, >>>> >>>> David
-- //CL
-- //CL
I don’t buy that excuse either. If Level 3 is doing fiber maintenance on any route and takes down your entire network, then you have a pretty poor backbone design. It is not Level 3s fault if you design your network such that a single route loss causes a huge outage. Steven Naslund Chicago IL From: Chris Lane [mailto:clane1875@gmail.com] Sent: Thursday, August 28, 2014 8:51 AM To: Warren Bailey Cc: Chris Garrett; Naslund, Steve; nanog@nanog.org Subject: Re: Time Warner outage? Agreed on DHCP, just passing along something i had heard about..... With that said, why wouldn't the TW guys just post something oh right they did, blame the other guy "Level3" On Thu, Aug 28, 2014 at 9:42 AM, Warren Bailey <wbailey@satelliteintelligencegroup.com<mailto:wbailey@satelliteintelligencegroup.com>> wrote: Sounds more likely than "we broke the dhcp server". Sent from my T-Mobile 4G LTE Device -------- Original message -------- From: Chris Garrett <chris@aperturefiber.com<mailto:chris@aperturefiber.com>> Date: 08/28/2014 6:39 AM (GMT-08:00) To: "Naslund, Steve" <SNaslund@medline.com<mailto:SNaslund@medline.com>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Time Warner outage? I have never seen BGP drop peers because of a DHCP failure. I would love to see the breakdown on that. TW NOC had said it was a Level 3 fiber maintenance gone bad that took down their peering connections when we called in at the time of the event. Can anyone confirm one way or the other? On Aug 28, 2014, at 8:29 AM, Naslund, Steve <SNaslund@medline.com<mailto:SNaslund@medline.com>> wrote:
Something sounds really unlikely about that. Lack of DHCP would not cause reachability problems except for the clients. The trace below looks like a transit connection that should be unaffected by DHCP. Looks more like a routing issue. Also sounds unlikely that one DHCP server would be covering this large a scope.
Steven Naslund Chicago IL
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org<mailto:nanog-bounces@nanog.org>] On Behalf Of Chris Lane Sent: Thursday, August 28, 2014 8:23 AM To: Stephen Satchell Cc: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Time Warner outage?
I heard the following,
It was actually an engineer doing maintenance on the dhpc server that stopped customers from getting an IP address when the connected to the network between 5:30 and 7:00. The funny part is upper Managment heard about it on the today show
On Thu, Aug 28, 2014 at 8:46 AM, Stephen Satchell <list@satchell.net<mailto:list@satchell.net>> wrote:
This just keeps getting better and better:
"Yahoo Logo Will be right back...
Thank you for your patience.
Our engineers are working quickly to resolve the issue."
My upstream is Charter Business...
On 08/28/2014 04:05 AM, Chris Garrett wrote:
I believe this is what is commonly referred to in our industry as a “resume generating event”.
On Aug 27, 2014, at 8:24 AM, Alexandru Suciu <alex.suciu.sum@gmail.com<mailto:alex.suciu.sum@gmail.com>> wrote:
Can't wait to see the postmortem report on this one.
On 8/27/2014 3:05 PM, Adam Greene wrote:
Came back for us, too, at about 7am.
Outage was a biggie this time:
https://news.yahoo.com/time-warner-cable-suffering-massive-nationwide- outage
-110549606.html
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org<mailto:nanog-bounces@nanog.org>] On Behalf Of David Coulson Sent: Wednesday, August 27, 2014 6:59 AM To: Rick Coloccia Cc: <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Time Warner outage?
Just came back up for me.
Sent from my iPad
On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia@geneseo.edu<mailto:coloccia@geneseo.edu>> wrote:
BGPMON shows my routes falling off the net at around 5:49am.
We now sit at their mercy....
-- Sent from my iPhone
> On Aug 27, 2014, at 6:46 AM, "Adam Greene" > <maillist@webjogger.net<mailto:maillist@webjogger.net>> wrote: > Same here. Seems like no traffic is exiting TWC: > > Tracing route to ns03.savvis.net<http://ns03.savvis.net> [204.70.25.234<tel:%5B204.70.25.234>] over a maximum > of 30 > hops: > > 1 1 ms 1 ms 1 ms 10049.webjogger.net<http://10049.webjogger.net> [204.8.80.49] > 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com<http://cpe-24-29-112-25.nyc.res.rr.com> > [24.29.112.25] > 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com<http://rdc-69-193-225-74.nyc.bc.twcable.com> > [69.193.225.74] > 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com<http://rdc-69-193-225-137.nyc.bc.twcable.com> > [69.193.225.137] > 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com<http://rdc-69-193-225-222.nyc.bc.twcable.com> > [69.193.225.222] > 6 * * * Request timed out. > 7 * * * Request timed out. > 8 * * * Request timed out. > 9 * * * Request timed out. > > -----Original Message----- > From: NANOG [mailto:nanog-bounces@nanog.org<mailto:nanog-bounces@nanog.org>] On Behalf Of Rick > Coloccia > Sent: Wednesday, August 27, 2014 6:43 AM > To: nanog@nanog.org<mailto:nanog@nanog.org> > Subject: Re: Time Warner outage? > > My whole campus (~10000 users) is down... Since roughly 6am. > TWC is our upstream. > > -- > Sent from my iPhone > >> On Aug 27, 2014, at 6:28 AM, Rob Barbeau >> <rob.barbeau@gmail.com<mailto:rob.barbeau@gmail.com>> wrote: >> >> David, >> >> I have a branch office in Syracuse,NY that appears to be down >> at the moment that uses a time warner business connection for >> internet access. >> >> -rob >> On Aug 27, 2014 5:20 AM, "David Hubbard" >> <dhubbard@dino.hostasaurus.com<mailto:dhubbard@dino.hostasaurus.com>> >> wrote: >> >>> Hey all, anyone else having issues with Time Warner >>> residential or business connections? One of our offices is >>> down and the route is not currently in bgp. >>> http://downdetector.com/status/time-warner-cable >>> shows thousands of reports of outages on the consumer side >>> starting an hour or so ago so I figure it's a larger issue >>> than just my one office; couldn't reach anyone by phone. >>> >>> Thanks, >>> >>> David
-- //CL
-- //CL
As others have said, it's difficult to rationalize a situation where dhcp breaks and your existing routing breaks as well. Especially one that kills the Internet for a couple 100k subs. I'd shoot flames, but all too often I've been the guy working something like this after some maintenance became "service affecting".. Lol If you've ever been responsible for a large Internet connection with a lot of subs, have a beer for the guy answering emails from some no name middle manager about his new found PEP right now. ;) Sent from my T-Mobile 4G LTE Device -------- Original message -------- From: "Naslund, Steve" <SNaslund@medline.com> Date: 08/28/2014 7:06 AM (GMT-08:00) To: nanog@nanog.org Subject: RE: Time Warner outage? I don’t buy that excuse either. If Level 3 is doing fiber maintenance on any route and takes down your entire network, then you have a pretty poor backbone design. It is not Level 3s fault if you design your network such that a single route loss causes a huge outage. Steven Naslund Chicago IL From: Chris Lane [mailto:clane1875@gmail.com] Sent: Thursday, August 28, 2014 8:51 AM To: Warren Bailey Cc: Chris Garrett; Naslund, Steve; nanog@nanog.org Subject: Re: Time Warner outage? Agreed on DHCP, just passing along something i had heard about..... With that said, why wouldn't the TW guys just post something oh right they did, blame the other guy "Level3" On Thu, Aug 28, 2014 at 9:42 AM, Warren Bailey <wbailey@satelliteintelligencegroup.com<mailto:wbailey@satelliteintelligencegroup.com>> wrote: Sounds more likely than "we broke the dhcp server". Sent from my T-Mobile 4G LTE Device -------- Original message -------- From: Chris Garrett <chris@aperturefiber.com<mailto:chris@aperturefiber.com>> Date: 08/28/2014 6:39 AM (GMT-08:00) To: "Naslund, Steve" <SNaslund@medline.com<mailto:SNaslund@medline.com>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Time Warner outage? I have never seen BGP drop peers because of a DHCP failure. I would love to see the breakdown on that. TW NOC had said it was a Level 3 fiber maintenance gone bad that took down their peering connections when we called in at the time of the event. Can anyone confirm one way or the other? On Aug 28, 2014, at 8:29 AM, Naslund, Steve <SNaslund@medline.com<mailto:SNaslund@medline.com>> wrote:
Something sounds really unlikely about that. Lack of DHCP would not cause reachability problems except for the clients. The trace below looks like a transit connection that should be unaffected by DHCP. Looks more like a routing issue. Also sounds unlikely that one DHCP server would be covering this large a scope.
Steven Naslund Chicago IL
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org<mailto:nanog-bounces@nanog.org>] On Behalf Of Chris Lane Sent: Thursday, August 28, 2014 8:23 AM To: Stephen Satchell Cc: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Time Warner outage?
I heard the following,
It was actually an engineer doing maintenance on the dhpc server that stopped customers from getting an IP address when the connected to the network between 5:30 and 7:00. The funny part is upper Managment heard about it on the today show
On Thu, Aug 28, 2014 at 8:46 AM, Stephen Satchell <list@satchell.net<mailto:list@satchell.net>> wrote:
This just keeps getting better and better:
"Yahoo Logo Will be right back...
Thank you for your patience.
Our engineers are working quickly to resolve the issue."
My upstream is Charter Business...
On 08/28/2014 04:05 AM, Chris Garrett wrote:
I believe this is what is commonly referred to in our industry as a “resume generating event”.
On Aug 27, 2014, at 8:24 AM, Alexandru Suciu <alex.suciu.sum@gmail.com<mailto:alex.suciu.sum@gmail.com>> wrote:
Can't wait to see the postmortem report on this one.
On 8/27/2014 3:05 PM, Adam Greene wrote:
Came back for us, too, at about 7am.
Outage was a biggie this time:
https://news.yahoo.com/time-warner-cable-suffering-massive-nationwide- outage
-110549606.html
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org<mailto:nanog-bounces@nanog.org>] On Behalf Of David Coulson Sent: Wednesday, August 27, 2014 6:59 AM To: Rick Coloccia Cc: <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Time Warner outage?
Just came back up for me.
Sent from my iPad
On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia@geneseo.edu<mailto:coloccia@geneseo.edu>> wrote:
BGPMON shows my routes falling off the net at around 5:49am.
We now sit at their mercy....
-- Sent from my iPhone
> On Aug 27, 2014, at 6:46 AM, "Adam Greene" > <maillist@webjogger.net<mailto:maillist@webjogger.net>> wrote: > Same here. Seems like no traffic is exiting TWC: > > Tracing route to ns03.savvis.net<http://ns03.savvis.net> [204.70.25.234<tel:%5B204.70.25.234>] over a maximum > of 30 > hops: > > 1 1 ms 1 ms 1 ms 10049.webjogger.net<http://10049.webjogger.net> [204.8.80.49] > 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com<http://cpe-24-29-112-25.nyc.res.rr.com> > [24.29.112.25] > 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com<http://rdc-69-193-225-74.nyc.bc.twcable.com> > [69.193.225.74] > 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com<http://rdc-69-193-225-137.nyc.bc.twcable.com> > [69.193.225.137] > 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com<http://rdc-69-193-225-222.nyc.bc.twcable.com> > [69.193.225.222] > 6 * * * Request timed out. > 7 * * * Request timed out. > 8 * * * Request timed out. > 9 * * * Request timed out. > > -----Original Message----- > From: NANOG [mailto:nanog-bounces@nanog.org<mailto:nanog-bounces@nanog.org>] On Behalf Of Rick > Coloccia > Sent: Wednesday, August 27, 2014 6:43 AM > To: nanog@nanog.org<mailto:nanog@nanog.org> > Subject: Re: Time Warner outage? > > My whole campus (~10000 users) is down... Since roughly 6am. > TWC is our upstream. > > -- > Sent from my iPhone > >> On Aug 27, 2014, at 6:28 AM, Rob Barbeau >> <rob.barbeau@gmail.com<mailto:rob.barbeau@gmail.com>> wrote: >> >> David, >> >> I have a branch office in Syracuse,NY that appears to be down >> at the moment that uses a time warner business connection for >> internet access. >> >> -rob >> On Aug 27, 2014 5:20 AM, "David Hubbard" >> <dhubbard@dino.hostasaurus.com<mailto:dhubbard@dino.hostasaurus.com>> >> wrote: >> >>> Hey all, anyone else having issues with Time Warner >>> residential or business connections? One of our offices is >>> down and the route is not currently in bgp. >>> http://downdetector.com/status/time-warner-cable >>> shows thousands of reports of outages on the consumer side >>> starting an hour or so ago so I figure it's a larger issue >>> than just my one office; couldn't reach anyone by phone. >>> >>> Thanks, >>> >>> David
-- //CL
-- //CL
I am not cleared to give further details, but in the hopes of providing a little more accurate info, I can point you at the following blog post http://www.twcableuntangled.com/2014/08/twc-identifies-cause-of-internet-ou tage/ Thanks, Wes George Anything below this line has been added by my company’s mail server, I have no control over it. ----------- This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.
----- Original Message -----
From: "Steve Naslund" <SNaslund@medline.com>
I don’t buy that excuse either. If Level 3 is doing fiber maintenance on any route and takes down your entire network, then you have a pretty poor backbone design. It is not Level 3s fault if you design your network such that a single route loss causes a huge outage.
Does the T-Bone run over L3? I thought it was either owned or dark-leased fiber and their own routers... Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
Based on the link that Wes shared, sounds like someone made a BGP boo boo. Not that I routed damn near the whole internet down a customers DS3 or anything once, but I can see how it could happen and then propagate out of control before it was caught. On Aug 28, 2014, at 11:57 AM, Jay Ashworth <jra@baylink.com> wrote:
----- Original Message -----
From: "Steve Naslund" <SNaslund@medline.com>
I don’t buy that excuse either. If Level 3 is doing fiber maintenance on any route and takes down your entire network, then you have a pretty poor backbone design. It is not Level 3s fault if you design your network such that a single route loss causes a huge outage.
Does the T-Bone run over L3? I thought it was either owned or dark-leased fiber and their own routers...
Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
Looks like TW is having further issues this afternoon. Eric Stoltz Neovera On 08/28/2014 01:37 PM, Chris Garrett wrote:
Based on the link that Wes shared, sounds like someone made a BGP boo boo.
Not that I routed damn near the whole internet down a customers DS3 or anything once, but I can see how it could happen and then propagate out of control before it was caught.
On Aug 28, 2014, at 11:57 AM, Jay Ashworth <jra@baylink.com> wrote:
----- Original Message -----
From: "Steve Naslund" <SNaslund@medline.com> I don’t buy that excuse either. If Level 3 is doing fiber maintenance on any route and takes down your entire network, then you have a pretty poor backbone design. It is not Level 3s fault if you design your network such that a single route loss causes a huge outage. Does the T-Bone run over L3? I thought it was either owned or dark-leased fiber and their own routers...
Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
What we noticed on home RoadRunner and on business TWC fiber about 6:15 AM EDT on the 27th: We could access systems from one to another by IP address but not DNS. We could ping other systems by IP. Home DHCP was up but TWC DNS was down at home and at work on the business fiber. Couldn't even reach TWC DNS servers by IP address. I'm not sure why two different classes of systems, business and residential, thirty miles apart would both be down on DNS but up on routing since the DNS servers are on different subnets. (We use TWC business DNS as Forwarders.) It was acting like a DNS failure more than a routing failure for us. I vaguely remembered something about Cox DNS having an outage a few days earlier that someone in the Middle East took credit for but I can't find the article now. I was wondering if TWC was putting in a mitigation and got it wrong, or TWC did not put in a mitigation and the Middle Easters knocked on their door. My impression is that it was acting like all DNS traffic in and out of TWC was getting blocked but it cleared up too soon to perform further testing. Ray
Date: Thu, 28 Aug 2014 15:17:37 -0400 From: eric.stoltz@neovera.com To: nanog@nanog.org Subject: Re: Time Warner outage?
Looks like TW is having further issues this afternoon.
Eric Stoltz Neovera
On 08/28/2014 01:37 PM, Chris Garrett wrote:
Based on the link that Wes shared, sounds like someone made a BGP boo boo.
Not that I routed damn near the whole internet down a customers DS3 or anything once, but I can see how it could happen and then propagate out of control before it was caught.
On Aug 28, 2014, at 11:57 AM, Jay Ashworth <jra@baylink.com> wrote:
----- Original Message -----
From: "Steve Naslund" <SNaslund@medline.com> I don’t buy that excuse either. If Level 3 is doing fiber maintenance on any route and takes down your entire network, then you have a pretty poor backbone design. It is not Level 3s fault if you design your network such that a single route loss causes a huge outage. Does the T-Bone run over L3? I thought it was either owned or dark-leased fiber and their own routers...
Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
hehe... We call them 'Character Builder Events'. Everyone gets to find out how much pressure they can take, beak new grounds, what they are made off, etc etc. 'Shit Storms' come in waves..... typically these are not caused by singular events, but more like a 'Series of Unfortunate Events'. We don't wish this on any carrier, but sooner or later everyone has to go thru it. Faisal Imtiaz Snappy Internet & Telecom ----- Original Message -----
From: "Chris Garrett" <chris@aperturefiber.com> To: "Alexandru Suciu" <alex.suciu.sum@gmail.com> Cc: nanog@nanog.org Sent: Thursday, August 28, 2014 7:05:30 AM Subject: Re: Time Warner outage?
I believe this is what is commonly referred to in our industry as a “resume generating event”.
On Aug 27, 2014, at 8:24 AM, Alexandru Suciu <alex.suciu.sum@gmail.com> wrote:
Can't wait to see the postmortem report on this one.
On 8/27/2014 3:05 PM, Adam Greene wrote:
Came back for us, too, at about 7am.
Outage was a biggie this time:
https://news.yahoo.com/time-warner-cable-suffering-massive-nationwide-outage -110549606.html
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of David Coulson Sent: Wednesday, August 27, 2014 6:59 AM To: Rick Coloccia Cc: <nanog@nanog.org> Subject: Re: Time Warner outage?
Just came back up for me.
Sent from my iPad
On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia@geneseo.edu> wrote:
BGPMON shows my routes falling off the net at around 5:49am.
We now sit at their mercy....
-- Sent from my iPhone
On Aug 27, 2014, at 6:46 AM, "Adam Greene" <maillist@webjogger.net> wrote: Same here. Seems like no traffic is exiting TWC:
Tracing route to ns03.savvis.net [204.70.25.234] over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 10049.webjogger.net [204.8.80.49] 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com [24.29.112.25] 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com [69.193.225.74] 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com [69.193.225.137] 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com [69.193.225.222] 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out.
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rick Coloccia Sent: Wednesday, August 27, 2014 6:43 AM To: nanog@nanog.org Subject: Re: Time Warner outage?
My whole campus (~10000 users) is down... Since roughly 6am. TWC is our upstream.
-- Sent from my iPhone
On Aug 27, 2014, at 6:28 AM, Rob Barbeau <rob.barbeau@gmail.com> wrote:
David,
I have a branch office in Syracuse,NY that appears to be down at the moment that uses a time warner business connection for internet access.
-rob On Aug 27, 2014 5:20 AM, "David Hubbard" <dhubbard@dino.hostasaurus.com> wrote:
> Hey all, anyone else having issues with Time Warner residential or > business connections? One of our offices is down and the route is > not currently in bgp. > http://downdetector.com/status/time-warner-cable > shows thousands of reports of outages on the consumer side starting > an hour or so ago so I figure it's a larger issue than just my one > office; couldn't reach anyone by phone. > > Thanks, > > David
All of WNY seems to be dead. Craps out what looks like inside TWC, so looks like they lost something big. --- Sent from Boxer | http://getboxer.com<http://d.getboxer.com/1dmRL48> On August 27, 2014 at 6:43:12 AM EDT, Rick Coloccia <coloccia@geneseo.edu> wrote: My whole campus (~10000 users) is down... Since roughly 6am. TWC is our upstream. -- Sent from my iPhone
On Aug 27, 2014, at 6:28 AM, Rob Barbeau wrote:
David,
I have a branch office in Syracuse,NY that appears to be down at the moment that uses a time warner business connection for internet access.
-rob On Aug 27, 2014 5:20 AM, "David Hubbard" wrote:
Hey all, anyone else having issues with Time Warner residential or business connections? One of our offices is down and the route is not currently in bgp. http://downdetector.com/status/time-warner-cable shows thousands of reports of outages on the consumer side starting an hour or so ago so I figure it's a larger issue than just my one office; couldn't reach anyone by phone.
Thanks,
David
I've have residential twc in Cleveland. My router has an ip in the 104.139.34/24 network that isn't being advertised via bgp anymore either. I can still trace route out from here half a dozen hops, so seems like an edge/peering issue somewhere. Sent from my iPad
On Aug 27, 2014, at 6:17 AM, David Hubbard <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone else having issues with Time Warner residential or business connections? One of our offices is down and the route is not currently in bgp. http://downdetector.com/status/time-warner-cable shows thousands of reports of outages on the consumer side starting an hour or so ago so I figure it's a larger issue than just my one office; couldn't reach anyone by phone.
Thanks,
David
I had a CNN breaking news email on my phone about this when I woke up, their backbone apparently was down from 2 ish to 4 ish AM. On August 27, 2014 6:17:00 AM EDT, David Hubbard <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone else having issues with Time Warner residential or business connections? One of our offices is down and the route is not currently in bgp. http://downdetector.com/status/time-warner-cable shows thousands of reports of outages on the consumer side starting an hour or so ago so I figure it's a larger issue than just my one office; couldn't reach anyone by phone.
Thanks,
David
-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
TCAM? Those of you with TW, how many routes are they advertising to you via BGP? Kevin On 8/27/14, 9:56 AM, "Jay Ashworth" <jra@baylink.com> wrote:
I had a CNN breaking news email on my phone about this when I woke up, their backbone apparently was down from 2 ish to 4 ish AM.
On August 27, 2014 6:17:00 AM EDT, David Hubbard <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone else having issues with Time Warner residential or business connections? One of our offices is down and the route is not currently in bgp. http://downdetector.com/status/time-warner-cable shows thousands of reports of outages on the consumer side starting an hour or so ago so I figure it's a larger issue than just my one office; couldn't reach anyone by phone.
Thanks,
David
-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and destroy any copies of this document.
The issue reported on our side was impact to TW from a L3 maintenance that caused the issue. No details beyond that to share unfortunately On Aug 27, 2014, at 9:34 AM, Irwin, Kevin <Kevin.Irwin@cinbell.com> wrote:
TCAM? Those of you with TW, how many routes are they advertising to you via BGP?
Kevin
On 8/27/14, 9:56 AM, "Jay Ashworth" <jra@baylink.com> wrote:
I had a CNN breaking news email on my phone about this when I woke up, their backbone apparently was down from 2 ish to 4 ish AM.
On August 27, 2014 6:17:00 AM EDT, David Hubbard <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone else having issues with Time Warner residential or business connections? One of our offices is down and the route is not currently in bgp. http://downdetector.com/status/time-warner-cable shows thousands of reports of outages on the consumer side starting an hour or so ago so I figure it's a larger issue than just my one office; couldn't reach anyone by phone.
Thanks,
David
-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and destroy any copies of this document.
participants (18)
-
Adam Greene -
Alexandru Suciu -
Chris Garrett -
Chris Lane -
David Coulson -
David Hubbard -
Eric Stoltz -
Faisal Imtiaz -
George, Wes -
Irwin, Kevin -
Jay Ashworth -
Naslund, Steve -
Ray -
Rick Coloccia -
Rob Barbeau -
Stephen Satchell -
Tom Beecher -
Warren Bailey