-----Original Message----- From: nanog-request@nanog.org [mailto:nanog-request@nanog.org] Sent: Monday, November 07, 2011 10:05 AM To: nanog@nanog.org Subject: NANOG Digest, Vol 46, Issue 15
Send NANOG mailing list submissions to nanog@nanog.org
To subscribe or unsubscribe via the World Wide Web, visit https://mailman.nanog.org/mailman/listinfo/nanog or, via email, send a message with subject or body 'help' to nanog-request@nanog.org
You can reach the person managing the list at nanog-owner@nanog.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of NANOG digest..."
Today's Topics:
1. Re: Time Warner Telecom problems (Jon Lewis) 2. Re: Performance Issues - PTR Records (Leigh Porter) 3. Re: Time Warner Telecom problems (Ray Van Dolson) 4. General Internet Instability (Jared Mauch) 5. Re: TATA problems? (Pierre-Yves Maunier) 6. Re: TATA problems? (Leigh Porter) 7. Re: Time Warner Telecom problems (Joe Greco) 8. Re: TATA problems? (Kelly Kane) 9. Re: Time Warner Telecom problems (Blake Hudson) 10. RE: Time Warner Telecom problems (Thomas York)
----------------------------------------------------------------------
Message: 1 Date: Mon, 7 Nov 2011 10:12:30 -0500 (EST) From: Jon Lewis <jlewis@lewis.org> To: Peter Pauly <ppauly@gmail.com> Cc: nanog@nanog.org Subject: Re: Time Warner Telecom problems Message-ID: <Pine.LNX.4.61.1111071005540.24418@soloth.lewis.org> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
On Mon, 7 Nov 2011, Peter Pauly wrote:
Gizmodo is reporting problems at Time Warner Telecom .... we're suffering from it too and calls to the NOC have not been answered so far... does anyone have any further information?
http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
I noticed just a little while ago that we're having a lot of DNS fail. Initial findings were that several of the root-servers we were trying to reach via our TWTelecom link were unreachable after 2 hops into TWT.
4 64-128-130-233.static.twtelecom.NET (64.128.130.233) 2.399 ms 2.298 ms 2.338 ms 5 mia2-pr1-xe-1-3-0-0.us.twtelecom.net (66.192.253.18) 11.571 ms 11.552 ms 9.467 ms 6 * * * 7 * * * 8 * * *
For instance, a.root-servers.net is pingable from a rackspace server, but not from our network (unless I shut off TWT, at which point it is, but it's apparently not the same a.root-servers.net instance rackspace sees). I assume this is one of the root-servers being anycast.
Shutting off our BGP with TWT didn't appear to help (though the root-servers became reachable)...so I assume there's more going on
just TWT routing fail.
---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
------------------------------
Message: 2 Date: Mon, 7 Nov 2011 15:29:30 +0000 From: Leigh Porter <leigh.porter@ukbroadband.com> To: Bj?rn Mork <bjorn@mork.no> Cc: "nanog@nanog.org" <nanog@nanog.org> Subject: Re: Performance Issues - PTR Records Message-ID: <53A4963F-4969-4A60-BF06-E690C7324863@ukbroadband.com> Content-Type: text/plain; charset="iso-8859-1"
On 7 Nov 2011, at 14:03, "Bj?rn Mork" <bjorn@mork.no> wrote:
Leigh Porter <leigh.porter@ukbroadband.com> writes:
Indeed, there is no way I would allow that either. But really, providing a reverse zone and forward zone to match is a case of five minutes and a shell script or a DNS that as Steinar said, will synthesise results.
It's really not all that difficult..
No, not at all. It's just totally pointless. Any IPv6 address is just as pretty as a synthesized name. Maybe even prettier. Do you prefer "2001:db8:1::2" or "20010db8000100000000000000000002.rev.example.com"?
If we're going to provide any reverse DNS for end users, then it is because we can create names which actually improves something.
Bj?rn
Yup it is pointless.. Mine are all ipadrress.domain which is of course, pointless.. I suppose at least somebody would glean that perhaps its a home user rather than a business or server on that address but that's all.
With IPv6 arguably even more pointless as you say.
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________
------------------------------
Message: 3 Date: Mon, 7 Nov 2011 07:28:18 -0800 From: Ray Van Dolson <rvandolson@esri.com> To: nanog@nanog.org Subject: Re: Time Warner Telecom problems Message-ID: <20111107152817.GA29715@esri.com> Content-Type: text/plain; charset=us-ascii
On Mon, Nov 07, 2011 at 07:04:19AM -0800, Peter Pauly wrote:
Gizmodo is reporting problems at Time Warner Telecom .... we're suffering from it too and calls to the NOC have not been answered so far... does anyone have any further information?
http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
FWIW, my home TWC connection dropped this morning for about 15 minutes (Southern California around 6:30AM'ish). Still could ping the default gateway, but packets weren't traversing much beyond that.
Didn't investigate further, just headed into work.
Ray
------------------------------
Message: 4 Date: Mon, 7 Nov 2011 10:31:31 -0500 From: Jared Mauch <jared@puck.nether.net> To: Tom Hill <tom@ninjabadger.net> Cc: nanog@nanog.org Subject: General Internet Instability Message-ID: <B6567BC5-75E9-4E58-AFA9-5ADC0C2A7BDD@puck.nether.net> Content-Type: text/plain; charset=us-ascii
On Nov 7, 2011, at 10:08 AM, Tom Hill wrote:
On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
We seem to be having some problems with our tata links - first seen in EU about 45 minutes ago, now we're seeing problems in NA. I'm focused on DNS, so I'm seeing a lot of timeouts/servfails, but our networking folks are talking about links dropping.
Anyone else seeing oddness on the NA Internet right now?
http://downrightnow.com/ confirms - something is up.
There are widespread issues across the Internet; certain versions of Juniper firmware have core dumped after seeing a particular BGP 'UPDATE' message.
(That's the running theory at least).
It's affected multiple service providers, globally, not just those connected to TATA.
Pretty much any major BGP event will impact multiple providers.
A threshold you should use to view the general instability (which I find valuable, you may as well) is route views data.
If you look at the BGP UPDATES archive sizes, you can see when something happens, e.g.:
http://archive.routeviews.org/bgpdata/2011.11/UPDATES/
Take a look at the size of the updates.20111107.1400.bz2 file and the 1415 file. They are abnormally large compared to a normal period of time. This shows there were a lot of updates out there being processed and a reference to levels of instability.
If you are not feeding route views or similar community projects,
Level 3 was down in KC, Chi, and San Jose (at least) for us between about 8:10 and 8:40, plus or minus. Brought down SureWest in KC too. -Steve than please
consider doing so. It helps paint the view for those doing analysis.
- Jared
------------------------------
Message: 5 Date: Mon, 7 Nov 2011 16:33:15 +0100 From: Pierre-Yves Maunier <nanog@maunier.org> To: Tom Hill <tom@ninjabadger.net> Cc: nanog@nanog.org Subject: Re: TATA problems? Message-ID:
<CAEGdXrM+2rFXG2=d80kO_1ObtAv=24npNqZsLCTXUxeKS9q9yA@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
2011/11/7 Tom Hill <tom@ninjabadger.net>
On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
We seem to be having some problems with our tata links - first seen in EU about 45 minutes ago, now we're seeing problems in NA. I'm focused on DNS, so I'm seeing a lot of timeouts/servfails, but our networking folks are talking about links dropping.
Anyone else seeing oddness on the NA Internet right now?
http://downrightnow.com/ confirms - something is up.
There are widespread issues across the Internet; certain versions of Juniper firmware have core dumped after seeing a particular BGP 'UPDATE' message.
(That's the running theory at least).
It's affected multiple service providers, globally, not just those connected to TATA.
Tom
On our side all our 10.3R2.11 core dumped which made all our interfaces flapped. I've been told 10.4R1.9 is affected too.
-- Pierre-Yves Maunier
------------------------------
Message: 6 Date: Mon, 7 Nov 2011 15:45:18 +0000 From: Leigh Porter <leigh.porter@ukbroadband.com> To: Pierre-Yves Maunier <nanog@maunier.org> Cc: "nanog@nanog.org" <nanog@nanog.org> Subject: Re: TATA problems? Message-ID: <7994AF08-0622-434F-974F-FC9269469176@ukbroadband.com> Content-Type: text/plain; charset="us-ascii"
My 10.4r1.9 boxes died also but I saw interfaces go down whilst bgpd seemed stable.
-- Leigh
On 7 Nov 2011, at 15:34, "Pierre-Yves Maunier" <nanog@maunier.org> wrote:
2011/11/7 Tom Hill <tom@ninjabadger.net>
On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
We seem to be having some problems with our tata links - first seen in EU about 45 minutes ago, now we're seeing problems in NA. I'm focused on DNS, so I'm seeing a lot of timeouts/servfails, but our networking folks are talking about links dropping.
Anyone else seeing oddness on the NA Internet right now?
http://downrightnow.com/ confirms - something is up.
There are widespread issues across the Internet; certain versions of Juniper firmware have core dumped after seeing a particular BGP 'UPDATE' message.
(That's the running theory at least).
It's affected multiple service providers, globally, not just those connected to TATA.
Tom
On our side all our 10.3R2.11 core dumped which made all our interfaces flapped. I've been told 10.4R1.9 is affected too.
-- Pierre-Yves Maunier
This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________
------------------------------
Message: 7 Date: Mon, 7 Nov 2011 09:54:25 -0600 (CST) From: Joe Greco <jgreco@ns.sol.net> To: ppauly@gmail.com (Peter Pauly) Cc: nanog@nanog.org Subject: Re: Time Warner Telecom problems Message-ID: <201111071554.pA7FsPHb045359@aurora.sol.net> Content-Type: text/plain; charset=us-ascii
Gizmodo is reporting problems at Time Warner Telecom .... we're suffering from it too and calls to the NOC have not been answered so far...
does
anyone have any further information?
http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
Actually, it looks to me like they mean "Time Warner", because that's what they said.
The company once known as "Time Warner Telecom" has always been a different entity, and hasn't been known as that in some time, now being called "twtelecom." Much of that company is what was once known as inc.net, a Milwaukee area provider of the '90's.
Time Warner Cable appears to have experienced an implosion this morning, being out of service for about 11 minutes. During that time, packets originating here in Milwaukee quickly died in Chicago;
1 76.46.192.1 8.320 ms 9.900 ms 7.974 ms 2 24.160.230.32 7.967 ms 5.975 ms 8.479 ms 3 24.160.229.132 8.471 ms 7.969 ms 10.991 ms 4 24.160.229.193 9.972 ms 9.973 ms 24.160.229.197 9.985 ms 5 * * * 6 * * *
while packets destined for RR all seemed to be headed out to SJC, from what I can tell.
... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
------------------------------
Message: 8 Date: Mon, 7 Nov 2011 07:55:33 -0800 From: Kelly Kane <kelly@hawknetworks.com> To: Tim Vollebregt <tim@interworx.nl> Cc: nanog@nanog.org Subject: Re: TATA problems? Message-ID:
Content-Type: text/plain; charset=UTF-8
On Mon, Nov 7, 2011 at 07:06, Tim Vollebregt <tim@interworx.nl> wrote:
On #IX there are rumours about Junos version 10.3R2.11 being core
dumped and
rebooted, which makes sense.
Perhaps related to Juniper PSN-2011-08-327? Did the whole router reboot, or just the service module?
We saw one TATA session, and one Abovenet session flap.
Kelly
------------------------------
Message: 9 Date: Mon, 07 Nov 2011 10:02:13 -0600 From: Blake Hudson <blake@ispn.net> To: nanog@nanog.org Subject: Re: Time Warner Telecom problems Message-ID: <4EB80105.8060703@ispn.net> Content-Type: text/plain; charset=UTF-8; format=flowed
Joe Greco wrote the following on 11/7/2011 9:54 AM:
Gizmodo is reporting problems at Time Warner Telecom .... we're suffering from it too and calls to the NOC have not been answered so far... does anyone have any further information?
http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us Actually, it looks to me like they mean "Time Warner", because
what they said.
The company once known as "Time Warner Telecom" has always been a different entity, and hasn't been known as that in some time, now being called "twtelecom." Much of that company is what was once known as inc.net, a Milwaukee area provider of the '90's.
Time Warner Cable appears to have experienced an implosion this morning, being out of service for about 11 minutes. During that time,
originating here in Milwaukee quickly died in Chicago;
Using the looking glass from TWtelecom, we saw 30-60min outage (roughly 8:30AM to 9:30AM CST) between the Kansas City location and our own server room in Kansas City. Other TWtelecom locations appeared to be unaffected. Perhaps TWtelecom is served by Timewarner or shares equipment in KC. Either way, none of our KC customers who were served via TWtelecom or Timewarner were able to reach us. Packets would hit Level 3 Communications and die in either direction at the border between L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa. http://lglass.twtelecom.net/
------------------------------
Message: 10 Date: Mon, 7 Nov 2011 11:04:59 -0500 From: "Thomas York" <straterra@fuhell.com> To: "'Blake Hudson'" <blake@ispn.net>, <nanog@nanog.org> Subject: RE: Time Warner Telecom problems Message-ID: <c23401cc9d67$046baea0$0d430be0$@fuhell.com> Content-Type: text/plain; charset="utf-8"
FWIW, We saw issues here in Indianapolis between TWTC and L3 up until a few minutes ago.
--Thomas York
-----Original Message----- From: Blake Hudson [mailto:blake@ispn.net] Sent: Monday, November 07, 2011 11:02 AM To: nanog@nanog.org Subject: Re: Time Warner Telecom problems
Joe Greco wrote the following on 11/7/2011 9:54 AM:
Gizmodo is reporting problems at Time Warner Telecom .... we're suffering from it too and calls to the NOC have not been answered so far... does anyone have any further information?
http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us Actually, it looks to me like they mean "Time Warner", because
<CAKfXD0zrH9TJkN=7doGe1uAq8zxVszyvtJYfLLTbiC6+UMy4ww@mail.gmail.com> that's packets that's
what they said.
The company once known as "Time Warner Telecom" has always been a different entity, and hasn't been known as that in some time, now being called "twtelecom." Much of that company is what was once known as inc.net, a Milwaukee area provider of the '90's.
Time Warner Cable appears to have experienced an implosion this morning, being out of service for about 11 minutes. During that time, packets originating here in Milwaukee quickly died in Chicago;
Using the looking glass from TWtelecom, we saw 30-60min outage (roughly 8:30AM to 9:30AM CST) between the Kansas City location and our own server room in Kansas City. Other TWtelecom locations appeared to be unaffected. Perhaps TWtelecom is served by Timewarner or shares equipment in KC. Either way, none of our KC customers who were served via TWtelecom or Timewarner were able to reach us. Packets would hit Level 3 Communications and die in either direction at the border between L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa. http://lglass.twtelecom.net/
End of NANOG Digest, Vol 46, Issue 15 *************************************