Has Level3 done away with traceroute??
Hello All, Recently I was troubleshooting a network event for a client of our who resides on the Level3 network. While trying to verify the path, I noticed I am no longer able to traceroute through the Level3 network. The funny thing is this is not just isolated to the /32. It appears to be that the entire 4.0.0.0/9 network is no longer able to traceroute through. Everything dies on their edge network. This appears to be isolated to traceroute. I have check this in NA and EU. My carrier contacted Level3 who pretty much stated that they can’t provide anything. I have checked multiple looking glasses and other online tools and none of them make it. Even Level3 looking glass drops the packets. Does anyone know anything about this? I’m pretty sure this is the first time we are seeing this. Random 4.0.0.0/9 address. NTT looking glass Tracing the route to 4.35.230.7 1 * ae-2.a00.snjsca04.us.bb.gin.ntt.net (129.250.3.58) 3 msec 1 msec 2 * * * 3 * * * 4 * * * 5 * * * 6 * * * TATA looking glass traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets 1 if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89) 2.056 ms if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173) 1.253 ms 1.177 ms MPLS Label=616998 CoS=0 TTL=1 S=1 2 195.219.50.50 (195.219.50.50) 1.214 ms 1.247 ms 1.535 ms 3 195.219.50.50 (195.219.50.50) 1.144 ms * 2.246 ms 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * Telia looking glass traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets 1 if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89) 2.056 ms if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173) 1.253 ms 1.177 ms MPLS Label=616998 CoS=0 TTL=1 S=1 2 195.219.50.50 (195.219.50.50) 1.214 ms 1.247 ms 1.535 ms 3 195.219.50.50 (195.219.50.50) 1.144 ms * 2.246 ms 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * Level3 looking glass Traceroute results from Atlanta, GA to 4.200.65.42(dialup-4.200.65.42.Dial1.LosAngeles1) 1 0.0.0.0 * * * 2 0.0.0.0 * * * 3 0.0.0.0 * * * 4 0.0.0.0 * * * 5 0.0.0.0 * * * 6 0.0.0.0 * * * 7 0.0.0.0 * * * 8 0.0.0.0 * * * 9 0.0.0.0 * * * 10 0.0.0.0 * * * 11 0.0.0.0 * * * -- Donovan Van Dyk SOC Network Engineer Fort Lauderdale, FL USA [cid:image001.png@01D332DA.F4DD00A0] The information contained in this electronic mail transmission and its attachments may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient (or an individual responsible for delivery of the message to such person), you are strictly prohibited from copying, disseminating or distributing this communication. If you have received this communication in error, please notify the sender immediately and destroy all electronic, paper or other versions.
I’m seeing the same thing across our Level 3 circuits, even if the traffic never leaves their network. This was not the case as recently as two days ago when I had a reachability ticket open with them. My SE says they’re having an internal issue that’s being worked on; didn’t provide detail. On 9/21/17, 1:13 PM, "NANOG on behalf of Van Dyk, Donovan via NANOG" <nanog-bounces@nanog.org on behalf of nanog@nanog.org> wrote: Hello All, Recently I was troubleshooting a network event for a client of our who resides on the Level3 network. While trying to verify the path, I noticed I am no longer able to traceroute through the Level3 network. The funny thing is this is not just isolated to the /32. It appears to be that the entire 4.0.0.0/9 network is no longer able to traceroute through. Everything dies on their edge network. This appears to be isolated to traceroute. I have check this in NA and EU. My carrier contacted Level3 who pretty much stated that they can’t provide anything. I have checked multiple looking glasses and other online tools and none of them make it. Even Level3 looking glass drops the packets. Does anyone know anything about this? I’m pretty sure this is the first time we are seeing this. Random 4.0.0.0/9 address. NTT looking glass Tracing the route to 4.35.230.7 1 * ae-2.a00.snjsca04.us.bb.gin.ntt.net (129.250.3.58) 3 msec 1 msec 2 * * * 3 * * * 4 * * * 5 * * * 6 * * * TATA looking glass traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets 1 if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89) 2.056 ms if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173) 1.253 ms 1.177 ms MPLS Label=616998 CoS=0 TTL=1 S=1 2 195.219.50.50 (195.219.50.50) 1.214 ms 1.247 ms 1.535 ms 3 195.219.50.50 (195.219.50.50) 1.144 ms * 2.246 ms 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * Telia looking glass traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets 1 if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89) 2.056 ms if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173) 1.253 ms 1.177 ms MPLS Label=616998 CoS=0 TTL=1 S=1 2 195.219.50.50 (195.219.50.50) 1.214 ms 1.247 ms 1.535 ms 3 195.219.50.50 (195.219.50.50) 1.144 ms * 2.246 ms 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * Level3 looking glass Traceroute results from Atlanta, GA to 4.200.65.42(dialup-4.200.65.42.Dial1.LosAngeles1) 1 0.0.0.0 * * * 2 0.0.0.0 * * * 3 0.0.0.0 * * * 4 0.0.0.0 * * * 5 0.0.0.0 * * * 6 0.0.0.0 * * * 7 0.0.0.0 * * * 8 0.0.0.0 * * * 9 0.0.0.0 * * * 10 0.0.0.0 * * * 11 0.0.0.0 * * * -- Donovan Van Dyk SOC Network Engineer Fort Lauderdale, FL USA [cid:image001.png@01D332DA.F4DD00A0] The information contained in this electronic mail transmission and its attachments may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient (or an individual responsible for delivery of the message to such person), you are strictly prohibited from copying, disseminating or distributing this communication. If you have received this communication in error, please notify the sender immediately and destroy all electronic, paper or other versions.
Hi, Did a measurement with RIPE Atlas and it seems like a worldwide problem, based on data from 100 probes. https://atlas.ripe.net/measurements/9324230/#!tracemon Dne 9/21/17 v 19:10 Van Dyk, Donovan via NANOG napsal(a):
Hello All,
Recently I was troubleshooting a network event for a client of our who resides on the Level3 network. While trying to verify the path, I noticed I am no longer able to traceroute through the Level3 network. The funny thing is this is not just isolated to the /32. It appears to be that the entire 4.0.0.0/9 network is no longer able to traceroute through. Everything dies on their edge network.
This appears to be isolated to traceroute. I have check this in NA and EU.
My carrier contacted Level3 who pretty much stated that they can’t provide anything.
I have checked multiple looking glasses and other online tools and none of them make it. Even Level3 looking glass drops the packets.
Does anyone know anything about this? I’m pretty sure this is the first time we are seeing this.
Random 4.0.0.0/9 address.
NTT looking glass Tracing the route to 4.35.230.7
1 * ae-2.a00.snjsca04.us.bb.gin.ntt.net (129.250.3.58) 3 msec 1 msec 2 * * * 3 * * * 4 * * * 5 * * * 6 * * *
TATA looking glass traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets 1 if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89) 2.056 ms if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173) 1.253 ms 1.177 ms MPLS Label=616998 CoS=0 TTL=1 S=1 2 195.219.50.50 (195.219.50.50) 1.214 ms 1.247 ms 1.535 ms 3 195.219.50.50 (195.219.50.50) 1.144 ms * 2.246 ms 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * *
Telia looking glass traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets 1 if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89) 2.056 ms if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173) 1.253 ms 1.177 ms MPLS Label=616998 CoS=0 TTL=1 S=1 2 195.219.50.50 (195.219.50.50) 1.214 ms 1.247 ms 1.535 ms 3 195.219.50.50 (195.219.50.50) 1.144 ms * 2.246 ms 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * *
Level3 looking glass Traceroute results from Atlanta, GA to 4.200.65.42(dialup-4.200.65.42.Dial1.LosAngeles1)
1 0.0.0.0 * * * 2 0.0.0.0 * * * 3 0.0.0.0 * * * 4 0.0.0.0 * * * 5 0.0.0.0 * * * 6 0.0.0.0 * * * 7 0.0.0.0 * * * 8 0.0.0.0 * * * 9 0.0.0.0 * * * 10 0.0.0.0 * * * 11 0.0.0.0 * * *
-- Donovan Van Dyk SOC Network Engineer Fort Lauderdale, FL USA
[cid:image001.png@01D332DA.F4DD00A0] The information contained in this electronic mail transmission and its attachments may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient (or an individual responsible for delivery of the message to such person), you are strictly prohibited from copying, disseminating or distributing this communication. If you have received this communication in error, please notify the sender immediately and destroy all electronic, paper or other versions.
-- Best Regards, Filip Hruska Linux System Administrator
I just performed a few traceroutes. Comcast to 4.2.2.2 5. hu-0-11-0-0-ar03.ivyland.pa.panjde.comcast.net 6. xe-4-0-0.edge1.Toronto.Level3.net 7. ??? 8. b.resolvers.Level3.net Comcast to Level3 customer 5. hu-0-11-0-0-ar03.ivyland.pa.panjde.comcast.net 6. xe-4-0-0.edge1.Toronto.Level3.net 7. ae-11-11.bar1.Philadelphia1.Level3.net 8. XXXXXXXX.bar1.Philadelphia1.Level3.net I see the problem for sure FROM a Level3 Philadelphia customer to other destinations though. On 9/21/2017 1:45 PM, Filip Hruska wrote:
Hi,
Did a measurement with RIPE Atlas and it seems like a worldwide problem, based on data from 100 probes.
https://atlas.ripe.net/measurements/9324230/#!tracemon
Dne 9/21/17 v 19:10 Van Dyk, Donovan via NANOG napsal(a):
Hello All,
Recently I was troubleshooting a network event for a client of our who resides on the Level3 network. While trying to verify the path, I noticed I am no longer able to traceroute through the Level3 network. The funny thing is this is not just isolated to the /32. It appears to be that the entire 4.0.0.0/9 network is no longer able to traceroute through. Everything dies on their edge network.
This appears to be isolated to traceroute. I have check this in NA and EU.
My carrier contacted Level3 who pretty much stated that they can’t provide anything.
I have checked multiple looking glasses and other online tools and none of them make it. Even Level3 looking glass drops the packets.
Does anyone know anything about this? I’m pretty sure this is the first time we are seeing this.
Random 4.0.0.0/9 address.
NTT looking glass Tracing the route to 4.35.230.7
1 * ae-2.a00.snjsca04.us.bb.gin.ntt.net (129.250.3.58) 3 msec 1 msec 2 * * * 3 * * * 4 * * * 5 * * * 6 * * *
TATA looking glass traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets 1 if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89) 2.056 ms if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173) 1.253 ms 1.177 ms MPLS Label=616998 CoS=0 TTL=1 S=1 2 195.219.50.50 (195.219.50.50) 1.214 ms 1.247 ms 1.535 ms 3 195.219.50.50 (195.219.50.50) 1.144 ms * 2.246 ms 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * *
Telia looking glass traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets 1 if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89) 2.056 ms if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173) 1.253 ms 1.177 ms MPLS Label=616998 CoS=0 TTL=1 S=1 2 195.219.50.50 (195.219.50.50) 1.214 ms 1.247 ms 1.535 ms 3 195.219.50.50 (195.219.50.50) 1.144 ms * 2.246 ms 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * *
Level3 looking glass Traceroute results from Atlanta, GA to 4.200.65.42(dialup-4.200.65.42.Dial1.LosAngeles1)
1 0.0.0.0 * * * 2 0.0.0.0 * * * 3 0.0.0.0 * * * 4 0.0.0.0 * * * 5 0.0.0.0 * * * 6 0.0.0.0 * * * 7 0.0.0.0 * * * 8 0.0.0.0 * * * 9 0.0.0.0 * * * 10 0.0.0.0 * * * 11 0.0.0.0 * * *
-- Donovan Van Dyk SOC Network Engineer Fort Lauderdale, FL USA
[cid:image001.png@01D332DA.F4DD00A0] The information contained in this electronic mail transmission and its attachments may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient (or an individual responsible for delivery of the message to such person), you are strictly prohibited from copying, disseminating or distributing this communication. If you have received this communication in error, please notify the sender immediately and destroy all electronic, paper or other versions.
I am able to traceroute in and out of Level3, but it seems like some L3 internal hops are missing, as I appear to go straight from the L3 edge router to my L3 customer agg router: traceroute to 206.83.0.42 (206.83.0.42), 64 hops max, 52 byte packets 1 47.155.227.1 (47.155.227.1) 4.318 ms 4.661 ms 4.715 ms 2 172.102.107.166 (172.102.107.166) 10.202 ms 7.521 ms 172.102.102.240 (172.102.102.240) 7.240 ms 3 ae8---0.scr02.lsan.ca.frontiernet.net (74.40.3.49) 9.609 ms 7.501 ms ae8---0.scr01.lsan.ca.frontiernet.net (74.40.3.37) 5.298 ms 4 ae0---0.cbr01.lsan.ca.frontiernet.net (74.40.3.198) 7.169 ms 7.015 ms 7.277 ms 5 * * * 6 ae-4-90.edge1.losangeles9.level3.net (4.69.144.202) 8.384 ms 7.012 ms ae-2-70.edge1.losangeles9.level3.net (4.69.144.74) 9.865 ms 7 ae-2-70.edge1.losangeles9.level3.net (4.69.144.74) 7.471 ms ae-3-80.edge1.losangeles9.level3.net (4.69.144.138) 6.551 ms ae-2-70.edge1.losangeles9.level3.net (4.69.144.74) 10.520 ms 8 4.68.111.22 (4.68.111.22) 9.010 ms 7.445 ms 7.314 ms 9 sba1-ar1-xe-11-0-0-0.us.twtelecom.net (35.248.2.6) 9.788 ms 9.536 ms 10.266 ms 10 206-190-77-10.static.twtelecom.net (206.190.77.10) 14.739 ms 12.525 ms 9.979 ms 11 iris1.jet.net (206.83.0.42) 10.285 ms 9.880 ms 10.063 ms traceroute to 47.155.227.1 (47.155.227.1), 64 hops max, 40 byte packets 1 router1.sb.becknet.com (206.83.0.1) 1.270 ms 1.023 ms 0.352 ms 2 206-190-77-9.static.twtelecom.net (206.190.77.9) 0.799 ms 0.741 ms 0.785 ms 3 ae1-90g.ar7.lax1.gblx.net (67.17.75.18) 3.052 ms 3.126 ms 2.957 ms 4 ae10.edge1.losangeles9.level3.net (4.68.111.21) 3.456 ms 3.043 ms 2.981 ms 5 * * * 6 4.16.30.130 (4.16.30.130) 27.564 ms 3.574 ms 3.445 ms 7 ae1---0.scr02.lsan.ca.frontiernet.net (74.40.3.213) 3.489 ms ae1---0.scr01.lsan.ca.frontiernet.net (74.40.3 .197) 3.396 ms ae1---0.scr02.lsan.ca.frontiernet.net (74.40.3.213) 3.428 ms 8 be10---0.lcr21.lsan.ca.frontiernet.net (74.40.3.38) 11.736 ms 5.026 ms be10---0.lcr22.lsan.ca.frontiernet. net (74.40.3.50) 9.858 ms 9 47.155.227.1 (47.155.227.1) 6.545 ms 5.398 ms 6.499 ms -mel beckman
On Sep 21, 2017, at 10:45 AM, Filip Hruska <fhr@fhrnet.eu> wrote:
Hi,
Did a measurement with RIPE Atlas and it seems like a worldwide problem, based on data from 100 probes.
https://atlas.ripe.net/measurements/9324230/#!tracemon
Dne 9/21/17 v 19:10 Van Dyk, Donovan via NANOG napsal(a):
Hello All,
Recently I was troubleshooting a network event for a client of our who resides on the Level3 network. While trying to verify the path, I noticed I am no longer able to traceroute through the Level3 network. The funny thing is this is not just isolated to the /32. It appears to be that the entire 4.0.0.0/9 network is no longer able to traceroute through. Everything dies on their edge network.
This appears to be isolated to traceroute. I have check this in NA and EU.
My carrier contacted Level3 who pretty much stated that they can’t provide anything.
I have checked multiple looking glasses and other online tools and none of them make it. Even Level3 looking glass drops the packets.
Does anyone know anything about this? I’m pretty sure this is the first time we are seeing this.
Random 4.0.0.0/9 address.
NTT looking glass Tracing the route to 4.35.230.7
1 * ae-2.a00.snjsca04.us.bb.gin.ntt.net (129.250.3.58) 3 msec 1 msec 2 * * * 3 * * * 4 * * * 5 * * * 6 * * *
TATA looking glass traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets 1 if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89) 2.056 ms if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173) 1.253 ms 1.177 ms MPLS Label=616998 CoS=0 TTL=1 S=1 2 195.219.50.50 (195.219.50.50) 1.214 ms 1.247 ms 1.535 ms 3 195.219.50.50 (195.219.50.50) 1.144 ms * 2.246 ms 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * *
Telia looking glass traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets 1 if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89) 2.056 ms if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173) 1.253 ms 1.177 ms MPLS Label=616998 CoS=0 TTL=1 S=1 2 195.219.50.50 (195.219.50.50) 1.214 ms 1.247 ms 1.535 ms 3 195.219.50.50 (195.219.50.50) 1.144 ms * 2.246 ms 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * *
Level3 looking glass Traceroute results from Atlanta, GA to 4.200.65.42(dialup-4.200.65.42.Dial1.LosAngeles1)
1 0.0.0.0 * * * 2 0.0.0.0 * * * 3 0.0.0.0 * * * 4 0.0.0.0 * * * 5 0.0.0.0 * * * 6 0.0.0.0 * * * 7 0.0.0.0 * * * 8 0.0.0.0 * * * 9 0.0.0.0 * * * 10 0.0.0.0 * * * 11 0.0.0.0 * * *
-- Donovan Van Dyk SOC Network Engineer Fort Lauderdale, FL USA
[cid:image001.png@01D332DA.F4DD00A0] The information contained in this electronic mail transmission and its attachments may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient (or an individual responsible for delivery of the message to such person), you are strictly prohibited from copying, disseminating or distributing this communication. If you have received this communication in error, please notify the sender immediately and destroy all electronic, paper or other versions.
-- Best Regards, Filip Hruska Linux System Administrator
participants (5)
-
David Hubbard -
Filip Hruska -
Mel Beckman -
ML -
Van Dyk, Donovan