I can see the Google IP space (64.71.240.0/20) from Verizon/AS701, but not from Rogers/AS812 in Toronto. I've done a few other test traceroutes through Rogers to verify that they didn't filter UDP/ICMP. At this point nothing would surprise me from Rogers. AS701 ===== TOR2-CORE-R1#traceroute www.google.com Translating "www.google.com"...domain server (8.8.8.8) [OK] Type escape sequence to abort. Tracing the route to www.google.com (74.125.26.99) VRF info: (vrf in name/id, vrf out name/id) 1 x.x.x.x [AS701] 4 msec 0 msec 0 msec 2 0.ge-11-0-0.XT4.TOR2.ALTER.NET (152.63.133.78) [AS 701] 4 msec 0 msec 4 msec 3 0.so-4-0-3.XT2.NYC4.ALTER.NET (152.63.0.73) [AS 701] 16 msec 16 msec 16 msec 4 TenGigE0-7-1-0.GW8.NYC4.ALTER.NET (152.63.21.125) [AS 701] 20 msec TenGigE0-5-1-0.GW8.NYC4.ALTER.NET (152.63.21.73) [AS 701] 16 msec TenGigE0-5-4-0.GW8.NYC4.ALTER.NET (152.63.18.206) [AS 701] 20 msec 5 72.14.238.232 [AS 15169] 16 msec 16 msec 20 msec 6 72.14.236.208 [AS 15169] [MPLS: Label 680976 Exp 4] 16 msec 16 msec 72.14.236.206 [AS 15169] [MPLS: Label 533197 Exp 4] 20 msec 7 209.85.249.11 [AS 15169] [MPLS: Label 16668 Exp 4] 24 msec 72.14.239.93 [AS 15169] [MPLS: Label 14644 Exp 4] 24 msec 209.85.249.11 [AS 15169] [MPLS: Label 13978 Exp 4] 24 msec 8 209.85.243.114 [AS 15169] [MPLS: Label 568789 Exp 4] 32 msec 209.85.241.222 [AS 15169] [MPLS: Label 632535 Exp 4] 32 msec 32 msec 9 216.239.48.159 [AS 15169] 36 msec 32 msec 216.239.48.59 [AS 15169] 32 msec 10 www.google.com (74.125.26.147) [AS 15169] 32 msec 32 msec 32 msec AS812 (Rogers Looking Glass https://supernoc.rogerstelecom.net/lg/) =================================================== *Query:* *tr 64.71.249.45* Type escape sequence to abort. Tracing the route to 64.71.249.45 1 64.71.255.62 0 msec 0 msec 0 msec 2 ge-4-3-0.gw02.ym.phub.net.cable.ROGERS.com (66.185.82.237) 4 msec 4 msec 4 msec 3 69.63.250.189 4 msec 4 msec 4 msec 4 69.63.250.174 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Cheers, - MJ -----Original Message-----
From: Grant Ridder [mailto:shortdudey123@gmail.com] Sent: July-10-13 10:57 AM To: John York Cc: nanog@nanog.org Subject: Re: google troubles?
Does anyone have traceroutes showing where the issues are?
-Grant
On Wed, Jul 10, 2013 at 7:45 AM, John York <johny@griffintechnology.com>wrote:
We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working.
...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes.
On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper <blair.trosper@gmail.com
wrote:
Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine.
It's anecdotal, but it would seem like there's an issue based on these reports.
Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html
Anyone know what's up? Fiber cut? DC outages?
-- blair
--
John York
Information Technology | Network Administrator
Phone: 615-399-7000 x:333
Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA
This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
I just realized that it's not Google IP space (74.125.0.0/16), Rogers is hijacking the DNS and resolving www.google.com to space within 64.71.240.0/20 which is Rogers IP space! (note the name server set as 8.8.8.8) davinci#traceroute www.google.com Translating "www.google.com"...domain server (8.8.8.8) [OK] Type escape sequence to abort. Tracing the route to www.google.com (66.185.85.29) 1 x.x.x.x [AS 812] 4 msec 4 msec 4 msec 2 so-4-0-2.gw02.ym.phub.net.cable.rogers.com (66.185.82.129) [AS 812] 8 msec 8 msec 4 msec 3 69.63.252.222 [AS 812] 4 msec 0 msec 8 msec 4 69.63.250.162 [AS 812] 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * TOR2-CORE-R1#show ip bgp 66.185.85.29 BGP routing table entry for 66.185.80.0/20, version 13095115 Paths: (1 available, best #1, table default) Advertised to update-groups: 14 701 6461 812 205.205.23.121 from 205.205.23.121 (137.39.8.42) Origin IGP, localpref 100, valid, external, best TOR2-CORE-R1# Thanks, - Mike On Wed, Jul 10, 2013 at 11:28 AM, Mike Jackson <mike@routed.ca> wrote:
I can see the Google IP space (64.71.240.0/20) from Verizon/AS701, but not from Rogers/AS812 in Toronto. I've done a few other test traceroutes through Rogers to verify that they didn't filter UDP/ICMP. At this point nothing would surprise me from Rogers.
AS701 =====
TOR2-CORE-R1#traceroute www.google.com Translating "www.google.com"...domain server (8.8.8.8) [OK]
Type escape sequence to abort. Tracing the route to www.google.com (74.125.26.99) VRF info: (vrf in name/id, vrf out name/id) 1 x.x.x.x [AS701] 4 msec 0 msec 0 msec 2 0.ge-11-0-0.XT4.TOR2.ALTER.NET (152.63.133.78) [AS 701] 4 msec 0 msec 4 msec 3 0.so-4-0-3.XT2.NYC4.ALTER.NET (152.63.0.73) [AS 701] 16 msec 16 msec 16 msec 4 TenGigE0-7-1-0.GW8.NYC4.ALTER.NET (152.63.21.125) [AS 701] 20 msec TenGigE0-5-1-0.GW8.NYC4.ALTER.NET (152.63.21.73) [AS 701] 16 msec TenGigE0-5-4-0.GW8.NYC4.ALTER.NET (152.63.18.206) [AS 701] 20 msec 5 72.14.238.232 [AS 15169] 16 msec 16 msec 20 msec 6 72.14.236.208 [AS 15169] [MPLS: Label 680976 Exp 4] 16 msec 16 msec 72.14.236.206 [AS 15169] [MPLS: Label 533197 Exp 4] 20 msec 7 209.85.249.11 [AS 15169] [MPLS: Label 16668 Exp 4] 24 msec 72.14.239.93 [AS 15169] [MPLS: Label 14644 Exp 4] 24 msec 209.85.249.11 [AS 15169] [MPLS: Label 13978 Exp 4] 24 msec 8 209.85.243.114 [AS 15169] [MPLS: Label 568789 Exp 4] 32 msec 209.85.241.222 [AS 15169] [MPLS: Label 632535 Exp 4] 32 msec 32 msec 9 216.239.48.159 [AS 15169] 36 msec 32 msec 216.239.48.59 [AS 15169] 32 msec 10 www.google.com (74.125.26.147) [AS 15169] 32 msec 32 msec 32 msec
AS812 (Rogers Looking Glass https://supernoc.rogerstelecom.net/lg/) ===================================================
*Query:* *tr 64.71.249.45*
Type escape sequence to abort. Tracing the route to 64.71.249.45
1 64.71.255.62 0 msec 0 msec 0 msec 2 ge-4-3-0.gw02.ym.phub.net.cable.ROGERS.com (66.185.82.237) 4 msec 4 msec 4 msec 3 69.63.250.189 4 msec 4 msec 4 msec 4 69.63.250.174 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * *
Cheers,
- MJ
-----Original Message-----
From: Grant Ridder [mailto:shortdudey123@gmail.com] Sent: July-10-13 10:57 AM To: John York Cc: nanog@nanog.org Subject: Re: google troubles?
Does anyone have traceroutes showing where the issues are?
-Grant
On Wed, Jul 10, 2013 at 7:45 AM, John York <johny@griffintechnology.com>wrote:
We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working.
...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes.
On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper <blair.trosper@gmail.com
wrote:
Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine.
It's anecdotal, but it would seem like there's an issue based on these reports.
Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html
Anyone know what's up? Fiber cut? DC outages?
-- blair
--
John York
Information Technology | Network Administrator
Phone: 615-399-7000 x:333
Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA
This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
On Wed, Jul 10, 2013 at 12:20 PM, Mike Jackson <mike@routed.ca> wrote:
I just realized that it's not Google IP space (74.125.0.0/16), Rogers is hijacking the DNS and resolving www.google.com to space within 64.71.240.0/20 which is Rogers IP space! (note the name server set as 8.8.8.8)
so: 1) rogers is hijacking traffic to 8.8.8.8 2) the copy of 8.8.8.8 in rogers-land is replying with incorrect information for google properties (at least). err... any idea if it's lying about other things too? :)
davinci#traceroute www.google.com
Translating "www.google.com"...domain server (8.8.8.8) [OK]
Type escape sequence to abort. Tracing the route to www.google.com (66.185.85.29)
1 x.x.x.x [AS 812] 4 msec 4 msec 4 msec 2 so-4-0-2.gw02.ym.phub.net.cable.rogers.com (66.185.82.129) [AS 812] 8 msec 8 msec 4 msec 3 69.63.252.222 [AS 812] 4 msec 0 msec 8 msec 4 69.63.250.162 [AS 812] 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * *
TOR2-CORE-R1#show ip bgp 66.185.85.29 BGP routing table entry for 66.185.80.0/20, version 13095115 Paths: (1 available, best #1, table default) Advertised to update-groups: 14 701 6461 812 205.205.23.121 from 205.205.23.121 (137.39.8.42) Origin IGP, localpref 100, valid, external, best TOR2-CORE-R1#
Thanks,
- Mike
On Wed, Jul 10, 2013 at 11:28 AM, Mike Jackson <mike@routed.ca> wrote:
I can see the Google IP space (64.71.240.0/20) from Verizon/AS701, but not from Rogers/AS812 in Toronto. I've done a few other test traceroutes through Rogers to verify that they didn't filter UDP/ICMP. At this point nothing would surprise me from Rogers.
AS701 =====
TOR2-CORE-R1#traceroute www.google.com Translating "www.google.com"...domain server (8.8.8.8) [OK]
Type escape sequence to abort. Tracing the route to www.google.com (74.125.26.99) VRF info: (vrf in name/id, vrf out name/id) 1 x.x.x.x [AS701] 4 msec 0 msec 0 msec 2 0.ge-11-0-0.XT4.TOR2.ALTER.NET (152.63.133.78) [AS 701] 4 msec 0 msec 4 msec 3 0.so-4-0-3.XT2.NYC4.ALTER.NET (152.63.0.73) [AS 701] 16 msec 16 msec 16 msec 4 TenGigE0-7-1-0.GW8.NYC4.ALTER.NET (152.63.21.125) [AS 701] 20 msec TenGigE0-5-1-0.GW8.NYC4.ALTER.NET (152.63.21.73) [AS 701] 16 msec TenGigE0-5-4-0.GW8.NYC4.ALTER.NET (152.63.18.206) [AS 701] 20 msec 5 72.14.238.232 [AS 15169] 16 msec 16 msec 20 msec 6 72.14.236.208 [AS 15169] [MPLS: Label 680976 Exp 4] 16 msec 16 msec 72.14.236.206 [AS 15169] [MPLS: Label 533197 Exp 4] 20 msec 7 209.85.249.11 [AS 15169] [MPLS: Label 16668 Exp 4] 24 msec 72.14.239.93 [AS 15169] [MPLS: Label 14644 Exp 4] 24 msec 209.85.249.11 [AS 15169] [MPLS: Label 13978 Exp 4] 24 msec 8 209.85.243.114 [AS 15169] [MPLS: Label 568789 Exp 4] 32 msec 209.85.241.222 [AS 15169] [MPLS: Label 632535 Exp 4] 32 msec 32 msec 9 216.239.48.159 [AS 15169] 36 msec 32 msec 216.239.48.59 [AS 15169] 32 msec 10 www.google.com (74.125.26.147) [AS 15169] 32 msec 32 msec 32 msec
AS812 (Rogers Looking Glass https://supernoc.rogerstelecom.net/lg/) ===================================================
*Query:* *tr 64.71.249.45*
Type escape sequence to abort. Tracing the route to 64.71.249.45
1 64.71.255.62 0 msec 0 msec 0 msec 2 ge-4-3-0.gw02.ym.phub.net.cable.ROGERS.com (66.185.82.237) 4 msec 4 msec 4 msec 3 69.63.250.189 4 msec 4 msec 4 msec 4 69.63.250.174 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * *
Cheers,
- MJ
-----Original Message-----
From: Grant Ridder [mailto:shortdudey123@gmail.com] Sent: July-10-13 10:57 AM To: John York Cc: nanog@nanog.org Subject: Re: google troubles?
Does anyone have traceroutes showing where the issues are?
-Grant
On Wed, Jul 10, 2013 at 7:45 AM, John York <johny@griffintechnology.com>wrote:
We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working.
...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes.
On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper <blair.trosper@gmail.com
wrote:
Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine.
It's anecdotal, but it would seem like there's an issue based on these reports.
Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html
Anyone know what's up? Fiber cut? DC outages?
-- blair
--
John York
Information Technology | Network Administrator
Phone: 615-399-7000 x:333
Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA
This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
On Wed, Jul 10, 2013 at 2:18 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jul 10, 2013 at 12:20 PM, Mike Jackson <mike@routed.ca> wrote:
I just realized that it's not Google IP space (74.125.0.0/16), Rogers is hijacking the DNS and resolving www.google.com to space within 64.71.240.0/20 which is Rogers IP space! (note the name server set as 8.8.8.8)
so: 1) rogers is hijacking traffic to 8.8.8.8 2) the copy of 8.8.8.8 in rogers-land is replying with incorrect information for google properties (at least).
err... any idea if it's lying about other things too? :)
oops, so a polite caller noted that google often ships people at a local 'google global cache' node if they have one... it seems like that might be what's going on here with 8.8.8.8 sending you to 64.71 and/or 66.185 addresses. so maybe rogers isn't doing something untoward after all! -chris (thanks for the headslap polite caller)
davinci#traceroute www.google.com
Translating "www.google.com"...domain server (8.8.8.8) [OK]
Type escape sequence to abort. Tracing the route to www.google.com (66.185.85.29)
1 x.x.x.x [AS 812] 4 msec 4 msec 4 msec 2 so-4-0-2.gw02.ym.phub.net.cable.rogers.com (66.185.82.129) [AS 812] 8 msec 8 msec 4 msec 3 69.63.252.222 [AS 812] 4 msec 0 msec 8 msec 4 69.63.250.162 [AS 812] 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * *
TOR2-CORE-R1#show ip bgp 66.185.85.29 BGP routing table entry for 66.185.80.0/20, version 13095115 Paths: (1 available, best #1, table default) Advertised to update-groups: 14 701 6461 812 205.205.23.121 from 205.205.23.121 (137.39.8.42) Origin IGP, localpref 100, valid, external, best TOR2-CORE-R1#
Thanks,
- Mike
On Wed, Jul 10, 2013 at 11:28 AM, Mike Jackson <mike@routed.ca> wrote:
I can see the Google IP space (64.71.240.0/20) from Verizon/AS701, but not from Rogers/AS812 in Toronto. I've done a few other test traceroutes through Rogers to verify that they didn't filter UDP/ICMP. At this point nothing would surprise me from Rogers.
AS701 =====
TOR2-CORE-R1#traceroute www.google.com Translating "www.google.com"...domain server (8.8.8.8) [OK]
Type escape sequence to abort. Tracing the route to www.google.com (74.125.26.99) VRF info: (vrf in name/id, vrf out name/id) 1 x.x.x.x [AS701] 4 msec 0 msec 0 msec 2 0.ge-11-0-0.XT4.TOR2.ALTER.NET (152.63.133.78) [AS 701] 4 msec 0 msec 4 msec 3 0.so-4-0-3.XT2.NYC4.ALTER.NET (152.63.0.73) [AS 701] 16 msec 16 msec 16 msec 4 TenGigE0-7-1-0.GW8.NYC4.ALTER.NET (152.63.21.125) [AS 701] 20 msec TenGigE0-5-1-0.GW8.NYC4.ALTER.NET (152.63.21.73) [AS 701] 16 msec TenGigE0-5-4-0.GW8.NYC4.ALTER.NET (152.63.18.206) [AS 701] 20 msec 5 72.14.238.232 [AS 15169] 16 msec 16 msec 20 msec 6 72.14.236.208 [AS 15169] [MPLS: Label 680976 Exp 4] 16 msec 16 msec 72.14.236.206 [AS 15169] [MPLS: Label 533197 Exp 4] 20 msec 7 209.85.249.11 [AS 15169] [MPLS: Label 16668 Exp 4] 24 msec 72.14.239.93 [AS 15169] [MPLS: Label 14644 Exp 4] 24 msec 209.85.249.11 [AS 15169] [MPLS: Label 13978 Exp 4] 24 msec 8 209.85.243.114 [AS 15169] [MPLS: Label 568789 Exp 4] 32 msec 209.85.241.222 [AS 15169] [MPLS: Label 632535 Exp 4] 32 msec 32 msec 9 216.239.48.159 [AS 15169] 36 msec 32 msec 216.239.48.59 [AS 15169] 32 msec 10 www.google.com (74.125.26.147) [AS 15169] 32 msec 32 msec 32 msec
AS812 (Rogers Looking Glass https://supernoc.rogerstelecom.net/lg/) ===================================================
*Query:* *tr 64.71.249.45*
Type escape sequence to abort. Tracing the route to 64.71.249.45
1 64.71.255.62 0 msec 0 msec 0 msec 2 ge-4-3-0.gw02.ym.phub.net.cable.ROGERS.com (66.185.82.237) 4 msec 4 msec 4 msec 3 69.63.250.189 4 msec 4 msec 4 msec 4 69.63.250.174 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * *
Cheers,
- MJ
-----Original Message-----
From: Grant Ridder [mailto:shortdudey123@gmail.com] Sent: July-10-13 10:57 AM To: John York Cc: nanog@nanog.org Subject: Re: google troubles?
Does anyone have traceroutes showing where the issues are?
-Grant
On Wed, Jul 10, 2013 at 7:45 AM, John York <johny@griffintechnology.com>wrote:
We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working.
...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes.
On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper <blair.trosper@gmail.com
wrote:
Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine.
It's anecdotal, but it would seem like there's an issue based on these reports.
Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html
Anyone know what's up? Fiber cut? DC outages?
-- blair
--
John York
Information Technology | Network Administrator
Phone: 615-399-7000 x:333
Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA
This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
That makes sense, I figured it was some type of a CDN caching environment. I've found that youtube.com is in the same boat: Translating "www.youtube.com"...domain server (8.8.8.8) [OK] Type escape sequence to abort. Tracing the route to youtube-ui.l.google.com (24.156.153.40) [synkro:ROOT](~): jwhois 24.156.153.40 | grep -i orgname OrgName: Rogers Cable Communications Inc. [synkro:ROOT](~): Thanks! - MJ On Wed, Jul 10, 2013 at 2:39 PM, Christopher Morrow <morrowc.lists@gmail.com
wrote:
On Wed, Jul 10, 2013 at 2:18 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jul 10, 2013 at 12:20 PM, Mike Jackson <mike@routed.ca> wrote:
I just realized that it's not Google IP space (74.125.0.0/16), Rogers is hijacking the DNS and resolving www.google.com to space within 64.71.240.0/20 which is Rogers IP space! (note the name server set as 8.8.8.8)
so: 1) rogers is hijacking traffic to 8.8.8.8 2) the copy of 8.8.8.8 in rogers-land is replying with incorrect information for google properties (at least).
err... any idea if it's lying about other things too? :)
oops, so a polite caller noted that google often ships people at a local 'google global cache' node if they have one... it seems like that might be what's going on here with 8.8.8.8 sending you to 64.71 and/or 66.185 addresses.
so maybe rogers isn't doing something untoward after all!
-chris (thanks for the headslap polite caller)
davinci#traceroute www.google.com
Translating "www.google.com"...domain server (8.8.8.8) [OK]
Type escape sequence to abort. Tracing the route to www.google.com (66.185.85.29)
1 x.x.x.x [AS 812] 4 msec 4 msec 4 msec 2 so-4-0-2.gw02.ym.phub.net.cable.rogers.com (66.185.82.129) [AS 812] 8 msec 8 msec 4 msec 3 69.63.252.222 [AS 812] 4 msec 0 msec 8 msec 4 69.63.250.162 [AS 812] 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * *
TOR2-CORE-R1#show ip bgp 66.185.85.29 BGP routing table entry for 66.185.80.0/20, version 13095115 Paths: (1 available, best #1, table default) Advertised to update-groups: 14 701 6461 812 205.205.23.121 from 205.205.23.121 (137.39.8.42) Origin IGP, localpref 100, valid, external, best TOR2-CORE-R1#
Thanks,
- Mike
On Wed, Jul 10, 2013 at 11:28 AM, Mike Jackson <mike@routed.ca> wrote:
I can see the Google IP space (64.71.240.0/20) from Verizon/AS701, but not from Rogers/AS812 in Toronto. I've done a few other test traceroutes through Rogers to verify that they didn't filter UDP/ICMP. At this point nothing would surprise me from Rogers.
AS701 =====
TOR2-CORE-R1#traceroute www.google.com Translating "www.google.com"...domain server (8.8.8.8) [OK]
Type escape sequence to abort. Tracing the route to www.google.com (74.125.26.99) VRF info: (vrf in name/id, vrf out name/id) 1 x.x.x.x [AS701] 4 msec 0 msec 0 msec 2 0.ge-11-0-0.XT4.TOR2.ALTER.NET (152.63.133.78) [AS 701] 4 msec 0 msec 4 msec 3 0.so-4-0-3.XT2.NYC4.ALTER.NET (152.63.0.73) [AS 701] 16 msec 16 msec 16 msec 4 TenGigE0-7-1-0.GW8.NYC4.ALTER.NET (152.63.21.125) [AS 701] 20 msec TenGigE0-5-1-0.GW8.NYC4.ALTER.NET (152.63.21.73) [AS 701] 16 msec TenGigE0-5-4-0.GW8.NYC4.ALTER.NET (152.63.18.206) [AS 701] 20 msec 5 72.14.238.232 [AS 15169] 16 msec 16 msec 20 msec 6 72.14.236.208 [AS 15169] [MPLS: Label 680976 Exp 4] 16 msec 16 msec 72.14.236.206 [AS 15169] [MPLS: Label 533197 Exp 4] 20 msec 7 209.85.249.11 [AS 15169] [MPLS: Label 16668 Exp 4] 24 msec 72.14.239.93 [AS 15169] [MPLS: Label 14644 Exp 4] 24 msec 209.85.249.11 [AS 15169] [MPLS: Label 13978 Exp 4] 24 msec 8 209.85.243.114 [AS 15169] [MPLS: Label 568789 Exp 4] 32 msec 209.85.241.222 [AS 15169] [MPLS: Label 632535 Exp 4] 32 msec 32 msec 9 216.239.48.159 [AS 15169] 36 msec 32 msec 216.239.48.59 [AS 15169] 32 msec 10 www.google.com (74.125.26.147) [AS 15169] 32 msec 32 msec 32 msec
AS812 (Rogers Looking Glass https://supernoc.rogerstelecom.net/lg/) ===================================================
*Query:* *tr 64.71.249.45*
Type escape sequence to abort. Tracing the route to 64.71.249.45
1 64.71.255.62 0 msec 0 msec 0 msec 2 ge-4-3-0.gw02.ym.phub.net.cable.ROGERS.com (66.185.82.237) 4 msec 4 msec 4 msec 3 69.63.250.189 4 msec 4 msec 4 msec 4 69.63.250.174 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * *
Cheers,
- MJ
-----Original Message-----
From: Grant Ridder [mailto:shortdudey123@gmail.com] Sent: July-10-13 10:57 AM To: John York Cc: nanog@nanog.org Subject: Re: google troubles?
Does anyone have traceroutes showing where the issues are?
-Grant
On Wed, Jul 10, 2013 at 7:45 AM, John York <johny@griffintechnology.com>wrote:
We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working.
...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes.
On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper <blair.trosper@gmail.com >wrote:
> Seeing lots of reports of people unable to get to many Google services. > Seems to be affecting Comcast users disproportionately. It's fine > for me, > but a lot of my staff are basically out of luck...but according to > the Google Apps Status page, everything is fine. > > It's anecdotal, but it would seem like there's an issue based on > these reports. > > Oh, and this: > http://www.cnn.com/2013/07/10/tech/web/google-down/index.html > > Anyone know what's up? Fiber cut? DC outages? > > -- blair >
--
John York
Information Technology | Network Administrator
Phone: 615-399-7000 x:333
Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA
This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
On Wed, Jul 10, 2013 at 2:55 PM, Mike Jackson <mike@routed.ca> wrote:
That makes sense, I figured it was some type of a CDN caching environment.
I forget that we do this at times... :)
I've found that youtube.com is in the same boat:
Translating "www.youtube.com"...domain server (8.8.8.8) [OK]
yup, so this makes sense as well... good.
Hey Chris, long time!
From what I can tell, it's only Google Services (that I've found so far; other things appear to resolve correctly). I'm wondering if they're bouncing Google based traffic through some type of caching / accelerator? Or maybe it's an NSA/DPI box ;)
I've tested Maps, Gmail, Translate as well as www.google.com, www.google.caand they're all hijacked replies ---->> Translating "www.google.com"...domain server (8.8.8.8) [OK] (www.google.com) Type escape sequence to abort. Tracing the route to www.google.com (66.185.84.44) (www.google.ca) Type escape sequence to abort. Tracing the route to www.google.ca (66.185.95.44) (gmail) Type escape sequence to abort. Tracing the route to www3.l.google.com (66.185.85.39) (maps) Type escape sequence to abort. Tracing the route to maps.l.google.com (64.71.249.114) (translate) Type escape sequence to abort. Tracing the route to www3.l.google.com (66.185.84.30) Cheers, - Mike On Wed, Jul 10, 2013 at 2:18 PM, Christopher Morrow <morrowc.lists@gmail.com
wrote:
On Wed, Jul 10, 2013 at 12:20 PM, Mike Jackson <mike@routed.ca> wrote:
I just realized that it's not Google IP space (74.125.0.0/16), Rogers is hijacking the DNS and resolving www.google.com to space within 64.71.240.0/20 which is Rogers IP space! (note the name server set as 8.8.8.8)
so: 1) rogers is hijacking traffic to 8.8.8.8 2) the copy of 8.8.8.8 in rogers-land is replying with incorrect information for google properties (at least).
err... any idea if it's lying about other things too? :)
davinci#traceroute www.google.com
Translating "www.google.com"...domain server (8.8.8.8) [OK]
Type escape sequence to abort. Tracing the route to www.google.com (66.185.85.29)
1 x.x.x.x [AS 812] 4 msec 4 msec 4 msec 2 so-4-0-2.gw02.ym.phub.net.cable.rogers.com (66.185.82.129) [AS 812] 8 msec 8 msec 4 msec 3 69.63.252.222 [AS 812] 4 msec 0 msec 8 msec 4 69.63.250.162 [AS 812] 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * *
TOR2-CORE-R1#show ip bgp 66.185.85.29 BGP routing table entry for 66.185.80.0/20, version 13095115 Paths: (1 available, best #1, table default) Advertised to update-groups: 14 701 6461 812 205.205.23.121 from 205.205.23.121 (137.39.8.42) Origin IGP, localpref 100, valid, external, best TOR2-CORE-R1#
Thanks,
- Mike
On Wed, Jul 10, 2013 at 11:28 AM, Mike Jackson <mike@routed.ca> wrote:
I can see the Google IP space (64.71.240.0/20) from Verizon/AS701, but not from Rogers/AS812 in Toronto. I've done a few other test traceroutes through Rogers to verify that they didn't filter UDP/ICMP. At this point nothing would surprise me from Rogers.
AS701 =====
TOR2-CORE-R1#traceroute www.google.com Translating "www.google.com"...domain server (8.8.8.8) [OK]
Type escape sequence to abort. Tracing the route to www.google.com (74.125.26.99) VRF info: (vrf in name/id, vrf out name/id) 1 x.x.x.x [AS701] 4 msec 0 msec 0 msec 2 0.ge-11-0-0.XT4.TOR2.ALTER.NET (152.63.133.78) [AS 701] 4 msec 0 msec 4 msec 3 0.so-4-0-3.XT2.NYC4.ALTER.NET (152.63.0.73) [AS 701] 16 msec 16 msec 16 msec 4 TenGigE0-7-1-0.GW8.NYC4.ALTER.NET (152.63.21.125) [AS 701] 20 msec TenGigE0-5-1-0.GW8.NYC4.ALTER.NET (152.63.21.73) [AS 701] 16 msec TenGigE0-5-4-0.GW8.NYC4.ALTER.NET (152.63.18.206) [AS 701] 20 msec 5 72.14.238.232 [AS 15169] 16 msec 16 msec 20 msec 6 72.14.236.208 [AS 15169] [MPLS: Label 680976 Exp 4] 16 msec 16 msec 72.14.236.206 [AS 15169] [MPLS: Label 533197 Exp 4] 20 msec 7 209.85.249.11 [AS 15169] [MPLS: Label 16668 Exp 4] 24 msec 72.14.239.93 [AS 15169] [MPLS: Label 14644 Exp 4] 24 msec 209.85.249.11 [AS 15169] [MPLS: Label 13978 Exp 4] 24 msec 8 209.85.243.114 [AS 15169] [MPLS: Label 568789 Exp 4] 32 msec 209.85.241.222 [AS 15169] [MPLS: Label 632535 Exp 4] 32 msec 32 msec 9 216.239.48.159 [AS 15169] 36 msec 32 msec 216.239.48.59 [AS 15169] 32 msec 10 www.google.com (74.125.26.147) [AS 15169] 32 msec 32 msec 32 msec
AS812 (Rogers Looking Glass https://supernoc.rogerstelecom.net/lg/) ===================================================
*Query:* *tr 64.71.249.45*
Type escape sequence to abort. Tracing the route to 64.71.249.45
1 64.71.255.62 0 msec 0 msec 0 msec 2 ge-4-3-0.gw02.ym.phub.net.cable.ROGERS.com (66.185.82.237) 4 msec 4 msec 4 msec 3 69.63.250.189 4 msec 4 msec 4 msec 4 69.63.250.174 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * *
Cheers,
- MJ
-----Original Message-----
From: Grant Ridder [mailto:shortdudey123@gmail.com] Sent: July-10-13 10:57 AM To: John York Cc: nanog@nanog.org Subject: Re: google troubles?
Does anyone have traceroutes showing where the issues are?
-Grant
On Wed, Jul 10, 2013 at 7:45 AM, John York <johny@griffintechnology.com>wrote:
We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working.
...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes.
On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper <blair.trosper@gmail.com
wrote:
Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine.
It's anecdotal, but it would seem like there's an issue based on these reports.
Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html
Anyone know what's up? Fiber cut? DC outages?
-- blair
--
John York
Information Technology | Network Administrator
Phone: 615-399-7000 x:333
Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA
This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
On Wed, Jul 10, 2013 at 2:41 PM, Mike Jackson <mike@routed.ca> wrote:
Hey Chris, long time!
From what I can tell, it's only Google Services (that I've found so far; other things appear to resolve correctly). I'm wondering if they're bouncing Google based traffic through some type of caching / accelerator? Or maybe it's an NSA/DPI box ;)
in .CA? :) don't you mean the meat-helmet-wearing CSE folks? :)
I've tested Maps, Gmail, Translate as well as www.google.com, www.google.ca and they're all hijacked replies ---->>
yup... well, not hijacked in a bad sense... it's actually supposed to be making things better.
Translating "www.google.com"...domain server (8.8.8.8) [OK]
(www.google.com)
Type escape sequence to abort. Tracing the route to www.google.com (66.185.84.44)
(www.google.ca)
Type escape sequence to abort. Tracing the route to www.google.ca (66.185.95.44)
(gmail)
Type escape sequence to abort. Tracing the route to www3.l.google.com (66.185.85.39)
(maps)
Type escape sequence to abort. Tracing the route to maps.l.google.com (64.71.249.114)
(translate)
Type escape sequence to abort. Tracing the route to www3.l.google.com (66.185.84.30)
those all seem properly done, yes. -chris
participants (2)
-
Christopher Morrow -
Mike Jackson