Anyone from AS7160 (Oracle) around ?
Hi, I have been trying to get a hold of someone who looks after ASN 7160 since last Thursday both directly (OrgTechEmail), and indirectly via upstreams with no luck. I am trying to resolve or at least understand a routing reachability issue between our two networks. It seems packets from ASN7160 are not able to get back to some of my netblocks in AS 11647. e.g. eg. this is fine % traceroute -q1 -s 98.159.240.105 -Picmp 68.233.77.173 traceroute to 68.233.77.173 (68.233.77.173) from 98.159.240.105, 64 hops max, 72 byte packets 1 cogent-vl38-tor-hespler-v38 (205.211.165.117) 0.091 ms 2 gi1-18.mag02.yyz02.atlas.cogentco.com (38.104.158.77) 0.701 ms 3 te0-7-0-12.ccr22.yyz02.atlas.cogentco.com (154.54.40.165) 0.765 ms 4 be2080.ccr42.ord01.atlas.cogentco.com (154.54.42.5) 15.871 ms 5 be2003.ccr21.ord03.atlas.cogentco.com (154.54.29.22) 17.882 ms 6 38.104.102.102 (38.104.102.102) 17.395 ms 7 border2.te7-1-bbnet1.chg004.pnap.net (64.94.32.19) 16.335 ms 8 oraclebol-7.border2.chg004.pnap.net (74.217.8.106) 16.945 ms 9 VIP-CH-77-173.taleo.net (68.233.77.173) 17.014 ms This is not % traceroute -q1 -s 205.211.165.119 -Picmp 68.233.77.173 traceroute to 68.233.77.173 (68.233.77.173) from 205.211.165.119, 64 hops max, 72 byte packets 1 cogent-vl38-tor-hespler-v38 (205.211.165.117) 0.145 ms 2 gi1-18.mag02.yyz02.atlas.cogentco.com (38.104.158.77) 7.926 ms 3 te0-7-0-12.ccr22.yyz02.atlas.cogentco.com (154.54.40.165) 1.081 ms 4 be2080.ccr42.ord01.atlas.cogentco.com (154.54.42.5) 15.699 ms 5 be2003.ccr21.ord03.atlas.cogentco.com (154.54.29.22) 15.394 ms 6 * 7 * 8 * Same with 64.7.137.0/24 and 64.7.135.0/24 for some reason, but not all subnets within 64.7.128.0/19 are blocked. % traceroute -q1 -s 64.7.135.1 -Picmp 68.233.77.173 traceroute to 68.233.77.173 (68.233.77.173) from 64.7.135.1, 64 hops max, 60 byte packets 1 iolite3 (199.212.135.73) 0.234 ms 2 cogent-vl108 (67.43.129.246) 2.575 ms 3 gi1-18.mag02.yyz02.atlas.cogentco.com (38.104.158.77) 2.844 ms 4 te0-7-0-12.ccr21.yyz02.atlas.cogentco.com (154.54.40.137) 3.460 ms 5 be2079.ccr41.ord01.atlas.cogentco.com (154.54.27.181) 17.338 ms 6 be2005.ccr21.ord03.atlas.cogentco.com (66.28.4.74) 17.962 ms 7 * vs % traceroute -q1 -s 64.7.138.225 -Picmp 68.233.77.173 traceroute to 68.233.77.173 (68.233.77.173) from 64.7.138.225, 64 hops max, 60 byte packets 1 iolite3 (199.212.135.73) 0.193 ms 2 cogent-vl108 (67.43.129.246) 2.210 ms 3 gi1-18.mag02.yyz02.atlas.cogentco.com (38.104.158.77) 2.469 ms 4 te0-7-0-12.ccr22.yyz02.atlas.cogentco.com (154.54.40.165) 3.091 ms 5 be2080.ccr42.ord01.atlas.cogentco.com (154.54.42.5) 17.566 ms 6 be2003.ccr21.ord03.atlas.cogentco.com (154.54.29.22) 18.457 ms 7 38.104.102.102 (38.104.102.102) 17.831 ms 8 border2.te8-1-bbnet2.chg004.pnap.net (64.94.32.83) 22.324 ms 9 oraclebol-7.border2.chg004.pnap.net (74.217.8.106) 19.091 ms 10 VIP-CH-77-173.taleo.net (68.233.77.173) 20.318 ms The issue started around July 3rd some time. I have tried the listed contacts OrgTechHandle: NOC2096-ARIN OrgTechName: Network Operation Center OrgTechPhone: +1-877-524-5665 OrgTechEmail: network-contact_ww@oracle.com OrgTechRef: http://whois.arin.net/rest/poc/NOC2096-ARIN with no luck since last Thursday. The toll free people were trying their best to understand who within Oracle I was trying to reach, but its not part of their normal decision tree. via TATA and abovenet gives similar results. traceroute -q1 -Picmp -s 205.211.165.121 68.233.77.173 traceroute to 68.233.77.173 (68.233.77.173) from 205.211.165.121, 64 hops max, 72 byte packets 1 ix-11-2-3-0.tcore1.TNK-Toronto.as6453.net (209.58.16.21) 0.170 ms 2 if-5-0-0-5.core4.TNK-Toronto.as6453.net (63.243.172.25) 30.856 ms 3 if-0-3-2-0.tcore1.CT8-Chicago.as6453.net (63.243.172.34) 14.203 ms 4 63.243.129.14 (63.243.129.14) 13.967 ms 5 ae4.cr1.ord2.us.above.net (64.125.28.49) 12.203 ms 6 ae9.mpr1.ord11.us.above.net (64.125.24.106) 13.043 ms 7 ae4.mpr1.ord5.us.above.net (64.125.24.94) 15.027 ms 8 * traceroute -q1 -Picmp -s 67.43.129.244 68.233.77.173 traceroute to 68.233.77.173 (68.233.77.173) from 67.43.129.244, 64 hops max, 72 byte packets 1 ix-11-2-3-0.tcore1.TNK-Toronto.as6453.net (209.58.16.21) 29.514 ms 2 if-11-0-0-4.core4.TNK-Toronto.as6453.net (64.86.33.42) 0.376 ms 3 if-2-3-2-0.tcore1.CT8-Chicago.as6453.net (63.243.172.42) 12.124 ms 4 63.243.129.14 (63.243.129.14) 13.624 ms 5 ae4.cr1.ord2.us.above.net (64.125.28.49) 12.607 ms 6 ae9.mpr1.ord11.us.above.net (64.125.24.106) 14.290 ms 7 ae4.mpr1.ord5.us.above.net (64.125.24.94) 13.500 ms 8 208.185.21.162 (208.185.21.162) 13.476 ms 9 VIP-CH-77-173.taleo.net (68.233.77.173) 13.778 ms ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/
On Jul 8, 2014, at 9:15 AM, Mike Tancsa <mike@sentex.net> wrote:
Hi, I have been trying to get a hold of someone who looks after ASN 7160 since last Thursday both directly (OrgTechEmail), and indirectly via upstreams with no luck. I am trying to resolve or at least understand a routing reachability issue between our two networks. It seems packets from ASN7160 are not able to get back to some of my netblocks in AS 11647. e.g.
I created a public measurement for you to view the results from various locations: https://atlas.ripe.net/api/v1/measurement/1695662/result/ or if you get a free atlas.ripe.net account you can visualize it better here: https://atlas.ripe.net/atlas/udm.html?msm_id=1695662 - Jared
Thanks to an unnamed frontline staff member who worked hard to find the right people at Oracle, I found the right people at Oracle-- She had no idea what I was talking about, but knew how to figure out how to find who I needed and didnt give up! It seems Oracle is being sent bogus routing information from their PNAP peer. They are learning, what seems to be a random subnet of prefixes (two of which I am not even announcing-- 64.7.135.0/24 and 64.7.137.0/24) that are learned from Torix. The path that 7160 sees is 19024 29791 8001 11670 11647 They see the following prefixes leaking out of Torix. But if they do a traceroute, the packets just bounce around between Voxel and PNAP. Traceroute below. I have emailed the listed POCs, but no response. Anyone here from those 2 networks ? 64.7.128.0/24 *[BGP/170] 1w4d 11:08:57, localpref 100 AS path: 19024 29791 8001 11670 11647 I > to 74.217.8.105 via xe-2/2/0.0 64.7.135.0/24 *[BGP/170] 1w4d 12:03:19, localpref 100 AS path: 19024 29791 8001 11670 11647 I > to 74.217.8.105 via xe-2/2/0.0 64.7.137.0/24 *[BGP/170] 1w4d 13:52:08, localpref 100 AS path: 19024 29791 8001 11670 11647 I > to 74.217.8.105 via xe-2/2/0.0 198.235.180.0/24 *[BGP/170] 1w4d 06:07:16, localpref 100 AS path: 19024 29791 8001 11670 11647 I > to 74.217.8.105 via xe-2/2/0.0 198.235.181.0/24 *[BGP/170] 1w3d 06:53:31, localpref 100 AS path: 19024 29791 8001 11670 11647 I > to 74.217.8.105 via xe-2/2/0.0 198.235.183.0/24 *[BGP/170] 1w4d 02:44:55, localpref 100 AS path: 19024 29791 8001 11670 11647 I > to 74.217.8.105 via xe-2/2/0.0 204.138.108.0/24 [BGP/170] 1w4d 05:18:36, localpref 100 AS path: 19024 29791 8001 11670 11647 I > to 74.217.8.105 via xe-2/2/0.0 205.211.165.0/24 *[BGP/170] 1w4d 08:04:32, localpref 100 AS path: 19024 29791 8001 11670 11647 I > to 74.217.8.105 via xe-2/2/0.0 traceroute to 64.7.135.1 (64.7.135.1), 30 hops max, 40 byte packets 1 74.217.8.105 (74.217.8.105) 0.642 ms 0.514 ms 0.503 ms 2 64.94.32.14 (64.94.32.14) 1.479 ms 1.564 ms 1.503 ms 3 208.122.29.21 (208.122.29.21) 11.579 ms 1.428 ms 1.388 ms 4 66.151.28.149 (66.151.28.149) 1.773 ms 66.151.28.141 (66.151.28.141) 1.580 ms 1.524 ms 5 64.94.32.14 (64.94.32.14) 1.448 ms 1.554 ms 1.558 ms 6 208.122.29.21 (208.122.29.21) 10.274 ms 1.245 ms 1.232 ms 7 66.151.28.149 (66.151.28.149) 1.831 ms 1.800 ms 1.785 ms 8 64.94.32.78 (64.94.32.78) 2.027 ms 64.94.32.14 (64.94.32.14) 1.519 ms 1.605 ms 9 208.122.29.21 (208.122.29.21) 8.869 ms 1.291 ms 1.269 ms 10 66.151.28.149 (66.151.28.149) 1.872 ms 3.619 ms 1.869 ms 11 64.94.32.78 (64.94.32.78) 2.080 ms 1.956 ms 3.242 ms 12 208.122.29.21 (208.122.29.21) 6.033 ms 1.332 ms 1.302 ms 13 66.151.28.141 (66.151.28.141) 1.818 ms 2.006 ms 1.882 ms 14 64.94.32.14 (64.94.32.14) 1.699 ms 1.670 ms 1.615 ms 15 208.122.29.21 (208.122.29.21) 9.590 ms 1.566 ms 1.540 ms 16 66.151.28.149 (66.151.28.149) 1.891 ms 1.978 ms 1.869 ms 17 64.94.32.78 (64.94.32.78) 2.139 ms 64.94.32.14 (64.94.32.14) 1.726 ms 1.741 ms 18 208.122.29.21 (208.122.29.21) 8.236 ms 1.416 ms 1.403 ms 19 66.151.28.149 (66.151.28.149) 2.251 ms 1.987 ms 1.968 ms 20 64.94.32.78 (64.94.32.78) 2.154 ms 64.94.32.14 (64.94.32.14) 1.818 ms 1.754 ms 21 208.122.29.21 (208.122.29.21) 7.431 ms 1.684 ms 1.434 ms 22 66.151.28.141 (66.151.28.141) 1.758 ms 1.764 ms 2.006 ms 23 64.94.32.14 (64.94.32.14) 1.798 ms 1.691 ms 64.94.32.78 (64.94.32.78) 2.201 ms 24 208.122.29.21 (208.122.29.21) 8.134 ms 1.469 ms 1.485 ms 25 66.151.28.141 (66.151.28.141) 10.774 ms 1.885 ms 66.151.28.149 (66.151.28.149) 1.794 ms 26 64.94.32.14 (64.94.32.14) 1.720 ms 64.94.32.78 (64.94.32.78) 3.555 ms 64.94.32.14 (64.94.32.14) 1.831 ms 27 208.122.29.21 (208.122.29.21) 1.762 ms 1.696 ms 1.714 ms 28 66.151.28.149 (66.151.28.149) 2.074 ms 66.151.28.141 (66.151.28.141) 2.286 ms 1.919 ms 29 64.94.32.78 (64.94.32.78) 3.616 ms 2.337 ms 64.94.32.14 (64.94.32.14) 1.911 ms 30 208.122.29.21 (208.122.29.21) 1.737 ms 1.528 ms 1.554 ms ---Mike On 7/8/2014 9:15 AM, Mike Tancsa wrote:
Hi, I have been trying to get a hold of someone who looks after ASN 7160 since last Thursday both directly (OrgTechEmail), and indirectly via upstreams with no luck. I am trying to resolve or at least understand a routing reachability issue between our two networks. It seems packets from ASN7160 are not able to get back to some of my netblocks in AS 11647. e.g.
eg. this is fine % traceroute -q1 -s 98.159.240.105 -Picmp 68.233.77.173 traceroute to 68.233.77.173 (68.233.77.173) from 98.159.240.105, 64 hops max, 72 byte packets 1 cogent-vl38-tor-hespler-v38 (205.211.165.117) 0.091 ms 2 gi1-18.mag02.yyz02.atlas.cogentco.com (38.104.158.77) 0.701 ms 3 te0-7-0-12.ccr22.yyz02.atlas.cogentco.com (154.54.40.165) 0.765 ms 4 be2080.ccr42.ord01.atlas.cogentco.com (154.54.42.5) 15.871 ms 5 be2003.ccr21.ord03.atlas.cogentco.com (154.54.29.22) 17.882 ms 6 38.104.102.102 (38.104.102.102) 17.395 ms 7 border2.te7-1-bbnet1.chg004.pnap.net (64.94.32.19) 16.335 ms 8 oraclebol-7.border2.chg004.pnap.net (74.217.8.106) 16.945 ms 9 VIP-CH-77-173.taleo.net (68.233.77.173) 17.014 ms
This is not % traceroute -q1 -s 205.211.165.119 -Picmp 68.233.77.173 traceroute to 68.233.77.173 (68.233.77.173) from 205.211.165.119, 64 hops max, 72 byte packets 1 cogent-vl38-tor-hespler-v38 (205.211.165.117) 0.145 ms 2 gi1-18.mag02.yyz02.atlas.cogentco.com (38.104.158.77) 7.926 ms 3 te0-7-0-12.ccr22.yyz02.atlas.cogentco.com (154.54.40.165) 1.081 ms 4 be2080.ccr42.ord01.atlas.cogentco.com (154.54.42.5) 15.699 ms 5 be2003.ccr21.ord03.atlas.cogentco.com (154.54.29.22) 15.394 ms 6 * 7 * 8 *
Same with 64.7.137.0/24 and 64.7.135.0/24 for some reason, but not all subnets within 64.7.128.0/19 are blocked.
% traceroute -q1 -s 64.7.135.1 -Picmp 68.233.77.173 traceroute to 68.233.77.173 (68.233.77.173) from 64.7.135.1, 64 hops max, 60 byte packets 1 iolite3 (199.212.135.73) 0.234 ms 2 cogent-vl108 (67.43.129.246) 2.575 ms 3 gi1-18.mag02.yyz02.atlas.cogentco.com (38.104.158.77) 2.844 ms 4 te0-7-0-12.ccr21.yyz02.atlas.cogentco.com (154.54.40.137) 3.460 ms 5 be2079.ccr41.ord01.atlas.cogentco.com (154.54.27.181) 17.338 ms 6 be2005.ccr21.ord03.atlas.cogentco.com (66.28.4.74) 17.962 ms 7 * vs % traceroute -q1 -s 64.7.138.225 -Picmp 68.233.77.173 traceroute to 68.233.77.173 (68.233.77.173) from 64.7.138.225, 64 hops max, 60 byte packets 1 iolite3 (199.212.135.73) 0.193 ms 2 cogent-vl108 (67.43.129.246) 2.210 ms 3 gi1-18.mag02.yyz02.atlas.cogentco.com (38.104.158.77) 2.469 ms 4 te0-7-0-12.ccr22.yyz02.atlas.cogentco.com (154.54.40.165) 3.091 ms 5 be2080.ccr42.ord01.atlas.cogentco.com (154.54.42.5) 17.566 ms 6 be2003.ccr21.ord03.atlas.cogentco.com (154.54.29.22) 18.457 ms 7 38.104.102.102 (38.104.102.102) 17.831 ms 8 border2.te8-1-bbnet2.chg004.pnap.net (64.94.32.83) 22.324 ms 9 oraclebol-7.border2.chg004.pnap.net (74.217.8.106) 19.091 ms 10 VIP-CH-77-173.taleo.net (68.233.77.173) 20.318 ms
The issue started around July 3rd some time. I have tried the listed contacts
OrgTechHandle: NOC2096-ARIN OrgTechName: Network Operation Center OrgTechPhone: +1-877-524-5665 OrgTechEmail: network-contact_ww@oracle.com OrgTechRef: http://whois.arin.net/rest/poc/NOC2096-ARIN
with no luck since last Thursday. The toll free people were trying their best to understand who within Oracle I was trying to reach, but its not part of their normal decision tree.
via TATA and abovenet gives similar results.
traceroute -q1 -Picmp -s 205.211.165.121 68.233.77.173 traceroute to 68.233.77.173 (68.233.77.173) from 205.211.165.121, 64 hops max, 72 byte packets 1 ix-11-2-3-0.tcore1.TNK-Toronto.as6453.net (209.58.16.21) 0.170 ms 2 if-5-0-0-5.core4.TNK-Toronto.as6453.net (63.243.172.25) 30.856 ms 3 if-0-3-2-0.tcore1.CT8-Chicago.as6453.net (63.243.172.34) 14.203 ms 4 63.243.129.14 (63.243.129.14) 13.967 ms 5 ae4.cr1.ord2.us.above.net (64.125.28.49) 12.203 ms 6 ae9.mpr1.ord11.us.above.net (64.125.24.106) 13.043 ms 7 ae4.mpr1.ord5.us.above.net (64.125.24.94) 15.027 ms 8 * traceroute -q1 -Picmp -s 67.43.129.244 68.233.77.173 traceroute to 68.233.77.173 (68.233.77.173) from 67.43.129.244, 64 hops max, 72 byte packets 1 ix-11-2-3-0.tcore1.TNK-Toronto.as6453.net (209.58.16.21) 29.514 ms 2 if-11-0-0-4.core4.TNK-Toronto.as6453.net (64.86.33.42) 0.376 ms 3 if-2-3-2-0.tcore1.CT8-Chicago.as6453.net (63.243.172.42) 12.124 ms 4 63.243.129.14 (63.243.129.14) 13.624 ms 5 ae4.cr1.ord2.us.above.net (64.125.28.49) 12.607 ms 6 ae9.mpr1.ord11.us.above.net (64.125.24.106) 14.290 ms 7 ae4.mpr1.ord5.us.above.net (64.125.24.94) 13.500 ms 8 208.185.21.162 (208.185.21.162) 13.476 ms 9 VIP-CH-77-173.taleo.net (68.233.77.173) 13.778 ms
---Mike
-- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/
It seems Oracle is being sent bogus routing information from their PNAP peer. They are learning, what seems to be a random subnet of prefixes (two of which I am not even announcing-- 64.7.135.0/24 and 64.7.137.0/24) that are learned from Torix. The path that 7160 sees is
19024 29791 8001 11670 11647
The nice people at pnap actually took my call--I have had a couple in the past say, "Sorry, you are not our customer. <click>".. They found an issue with their routing engine injecting stale / bogus info into parts of their network and corrected it. ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/
participants (2)
-
Jared Mauch -
Mike Tancsa