Long hops on international paths
Hello, I am a researcher at the University of Wisconsin. My colleagues at Northwestern University and I are studying international Internet connectivity and would appreciate your perspective on a recent finding. We're using traceroute data from CAIDA's Ark project for our work. We've observed that many international links (i.e., a single hop on an end-to-end path that connects two countries where end points on the hop are identified via rDNS) tend to originate/terminate at the same routers. Said another way, we are observing a relatively small set of routers in different countries tend to have a majority of the international connections - this is especially the case for hops that terminate in the US. For example, there is a router operated by Telia (AS1299) in Chicago that has a high concentration of such links. We were a bit surprised by this finding since even though it makes sense that the set of providers is relatively small (i.e., those that offer global connectivity), we assumed that the set of routers that used for international connectivity within any one country would tend to be more widely distributed (at least with respect to how they appear in traceroute data - MPLS notwithstanding). We're interested in whether or not this is indeed standard practice and if so, the cost/benefit for configuring international connectivity in this way? Any thoughts or insights you might have would be greatly appreciated - off-list responses are welcome. Thank you. Regards, PB Paul Barford University of Wisconsin - Madison
PAUL R BARFORD wrote on 17/01/2022 18:02:
For example, there is a router operated by Telia (AS1299) in Chicago that has a high concentration of such links.
this doesn't appear to match 1299's public network topology: https://www.teliacarrier.com/our-network.html Is ttl decrement disabled on the test paths you're measuring? Broadly speaking, if you have a point-to-point link from one location to another (or parallel set of links with a common failure path, e.g. waves on a specific fibre path), there's a single router at each end. Nick
Hello Nick, I've added my collaborators to this reply - Esteban can comment on your observation re. Telia. TTL is decremented on the paths we're analyzing. What we're curious about is why we're seeing a concentration of hops at a small number of routers that appear on international paths. I expected that when we looked at paths between e.g., US-Asia, US-Europe, US-South America (considering measurements from Ark nodes doing traceroutes to/from those locations), we would see the first instances of routers located the US along the west coast, east coast and south coast respectively. We did not expect to see Chicago as a first hop location in the US and are wondering e.g., if large providers do this to simplify their operations. Hopefully that makes sense. Any further thoughts are appreciated. Regards, PB ________________________________ From: Nick Hilliard <nick@foobar.org> Sent: Monday, January 17, 2022 12:36 PM To: PAUL R BARFORD <pb@cs.wisc.edu> Cc: nanog@nanog.org <nanog@nanog.org> Subject: Re: Long hops on international paths PAUL R BARFORD wrote on 17/01/2022 18:02:
For example, there is a router operated by Telia (AS1299) in Chicago that has a high concentration of such links.
this doesn't appear to match 1299's public network topology: https://www.teliacarrier.com/our-network.html Is ttl decrement disabled on the test paths you're measuring? Broadly speaking, if you have a point-to-point link from one location to another (or parallel set of links with a common failure path, e.g. waves on a specific fibre path), there's a single router at each end. Nick
On Mon, 17 Jan 2022 at 20:00, PAUL R BARFORD <pb@cs.wisc.edu> wrote:
What we're curious about is why we're seeing a concentration of hops at a small number of routers that appear on international paths.
I suggest you share a few actual examples (IP addresses, traceroutes). I don't think discussing your conclusion based on data we don't have makes sense. Lukas
Please find the examples for the case of Telia below. FROM jfk-us (jfk-us.team-probing.c008820.20201002.warts.gz) traceroute from 216.66.30.102 (Ark probe hosted in New York City, NY, US. No AS info found) to 223.114.235.32 (MAXMIXD: Turpan, CN) 1 216.66.30.101 0.365 ms 2 62.115.49.173 3.182 ms 3 * 4 62.115.137.59 17.453 ms [x] (chi-b23-link.ip.twelve99.net., CAIDA-GEOLOC -> Chicago, IL, US) 5 62.115.117.48 59.921 ms [x] (sea-b2-link.ip.twelve99.net., RIPE-IPMAP -> Seattle, WA, US) 6 62.115.171.221 69.993 ms 7 223.120.6.53 69.378 ms 8 223.120.12.34 226.225 ms 9 221.183.55.110 237.475 ms 10 221.183.25.201 238.697 ms 11 221.176.16.213 242.296 ms 12 221.183.36.62 352.695 ms 13 221.183.39.2 300.166 ms 14 117.191.8.118 316.270 ms 15 * 16 * 17 * 18 * 19 * FROM ord-us (ord-us.team-probing.c008820.20201002.warts.gz) traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 109.25.215.237 (237.215.25.109.rev.sfr.net., MAXMIXD: La Crau, FR) 1 140.192.218.129 0.795 ms 2 140.192.9.124 0.603 ms 3 64.124.44.158 1.099 ms 4 64.125.31.172 3.047 ms 5 * 6 64.125.15.65 1.895 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.118.59 99.242 ms [x] (prs-b3-link.ip.twelve99.net., CAIDA-GEOLOC -> Paris, FR) 8 62.115.154.23 105.214 ms 9 77.136.10.6 119.021 ms 10 77.136.10.6 118.830 ms 11 80.118.89.202 118.690 ms 12 80.118.89.234 118.986 ms 13 109.24.108.66 119.159 ms 14 109.25.215.237 126.085 ms traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 84.249.89.93 (dsl-tkubng12-54f959-93.dhcp.inet.fi., MAXMIXD: Turku, FI) 1 140.192.218.129 0.243 ms 2 140.192.9.124 0.326 ms 3 64.124.44.158 0.600 ms 4 * 5 * 6 64.125.15.65 1.792 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.123.27 121.199 ms [x] (hls-b4-link.ip.twelve99.net., CAIDA-GEOLOC -> Helsinki, FI) 8 * 9 141.208.193.190 127.723 ms 10 84.249.89.93 139.051 ms traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US) to 193.28.231.50 (MAXMIXD: None, HU) 1 140.192.218.129 0.240 ms 2 140.192.9.124 0.333 ms 3 64.124.44.158 0.648 ms 4 * 5 64.125.25.75 0.752 ms 6 64.125.15.65 1.877 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.119.39 123.952 ms [x] (bpt-b2-link.ip.twelve99.net., **I suspect it is in Budapest, HU**) 8 62.115.39.122 117.171 ms 9 88.151.96.148 117.202 ms 10 88.151.96.213 124.787 ms 11 * 12 * 13 * 14 * 15 * traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 152.195.4.11 (MAXMIXD: Los Angeles, CA, US) 1 140.192.218.129 0.224 ms 2 140.192.9.124 0.545 ms 3 64.124.44.158 0.640 ms 4 * 5 * 6 64.125.15.65 1.786 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.118.247 54.597 ms [x] (las-b22-link.ip.twelve99.net., CAIDA-GEOLOC -> Los Angeles, CA, US) 8 62.115.11.129 55.979 ms 9 * 10 * 11 * 12 * 13 * traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 47.31.143.217 (MAXMIXD: Delhi, IN) 1 140.192.218.129 2.277 ms 2 140.192.9.124 0.449 ms 3 64.124.44.158 0.576 ms 4 * 5 * 6 64.125.15.65 1.814 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.114.41 210.056 ms [x] (snge-b5-link.ip.twelve99.net.,) 8 62.115.177.11 200.840 ms 9 103.198.140.16 233.636 ms 10 103.198.140.16 232.871 ms 11 103.198.140.171 232.648 ms 12 * 13 * 14 * 15 * 16 * ________________________________ From: Lukas Tribus <lukas@ltri.eu> Sent: Monday, January 17, 2022 1:52 PM To: PAUL R BARFORD <pb@cs.wisc.edu> Cc: Nick Hilliard <nick@foobar.org>; nanog@nanog.org <nanog@nanog.org>; Esteban Carisimo <esteban.carisimo@northwestern.edu>; Fabian E. Bustamante <fabianb@cs.northwestern.edu> Subject: Re: Long hops on international paths On Mon, 17 Jan 2022 at 20:00, PAUL R BARFORD <pb@cs.wisc.edu> wrote:
What we're curious about is why we're seeing a concentration of hops at a small number of routers that appear on international paths.
I suggest you share a few actual examples (IP addresses, traceroutes). I don't think discussing your conclusion based on data we don't have makes sense. Lukas
Looking at your 1 repeat ORD example: On Tue, Jan 18, 2022 at 12:17 AM PAUL R BARFORD <pb@cs.wisc.edu> wrote:
6 64.125.15.65 1.895 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.118.59 99.242 ms [x] (prs-b3-link.ip.twelve99.net., CAIDA-GEOLOC -> Paris, FR)
65.15.125.64.in-addr.arpa name = zayo.telia.ter1.ord7.us.zip.zayo.com. 64.15.125.64.in-addr.arpa name = ae51.zayo.ter1.ord7.us.zip.zayo.com. it looks like the probes you selected (at least the depaul univ one(s)) are finding the 'best path' to whatever destinations via depaul -> zayo -> telia. It looks like zayo/telia interconnect at that /31. Based on: https://www.teliacarrier.com/dam/jcr:fc260a69-98a2-47d3-8d30-ca7095318413/te... i'd guess that: 1) telia has an mpls core with no-decrement-ttl enabled 2) the hidden hosp include NYC and possibly cleveland/wdc 3) judging the path information purely on traceroute hops is error prone.
1) all (meaning all hitting the zayo.telia) your traceroutes originate from University in Chicago 2) the zayo.telia device is physically close to the university 3) we should expect physically close-by backbone device to be present in disproportionate amount of traceroutes 4) almost certainly zayo.telia is imposing the MPLS label of TTL 255, _NOT_ copying IP TTL, therefore until MPLS label is popped, TTL is not expiring. I.e. you are seeing ingressPE and egress PE ot Telia, you are not seeing any P routers. This is not esoteric knowledge, but a fairly basic Internet concept. I am worried you are missing too much context to produce actionable output from your work. It might be interesting to see your curriculum, why this confusion arose, why it seems logical that the reason must be that almost all waves are terminated there, because it would not seem logical for people practising in the field who have even cursory understanding, this implies problems in the curriculum. On Tue, 18 Jan 2022 at 07:21, PAUL R BARFORD <pb@cs.wisc.edu> wrote:
Please find the examples for the case of Telia below.
FROM jfk-us (jfk-us.team-probing.c008820.20201002.warts.gz)
traceroute from 216.66.30.102 (Ark probe hosted in New York City, NY, US. No AS info found) to 223.114.235.32 (MAXMIXD: Turpan, CN)
1 216.66.30.101 0.365 ms
2 62.115.49.173 3.182 ms
3 *
4 62.115.137.59 17.453 ms [x] (chi-b23-link.ip.twelve99.net., CAIDA-GEOLOC -> Chicago, IL, US)
5 62.115.117.48 59.921 ms [x] (sea-b2-link.ip.twelve99.net., RIPE-IPMAP -> Seattle, WA, US)
6 62.115.171.221 69.993 ms
7 223.120.6.53 69.378 ms
8 223.120.12.34 226.225 ms
9 221.183.55.110 237.475 ms
10 221.183.25.201 238.697 ms
11 221.176.16.213 242.296 ms
12 221.183.36.62 352.695 ms
13 221.183.39.2 300.166 ms
14 117.191.8.118 316.270 ms
15 *
16 *
17 *
18 *
19 *
FROM ord-us (ord-us.team-probing.c008820.20201002.warts.gz)
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 109.25.215.237 (237.215.25.109.rev.sfr.net., MAXMIXD: La Crau, FR)
1 140.192.218.129 0.795 ms
2 140.192.9.124 0.603 ms
3 64.124.44.158 1.099 ms
4 64.125.31.172 3.047 ms
5 *
6 64.125.15.65 1.895 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.118.59 99.242 ms [x] (prs-b3-link.ip.twelve99.net., CAIDA-GEOLOC -> Paris, FR)
8 62.115.154.23 105.214 ms
9 77.136.10.6 119.021 ms
10 77.136.10.6 118.830 ms
11 80.118.89.202 118.690 ms
12 80.118.89.234 118.986 ms
13 109.24.108.66 119.159 ms
14 109.25.215.237 126.085 ms
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 84.249.89.93 (dsl-tkubng12-54f959-93.dhcp.inet.fi., MAXMIXD: Turku, FI)
1 140.192.218.129 0.243 ms
2 140.192.9.124 0.326 ms
3 64.124.44.158 0.600 ms
4 *
5 *
6 64.125.15.65 1.792 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.123.27 121.199 ms [x] (hls-b4-link.ip.twelve99.net., CAIDA-GEOLOC -> Helsinki, FI)
8 *
9 141.208.193.190 127.723 ms
10 84.249.89.93 139.051 ms
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US) to 193.28.231.50 (MAXMIXD: None, HU)
1 140.192.218.129 0.240 ms
2 140.192.9.124 0.333 ms
3 64.124.44.158 0.648 ms
4 *
5 64.125.25.75 0.752 ms
6 64.125.15.65 1.877 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.119.39 123.952 ms [x] (bpt-b2-link.ip.twelve99.net., **I suspect it is in Budapest, HU**)
8 62.115.39.122 117.171 ms
9 88.151.96.148 117.202 ms
10 88.151.96.213 124.787 ms
11 *
12 *
13 *
14 *
15 *
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 152.195.4.11 (MAXMIXD: Los Angeles, CA, US)
1 140.192.218.129 0.224 ms
2 140.192.9.124 0.545 ms
3 64.124.44.158 0.640 ms
4 *
5 *
6 64.125.15.65 1.786 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.118.247 54.597 ms [x] (las-b22-link.ip.twelve99.net., CAIDA-GEOLOC -> Los Angeles, CA, US)
8 62.115.11.129 55.979 ms
9 *
10 *
11 *
12 *
13 *
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 47.31.143.217 (MAXMIXD: Delhi, IN)
1 140.192.218.129 2.277 ms
2 140.192.9.124 0.449 ms
3 64.124.44.158 0.576 ms
4 *
5 *
6 64.125.15.65 1.814 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.114.41 210.056 ms [x] (snge-b5-link.ip.twelve99.net.,)
8 62.115.177.11 200.840 ms
9 103.198.140.16 233.636 ms
10 103.198.140.16 232.871 ms
11 103.198.140.171 232.648 ms
12 *
13 *
14 *
15 *
16 *
________________________________ From: Lukas Tribus <lukas@ltri.eu> Sent: Monday, January 17, 2022 1:52 PM To: PAUL R BARFORD <pb@cs.wisc.edu> Cc: Nick Hilliard <nick@foobar.org>; nanog@nanog.org <nanog@nanog.org>; Esteban Carisimo <esteban.carisimo@northwestern.edu>; Fabian E. Bustamante <fabianb@cs.northwestern.edu> Subject: Re: Long hops on international paths
On Mon, 17 Jan 2022 at 20:00, PAUL R BARFORD <pb@cs.wisc.edu> wrote:
What we're curious about is why we're seeing a concentration of hops at a small number of routers that appear on international paths.
I suggest you share a few actual examples (IP addresses, traceroutes).
I don't think discussing your conclusion based on data we don't have makes sense.
Lukas
-- ++ytti
Hello Saku, Thank you for the summary. We're clear about the fact that what we're seeing are MLPS paths - that was not in question. What we are not clear about and the reason for the post is why the provider - zayo.telia in this case - would decide to configure MPLS paths between Chicago and distant international locations. We assumed we would see hops in traceroute between Chicago and coastal locations and then hops that transited submarine infrastructure followed by hops to large population centers. Regards, PB ________________________________ From: Saku Ytti <saku@ytti.fi> Sent: Tuesday, January 18, 2022 12:50 AM To: PAUL R BARFORD <pb@cs.wisc.edu> Cc: Lukas Tribus <lukas@ltri.eu>; Esteban Carisimo <esteban.carisimo@northwestern.edu>; nanog@nanog.org <nanog@nanog.org>; Fabian E. Bustamante <fabianb@cs.northwestern.edu> Subject: Re: Long hops on international paths 1) all (meaning all hitting the zayo.telia) your traceroutes originate from University in Chicago 2) the zayo.telia device is physically close to the university 3) we should expect physically close-by backbone device to be present in disproportionate amount of traceroutes 4) almost certainly zayo.telia is imposing the MPLS label of TTL 255, _NOT_ copying IP TTL, therefore until MPLS label is popped, TTL is not expiring. I.e. you are seeing ingressPE and egress PE ot Telia, you are not seeing any P routers. This is not esoteric knowledge, but a fairly basic Internet concept. I am worried you are missing too much context to produce actionable output from your work. It might be interesting to see your curriculum, why this confusion arose, why it seems logical that the reason must be that almost all waves are terminated there, because it would not seem logical for people practising in the field who have even cursory understanding, this implies problems in the curriculum. On Tue, 18 Jan 2022 at 07:21, PAUL R BARFORD <pb@cs.wisc.edu> wrote:
Please find the examples for the case of Telia below.
FROM jfk-us (jfk-us.team-probing.c008820.20201002.warts.gz)
traceroute from 216.66.30.102 (Ark probe hosted in New York City, NY, US. No AS info found) to 223.114.235.32 (MAXMIXD: Turpan, CN)
1 216.66.30.101 0.365 ms
2 62.115.49.173 3.182 ms
3 *
4 62.115.137.59 17.453 ms [x] (chi-b23-link.ip.twelve99.net., CAIDA-GEOLOC -> Chicago, IL, US)
5 62.115.117.48 59.921 ms [x] (sea-b2-link.ip.twelve99.net., RIPE-IPMAP -> Seattle, WA, US)
6 62.115.171.221 69.993 ms
7 223.120.6.53 69.378 ms
8 223.120.12.34 226.225 ms
9 221.183.55.110 237.475 ms
10 221.183.25.201 238.697 ms
11 221.176.16.213 242.296 ms
12 221.183.36.62 352.695 ms
13 221.183.39.2 300.166 ms
14 117.191.8.118 316.270 ms
15 *
16 *
17 *
18 *
19 *
FROM ord-us (ord-us.team-probing.c008820.20201002.warts.gz)
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 109.25.215.237 (237.215.25.109.rev.sfr.net., MAXMIXD: La Crau, FR)
1 140.192.218.129 0.795 ms
2 140.192.9.124 0.603 ms
3 64.124.44.158 1.099 ms
4 64.125.31.172 3.047 ms
5 *
6 64.125.15.65 1.895 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.118.59 99.242 ms [x] (prs-b3-link.ip.twelve99.net., CAIDA-GEOLOC -> Paris, FR)
8 62.115.154.23 105.214 ms
9 77.136.10.6 119.021 ms
10 77.136.10.6 118.830 ms
11 80.118.89.202 118.690 ms
12 80.118.89.234 118.986 ms
13 109.24.108.66 119.159 ms
14 109.25.215.237 126.085 ms
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 84.249.89.93 (dsl-tkubng12-54f959-93.dhcp.inet.fi., MAXMIXD: Turku, FI)
1 140.192.218.129 0.243 ms
2 140.192.9.124 0.326 ms
3 64.124.44.158 0.600 ms
4 *
5 *
6 64.125.15.65 1.792 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.123.27 121.199 ms [x] (hls-b4-link.ip.twelve99.net., CAIDA-GEOLOC -> Helsinki, FI)
8 *
9 141.208.193.190 127.723 ms
10 84.249.89.93 139.051 ms
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US) to 193.28.231.50 (MAXMIXD: None, HU)
1 140.192.218.129 0.240 ms
2 140.192.9.124 0.333 ms
3 64.124.44.158 0.648 ms
4 *
5 64.125.25.75 0.752 ms
6 64.125.15.65 1.877 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.119.39 123.952 ms [x] (bpt-b2-link.ip.twelve99.net., **I suspect it is in Budapest, HU**)
8 62.115.39.122 117.171 ms
9 88.151.96.148 117.202 ms
10 88.151.96.213 124.787 ms
11 *
12 *
13 *
14 *
15 *
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 152.195.4.11 (MAXMIXD: Los Angeles, CA, US)
1 140.192.218.129 0.224 ms
2 140.192.9.124 0.545 ms
3 64.124.44.158 0.640 ms
4 *
5 *
6 64.125.15.65 1.786 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.118.247 54.597 ms [x] (las-b22-link.ip.twelve99.net., CAIDA-GEOLOC -> Los Angeles, CA, US)
8 62.115.11.129 55.979 ms
9 *
10 *
11 *
12 *
13 *
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 47.31.143.217 (MAXMIXD: Delhi, IN)
1 140.192.218.129 2.277 ms
2 140.192.9.124 0.449 ms
3 64.124.44.158 0.576 ms
4 *
5 *
6 64.125.15.65 1.814 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.114.41 210.056 ms [x] (snge-b5-link.ip.twelve99.net.,)
8 62.115.177.11 200.840 ms
9 103.198.140.16 233.636 ms
10 103.198.140.16 232.871 ms
11 103.198.140.171 232.648 ms
12 *
13 *
14 *
15 *
16 *
________________________________ From: Lukas Tribus <lukas@ltri.eu> Sent: Monday, January 17, 2022 1:52 PM To: PAUL R BARFORD <pb@cs.wisc.edu> Cc: Nick Hilliard <nick@foobar.org>; nanog@nanog.org <nanog@nanog.org>; Esteban Carisimo <esteban.carisimo@northwestern.edu>; Fabian E. Bustamante <fabianb@cs.northwestern.edu> Subject: Re: Long hops on international paths
On Mon, 17 Jan 2022 at 20:00, PAUL R BARFORD <pb@cs.wisc.edu> wrote:
What we're curious about is why we're seeing a concentration of hops at a small number of routers that appear on international paths.
I suggest you share a few actual examples (IP addresses, traceroutes).
I don't think discussing your conclusion based on data we don't have makes sense.
Lukas
-- ++ytti
Hey Paul,
Thank you for the summary. We're clear about the fact that what we're seeing are MLPS paths - that was not in question. What we are not clear about and the reason for the post is why the provider - zayo.telia in this case - would decide to configure MPLS paths between Chicago and distant international locations. We assumed we would see hops in traceroute between Chicago and coastal locations and then hops that transited submarine infrastructure followed by hops to large population centers.
MPLS is the default, not exception. And like any other form of tunnelling, MPLS decouples underlay and overlay. This means, the key value proposal of tunneling is that the devices between tunnel end points do not know the original sender or final receiver. This means, when TTL expires in transit, the P device may not know how to return packet to sender. There are three cases here 1) MPLS-TTL does not expire in transit => easy 2) MPLS-TTL expires in transit 2a) generate TTL exceeded and put it back to tunnel, sending it to egressPE, which is guaranteed to know how to return to sender 2b) randomly assume that you know how to reach the sender and try to send the TTL exceeded directly with 2a) all P hops display egressPE latency, but it works. With 2b) it might not work, as P might not know how to return. Some devices, like Cisco, allows you heuristically to decide if to tunnel ICMP or not, based on stack depth, but this does not work. As default table during repair is as deep as vrf without repair, so we cannot really discriminate. So the best solution is to not expire in transit (the norm in tunneling), i.e. set MPLS-TTL to 255. 2nd best is to tunnel, but the RTT will confuse uneducated, or as it may be, hjghly educated, users. We could implement something like https://ytti.github.io/icmp-eo-timestamp/draft-ytti-intarea-icmp-eo-timestam... to offer correct forward latencies to P/LSR in 2a scenario.
Regards, PB ________________________________ From: Saku Ytti <saku@ytti.fi> Sent: Tuesday, January 18, 2022 12:50 AM To: PAUL R BARFORD <pb@cs.wisc.edu> Cc: Lukas Tribus <lukas@ltri.eu>; Esteban Carisimo <esteban.carisimo@northwestern.edu>; nanog@nanog.org <nanog@nanog.org>; Fabian E. Bustamante <fabianb@cs.northwestern.edu> Subject: Re: Long hops on international paths
1) all (meaning all hitting the zayo.telia) your traceroutes originate from University in Chicago 2) the zayo.telia device is physically close to the university 3) we should expect physically close-by backbone device to be present in disproportionate amount of traceroutes 4) almost certainly zayo.telia is imposing the MPLS label of TTL 255, _NOT_ copying IP TTL, therefore until MPLS label is popped, TTL is not expiring. I.e. you are seeing ingressPE and egress PE ot Telia, you are not seeing any P routers.
This is not esoteric knowledge, but a fairly basic Internet concept. I am worried you are missing too much context to produce actionable output from your work. It might be interesting to see your curriculum, why this confusion arose, why it seems logical that the reason must be that almost all waves are terminated there, because it would not seem logical for people practising in the field who have even cursory understanding, this implies problems in the curriculum.
On Tue, 18 Jan 2022 at 07:21, PAUL R BARFORD <pb@cs.wisc.edu> wrote:
Please find the examples for the case of Telia below.
FROM jfk-us (jfk-us.team-probing.c008820.20201002.warts.gz)
traceroute from 216.66.30.102 (Ark probe hosted in New York City, NY, US. No AS info found) to 223.114.235.32 (MAXMIXD: Turpan, CN)
1 216.66.30.101 0.365 ms
2 62.115.49.173 3.182 ms
3 *
4 62.115.137.59 17.453 ms [x] (chi-b23-link.ip.twelve99.net., CAIDA-GEOLOC -> Chicago, IL, US)
5 62.115.117.48 59.921 ms [x] (sea-b2-link.ip.twelve99.net., RIPE-IPMAP -> Seattle, WA, US)
6 62.115.171.221 69.993 ms
7 223.120.6.53 69.378 ms
8 223.120.12.34 226.225 ms
9 221.183.55.110 237.475 ms
10 221.183.25.201 238.697 ms
11 221.176.16.213 242.296 ms
12 221.183.36.62 352.695 ms
13 221.183.39.2 300.166 ms
14 117.191.8.118 316.270 ms
15 *
16 *
17 *
18 *
19 *
FROM ord-us (ord-us.team-probing.c008820.20201002.warts.gz)
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 109.25.215.237 (237.215.25.109.rev.sfr.net., MAXMIXD: La Crau, FR)
1 140.192.218.129 0.795 ms
2 140.192.9.124 0.603 ms
3 64.124.44.158 1.099 ms
4 64.125.31.172 3.047 ms
5 *
6 64.125.15.65 1.895 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.118.59 99.242 ms [x] (prs-b3-link.ip.twelve99.net., CAIDA-GEOLOC -> Paris, FR)
8 62.115.154.23 105.214 ms
9 77.136.10.6 119.021 ms
10 77.136.10.6 118.830 ms
11 80.118.89.202 118.690 ms
12 80.118.89.234 118.986 ms
13 109.24.108.66 119.159 ms
14 109.25.215.237 126.085 ms
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 84.249.89.93 (dsl-tkubng12-54f959-93.dhcp.inet.fi., MAXMIXD: Turku, FI)
1 140.192.218.129 0.243 ms
2 140.192.9.124 0.326 ms
3 64.124.44.158 0.600 ms
4 *
5 *
6 64.125.15.65 1.792 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.123.27 121.199 ms [x] (hls-b4-link.ip.twelve99.net., CAIDA-GEOLOC -> Helsinki, FI)
8 *
9 141.208.193.190 127.723 ms
10 84.249.89.93 139.051 ms
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US) to 193.28.231.50 (MAXMIXD: None, HU)
1 140.192.218.129 0.240 ms
2 140.192.9.124 0.333 ms
3 64.124.44.158 0.648 ms
4 *
5 64.125.25.75 0.752 ms
6 64.125.15.65 1.877 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.119.39 123.952 ms [x] (bpt-b2-link.ip.twelve99.net., **I suspect it is in Budapest, HU**)
8 62.115.39.122 117.171 ms
9 88.151.96.148 117.202 ms
10 88.151.96.213 124.787 ms
11 *
12 *
13 *
14 *
15 *
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 152.195.4.11 (MAXMIXD: Los Angeles, CA, US)
1 140.192.218.129 0.224 ms
2 140.192.9.124 0.545 ms
3 64.124.44.158 0.640 ms
4 *
5 *
6 64.125.15.65 1.786 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.118.247 54.597 ms [x] (las-b22-link.ip.twelve99.net., CAIDA-GEOLOC -> Los Angeles, CA, US)
8 62.115.11.129 55.979 ms
9 *
10 *
11 *
12 *
13 *
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 47.31.143.217 (MAXMIXD: Delhi, IN)
1 140.192.218.129 2.277 ms
2 140.192.9.124 0.449 ms
3 64.124.44.158 0.576 ms
4 *
5 *
6 64.125.15.65 1.814 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.114.41 210.056 ms [x] (snge-b5-link.ip.twelve99.net.,)
8 62.115.177.11 200.840 ms
9 103.198.140.16 233.636 ms
10 103.198.140.16 232.871 ms
11 103.198.140.171 232.648 ms
12 *
13 *
14 *
15 *
16 *
________________________________ From: Lukas Tribus <lukas@ltri.eu> Sent: Monday, January 17, 2022 1:52 PM To: PAUL R BARFORD <pb@cs.wisc.edu> Cc: Nick Hilliard <nick@foobar.org>; nanog@nanog.org <nanog@nanog.org>; Esteban Carisimo <esteban.carisimo@northwestern.edu>; Fabian E. Bustamante <fabianb@cs.northwestern.edu> Subject: Re: Long hops on international paths
On Mon, 17 Jan 2022 at 20:00, PAUL R BARFORD <pb@cs.wisc.edu> wrote:
What we're curious about is why we're seeing a concentration of hops at a small number of routers that appear on international paths.
I suggest you share a few actual examples (IP addresses, traceroutes).
I don't think discussing your conclusion based on data we don't have makes sense.
Lukas
-- ++ytti
-- ++ytti
On 1/18/22 16:28, Saku Ytti wrote:
2) MPLS-TTL expires in transit 2a) generate TTL exceeded and put it back to tunnel, sending it to egressPE, which is guaranteed to know how to return to sender
This is what we run - our core is BGP-free (LDP), so the traceroute value customers see will be the egress PE router.
2nd best is to tunnel, but the RTT will confuse uneducated, or as it may be, hjghly educated, users.
It is very confusing, even for some educated users. It's become less of problem, as years of educating users seems to have bedded in.
We could implement something like https://ytti.github.io/icmp-eo-timestamp/draft-ytti-intarea-icmp-eo-timestam... to offer correct forward latencies to P/LSR in 2a scenario.
I recall you and I chatted about this some 3 or so years ago. Do you know if anyone of the (un)usual suspects have implemented? Mark.
On Tue, 18 Jan 2022 at 17:27, Mark Tinka <mark@tinka.africa> wrote:
https://ytti.github.io/icmp-eo-timestamp/draft-ytti-intarea-icmp-eo-timestam...
I recall you and I chatted about this some 3 or so years ago. Do you know if anyone of the (un)usual suspects have implemented?
I've not even submitted it, so that's a no. -- ++ytti
On 1/18/22 16:28, Saku Ytti wrote:
2) MPLS-TTL expires in transit 2a) generate TTL exceeded and put it back to tunnel, sending it to egressPE, which is guaranteed to know how to return to sender
This is what we run - our core is BGP-free (LDP), so the traceroute value customers see will be the egress PE router.
2nd best is to tunnel, but the RTT will confuse uneducated, or as it may be, hjghly educated, users.
It is very confusing, even for some educated users. It's become less of problem, as years of educating users seems to have bedded in.
We could implement something like https://ytti.github.io/icmp-eo-timestamp/draft-ytti-intarea-icmp-eo-timestam... to offer correct forward latencies to P/LSR in 2a scenario.
I recall you and I chatted about this some 3 or so years ago. Do you know if anyone of the (un)usual suspects have implemented? Mark.
Paul- You said: "... would decide to configure MPLS paths between Chicago and distant international locations ..." AS3128 runs MPLS and it's probable someone might correct me here, but for a IGP backbone area I think it's common for there to be a full mesh of LSPs via either LDP, RSVP, SR etc. AS3128 is a small regional and we operate in that way across 60+ nodes. I don't know if it's common for someone with a global footprint like 1299 to have a contiguous global MPLS backbone, but the point of my reply was to say it's not impossible to think 1299 has a global MPLS mesh between major POPs. -Michael
-----Original Message----- From: NANOG <nanog-bounces+michael.hare=wisc.edu@nanog.org> On Behalf Of PAUL R BARFORD Sent: Tuesday, January 18, 2022 8:16 AM To: Saku Ytti <saku@ytti.fi> Cc: Esteban Carisimo <esteban.carisimo@northwestern.edu>; nanog@nanog.org; Fabian E. Bustamante <fabianb@cs.northwestern.edu> Subject: Re: Long hops on international paths
Hello Saku,
Thank you for the summary. We're clear about the fact that what we're seeing are MLPS paths - that was not in question. What we are not clear about and the reason for the post is why the provider - zayo.telia in this case - would decide to configure MPLS paths between Chicago and distant international locations. We assumed we would see hops in traceroute between Chicago and coastal locations and then hops that transited submarine infrastructure followed by hops to large population centers.
Regards, PB ________________________________
From: Saku Ytti <saku@ytti.fi> Sent: Tuesday, January 18, 2022 12:50 AM To: PAUL R BARFORD <pb@cs.wisc.edu> Cc: Lukas Tribus <lukas@ltri.eu>; Esteban Carisimo <esteban.carisimo@northwestern.edu>; nanog@nanog.org <nanog@nanog.org>; Fabian E. Bustamante <fabianb@cs.northwestern.edu> Subject: Re: Long hops on international paths
1) all (meaning all hitting the zayo.telia) your traceroutes originate from University in Chicago 2) the zayo.telia device is physically close to the university 3) we should expect physically close-by backbone device to be present in disproportionate amount of traceroutes 4) almost certainly zayo.telia is imposing the MPLS label of TTL 255, _NOT_ copying IP TTL, therefore until MPLS label is popped, TTL is not expiring. I.e. you are seeing ingressPE and egress PE ot Telia, you are not seeing any P routers.
This is not esoteric knowledge, but a fairly basic Internet concept. I am worried you are missing too much context to produce actionable output from your work. It might be interesting to see your curriculum, why this confusion arose, why it seems logical that the reason must be that almost all waves are terminated there, because it would not seem logical for people practising in the field who have even cursory understanding, this implies problems in the curriculum.
On Tue, 18 Jan 2022 at 07:21, PAUL R BARFORD <pb@cs.wisc.edu> wrote:
Please find the examples for the case of Telia below.
FROM jfk-us (jfk-us.team-probing.c008820.20201002.warts.gz)
traceroute from 216.66.30.102 (Ark probe hosted in New York City, NY, US.
No AS info found) to 223.114.235.32 (MAXMIXD: Turpan, CN)
1 216.66.30.101 0.365 ms
2 62.115.49.173 3.182 ms
3 *
4 62.115.137.59 17.453 ms [x] (chi-b23-link.ip.twelve99.net., CAIDA-
GEOLOC -> Chicago, IL, US)
5 62.115.117.48 59.921 ms [x] (sea-b2-link.ip.twelve99.net., RIPE-IPMAP ->
Seattle, WA, US)
6 62.115.171.221 69.993 ms
7 223.120.6.53 69.378 ms
8 223.120.12.34 226.225 ms
9 221.183.55.110 237.475 ms
10 221.183.25.201 238.697 ms
11 221.176.16.213 242.296 ms
12 221.183.36.62 352.695 ms
13 221.183.39.2 300.166 ms
14 117.191.8.118 316.270 ms
15 *
16 *
17 *
18 *
19 *
FROM ord-us (ord-us.team-probing.c008820.20201002.warts.gz)
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at
Depaul University-AS20120) to 109.25.215.237 (237.215.25.109.rev.sfr.net., MAXMIXD: La Crau, FR)
1 140.192.218.129 0.795 ms
2 140.192.9.124 0.603 ms
3 64.124.44.158 1.099 ms
4 64.125.31.172 3.047 ms
5 *
6 64.125.15.65 1.895 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com.,
CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.118.59 99.242 ms [x] (prs-b3-link.ip.twelve99.net., CAIDA-
GEOLOC -> Paris, FR)
8 62.115.154.23 105.214 ms
9 77.136.10.6 119.021 ms
10 77.136.10.6 118.830 ms
11 80.118.89.202 118.690 ms
12 80.118.89.234 118.986 ms
13 109.24.108.66 119.159 ms
14 109.25.215.237 126.085 ms
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at
Depaul University-AS20120) to 84.249.89.93 (dsl-tkubng12-54f959- 93.dhcp.inet.fi., MAXMIXD: Turku, FI)
1 140.192.218.129 0.243 ms
2 140.192.9.124 0.326 ms
3 64.124.44.158 0.600 ms
4 *
5 *
6 64.125.15.65 1.792 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com.,
CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.123.27 121.199 ms [x] (hls-b4-link.ip.twelve99.net., CAIDA-
GEOLOC -> Helsinki, FI)
8 *
9 141.208.193.190 127.723 ms
10 84.249.89.93 139.051 ms
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US) to
193.28.231.50 (MAXMIXD: None, HU)
1 140.192.218.129 0.240 ms
2 140.192.9.124 0.333 ms
3 64.124.44.158 0.648 ms
4 *
5 64.125.25.75 0.752 ms
6 64.125.15.65 1.877 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com.,
CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.119.39 123.952 ms [x] (bpt-b2-link.ip.twelve99.net., **I suspect
it is in Budapest, HU**)
8 62.115.39.122 117.171 ms
9 88.151.96.148 117.202 ms
10 88.151.96.213 124.787 ms
11 *
12 *
13 *
14 *
15 *
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at
Depaul University-AS20120) to 152.195.4.11 (MAXMIXD: Los Angeles, CA, US)
1 140.192.218.129 0.224 ms
2 140.192.9.124 0.545 ms
3 64.124.44.158 0.640 ms
4 *
5 *
6 64.125.15.65 1.786 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com.,
CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.118.247 54.597 ms [x] (las-b22-link.ip.twelve99.net., CAIDA-
GEOLOC -> Los Angeles, CA, US)
8 62.115.11.129 55.979 ms
9 *
10 *
11 *
12 *
13 *
traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at
Depaul University-AS20120) to 47.31.143.217 (MAXMIXD: Delhi, IN)
1 140.192.218.129 2.277 ms
2 140.192.9.124 0.449 ms
3 64.124.44.158 0.576 ms
4 *
5 *
6 64.125.15.65 1.814 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com.,
CAIDA-GEOLOC -> Chicago, IL, US)
7 62.115.114.41 210.056 ms [x] (snge-b5-link.ip.twelve99.net.,)
8 62.115.177.11 200.840 ms
9 103.198.140.16 233.636 ms
10 103.198.140.16 232.871 ms
11 103.198.140.171 232.648 ms
12 *
13 *
14 *
15 *
16 *
________________________________ From: Lukas Tribus <lukas@ltri.eu> Sent: Monday, January 17, 2022 1:52 PM To: PAUL R BARFORD <pb@cs.wisc.edu> Cc: Nick Hilliard <nick@foobar.org>; nanog@nanog.org
<nanog@nanog.org>; Esteban Carisimo <esteban.carisimo@northwestern.edu>; Fabian E. Bustamante <fabianb@cs.northwestern.edu>
Subject: Re: Long hops on international paths
On Mon, 17 Jan 2022 at 20:00, PAUL R BARFORD <pb@cs.wisc.edu> wrote:
What we're curious about is why we're seeing a concentration of hops at a small number of routers that appear on international paths.
I suggest you share a few actual examples (IP addresses, traceroutes).
I don't think discussing your conclusion based on data we don't have makes sense.
Lukas
-- ++ytti
On Tue, 18 Jan 2022 at 17:14, Michael Hare <michael.hare@wisc.edu> wrote:
AS3128 runs MPLS and it's probable someone might correct me here, but for a IGP backbone area I think it's common for there to be a full mesh of LSPs via either LDP, RSVP, SR etc. AS3128 is a small regional and we operate in that way across 60+ nodes. I don't know if it's common for someone with a global footprint like 1299 to have a contiguous global MPLS backbone, but the point of my reply was to say it's not impossible to think 1299 has a global MPLS mesh between major POPs.
Yes, this is the common case, not an exception. -- ++ytti
Thanks Saku and Michael. ________________________________ From: Saku Ytti <saku@ytti.fi> Sent: Tuesday, January 18, 2022 9:17 AM To: Michael Hare <michael.hare@wisc.edu> Cc: PAUL R BARFORD <pb@cs.wisc.edu>; Esteban Carisimo <esteban.carisimo@northwestern.edu>; nanog@nanog.org <nanog@nanog.org>; Fabian E. Bustamante <fabianb@cs.northwestern.edu> Subject: Re: Long hops on international paths On Tue, 18 Jan 2022 at 17:14, Michael Hare <michael.hare@wisc.edu> wrote:
AS3128 runs MPLS and it's probable someone might correct me here, but for a IGP backbone area I think it's common for there to be a full mesh of LSPs via either LDP, RSVP, SR etc. AS3128 is a small regional and we operate in that way across 60+ nodes. I don't know if it's common for someone with a global footprint like 1299 to have a contiguous global MPLS backbone, but the point of my reply was to say it's not impossible to think 1299 has a global MPLS mesh between major POPs.
Yes, this is the common case, not an exception. -- ++ytti
On 1/18/22 17:14, Michael Hare via NANOG wrote:
Paul-
You said: "... would decide to configure MPLS paths between Chicago and distant international locations ..."
AS3128 runs MPLS and it's probable someone might correct me here, but for a IGP backbone area I think it's common for there to be a full mesh of LSPs via either LDP, RSVP, SR etc. AS3128 is a small regional and we operate in that way across 60+ nodes. I don't know if it's common for someone with a global footprint like 1299 to have a contiguous global MPLS backbone, but the point of my reply was to say it's not impossible to think 1299 has a global MPLS mesh between major POPs.
100%. Mark.
Chicago is a fairly major POP that *MAY* very well have waves right to other major POPs. Can you retest from a *not* major POP? They're not likely to have a wave from Indy, St. Louis, Des Moines, etc. going to Paris, Singapore, Helsinki, Budapest, etc. Then you could *maybe* determine if it's a wave or MPLS. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "PAUL R BARFORD" <pb@cs.wisc.edu> To: "Lukas Tribus" <lukas@ltri.eu> Cc: "Esteban Carisimo" <esteban.carisimo@northwestern.edu>, nanog@nanog.org, "Fabian E. Bustamante" <fabianb@cs.northwestern.edu> Sent: Monday, January 17, 2022 11:17:18 PM Subject: Re: Long hops on international paths Please find the examples for the case of Telia below. FROM jfk-us (jfk-us.team-probing.c008820.20201002.warts.gz) traceroute from 216.66.30.102 (Ark probe hosted in New York City, NY, US. No AS info found) to 223.114.235.32 (MAXMIXD: Turpan, CN) 1 216.66.30.101 0.365 ms 2 62.115.49.173 3.182 ms 3 * 4 62.115.137.59 17.453 ms [x] (chi-b23-link.ip.twelve99.net., CAIDA-GEOLOC -> Chicago, IL, US) 5 62.115.117.48 59.921 ms [x] (sea-b2-link.ip.twelve99.net., RIPE-IPMAP -> Seattle, WA, US) 6 62.115.171.221 69.993 ms 7 223.120.6.53 69.378 ms 8 223.120.12.34 226.225 ms 9 221.183.55.110 237.475 ms 10 221.183.25.201 238.697 ms 11 221.176.16.213 242.296 ms 12 221.183.36.62 352.695 ms 13 221.183.39.2 300.166 ms 14 117.191.8.118 316.270 ms 15 * 16 * 17 * 18 * 19 * FROM ord-us (ord-us.team-probing.c008820.20201002.warts.gz) traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 109.25.215.237 (237.215.25.109.rev.sfr.net., MAXMIXD: La Crau, FR) 1 140.192.218.129 0.795 ms 2 140.192.9.124 0.603 ms 3 64.124.44.158 1.099 ms 4 64.125.31.172 3.047 ms 5 * 6 64.125.15.65 1.895 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.118.59 99.242 ms [x] (prs-b3-link.ip.twelve99.net., CAIDA-GEOLOC -> Paris, FR) 8 62.115.154.23 105.214 ms 9 77.136.10.6 119.021 ms 10 77.136.10.6 118.830 ms 11 80.118.89.202 118.690 ms 12 80.118.89.234 118.986 ms 13 109.24.108.66 119.159 ms 14 109.25.215.237 126.085 ms traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 84.249.89.93 (dsl-tkubng12-54f959-93.dhcp.inet.fi., MAXMIXD: Turku, FI) 1 140.192.218.129 0.243 ms 2 140.192.9.124 0.326 ms 3 64.124.44.158 0.600 ms 4 * 5 * 6 64.125.15.65 1.792 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.123.27 121.199 ms [x] (hls-b4-link.ip.twelve99.net., CAIDA-GEOLOC -> Helsinki, FI) 8 * 9 141.208.193.190 127.723 ms 10 84.249.89.93 139.051 ms traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US) to 193.28.231.50 (MAXMIXD: None, HU) 1 140.192.218.129 0.240 ms 2 140.192.9.124 0.333 ms 3 64.124.44.158 0.648 ms 4 * 5 64.125.25.75 0.752 ms 6 64.125.15.65 1.877 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.119.39 123.952 ms [x] (bpt-b2-link.ip.twelve99.net., **I suspect it is in Budapest, HU**) 8 62.115.39.122 117.171 ms 9 88.151.96.148 117.202 ms 10 88.151.96.213 124.787 ms 11 * 12 * 13 * 14 * 15 * traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 152.195.4.11 (MAXMIXD: Los Angeles, CA, US) 1 140.192.218.129 0.224 ms 2 140.192.9.124 0.545 ms 3 64.124.44.158 0.640 ms 4 * 5 * 6 64.125.15.65 1.786 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.118.247 54.597 ms [x] (las-b22-link.ip.twelve99.net., CAIDA-GEOLOC -> Los Angeles, CA, US) 8 62.115.11.129 55.979 ms 9 * 10 * 11 * 12 * 13 * traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 47.31.143.217 (MAXMIXD: Delhi, IN) 1 140.192.218.129 2.277 ms 2 140.192.9.124 0.449 ms 3 64.124.44.158 0.576 ms 4 * 5 * 6 64.125.15.65 1.814 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.114.41 210.056 ms [x] (snge-b5-link.ip.twelve99.net.,) 8 62.115.177.11 200.840 ms 9 103.198.140.16 233.636 ms 10 103.198.140.16 232.871 ms 11 103.198.140.171 232.648 ms 12 * 13 * 14 * 15 * 16 * From: Lukas Tribus <lukas@ltri.eu> Sent: Monday, January 17, 2022 1:52 PM To: PAUL R BARFORD <pb@cs.wisc.edu> Cc: Nick Hilliard <nick@foobar.org>; nanog@nanog.org <nanog@nanog.org>; Esteban Carisimo <esteban.carisimo@northwestern.edu>; Fabian E. Bustamante <fabianb@cs.northwestern.edu> Subject: Re: Long hops on international paths On Mon, 17 Jan 2022 at 20:00, PAUL R BARFORD <pb@cs.wisc.edu> wrote:
What we're curious about is why we're seeing a concentration of hops at a small number of routers that appear on international paths.
I suggest you share a few actual examples (IP addresses, traceroutes). I don't think discussing your conclusion based on data we don't have makes sense. Lukas
Hi Paul, Just curious. How do you determine they are the same routers? Is it based on IP address or MAC addresses? Or using CAIDA’s router alias database? Also how do you draw the conclusion that the AS1299 router is indeed in Chicago? IP-geolocation based on rDNS is not always accurate though. Pengxiong On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD <pb@cs.wisc.edu> wrote:
Hello,
I am a researcher at the University of Wisconsin. My colleagues at Northwestern University and I are studying international Internet connectivity and would appreciate your perspective on a recent finding.
We're using traceroute data from CAIDA's Ark project for our work. We've observed that many international links (i.e., a single hop on an end-to-end path that connects two countries where end points on the hop are identified via rDNS) tend to originate/terminate at the same routers. Said another way, we are observing a relatively small set of routers in different countries tend to have a majority of the international connections - this is especially the case for hops that terminate in the US. For example, there is a router operated by Telia (AS1299) in Chicago that has a high concentration of such links. We were a bit surprised by this finding since even though it makes sense that the set of providers is relatively small (i.e., those that offer global connectivity), we assumed that the set of routers that used for international connectivity within any one country would tend to be more widely distributed (at least with respect to how they appear in traceroute data - MPLS notwithstanding).
We're interested in whether or not this is indeed standard practice and if so, the cost/benefit for configuring international connectivity in this way?
Any thoughts or insights you might have would be greatly appreciated - off-list responses are welcome.
Thank you.
Regards, PB
Paul Barford University of Wisconsin - Madison
--
Regards, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside
Carrier class core routers still cost half a million dollars each or (way) more, so it’s not uncommon for there to be 2-4 in a metro. And there are only a few metros that have undersea cable landing stations. We deploy a minimum of a pair of core routers everywhere, but with our BGP/OSPF/iBGP core your path through us generally won’t change even though there’s an alternative path with slightly lower route pref. (Absent loss of both physical path and physical alternate) Ms. Lady Benjamin PD Cannon of Glencoe, ASCE 6x7 Networks & 6x7 Telecom, LLC CEO lb@6by7.net "The only fully end-to-end encrypted global telecommunications company in the world.” FCC License KJ6FJJ Sent from my iPhone via RFC1149.
On Jan 17, 2022, at 1:23 PM, Pengxiong Zhu <pzhu011@ucr.edu> wrote:
Hi Paul,
Just curious. How do you determine they are the same routers? Is it based on IP address or MAC addresses? Or using CAIDA’s router alias database?
Also how do you draw the conclusion that the AS1299 router is indeed in Chicago? IP-geolocation based on rDNS is not always accurate though.
Pengxiong
On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD <pb@cs.wisc.edu> wrote: Hello,
I am a researcher at the University of Wisconsin. My colleagues at Northwestern University and I are studying international Internet connectivity and would appreciate your perspective on a recent finding.
We're using traceroute data from CAIDA's Ark project for our work. We've observed that many international links (i.e., a single hop on an end-to-end path that connects two countries where end points on the hop are identified via rDNS) tend to originate/terminate at the same routers. Said another way, we are observing a relatively small set of routers in different countries tend to have a majority of the international connections - this is especially the case for hops that terminate in the US. For example, there is a router operated by Telia (AS1299) in Chicago that has a high concentration of such links. We were a bit surprised by this finding since even though it makes sense that the set of providers is relatively small (i.e., those that offer global connectivity), we assumed that the set of routers that used for international connectivity within any one country would tend to be more widely distributed (at least with respect to how they appear in traceroute data - MPLS notwithstanding).
We're interested in whether or not this is indeed standard practice and if so, the cost/benefit for configuring international connectivity in this way?
Any thoughts or insights you might have would be greatly appreciated - off-list responses are welcome.
Thank you.
Regards, PB
Paul Barford University of Wisconsin - Madison
--
Regards, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside
Dear Pengxiong, Thanks for your questions: 1. We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the MIDAR alias resolution method to infer IP addresses assigned to the same router. 2. We understand the concerns about IP geolocation. Interfaces of the router in question are assigned similar domain names e.g., “chi-b2-link.ip.twelve99.net” (62.115.50.61). We also used CAIDA’s ITDK, which provides geolocation information, and indicates that this router is located in Chicago. We cross-reference with Maxmind where possible. In this particular case, there is the telltale in the use of "chi" in the domain name. 3. Hope that helps. Regards, PB ________________________________ From: Pengxiong Zhu <pzhu011@ucr.edu> Sent: Monday, January 17, 2022 3:23 PM To: PAUL R BARFORD <pb@cs.wisc.edu> Cc: nanog@nanog.org <nanog@nanog.org> Subject: Re: Long hops on international paths Hi Paul, Just curious. How do you determine they are the same routers? Is it based on IP address or MAC addresses? Or using CAIDA’s router alias database? Also how do you draw the conclusion that the AS1299 router is indeed in Chicago? IP-geolocation based on rDNS is not always accurate though. Pengxiong On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> wrote: Hello, I am a researcher at the University of Wisconsin. My colleagues at Northwestern University and I are studying international Internet connectivity and would appreciate your perspective on a recent finding. We're using traceroute data from CAIDA's Ark project for our work. We've observed that many international links (i.e., a single hop on an end-to-end path that connects two countries where end points on the hop are identified via rDNS) tend to originate/terminate at the same routers. Said another way, we are observing a relatively small set of routers in different countries tend to have a majority of the international connections - this is especially the case for hops that terminate in the US. For example, there is a router operated by Telia (AS1299) in Chicago that has a high concentration of such links. We were a bit surprised by this finding since even though it makes sense that the set of providers is relatively small (i.e., those that offer global connectivity), we assumed that the set of routers that used for international connectivity within any one country would tend to be more widely distributed (at least with respect to how they appear in traceroute data - MPLS notwithstanding). We're interested in whether or not this is indeed standard practice and if so, the cost/benefit for configuring international connectivity in this way? Any thoughts or insights you might have would be greatly appreciated - off-list responses are welcome. Thank you. Regards, PB Paul Barford University of Wisconsin - Madison -- Regards, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside
On Mon, Jan 17, 2022 at 5:31 PM PAUL R BARFORD <pb@cs.wisc.edu> wrote:
Dear Pengxiong,
Thanks for your questions:
1. We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the MIDAR alias resolution method to infer IP addresses assigned to the same router. 2. We understand the concerns about IP geolocation. Interfaces of the router in question are assigned similar domain names e.g., “ chi-b2-link.ip.twelve99.net” (62.115.50.61). We also used CAIDA’s ITDK, which provides geolocation information, and indicates that this router is located in Chicago. We cross-reference with Maxmind where possible. In this particular case, there is the telltale in the use of "chi" in the domain name. 3.
I think nick's point about ttl expiry and missing some context on topology still stands. I'd be that the paths between 2 continents do not actually land in chicago... that you're seeing (or not seeing) missing hops between the coast(s) and chicago inside 1299's network in the US.
1.
Hope that helps.
Regards, PB ------------------------------ *From:* Pengxiong Zhu <pzhu011@ucr.edu> *Sent:* Monday, January 17, 2022 3:23 PM *To:* PAUL R BARFORD <pb@cs.wisc.edu> *Cc:* nanog@nanog.org <nanog@nanog.org> *Subject:* Re: Long hops on international paths
Hi Paul,
Just curious. How do you determine they are the same routers? Is it based on IP address or MAC addresses? Or using CAIDA’s router alias database?
Also how do you draw the conclusion that the AS1299 router is indeed in Chicago? IP-geolocation based on rDNS is not always accurate though.
Pengxiong
On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD <pb@cs.wisc.edu> wrote:
Hello,
I am a researcher at the University of Wisconsin. My colleagues at Northwestern University and I are studying international Internet connectivity and would appreciate your perspective on a recent finding.
We're using traceroute data from CAIDA's Ark project for our work. We've observed that many international links (i.e., a single hop on an end-to-end path that connects two countries where end points on the hop are identified via rDNS) tend to originate/terminate at the same routers. Said another way, we are observing a relatively small set of routers in different countries tend to have a majority of the international connections - this is especially the case for hops that terminate in the US. For example, there is a router operated by Telia (AS1299) in Chicago that has a high concentration of such links. We were a bit surprised by this finding since even though it makes sense that the set of providers is relatively small (i.e., those that offer global connectivity), we assumed that the set of routers that used for international connectivity within any one country would tend to be more widely distributed (at least with respect to how they appear in traceroute data - MPLS notwithstanding).
We're interested in whether or not this is indeed standard practice and if so, the cost/benefit for configuring international connectivity in this way?
Any thoughts or insights you might have would be greatly appreciated - off-list responses are welcome.
Thank you.
Regards, PB
Paul Barford University of Wisconsin - Madison
--
Regards, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside
What we're considering specifically are consecutive (layer 3) hops as identified by traceroute. Thus, TTL is decremented by 1 and no more than 1 (i.e., we have to get full information (not *****) from consecutive hops to consider the link). I have asked my colleague to put together a set of examples. We assume that there are multiple layer 1 and 2 links, and possibly layer 3 hops masked from traceroute by MPLS. But what we're seeing in terms of hops exposed by traceroute make it look like a single (TTL decremented by 1) hop. I'll post the examples when I get them. PB ________________________________ From: morrowc.lists@gmail.com <morrowc.lists@gmail.com> Sent: Monday, January 17, 2022 5:13 PM To: PAUL R BARFORD <pb@cs.wisc.edu> Cc: Pengxiong Zhu <pzhu011@ucr.edu>; nanog@nanog.org <nanog@nanog.org> Subject: Re: Long hops on international paths On Mon, Jan 17, 2022 at 5:31 PM PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> wrote: Dear Pengxiong, Thanks for your questions: 1. We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the MIDAR alias resolution method to infer IP addresses assigned to the same router. 2. We understand the concerns about IP geolocation. Interfaces of the router in question are assigned similar domain names e.g., “chi-b2-link.ip.twelve99.net<http://chi-b2-link.ip.twelve99.net>” (62.115.50.61). We also used CAIDA’s ITDK, which provides geolocation information, and indicates that this router is located in Chicago. We cross-reference with Maxmind where possible. In this particular case, there is the telltale in the use of "chi" in the domain name. 3. I think nick's point about ttl expiry and missing some context on topology still stands. I'd be that the paths between 2 continents do not actually land in chicago... that you're seeing (or not seeing) missing hops between the coast(s) and chicago inside 1299's network in the US. 1. Hope that helps. Regards, PB ________________________________ From: Pengxiong Zhu <pzhu011@ucr.edu<mailto:pzhu011@ucr.edu>> Sent: Monday, January 17, 2022 3:23 PM To: PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Long hops on international paths Hi Paul, Just curious. How do you determine they are the same routers? Is it based on IP address or MAC addresses? Or using CAIDA’s router alias database? Also how do you draw the conclusion that the AS1299 router is indeed in Chicago? IP-geolocation based on rDNS is not always accurate though. Pengxiong On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> wrote: Hello, I am a researcher at the University of Wisconsin. My colleagues at Northwestern University and I are studying international Internet connectivity and would appreciate your perspective on a recent finding. We're using traceroute data from CAIDA's Ark project for our work. We've observed that many international links (i.e., a single hop on an end-to-end path that connects two countries where end points on the hop are identified via rDNS) tend to originate/terminate at the same routers. Said another way, we are observing a relatively small set of routers in different countries tend to have a majority of the international connections - this is especially the case for hops that terminate in the US. For example, there is a router operated by Telia (AS1299) in Chicago that has a high concentration of such links. We were a bit surprised by this finding since even though it makes sense that the set of providers is relatively small (i.e., those that offer global connectivity), we assumed that the set of routers that used for international connectivity within any one country would tend to be more widely distributed (at least with respect to how they appear in traceroute data - MPLS notwithstanding). We're interested in whether or not this is indeed standard practice and if so, the cost/benefit for configuring international connectivity in this way? Any thoughts or insights you might have would be greatly appreciated - off-list responses are welcome. Thank you. Regards, PB Paul Barford University of Wisconsin - Madison -- Regards, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside
I think a large part of your problem is that you’re using trace route to try and determine the full topology of a large complex network. It won’t show the full topology. On Mon, Jan 17, 2022 at 7:43 PM PAUL R BARFORD <pb@cs.wisc.edu> wrote:
What we're considering specifically are consecutive (layer 3) hops as identified by traceroute. Thus, TTL is decremented by 1 and no more than 1 (i.e., we have to get full information (not *****) from consecutive hops to consider the link). I have asked my colleague to put together a set of examples. We assume that there are multiple layer 1 and 2 links, and possibly layer 3 hops masked from traceroute by MPLS. But what we're seeing in terms of hops exposed by traceroute make it look like a single (TTL decremented by 1) hop.
I'll post the examples when I get them.
PB ------------------------------ *From:* morrowc.lists@gmail.com <morrowc.lists@gmail.com> *Sent:* Monday, January 17, 2022 5:13 PM *To:* PAUL R BARFORD <pb@cs.wisc.edu> *Cc:* Pengxiong Zhu <pzhu011@ucr.edu>; nanog@nanog.org <nanog@nanog.org>
*Subject:* Re: Long hops on international paths
On Mon, Jan 17, 2022 at 5:31 PM PAUL R BARFORD <pb@cs.wisc.edu> wrote:
Dear Pengxiong,
Thanks for your questions:
1. We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the MIDAR alias resolution method to infer IP addresses assigned to the same router. 2. We understand the concerns about IP geolocation. Interfaces of the router in question are assigned similar domain names e.g., “ chi-b2-link.ip.twelve99.net” (62.115.50.61). We also used CAIDA’s ITDK, which provides geolocation information, and indicates that this router is located in Chicago. We cross-reference with Maxmind where possible. In this particular case, there is the telltale in the use of "chi" in the domain name. 3.
I think nick's point about ttl expiry and missing some context on topology still stands. I'd be that the paths between 2 continents do not actually land in chicago... that you're seeing (or not seeing) missing hops between the coast(s) and chicago inside 1299's network in the US.
1.
Hope that helps.
Regards, PB ------------------------------ *From:* Pengxiong Zhu <pzhu011@ucr.edu> *Sent:* Monday, January 17, 2022 3:23 PM *To:* PAUL R BARFORD <pb@cs.wisc.edu> *Cc:* nanog@nanog.org <nanog@nanog.org> *Subject:* Re: Long hops on international paths
Hi Paul,
Just curious. How do you determine they are the same routers? Is it based on IP address or MAC addresses? Or using CAIDA’s router alias database?
Also how do you draw the conclusion that the AS1299 router is indeed in Chicago? IP-geolocation based on rDNS is not always accurate though.
Pengxiong
On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD <pb@cs.wisc.edu> wrote:
Hello,
I am a researcher at the University of Wisconsin. My colleagues at Northwestern University and I are studying international Internet connectivity and would appreciate your perspective on a recent finding.
We're using traceroute data from CAIDA's Ark project for our work. We've observed that many international links (i.e., a single hop on an end-to-end path that connects two countries where end points on the hop are identified via rDNS) tend to originate/terminate at the same routers. Said another way, we are observing a relatively small set of routers in different countries tend to have a majority of the international connections - this is especially the case for hops that terminate in the US. For example, there is a router operated by Telia (AS1299) in Chicago that has a high concentration of such links. We were a bit surprised by this finding since even though it makes sense that the set of providers is relatively small (i.e., those that offer global connectivity), we assumed that the set of routers that used for international connectivity within any one country would tend to be more widely distributed (at least with respect to how they appear in traceroute data - MPLS notwithstanding).
We're interested in whether or not this is indeed standard practice and if so, the cost/benefit for configuring international connectivity in this way?
Any thoughts or insights you might have would be greatly appreciated - off-list responses are welcome.
Thank you.
Regards, PB
Paul Barford University of Wisconsin - Madison
--
Regards, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside
Hello David, Understanding the physical topology of the network is not our objective. What we're trying to understand is the logical topology revealed by traceroute (we are well-aware of traceroute limitations) and why a relatively small set of routers in different countries tend to have the majority of the international connections. Our expectation was that layer 3 connectivity revealed in traceroute to be relatively evenly spread out along coasts and near submarine landing points. We're not seeing that. So, the question is what is the cost/benefit to providers to configure/maintain routes (that include long MPLS tunnels) that tend to concentrate international connectivity at a relatively small number of routers? Regards, PB ________________________________ From: davidbass570@gmail.com <davidbass570@gmail.com> Sent: Tuesday, January 18, 2022 8:22 AM To: PAUL R BARFORD <pb@cs.wisc.edu> Cc: morrowc.lists@gmail.com <morrowc.lists@gmail.com>; nanog@nanog.org <nanog@nanog.org> Subject: Re: Long hops on international paths I think a large part of your problem is that you’re using trace route to try and determine the full topology of a large complex network. It won’t show the full topology. On Mon, Jan 17, 2022 at 7:43 PM PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> wrote: What we're considering specifically are consecutive (layer 3) hops as identified by traceroute. Thus, TTL is decremented by 1 and no more than 1 (i.e., we have to get full information (not *****) from consecutive hops to consider the link). I have asked my colleague to put together a set of examples. We assume that there are multiple layer 1 and 2 links, and possibly layer 3 hops masked from traceroute by MPLS. But what we're seeing in terms of hops exposed by traceroute make it look like a single (TTL decremented by 1) hop. I'll post the examples when I get them. PB ________________________________ From: morrowc.lists@gmail.com<mailto:morrowc.lists@gmail.com> <morrowc.lists@gmail.com<mailto:morrowc.lists@gmail.com>> Sent: Monday, January 17, 2022 5:13 PM To: PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> Cc: Pengxiong Zhu <pzhu011@ucr.edu<mailto:pzhu011@ucr.edu>>; nanog@nanog.org<mailto:nanog@nanog.org> <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Long hops on international paths On Mon, Jan 17, 2022 at 5:31 PM PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> wrote: Dear Pengxiong, Thanks for your questions: 1. We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the MIDAR alias resolution method to infer IP addresses assigned to the same router. 2. We understand the concerns about IP geolocation. Interfaces of the router in question are assigned similar domain names e.g., “chi-b2-link.ip.twelve99.net<http://chi-b2-link.ip.twelve99.net>” (62.115.50.61). We also used CAIDA’s ITDK, which provides geolocation information, and indicates that this router is located in Chicago. We cross-reference with Maxmind where possible. In this particular case, there is the telltale in the use of "chi" in the domain name. 3. I think nick's point about ttl expiry and missing some context on topology still stands. I'd be that the paths between 2 continents do not actually land in chicago... that you're seeing (or not seeing) missing hops between the coast(s) and chicago inside 1299's network in the US. 1. Hope that helps. Regards, PB ________________________________ From: Pengxiong Zhu <pzhu011@ucr.edu<mailto:pzhu011@ucr.edu>> Sent: Monday, January 17, 2022 3:23 PM To: PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Long hops on international paths Hi Paul, Just curious. How do you determine they are the same routers? Is it based on IP address or MAC addresses? Or using CAIDA’s router alias database? Also how do you draw the conclusion that the AS1299 router is indeed in Chicago? IP-geolocation based on rDNS is not always accurate though. Pengxiong On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> wrote: Hello, I am a researcher at the University of Wisconsin. My colleagues at Northwestern University and I are studying international Internet connectivity and would appreciate your perspective on a recent finding. We're using traceroute data from CAIDA's Ark project for our work. We've observed that many international links (i.e., a single hop on an end-to-end path that connects two countries where end points on the hop are identified via rDNS) tend to originate/terminate at the same routers. Said another way, we are observing a relatively small set of routers in different countries tend to have a majority of the international connections - this is especially the case for hops that terminate in the US. For example, there is a router operated by Telia (AS1299) in Chicago that has a high concentration of such links. We were a bit surprised by this finding since even though it makes sense that the set of providers is relatively small (i.e., those that offer global connectivity), we assumed that the set of routers that used for international connectivity within any one country would tend to be more widely distributed (at least with respect to how they appear in traceroute data - MPLS notwithstanding). We're interested in whether or not this is indeed standard practice and if so, the cost/benefit for configuring international connectivity in this way? Any thoughts or insights you might have would be greatly appreciated - off-list responses are welcome. Thank you. Regards, PB Paul Barford University of Wisconsin - Madison -- Regards, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside
PAUL R BARFORD wrote on 18/01/2022 14:48:
So, the question is what is the cost/benefit to providers to configure/maintain routes (that include long MPLS tunnels) that tend to concentrate international connectivity at a relatively small number of routers?
the cost of mpls TE is pretty low: a couple of extra config lines per LSP. The benefit can be substantial in terms of having fine-grained control of how packets traverse a network, and allow optimisation of specific policy outcomes, e.g. cost / latency / throughput / pktloss / qos / etc. Nick
On Tue, Jan 18, 2022 at 6:49 AM PAUL R BARFORD <pb@cs.wisc.edu> wrote:
So, the question is what is the cost/benefit to providers to configure/maintain routes (that include long MPLS tunnels) that tend to concentrate international connectivity at a relatively small number of routers?
Most likely that's not what's happening. None of your traces showed traffic going to Chicago and then doubling back. I can't say with certainty what this means but my educated guess is this: Chicago is a major peering point for the networks in question, so packets headed more or less that direction tend to hop over there instead of somewhere else. Having crossed over, they enter the MPLS system which is opaque to traceroute, so they're not seen again until they exit MPLS. Traffic originating closer to the final destination goes to a scattering of other MPLS entry points instead, so Chicago looks outsized in the data. The actual traffic flow, opaque inside MPLS, goes through the nodes you'd expect, close to the cable landings. So it's not a question of whether the traffic has an international destination, it's a question of whether that backbone is in the path to the destination (international or not). Rephrase your question as: is there a cost/benefit to providers having a small number of very large peering points accepting traffic into their networks in addition to the myriad small ones. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
Peering connection, I think, can explain this. With some notable exceptions (all of whom participate here, I think), carriers still don’t throw around 100G+ peering routers around like sprinkles on a donut. (Even those big networks don’t, it just looks like they do because they’re freaking huge.) I suspect what you may be seeing is NOT “international carriers all concentrate on a single router” at all, but rather a) “large carriers tend to interconnect at only a few points” and b) “the best path from North America to Europe will therefore tend to always go through the same small set of inter-AS peering routers on this side”. What you’ve described, purely in the public-visible layer 3 internet, is normal in my experience. I would fully expect upwards of 90% of my daily traffic crosses one of 3 peering routers in my network, no surprise there, but ALSO I estimate at least 80% of it crosses one of no more than 5 or 6 routers even as I get 5, 6, 7, 8 hops away from me. The internet isn’t as diversely-pathed as it once was. In the MPLS carrier case, I’m aware of a couple that use MPLS to tunnel peering traffic from their edge back to a centralized “core” router that speaks BGP. Not sure how common this is, I have a very small sample set. But in this example, no matter how diverse the carrier is at L1/L2, your L3 investigation will always hit the same much-smaller set of routers. To directly answer your question, the cost/benefit is driven by the fact that running BGP is (relatively) complicated, error-prone and expensive, compared to not running BGP. And those routers running BGP are (broadly speaking) the routers controlling inter-carrier traffic. So the “chokepoints” naturally occur as an emergent property of each carrier controlling their own operational and financial risk. Very much depends on the carrier and their operational philosophy. I know nearly nothing about Telia, but Zayo doesn’t (didn’t? did they get bought?) have a lot of peering routers – they were historically more of an L2 and/or Private-network operator, as I understand it. (I was never a customer, so that’s hearsay.) -Adam Adam Thompson Consultant, Infrastructure Services [MERLIN] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.ca<mailto:athompson@merlin.mb.ca> www.merlin.mb.ca<http://www.merlin.mb.ca/> From: NANOG <nanog-bounces+athompson=merlin.mb.ca@nanog.org> On Behalf Of PAUL R BARFORD Sent: Tuesday, January 18, 2022 8:49 AM To: davidbass570@gmail.com Cc: nanog@nanog.org Subject: Re: Long hops on international paths Hello David, Understanding the physical topology of the network is not our objective. What we're trying to understand is the logical topology revealed by traceroute (we are well-aware of traceroute limitations) and why a relatively small set of routers in different countries tend to have the majority of the international connections. Our expectation was that layer 3 connectivity revealed in traceroute to be relatively evenly spread out along coasts and near submarine landing points. We're not seeing that. So, the question is what is the cost/benefit to providers to configure/maintain routes (that include long MPLS tunnels) that tend to concentrate international connectivity at a relatively small number of routers? Regards, PB ________________________________ From: davidbass570@gmail.com<mailto:davidbass570@gmail.com> <davidbass570@gmail.com<mailto:davidbass570@gmail.com>> Sent: Tuesday, January 18, 2022 8:22 AM To: PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> Cc: morrowc.lists@gmail.com<mailto:morrowc.lists@gmail.com> <morrowc.lists@gmail.com<mailto:morrowc.lists@gmail.com>>; nanog@nanog.org<mailto:nanog@nanog.org> <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Long hops on international paths I think a large part of your problem is that you’re using trace route to try and determine the full topology of a large complex network. It won’t show the full topology. On Mon, Jan 17, 2022 at 7:43 PM PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> wrote: What we're considering specifically are consecutive (layer 3) hops as identified by traceroute. Thus, TTL is decremented by 1 and no more than 1 (i.e., we have to get full information (not *****) from consecutive hops to consider the link). I have asked my colleague to put together a set of examples. We assume that there are multiple layer 1 and 2 links, and possibly layer 3 hops masked from traceroute by MPLS. But what we're seeing in terms of hops exposed by traceroute make it look like a single (TTL decremented by 1) hop. I'll post the examples when I get them. PB ________________________________ From: morrowc.lists@gmail.com<mailto:morrowc.lists@gmail.com> <morrowc.lists@gmail.com<mailto:morrowc.lists@gmail.com>> Sent: Monday, January 17, 2022 5:13 PM To: PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> Cc: Pengxiong Zhu <pzhu011@ucr.edu<mailto:pzhu011@ucr.edu>>; nanog@nanog.org<mailto:nanog@nanog.org> <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Long hops on international paths On Mon, Jan 17, 2022 at 5:31 PM PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> wrote: Dear Pengxiong, Thanks for your questions: 1. We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the MIDAR alias resolution method to infer IP addresses assigned to the same router. 2. We understand the concerns about IP geolocation. Interfaces of the router in question are assigned similar domain names e.g., “chi-b2-link.ip.twelve99.net<http://chi-b2-link.ip.twelve99.net>” (62.115.50.61). We also used CAIDA’s ITDK, which provides geolocation information, and indicates that this router is located in Chicago. We cross-reference with Maxmind where possible. In this particular case, there is the telltale in the use of "chi" in the domain name. 3. I think nick's point about ttl expiry and missing some context on topology still stands. I'd be that the paths between 2 continents do not actually land in chicago... that you're seeing (or not seeing) missing hops between the coast(s) and chicago inside 1299's network in the US. 1. Hope that helps. Regards, PB ________________________________ From: Pengxiong Zhu <pzhu011@ucr.edu<mailto:pzhu011@ucr.edu>> Sent: Monday, January 17, 2022 3:23 PM To: PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Long hops on international paths Hi Paul, Just curious. How do you determine they are the same routers? Is it based on IP address or MAC addresses? Or using CAIDA’s router alias database? Also how do you draw the conclusion that the AS1299 router is indeed in Chicago? IP-geolocation based on rDNS is not always accurate though. Pengxiong On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> wrote: Hello, I am a researcher at the University of Wisconsin. My colleagues at Northwestern University and I are studying international Internet connectivity and would appreciate your perspective on a recent finding. We're using traceroute data from CAIDA's Ark project for our work. We've observed that many international links (i.e., a single hop on an end-to-end path that connects two countries where end points on the hop are identified via rDNS) tend to originate/terminate at the same routers. Said another way, we are observing a relatively small set of routers in different countries tend to have a majority of the international connections - this is especially the case for hops that terminate in the US. For example, there is a router operated by Telia (AS1299) in Chicago that has a high concentration of such links. We were a bit surprised by this finding since even though it makes sense that the set of providers is relatively small (i.e., those that offer global connectivity), we assumed that the set of routers that used for international connectivity within any one country would tend to be more widely distributed (at least with respect to how they appear in traceroute data - MPLS notwithstanding). We're interested in whether or not this is indeed standard practice and if so, the cost/benefit for configuring international connectivity in this way? Any thoughts or insights you might have would be greatly appreciated - off-list responses are welcome. Thank you. Regards, PB Paul Barford University of Wisconsin - Madison -- Regards, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside
Hello Adam, Thanks for your follow-up - it helps to clarify that we've been observing in our analysis. The tradeoff between cost (100G+ links), manageability (related to BGP) and risk (i.e., concentration of high-value links in a few locations) is really interesting and not something that we were aware of. PB ________________________________ From: athompson@merlin.mb.ca <athompson@merlin.mb.ca> Sent: Tuesday, January 25, 2022 10:54 AM To: PAUL R BARFORD <pb@cs.wisc.edu>; davidbass570@gmail.com <davidbass570@gmail.com> Cc: nanog@nanog.org <nanog@nanog.org> Subject: RE: Long hops on international paths Peering connection, I think, can explain this. With some notable exceptions (all of whom participate here, I think), carriers still don’t throw around 100G+ peering routers around like sprinkles on a donut. (Even those big networks don’t, it just looks like they do because they’re freaking huge.) I suspect what you may be seeing is NOT “international carriers all concentrate on a single router” at all, but rather a) “large carriers tend to interconnect at only a few points” and b) “the best path from North America to Europe will therefore tend to always go through the same small set of inter-AS peering routers on this side”. What you’ve described, purely in the public-visible layer 3 internet, is normal in my experience. I would fully expect upwards of 90% of my daily traffic crosses one of 3 peering routers in my network, no surprise there, but ALSO I estimate at least 80% of it crosses one of no more than 5 or 6 routers even as I get 5, 6, 7, 8 hops away from me. The internet isn’t as diversely-pathed as it once was. In the MPLS carrier case, I’m aware of a couple that use MPLS to tunnel peering traffic from their edge back to a centralized “core” router that speaks BGP. Not sure how common this is, I have a very small sample set. But in this example, no matter how diverse the carrier is at L1/L2, your L3 investigation will always hit the same much-smaller set of routers. To directly answer your question, the cost/benefit is driven by the fact that running BGP is (relatively) complicated, error-prone and expensive, compared to not running BGP. And those routers running BGP are (broadly speaking) the routers controlling inter-carrier traffic. So the “chokepoints” naturally occur as an emergent property of each carrier controlling their own operational and financial risk. Very much depends on the carrier and their operational philosophy. I know nearly nothing about Telia, but Zayo doesn’t (didn’t? did they get bought?) have a lot of peering routers – they were historically more of an L2 and/or Private-network operator, as I understand it. (I was never a customer, so that’s hearsay.) -Adam Adam Thompson Consultant, Infrastructure Services [MERLIN] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.ca<mailto:athompson@merlin.mb.ca> www.merlin.mb.ca<http://www.merlin.mb.ca/> From: NANOG <nanog-bounces+athompson=merlin.mb.ca@nanog.org> On Behalf Of PAUL R BARFORD Sent: Tuesday, January 18, 2022 8:49 AM To: davidbass570@gmail.com Cc: nanog@nanog.org Subject: Re: Long hops on international paths Hello David, Understanding the physical topology of the network is not our objective. What we're trying to understand is the logical topology revealed by traceroute (we are well-aware of traceroute limitations) and why a relatively small set of routers in different countries tend to have the majority of the international connections. Our expectation was that layer 3 connectivity revealed in traceroute to be relatively evenly spread out along coasts and near submarine landing points. We're not seeing that. So, the question is what is the cost/benefit to providers to configure/maintain routes (that include long MPLS tunnels) that tend to concentrate international connectivity at a relatively small number of routers? Regards, PB ________________________________ From: davidbass570@gmail.com<mailto:davidbass570@gmail.com> <davidbass570@gmail.com<mailto:davidbass570@gmail.com>> Sent: Tuesday, January 18, 2022 8:22 AM To: PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> Cc: morrowc.lists@gmail.com<mailto:morrowc.lists@gmail.com> <morrowc.lists@gmail.com<mailto:morrowc.lists@gmail.com>>; nanog@nanog.org<mailto:nanog@nanog.org> <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Long hops on international paths I think a large part of your problem is that you’re using trace route to try and determine the full topology of a large complex network. It won’t show the full topology. On Mon, Jan 17, 2022 at 7:43 PM PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> wrote: What we're considering specifically are consecutive (layer 3) hops as identified by traceroute. Thus, TTL is decremented by 1 and no more than 1 (i.e., we have to get full information (not *****) from consecutive hops to consider the link). I have asked my colleague to put together a set of examples. We assume that there are multiple layer 1 and 2 links, and possibly layer 3 hops masked from traceroute by MPLS. But what we're seeing in terms of hops exposed by traceroute make it look like a single (TTL decremented by 1) hop. I'll post the examples when I get them. PB ________________________________ From: morrowc.lists@gmail.com<mailto:morrowc.lists@gmail.com> <morrowc.lists@gmail.com<mailto:morrowc.lists@gmail.com>> Sent: Monday, January 17, 2022 5:13 PM To: PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> Cc: Pengxiong Zhu <pzhu011@ucr.edu<mailto:pzhu011@ucr.edu>>; nanog@nanog.org<mailto:nanog@nanog.org> <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Long hops on international paths On Mon, Jan 17, 2022 at 5:31 PM PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> wrote: Dear Pengxiong, Thanks for your questions: 1. We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the MIDAR alias resolution method to infer IP addresses assigned to the same router. 2. We understand the concerns about IP geolocation. Interfaces of the router in question are assigned similar domain names e.g., “chi-b2-link.ip.twelve99.net<http://chi-b2-link.ip.twelve99.net>” (62.115.50.61). We also used CAIDA’s ITDK, which provides geolocation information, and indicates that this router is located in Chicago. We cross-reference with Maxmind where possible. In this particular case, there is the telltale in the use of "chi" in the domain name. 3. I think nick's point about ttl expiry and missing some context on topology still stands. I'd be that the paths between 2 continents do not actually land in chicago... that you're seeing (or not seeing) missing hops between the coast(s) and chicago inside 1299's network in the US. 1. Hope that helps. Regards, PB ________________________________ From: Pengxiong Zhu <pzhu011@ucr.edu<mailto:pzhu011@ucr.edu>> Sent: Monday, January 17, 2022 3:23 PM To: PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Long hops on international paths Hi Paul, Just curious. How do you determine they are the same routers? Is it based on IP address or MAC addresses? Or using CAIDA’s router alias database? Also how do you draw the conclusion that the AS1299 router is indeed in Chicago? IP-geolocation based on rDNS is not always accurate though. Pengxiong On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD <pb@cs.wisc.edu<mailto:pb@cs.wisc.edu>> wrote: Hello, I am a researcher at the University of Wisconsin. My colleagues at Northwestern University and I are studying international Internet connectivity and would appreciate your perspective on a recent finding. We're using traceroute data from CAIDA's Ark project for our work. We've observed that many international links (i.e., a single hop on an end-to-end path that connects two countries where end points on the hop are identified via rDNS) tend to originate/terminate at the same routers. Said another way, we are observing a relatively small set of routers in different countries tend to have a majority of the international connections - this is especially the case for hops that terminate in the US. For example, there is a router operated by Telia (AS1299) in Chicago that has a high concentration of such links. We were a bit surprised by this finding since even though it makes sense that the set of providers is relatively small (i.e., those that offer global connectivity), we assumed that the set of routers that used for international connectivity within any one country would tend to be more widely distributed (at least with respect to how they appear in traceroute data - MPLS notwithstanding). We're interested in whether or not this is indeed standard practice and if so, the cost/benefit for configuring international connectivity in this way? Any thoughts or insights you might have would be greatly appreciated - off-list responses are welcome. Thank you. Regards, PB Paul Barford University of Wisconsin - Madison -- Regards, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside
I guess it depends what you’re considering a “very few” number of routers but this seems to be an expected outcome. While there are a large number of wet cable landing stations, they are highly concentrated near a small number of metro areas, and with the exception of capacity owned by the ILECs, the supermajority of routers terminating that capacity in the US are going to live in fewer than ten discrete carrier hotel locations. (It’s worth noting that terrestrial capacity coming in from Mexico and Canada also terminates in a small number of locations, although the overlap between the two lists is fairly small). In addition, while the links likely terminate in multiple devices at a given location, carriers more likely to undersubscribe transoceanic core capacity than other areas of the core, which means that for many carriers it’s unlikely you’d see multiple paths show up in a trace unless you catch it during an outage situation. That said, seeing transoceanic links terminate in Chicago is likely an artifact of hops missing in a trace; although I am familiar with a couple of more niche providers that extend transoceanic capacity into non-coastal markets on optical gear in order to meet specific performance needs, this is unlikely to be seen in the network of a Tier 1 or similarly scaled network. Dave Cohen craetdave@gmail.com
On Jan 17, 2022, at 6:15 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Mon, Jan 17, 2022 at 5:31 PM PAUL R BARFORD <pb@cs.wisc.edu> wrote: Dear Pengxiong,
Thanks for your questions:
We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the MIDAR alias resolution method to infer IP addresses assigned to the same router. We understand the concerns about IP geolocation. Interfaces of the router in question are assigned similar domain names e.g., “chi-b2-link.ip.twelve99.net” (62.115.50.61). We also used CAIDA’s ITDK, which provides geolocation information, and indicates that this router is located in Chicago. We cross-reference with Maxmind where possible. In this particular case, there is the telltale in the use of "chi" in the domain name.
I think nick's point about ttl expiry and missing some context on topology still stands. I'd be that the paths between 2 continents do not actually land in chicago... that you're seeing (or not seeing) missing hops between the coast(s) and chicago inside 1299's network in the US.
Hope that helps.
Regards, PB From: Pengxiong Zhu <pzhu011@ucr.edu> Sent: Monday, January 17, 2022 3:23 PM To: PAUL R BARFORD <pb@cs.wisc.edu> Cc: nanog@nanog.org <nanog@nanog.org> Subject: Re: Long hops on international paths
Hi Paul,
Just curious. How do you determine they are the same routers? Is it based on IP address or MAC addresses? Or using CAIDA’s router alias database?
Also how do you draw the conclusion that the AS1299 router is indeed in Chicago? IP-geolocation based on rDNS is not always accurate though.
Pengxiong
On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD <pb@cs.wisc.edu> wrote: Hello,
I am a researcher at the University of Wisconsin. My colleagues at Northwestern University and I are studying international Internet connectivity and would appreciate your perspective on a recent finding.
We're using traceroute data from CAIDA's Ark project for our work. We've observed that many international links (i.e., a single hop on an end-to-end path that connects two countries where end points on the hop are identified via rDNS) tend to originate/terminate at the same routers. Said another way, we are observing a relatively small set of routers in different countries tend to have a majority of the international connections - this is especially the case for hops that terminate in the US. For example, there is a router operated by Telia (AS1299) in Chicago that has a high concentration of such links. We were a bit surprised by this finding since even though it makes sense that the set of providers is relatively small (i.e., those that offer global connectivity), we assumed that the set of routers that used for international connectivity within any one country would tend to be more widely distributed (at least with respect to how they appear in traceroute data - MPLS notwithstanding).
We're interested in whether or not this is indeed standard practice and if so, the cost/benefit for configuring international connectivity in this way?
Any thoughts or insights you might have would be greatly appreciated - off-list responses are welcome.
Thank you.
Regards, PB
Paul Barford University of Wisconsin - Madison
--
Regards, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside
participants (14)
-
Adam Thompson -
Christopher Morrow -
Dave Cohen -
David Bass -
Lady Benjamin Cannon of Glencoe, ASCE -
Lukas Tribus -
Mark Tinka -
Michael Hare -
Mike Hammett -
Nick Hilliard -
PAUL R BARFORD -
Pengxiong Zhu -
Saku Ytti -
William Herrin