DDoS traffic is coming from China to the outside world, which should saturate the upstream link of China, however, what we observed is that the upstream link has high and stable performance, while the downstream link of China, which is traffic coming from the outside world to China, is suffering from slow speed. Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside On Mon, Mar 2, 2020 at 8:11 AM Compton, Rich A <Rich.Compton@charter.com> wrote:

My guess is that it’s all the DDoS traffic coming from China saturating the links.

*From: *NANOG Email List <nanog-bounces@nanog.org> on behalf of Pengxiong Zhu <pzhu011@ucr.edu> *Date: *Monday, March 2, 2020 at 8:58 AM *To: *NANOG list <nanog@nanog.org> *Cc: *Zhiyun Qian <zhiyunq@cs.ucr.edu> *Subject: *China’s Slow Transnational Network

Hi all,

We are a group of researchers at University of California, Riverside who have been working on measuring the transnational network performance (and have previously asked questions on the mailing list). Our work has now led to a publication in Sigmetrics 2020 and we are eager to share some

interesting findings.

We find China's transnational networks have extremely poor performance when accessing foreign sites, where the throughput is often persistently

low (e.g., for the majority of the daytime). Compared to other countries we measured including both developed and developing, China's transnational network performance is among the worst (comparable and even worse than some African countries).

Measuring from more than 400 pairs of mainland China and foreign nodes over more than 53 days, our result shows when data transferring from foreign nodes to China, 79% of measured connections has throughput lower than the 1Mbps, sometimes it is even much lower. The slow speed occurs only during certain times and forms a diurnal pattern that resembles congestion (irrespective of network protocol and content), please see the following figure. The diurnal pattern is fairly stable, 80% to 95% of the transnational connections have a less than 3 hours standard deviation of the slowdown hours each day over the entire duration. However, the speed rises up from 1Mbps to 4Mbps in about half an hour.

[image: blob:null/71cf5a6a-3841-41ce-a1d4-207b59182189]

We are able to confirm that high packet loss rates and delays are incurred in the foreign-to-China direction only. Moreover, the end-to-end loss rate could rise up to 40% during the slow period, with ~15% on average.

There are a few things noteworthy regarding the phenomenon. First of all, all traffic types are treated equally, HTTP(S), VPN, etc., which means it is discriminating or differentiating any specific kinds of traffic. Second, we found for 71% of connections, the bottleneck is located inside China (the second hop after entering China or further), which means that it is mostly unrelated to the transnational link itself (e.g., submarine cable). Yet we never observed any such domestic traffic slowdowns within China.

Assuming this is due to congestion, it is unclear why the infrastructures within China that handles transnational traffic is not even capable to handle the capacity of transnational links, e.g., submarine cable, which maybe the most expensive investment themselves.

Here is the link to our paper:

https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf

We appreciate any comments or feedback.

--

Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

Yes, we agree. The poor transnational Internet performance effectively puts any foreign business that does not have a physical presence (i.e., servers) in China at a disadvantage. The challenge is to find out direct evidence to prove mandatory content filtering at the border, if the government is actually doing it. Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside On Mon, Mar 2, 2020 at 8:38 AM Matt Corallo <nanog@as397444.net> wrote:

It also gives local competitors a leg up by helping domestic apps perform better simply by being hosted domestically (or making foreign players host inside China).

On Mar 2, 2020, at 11:27, Ben Cannon <ben@6by7.net> wrote:

 It’s the Government doing mandatory content filtering at the border. Their hardware is either deliberately or accidentally poor-performing.

I believe providing limited and throttled external connectivity may be deliberate; think of how that curtails for one thing; streaming video?

-Ben.

-Ben Cannon CEO 6x7 Networks & 6x7 Telecom, LLC ben@6by7.net

On Mar 1, 2020, at 9:00 PM, Pengxiong Zhu <pzhu011@ucr.edu> wrote:

Hi all,

We are a group of researchers at University of California, Riverside who have been working on measuring the transnational network performance (and have previously asked questions on the mailing list). Our work has now led to a publication in Sigmetrics 2020 and we are eager to share some interesting findings.

We find China's transnational networks have extremely poor performance when accessing foreign sites, where the throughput is often persistently low (e.g., for the majority of the daytime). Compared to other countries we measured including both developed and developing, China's transnational network performance is among the worst (comparable and even worse than some African countries).

Measuring from more than 400 pairs of mainland China and foreign nodes over more than 53 days, our result shows when data transferring from foreign nodes to China, 79% of measured connections has throughput lower than the 1Mbps, sometimes it is even much lower. The slow speed occurs only during certain times and forms a diurnal pattern that resembles congestion (irrespective of network protocol and content), please see the following figure. The diurnal pattern is fairly stable, 80% to 95% of the transnational connections have a less than 3 hours standard deviation of the slowdown hours each day over the entire duration. However, the speed rises up from 1Mbps to 4Mbps in about half an hour.

We are able to confirm that high packet loss rates and delays are incurred in the foreign-to-China direction only. Moreover, the end-to-end loss rate could rise up to 40% during the slow period, with ~15% on average.

There are a few things noteworthy regarding the phenomenon. First of all, all traffic types are treated equally, HTTP(S), VPN, etc., which means it is discriminating or differentiating any specific kinds of traffic. Second, we found for 71% of connections, the bottleneck is located inside China (the second hop after entering China or further), which means that it is mostly unrelated to the transnational link itself (e.g., submarine cable). Yet we never observed any such domestic traffic slowdowns within China. Assuming this is due to congestion, it is unclear why the infrastructures within China that handles transnational traffic is not even capable to handle the capacity of transnational links, e.g., submarine cable, which maybe the most expensive investment themselves.

Here is the link to our paper: https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf

We appreciate any comments or feedback. --

Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside

You seem to be implying that you don't believe/can't see the GFW

No, that's not what I meant. I thought mandatory content filtering at the border means traffic throttling at the border, deliberately or accidentally rate-limiting the traffic, now I think he was referring to GFW and the side effect of deep packet inspection. In fact, we designed a small experiment to locate the hops with GFW presence, and then try to match them with the bottleneck hops. We found only in 34.45% of the cases, the GFW hops match the bottleneck hops. Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside On Mon, Mar 2, 2020 at 1:13 PM Matt Corallo <nanog@as397444.net> wrote:

...

find out direct evidence of mandatory content filtering at the border

You seem to be implying that you don't believe/can't see the GFW, which seems surprising. I've personally had issues with traffic crossing it getting RST'd (luckily I was fortunate enough to cross through a GFW instance which was easy to avoid with a simple iptables DROP), but its also one of the most well-studied bits of opaque internet censorship gear in the world. I'm not sure how you could possibly miss it.

Matt

On 3/2/20 2:55 PM, Pengxiong Zhu wrote:

...

Yes, we agree. The poor transnational Internet performance effectively puts any foreign business that does not have a physical presence (i.e., servers) in China at a disadvantage. The challenge is to find out direct evidence to prove mandatory content filtering at the border, if the government is actually doing it.

Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside

On Mon, Mar 2, 2020 at 8:38 AM Matt Corallo <nanog@as397444.net <mailto:nanog@as397444.net>> wrote:

It also gives local competitors a leg up by helping domestic apps perform better simply by being hosted domestically (or making foreign players host inside China).

...

On Mar 2, 2020, at 11:27, Ben Cannon <ben@6by7.net <mailto:ben@6by7.net>> wrote:

 It’s the Government doing mandatory content filtering at the border. Their hardware is either deliberately or accidentally poor-performing.

I believe providing limited and throttled external connectivity may be deliberate; think of how that curtails for one thing; streaming video?

-Ben.

-Ben Cannon CEO 6x7 Networks & 6x7 Telecom, LLC ben@6by7.net <mailto:ben@6by7.net>

...

On Mar 1, 2020, at 9:00 PM, Pengxiong Zhu <pzhu011@ucr.edu <mailto:pzhu011@ucr.edu>> wrote:

Hi all,

We are a group of researchers at University of California, Riverside who have been working on measuring the transnational network performance (and have previously asked questions on the mailing list). Our work has now led to a publication in Sigmetrics 2020 and we are eager to share some interesting findings.

We find China's transnational networks have extremely poor performance when accessing foreign sites, where the throughput is often persistently low (e.g., for the majority of the daytime). Compared to other countries we measured including both developed and developing, China's transnational network performance is among the worst (comparable and even worse than some African countries).

Measuring from more than 400 pairs of mainland China and foreign nodes over more than 53 days, our result shows when data transferring from foreign nodes to China, 79% of measured connections has throughput lower than the 1Mbps, sometimes it is even much lower. The slow speed occurs only during certain times and forms a diurnal pattern that resembles congestion (irrespective of network protocol and content), please see the following figure. The diurnal pattern is fairly stable, 80% to 95% of the transnational connections have a less than 3 hours standard deviation of the slowdown hours each day over the entire duration. However, the speed rises up from 1Mbps to 4Mbps in about half an hour.

We are able to confirm that high packet loss rates and delays are incurred in the foreign-to-China direction only. Moreover, the end-to-end loss rate could rise up to 40% during the slow period, with ~15% on average.

There are a few things noteworthy regarding the phenomenon. First of all, all traffic types are treated equally, HTTP(S), VPN, etc., which means it is discriminating or differentiating any specific kinds of traffic. Second, we found for 71% of connections, the bottleneck is located inside China (the second hop after entering China or further), which means that it is mostly unrelated to the transnational link itself (e.g., submarine cable). Yet we never observed any such domestic traffic slowdowns within China. Assuming this is due to congestion, it is unclear why the infrastructures within China that handles transnational traffic is not even capable to handle the capacity of transnational links, e.g., submarine cable, which maybe the most expensive investment themselves.

Here is the link to our paper: https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf

We appreciate any comments or feedback. --

Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside

On 3/Mar/20 00:57, Tom Paseka via NANOG wrote:

Prices are set artificially high, so their interconnection partners wont purchase enough capacity. additionally, the three don't purchase enough to cover demand for their own network. Results in congestion.

We've seen somewhat similar behaviour from these networks when peering outside of China, perhaps, to influence the flow of money and traffic. We have zero patience for such things. Mark.

On 15/Mar/20 09:55, Pengxiong Zhu wrote:

I know Caida has one paper on the congestion on Africa's IXPs substrate.

I can't think of a single IXP in Africa that is "congested". Do you have more data?

However, we did find Kenya, Nigeria and South Africa have better transnational performance than China,...

Not having great big firewalls tends to help :-).

while the performance of Ghana and Egypt was worse than China, at least that's what we saw from the web4africa VPSes we brought.

While I can't speak to the national backbones of Ghana and Egypt, it would be good to obtain multiple perspectives, just to be sure.

Sorry I am a bit confused here. What do you mean by "these networks"? When you say "peering outside of China", who is peering who exactly?

I know about Chinese operators who will deliberately congest peering ports to influence 3rd party network behaviour. Mark.

On 15/Mar/20 22:51, Frank Habicht wrote:

thanks for the "quotes", Mark. I agree.

https://www.caida.org/publications/presentations/2018/investigating_causes_c...

page 23: Results Overview • No evidence of widespread congestion - 2.2% of discovered link showed evidence of congestion at the end of our measurements campaign

page 34: Conclusions • Measured IXPs were congestion-free, which promotes peering in the region

https://conferences.sigcomm.org/imc/2017/papers/imc17-final182.pdf

my conclusion: s/congestion/congestion or the lack thereof/g

Frank Habicht

PS: yes, i could name peers that once had inadequate links into an IXP. but for how long did that happen? (yes..., any minute is too long...)

Indeed. There was a time when backhaul links between ISP routers at the exchange point and their nearest PoP were based on E1's, wireless, e.t.c. But that could be said of, pretty much, every exchange point that kicked off inside of the last 2.5 decades. Nowadays, such links, if they exist, are the very deep exception, not the rule. Mark.

On 21/Mar/20 09:09, Pengxiong Zhu wrote:

How do they deliberately congest peering ports? Do you hear from those Chinese operators or you observe this from the traffic?

Simple - let them run at 350% of capacity and pipeline upgrades for Lord knows how long :-). On a serious note, let's have a beer.

Seems like you also think GFW is part of the cause,

I do - each of my trips to China have questioned the role of my VPN for my online experience.

however, we don't have direct evidence.

I won't argue with you there, you did the groundwork. I'm just being anecdotal.

Just curious, What is your "problems"? I thought it's congestion.

Accessibility and penetration rates. Mark.

Thank you for your insights. We are not so familiar with interconnect and peering, we will ask you some questions for clarification first. Hope you don't mind. :-) When there is a tri-opoly, with no opportunity of competition, its easily

possible to set prices which are very different than market conditions.

I assume the tri-opoly involves a Chinese ISP, outside ISP A, outside ISP B. Who is competing with whom? Why its easily possible to set prices which are very different than market conditions?

additionally, the three don't purchase enough to cover demand for their own network.

Do you mean that the three don't purchase enough capacity for their traffic going out of their network(China->Outside)? If this is what you mean, however, we don't observe low speed in that direction. We assume there is not so much traffic going out of China, comparing to the traffic coming in. Also, why would the three purchase outbound traffic if they set their inbound traffic artificially high? They could charge some peers less for the outbound traffic to solve the problem. Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside On Mon, Mar 2, 2020 at 2:58 PM Tom Paseka <tom@cloudflare.com> wrote:

Most of the performance hit is because of commercial actions, not censorship.

When there is a tri-opoly, with no opportunity of competition, its easily possible to set prices which are very different than market conditions. This is what is happening here.

Prices are set artificially high, so their interconnection partners wont purchase enough capacity. additionally, the three don't purchase enough to cover demand for their own network. Results in congestion.

On Mon, Mar 2, 2020 at 2:49 PM Pengxiong Zhu <pzhu011@ucr.edu> wrote:

...

You seem to be implying that you don't believe/can't see the GFW

No, that's not what I meant. I thought mandatory content filtering at the border means traffic throttling at the border, deliberately or accidentally rate-limiting the traffic, now I think he was referring to GFW and the side effect of deep packet inspection.

In fact, we designed a small experiment to locate the hops with GFW presence, and then try to match them with the bottleneck hops. We found only in 34.45% of the cases, the GFW hops match the bottleneck hops.

Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside

On Mon, Mar 2, 2020 at 1:13 PM Matt Corallo <nanog@as397444.net> wrote:

...

...

find out direct evidence of mandatory content filtering at the border

You seem to be implying that you don't believe/can't see the GFW, which seems surprising. I've personally had issues with traffic crossing it getting RST'd (luckily I was fortunate enough to cross through a GFW instance which was easy to avoid with a simple iptables DROP), but its also one of the most well-studied bits of opaque internet censorship gear in the world. I'm not sure how you could possibly miss it.

Matt

On 3/2/20 2:55 PM, Pengxiong Zhu wrote:

...

Yes, we agree. The poor transnational Internet performance effectively puts any foreign business that does not have a physical presence (i.e., servers) in China at a disadvantage. The challenge is to find out direct evidence to prove mandatory content filtering at the border, if the government is actually doing it.

Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside

On Mon, Mar 2, 2020 at 8:38 AM Matt Corallo <nanog@as397444.net <mailto:nanog@as397444.net>> wrote:

It also gives local competitors a leg up by helping domestic apps perform better simply by being hosted domestically (or making foreign players host inside China).

...

On Mar 2, 2020, at 11:27, Ben Cannon <ben@6by7.net <mailto:ben@6by7.net>> wrote:

 It’s the Government doing mandatory content filtering at the border. Their hardware is either deliberately or accidentally poor-performing.

I believe providing limited and throttled external connectivity may be deliberate; think of how that curtails for one thing; streaming video?

-Ben.

-Ben Cannon CEO 6x7 Networks & 6x7 Telecom, LLC ben@6by7.net <mailto:ben@6by7.net>

...

On Mar 1, 2020, at 9:00 PM, Pengxiong Zhu <pzhu011@ucr.edu <mailto:pzhu011@ucr.edu>> wrote:

Hi all,

We are a group of researchers at University of California, Riverside who have been working on measuring the transnational network performance (and have previously asked questions on the mailing list). Our work has now led to a publication in Sigmetrics 2020 and we are eager to share some interesting findings.

We find China's transnational networks have extremely poor performance when accessing foreign sites, where the throughput is often persistently low (e.g., for the majority of the daytime). Compared to other countries we measured including both developed and developing, China's transnational network performance is among the worst (comparable and even worse than some African countries).

Measuring from more than 400 pairs of mainland China and foreign nodes over more than 53 days, our result shows when data transferring from foreign nodes to China, 79% of measured connections has throughput lower than the 1Mbps, sometimes it is even much lower. The slow speed occurs only during certain times and forms a diurnal pattern that resembles congestion (irrespective of network protocol and content), please see the following figure. The diurnal pattern is fairly stable, 80% to 95% of the transnational connections have a less than 3 hours standard deviation of the slowdown hours each day over the entire duration. However, the speed rises up from 1Mbps to 4Mbps in about half an hour.

We are able to confirm that high packet loss rates and delays are incurred in the foreign-to-China direction only. Moreover, the end-to-end loss rate could rise up to 40% during the slow period, with ~15% on average.

There are a few things noteworthy regarding the phenomenon. First of all, all traffic types are treated equally, HTTP(S), VPN, etc., which means it is discriminating or differentiating any specific kinds of traffic. Second, we found for 71% of connections, the bottleneck is located inside China (the second hop after entering China or further), which means that it is mostly unrelated to the transnational link itself (e.g., submarine cable). Yet we never observed any such domestic traffic slowdowns within China. Assuming this is due to congestion, it is unclear why the infrastructures within China that handles transnational traffic is not even capable to handle the capacity of transnational links, e.g., submarine cable, which maybe the most expensive investment themselves.

Here is the link to our paper: https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf

We appreciate any comments or feedback. --

Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside

Yes, the sentence is missing a ‘not’. Sorry about that. It’s not discriminating or differentiating any specific kinds of traffic. On Mon, Mar 2, 2020 at 10:56 AM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote:

On Sun, 01 Mar 2020 21:00:05 -0800, Pengxiong Zhu said:

...

There are a few things noteworthy regarding the phenomenon. First of all, all traffic types are treated equally, HTTP(S), VPN, etc., which means it is discriminating or differentiating any specific kinds of traffic.

This sentence is missing a 'not'. However, I can't tell if it's "not treated equally" or "not discriminating"....

-- Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside

Yes, CERNET has indeed smaller slowdown period(4 hours) than commodity networks(12 hours), but still has slowdown. Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside On Mon, Mar 2, 2020 at 2:36 PM David Burns <davburns@gmail.com> wrote:

Did you compare CERNET with commodity networks? (My anecdotal observations from a couple years ago suggest that Internet2 to CERNET is very good when other paths are poor to unusable.)

--David Burns

On Mon, Mar 2, 2020 at 7:58 AM Pengxiong Zhu <pzhu011@ucr.edu> wrote:

...

Hi all,

We are a group of researchers at University of California, Riverside who have been working on measuring the transnational network performance (and have previously asked questions on the mailing list). Our work has now led to a publication in Sigmetrics 2020 and we are eager to share some interesting findings.

We find China's transnational networks have extremely poor performance when accessing foreign sites, where the throughput is often persistently low (e.g., for the majority of the daytime). Compared to other countries we measured including both developed and developing, China's transnational network performance is among the worst (comparable and even worse than some African countries).

Measuring from more than 400 pairs of mainland China and foreign nodes over more than 53 days, our result shows when data transferring from foreign nodes to China, 79% of measured connections has throughput lower than the 1Mbps, sometimes it is even much lower. The slow speed occurs only during certain times and forms a diurnal pattern that resembles congestion (irrespective of network protocol and content), please see the following figure. The diurnal pattern is fairly stable, 80% to 95% of the transnational connections have a less than 3 hours standard deviation of the slowdown hours each day over the entire duration. However, the speed rises up from 1Mbps to 4Mbps in about half an hour.

We are able to confirm that high packet loss rates and delays are incurred in the foreign-to-China direction only. Moreover, the end-to-end loss rate could rise up to 40% during the slow period, with ~15% on average.

There are a few things noteworthy regarding the phenomenon. First of all, all traffic types are treated equally, HTTP(S), VPN, etc., which means it is discriminating or differentiating any specific kinds of traffic. Second, we found for 71% of connections, the bottleneck is located inside China (the second hop after entering China or further), which means that it is mostly unrelated to the transnational link itself (e.g., submarine cable). Yet we never observed any such domestic traffic slowdowns within China. Assuming this is due to congestion, it is unclear why the infrastructures within China that handles transnational traffic is not even capable to handle the capacity of transnational links, e.g., submarine cable, which maybe the most expensive investment themselves.

Here is the link to our paper: https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf

We appreciate any comments or feedback. --

Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside