Re: COVID-19 vs. our Networks
The access facility and the underlying long haul are telecommunications services. The application provided using that facility may or may not be. The congestion you were experiencing was not with the telecommunications facility itself, but with the application running on it, and was as you state, outside of your network - on a CDN hosted service. Your argument is with your third party hosted service. Their argument is with their CDN. Internet exchange points are not regulated. Interconnections between ISPs and CDNs are private agreements, and are always at risk of congestion and commercial dispute between the parties. There is a long history of this. If you have a direct layer 2 or 3 private line to your hosted service provider's CDN, and it was not performing as per the SLA, then you need to take that up with them. If the underlying telecommunications facility failed, and was classified as critical infrastructure, and not restored in a timely manner, then you need to take that up with the provider of that infrastructure. I'm not trying to be difficult, but the fact remains that there is a distinction between telecommunications services, and Internet services. The fact that Internet services (and I'm not talking about any one particular DIA circuit, but the rather the global network of networks) work so well most of the time, such that people tend to start treating it as a substitute for telecommunications services is pretty impressive. There are cost/benefit tradeoffs for using cloud hosted services and public Internet infrastructure. You save money by not operating the data centre yourself, but you pay for it in reliability. Your organization may have made that choice, but to say that because you chose to put critical applications on Internet infrastructure, other users of the Internet should take a back seat to your needs seems to be a bit of a stretch. Again, if your provider sold this to you as something that was NOT relying on public Internet, and was Layer 2/3 private managed services with dedicated bandwidth, then you need to have a conversation with them. At 02:01 PM 14/03/2020, Mike Bolitho wrote:
First of all, we use a mixture of layer 2/3 private lines and DIA circuits. You don't know our infrastructure, stop being condescending. It goes against the spirit of this mailing list.
Second, yes, the Internet is protected. Both public and private lines. I know this because we have TSP coded circuits and I spent four years at a Tier I ISP servicing TSP coded circuits
Third, the trouble we had was a third party service having congestion issues. They are hosted by the same CDN as Call of Duty. The problem was both outside of our control and our third party service's control. The chokepoint was between ISPs/IXPs and the CDN. I've seen this time and time again while working at the aforementioned ISP. Saturated links on ISP/IXP/CDN networks. This is where the TSP code comes in. In this day and age of cloud services, it is financially unfeasible for every company to have a private line to every single cloud provider. That's preposterous to even suggest.
- Mike Bolitho
On Sat, Mar 14, 2020 at 10:40 AM Clayton Zekelman <<mailto:clayton@mnsi.net>clayton@mnsi.net> wrote:
The Internet is not a telecommunications service, according to your FCC. If you want predictability, buy WAN circuits, not Internet circuits.  If your provider is co-mingling Internet and WAN traffic (i.e. circuits with defined endpoints vs. public Internet or VPN), then you need to talk to them about their prioritization.
If you have mission critical applications, put them on mission critical infrastructure, not the public Internet.
Oh, that's right - Internet circuits are cheaper than WAN circuits
-- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409
I don't want to get in a fight, but absolutely: Folks saw congestion from a massive free content drop this past week. But as folks had called out, that was the CDN angle of distributing that content rather than the actual game play. There is a rather long discussion about that in the "akamai yesterday - what in the world was that" thread from Jan/Feb/March. I don't want to trivialize the challenges people may have from knock-on effects of upstream providers facing congestion or CDNs that have co-located content on nodes serving those bits with other bits, but did want to carve out regular online gameplay from content distribution. Whether people end up adjusting plans around large content distribution at this time, I guess remains to be seen. On Sat., Mar. 14, 2020, 11:49 Clayton Zekelman <clayton@mnsi.net> wrote:
The access facility and the underlying long haul are telecommunications services. The application provided using that facility may or may not be. The congestion you were experiencing was not with the telecommunications facility itself, but with the application running on it, and was as you state, outside of your network - on a CDN hosted service. Your argument is with your third party hosted service. Their argument is with their CDN.
Internet exchange points are not regulated. Interconnections between ISPs and CDNs are private agreements, and are always at risk of congestion and commercial dispute between the parties. There is a long history of this.
If you have a direct layer 2 or 3 private line to your hosted service provider's CDN, and it was not performing as per the SLA, then you need to take that up with them.
If the underlying telecommunications facility failed, and was classified as critical infrastructure, and not restored in a timely manner, then you need to take that up with the provider of that infrastructure.
I'm not trying to be difficult, but the fact remains that there is a distinction between telecommunications services, and Internet services. The fact that Internet services (and I'm not talking about any one particular DIA circuit, but the rather the global network of networks) work so well most of the time, such that people tend to start treating it as a substitute for telecommunications services is pretty impressive.
There are cost/benefit tradeoffs for using cloud hosted services and public Internet infrastructure. You save money by not operating the data centre yourself, but you pay for it in reliability. Your organization may have made that choice, but to say that because you chose to put critical applications on Internet infrastructure, other users of the Internet should take a back seat to your needs seems to be a bit of a stretch. Again, if your provider sold this to you as something that was NOT relying on public Internet, and was Layer 2/3 private managed services with dedicated bandwidth, then you need to have a conversation with them.
At 02:01 PM 14/03/2020, Mike Bolitho wrote:
First of all, we use a mixture of layer 2/3 private lines and DIA circuits. You don't know our infrastructure, stop being condescending. It goes against the spirit of this mailing list.
Second, yes, the Internet is protected. Both public and private lines. I know this because we have TSP coded circuits and I spent four years at a Tier I ISP servicing TSP coded circuits
Third, the trouble we had was a third party service having congestion issues. They are hosted by the same CDN as Call of Duty. The problem was both outside of our control and our third party service's control. The chokepoint was between ISPs/IXPs and the CDN. I've seen this time and time again while working at the aforementioned ISP. Saturated links on ISP/IXP/CDN networks. This is where the TSP code comes in. In this day and age of cloud services, it is financially unfeasible for every company to have a private line to every single cloud provider. That's preposterous to even suggest.
- Mike Bolitho
On Sat, Mar 14, 2020 at 10:40 AM Clayton Zekelman <clayton@mnsi.net> wrote:
The Internet is not a telecommunications service, according to your FCC. If you want predictability, buy WAN circuits, not Internet circuits.  If your provider is co-mingling Internet and WAN traffic (i.e. circuits with defined endpoints vs. public Internet or VPN), then you need to talk to them about their prioritization.
If you have mission critical applications, put them on mission critical infrastructure, not the public Internet.
Oh, that's right - Internet circuits are cheaper than WAN circuits
--
Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4
tel. 519-985-8410 fax. 519-985-8409
Folks saw congestion from a massive free content drop this past week. But as folks had called out, that was the CDN angle of distributing that content rather than the actual game play. There is a rather long discussion about that in the "akamai yesterday - what in the world was that" thread from Jan/Feb/March. Whether people end up adjusting plans around large content distribution at this time, I guess remains to be seen.
(*this is 100% my personal opinion**) Perhaps we are approaching at time when large content distribution events such as the one noted above will be configured so as to minimize any potential network effects (e.g. pushing to off-peak or applying server-side per connection rate limits). The Internet is a shared resource and as such we all share responsibility to work together and act in a way as to maintain the performance & reliability of the Internet as a whole. JL
On Mon, Mar 16, 2020 at 8:51 AM Livingood, Jason < Jason_Livingood@comcast.com> wrote:
Folks saw congestion from a massive free content drop this past week. But as folks had called out, that was the CDN angle of distributing that content rather than the actual game play. There is a rather long discussion about that in the "akamai yesterday - what in the world was that" thread from Jan/Feb/March. Whether people end up adjusting plans around large content distribution at this time, I guess remains to be seen.
(*this is 100% my personal opinion**) Perhaps we are approaching at time when large content distribution events such as the one noted above will be configured so as to minimize any potential network effects (e.g. pushing to off-peak or applying server-side per connection rate limits). The Internet is a shared resource and as such we all share responsibility to work together and act in a way as to maintain the performance & reliability of the Internet as a whole.
JL
Sadly, both marketing headlines and peering contracts / requirements reward pushing ever higher peaks instead of smartly filling troughs. CB
participants (4)
-
Ca By -
Clayton Zekelman -
Hugo Slabbert -
Livingood, Jason