Re: COVID-19 vs. our Networks
Seems arbitrary. Lots of networks have lots of Netflix/etc capacity. Who determines what is "mission critical"? Our mission as an ISP is to deliver Internet to our customers. If they want to play online games or watch video, who am I to say that isn't critical to THEIR mission? The last thing we need are a bunch of kids in quarantine that have NOTHING to do because Mike Bolitho thinks their entertainment isn't part of the "mission" of the Internet. About the only thing that might be useful is something to smooth out the big jumps in utilization on game releases - but even that is something that can be managed by adding capacity. To quote Jay Leno - Crunch All You Want, We'll Make More. At 12:16 PM 14/03/2020, Mike Bolitho wrote:
Basically that. It's probably more streaming services that could crowd out what would be considered "mission critical" infrastructure. Maybe the Netflixs and Hulus of the world will limit 4K streaming or something along those lines. Basically cap resolution to 720p for the time being.
- Mike Bolitho
 The impact of all these bored school kids on
On Sat, Mar 14, 2020 at 1:06 AM Hugo Slabbert <<mailto:hugo@slabnet.com>hugo@slabnet.com> wrote: the networks due to gaming might cause some issues. I know that if I'm working from home and my videoconferencing slows down because of someones gaming, I'm taking the necessary action (read, change some rules on my firewall).Â
People are welcome to do whatever they want on their own networks. I just didn't get the suggestion that online gaming services would shut down. Or were you saying, Mike, that online gaming would crowd out other services and so "shut down" those other services?
On Fri., Mar. 13, 2020, 21:42 Owen DeLong <<mailto:owen@delong.com>owen@delong.com> wrote: You donât have kids, do you
They have the attention span off Koi these days. Theyâll play most games for about 15 minutes or so before downloading the next one. (At least thatâs been my observation of behavior among my GFâs daughter and her friends).
Owen
On Mar 13, 2020, at 20:31 , Darin Steffl <<mailto:darin.steffl@mnwifi.com>darin.steffl@mnwifi.com> wrote:
Playing games doesn't take much bandwidth. Downloading games does. So as long as everyone already has their games and there's no updates, playing the game is typically under 100 kbps which is negligible compared to streaming video which takes 1 to 25 mbps.Â
On Fri, Mar 13, 2020, 8:52 PM Sabri Berisha <<mailto:sabri@cluecentral.net>sabri@cluecentral.net> wrote: Hi,
I don't know where y'all live, but here in the SF Bay Area, pretty much all public and private schools have closed down. My school district (in Santa Clara County) will be closed until Spring Break.
The impact of all these bored school kids on the networks due to gaming might cause some issues. I know that if I'm working from home and my videoconferencing slows down because of someones gaming, I'm taking the necessary action (read, change some rules on my firewall).Â
Thanks,
Sabri
----- On Mar 13, 2020, at 4:12 PM, Hugo Slabbert <<mailto:hugo@slabnet.com>hugo@slabnet.com> wrote: I think under circumstances like this, I could definitely see some of the online based games shutting services down.
 How so?
Signed,
Someone who works for an online gaming company and has heard nothing of this.
-- Hugo Slabbert    | email, xmpp/jabber: <mailto:hugo@slabnet.com>hugo@slabnet.com pgp key: B178313E  | also on Signal
On Fri, Mar 13, 2020 at 2:52 PM Mike Bolitho <<mailto:mikebolitho@gmail.com>mikebolitho@gmail.com> wrote: I think under circumstances like this, I could definitely see some of the online based games shutting services down.
- Mike Bolitho
On Fri, Mar 13, 2020 at 2:41 PM Ahmed Borno <<mailto:amaged@gmail.com>amaged@gmail.com> wrote: Its already happening in Italy, and now that schools are shutting down here as well, its going to get interesting:Â <https://www.bloomberg.com/news/articles/2020-03-12/housebound-italian-kids-strain-network-with-fortnite-marathon>https://www.bloomberg.com/news/articles/2020-03-12/housebound-italian-kids-strain-network-with-fortnite-marathon
The ultimate traffic test is coming, looking forward to hearing about it on this thread.
Maybe its a good time to start a communication channel between content providers/gaming companies and ISPs/CDNs.
On Fri, Mar 13, 2020 at 11:22 AM Rubens Kuhl <<mailto:rubensk@gmail.com>rubensk@gmail.com> wrote:
On Thu, Mar 12, 2020 at 3:46 PM <mailto:g@1337.io>g@1337.io <<mailto:lists@1337.io>lists@1337.io> wrote: With talk of there being an involuntary statewide (WA) and then national quarantines (house arrest) for multiple weeks, has anyone put thought into the impacts of this on your networks if/when this comes to fruition?
We're already pushing the limits with telecommuters / those that are WFH, but I can only imagine what things will look like with everyone stuck at home for any duration of time.
People will turn to you and every other ISP hoping you keep them online. So besides demand issues, keeping your network up will be important to a whole lot of people.Â
Rubens Â
-- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409
*Seems arbitrary. Lots of networks have lots of Netflix/etc capacity. Who determines what is "mission critical"? Our mission as an ISP is to deliver Internet to our customers. If they want to play online games or watch video, who am I to say that isn't critical to THEIR mission?...*
*...The last thing we need are a bunch of kids in quarantine that have
NOTHING to do because Mike Bolitho thinks their entertainment isn't part of the "mission" of the Internet.*
We already have that. It's called Telecommunications Service Priority and this is the charge: Telecommunications Service Priority (TSP) is a program that authorizes
national security and emergency preparedness (NS/EP) organizations to receive priority treatment for vital voice and data circuits or other telecommunications services.
I work for a hospital, we ran into some issues last week due to congestion that was totally outside of our control that was off of our WAN (Thanks Call Of Duty). Now, the issue we ran into was not mission critical at the time but it was still disruptive. As more and more people are driven home during this time, more and more people will be using bandwidth intensive streaming and online gaming products. If more and more TSP coded entities are running into issues, ISPs, IXPs, and CDNs will be forced to act. For more information: https://www.cisa.gov/telecommunications-service-priority-tsp https://www.fcc.gov/general/telecommunications-service-priority https://en.wikipedia.org/wiki/Telecommunications_Service_Priority *These views are my own and do not reflect the opinions or official stances of my employer etc etc.* - Mike Bolitho On Sat, Mar 14, 2020 at 9:27 AM Clayton Zekelman <clayton@mnsi.net> wrote:
Seems arbitrary. Lots of networks have lots of Netflix/etc capacity. Who determines what is "mission critical"? Our mission as an ISP is to deliver Internet to our customers. If they want to play online games or watch video, who am I to say that isn't critical to THEIR mission?
The last thing we need are a bunch of kids in quarantine that have NOTHING to do because Mike Bolitho thinks their entertainment isn't part of the "mission" of the Internet.
About the only thing that might be useful is something to smooth out the big jumps in utilization on game releases - but even that is something that can be managed by adding capacity.
To quote Jay Leno - Crunch All You Want, We'll Make More.
At 12:16 PM 14/03/2020, Mike Bolitho wrote:
Basically that. It's probably more streaming services that could crowd out what would be considered "mission critical" infrastructure. Maybe the Netflixs and Hulus of the world will limit 4K streaming or something along those lines. Basically cap resolution to 720p for the time being.
- Mike Bolitho
On Sat, Mar 14, 2020 at 1:06 AM Hugo Slabbert <hugo@slabnet.com> wrote:
 The impact of all these bored school kids on the networks due to gaming might cause some issues. I know that if I'm working from home and my videoconferencing slows down because of someones gaming, I'm taking the necessary action (read, change some rules on my firewall).Â
People are welcome to do whatever they want on their own networks. I just didn't get the suggestion that online gaming services would shut down. Or were you saying, Mike, that online gaming would crowd out other services and so "shut down" those other services?
On Fri., Mar. 13, 2020, 21:42 Owen DeLong <owen@delong.com> wrote: You don’t have kids, do you…
They have the attention span off Koi these days. They’ll play most games for about 15 minutes or so before downloading the next one. (At least that’s been my observation of behavior among my GF’s daughter and her friends).
Owen
On Mar 13, 2020, at 20:31 , Darin Steffl <darin.steffl@mnwifi.com > wrote:
Playing games doesn't take much bandwidth. Downloading games does. So as long as everyone already has their games and there's no updates, playing the game is typically under 100 kbps which is negligible compared to streaming video which takes 1 to 25 mbps.Â
On Fri, Mar 13, 2020, 8:52 PM Sabri Berisha <sabri@cluecentral.net> wrote: Hi,
I don't know where y'all live, but here in the SF Bay Area, pretty much all public and private schools have closed down. My school district (in Santa Clara County) will be closed until Spring Break.
The impact of all these bored school kids on the networks due to gaming might cause some issues. I know that if I'm working from home and my videoconferencing slows down because of someones gaming, I'm taking the necessary action (read, change some rules on my firewall).Â
Thanks,
Sabri
----- On Mar 13, 2020, at 4:12 PM, Hugo Slabbert <hugo@slabnet.com> wrote: I think under circumstances like this, I could definitely see some of the online based games shutting services down.
 How so?
Signed,
Someone who works for an online gaming company and has heard nothing of this.
-- Hugo Slabbert    | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E  | also on Signal
On Fri, Mar 13, 2020 at 2:52 PM Mike Bolitho <mikebolitho@gmail.com> wrote: I think under circumstances like this, I could definitely see some of the online based games shutting services down.
- Mike Bolitho
On Fri, Mar 13, 2020 at 2:41 PM Ahmed Borno <amaged@gmail.com> wrote: Its already happening in Italy, and now that schools are shutting down here as well, its going to get interesting:Â
https://www.bloomberg.com/news/articles/2020-03-12/housebound-italian-kids-s...
The ultimate traffic test is coming, looking forward to hearing about it on this thread.
Maybe its a good time to start a communication channel between content providers/gaming companies and ISPs/CDNs.
On Fri, Mar 13, 2020 at 11:22 AM Rubens Kuhl <rubensk@gmail.com> wrote:
On Thu, Mar 12, 2020 at 3:46 PM g@1337.io <lists@1337.io> wrote: With talk of there being an involuntary statewide (WA) and then national quarantines (house arrest) for multiple weeks, has anyone put thought into the impacts of this on your networks if/when this comes to fruition?
We're already pushing the limits with telecommuters / those that are WFH, but I can only imagine what things will look like with everyone stuck at home for any duration of time.
People will turn to you and every other ISP hoping you keep them online. So besides demand issues, keeping your network up will be important to a whole lot of people.Â
Rubens Â
--
Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409
On 14/Mar/20 19:14, Mike Bolitho wrote:
/ /
I work for a hospital, we ran into some issues last week due to congestion that was totally outside of our control that was off of our WAN (Thanks Call Of Duty). Now, the issue we ran into was not mission critical at the time but it was still disruptive. As more and more people are driven home during this time, more and more people will be using bandwidth intensive streaming and online gaming products. If more and more TSP coded entities are running into issues, ISPs, IXPs, and CDNs will be forced to act.
Hmmh, if that level of priority is required, I'd probably build my own network, and not rely on public infrastructure like the Internet. Mark.
In $dayjob I constantly see the lack of understanding of the difference between what the Internet is and what a path engineered private circuit is (eg. pseudowire, wave, whatever). The latest fight is over SD-WAN and those who think it will replace MPLS entirely and they won't need those expensive routers anymore. But I digress. Mark's comment and others like it are the correct approach Mike. If your private WAN is most critical, then invest in and manage user complaints about poor Internet service. ISP's, IXP's and CDN's are not going to twist themselves into knots to solve your problems, even if someone calls it an emergency. Sorry. Stephen On 2020-03-15 02:01, Mark Tinka wrote:
On 14/Mar/20 19:14, Mike Bolitho wrote:
/ /
I work for a hospital, we ran into some issues last week due to congestion that was totally outside of our control that was off of our WAN (Thanks Call Of Duty). Now, the issue we ran into was not mission critical at the time but it was still disruptive. As more and more people are driven home during this time, more and more people will be using bandwidth intensive streaming and online gaming products. If more and more TSP coded entities are running into issues, ISPs, IXPs, and CDNs will be forced to act.
Hmmh, if that level of priority is required, I'd probably build my own network, and not rely on public infrastructure like the Internet.
Mark.
I think there's a bit of a misunderstanding of what I'm trying to say here. We have dual private lines from two Tier I providers. These interconnect all major hospitals and our data centers. We also have a third metro connection that connects things regionally. We have DIA on top of that. I think people are vastly underestimating just how much $aaS there is within the medical field. TeleDoc, translation services, remote radiologists, the way prescriptions get filled, how staffing works, third party providers basically hoteling within our facilities, critical staff VPNed in because the government has locked things down, etc. Then there's things that we don't use but I'm sure other providers do, GoToMeeting, O365, VaaS, etc. There's no practical way to engineer your WAN to facilitate dozens of connections to these services. This extends beyond just hospitals as well. Fire departments, police departments, water treatment etc. Regardless of whether or not those entities planned well (I think we did), the government should and will step in if critical services are degraded. And for what it's worth, Stephen, I know how things are built within the ISP world. I spent four years there. That doesn't change the fact that we're possibly heading into uncharted waters when it comes to utilization and the impact that will have on $aaS products that are interwoven into every single vertical, including entities that fall under TSP, critical national security and emergency preparedness functions, including those areas related to safety, maintenance of law and order, and public health. It's easy for all you guys to sit here and armchair quarterback other people's planning but when things really start to degrade, all bets are off. If you don't believe that, just look at the news. States are literally shutting down private businesses (restaurants, bars, night clubs, private schools) and banning people from associating in groups of larger than 50. *The opinions expressed here are my own and do not represent my employer or their views.* - Mike Bolitho On Sun, Mar 15, 2020 at 6:12 PM Stephen Fulton <sf@lists.esoteric.ca> wrote:
In $dayjob I constantly see the lack of understanding of the difference between what the Internet is and what a path engineered private circuit is (eg. pseudowire, wave, whatever). The latest fight is over SD-WAN and those who think it will replace MPLS entirely and they won't need those expensive routers anymore. But I digress.
Mark's comment and others like it are the correct approach Mike. If your private WAN is most critical, then invest in and manage user complaints about poor Internet service. ISP's, IXP's and CDN's are not going to twist themselves into knots to solve your problems, even if someone calls it an emergency. Sorry.
Stephen
On 2020-03-15 02:01, Mark Tinka wrote:
On 14/Mar/20 19:14, Mike Bolitho wrote:
/ /
I work for a hospital, we ran into some issues last week due to congestion that was totally outside of our control that was off of our WAN (Thanks Call Of Duty). Now, the issue we ran into was not mission critical at the time but it was still disruptive. As more and more people are driven home during this time, more and more people will be using bandwidth intensive streaming and online gaming products. If more and more TSP coded entities are running into issues, ISPs, IXPs, and CDNs will be forced to act.
Hmmh, if that level of priority is required, I'd probably build my own network, and not rely on public infrastructure like the Internet.
Mark.
On 2020-03-16, at 15:40, Mike Bolitho <mikebolitho@gmail.com> wrote:
I think people are vastly underestimating just how much $aaS there is within the medical field.
I recently had to reschedule an X-ray because the license manager for the X-ray machine was acting up. I don’t think people have a grasp for how much of the medical infrastructure no longer works when the Internet is down. Grüße, Carsten
On 16/Mar/20 16:54, Carsten Bormann wrote:
I recently had to reschedule an X-ray because the license manager for the X-ray machine was acting up. I don’t think people have a grasp for how much of the medical infrastructure no longer works when the Internet is down.
I get this, to some extent. But also, there is a reason hospitals, airports and military installations are either put on special power grids or invest plenty of money in backup power. If an x-ray machine won't work because the Internet is down, I'm not sure that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me. Mark.
If an x-ray machine won't work because the Internet is down, I'm not sure that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Totally agree with you. Unfortunately it's not a problem with the medical providers, it's a problem with the medical devices. Anybody who works in the healthcare vertical will tell you just how bad medical devices are to work with from an IT perspective. And that is part of my original comments. In your case, I am not sure I have an answer for you, unfortunately. The
public Internet is what it is, mostly best-effort. Your applications and use-cases certainly deserve better than that. I'm not sure how to achieve that as your industry shoves more and more activity into the public Internet domain, for one reason or another.
I don't know what it's going to take either. A general shift in mentality from the vendors we use I guess. I'm not sure how you get a bunch of medical providers to tell these companies they need to fix their stuff. You can't exactly use your wallet to force change either. There are only a handful of vendor options out there so there isn't a ton of choice. It's not like you can buy one of 50 different models of CT machines or EHR systems. Generally speaking it's not an issue. It's just in crazy times like these where, if congestion on the public internet gets too crazy, that certain platforms might need to be deemed "unnecessary". Is playing Fortnight a right? Is streaming a movie in 4K a right? In cases like San Francisco they have decided that leaving your home for anything other than work or medical care is no longer a right because you're now infringing on other's rights by potentially getting them sick. Maybe 4K Netflix fits into that category if you're causing problems for first responders and hospitals trying to save lives. - Mike Bolitho On Tue, Mar 17, 2020 at 2:22 AM Mark Tinka <mark.tinka@seacom.mu> wrote:
On 16/Mar/20 16:54, Carsten Bormann wrote:
I recently had to reschedule an X-ray because the license manager for the X-ray machine was acting up. I don’t think people have a grasp for how much of the medical infrastructure no longer works when the Internet is down.
I get this, to some extent. But also, there is a reason hospitals, airports and military installations are either put on special power grids or invest plenty of money in backup power.
If an x-ray machine won't work because the Internet is down, I'm not sure that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Mark.
On 17/Mar/20 17:38, Mike Bolitho wrote:
Totally agree with you. Unfortunately it's not a problem with the medical providers, it's a problem with the medical devices. Anybody who works in the healthcare vertical will tell you just how bad medical devices are to work with from an IT perspective. And that is part of my original comments.
I guess that means they don't support IPv6 :-)?
I don't know what it's going to take either. A general shift in mentality from the vendors we use I guess. I'm not sure how you get a bunch of medical providers to tell these companies they need to fix their stuff. You can't exactly use your wallet to force change either. There are only a handful of vendor options out there so there isn't a ton of choice. It's not like you can buy one of 50 different models of CT machines or EHR systems.
Ah, so equipment vendors are simply rolling out kit with an IP stack, without a care of how the hospitals will actually operate them on the Internet? Tick-in-the-box, type-thing :-)? Much like how gaming producers write code so that updates are whole blobs rather than incremental changes, without a care for the network operators/customers, because it's just easier? Or like how CPE manufacturers ship hardware with hard-coded DNS settings to make provisioning as zero-touch as possible. Or like how... I'd say someone should spend some time sensitizing the medical equipment OEM's about their potential impact on/by the Internet, but something tells me they won't care, nor will the doctors/hospitals they market to.
Generally speaking it's not an issue. It's just in crazy times like these where, if congestion on the public internet gets too crazy, that certain platforms might need to be deemed "unnecessary". Is playing Fortnight a right? Is streaming a movie in 4K a right? In cases like San Francisco they have decided that leaving your home for anything other than work or medical care is no longer a right because you're now infringing on other's rights by potentially getting them sick. Maybe 4K Netflix fits into that category if you're causing problems for first responders and hospitals trying to save lives.
The difference between the SFO gubbermint and the ISP's that operate around the world is one of governance scope. A city gubbermint may be able to impose rules and laws against its citizens. Whether they can do that to an ISP, especially an ISP that either is based out of state or out of the country, is where the issue lies. But even before all that - if an ISP's raison d'être is to deliver 4K Netflix to its users, and they pay their good money to vendors and providers to achieve this, who are we to tell them their business is deemed "unnecessary"? Mark.
The answer is don't shove application traffic that has tight service level requirements onto the public internet at large and expect the same performance as private circuits or other SLA protected services. On Tue, Mar 17, 2020 at 11:40 AM Mike Bolitho <mikebolitho@gmail.com> wrote:
If an x-ray machine won't work because the Internet is down, I'm not sure
that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Totally agree with you. Unfortunately it's not a problem with the medical providers, it's a problem with the medical devices. Anybody who works in the healthcare vertical will tell you just how bad medical devices are to work with from an IT perspective. And that is part of my original comments.
In your case, I am not sure I have an answer for you, unfortunately. The
public Internet is what it is, mostly best-effort. Your applications and use-cases certainly deserve better than that. I'm not sure how to achieve that as your industry shoves more and more activity into the public Internet domain, for one reason or another.
I don't know what it's going to take either. A general shift in mentality from the vendors we use I guess. I'm not sure how you get a bunch of medical providers to tell these companies they need to fix their stuff. You can't exactly use your wallet to force change either. There are only a handful of vendor options out there so there isn't a ton of choice. It's not like you can buy one of 50 different models of CT machines or EHR systems.
Generally speaking it's not an issue. It's just in crazy times like these where, if congestion on the public internet gets too crazy, that certain platforms might need to be deemed "unnecessary". Is playing Fortnight a right? Is streaming a movie in 4K a right? In cases like San Francisco they have decided that leaving your home for anything other than work or medical care is no longer a right because you're now infringing on other's rights by potentially getting them sick. Maybe 4K Netflix fits into that category if you're causing problems for first responders and hospitals trying to save lives.
- Mike Bolitho
On Tue, Mar 17, 2020 at 2:22 AM Mark Tinka <mark.tinka@seacom.mu> wrote:
On 16/Mar/20 16:54, Carsten Bormann wrote:
I recently had to reschedule an X-ray because the license manager for the X-ray machine was acting up. I don’t think people have a grasp for how much of the medical infrastructure no longer works when the Internet is down.
I get this, to some extent. But also, there is a reason hospitals, airports and military installations are either put on special power grids or invest plenty of money in backup power.
If an x-ray machine won't work because the Internet is down, I'm not sure that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Mark.
The answer is don't shove application traffic that has tight service level requirements onto the public internet at large and expect the same performance as private circuits or other SLA protected services.
I keep seeing this over and over again in this long thread. What's your suggestion? How does a hospital, with dozens of third party applications/devices across multiple cloud platforms do this? We have two redundant private lines out of each hospital connecting back to primary and DR DCs and a metro connecting everything together in each region. But for things we do not own that are not hosted locally, what are we supposed to do? We have to go out DIA to get there. Everything we own is connected via fully SLAed private lines. We have zero issues there. I think people vastly underestimate just how much in the healthcare vertical is outside of a medical providers control/ownership. - Mike Bolitho On Tue, Mar 17, 2020 at 9:54 AM Tom Beecher <beecher@beecher.cc> wrote:
The answer is don't shove application traffic that has tight service level requirements onto the public internet at large and expect the same performance as private circuits or other SLA protected services.
On Tue, Mar 17, 2020 at 11:40 AM Mike Bolitho <mikebolitho@gmail.com> wrote:
If an x-ray machine won't work because the Internet is down, I'm not sure
that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Totally agree with you. Unfortunately it's not a problem with the medical providers, it's a problem with the medical devices. Anybody who works in the healthcare vertical will tell you just how bad medical devices are to work with from an IT perspective. And that is part of my original comments.
In your case, I am not sure I have an answer for you, unfortunately. The
public Internet is what it is, mostly best-effort. Your applications and use-cases certainly deserve better than that. I'm not sure how to achieve that as your industry shoves more and more activity into the public Internet domain, for one reason or another.
I don't know what it's going to take either. A general shift in mentality from the vendors we use I guess. I'm not sure how you get a bunch of medical providers to tell these companies they need to fix their stuff. You can't exactly use your wallet to force change either. There are only a handful of vendor options out there so there isn't a ton of choice. It's not like you can buy one of 50 different models of CT machines or EHR systems.
Generally speaking it's not an issue. It's just in crazy times like these where, if congestion on the public internet gets too crazy, that certain platforms might need to be deemed "unnecessary". Is playing Fortnight a right? Is streaming a movie in 4K a right? In cases like San Francisco they have decided that leaving your home for anything other than work or medical care is no longer a right because you're now infringing on other's rights by potentially getting them sick. Maybe 4K Netflix fits into that category if you're causing problems for first responders and hospitals trying to save lives.
- Mike Bolitho
On Tue, Mar 17, 2020 at 2:22 AM Mark Tinka <mark.tinka@seacom.mu> wrote:
On 16/Mar/20 16:54, Carsten Bormann wrote:
I recently had to reschedule an X-ray because the license manager for the X-ray machine was acting up. I don’t think people have a grasp for how much of the medical infrastructure no longer works when the Internet is down.
I get this, to some extent. But also, there is a reason hospitals, airports and military installations are either put on special power grids or invest plenty of money in backup power.
If an x-ray machine won't work because the Internet is down, I'm not sure that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Mark.
You're facing essentially the same issue as many in non-healthcare do ; how to best talk to applications in Magic Cloud Land. Reaching the major cloud providers does not require DIA ; they all have presences on the major IXes, and direct peering could be an option too depending on your needs and traffic. I don't mean to be dismissive of the issues you face, I apologize if that's how it comes off. What you describe is certainly challenging, but I think that you will have better success with some of the options that are out there already than hoping for any resolution of intermittent congestion issues in the wild west of the DFZ. On Tue, Mar 17, 2020 at 1:03 PM Mike Bolitho <mikebolitho@gmail.com> wrote:
The answer is don't shove application traffic that has tight service level requirements onto the public internet at large and expect the same performance as private circuits or other SLA protected services.
I keep seeing this over and over again in this long thread. What's your suggestion? How does a hospital, with dozens of third party applications/devices across multiple cloud platforms do this?
We have two redundant private lines out of each hospital connecting back to primary and DR DCs and a metro connecting everything together in each region. But for things we do not own that are not hosted locally, what are we supposed to do? We have to go out DIA to get there. Everything we own is connected via fully SLAed private lines. We have zero issues there. I think people vastly underestimate just how much in the healthcare vertical is outside of a medical providers control/ownership.
- Mike Bolitho
On Tue, Mar 17, 2020 at 9:54 AM Tom Beecher <beecher@beecher.cc> wrote:
The answer is don't shove application traffic that has tight service level requirements onto the public internet at large and expect the same performance as private circuits or other SLA protected services.
On Tue, Mar 17, 2020 at 11:40 AM Mike Bolitho <mikebolitho@gmail.com> wrote:
If an x-ray machine won't work because the Internet is down, I'm not sure
that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Totally agree with you. Unfortunately it's not a problem with the medical providers, it's a problem with the medical devices. Anybody who works in the healthcare vertical will tell you just how bad medical devices are to work with from an IT perspective. And that is part of my original comments.
In your case, I am not sure I have an answer for you, unfortunately. The
public Internet is what it is, mostly best-effort. Your applications and use-cases certainly deserve better than that. I'm not sure how to achieve that as your industry shoves more and more activity into the public Internet domain, for one reason or another.
I don't know what it's going to take either. A general shift in mentality from the vendors we use I guess. I'm not sure how you get a bunch of medical providers to tell these companies they need to fix their stuff. You can't exactly use your wallet to force change either. There are only a handful of vendor options out there so there isn't a ton of choice. It's not like you can buy one of 50 different models of CT machines or EHR systems.
Generally speaking it's not an issue. It's just in crazy times like these where, if congestion on the public internet gets too crazy, that certain platforms might need to be deemed "unnecessary". Is playing Fortnight a right? Is streaming a movie in 4K a right? In cases like San Francisco they have decided that leaving your home for anything other than work or medical care is no longer a right because you're now infringing on other's rights by potentially getting them sick. Maybe 4K Netflix fits into that category if you're causing problems for first responders and hospitals trying to save lives.
- Mike Bolitho
On Tue, Mar 17, 2020 at 2:22 AM Mark Tinka <mark.tinka@seacom.mu> wrote:
On 16/Mar/20 16:54, Carsten Bormann wrote:
I recently had to reschedule an X-ray because the license manager for the X-ray machine was acting up. I don’t think people have a grasp for how much of the medical infrastructure no longer works when the Internet is down.
I get this, to some extent. But also, there is a reason hospitals, airports and military installations are either put on special power grids or invest plenty of money in backup power.
If an x-ray machine won't work because the Internet is down, I'm not sure that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Mark.
You're facing essentially the same issue as many in non-healthcare do ; how to best talk to applications in Magic Cloud Land. Reaching the major cloud providers does not require DIA ; they all have presences on the major IXes, and direct peering could be an option too depending on your needs and traffic.
I totally agree and 99.999% of the time, congestion on the Internet is a nuisance, not a critical problem. I'm not sitting here complaining that my public internet circuits don't have SLAs or that we run into some packet loss and latency here and there under normal operations. That's obviously to be expected. But this whole topic is around what to do when a once in a lifetime pandemic hits and we're faced with unseen levels of congestion across the country's infrastructure. I mean the thread is titled COVID-19 Vs Our Networks. That's why I brought up the possible application of TSP to tell some of the big CDNs that maybe they should limit 4K streaming or big DLCs during a pandemic. That's it. And yet I'm getting chastised (not necessarily by you) for suggesting that hospitals, governments, water treatment plants, power plants, first responders, etc are actually more important during times like this. - Mike Bolitho On Tue, Mar 17, 2020 at 10:35 AM Tom Beecher <beecher@beecher.cc> wrote:
You're facing essentially the same issue as many in non-healthcare do ; how to best talk to applications in Magic Cloud Land. Reaching the major cloud providers does not require DIA ; they all have presences on the major IXes, and direct peering could be an option too depending on your needs and traffic.
I don't mean to be dismissive of the issues you face, I apologize if that's how it comes off. What you describe is certainly challenging, but I think that you will have better success with some of the options that are out there already than hoping for any resolution of intermittent congestion issues in the wild west of the DFZ.
On Tue, Mar 17, 2020 at 1:03 PM Mike Bolitho <mikebolitho@gmail.com> wrote:
The answer is don't shove application traffic that has tight service level requirements onto the public internet at large and expect the same performance as private circuits or other SLA protected services.
I keep seeing this over and over again in this long thread. What's your suggestion? How does a hospital, with dozens of third party applications/devices across multiple cloud platforms do this?
We have two redundant private lines out of each hospital connecting back to primary and DR DCs and a metro connecting everything together in each region. But for things we do not own that are not hosted locally, what are we supposed to do? We have to go out DIA to get there. Everything we own is connected via fully SLAed private lines. We have zero issues there. I think people vastly underestimate just how much in the healthcare vertical is outside of a medical providers control/ownership.
- Mike Bolitho
On Tue, Mar 17, 2020 at 9:54 AM Tom Beecher <beecher@beecher.cc> wrote:
The answer is don't shove application traffic that has tight service level requirements onto the public internet at large and expect the same performance as private circuits or other SLA protected services.
On Tue, Mar 17, 2020 at 11:40 AM Mike Bolitho <mikebolitho@gmail.com> wrote:
If an x-ray machine won't work because the Internet is down, I'm not sure
that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Totally agree with you. Unfortunately it's not a problem with the medical providers, it's a problem with the medical devices. Anybody who works in the healthcare vertical will tell you just how bad medical devices are to work with from an IT perspective. And that is part of my original comments.
In your case, I am not sure I have an answer for you, unfortunately.
The public Internet is what it is, mostly best-effort. Your applications and use-cases certainly deserve better than that. I'm not sure how to achieve that as your industry shoves more and more activity into the public Internet domain, for one reason or another.
I don't know what it's going to take either. A general shift in mentality from the vendors we use I guess. I'm not sure how you get a bunch of medical providers to tell these companies they need to fix their stuff. You can't exactly use your wallet to force change either. There are only a handful of vendor options out there so there isn't a ton of choice. It's not like you can buy one of 50 different models of CT machines or EHR systems.
Generally speaking it's not an issue. It's just in crazy times like these where, if congestion on the public internet gets too crazy, that certain platforms might need to be deemed "unnecessary". Is playing Fortnight a right? Is streaming a movie in 4K a right? In cases like San Francisco they have decided that leaving your home for anything other than work or medical care is no longer a right because you're now infringing on other's rights by potentially getting them sick. Maybe 4K Netflix fits into that category if you're causing problems for first responders and hospitals trying to save lives.
- Mike Bolitho
On Tue, Mar 17, 2020 at 2:22 AM Mark Tinka <mark.tinka@seacom.mu> wrote:
On 16/Mar/20 16:54, Carsten Bormann wrote:
I recently had to reschedule an X-ray because the license manager for the X-ray machine was acting up. I don’t think people have a grasp for how much of the medical infrastructure no longer works when the Internet is down.
I get this, to some extent. But also, there is a reason hospitals, airports and military installations are either put on special power grids or invest plenty of money in backup power.
If an x-ray machine won't work because the Internet is down, I'm not sure that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Mark.
On 17/Mar/20 19:46, Mike Bolitho wrote:
I totally agree and 99.999% of the time, congestion on the Internet is a nuisance, not a critical problem. I'm not sitting here complaining that my public internet circuits don't have SLAs or that we run into some packet loss and latency here and there under normal operations. That's obviously to be expected. But this whole topic is around what to do when a once in a lifetime pandemic hits and we're faced with unseen levels of congestion across the country's infrastructure. I mean the thread is titled COVID-19 Vs Our Networks. That's why I brought up the possible application of TSP to tell some of the big CDNs that maybe they should limit 4K streaming or big DLCs during a pandemic. That's it. And yet I'm getting chastised (not necessarily by you) for suggesting that hospitals, governments, water treatment plants, power plants, first responders, etc are actually more important during times like this.
To me, sounds like a potential business case for an existing or new CDN provider focused squarely on healthcare, and other such critical services :-). As is always the case with invention, "I didn't like what I found, so I built a better one". Mark.
On Tue, Mar 17, 2020 at 10:52 AM Mike Bolitho <mikebolitho@gmail.com> wrote:
You're facing essentially the same issue as many in non-healthcare do ; how to best talk to applications in Magic Cloud Land. Reaching the major cloud providers does not require DIA ; they all have presences on the major IXes, and direct peering could be an option too depending on your needs and traffic.
I totally agree and 99.999% of the time, congestion on the Internet is a nuisance, not a critical problem. I'm not sitting here complaining that my public internet circuits don't have SLAs or that we run into some packet loss and latency here and there under normal operations. That's obviously to be expected. But this whole topic is around what to do when a once in a lifetime pandemic hits and we're faced with unseen levels of congestion across the country's infrastructure. I mean the thread is titled COVID-19 Vs Our Networks. That's why I brought up the possible application of TSP to tell some of the big CDNs that maybe they should limit 4K streaming or big DLCs during a pandemic. That's it. And yet I'm getting chastised (not necessarily by you) for suggesting that hospitals, governments, water treatment plants, power plants, first responders, etc are actually more important during times like this.
- Mike Bolitho
I think it's time to re-stock on the "The Cloud Is Just Someone Else's Computer In A Different Building" stickers... While having streaming services voluntarily ratchet their bitrates down during the crisis is a nice enough response, I think the deeper underlying issue is that any system that is CRITICAL for maintaining health and safety during a pandemic or other crisis MUST be capable of operating standalone in case the rest of the infrastructure has melted down. X-Ray systems at hospitals that refuse to work when they can't talk to a license server in the cloud? Nope. If there's government intervention and regulation that comes out of this, it should focus not on TSP responses during a crisis, but on ensuring that manufacturers of healthcare devices do not prioritize making money over saving lives. IF there is regulation to be made after this, THAT is what it needs to focus on. Internet congestion is a symptom, not the cause of this thread. Fix the real problem. CRITICAL health care systems must be capable of operating on their own during a state of emergency, not held captive to the profit motives of rich executives. :/ Matt who finds it appalling that we consider it more important to make money than to save lives. :(
On Tue, Mar 17, 2020 at 10:35 AM Tom Beecher <beecher@beecher.cc> wrote:
You're facing essentially the same issue as many in non-healthcare do ; how to best talk to applications in Magic Cloud Land. Reaching the major cloud providers does not require DIA ; they all have presences on the major IXes, and direct peering could be an option too depending on your needs and traffic.
I don't mean to be dismissive of the issues you face, I apologize if that's how it comes off. What you describe is certainly challenging, but I think that you will have better success with some of the options that are out there already than hoping for any resolution of intermittent congestion issues in the wild west of the DFZ.
Internet congestion is a symptom, not the cause of this thread.
[JL] I'm wondering if one of the issues is problems with legacy TCP congestion control algorithms. The industry has been poking at that for awhile and approaches range from BBR to fq_codel. This is worth exploring a bit more IMO.
On Sat, 21 Mar 2020 at 00:15, Matthew Petach <mpetach@netflight.com> wrote:
who finds it appalling that we consider it more important to make money than to save lives. :(
If we (me included) would be half as angry about those who have less than we have, as we are about those who have more than we have, inequality wouldn't exist. We are the beneficiaries of immense suffering of millions of people, things are artificially cheap for us to buy at human cost 'somewhere else'. We are not the heroes of this story. -- ++ytti
On 21/Mar/20 17:53, Saku Ytti wrote:
If we (me included) would be half as angry about those who have less than we have, as we are about those who have more than we have, inequality wouldn't exist. We are the beneficiaries of immense suffering of millions of people, things are artificially cheap for us to buy at human cost 'somewhere else'. We are not the heroes of this story.
I clean my pool every morning. When I am holding the rod closer to the leaf basket, I use less energy. When I am holding the rod farther from the leaf basket, not only do I use more energy, but I have less control of the guidance of the rod and basket, as it resists the water. The businesses that will succeed in this new digital economy will be those that empathize with customers, listen to them, engage them, and provide them with value, regardless of their economic (mis)fortunes. Those businesses that continue to push product and detach themselves from how customers want to engage with them will quickly become irrelevant. In the past, countries were set apart by their physical infrastructure; those that had better infrastructure had a higher chance of succeeding relative to those that didn't. Today (and in the future), regardless of the tangible infrastructure one country has vs. another, the Internet is the single most important thing that levels the playing field. Today, a kid in Kigali has about the same opportunities as a kid in San Francisco, to harness the Internet in order to make each of their lives better. The Internet in Australia is exactly the same as the Internet in Sao Paulo. Mark.
On 17/Mar/20 19:35, Tom Beecher wrote:
You're facing essentially the same issue as many in non-healthcare do ; how to best talk to applications in Magic Cloud Land. Reaching the major cloud providers does not require DIA ; they all have presences on the major IXes, and direct peering could be an option too depending on your needs and traffic.
I don't mean to be dismissive of the issues you face, I apologize if that's how it comes off. What you describe is certainly challenging, but I think that you will have better success with some of the options that are out there already than hoping for any resolution of intermittent congestion issues in the wild west of the DFZ.
Sounds like a use-case for the cloud providers' so-called "Express Route" services. But that will only be really successful if the majority of the services that the hospitals need are hosted on the cloud platforms that offer these Express Route things. Then again, the "private" link between a cloud provider and their customer is typically an MPLS-based one, which is subject to emergent issues that may impact the operators' backbone. If the service is not on a cloud provider (or one that can offer an Express Route thing), then we're back to square one. Mark.
On Mar 17, 2020, at 10:03 , Mike Bolitho <mikebolitho@gmail.com> wrote:
The answer is don't shove application traffic that has tight service level requirements onto the public internet at large and expect the same performance as private circuits or other SLA protected services.
I keep seeing this over and over again in this long thread. What's your suggestion? How does a hospital, with dozens of third party applications/devices across multiple cloud platforms do this?
Step one: Consumers _AND_ especially mission critical consumers must start refusing to purchase devices which have inherent dependency on a vendor-cloud (or any cloud for that matter).
We have two redundant private lines out of each hospital connecting back to primary and DR DCs and a metro connecting everything together in each region. But for things we do not own that are not hosted locally, what are we supposed to do? We have to go out DIA to get there. Everything we own is connected via fully SLAed private lines. We have zero issues there. I think people vastly underestimate just how much in the healthcare vertical is outside of a medical providers control/ownership.
Stop treating things you don’t own and things that aren’t hosted locally as “reliable” and make sure that they are not in the mission critical chain of urgent patient care. Anything in the healthcare vertical that is outside of the medical providers control/ownership is a result of the medical provider buying into that model on some level. STOP DOING THAT. (How am I suddenly reminded of the old adage “Doctor, doctor, it hurts when I do this!”…) I understand how the allure of lower costs and the frustration of “every vendor does this, we can’t find one who doesn’t” plays out. However, the only way “every vendor does it” will continue is if every vendor continues to be able to make sales without changing. Owen
- Mike Bolitho
On Tue, Mar 17, 2020 at 9:54 AM Tom Beecher <beecher@beecher.cc> wrote: The answer is don't shove application traffic that has tight service level requirements onto the public internet at large and expect the same performance as private circuits or other SLA protected services.
On Tue, Mar 17, 2020 at 11:40 AM Mike Bolitho <mikebolitho@gmail.com <mailto:mikebolitho@gmail.com>> wrote: If an x-ray machine won't work because the Internet is down, I'm not sure that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Totally agree with you. Unfortunately it's not a problem with the medical providers, it's a problem with the medical devices. Anybody who works in the healthcare vertical will tell you just how bad medical devices are to work with from an IT perspective. And that is part of my original comments.
In your case, I am not sure I have an answer for you, unfortunately. The public Internet is what it is, mostly best-effort. Your applications and use-cases certainly deserve better than that. I'm not sure how to achieve that as your industry shoves more and more activity into the public Internet domain, for one reason or another.
I don't know what it's going to take either. A general shift in mentality from the vendors we use I guess. I'm not sure how you get a bunch of medical providers to tell these companies they need to fix their stuff. You can't exactly use your wallet to force change either. There are only a handful of vendor options out there so there isn't a ton of choice. It's not like you can buy one of 50 different models of CT machines or EHR systems.
Generally speaking it's not an issue. It's just in crazy times like these where, if congestion on the public internet gets too crazy, that certain platforms might need to be deemed "unnecessary". Is playing Fortnight a right? Is streaming a movie in 4K a right? In cases like San Francisco they have decided that leaving your home for anything other than work or medical care is no longer a right because you're now infringing on other's rights by potentially getting them sick. Maybe 4K Netflix fits into that category if you're causing problems for first responders and hospitals trying to save lives.
- Mike Bolitho
On Tue, Mar 17, 2020 at 2:22 AM Mark Tinka <mark.tinka@seacom.mu <mailto:mark.tinka@seacom.mu>> wrote:
On 16/Mar/20 16:54, Carsten Bormann wrote:
I recently had to reschedule an X-ray because the license manager for the X-ray machine was acting up. I don’t think people have a grasp for how much of the medical infrastructure no longer works when the Internet is down.
I get this, to some extent. But also, there is a reason hospitals, airports and military installations are either put on special power grids or invest plenty of money in backup power.
If an x-ray machine won't work because the Internet is down, I'm not sure that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Mark.
On Tue, Mar 17, 2020 at 11:35:59AM -0700, Owen DeLong wrote:
Anything in the healthcare vertical that is outside of the medical providers control/ownership is a result of the medical provider buying into that model on some level. STOP DOING THAT. (How am I suddenly reminded of the old adage ???Doctor, doctor, it hurts when I do this!??????)
I understand how the allure of lower costs and the frustration of ???every vendor does this, we can???t find one who doesn???t??? plays out. However, the only way ???every vendor does it??? will continue is if every vendor continues to be able to make sales without changing.
Fought this battle, lost this battle. Why? Because the people with the authority to make purchasing decisions are not the people who will be on the phone to some vendor's tech support at 3 AM on a Sunday morning, frantically pleading with them to fix a problem because they really need that piece of equipment to work right now. Decisions are no longer based on the greater good or on anticipating worst case scenarios or on maximizing preparedness or anything that we might hope they're based on. They're based, coldly and calculatingly, on money. If you want this to change -- and I sure would like it to change -- then money needs to be entirely removed from that calculation. That is a problem whose solution lies outside the scope of NANOG. Meanwhile, I've updated this: Covid19 http://www.firemountain.net/covid19.html to include some more resources, including CORD-19, which compiles tens of thousands of papers on the virus in one place. I've also included a link to the relevant Folding@Home project -- which could probably use as much CPU as you can throw at it. ---rsk
On Tuesday, 17 March, 2020 15:48, Rich Kulawiec <rsk@gsp.org> wrote:
On Tue, Mar 17, 2020 at 11:35:59AM -0700, Owen DeLong wrote:
Anything in the healthcare vertical that is outside of the medical providers control/ownership is a result of the medical provider buying into that model on some level. STOP DOING THAT. (How am I suddenly reminded of the old adage ???Doctor, doctor, it hurts when I do this!??????)
I understand how the allure of lower costs and the frustration of ???every vendor does this, we can???t find one who doesn???t??? plays out.
However, the only way ???every vendor does it??? will continue is if every vendor continues to be able to make sales without changing.
Fought this battle, lost this battle.
Why?
Because the people with the authority to make purchasing decisions are not the people who will be on the phone to some vendor's tech support at 3 AM on a Sunday morning, frantically pleading with them to fix a problem because they really need that piece of equipment to work right now.
So you failed because you did not require the person making the decision to take responsibility for their decision. That is, your organization has a severely flawed process wherein the "R" for making the decision is not the same person as has the "R" for the repercussions.
Decisions are no longer based on the greater good or on anticipating worst case scenarios or on maximizing preparedness or anything that we might hope they're based on. They're based, coldly and calculatingly, on money.
No, they are based on whatever the specification for making decisions happens to be. If you have chosen that basis to be "cheapest bidder", then that is what you can expect to receive.
If you want this to change -- and I sure would like it to change -- then money needs to be entirely removed from that calculation. That is a problem whose solution lies outside the scope of NANOG.
No. One simply has to assign a "cost" to "suitability for use". For example, if you put out an RFQ for a CT Machine and someone bids a bag of peanuts for $1.50, that is probably the lowest bid, and that is what you will get if you choose based entirely on the lowest bid. However, if you also require that the purchased machine also actually be capable of performing Computed Tomography then clearly that $1.50 bid will be rejected. You simply have to define what you want to achieve, then do it. -- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.
On Wed, Mar 18, 2020 at 03:43:37AM -0600, Keith Medcalf wrote:
So you failed because you did not require the person making the decision to take responsibility for their decision. That is, your organization has a severely flawed process wherein the "R" for making the decision is not the same person as has the "R" for the repercussions.
The use of "you/your" here and throughout is misplaced and inappropriate. Also: this not an isolated or unique experience. It's this way pretty much everywhere in the US now. And I can disapprove of it, you can disapprove of it, we can all disapprove of it, but like I said, until money is completely removed from the calculation, this is how it will be. Critiques of process and role and organization and everything else are interesting, maybe even correct -- but will change nothing. ---rsk
On Wednesday, 18 March, 2020 05:24, Rich Kulawiec <rsk@gsp.org> wrote:
On Wed, Mar 18, 2020 at 03:43:37AM -0600, Keith Medcalf wrote:
So you failed because you did not require the person making the decision to take responsibility for their decision. That is, your organization has a severely flawed process wherein the "R" for making the decision is not the same person as has the "R" for the repercussions.
The use of "you/your" here and throughout is misplaced and inappropriate.
It is the "Royal You". However, you can replace that with generics if y'all wish. The point is that the root of the problem is the failure of the organizational decision maker to take responsibility for their decision. As a business person (now retired) once told me about the things he sells in his shop, "I will not sell that here because in my opinion it is crap. If you want that, you can go to the shop next door. They will be quite willing to sell that crap to you, but don't come complaining to me when your failure to take my advice comes back to bite you in the ass."
Also: this not an isolated or unique experience. It's this way pretty much everywhere in the US now. And I can disapprove of it, you can disapprove of it, we can all disapprove of it, but like I said, until money is completely removed from the calculation, this is how it will be. Critiques of process and role and organization and everything else are interesting, maybe even correct -- but will change nothing.
Yes, it is generally an USian problem. While I cannot speak to its prevelance in the US I can attest to the fact that USians try to bring this philosophy with them were ever they go and that such thinking has to be repelled with large bats. I have had to deal with such things several times and my response is quite simple: My name will not be associated in any way with that stupidity other than complete opposition to it. If you want me to sign off on it, then I will not. And if you decide to do it anyway then do not ask me to have anything to do with the mess that ensues because the only action you will get from me is "told you so -- you made your bed now go sleep in it". Generally the encroachment of ill-conceived plans is staved off until the resistant retire leaving the inmates in charge of the asylum. -- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.
On 18/Mar/20 17:40, Keith Medcalf wrote:
Yes, it is generally an USian problem. While I cannot speak to its prevelance in the US I can attest to the fact that USians try to bring this philosophy with them were ever they go and that such thinking has to be repelled with large bats.
No country, business or business model is immune, anywhere around the world. In this new economy, your competitor is anyone, anywhere, with an idea and an Internet connection. So take heart - this won't be unique to America :-).
I have had to deal with such things several times and my response is quite simple: My name will not be associated in any way with that stupidity other than complete opposition to it. If you want me to sign off on it, then I will not. And if you decide to do it anyway then do not ask me to have anything to do with the mess that ensues because the only action you will get from me is "told you so -- you made your bed now go sleep in it".
Generally the encroachment of ill-conceived plans is staved off until the resistant retire leaving the inmates in charge of the asylum.
The kids today don't care about your job title, how many degrees, masters or PhD's you obtained to get there, the name of your company (unless it rhymes with an app they adore), if you or your company are famous, or whether you make thousands, millions or billions. All they care about is if you give them value. Businesses with product-based models that try to stay relevant through cost-cutting and "focused sales" will be undone by unassuming "pretenders" who know how to harness technology to deliver value to a customer that lives thousands of miles away, in another land; the kinds of "pretenders" who are not interested in stamping their authority or superiority on a market. The world has changed. Adapt or die :-). Mark.
On 18/Mar/20 13:24, Rich Kulawiec wrote:
The use of "you/your" here and throughout is misplaced and inappropriate.
Also: this not an isolated or unique experience. It's this way pretty much everywhere in the US now. And I can disapprove of it, you can disapprove of it, we can all disapprove of it, but like I said, until money is completely removed from the calculation, this is how it will be. Critiques of process and role and organization and everything else are interesting, maybe even correct -- but will change nothing.
Not just the U.S., mate. Not just the U.S. Some families in Ethiopia and Indonesia would be happy to be proof. Mark.
On 18/Mar/20 11:43, Keith Medcalf wrote:
No. One simply has to assign a "cost" to "suitability for use". For example, if you put out an RFQ for a CT Machine and someone bids a bag of peanuts for $1.50, that is probably the lowest bid, and that is what you will get if you choose based entirely on the lowest bid. However, if you also require that the purchased machine also actually be capable of performing Computed Tomography then clearly that $1.50 bid will be rejected.
You simply have to define what you want to achieve, then do it.
If only it were that simple, with 2020 corporate life. What I can say tends to work is: "You simply have to define what you want to achieve, scream, yell and shout, then do it, then scream, yell and shout some more, until you can't tell whether you'll leave the job from being fed up or being asked to walk". That has a slightly better chance of succeeding more than failing. I'm old now - I ask once, maybe twice if I've had a beer. Then I carry on, and we meet in 12 months when it all goes to hell :-). Mark.
On 17/Mar/20 23:47, Rich Kulawiec wrote:
Decisions are no longer based on the greater good or on anticipating worst case scenarios or on maximizing preparedness or anything that we might hope they're based on. They're based, coldly and calculatingly, on money.
If you want this to change -- and I sure would like it to change -- then money needs to be entirely removed from that calculation. That is a problem whose solution lies outside the scope of NANOG.
<off_topic> I've been saying, quietly to friends for a while now, that this system of money is stretching all of us to the limit, because we are all chasing it. In this age of Coronavirus, what good is money if the supermarket shelves are empty? What good is money if people are dead? Also, money works because not all of us have it - and yet the goal of any society is to put it in everyone's hands. So then, what's the real value [of money] if everyone has the ability to pay everyone? Between the Coronavirus and the love spats between Russia and Saudi Arabia, "artificial" markets have lost 20% of their value. Some economies with less advanced markets are finding value in other "less artificial" things, so they can get their toilet paper, hehe :-). </off_topic> Seriously though, I echo Rich's comments - this isn't a problem NANOG can solve, but if the Coronavirus has taught us anything, it's that we seriously need to re-evaluate this "modern humanity" thing we are all trying to build, and live. Mark.
On 17/Mar/20 20:35, Owen DeLong wrote:
Step one: Consumers _AND_ especially mission critical consumers must start refusing to purchase devices which have inherent dependency on a vendor-cloud (or any cloud for that matter).
Good advice for mission-critical consumers. But the kids don't care how the information gets to them, as long as it gets to them.
Stop treating things you don’t own and things that aren’t hosted locally as “reliable” and make sure that they are not in the mission critical chain of urgent patient care.
Anything in the healthcare vertical that is outside of the medical providers control/ownership is a result of the medical provider buying into that model on some level. STOP DOING THAT. (How am I suddenly reminded of the old adage “Doctor, doctor, it hurts when I do this!”…)
I understand how the allure of lower costs and the frustration of “every vendor does this, we can’t find one who doesn’t” plays out. However, the only way “every vendor does it” will continue is if every vendor continues to be able to make sales without changing.
Product-based mindset from an industrial era is very hard to shake, even though as consumers of commoditized information in 2020, ourselves, we actually employ a value-based mindset for our personal consumption, which we are unable to use to convert our own businesses into. Funny that, eh... Your competitor is no longer the shop down the road. It's anyone, anywhere, with an Internet connection and an idea. No one is immune from this, not even healthcare providers or the OEM's they choose to buy from. Cutting costs is not how you stay relevant. But, it's what product-based businesses know to do, because the alternative is simply too daring to consider :-). Mark.
On Mar 18, 2020, at 9:24 AM, Mark Tinka <mark.tinka@seacom.mu> wrote:
On 17/Mar/20 20:35, Owen DeLong wrote:
Step one: Consumers _AND_ especially mission critical consumers must start refusing to purchase devices which have inherent dependency on a vendor-cloud (or any cloud for that matter).
Good advice for mission-critical consumers.
Stop treating things you don’t own and things that aren’t hosted locally as “reliable” and make sure that they are not in the mission critical chain of urgent patient care.
We have told our readers (and, really, anyone who will listen) for years that 'the cloud' is just another term for 'somebody else's computer'. Sometimes (often) people really need to hear it in such simple terms. Anne -- Anne P. Mitchell, Attorney at Law Dean of Cyberlaw & Cybersecurity, Lincoln Law School Policy Drafting and Review for Businesses Author: Section 6 of the CAN-SPAM Act of 2003 (the Federal anti-spam law) Legislative Consultant, GDPR, CCPA (CA) & CCDPA (CO) Compliance Consultant Board of Directors, Denver Internet Exchange
On Tuesday, 17 March, 2020 11:04, Mike Bolitho <mikebolitho@gmail.com> wrote:
The answer is don't shove application traffic that has tight service level requirements onto the public internet at large and expect the same performance as private circuits or other SLA protected services.
I keep seeing this over and over again in this long thread. What's your suggestion? How does a hospital, with dozens of third party applications/devices across multiple cloud platforms do this?
Do what everyone else that has "critical infrastructure" does. Put a requirement in the RFP that the thing you want to buy must continue to operate even when totally isolated from the outside world. And then do not select to purchase products that do not meet this requirement. It is quite simple actually. We do this all the time with great success. -- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.
Join an IX your provider is on? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mike Bolitho" <mikebolitho@gmail.com> To: "Tom Beecher" <beecher@beecher.cc> Cc: "NANOG" <nanog@nanog.org> Sent: Tuesday, March 17, 2020 12:03:46 PM Subject: Re: COVID-19 vs. our Networks
The answer is don't shove application traffic that has tight service level requirements onto the public internet at large and expect the same performance as private circuits or other SLA protected services.
I keep seeing this over and over again in this long thread. What's your suggestion? How does a hospital, with dozens of third party applications/devices across multiple cloud platforms do this? We have two redundant private lines out of each hospital connecting back to primary and DR DCs and a metro connecting everything together in each region. But for things we do not own that are not hosted locally, what are we supposed to do? We have to go out DIA to get there. Everything we own is connected via fully SLAed private lines. We have zero issues there. I think people vastly underestimate just how much in the healthcare vertical is outside of a medical providers control/ownership. - Mike Bolitho On Tue, Mar 17, 2020 at 9:54 AM Tom Beecher <beecher@beecher.cc> wrote: The answer is don't shove application traffic that has tight service level requirements onto the public internet at large and expect the same performance as private circuits or other SLA protected services. On Tue, Mar 17, 2020 at 11:40 AM Mike Bolitho < mikebolitho@gmail.com > wrote: <blockquote> <blockquote> If an x-ray machine won't work because the Internet is down, I'm not sure that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me. Totally agree with you. Unfortunately it's not a problem with the medical providers, it's a problem with the medical devices. Anybody who works in the healthcare vertical will tell you just how bad medical devices are to work with from an IT perspective. And that is part of my original comments. <blockquote> In your case, I am not sure I have an answer for you, unfortunately. The public Internet is what it is, mostly best-effort. Your applications and use-cases certainly deserve better than that. I'm not sure how to achieve that as your industry shoves more and more activity into the public Internet domain, for one reason or another. </blockquote> I don't know what it's going to take either. A general shift in mentality from the vendors we use I guess. I'm not sure how you get a bunch of medical providers to tell these companies they need to fix their stuff. You can't exactly use your wallet to force change either. There are only a handful of vendor options out there so there isn't a ton of choice. It's not like you can buy one of 50 different models of CT machines or EHR systems. Generally speaking it's not an issue. It's just in crazy times like these where, if congestion on the public internet gets too crazy, that certain platforms might need to be deemed "unnecessary". Is playing Fortnight a right? Is streaming a movie in 4K a right? In cases like San Francisco they have decided that leaving your home for anything other than work or medical care is no longer a right because you're now infringing on other's rights by potentially getting them sick. Maybe 4K Netflix fits into that category if you're causing problems for first responders and hospitals trying to save lives. - Mike Bolitho On Tue, Mar 17, 2020 at 2:22 AM Mark Tinka < mark.tinka@seacom.mu > wrote: <blockquote> On 16/Mar/20 16:54, Carsten Bormann wrote:
I recently had to reschedule an X-ray because the license manager for the X-ray machine was acting up. I don’t think people have a grasp for how much of the medical infrastructure no longer works when the Internet is down.
I get this, to some extent. But also, there is a reason hospitals, airports and military installations are either put on special power grids or invest plenty of money in backup power. If an x-ray machine won't work because the Internet is down, I'm not sure that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me. Mark. </blockquote> </blockquote> </blockquote>
On Tue, Mar 17, 2020, at 19:59, Mike Hammett wrote:
Join an IX your provider is on?
As someone that works for an IXP these days, I would prefer *NOT* having to deal with people that do not understand the Internet ecosystem. Which hospitals, and most businesses are. An IXP is not an ISP targeting business/corporate. We're already dealing with people that do not understand what an IXP does, and open tickets every time a direct BGP session (one between 2 peers, not involving the route-server) goes down. Even had "Google is slow" tickets. Joining an IX purely for PNI/NNI interconnection may be an option, but only if you are 100% sure that the other party agrees an PNI/NNI over an IX. Some do, some don't, most don't even know it's a possibility.
I think we need an emai list with both skillsets on it? REmember this affects each one of us. Alex, LF/HF 1 Le 24/03/2020 à 14:18, Radu-Adrian Feurdean a écrit :
On Tue, Mar 17, 2020, at 19:59, Mike Hammett wrote:
Join an IX your provider is on? As someone that works for an IXP these days, I would prefer *NOT* having to deal with people that do not understand the Internet ecosystem. Which hospitals, and most businesses are. An IXP is not an ISP targeting business/corporate. We're already dealing with people that do not understand what an IXP does, and open tickets every time a direct BGP session (one between 2 peers, not involving the route-server) goes down. Even had "Google is slow" tickets. Joining an IX purely for PNI/NNI interconnection may be an option, but only if you are 100% sure that the other party agrees an PNI/NNI over an IX. Some do, some don't, most don't even know it's a possibility.
On 17/Mar/20 19:03, Mike Bolitho wrote:
I keep seeing this over and over again in this long thread. What's your suggestion? How does a hospital, with dozens of third party applications/devices across multiple cloud platforms do this?
We have two redundant private lines out of each hospital connecting back to primary and DR DCs and a metro connecting everything together in each region. But for things we do not own that are not hosted locally, what are we supposed to do? We have to go out DIA to get there. Everything we own is connected via fully SLAed private lines. We have zero issues there. I think people vastly underestimate just how much in the healthcare vertical is outside of a medical providers control/ownership.
On my WhatsApp profile, one of my tag lines is "Hater of the 'What Do You Do?' culture and WhatsApp Calling". I detest both equally. But focusing on the latter, if you call me on WhatsApp, I'll cut you off and call you back via GSM. The only time I'll entertain any WhatsApp calls is if GSM coverage is poor, or if I know you are traveling and can't roam, but have wi-fi. I'd never blame my mobile providers for poor quality WhatsApp calls, nor would I do the same to my ISP. I have zero patience for WhatsApp voice calls to sort themselves out when initiated, and yet plenty of people enjoy using it for whatever reasons, mostly to "save money". Personally, wasting time exchanging "Can you hear me now?" is more costly than having a short and concise call over GSM. If we are going to talk for hours, let's have a beer. The point is, as much as some "critical" conversations (want to) take place on WhatsApp, Facebook have zero control of the quality of that experience once the bits leave their data centre. I don't know if they will ever fix that given all the variables that exist thousands of miles from where the service is hosted, but you might not be forgiven for thinking you can run a voice-based business on WhatsApp. In fact, recording a voice note and sending it via WhatsApp is like two-way walkie-talkie radio, but perhaps more reliable :-). I really don't know how to fix this for hospitals relying on best-effort infrastructure to deliver critical, priority services to their patients. Mark.
On 3/17/20 10:03 AM, Mike Bolitho wrote:
We have two redundant private lines out of each hospital connecting back to primary and DR DCs and a metro connecting everything together in each region. But for things we do not own that are not hosted locally, what are we supposed to do? We have to go out DIA to get there. Everything we own is connected via fully SLAed private lines. We have zero issues there. I think people vastly underestimate just how much in the healthcare vertical is outside of a medical providers control/ownership.
Do all the SLA's in the world even matter if the contract has a force majeure clause?
Depends on the verbiage of the clause. On Wed, Mar 18, 2020 at 10:41 AM Seth Mattinen <sethm@rollernet.us> wrote:
On 3/17/20 10:03 AM, Mike Bolitho wrote:
We have two redundant private lines out of each hospital connecting back to primary and DR DCs and a metro connecting everything together in each region. But for things we do not own that are not hosted locally, what are we supposed to do? We have to go out DIA to get there. Everything we own is connected via fully SLAed private lines. We have zero issues there. I think people vastly underestimate just how much in the healthcare vertical is outside of a medical providers control/ownership.
Do all the SLA's in the world even matter if the contract has a force majeure clause?
On 18/Mar/20 16:35, Seth Mattinen wrote:
Do all the SLA's in the world even matter if the contract has a force majeure clause?
Feel-good-tick-in-the-box type-thing... like that time a network operator is asked if any part of their network/service touches any equipment manufactured by a well-known Chinese OEM :-). Tick in the box :-)... Mark.
On Tue, Mar 17, 2020 at 08:38:28AM -0700, Mike Bolitho wrote:
Anybody who works in the healthcare vertical will tell you just how bad medical devices are to work with from an IT perspective.
Medical devices are appallingly bad to work with from an IT perspective. They're designed and built to work in idealized environments that don't exist, they make unduly optimistic assumptions, they completely fail to account for hostile actors, and whenever possible they are gratuitously incompatible to ensure vendor lock-in. That's the good news. Here's the bad news: in about 2-3 weeks, when our health care systems are stretched to the breaking point, there will be a window of opportunity for adversaries to maximize the damage. ---rsk
On 03/17/20 14:38 -0400, Rich Kulawiec wrote:
On Tue, Mar 17, 2020 at 08:38:28AM -0700, Mike Bolitho wrote:
Anybody who works in the healthcare vertical will tell you just how bad medical devices are to work with from an IT perspective.
Medical devices are appallingly bad to work with from an IT perspective.
They're designed and built to work in idealized environments that don't exist, they make unduly optimistic assumptions, they completely fail to account for hostile actors, and whenever possible they are gratuitously incompatible to ensure vendor lock-in.
That's the good news. Here's the bad news: in about 2-3 weeks, when our health care systems are stretched to the breaking point, there will be a window of opportunity for adversaries to maximize the damage.
On a slightly tangential topic, we had a dictionary attack against customer voice accounts over night, presumably to implement toll fraud. We were in the middle of working out work-from-home plans and were quite distracted with other things. We managed to get on top of it quickly once someone noticed. Attackers taking advantage of this situation is a serious concern. -- Dan White Network Admin Lead
On 3/17/2020 1:54 PM, Dan White wrote:
On 03/17/20 14:38 -0400, Rich Kulawiec wrote:
On Tue, Mar 17, 2020 at 08:38:28AM -0700, Mike Bolitho wrote:
Anybody who works in the healthcare vertical will tell you just how bad medical devices are to work with from an IT perspective.
Medical devices are appallingly bad to work with from an IT perspective.
They're designed and built to work in idealized environments that don't exist, they make unduly optimistic assumptions, they completely fail to account for hostile actors, and whenever possible they are gratuitously incompatible to ensure vendor lock-in.
That's the good news. Here's the bad news: in about 2-3 weeks, when our health care systems are stretched to the breaking point, there will be a window of opportunity for adversaries to maximize the damage.
On a slightly tangential topic, we had a dictionary attack against customer voice accounts over night, presumably to implement toll fraud. We were in the middle of working out work-from-home plans and were quite distracted with other things. We managed to get on top of it quickly once someone noticed.
Attackers taking advantage of this situation is a serious concern.
Dan, we're aware of another telco that ran into a similar fraud situation last week. They stood up some more restrictive ACLs to combat the fraud, but broke VoIP RTP in the process. 'Hit em while they're occupied' type of attacks I guess should be expected right now. As my grandmother would say: an ounce of prevention is worth a pound of cure.
On 03/18/20 09:29 -0500, Blake Hudson wrote:
On 3/17/2020 1:54 PM, Dan White wrote:
On 03/17/20 14:38 -0400, Rich Kulawiec wrote:
On Tue, Mar 17, 2020 at 08:38:28AM -0700, Mike Bolitho wrote:
That's the good news. Here's the bad news: in about 2-3 weeks, when our health care systems are stretched to the breaking point, there will be a window of opportunity for adversaries to maximize the damage.
On a slightly tangential topic, we had a dictionary attack against customer voice accounts over night, presumably to implement toll fraud. We were in the middle of working out work-from-home plans and were quite distracted with other things. We managed to get on top of it quickly once someone noticed.
Attackers taking advantage of this situation is a serious concern.
Dan, we're aware of another telco that ran into a similar fraud situation last week. They stood up some more restrictive ACLs to combat the fraud, but broke VoIP RTP in the process. 'Hit em while they're occupied' type of attacks I guess should be expected right now. As my grandmother would say: an ounce of prevention is worth a pound of cure.
Hey Blake, I appreciate that. We've got two tendencies going on here at the moment: 1) Man the ship of operations. Stay alert and fix the problems that arise. Be totally reactive, and be the "hero". 2) Increase visibility and focus on network design. Move planned upgrades up a few weeks/months. Be proactive. Option 2 is the better long term option, but the risk is that any change, while short staffed is going to run the risk of unintended consequences. Basically it's "If it's not broke, don't fix it." and "Be very paranoid about what you touch. -- Dan White Network Admin Lead
On 17/Mar/20 20:54, Dan White wrote:
Attackers taking advantage of this situation is a serious concern.
In South Africa, we have people claiming to be from the Department of Health and one other reputable medical care group, going door-to-door offering Coronavirus testing: https://www.iol.co.za/dailynews/news/kwazulu-natal/south-africans-warned-abo... Be alert; the scammers are. Mark.
I saw on TV official requests to police radio to remove masks (Idont know why, because they dont have any anyways) Le 18/03/2020 à 16:29, Mark Tinka a écrit :
On 17/Mar/20 20:54, Dan White wrote:
Attackers taking advantage of this situation is a serious concern.
In South Africa, we have people claiming to be from the Department of Health and one other reputable medical care group, going door-to-door offering Coronavirus testing:
https://www.iol.co.za/dailynews/news/kwazulu-natal/south-africans-warned-abo...
Be alert; the scammers are.
Mark.
On Mar 17, 2020, at 02:20 , Mark Tinka <mark.tinka@seacom.mu> wrote:
On 16/Mar/20 16:54, Carsten Bormann wrote:
I recently had to reschedule an X-ray because the license manager for the X-ray machine was acting up. I don’t think people have a grasp for how much of the medical infrastructure no longer works when the Internet is down.
I get this, to some extent. But also, there is a reason hospitals, airports and military installations are either put on special power grids or invest plenty of money in backup power.
I don’t get this… X-Ray machines (and other critical medical equipment) should operate in a fail-safe mode where a license screw up doesn’t prevent the machine from operating. If the hospital hasn’t paid up, find a way to go after the hospital, but don’t kill patients to collect your fee.
If an x-ray machine won't work because the Internet is down, I'm not sure that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Why should there be a license server at all? Why should an X-ray machine have an external dependency like that in the first place, even if it’s a local server? Owen
Because the hospitals don't own the machines and the companies that do, charge the hospital per x-ray. The hospitals moved to this model to reduce their costs during "quiet" periods. And by doing so, put their patients in jeopardy. On Tue, Mar 17, 2020, 2:07 PM Owen DeLong <owen@delong.com> wrote:
On Mar 17, 2020, at 02:20 , Mark Tinka <mark.tinka@seacom.mu> wrote:
On 16/Mar/20 16:54, Carsten Bormann wrote:
I recently had to reschedule an X-ray because the license manager for the X-ray machine was acting up. I don’t think people have a grasp for how much of the medical infrastructure no longer works when the Internet is down.
I get this, to some extent. But also, there is a reason hospitals, airports and military installations are either put on special power grids or invest plenty of money in backup power.
I don’t get this… X-Ray machines (and other critical medical equipment) should operate in a fail-safe mode where a license screw up doesn’t prevent the machine from operating.
If the hospital hasn’t paid up, find a way to go after the hospital, but don’t kill patients to collect your fee.
If an x-ray machine won't work because the Internet is down, I'm not sure that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Why should there be a license server at all? Why should an X-ray machine have an external dependency like that in the first place, even if it’s a local server?
Owen
On 17/Mar/20 20:26, Shane Ronan wrote:
Because the hospitals don't own the machines and the companies that do, charge the hospital per x-ray. The hospitals moved to this model to reduce their costs during "quiet" periods. And by doing so, put their patients in jeopardy.
Can be said of, pretty much, any industry in 2020 that sells products (and not value). Remember that plane that was designed -MAX? Wonder how that happened. Mark.
Why should there be a license server at all? Why should an X-ray machine have an external dependency like that in the first place, even if it’s a local server?
In a world where you can license device performance by the megabit/sec/day, or even have to purchase per-use factory reset keys since the manufacture has stripped product owners of that right too, this doesn't totally surprise me. There would have to be a flip side to that coin - I would have to guess (read: guess) it's a 'n' x-rays/day to "cut costs to the end user." Great practice on paper for little guys, but beyond that... -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Owen DeLong Sent: Tuesday, March 17, 2020 11:06 AM To: Mark Tinka Cc: nanog@nanog.org Subject: Re: COVID-19 vs. our Networks
On Mar 17, 2020, at 02:20 , Mark Tinka <mark.tinka@seacom.mu> wrote:
On 16/Mar/20 16:54, Carsten Bormann wrote:
I recently had to reschedule an X-ray because the license manager for the X-ray machine was acting up. I don’t think people have a grasp for how much of the medical infrastructure no longer works when the Internet is down.
I get this, to some extent. But also, there is a reason hospitals, airports and military installations are either put on special power grids or invest plenty of money in backup power.
I don’t get this… X-Ray machines (and other critical medical equipment) should operate in a fail-safe mode where a license screw up doesn’t prevent the machine from operating. If the hospital hasn’t paid up, find a way to go after the hospital, but don’t kill patients to collect your fee.
If an x-ray machine won't work because the Internet is down, I'm not sure that is responsible. As inefficient as it may be to have a license server on-prem if there is an option to check against one in the public cloud, for a medical use-case, that would make more sense to me.
Why should there be a license server at all? Why should an X-ray machine have an external dependency like that in the first place, even if it’s a local server? Owen
On 17/Mar/20 20:33, Emille Blanc wrote:
In a world where you can license device performance by the megabit/sec/day, or even have to purchase per-use factory reset keys since the manufacture has stripped product owners of that right too, this doesn't totally surprise me.
There would have to be a flip side to that coin - I would have to guess (read: guess) it's a 'n' x-rays/day to "cut costs to the end user." Great practice on paper for little guys, but beyond that...
In the industrial era, it was "knowledge & expertise". In the digital era, it's "curiosity and creativity". Access to information is ubiquitous and exponential. Knowledge has been commoditized. How does that hurt the medical industry, you wonder? Well, a webachondriac will use the Internet to easily self-diagnose, use an app to order medication online, and have it delivered, all without ever leaving his/her house. How many businesses have lost out on his dime in that process? The local GP down the corner. The local pharmacy up the corner. And all the supply chain in between. If traditional businesses don't adapt, they will become irrelevant. Cutting costs (as the hospitals and, pretty much, any industry is doing) is the first path to staving off the death spiral. And then the massacre follows. Mark.
On 17/Mar/20 20:06, Owen DeLong wrote:
I don’t get this… X-Ray machines (and other critical medical equipment) should operate in a fail-safe mode where a license screw up doesn’t prevent the machine from operating.
If the hospital hasn’t paid up, find a way to go after the hospital, but don’t kill patients to collect your fee.
For my very simple 1+1 mind, I totally agree. Perhaps, it's far easier to collect (overdue) fees with a gun to your head, if I don't actually need to point one at you.
Why should there be a license server at all? Why should an X-ray machine have an external dependency like that in the first place, even if it’s a local server?
My Google OnHub wireless AP is completely unmanageable if I (against Google's advice) run it in Bridged mode. If I want to be able to reach it and manage it with an app or a web site, it needs to run as a router, even if all I want from it is to be an AP. You can guess who long mine have gone without a software update, then... Who knows why people come up with the BS they do? Mark.
Is it so difficult to put an "override, but keep counting" button on a device like this? On Wed, Mar 18, 2020 at 8:04 AM Mark Tinka <mark.tinka@seacom.mu> wrote:
On 17/Mar/20 20:06, Owen DeLong wrote:
I don’t get this… X-Ray machines (and other critical medical equipment) should operate in a fail-safe mode where a license screw up doesn’t prevent the machine from operating.
If the hospital hasn’t paid up, find a way to go after the hospital, but don’t kill patients to collect your fee.
For my very simple 1+1 mind, I totally agree.
Perhaps, it's far easier to collect (overdue) fees with a gun to your head, if I don't actually need to point one at you.
Why should there be a license server at all? Why should an X-ray machine have an external dependency like that in the first place, even if it’s a local server?
My Google OnHub wireless AP is completely unmanageable if I (against Google's advice) run it in Bridged mode. If I want to be able to reach it and manage it with an app or a web site, it needs to run as a router, even if all I want from it is to be an AP. You can guess who long mine have gone without a software update, then...
Who knows why people come up with the BS they do?
Mark.
-- Jeff Shultz -- Like us on Social Media for News, Promotions, and other information!! <https://www.facebook.com/SCTCWEB/> <https://www.instagram.com/sctc_503/> <https://www.yelp.com/biz/sctc-stayton-3> <https://www.youtube.com/c/sctcvideos> _**** This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. ****_
Mike- The TSP program provides for priority treatment for only 2 things : provisioning of new capacity, and restoration of capacity. It provides no accommodations for intermittent degradation events upstream. Source : DHC Office of Emergency Communications, TSP Program Office, TSP Vendor Handbook. https://www.cisa.gov/sites/default/files/publications/OEC%20Service%20Vendor... On Mon, Mar 16, 2020 at 10:42 AM Mike Bolitho <mikebolitho@gmail.com> wrote:
I think there's a bit of a misunderstanding of what I'm trying to say here. We have dual private lines from two Tier I providers. These interconnect all major hospitals and our data centers. We also have a third metro connection that connects things regionally. We have DIA on top of that. I think people are vastly underestimating just how much $aaS there is within the medical field. TeleDoc, translation services, remote radiologists, the way prescriptions get filled, how staffing works, third party providers basically hoteling within our facilities, critical staff VPNed in because the government has locked things down, etc. Then there's things that we don't use but I'm sure other providers do, GoToMeeting, O365, VaaS, etc. There's no practical way to engineer your WAN to facilitate dozens of connections to these services.
This extends beyond just hospitals as well. Fire departments, police departments, water treatment etc. Regardless of whether or not those entities planned well (I think we did), the government should and will step in if critical services are degraded. And for what it's worth, Stephen, I know how things are built within the ISP world. I spent four years there. That doesn't change the fact that we're possibly heading into uncharted waters when it comes to utilization and the impact that will have on $aaS products that are interwoven into every single vertical, including entities that fall under TSP, critical national security and emergency preparedness functions, including those areas related to safety, maintenance of law and order, and public health. It's easy for all you guys to sit here and armchair quarterback other people's planning but when things really start to degrade, all bets are off. If you don't believe that, just look at the news. States are literally shutting down private businesses (restaurants, bars, night clubs, private schools) and banning people from associating in groups of larger than 50.
*The opinions expressed here are my own and do not represent my employer or their views.*
- Mike Bolitho
On Sun, Mar 15, 2020 at 6:12 PM Stephen Fulton <sf@lists.esoteric.ca> wrote:
In $dayjob I constantly see the lack of understanding of the difference between what the Internet is and what a path engineered private circuit is (eg. pseudowire, wave, whatever). The latest fight is over SD-WAN and those who think it will replace MPLS entirely and they won't need those expensive routers anymore. But I digress.
Mark's comment and others like it are the correct approach Mike. If your private WAN is most critical, then invest in and manage user complaints about poor Internet service. ISP's, IXP's and CDN's are not going to twist themselves into knots to solve your problems, even if someone calls it an emergency. Sorry.
Stephen
On 2020-03-15 02:01, Mark Tinka wrote:
On 14/Mar/20 19:14, Mike Bolitho wrote:
/ /
I work for a hospital, we ran into some issues last week due to congestion that was totally outside of our control that was off of our WAN (Thanks Call Of Duty). Now, the issue we ran into was not mission critical at the time but it was still disruptive. As more and more people are driven home during this time, more and more people will be using bandwidth intensive streaming and online gaming products. If more and more TSP coded entities are running into issues, ISPs, IXPs, and CDNs will be forced to act.
Hmmh, if that level of priority is required, I'd probably build my own network, and not rely on public infrastructure like the Internet.
Mark.
On this subject, this is worth a read: https://transition.fcc.gov/pshs/docs/emergency-information/Pandemic_Comms_Im... Department of Homeland Security Pandemic Influenza Impact on Communications Networks Study Dec 2007 JL From: NANOG <nanog-bounces@nanog.org> on behalf of Tom Beecher <beecher@beecher.cc> Date: Monday, March 16, 2020 at 2:10 PM To: Mike Bolitho <mikebolitho@gmail.com> Cc: NANOG <nanog@nanog.org> Subject: Re: COVID-19 vs. our Networks Mike- The TSP program provides for priority treatment for only 2 things : provisioning of new capacity, and restoration of capacity. It provides no accommodations for intermittent degradation events upstream. Source : DHC Office of Emergency Communications, TSP Program Office, TSP Vendor Handbook. https://www.cisa.gov/sites/default/files/publications/OEC%20Service%20Vendor%20Handbook%20for%20TSP%2010-23-2017%20FINAL%20508C.pdf<https://urldefense.com/v3/__https:/www.cisa.gov/sites/default/files/publications/OEC*20Service*20Vendor*20Handbook*20for*20TSP*2010-23-2017*20FINAL*20508C.pdf__;JSUlJSUlJSU!!CQl3mcHX2A!XxhMP9AfXmBNjiociWHdq1-r4dwGmRnCRuDMe2bkws8JFS7ZeBDDcSORQWi3sQehSRlv9eI9$> On Mon, Mar 16, 2020 at 10:42 AM Mike Bolitho <mikebolitho@gmail.com<mailto:mikebolitho@gmail.com>> wrote: I think there's a bit of a misunderstanding of what I'm trying to say here. We have dual private lines from two Tier I providers. These interconnect all major hospitals and our data centers. We also have a third metro connection that connects things regionally. We have DIA on top of that. I think people are vastly underestimating just how much $aaS there is within the medical field. TeleDoc, translation services, remote radiologists, the way prescriptions get filled, how staffing works, third party providers basically hoteling within our facilities, critical staff VPNed in because the government has locked things down, etc. Then there's things that we don't use but I'm sure other providers do, GoToMeeting, O365, VaaS, etc. There's no practical way to engineer your WAN to facilitate dozens of connections to these services. This extends beyond just hospitals as well. Fire departments, police departments, water treatment etc. Regardless of whether or not those entities planned well (I think we did), the government should and will step in if critical services are degraded. And for what it's worth, Stephen, I know how things are built within the ISP world. I spent four years there. That doesn't change the fact that we're possibly heading into uncharted waters when it comes to utilization and the impact that will have on $aaS products that are interwoven into every single vertical, including entities that fall under TSP, critical national security and emergency preparedness functions, including those areas related to safety, maintenance of law and order, and public health. It's easy for all you guys to sit here and armchair quarterback other people's planning but when things really start to degrade, all bets are off. If you don't believe that, just look at the news. States are literally shutting down private businesses (restaurants, bars, night clubs, private schools) and banning people from associating in groups of larger than 50. The opinions expressed here are my own and do not represent my employer or their views. - Mike Bolitho On Sun, Mar 15, 2020 at 6:12 PM Stephen Fulton <sf@lists.esoteric.ca<mailto:sf@lists.esoteric.ca>> wrote: In $dayjob I constantly see the lack of understanding of the difference between what the Internet is and what a path engineered private circuit is (eg. pseudowire, wave, whatever). The latest fight is over SD-WAN and those who think it will replace MPLS entirely and they won't need those expensive routers anymore. But I digress. Mark's comment and others like it are the correct approach Mike. If your private WAN is most critical, then invest in and manage user complaints about poor Internet service. ISP's, IXP's and CDN's are not going to twist themselves into knots to solve your problems, even if someone calls it an emergency. Sorry. Stephen On 2020-03-15 02:01, Mark Tinka wrote:
On 14/Mar/20 19:14, Mike Bolitho wrote:
/ /
I work for a hospital, we ran into some issues last week due to congestion that was totally outside of our control that was off of our WAN (Thanks Call Of Duty). Now, the issue we ran into was not mission critical at the time but it was still disruptive. As more and more people are driven home during this time, more and more people will be using bandwidth intensive streaming and online gaming products. If more and more TSP coded entities are running into issues, ISPs, IXPs, and CDNs will be forced to act.
Hmmh, if that level of priority is required, I'd probably build my own network, and not rely on public infrastructure like the Internet.
Mark.
On 16/Mar/20 16:40, Mike Bolitho wrote:
I think there's a bit of a misunderstanding of what I'm trying to say here. We have dual private lines from two Tier I providers. These interconnect all major hospitals and our data centers. We also have a third metro connection that connects things regionally. We have DIA on top of that. I think people are vastly underestimating just how much $aaS there is within the medical field. TeleDoc, translation services, remote radiologists, the way prescriptions get filled, how staffing works, third party providers basically hoteling within our facilities, critical staff VPNed in because the government has locked things down, etc. Then there's things that we don't use but I'm sure other providers do, GoToMeeting, O365, VaaS, etc.There's no practical way to engineer your WAN to facilitate dozens of connections to these services.
This extends beyond just hospitals as well. Fire departments, police departments, water treatment etc. Regardless of whether or not those entities planned well(I think we did), the government should and will step in if critical services are degraded. And for what it's worth, Stephen, I know how things are built within the ISP world. I spent four years there. That doesn't change the fact that we're possibly heading into uncharted waters when it comes to utilization and the impactthat will have on $aaS products that are interwoven into every single vertical, including entities that fall under TSP, critical national security and emergency preparedness functions, including those areas related to safety, maintenance of law and order, and public health.It's easy for all you guys to sit here and armchair quarterback other people's planning but when things really start to degrade, all bets are off.If you don't believe that, just look at the news. States are literally shutting down private businesses (restaurants, bars, night clubs, private schools) and banning people from associating in groups of larger than 50.
The Internet has infiltrated every industry, every business, and every business model. While it's a great way to connect a lot of people and things at scale for the lowest cost possible, there are some industries that still require a certain caliber of reliability that the public Internet may not be best suited to provide. In your case, I am not sure I have an answer for you, unfortunately. The public Internet is what it is, mostly best-effort. Your applications and use-cases certainly deserve better than that. I'm not sure how to achieve that as your industry shoves more and more activity into the public Internet domain, for one reason or another. Mark.
On 17/03/2020 09:17, Mark Tinka wrote:
On 16/Mar/20 16:40, Mike Bolitho wrote:
I think there's a bit of a misunderstanding of what I'm trying to say here. We have dual private lines from two Tier I providers. These interconnect all major hospitals and our data centers. We also have a third metro connection that connects things regionally. We have DIA on top of that. I think people are vastly underestimating just how much $aaS there is within the medical field. TeleDoc, translation services, remote radiologists, the way prescriptions get filled, how staffing works, third party providers basically hoteling within our facilities, critical staff VPNed in because the government has locked things down, etc. Then there's things that we don't use but I'm sure other providers do, GoToMeeting, O365, VaaS, etc.There's no practical way to engineer your WAN to facilitate dozens of connections to these services.
This extends beyond just hospitals as well. Fire departments, police departments, water treatment etc. Regardless of whether or not those entities planned well(I think we did), the government should and will step in if critical services are degraded. And for what it's worth, Stephen, I know how things are built within the ISP world. I spent four years there. That doesn't change the fact that we're possibly heading into uncharted waters when it comes to utilization and the impactthat will have on $aaS products that are interwoven into every single vertical, including entities that fall under TSP, critical national security and emergency preparedness functions, including those areas related to safety, maintenance of law and order, and public health.It's easy for all you guys to sit here and armchair quarterback other people's planning but when things really start to degrade, all bets are off.If you don't believe that, just look at the news. States are literally shutting down private businesses (restaurants, bars, night clubs, private schools) and banning people from associating in groups of larger than 50.
The Internet has infiltrated every industry, every business, and every business model.
While it's a great way to connect a lot of people and things at scale for the lowest cost possible, there are some industries that still require a certain caliber of reliability that the public Internet may not be best suited to provide.
In your case, I am not sure I have an answer for you, unfortunately. The public Internet is what it is, mostly best-effort. Your applications and use-cases certainly deserve better than that. I'm not sure how to achieve that as your industry shoves more and more activity into the public Internet domain, for one reason or another.
Mark.
In theory best-effort Internet is seen as only part of a broader Internet model including open peering and so on. The idea for open Internet is it offers a form of digital herd immunity (to coin a current phrase being misused by UK Government circles in recent days) that offers a level of shared redundancy of spare capacity so that issues can be taken out of route until fixed but the edge still maintains high quality connectivity. In one sense the Internet model provides an informal community insurance across the provider / access sector. Although of course the legacy telco regulated protected infrastructure has remained a nub of resistance to open anything. Some short term financial optimisations between networks may turn out to be counter productive across time and "events". Which begs a question whether the winner takes all model that has emerged can live with a plural supply chain of network infrastructures. I suspect the concentration over recent years has created greater fragility for all of us judging comments in this thread and elsewhere. Can we survive covid 19 and maintain selfish networks over open ones? C
On 17/Mar/20 12:37, Christian wrote:
In theory best-effort Internet is seen as only part of a broader Internet model including open peering and so on. The idea for open Internet is it offers a form of digital herd immunity (to coin a current phrase being misused by UK Government circles in recent days) that offers a level of shared redundancy of spare capacity so that issues can be taken out of route until fixed but the edge still maintains high quality connectivity. In one sense the Internet model provides an informal community insurance across the provider / access sector. Although of course the legacy telco regulated protected infrastructure has remained a nub of resistance to open anything.
Some short term financial optimisations between networks may turn out to be counter productive across time and "events". Which begs a question whether the winner takes all model that has emerged can live with a plural supply chain of network infrastructures.
I suspect the concentration over recent years has created greater fragility for all of us judging comments in this thread and elsewhere. Can we survive covid 19 and maintain selfish networks over open ones?
Even if any organization tried to, they can't avoid the allure of the Internet to optimize costs, because their customers (the kids) are going to keep looking for value many of these organizations do not know how to deliver in this new economic era. So reducing costs is the first thing they will do in order to meet budgets, before the real massacre comes. A lot of Internet traffic is coalescing around a handful of service providers, as you rightly point out. They can enhance performance by building data centres closer to customers in big cities, and then toy with the idea of having even smaller edge clusters spread across wide metros. While that does improve availability and performance, I don't think it really pushes the Internet beyond the realm of "best-effort". The Internet is not a centralized, government-based entity. On that basis, it scales very well on a global scale, but conversely, cannot be tuned to operate in the way traditional telco's do/did, along with the compromise critical services like health, military and aviation have to make, to that effect. Mark.
On 2020-03-17, at 12:36, Mark Tinka <mark.tinka@seacom.mu> wrote:
While that does improve availability and performance, I don't think it really pushes the Internet beyond the realm of "best-effort”.
Folks, my supermarket is “best-effort”. I expect exactly the same level of service from my Internet that I expect from the system of supermarkets I have available to me. Grüße, Carsten
participants (23)
-
Alexandre Petrescu -
Anne P. Mitchell, Esq. -
Blake Hudson -
Carsten Bormann -
Christian -
Clayton Zekelman -
Dan White -
Emille Blanc -
Jeff Shultz -
Keith Medcalf -
Livingood, Jason -
Mark Tinka -
Matthew Petach -
Mike Bolitho -
Mike Hammett -
Owen DeLong -
Radu-Adrian Feurdean -
Rich Kulawiec -
Saku Ytti -
Seth Mattinen -
Shane Ronan -
Stephen Fulton -
Tom Beecher