Florida: Voter registration website overwhelmed at deadline
Every election has problems. Most of the time, those problems aren't noticed. Elections rely on a lot of back-end infrastructure, besides the actual voting itself. It could be a DDOS attack, or simply duct-taped systems having trouble with the load. Voting early (mail, drop-off, in-person) means more time to fix glitches. https://apnews.com/article/virus-outbreak-election-2020-florida-elections-ro... The FBI and the Cybersecurity and Infrastructure Security Agency warned elections officials nationwide last week that cyberattacks could disrupt their systems during the run-up to the election. They particularly noted “distributed denial-of-service” attacks, which inundate a computer system with requests, potentially clogging up servers until the system becomes inaccessible to legitimate users.
no indication of a DoS attack. <https://www.cloudflare.com/> __________________ *Justin Paine* He/Him/His Head of Trust & Safety 101 Townsend St, San Francisco, CA 94107 <https://www.cloudflare.com/> *PGP:* BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D <https://keys.openpgp.org/vks/v1/by-fingerprint/BBAA6BCE33057FD66452711557B60114DE0B314D> On Tue, Oct 6, 2020 at 9:51 AM Sean Donelan <sean@donelan.com> wrote:
Every election has problems. Most of the time, those problems aren't noticed. Elections rely on a lot of back-end infrastructure, besides the actual voting itself.
It could be a DDOS attack, or simply duct-taped systems having trouble with the load.
Voting early (mail, drop-off, in-person) means more time to fix glitches.
https://apnews.com/article/virus-outbreak-election-2020-florida-elections-ro...
The FBI and the Cybersecurity and Infrastructure Security Agency warned elections officials nationwide last week that cyberattacks could disrupt their systems during the run-up to the election. They particularly noted “distributed denial-of-service” attacks, which inundate a computer system with requests, potentially clogging up servers until the system becomes inaccessible to legitimate users.
Florida has had notoriously unreliable state I.T. infrastructure for years. Florida's unemployment websites were broken for months during the Spring 2020 COVID unemployment demand surge. So its very likely crappy state I.T. infrastructure problems being stressed by high volume. Florida voting age population (2016): 15,839,713 Florida registered voters (2016): 12,863,773 Florida registered voters (2019): 13,536,830 Florida Secretary of State Laurel Lee, who oversees the voting system, said the online registration system “was accessed by an unprecedented 1.1 million requests per hour” during the last few hours of Monday. On Tue, 6 Oct 2020, Justin Paine wrote:
no indication of a DoS attack.
__________________ Justin Paine He/Him/His Head of Trust & Safety 101 Townsend St, San Francisco, CA 94107
PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
On 06/10/2020, Sean Donelan <sean@donelan.com> wrote:
Florida has had notoriously unreliable state I.T. infrastructure for years. Florida's unemployment websites were broken for months during the Spring 2020 COVID unemployment demand surge. So its very likely crappy state I.T. infrastructure problems being stressed by high volume.
Florida voting age population (2016): 15,839,713 Florida registered voters (2016): 12,863,773 Florida registered voters (2019): 13,536,830
Florida Secretary of State Laurel Lee, who oversees the voting system, said the online registration system “was accessed by an unprecedented 1.1 million requests per hour” during the last few hours of Monday.
People act like 1.1 million requests per hour is a huge number. That's only 305 requests per second! Cheapest NVMe SSDs are capable of 160k+ IOPS. You can literally serve the whole thing from a single server on a 100Mbps line, if you design it properly, and don't waste bandwidth on stock images and silly front-ends. Add a T1 to do replication on the side to an off-site location. 100 Mbit/s / 305 req/s = 40 KiloBytes/req -- should be enough to display/process any form; and you can even get higher speeds on a 5G mobile phone these days; 1.5 Mbit/s / 305 req/s = 0.6 KByte/req -- should be enough to replicate each registration; and why are we even talking about T1 in 2020?! Keep in mind that 1Gbps (e.g., 1000Mbit/s) is pretty much a minimum these days, so, you'd either have plenty of extra room to spare, or can do way more than an average of 1.1 million requests per hour. A Google search reveals you can even get 10 Gbps transit for only $900/mo from he.net these days, for example. P.S. At least here you may have to collect and distribute unique information to each visitor; but what excuse did PG&E had in 2019 when they couldn't distribute non-unique information about the preemptive power shutoffs that they've had about one year ago now?! What I'm always curious about, is how many servers do they actually have, and just how unreasonable do their numbers look when you lay it all out. You'd think paying a few mils to design the system could actually make it work properly when the time comes. Or are they somehow not aware that they have 16M voters, everyone always doing everything in the last minute?! C.
This is a great F’in email, Sean! Sent from my iPhone
On Oct 7, 2020, at 10:10 PM, Constantine A. Murenin <mureninc@gmail.com> wrote:
On 06/10/2020, Sean Donelan <sean@donelan.com> wrote:
Florida has had notoriously unreliable state I.T. infrastructure for years. Florida's unemployment websites were broken for months during the Spring 2020 COVID unemployment demand surge. So its very likely crappy state I.T. infrastructure problems being stressed by high volume.
Florida voting age population (2016): 15,839,713 Florida registered voters (2016): 12,863,773 Florida registered voters (2019): 13,536,830
Florida Secretary of State Laurel Lee, who oversees the voting system, said the online registration system “was accessed by an unprecedented 1.1 million requests per hour” during the last few hours of Monday.
People act like 1.1 million requests per hour is a huge number.
That's only 305 requests per second!
Cheapest NVMe SSDs are capable of 160k+ IOPS.
You can literally serve the whole thing from a single server on a 100Mbps line, if you design it properly, and don't waste bandwidth on stock images and silly front-ends.
Add a T1 to do replication on the side to an off-site location.
100 Mbit/s / 305 req/s = 40 KiloBytes/req -- should be enough to display/process any form; and you can even get higher speeds on a 5G mobile phone these days;
1.5 Mbit/s / 305 req/s = 0.6 KByte/req -- should be enough to replicate each registration; and why are we even talking about T1 in 2020?!
Keep in mind that 1Gbps (e.g., 1000Mbit/s) is pretty much a minimum these days, so, you'd either have plenty of extra room to spare, or can do way more than an average of 1.1 million requests per hour. A Google search reveals you can even get 10 Gbps transit for only $900/mo from he.net these days, for example.
P.S. At least here you may have to collect and distribute unique information to each visitor; but what excuse did PG&E had in 2019 when they couldn't distribute non-unique information about the preemptive power shutoffs that they've had about one year ago now?!
What I'm always curious about, is how many servers do they actually have, and just how unreasonable do their numbers look when you lay it all out. You'd think paying a few mils to design the system could actually make it work properly when the time comes. Or are they somehow not aware that they have 16M voters, everyone always doing everything in the last minute?!
C.
On Wed, 07 Oct 2020 22:10:07 -0700, "Constantine A. Murenin" said:
People act like 1.1 million requests per hour is a huge number.
That's only 305 requests per second!
Cheapest NVMe SSDs are capable of 160k+ IOPS.
You can literally serve the whole thing from a single server on a 100Mbps line, if you design it properly, and don't waste bandwidth on stock images and silly front-ends.
It isn't the stock images and silly front-ends that take all the effort. Those are pretty damned easy to serve up quickly. It's the twisty little maze of databases, all different. You asked for a driver's license number for ID? Well, that just bought you a call to the DMV's servers to check on the validity/status of that ID. Vetting the home address gets equally interesting, especially if it's a PO box or a "suite" at a mailbox-for-rent company. Vetting the existence of the last employer is going to take time as well. Are you going to get the unemployment system, the tax system, the DMV systems, and any others you need to talk to on this "one server"? Oh, and don't forget that the systems in the DMV and tax systems almost certainly have *other* systems they have to talk to.... Don't forget that these state agencies usually don't have the budget that Amazon or other large commercial organizations have, so you're looking at a *really* high chance that some server in the Department of Revenue isn't sized big/fast enough, so verifying the employer's existence hangs, so the front end hangs.... On top of all that, even if you're only a *little* bit too slow clearing requests, you end up sitting on a big pile of pending requests, which sucks up memory.. Get 305 requests per second, clear 304 per second, and in a few minutes you're throwing '502 Gateway Error' left right and center because things are wedged up....
I understand that there is underlying work that can't be sourced somewhere else, at least not trivially. How many of these overloaded web sites that we hear about (voter registration, unemployment registration, web sites announced in a big way, causing surges in traffic, etc.) have a CDN offloading the low-hanging fruit? I know that processing a voter registration is far more intensive than serving up static images, but surely a CDN taking the low hanging fruit would help to some degree. I'm assuming most of the people running these sites are clueless and haven't looked at this, but maybe they have. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Sean Donelan" <sean@donelan.com> To: nanog@nanog.org Sent: Tuesday, October 6, 2020 11:51:39 AM Subject: Florida: Voter registration website overwhelmed at deadline Every election has problems. Most of the time, those problems aren't noticed. Elections rely on a lot of back-end infrastructure, besides the actual voting itself. It could be a DDOS attack, or simply duct-taped systems having trouble with the load. Voting early (mail, drop-off, in-person) means more time to fix glitches. https://apnews.com/article/virus-outbreak-election-2020-florida-elections-ro... The FBI and the Cybersecurity and Infrastructure Security Agency warned elections officials nationwide last week that cyberattacks could disrupt their systems during the run-up to the election. They particularly noted “distributed denial-of-service” attacks, which inundate a computer system with requests, potentially clogging up servers until the system becomes inaccessible to legitimate users.
This is the same state that spend $60M-ish to revamp their entire unemployment system 6 years ago, only to have it completely collapse this year when 'rona landed. On Tue, Oct 6, 2020 at 4:19 PM Mike Hammett <nanog@ics-il.net> wrote:
I understand that there is underlying work that can't be sourced somewhere else, at least not trivially.
How many of these overloaded web sites that we hear about (voter registration, unemployment registration, web sites announced in a big way, causing surges in traffic, etc.) have a CDN offloading the low-hanging fruit?
I know that processing a voter registration is far more intensive than serving up static images, but surely a CDN taking the low hanging fruit would help to some degree. I'm assuming most of the people running these sites are clueless and haven't looked at this, but maybe they have.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> ------------------------------ *From: *"Sean Donelan" <sean@donelan.com> *To: *nanog@nanog.org *Sent: *Tuesday, October 6, 2020 11:51:39 AM *Subject: *Florida: Voter registration website overwhelmed at deadline
Every election has problems. Most of the time, those problems aren't noticed. Elections rely on a lot of back-end infrastructure, besides the actual voting itself.
It could be a DDOS attack, or simply duct-taped systems having trouble with the load.
Voting early (mail, drop-off, in-person) means more time to fix glitches.
https://apnews.com/article/virus-outbreak-election-2020-florida-elections-ro...
The FBI and the Cybersecurity and Infrastructure Security Agency warned elections officials nationwide last week that cyberattacks could disrupt their systems during the run-up to the election. They particularly noted “distributed denial-of-service” attacks, which inundate a computer system with requests, potentially clogging up servers until the system becomes inaccessible to legitimate users.
On Tue, 6 Oct 2020, Mike Hammett wrote:
How many of these overloaded web sites that we hear about (voter registration, unemployment registration, web sites announced in a big way, causing surges in traffic, etc.) have a CDN offloading the low-hanging fruit?
Florida uses a CDN, like essentially all states. There is also lot of duct-tape, inadequate funding and human error across all states (and probably no different for governments world-wide). I see problems with overloaded state/local emergency alert sites all the time. Periods of extreme stress is when many systems break. Its not necessarily due to malfeasance or malicious. Expect there will be problems with election infrastructure through Election Day, and afterwards. I fully expect US state/local election administrators to make it work this time too (eventually, maybe a few days or weeks). Be on the look out for clever nefarious actors. But the dumb mistake is the more likely explanation for most problems. And in either case, Murphy's Law applies. Have a backup plan because things will go wrong.
Those projects are always given to the lowest bidder, not the best (see ACA). ( Or the "friendliest" depending if the requirements are spelling out the name of the highest donator of which ever party is in power =D ) ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 2020-10-06 16:17, Mike Hammett wrote:
I understand that there is underlying work that can't be sourced somewhere else, at least not trivially.
How many of these overloaded web sites that we hear about (voter registration, unemployment registration, web sites announced in a big way, causing surges in traffic, etc.) have a CDN offloading the low-hanging fruit?
I know that processing a voter registration is far more intensive than serving up static images, but surely a CDN taking the low hanging fruit would help to some degree. I'm assuming most of the people running these sites are clueless and haven't looked at this, but maybe they have.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> ------------------------------------------------------------------------ *From: *"Sean Donelan" <sean@donelan.com> *To: *nanog@nanog.org *Sent: *Tuesday, October 6, 2020 11:51:39 AM *Subject: *Florida: Voter registration website overwhelmed at deadline
Every election has problems. Most of the time, those problems aren't noticed. Elections rely on a lot of back-end infrastructure, besides the actual voting itself.
It could be a DDOS attack, or simply duct-taped systems having trouble with the load.
Voting early (mail, drop-off, in-person) means more time to fix glitches.
https://apnews.com/article/virus-outbreak-election-2020-florida-elections-ro...
The FBI and the Cybersecurity and Infrastructure Security Agency warned elections officials nationwide last week that cyberattacks could disrupt their systems during the run-up to the election. They particularly noted “distributed denial-of-service” attacks, which inundate a computer system with requests, potentially clogging up servers until the system becomes inaccessible to legitimate users.
This is not the first, and won't be the last, problem that happens during this election season. Unless there is substantial and credible evidence otherwise, the normal reason for an technology outage will likely be the normal stupid reasons for technology outages. ======================== Florida’s chief information officer said Wednesday that misconfigured computer servers were to blame for the crash of the state’s voter registration system as the deadline approached for enrolling to cast ballots in next month’s presidential election. [...] James Grant told The Associated Press in an interview that the voter registration system worked as expected during that extra time after technicians reconfigured existing servers, expanding the network’s capacity and giving the system a “whole lot more horsepower.” He said it is unknown how many people were prevented from registering to vote during Monday’s bottleneck. Grant said no one had any intention of preventing people from registering to vote, but he acknowledged that the system failed on a critical day. “The servers were configured in a way that reduced its capacity to a fraction of a fraction of what it was capable of,” Grant said.
participants (8)
-
Alain Hebert -
Constantine A. Murenin -
Eric Litvin -
Justin Paine -
Mike Hammett -
Sean Donelan -
Tom Beecher -
Valdis Klētnieks