Virginia voter registration down due to cable cut
On the last day of Virginia voter registration, the state-wide voter registration system experienced a cable cut disrupting access to the state-wide database system. Absent clear and convincing evidence otherwise, the problems will likely be caused by the usual stupid stuff. VITA @VITAagency A fiber cut near Rt. 10 in Chester near the Commonwealth Enterprise Solutions Center (CESC) is impacting data circuits and virtual private network (VPN) connectivity for multiple Commonwealth agencies. VITA @VITAagency Technicians are on site and working to repair the cut; updates will be provided as work progresses.
<cough>spof</cough> the vita folk have a history of 'not really understanding large scale compute/network operations' :( On Tue, Oct 13, 2020 at 11:06 AM Sean Donelan <sean@donelan.com> wrote:
On the last day of Virginia voter registration, the state-wide voter registration system experienced a cable cut disrupting access to the state-wide database system.
Absent clear and convincing evidence otherwise, the problems will likely be caused by the usual stupid stuff.
VITA @VITAagency A fiber cut near Rt. 10 in Chester near the Commonwealth Enterprise Solutions Center (CESC) is impacting data circuits and virtual private network (VPN) connectivity for multiple Commonwealth agencies.
VITA @VITAagency Technicians are on site and working to repair the cut; updates will be provided as work progresses.
On Tue, 13 Oct 2020, Christopher Morrow wrote:
<cough>spof</cough>
the vita folk have a history of 'not really understanding large scale compute/network operations' :(
Reportedly, the VITA data center and Virginia voter registration system is back up. According to VITA, a Verizon fiber was struck during a roadside utilities construction project near Route 10 in Chester, VA. As network engineers know, fiber cuts happen all the time due to construction. Malicious cuts and sabatoge occur, but are rare and usually obvious. Absent clear and compelling evidence, assume normal stupid reasons for outages. There are various long-term structural problems with election administration across 10,000+ jurisdictions in the United States. A lot of duct-tape and heroic work needed by election administrators to keep things running. Election administration in Australia, Norway and Luxembourg tend to score the best with 10 out of 10 according to international election observers. Election administration in the United States of American tends to score around 7 or 8. There will be problems and delays. Not perfect, embarrassing and USA should do better. But still a full and free election.
On Tue, Oct 13, 2020 at 2:41 PM Sean Donelan <sean@donelan.com> wrote:
On Tue, 13 Oct 2020, Christopher Morrow wrote:
<cough>spof</cough>
the vita folk have a history of 'not really understanding large scale compute/network operations' :(
Reportedly, the VITA data center and Virginia voter registration system is back up.
According to VITA, a Verizon fiber was struck during a roadside utilities construction project near Route 10 in Chester, VA. As network engineers know, fiber cuts happen all the time due to construction. Malicious cuts and sabatoge occur, but are rare and usually obvious. Absent clear and compelling evidence, assume normal stupid reasons for outages.
sorry I meant that: 1) yes clearly it's still the middle of roadwork/backhoe season, 2) i'm surprised that a single path failure for their production datacenter was enough to take the system offline. 'spof' there meant: "Wow, a single point of failure in their outside plant?"
There are various long-term structural problems with election administration across 10,000+ jurisdictions in the United States. A lot of duct-tape and heroic work needed by election administrators to keep things running.
Election administration in Australia, Norway and Luxembourg tend to score the best with 10 out of 10 according to international election observers.
Election administration in the United States of American tends to score around 7 or 8. There will be problems and delays. Not perfect, embarrassing and USA should do better. But still a full and free election.
On Tue, 13 Oct 2020 17:11:53 -0400, Christopher Morrow said:
sorry I meant that: 1) yes clearly it's still the middle of roadwork/backhoe season, 2) i'm surprised that a single path failure for their production datacenter was enough to take the system offline. 'spof' there meant: "Wow, a single point of failure in their outside plant?"
Given that back in 2010, they suffered a *disastrous* outage when a storage array failed and took multiple agencies with it.... https://www.computerworld.com/article/2515423/northrop-grumman-takes-blame-f... my reaction was more like <voice="Gomer Pyle"> Surprise, surprise, surprise... </voice> That one started when one storage array had a failed memory card, and the backup array encountered issues as well. There were a number of state agencies and universities that had fought for increased self-governance, and a *huge* part of that was "not be forced to outsource their internal IT to VITA", and those units were very glad they had won that fight....
On Tue, 13 Oct 2020, Valdis Klētnieks wrote:
my reaction was more like <voice="Gomer Pyle"> Surprise, surprise, surprise... </voice>
S.N.A.F.U. Other SNAFUs, Georgia had technical problems with its voter database systems during the first couple of days of early voting. Expect all sorts of minor problems throughout the election and afterwards. Nonetheless they are unlikely to significantly impact the results (hopefully), but will generate lots of noise. Its not just underfunded state I.T. systems. Even very large social media companies can have technical Oopsies. Again, hopefully Twitter won't fall down again during the evening of November 3rd. The digeratti will lose thier minds. Even if Twitter or another major social media platform does go belly-up, most likely it will be a normal technical problem. Wishing the FBI & CISA & OGAs watch officers a very boring night on November 3rd. In other news, New Zealand is having national elections this weekend. New Zealand is usually ranked in the top 10 best election administrations worldwide. NZ expects to have the majority of ballots counted within 2 hours of their polls closing on Saturday evening. Jealous of the Kiwis and their competently run elections. :-)
Hi, Beside being: . a country with 1/10th of the population; . centralized voting rules; . <yadi yada>... PS: And there is a lot in that <yadi yada> about the (publicly) unreal amount of insanity being pulled by the GOP this year. ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 2020-10-16 02:36, Sean Donelan wrote:
On Tue, 13 Oct 2020, Valdis Klētnieks wrote:
my reaction was more like <voice="Gomer Pyle"> Surprise, surprise, surprise... </voice>
S.N.A.F.U.
Other SNAFUs, Georgia had technical problems with its voter database systems during the first couple of days of early voting. Expect all sorts of minor problems throughout the election and afterwards. Nonetheless they are unlikely to significantly impact the results (hopefully), but will generate lots of noise.
Its not just underfunded state I.T. systems. Even very large social media companies can have technical Oopsies. Again, hopefully Twitter won't fall down again during the evening of November 3rd. The digeratti will lose thier minds.
Even if Twitter or another major social media platform does go belly-up, most likely it will be a normal technical problem. Wishing the FBI & CISA & OGAs watch officers a very boring night on November 3rd.
In other news, New Zealand is having national elections this weekend. New Zealand is usually ranked in the top 10 best election administrations worldwide. NZ expects to have the majority of ballots counted within 2 hours of their polls closing on Saturday evening.
Jealous of the Kiwis and their competently run elections. :-)
It’s not the population. It’s the number of positions/things you are voting for. New Zealand doesn’t vote for sheriffs, judges, mayors etc. AFAIK so there is much less to count. More population should lead to more polling stations. These need to be collated but that is a relatively quick job compared to counting the votes. Timezone spread also makes the night longer. If you have a result within 2 hours of the Hawaiian polls closing you are on par. -- Mark Andrews
On 17 Oct 2020, at 07:49, Alain Hebert <ahebert@pubnix.net> wrote:
Hi,
Beside being:
. a country with 1/10th of the population;
. centralized voting rules;
. <yadi yada>...
PS: And there is a lot in that <yadi yada> about the (publicly) unreal amount of insanity being pulled by the GOP this year.
----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 2020-10-16 02:36, Sean Donelan wrote:
On Tue, 13 Oct 2020, Valdis Klētnieks wrote:
my reaction was more like <voice="Gomer Pyle"> Surprise, surprise, surprise... </voice>
S.N.A.F.U.
Other SNAFUs, Georgia had technical problems with its voter database systems during the first couple of days of early voting. Expect all sorts of minor problems throughout the election and afterwards. Nonetheless they are unlikely to significantly impact the results (hopefully), but will generate lots of noise.
Its not just underfunded state I.T. systems. Even very large social media companies can have technical Oopsies. Again, hopefully Twitter won't fall down again during the evening of November 3rd. The digeratti will lose thier minds.
Even if Twitter or another major social media platform does go belly-up, most likely it will be a normal technical problem. Wishing the FBI & CISA & OGAs watch officers a very boring night on November 3rd.
In other news, New Zealand is having national elections this weekend. New Zealand is usually ranked in the top 10 best election administrations worldwide. NZ expects to have the majority of ballots counted within 2 hours of their polls closing on Saturday evening.
Jealous of the Kiwis and their competently run elections. :-)
Actually we count lots of things, though we keep civic separate from state (we did that last year).We delayed our election by 4 weeks because of covid.Voting is open for a number of weeks. About 72% of voters had voted before the last day.Voting is on paper.Voters don't have to register before voting (though that is prefered).Count started on the last day and reporting started at about 7.30pm (polls closed at 7pm)Counting was mostly done by midnight.This term we also have two referendums but they won't report back until the end of October.We don't have queues because we have so many places you can vote.https://i.stuff.co.nz/national/politics/300124486/election-2020-christchurch... were 3 locations within 750m of my house.Sorry to hear about your fibre...https://i.stuff.co.nz/business/industries/122715051/return-of-the-cyberattac... don't think a fibre cut would take down anything to serious in New Zealand.... but it does depend on what you consider important. ;)DD-- Don Gould5 Cargill PlaceRichmondChristchurch, New ZealandMobile/Telegram: + 64 21 114 0699www.bowenvale.co.nz -------- Original message --------From: Mark Andrews <marka@isc.org> Date: 17/10/20 11:29 am (GMT+12:00) To: ahebert@pubnix.net Cc: nanog@nanog.org Subject: Re: Virginia voter registration down due to cable cut It’s not the population. It’s the number of positions/things you are voting for. New Zealand doesn’t vote for sheriffs, judges, mayors etc. AFAIK so there is much less to count. More population should lead to more polling stations. These need to be collated but that is a relatively quick job compared to counting the votes. Timezone spread also makes the night longer. If you have a result within 2 hours of the Hawaiian polls closing you are on par. -- Mark AndrewsOn 17 Oct 2020, at 07:49, Alain Hebert <ahebert@pubnix.net> wrote: Hi, Beside being: . a country with 1/10th of the population; . centralized voting rules; . <yadi yada>... PS: And there is a lot in that <yadi yada> about the (publicly) unreal amount of insanity being pulled by the GOP this year. ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 2020-10-16 02:36, Sean Donelan wrote: On Tue, 13 Oct 2020, Valdis Klētnieks wrote: my reaction was more like <voice="Gomer Pyle"> Surprise, surprise, surprise... </voice> S.N.A.F.U. Other SNAFUs, Georgia had technical problems with its voter database systems during the first couple of days of early voting. Expect all sorts of minor problems throughout the election and afterwards. Nonetheless they are unlikely to significantly impact the results (hopefully), but will generate lots of noise. Its not just underfunded state I.T. systems. Even very large social media companies can have technical Oopsies. Again, hopefully Twitter won't fall down again during the evening of November 3rd. The digeratti will lose thier minds. Even if Twitter or another major social media platform does go belly-up, most likely it will be a normal technical problem. Wishing the FBI & CISA & OGAs watch officers a very boring night on November 3rd. In other news, New Zealand is having national elections this weekend. New Zealand is usually ranked in the top 10 best election administrations worldwide. NZ expects to have the majority of ballots counted within 2 hours of their polls closing on Saturday evening. Jealous of the Kiwis and their competently run elections. :-)
In other news, New Zealand is having national elections this weekend. New Zealand is usually ranked in the top 10 best election administrations worldwide. NZ expects to have the majority of ballots counted within 2 hours of their polls closing on Saturday evening.
I thought the HGIC (Head Ghoul In Charge) had cancelled all the elections in New Zealand and declared herself to the new Dictator Ghoul In Charge until ousted by violent uprising by the peasants? Has there been an uprising to oust the Head Ghoul via bullet-to-the-brain that was not reported in the socialist media? -- Be decisive. Make a decision, right or wrong. The road of life is paved with flat squirrels who could not make a decision.
To keep this on topic, no mention of any candidate or political party. Networks can impact elections, and elections can impact network operators. If @netblocks <www.netblocks.org> publishes a special report about your country's elections, that's a huge indicator of network (and election) problems. In the 1950's, often the first indication something was happening was the national radio station in a country suddenly started playing uninterrupted classical music. Now the modern equivalent is when mobile networks stop working and social media sites are blocked in a country. There are many day-to-day causes of network problems. I don't expect backhoes to stop cutting fiber cables on election day. I don't expect network engineers to stop making mistakes on election day. I don't expect software bugs to stop occurring on election day. In the USA, absent clear and convincing evidence otherwise, I expect any outages will be due to the normal things that cause outages on election day. If you do experience an outage, you probably want to have your P.R. people on call for a quicker than normal response.
On Sat, Oct 17, 2020 at 07:44:01PM -0400, Sean Donelan wrote:
In the USA, absent clear and convincing evidence otherwise, I expect any outages will be due to the normal things that cause outages on election day.
One of those things is the chronic underfunding of the systems/personnel involved. (This isn't the case everywhere of course but it's the case in a lot of places.) This leads to operations that are working -- barely -- and thus not resistant to stress, outages, and mistakes. We could argue that given the criticality of this particular function that resources should be more generously allocated, but unfortunately that's often an unsuccessful argument. ---rsk
In my experiences with both government IT and elections departments, funding may be a problem, but a definite problem is an unwillingness to relieve poorly performing staff. They could be doing so much better with the funding they have. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Rich Kulawiec" <rsk@gsp.org> To: nanog@nanog.org Sent: Monday, October 19, 2020 11:58:23 AM Subject: Re: Virginia voter registration down due to cable cut On Sat, Oct 17, 2020 at 07:44:01PM -0400, Sean Donelan wrote:
In the USA, absent clear and convincing evidence otherwise, I expect any outages will be due to the normal things that cause outages on election day.
One of those things is the chronic underfunding of the systems/personnel involved. (This isn't the case everywhere of course but it's the case in a lot of places.) This leads to operations that are working -- barely -- and thus not resistant to stress, outages, and mistakes. We could argue that given the criticality of this particular function that resources should be more generously allocated, but unfortunately that's often an unsuccessful argument. ---rsk
Again, hopefully Twitter won't fall down again during the evening of November 3rd. The digeratti will lose thier minds.
No offense towards our many friends at Twitter, but I think the best thing that could happen would be for Twitter to shut 'er down from November 1 through November 6. Remove one of the primary avenues for rapid disinformation and confusion, and allow tabulations to proceed without extra nonsense. This of course won't happen, but one can dream. On Fri, Oct 16, 2020 at 2:37 AM Sean Donelan <sean@donelan.com> wrote:
On Tue, 13 Oct 2020, Valdis Klētnieks wrote:
my reaction was more like <voice="Gomer Pyle"> Surprise, surprise, surprise... </voice>
S.N.A.F.U.
Other SNAFUs, Georgia had technical problems with its voter database systems during the first couple of days of early voting. Expect all sorts of minor problems throughout the election and afterwards. Nonetheless they are unlikely to significantly impact the results (hopefully), but will generate lots of noise.
Its not just underfunded state I.T. systems. Even very large social media companies can have technical Oopsies. Again, hopefully Twitter won't fall down again during the evening of November 3rd. The digeratti will lose thier minds.
Even if Twitter or another major social media platform does go belly-up, most likely it will be a normal technical problem. Wishing the FBI & CISA & OGAs watch officers a very boring night on November 3rd.
In other news, New Zealand is having national elections this weekend. New Zealand is usually ranked in the top 10 best election administrations worldwide. NZ expects to have the majority of ballots counted within 2 hours of their polls closing on Saturday evening.
Jealous of the Kiwis and their competently run elections. :-)
On Tue, Oct 13, 2020 at 5:15 PM Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Tue, Oct 13, 2020 at 2:41 PM Sean Donelan <sean@donelan.com> wrote:
On Tue, 13 Oct 2020, Christopher Morrow wrote:
<cough>spof</cough>
the vita folk have a history of 'not really understanding large scale compute/network operations' :(
Reportedly, the VITA data center and Virginia voter registration system is back up.
According to VITA, a Verizon fiber was struck during a roadside utilities construction project near Route 10 in Chester, VA. As network engineers know, fiber cuts happen all the time due to construction. Malicious cuts and sabatoge occur, but are rare and usually obvious. Absent clear and compelling evidence, assume normal stupid reasons for outages.
sorry I meant that: 1) yes clearly it's still the middle of roadwork/backhoe season, 2) i'm surprised that a single path failure for their production datacenter was enough to take the system offline. 'spof' there meant: "Wow, a single point of failure in their outside plant?"
... and there is now a lawsuit about this: Groups file lawsuit after Virginia state websites shut down for hours on last day of voter registration "An accidentally severed fiber-optic cable in Virginia effectively shut down most of the state’s online voter registration on its last day Tuesday, prompting voter advocates to file a lawsuit in federal court seeking an extension of the deadline that they argue thousands of voters missed because of the disruption."... https://www.washingtonpost.com/local/va-politics/virginia-state-websites-shu... W
There are various long-term structural problems with election administration across 10,000+ jurisdictions in the United States. A lot of duct-tape and heroic work needed by election administrators to keep things running.
Election administration in Australia, Norway and Luxembourg tend to score the best with 10 out of 10 according to international election observers.
Election administration in the United States of American tends to score around 7 or 8. There will be problems and delays. Not perfect, embarrassing and USA should do better. But still a full and free election.
-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf
participants (11)
-
Alain Hebert -
Christopher Morrow -
Don Gould -
Keith Medcalf -
Mark Andrews -
Mike Hammett -
Rich Kulawiec -
Sean Donelan -
Tom Beecher -
Valdis Klētnieks -
Warren Kumari