On 2/3/23 19:25, Brian Turnbow via NANOG wrote:

They have been discussing it here in Italy as well. The isp/telecommunication industry here is tryng to get Cos/pops/cabinets listed as critical infra and removed from rolling power cuts.

I would say plan for the worst, because there will always be some other department or governmental function that says they are more critical than the next one. And even if they may designate certain functions as critical, electrical distribution grids are not always built to that degree of granularity, and those critical functions are often caught in the crossfire as other non-critical loads are shed from the network. And even after all that, if the blackouts need to intensify, all bets are off, since the main purpose then becomes preventing grid frequency drop, rather than servicing loads.

Here this is mainly ran from pops that have ups and generator systems so several hours to days of uptime depending on site.

If major data centres need to spend more on fuel than they planned for, I'd suggest making a generous allocation for an increase in co-lo costs for your next budget cycle, as the data centres will, invariably, pass those costs on the longer the city has to shed power from the grid.

OTOH I have seen providers daisy chain customer sites in a ring that crash miserably when 2 customers loose power isolating all in between sites. But that is not the norm...

This is one of the reasons we refuse to turn customer sites into Metro-E PoP's, or PoP's of any kind.

Street cabinets for fttc services here have low times if any. Same thing for mini dslams mounted on poles in the middle of nowhere. 0 to 2 hours for these. Most have batteries/capacitors in the cabinet but not all and they are not designed for extended power outages 2 hours max. Some are remotely powered from the CO, but that does not seem to be a thing anymore. Too costly DSL ran from COs are protected as for fiber above.

A lot of this will be driven by what competitors do. If there is no competition that can keep their street cabinets going, the others won't. It's a great opportunity for anyone willing to make lemonade out of the situation.

Don't operate a 4g network, so take this info accordingly , but here it depends on the tower from what I have seen. All towers I have seen have battery backup , a lot have generators too. I would say they have higher times than the fttc times above.

In dense metro's, mobile sites will be well invested. It starts to get tricky when you go out into the sticks. Also, when the power goes out, so does the wi-fi. That means everybody moves their traffic away from the home wi-fi and on to the nearest cell tower. While radio bandwidth and signal coverage does not suffer that much, it hits the backhaul hard, between the tower and the mobile carrier's core. So nice flashing LTE/4G/5G signals on your phone translates to GPRS-esque performance. In some cases, we have seen mobile operators downgrade radio coverage to 3G, in order to manage this. Who knew 3G performed a bad as GPRS or EDGE, in 2023 :-)? In the most extreme of case, the cell site could run out of power as well, as there isn't enough time for the batteries to recharge between outage cycles, or the field teams can't replenish fuel for the generators in time. In the absolutely extreme cases (as we see here in South Africa, for example), cell sites can be raided and batteries stolen, especially if there is darkness all around. I would not expect this to be the case in the UK, especially if power outages are not the norm and people live a fairly middle-class life, but if I were Vodafone, for example, I'd have my risk department planning for such an eventuality already. Mark.

On 2/3/23 19:53, Saku Ytti wrote:

In practice I would default to expecting 0 min availability during power outage, regardless of how resilient my CPE is. We can scarcely make the Internet work at the best of times.

Agreed, this is a good place to start. It's a bit doom & gloom, but most people underestimate just how much work it takes power companies to generate and distribute energy. Consequently, they never have to think about generation and/or load management, if they had to do it on their own, meaning they will usually consider the cheapest solution possible, which for electricity, usually ends in tears. Mark.

On 2/3/23 21:11, Sabri Berisha wrote:

Living in an area served by PG&E, I've had my share of power cuts. At home I have a 600va UPS that protects my cable modem, RPI router, and POE switch which serves 2 APs. That lasts about 30 minutes, which gives me enough time to fire up my generator.

I'd assume it doesn't take you that long to fire up the genie, if you are home when the power goes out :-). Out of interest, depending on how long you've had the UPS, how many times have you changed the battery?

Tip of the day: I also have a 1000va UPS that protects my garage door opener. This makes it a lot easier to a. get a car out if needed, and b. get my generator out of the garage.

In South Africa, garage door motors historically come standard with a 12V 7Ah Lead Acid battery. What most people don't realize is that within 1.5 to 2 years, those batteries are dead, and since there was power most of the time, they never noticed, until the power went out and the battery did not have sufficient energy to drive the motor. It is not uncommon to see folk here moving to Li-Ion batteries with the same capacity to drive garage and gate motors. Personally, I try to only use Li-Ion packs in well ventilated, not-too-tight spaces, that come with a BMS and a charger that is predictable :-). So I avoid these little ones, and feel safer with Lead Acid batteries for garage and gate motors.

Lastly, in the spirit of happy wife, happy life, I have another 600va UPS that covers my tankless water heater. It heats using natural gas, but the control panel still needs power. That thing lasts pretty long.

There are tankless water heaters that can take standard AA batteries to spark the igniter as well, but yes, if yours is electric, makes sense to put a UPS on it. At my place, I use both a PV-based controller to heat traditional water tanks, and also have just as many tankless water heaters. The former is for end-of-day showers and dish washing once the sun has done its thing, the latter is for the morning jobs for the same things, or if it's cloudy/rainy outside. Even though I have whole-home backup, I did place a UPS on the PV-based controller, because even if it is connected to solar panels, it is still, in essence, an inverter, meaning it requires a grid before it can generate solar power. That is what the UPS does for it. It was my first deployment of renewable energy, so I needed the UPS before I backed up the entire home.

YMMV, of course, but I went through numerous outages recently. And by numerous, I mean enough for our City leadership to get pissed off at PG&E and demand explanations.

Judging by our situation, I'd take the city leadership demanding explanations any day :-). We are way past that, down here. It's so bad, folk are deploying self-generation and self-storage by the truck loads. It's the gold rush for solar and battery installers right now. People have all but given up on the power company (Eskom) ever going back to its glory days.

So far, my current ISP (Spectrum cable) has had 0 outages as a result of power loss. Which is pretty impressive, given the instability of the grid in this area.

Not bad. Mark.

On 2/4/23 07:29, William Herrin wrote:

If it's just a little gasoline generator, 30 minutes is about right. It takes 10 minutes to decide the power isn't coming back soon and another 10 to drag the generator out of the shed, hook up the wires and get it going even though it's cold, wet, and hasn't been run for several months. That leaves 10 minutes to spare figuring out how to convince the UPS that the generator power is good enough to retransfer and stay.

Indeed - I was guessing given how reliable PG&E have been for Sabri, a lot is probably pre-wired. I may be wrong. Mark.

On 2/4/23 07:48, William Herrin wrote:

Pre-wired makes it a standby generator, which 9 times out of 10 is automatic start with an automatic transfer switch. It's running within seconds whether you're home or not. Electricians cost too much and 20kva natural gas / propane generators with an ATS don't cost enough more than the portables.

Compare:

https://www.costco.com/honeywell-18kw-home-standby-generator-with-transfer-s...

and:

https://www.amazon.com/Honda-2200-Watt-120-Volt-Portable-Generator/dp/B079YF...

understanding that an electrician will cost you $2000-$3000 for the labor with any genset modification to the house wiring.

Aware of all this - I operate one or two submarine cable landing stations... What I mean by "pre-wired" is that, perhaps, the generator is pre-setup and wired into the house, but is not in standby mode to manage costs, and perhaps, to be reliable since ATS's are often dodgy. Maybe a manual start is required. Maybe a changeover switch has to be flipped. That sort of thing. Of course, we are speculating, and Sabri can answer best. Mark.

On 2/4/23 08:11, William Herrin wrote:

Not for more than a decade now, at least not in the U.S. When you're up to whole-house generator prices everyone expects electric start. Half the portables have electric start. Most lawnmowers have electric start. Once you have that, the cost to make the switch automatic instead of manual is trivial. And the mass-produced consumer-grade switches are quite reliable.

Fair point, but it is not uncommon (YMMV) for backup solutions to not be whole-home, and this includes generators. I have not lived in the U.S. on any meaningfully extended basis, so I can't speak to the degree to which folk that install backup power choose their cost/value matrix. But in many other places I've lived, especially where the power is frequently out, splitting the house loads into different panels a generator or small solar/battery system can support is commonplace; mainly for convenience but also to reduce the chances of a fire or electrical shock from having to run wires ad hoc. The same would also apply to renewables, especially where budget is limited and folk can't afford to backup the entire home. This is often cheaper than investing in large backup solutions, while still providing some degree of convenience to power what one would consider critical items, whatever that means to them. I've been helping quite a number of folk wire small inverters with limited power and backup battery time into DB boards that feed only lights and plugs to specifically drive wi-fi, TV, an IP phone and a fridge. The inverters and accompanying battery aren't terribly expensive here (US$500 - US$800 for a Lead Acid system, and up to US$1,400 for a Li-Ion one), and labour in Africa is dirt cheap (less than US$100 for an installation), so it's budget-friendly, and keeps basics going. One could even charge a laptop. In general, the main things folk will not backup are electric stoves, electric ovens, electric water heaters, electric space heaters and air conditioners. Obviously, in places with winter periods, serious plans have to be made to avoid death from cold. Mark.

On 2/4/23 23:58, Sabri Berisha wrote:

I'd say I have something in between. I have a WEN GN875i: https://www.amazon.com/WEN-GN875i-Transfer-Switch-Ready-8750-Watt-Generator/...

That's 7kw rated and 8.75kw peak. More than enough to support my home.

Yeah, plenty of juice.

I previously had one of those smaller 2200 watt generators. The problem with those is that you're now limited to 1600 watt running, which barely powers the fridge, lights, internet, and maybe some tv. Our power usually goes out when it's very warm, so I like some AC.

Makes sense.

Mine is electrical (but not automatic) start, I have to flip the main and a circuit breaker, which is protected by an interlock switch. Similar to https://www.amazon.com/Generator-Interlock-Compatible-panels-Professional-In...

The interlock switch ensures that I'm not backfeeding to the grid, and was necessary to pass inspection.

Usually I have it up and running within 10 minutes. That's how long it takes for my UPS script to kick in and start shutting down servers.

Awesome! Thanks for sharing. Mark.

On 2/5/23 07:02, Roy wrote:

My all electric house is in a rural area.  The generator that came with the place is a 20KW Onan,  The bad news is in can't handle the house.  I think it is the Aux Heat on the heat pump that is the problem.  I have to also power the well pump and the septic pump.

Is your house single or 3-phase? I'd be curious how much horsepower your well and septic pumps require. The most I've seen is 15hp @ 11kW, but that is pretty massive for an average home, even an off-grid one. Typical requirements would be in 0.75kW - 5kW range, which is a wide range. Do you know how much power the heat pump requires? I'd struggle to see how a 20kW generator struggles to to run a home, unless you've also got heated floors, saunas, steam baths, water and space heaters, electric stoves and ovens all running at the same time :-). Mark.

On Sun, 5 Feb 2023 at 07:50, Chris Adams <cma@cmadams.net> wrote:

Electric heat pumps are great for power efficiency until the temperature drops and they switch over to pure electric heat.

Here is graph from popular air heat pump Mitsubishi MSZ/MUZ 25 https://scanoffice.fi/wp-content/uploads/2022/09/rw-vtt-tuntikeskiarvo.jpg https://scanoffice.fi/wp-content/uploads/2022/09/rw-vttn-testitulos.png At -30c external, with +20c internal the units produce heat at approximately 2x of the electric input. But many other units do not perform that well even at -20c external. And these units are premium priced. Modern R32 units consistently outperform old R410A units. -- ++ytti

On 2/4/2023 9:31 PM, Mark Tinka wrote:

On 2/5/23 07:02, Roy wrote:

...

My all electric house is in a rural area.  The generator that came with the place is a 20KW Onan,  The bad news is in can't handle the house.  I think it is the Aux Heat on the heat pump that is the problem.  I have to also power the well pump and the septic pump.

Is your house single or 3-phase?

Single phase.  The house is 200A service and the barn is another 200A service

I'd be curious how much horsepower your well and septic pumps require. The most I've seen is 15hp @ 11kW, but that is pretty massive for an average home, even an off-grid one. Typical requirements would be in 0.75kW - 5kW range, which is a wide range.

Do you know how much power the heat pump requires?

I don't know how much the pumps require.  The water well is about 100 feet from the house and the pressure tank. The septic pump has to pump uphill to the drainage field.  Distance is about 250 feet and elevation gain of 100 feet or so. The heat pump doesn't seem to be a problem but the aux heat is on two 20amp 220v circuits.   There is a switch on the fan enclosure to disable the aux heat. Another biggie is the electric hot water heater. On 1/30 it never broke 32 degrees and the house used 145KWHR (average was 6KWH).  Thank goodness I am not far from the Columbia River and the BPA has a major substation about 5 miles away so I pay less than 10 cents per KWH Over 2022, I lost power about 8 times.  The longest outage was 15 hours.

I'd struggle to see how a 20kW generator struggles to to run a home, unless you've also got heated floors, saunas, steam baths, water and space heaters, electric stoves and ovens all running at the same time :-).

Mark.

On 2/5/23 09:32, John van Oppen wrote:

20 KW should easily cover the 9KW you could max draw with your strip heat. It is super uncommon to have even peak loads over 20 KW in a house. Even your peak day was only an average of 6 KW.

You might need some load shedding just to keep the big stuff from coming on all at once but that is pretty easy. If you have instant hot water that also could be a problem those are huge, typically 15-20 KW by themselves.

I agree - sounds like Roy would benefit from either doing some load shedding and/or switching out gear that is more power-efficient. Mark.

On 2/5/23 11:12, William Herrin wrote:

Hi Roy,

My guess is that your 20kw Onan isn't up to stably producing 20kw any more. Or perhaps the older transfer switch with its mechanical timer relays has gotten dodgy. The modern consumer gear is very reliable but the pre-2010s commercial/industrial gear (such as Onan) takes a lot of TLC to keep it working right.

Another thing that can happen is uneven load. The utility won't care that 80% of your draw is on one side of the circuit but that'll stop a generator in its tracks.

If you want to know for sure, get yourself an inductive clamp meter and start checking the amperages at the breakers. You can get a cheap one under 50 bucks. You just set it for AC amps, take the cover off your breaker panel and clamp the jaws around the "hot" wires coming from each breaker, one at a time.

As John mentioned, unless you have a really large house or a tankless electric water heater, it's really odd to overwhelm a 20kw generator. Even with the auxiliary resistive heat.

Agree with this assessment. 20kW rated is plenty of energy for even a demanding home. Mark.

----- On Feb 4, 2023, at 2:10 PM, Mark Tinka mark@tinka.africa wrote:

On 2/4/23 23:58, Sabri Berisha wrote:

Hi,

...

Usually I have it up and running within 10 minutes. That's how long it takes for my UPS script to kick in and start shutting down servers.

Awesome!

There were few raindrops, so we have an outage. Again. I timed it. It took me less than 4 minutes to get it up and running. Oh, and you were right about the UPS batteries. The UPS on top of my garage door opener died halfway through opening the door. Silicon Valley, the most technological place on earth, and I can't even have stable power. Thanks, Sabri

----- On Feb 3, 2023, at 9:05 PM, Mark Tinka mark@tinka.africa wrote:

On 2/3/23 21:11, Sabri Berisha wrote:

Hi Mark,

...

Living in an area served by PG&E, I've had my share of power cuts. At home I have a 600va UPS that protects my cable modem, RPI router, and POE switch which serves 2 APs. That lasts about 30 minutes, which gives me enough time to fire up my generator.

I'd assume it doesn't take you that long to fire up the genie, if you are home when the power goes out :-).

Yes, there have been times where I wasn't at home.

Out of interest, depending on how long you've had the UPS, how many times have you changed the battery?

All the "small" ones, I bought in 2019, they still work fine. I have one larger UPS for my homelab in my garage that I've had since 2014; I changed the batteries in that last year.

...

Tip of the day: I also have a 1000va UPS that protects my garage door opener. This makes it a lot easier to a. get a car out if needed, and b. get my generator out of the garage.

In South Africa, garage door motors historically come standard with a 12V 7Ah Lead Acid battery. What most people don't realize is that within 1.5 to 2 years, those batteries are dead, and since there was power most of the time, they never noticed, until the power went out and the battery did not have sufficient energy to drive the motor.

Those must be different from ours, because we don't have that...

...

So far, my current ISP (Spectrum cable) has had 0 outages as a result of power loss. Which is pretty impressive, given the instability of the grid in this area.

Not bad.

Pretty impressive. How do they do that in SA? Thanks, Sabri