https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678 This article details some of the issues with California's "new reality" of planned blackouts. One of the big things that came to light with these blackouts is that our network infrastructure's resilience is pretty lacking. While I was (surprisingly to me) ok with my DSL connection out in the boonies, lots and lots of people with cable weren't so lucky. And I'm not sure how bad the situation is with cellular infrastructure, but I assume it's not much better than cable. And I wouldn't doubt that other DSL deployments go dark when power is down. I have no clue with fiber. So I guess what I'm wondering is what can we do about this? What should we do about this? These days IP access is not just convenience, it's the way we go about our lives, just like electricity itself. At base, it seems to me that network operators should be required to keep the lights on in blackouts just like POTS operators do now. If I have power to light my modem or charge in my phone, I should be able to get onto the net. That seems like table stakes. One of the things we learned also is that the blackouts seem to last between 2-3 days apiece. I happen to have a generator since I'm out in the boonies and our power gets cut regularly because of snow, but not everyone has that luxury. I kind of want to think that my router+modem use about 20 watts, so powering it up would take about 1.5kwh for 3 days. a quick google look shows that I'd probably need to shell out $500 or so for a battery of that capacity, and that's doesn't include your phones, laptops, tv's, etc power needs. What does that mean? That is a major expense for a lot of people. On the bright side, I hear that power generator companies stocks have gone through the roof. On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh. Mike
On Wed, 25 Dec 2019 at 19:32, Michael Thomas <mike@mtcc.com> wrote:
On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh.
Do you have a source for this? It would seem that these power issues are rather unique to California not because of some "climate change" bogeyman, but rather because of a failed public policy at the state level. It would also seem that these issues of rolling blackouts aren't even new to California, either, as, apparently, it's already been the norm during 2000/2001: * https://en.wikipedia.org/wiki/California_electricity_crisis C.
On Wed, Dec 25, 2019 at 19:00 Constantine A. Murenin <mureninc@gmail.com> wrote:
On Wed, 25 Dec 2019 at 19:32, Michael Thomas <mike@mtcc.com> wrote:
On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh.
Do you have a source for this? It would seem that these power issues are rather unique to California not because of some "climate change" bogeyman, but rather because of a failed public policy at the state level.
It would also seem that these issues of rolling blackouts aren't even new to California, either, as, apparently, it's already been the norm during 2000/2001:
Having lived through the blackouts that was entirely different. 90% Enron manipulating the markets. There was plenty of capacity both in transmission and generation, but Enron manipulated prices and apparent supply to make money and screwed the whole state over. There was just about 2x the generating capacity, no real shortage. This time it’s PG&E all alone, but still fallout from back then. Too much liability and they’ve not maintained the infrastructure and so they decided that to reduce the liability costs it’s cheaper to blackout. Same story again different colors. PG&E making a mint while people get screwed (PG&E was mostly at the getting screwed end in 2000-2001)
* https://en.wikipedia.org/wiki/California_electricity_crisis
C.
-- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
On 12/25/19 6:16 PM, Michael Loftis wrote:
Having lived through the blackouts that was entirely different. 90% Enron manipulating the markets. There was plenty of capacity both in transmission and generation, but Enron manipulated prices and apparent supply to make money and screwed the whole state over. There was just about 2x the generating capacity, no real shortage.
This time it’s PG&E all alone, but still fallout from back then. Too much liability and they’ve not maintained the infrastructure and so they decided that to reduce the liability costs it’s cheaper to blackout. Same story again different colors. PG&E making a mint while people get screwed (PG&E was mostly at the getting screwed end in 2000-2001)
Yes, this is exactly right. My point here isn't to assign blame, but to ask what the hell we're going to do about it. Trying to score political points is disgusting. Mike
On 12/25/19 6:29 PM, Michael Thomas wrote:
Yes, this is exactly right. My point here isn't to assign blame, but to ask what the hell we're going to do about it. Trying to score political points is disgusting.
Do you live in California? Do you have your business in California? Take a look at neighboring states. I did. California madness is why I now live in Nevada. Our ecology doesn't have the Austrailian plant Eucalyptus. We do have tumbleweeds which pose their own fire risk. Place like Lake Tahoe is heavily forested -- the difference is that in Nevada there is active fuel control and controlled burns, so we have fewer burn-to-the-ground fires in populated areas. I used to make a living as a freelance writer. A GOOD living. In Nevada I'm outside the reach of CA AB 5 should I want to give up $DAYJOB. When I have to subcontract freelance work, I won't hire a California resident, because I don't want to be controlled by any "long-reach" laws. Because the law and Calufornia court decisions are currently silent about C, S, and LLC writers, I avoid them until the climate becomes clearer. Do I experience power outages? Yes. Longest duration? Several hours, when high winds cause damage (but that damage doesn't start wildfires). One very nice thing is that where I am, we have a geothermal power plant close by (on the order of 15 miles). No pre-emptive wide-area shutdowns, though. NV Energy has photoelectric arrays that are part of the utility, not only on private roof-tops.
On 12/25/19 7:10 PM, Stephen Satchell wrote:
On 12/25/19 6:29 PM, Michael Thomas wrote:
Yes, this is exactly right. My point here isn't to assign blame, but to ask what the hell we're going to do about it. Trying to score political points is disgusting.
Do you live in California?
Yes. Mike
On Wed, 25 Dec 2019 at 20:29, Michael Thomas <mike@mtcc.com> wrote:
On 12/25/19 6:16 PM, Michael Loftis wrote:
Having lived through the blackouts that was entirely different. 90% Enron manipulating the markets. There was plenty of capacity both in transmission and generation, but Enron manipulated prices and apparent supply to make money and screwed the whole state over. There was just about 2x the generating capacity, no real shortage.
This time it’s PG&E all alone, but still fallout from back then. Too much liability and they’ve not maintained the infrastructure and so they decided that to reduce the liability costs it’s cheaper to blackout. Same story again different colors. PG&E making a mint while people get screwed (PG&E was mostly at the getting screwed end in 2000-2001)
Yes, this is exactly right. My point here isn't to assign blame, but to ask what the hell we're going to do about it. Trying to score political points is disgusting.
Mike
The same thing we've always done and recommended — Vote With Your Wallet. Move to state that takes care of its infrastructure and doesn't have such a gridlock. Or remain in California if you think "climate deniers" (whatever that term may mean) are "disgusting". As a consumer and internet infrastructure operator, I don't particularly see or care about the difference between PG&E getting screwed or doing the screwing. It's the populace of the state that gets the resulting fallout in terms of the rolling blackouts. Which other state has had this in the last 20 years? I'm an ex-California resident myself here — voted with my wallet already. Love the idea and implementation of an independent power grid of my new home state. C.
On Wed, 25 Dec 2019 at 22:08, Michael Thomas <mike@mtcc.com> wrote:
On 12/25/19 7:26 PM, Constantine A. Murenin wrote:
I'm an ex-California resident myself here
Good riddance. This has nothing to do with the climate change that is actually happening here.
Mike
Well, enjoy your climate change! Surely has nothing to do with the ban of preemptive burning that works in adjacent states just fine! C
Same story again different colors. PG&E making a mint while people get screwed
I'm not quite sure that's an accurate statement. In 2000-2001, PG&E got screwed by Enron's market manipulation. ( Good job those who pushed so hard for deregulation of public utility services! ) PG&E is currently in bankruptcy proceedings, largely as a result of liabilities from wildfires in 2017 and 2018. Under California's application of inverse condemnation, a power utility is responsible for any damage caused by a wildfire if it was determined that their equipment was part of the cause. This applies even if the utility was in 100% compliance with all laws and regulations. So you have a terrible combination where housing prices in the state are driving more and more people to build in wildfire prone areas, climate change is increasing the frequency of weather conditions favorable to wildfire ignition, and the utility company that is being held financially liable for damages while at the same time not being allowed by the PUC to raise capital for infrastructure changes to reduce the chances of electrical equipment starting such things. The answer is easy. Money. If people want a power grid that is safe and reliable, then the utility should be given the funds to do it via rates and appropriate tax revenues. They should not be expected to turn profits like private enterprise. The power grid is for the benefit of all, not just the financial benefit of those who have equity stakes. This situation is the logical extension of 40+ years of America's only real product ; financial engineering. On Wed, Dec 25, 2019 at 9:18 PM Michael Loftis <mloftis@wgops.com> wrote:
On Wed, Dec 25, 2019 at 19:00 Constantine A. Murenin <mureninc@gmail.com> wrote:
On Wed, 25 Dec 2019 at 19:32, Michael Thomas <mike@mtcc.com> wrote:
On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh.
Do you have a source for this? It would seem that these power issues are rather unique to California not because of some "climate change" bogeyman, but rather because of a failed public policy at the state level.
It would also seem that these issues of rolling blackouts aren't even new to California, either, as, apparently, it's already been the norm during 2000/2001:
Having lived through the blackouts that was entirely different. 90% Enron manipulating the markets. There was plenty of capacity both in transmission and generation, but Enron manipulated prices and apparent supply to make money and screwed the whole state over. There was just about 2x the generating capacity, no real shortage.
This time it’s PG&E all alone, but still fallout from back then. Too much liability and they’ve not maintained the infrastructure and so they decided that to reduce the liability costs it’s cheaper to blackout. Same story again different colors. PG&E making a mint while people get screwed (PG&E was mostly at the getting screwed end in 2000-2001)
* https://en.wikipedia.org/wiki/California_electricity_crisis
C.
--
"Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
I'm pretty sure political bickering is well beyond the scope of the mailing list. Is anyone moderating this? - Mike Bolitho On Thu, Dec 26, 2019, 7:20 AM Tom Beecher <beecher@beecher.cc> wrote:
Same story again different colors. PG&E making a mint while people get
screwed
I'm not quite sure that's an accurate statement.
In 2000-2001, PG&E got screwed by Enron's market manipulation. ( Good job those who pushed so hard for deregulation of public utility services! )
PG&E is currently in bankruptcy proceedings, largely as a result of liabilities from wildfires in 2017 and 2018. Under California's application of inverse condemnation, a power utility is responsible for any damage caused by a wildfire if it was determined that their equipment was part of the cause. This applies even if the utility was in 100% compliance with all laws and regulations.
So you have a terrible combination where housing prices in the state are driving more and more people to build in wildfire prone areas, climate change is increasing the frequency of weather conditions favorable to wildfire ignition, and the utility company that is being held financially liable for damages while at the same time not being allowed by the PUC to raise capital for infrastructure changes to reduce the chances of electrical equipment starting such things.
The answer is easy. Money. If people want a power grid that is safe and reliable, then the utility should be given the funds to do it via rates and appropriate tax revenues. They should not be expected to turn profits like private enterprise. The power grid is for the benefit of all, not just the financial benefit of those who have equity stakes.
This situation is the logical extension of 40+ years of America's only real product ; financial engineering.
On Wed, Dec 25, 2019 at 9:18 PM Michael Loftis <mloftis@wgops.com> wrote:
On Wed, Dec 25, 2019 at 19:00 Constantine A. Murenin <mureninc@gmail.com> wrote:
On Wed, 25 Dec 2019 at 19:32, Michael Thomas <mike@mtcc.com> wrote:
On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh.
Do you have a source for this? It would seem that these power issues are rather unique to California not because of some "climate change" bogeyman, but rather because of a failed public policy at the state level.
It would also seem that these issues of rolling blackouts aren't even new to California, either, as, apparently, it's already been the norm during 2000/2001:
Having lived through the blackouts that was entirely different. 90% Enron manipulating the markets. There was plenty of capacity both in transmission and generation, but Enron manipulated prices and apparent supply to make money and screwed the whole state over. There was just about 2x the generating capacity, no real shortage.
This time it’s PG&E all alone, but still fallout from back then. Too much liability and they’ve not maintained the infrastructure and so they decided that to reduce the liability costs it’s cheaper to blackout. Same story again different colors. PG&E making a mint while people get screwed (PG&E was mostly at the getting screwed end in 2000-2001)
* https://en.wikipedia.org/wiki/California_electricity_crisis
C.
--
"Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
On 12/26/19 7:51 AM, Mike Bolitho wrote:
I'm pretty sure political bickering is well beyond the scope of the mailing list. Is anyone moderating this?
It certainly wasn't my intent or desire to have this turn political, and shame on the person who did. This is a serious networking related issue for California *right* *now*. It may become a serious networking related issue for a lot of other places too -- California is hardly unique in its wildland - urban interface issues, and lots of places burn just like California. And definitely lots of places have a 100+ years of fire suppression which is a policy thing, not a political thing. The question is what are network operators going to do? If the answer is "nothing", don't be surprised to get legislation shoved down your throats. Don't expect the bay area of all places to passively put up with all of this. If your network fails because of power going out and I can't call 911, you've got a big, big problem. Mike
As a small WISP operator in Northern California and well into the urban interface we fell victim to the PSPS this year. Thousands was spent on upgrading battery plants that would normally hold during a short outage and generator purchases, whether it be small inverter style generators for small sites to permanent standby site generators for those sites that are larger or a PITA to get to. We still have more work to do and hope to be better prepared for next summers rounds of shut offs. I am currently developing a portable trailer mounted solar/battery plant to replace the portable generators just for fuel cost savings since I spent just about $500/week in generator fuel alone for the largest outage. On Thu, Dec 26, 2019 at 9:59 AM Michael Thomas <mike@mtcc.com> wrote:
On 12/26/19 7:51 AM, Mike Bolitho wrote:
I'm pretty sure political bickering is well beyond the scope of the mailing list. Is anyone moderating this?
It certainly wasn't my intent or desire to have this turn political, and shame on the person who did. This is a serious networking related issue for California *right* *now*. It may become a serious networking related issue for a lot of other places too -- California is hardly unique in its wildland - urban interface issues, and lots of places burn just like California. And definitely lots of places have a 100+ years of fire suppression which is a policy thing, not a political thing.
The question is what are network operators going to do? If the answer is "nothing", don't be surprised to get legislation shoved down your throats. Don't expect the bay area of all places to passively put up with all of this. If your network fails because of power going out and I can't call 911, you've got a big, big problem.
Mike
--
Jason Wilson Remotely Located Providing High Speed Internet to out of the way places. 530-651-1736 530-748-9608 Cell www.remotelylocated.com
On 12/26/19 10:19 AM, Jason Wilson wrote:
As a small WISP operator in Northern California and well into the urban interface we fell victim to the PSPS this year. Thousands was spent on upgrading battery plants that would normally hold during a short outage and generator purchases, whether it be small inverter style generators for small sites to permanent standby site generators for those sites that are larger or a PITA to get to. We still have more work to do and hope to be better prepared for next summers rounds of shut offs. I am currently developing a portable trailer mounted solar/battery plant to replace the portable generators just for fuel cost savings since I spent just about $500/week in generator fuel alone for the largest outage.
Yeah, that's the biggest change: the outages are 2-3 days each. Maybe they'll get better but their stated goal is to inspect the entire distribution system before turning power back on, so that is going to take some time no matter what. And the fuel cost is definitely a consideration. Unfortunately October and November is pretty lousy for solar. Wind might be better since that is the reason they're doing the PSPS in the first place :) Mike
If that was a reference to my comments, it was certainly not my intention. I was striving to avoid it being seen as that, but apparently fell short. To reanswer the question posed though, is still the same ; $$$. If network operators take the position that the electric utility supply should be more reliable than it is, then they need to start influencing and lobbying for ways for that to happen. If not, they will have to increase investments into local generation or storage capacity to bridge those gaps. You seem to imply that regulation is inherently bad; however the scenario that you describe (power failures impacting 911 service) is only a concern to an operator if there is a legislatively define deterrent. On Thu, Dec 26, 2019 at 13:00 Michael Thomas <mike@mtcc.com> wrote:
On 12/26/19 7:51 AM, Mike Bolitho wrote:
I'm pretty sure political bickering is well beyond the scope of the mailing list. Is anyone moderating this?
It certainly wasn't my intent or desire to have this turn political, and shame on the person who did. This is a serious networking related issue for California *right* *now*. It may become a serious networking related issue for a lot of other places too -- California is hardly unique in its wildland - urban interface issues, and lots of places burn just like California. And definitely lots of places have a 100+ years of fire suppression which is a policy thing, not a political thing.
The question is what are network operators going to do? If the answer is "nothing", don't be surprised to get legislation shoved down your throats. Don't expect the bay area of all places to passively put up with all of this. If your network fails because of power going out and I can't call 911, you've got a big, big problem.
Mike
On 12/26/19 10:26 AM, Tom Beecher wrote:
If that was a reference to my comments, it was certainly not my intention. I was striving to avoid it being seen as that, but apparently fell short.
Not directed at you at all.
To reanswer the question posed though, is still the same ; $$$. If network operators take the position that the electric utility supply should be more reliable than it is, then they need to start influencing and lobbying for ways for that to happen. If not, they will have to increase investments into local generation or storage capacity to bridge those gaps.
You seem to imply that regulation is inherently bad; however the scenario that you describe (power failures impacting 911 service) is only a concern to an operator if there is a legislatively define deterrent.
Not at all. I'm saying that this problem will be solved one way or the other. Frankly it's surprising that anybody offering telephony service has gotten away with not fulfilling the battery backup mandate. I guess there must have been some wiggle room that the carriers took advantage of. And if so, legislation to fix that will be immanent. Mike
In article <CAL9Qcx5jGEZzoqmcvusfW9htTwoSVV9Mnf6Xca5VHQLLDSTySw@mail.gmail.com> you write:
To reanswer the question posed though, is still the same ; $$$. If network operators take the position that the electric utility supply should be more reliable than it is, then they need to start influencing and lobbying for ways for that to happen. If not, they will have to increase investments into local generation or storage capacity to bridge those gaps.
You seem to imply that regulation is inherently bad; however the scenario that you describe (power failures impacting 911 service) is only a concern to an operator if there is a legislatively define deterrent.
California suffers from an unusual combination of a dry climate that is getting dryer and political decisions that made sense in the short run but are now showing their long term consequences, notably land use that encourages sprawl and construction in ill-suited areas, and a regulator that keeps short term consumer prices down at the cost of reliability and long term stability. None of this should be a surprise to anyone familiar with the situation. Even well run US utilities are much less reliable than the norm in Europe or Japan. Where ISPs in the US are figuring out how to install batteries and backup generators or private windmills or whatever, their European peers pay somewhat higher utility bills and don't have to worry about the other stuff. You'll pay either way. European utilities aren't more reliable by accident; that's how they're regulated. Calfornia also offers an interesting natural experiment comparing privately run utilities PG&E and SCE and the city owned Los Angeles DWP.
On 12/26/19 11:18 AM, John Levine wrote:
To reanswer the question posed though, is still the same ; $$$. If network operators take the position that the electric utility supply should be more reliable than it is, then they need to start influencing and lobbying for ways for that to happen. If not, they will have to increase investments into local generation or storage capacity to bridge those gaps.
You seem to imply that regulation is inherently bad; however the scenario that you describe (power failures impacting 911 service) is only a concern to an operator if there is a legislatively define deterrent. California suffers from an unusual combination of a dry climate that is getting dryer and political decisions that made sense in the short run but are now showing their long term consequences, notably land use
In article <CAL9Qcx5jGEZzoqmcvusfW9htTwoSVV9Mnf6Xca5VHQLLDSTySw@mail.gmail.com> you write: that encourages sprawl and construction in ill-suited areas
, and a regulator that keeps short term consumer prices down at the cost of reliability and long term stability. None of this should be a surprise to anyone familiar with the situation. PG&E is especially egregious as it has extremely high rates and
If we stopped construction in all of the ill-suited areas, we'd stop construction all together, and tear down much more. We have it all here: earthquakes, floods, fires; often the trifecta. We could certainly be smarter, but the nature of the geography here is both a blessing and a curse. piss-poor maintenance. Where does all of that money go? Execs and shareholders. And if some random nyc hedge fund gets its way it's going to get even worse. I don't know what the ultimate solution is, but whatever it is cannot have those perverse incentives. Mike
In article <c0c79349-2a6c-4874-b4d9-8013532687c8@mtcc.com> you write:
run but are now showing their long term consequences, notably land use that encourages sprawl and construction in ill-suited areas
If we stopped construction in all of the ill-suited areas, we'd stop construction all together, and tear down much more. We have it all here: earthquakes, floods, fires; often the trifecta. We could certainly be smarter, but the nature of the geography here is both a blessing and a curse.
Among California's many problems is a bizarre terror of upzoning and infill construction, hence the sprawl. Here in my rustic bit of upstate New York you can build a two-family anywhere you can build a single family and the world has not come to an end.
PG&E is especially egregious as it has extremely high rates and piss-poor maintenance. Where does all of that money go? Execs and shareholders.
Evidently not since they've been through bankruptcy a few times. I think they're just institutionally incompetent as well as having an unusually environmentally hostile territory to serve. (Around here when the power company screws up, the power fails but the county does not catch fire.)
I don't know what the ultimate solution is, but whatever it is cannot have those perverse incentives.
The LA DWP seems to do OK. R's, John
On 12/26/19 4:06 PM, John Levine wrote:
In article <c0c79349-2a6c-4874-b4d9-8013532687c8@mtcc.com> you write:
run but are now showing their long term consequences, notably land use that encourages sprawl and construction in ill-suited areas If we stopped construction in all of the ill-suited areas, we'd stop construction all together, and tear down much more. We have it all here: earthquakes, floods, fires; often the trifecta. We could certainly be smarter, but the nature of the geography here is both a blessing and a curse. Among California's many problems is a bizarre terror of upzoning and infill construction, hence the sprawl. Here in my rustic bit of upstate New York you can build a two-family anywhere you can build a single family and the world has not come to an end.
NIMBYism. My previous state senator (Scott Wiener) has been trying all he can to make headway on that front. But NIMBY's are a strong force and don't cleave down party lines whatsoever.
PG&E is especially egregious as it has extremely high rates and piss-poor maintenance. Where does all of that money go? Execs and shareholders. Evidently not since they've been through bankruptcy a few times. I think they're just institutionally incompetent as well as having an unusually environmentally hostile territory to serve. (Around here when the power company screws up, the power fails but the county does not catch fire.)
Well that was true here until about 10-20 years ago too. Fire seasons are about 2 months longer, iirc. From beginning of May into first part of December. We almost never had fires in June but now they're fairly common. That's true for a lot of western US now.
I don't know what the ultimate solution is, but whatever it is cannot have those perverse incentives. The LA DWP seems to do OK.
As does Sacramento's SMUD. Part of the problem is that they are just so large. San Jose and SF are thinking very seriously about splitting off. Which would probably create yet another death spiral for PG&E because it would leave all of the expensive distribution (= out in the boonies, etc) and allow cities to cherry pick the cheaper distribution areas when it makes sense. The entire thing is a shitshow. Mike
This time it’s PG&E all alone, but still fallout from back then. Too much liability and they’ve not maintained the infrastructure and so they decided that to reduce the liability costs it’s cheaper to blackout. Same story again different colors. PG&E making a mint while people get screwed (PG&E was mostly at the getting screwed end in 2000-2001)
PG&E has been the one in the news, but SCE appears to have been making the same choices with about the same effects. The Thomas Fire was briefly the largest wildfire in state history, and the source (well, with the rain) of the Montecito mud flow a few weeks later. We're told that SCE seems to figure in that one and several others before and since. I go back and forth on who might be responsible. The electric utilities bear blame for their infrastructure; it should be underground, not strung from poles. I would put some to the state and the management of the various national forests and national parks in the area - one of the outcomes from a fire in 2007 or thereabouts was that the ecology folks had been protecting foliage, and that foliage burned and clogged streams, with all sorts of results. Surprise! If you're worried about ecology, you should support management of it. In California, there are also laws holding home-owners responsible for "defensible space" around their homes. https://www.google.com/search?q=california+brush+clearing+laws https://en.m.wikipedia.org/wiki/Thomas_Fire <https://en.m.wikipedia.org/wiki/Thomas_Fire> https://www.washingtonpost.com/graphics/2018/national/montecito-before-after... https://www.edhat.com/news/10-homes-destroyed-in-holiday-fire https://www.edhat.com/news/cave-fire-now-100-contained
On 12/26/19 6:38 PM, Fred Baker wrote:
This time it’s PG&E all alone, but still fallout from back then. Too much liability and they’ve not maintained the infrastructure and so they decided that to reduce the liability costs it’s cheaper to blackout. Same story again different colors. PG&E making a mint while people get screwed (PG&E was mostly at the getting screwed end in 2000-2001)
PG&E has been the one in the news, but SCE appears to have been making the same choices with about the same effects. The Thomas Fire was briefly the largest wildfire in state history, and the source (well, with the rain) of the Montecito mud flow a few weeks later. We're told that SCE seems to figure in that one and several others before and since.
I go back and forth on who might be responsible. The electric utilities bear blame for their infrastructure; it should be underground, not strung from poles. I would put some to the state and the management of the various national forests and national parks in the area - one of the outcomes from a fire in 2007 or thereabouts was that the ecology folks had been protecting foliage, and that foliage burned and clogged streams, with all sorts of results. Surprise! If you're worried about ecology, you should support management of it. In California, there are also laws holding home-owners responsible for "defensible space" around their homes.
When I lived in Socal, we certainly had hellacious brush fires when the Santa Ana winds blew. I don't remember any/many of them being attributed SCE though? Maybe I just wasn't paying attention? Do remember anything about that, Fred? We've forever had an urban-wildland interface problem -- I mean, how many times has Malibu burned down, it seemed like every other year. Apparently San Diego Gas and Electric has been something of a pioneer after the horrible Cedar fire, and apparently it's made a difference. Mike
This is all in conjunction with the CPUC. I believe it is also a part of a court order. I’ll need to find that later https://www.cpuc.ca.gov/deenergization/ I don’t want to copy the whole thing but this is the bulk of it before it goes Into when the outages were. Wildfires are more destructive and deadlier than in the past, and the threat of wildfires is more prevalent throughout the state and calendar year. The overall pattern shows the emerging effects of climate change in our daily lives. If you need information on disaster relief protections for customers of affected areas during any state of emergency, please read our blog <https://www.cpuc.ca.gov/cpucblog.aspx?id=6442463069&blogid=1551>. Throughout the year, the CPUC works with CalFire and the Office of Emergency Services to reduce the risk of utility infrastructure starting wildfires, to strengthen utility preparedness for emergencies, and to improve utility services during and after emergencies. Interagency coordination, and cooperation from the utilities is essential when the threat of wildfires is high. The State's investor-owned electric utilities, notably Pacific Gas and Electric Company (PG&E), Southern California Edison, and San Diego Gas & Electric (SDG&E), may shut off electric power, referred to as "de-energization" or Public Safety Power Shut-offs (PSPS), to protect public safety under California law, specifically California Public Utilities Code <http://leginfo.legislature.ca.gov/faces/codesTOCSelected.xhtml?tocCode=PUC&tocTitle=+Public+Utilities+Code+-+PUC> (PU Code) Sections 451 and 399.2(a). CPUC Actions On July 12, 2018, the CPUC adopted Resolution ESRB-8 <http://docs.cpuc.ca.gov/PublishedDocs/Published/G000/M217/K801/217801749.PDF>to strengthen customer notification requirements before de-energization events and ordered utilities to engage local communities in developing de-energization programs. Utilities must submit a report within 10 days after each de-energization event, and after high-fire-threat events where the utility provided notifications to local government, agencies, and customers of possible de-energization though no de-energization occurred. These reports are accessible below. On December 13, 2018, the CPUC opened a new Rulemaking <http://docs.cpuc.ca.gov/PublishedDocs/Published/G000/M251/K987/251987258.PDF> (R.18-12-005 <https://apps.cpuc.ca.gov/apex/f?p=401:56:0::NO:RP,57,RIR:P5_PROCEEDING_SELECT:R1812005>) to examine utilities' PSPS processes and practices, the impacts on communities and access and functional needs populations, efforts to reduce the need for de-energization, and mitigate measures to reduce the impacts when implemented. The Rulemaking will also review and improve existing reporting requirements. On May 30, 2019, in its ongoing efforts to expeditiously implement Senate Bill 901, the CPUC made its Phase I decision <http://docs.cpuc.ca.gov/PublishedDocs/Published/G000/M296/K867/296867634.PDF> in the proceeding, making improvements to utility communication and notification protocols to ensure that clear rules are in place as early as possible to prepare for the 2019 fire season. On August 14, 2019, the CPUC opened a second phase (Phase 2) in R.18-12-005 to address additional aspects of the utilities' PSPS processes and practices and to expand upon the guidelines adopted in Phase 1. In Phase 2 the CPUC will consider, among other issues, identification and communication with access and functional needs populations, communication with customers while the power is turned off, communication during reenergization, mitigation measures, coordination with emergency responders, and transmission-level de-energization. Proceeding documents are available on the Docket Card <https://apps.cpuc.ca.gov/apex/f?p=401:56:0::NO:RP,57,RIR:P5_PROCEEDING_SELECT:R1812005> . Send your comments on the proceeding to public.advisor@cpuc.ca.gov and refer to proceeding number R.18-12-005. The CPUC is working with the Office of Emergency Services (Cal OES) <http://www.caloes.ca.gov/>, CAL FIRE <http://www.fire.ca.gov/>, and first-responders to address potential impacts of utility de-energization practices on emergency response activities, including evacuations. The CPUC is also monitoring the development and will continuously assess implementation of de-energization programs by utilities, including performing a thorough review of de-energization events as they occur. On October 18, 2019, the CPUC held an Emergency Meeting to hear from top Pacific Gas and Electric Company (PG&E) executives to publicly address the mistakes and operational gaps identified in the utility’s October 2019 PSPS events and to provide lessons learned to ensure they are not repeated. More information about the meeting is available under "October 2019 PSPS Events" below. On Fri, Dec 27, 2019 at 2:57 PM Michael Thomas <mike@mtcc.com> wrote:
On 12/26/19 6:38 PM, Fred Baker wrote:
This time it’s PG&E all alone, but still fallout from back then. Too much liability and they’ve not maintained the infrastructure and so they decided that to reduce the liability costs it’s cheaper to blackout. Same story again different colors. PG&E making a mint while people get screwed (PG&E was mostly at the getting screwed end in 2000-2001)
PG&E has been the one in the news, but SCE appears to have been making the same choices with about the same effects. The Thomas Fire was briefly the largest wildfire in state history, and the source (well, with the rain) of the Montecito mud flow a few weeks later. We're told that SCE seems to figure in that one and several others before and since.
I go back and forth on who might be responsible. The electric utilities bear blame for their infrastructure; it should be underground, not strung from poles. I would put some to the state and the management of the various national forests and national parks in the area - one of the outcomes from a fire in 2007 or thereabouts was that the ecology folks had been protecting foliage, and that foliage burned and clogged streams, with all sorts of results. Surprise! If you're worried about ecology, you should support management of it. In California, there are also laws holding home-owners responsible for "defensible space" around their homes.
When I lived in Socal, we certainly had hellacious brush fires when the Santa Ana winds blew. I don't remember any/many of them being attributed SCE though? Maybe I just wasn't paying attention? Do remember anything about that, Fred? We've forever had an urban-wildland interface problem -- I mean, how many times has Malibu burned down, it seemed like every other year.
Apparently San Diego Gas and Electric has been something of a pioneer after the horrible Cedar fire, and apparently it's made a difference.
Mike
--
Jason Wilson Remotely Located Providing High Speed Internet to out of the way places. 530-651-1736 530-748-9608 Cell www.remotelylocated.com
* Jason Wilson:
This is all in conjunction with the CPUC. I believe it is also a part of a court order. I’ll need to find that later
I found the connection rather puzzling (that is, how switching off power distribution prevents wildfires or at least reduces their risk). I found some explanations here (downed lines, vegetation contact, conductor slap, repetitive faults, apparatus failures): <https://wildfiremitigation.tees.tamus.edu/faqs/how-power-lines-cause-wildfires>
In article <87y2up1vc4.fsf@mid.deneb.enyo.de> you write:
I found the connection rather puzzling (that is, how switching off power distribution prevents wildfires or at least reduces their risk). I found some explanations here (downed lines, vegetation contact, conductor slap, repetitive faults, apparatus failures):
<https://wildfiremitigation.tees.tamus.edu/faqs/how-power-lines-cause-wildfires>
Oh, you're in Europe. You wouldn't believe how cruddy US power distribution systems are. California is particularly bad becuase the populist state regulator has keep retail prices low at the cost of reliability, safety, and everything else. Also keep in mind that California has conditions seen nowhere in Europe: bone dry forests with 40C temperaturees and 100Kph winds, and a power company too underfunded to keep up with tree trimming. R's, John PS: You also wouldn't believe how cheap the power is. California's prices are high compared to most of the US, but it's still only about €0.15 per KWh.
----- On Jan 2, 2020, at 1:24 PM, John Levine johnl@iecc.com wrote:
PS: You also wouldn't believe how cheap the power is. California's prices are high compared to most of the US, but it's still only about €0.15 per KWh.
I don't know where you live, but I pay around 38 cents/KWh. Depending on your rate, that can go up to 53 cents/KWh during peak times. That's in Morgan Hill, CA, on PG&E power. See here: https://www.pge.com/tariffs/Res_Inclu_TOU_Current.xlsx Or the entire schedule: https://www.pge.com/tariffs/electric.shtml Thanks, Sabri
On 1/2/20 1:34 PM, Sabri Berisha wrote:
----- On Jan 2, 2020, at 1:24 PM, John Levine johnl@iecc.com wrote:
PS: You also wouldn't believe how cheap the power is. California's prices are high compared to most of the US, but it's still only about €0.15 per KWh. I don't know where you live, but I pay around 38 cents/KWh. Depending on your rate, that can go up to 53 cents/KWh during peak times.
That's in Morgan Hill, CA, on PG&E power.
See here: https://www.pge.com/tariffs/Res_Inclu_TOU_Current.xlsx Or the entire schedule: https://www.pge.com/tariffs/electric.shtml
Yeah, I looked it up too because that sounded just flat out wrong. And while the CPUC certainly has its share of the blame, PG&E is by far the most to blame for this. Mike
PS: You also wouldn't believe how cheap the power is. California's prices are high compared to most of the US, but it's still only about €0.15 per KWh.
I don't know where you live, but I pay around 38 cents/KWh. Depending on your rate, that can go up to 53 cents/KWh during peak times.
16x is supposed to be the average. I live in upstate New York where I pay about 8c/kwh and a fixed $15/mo connection charge. We have day/night rates available but they're not very different for retail customers. I get a slight discount due to credits from remote net metering at a nearby solar farm. Regards, John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies", Please consider the environment before reading this e-mail. https://jl.ly
On Thursday, 2 January, 2020 21:34, "Sabri Berisha" <sabri@cluecentral.net> said:
----- On Jan 2, 2020, at 1:24 PM, John Levine johnl@iecc.com wrote:
PS: You also wouldn't believe how cheap the power is. California's prices are high compared to most of the US, but it's still only about €0.15 per KWh.
I don't know where you live, but I pay around 38 cents/KWh. Depending on your rate, that can go up to 53 cents/KWh during peak times.
Data point for comparison: I'm in the UK, paying ~0.18GBP/KWh (0.21EUR or 0.24USD) - and that's on a tariff where I'm paying extra to ensure a certain amount of the power is sourced from renewables, I could get it cheaper. We do have a standing charge though, irrespective of usage, 0.22GBP/day in my case - I don't know if you have that on US electricity bills typically. It's a rare event and a cause of great annoyance if the power goes out. (Most recent events - and I'm talking 2 or 3 in the last decade or more - have been down to idiots trying to steal copper from live substations, which doesn't end well for anyone). The last time we had planned blackouts on any kind of scale was the 70s when due to the oil crisis and striking miners we couldn't source enough fossil-fuel to keep generating. Obviously completely different climate, geography, and (as I understand it) industry set-up - we have the national distribution infrastructure, last mile infrastructure, generation, and consumer-facing commercial / billing all as distinct entities, which brings its own set of benefits and challenges!
* John Levine:
In article <87y2up1vc4.fsf@mid.deneb.enyo.de> you write:
I found the connection rather puzzling (that is, how switching off power distribution prevents wildfires or at least reduces their risk). I found some explanations here (downed lines, vegetation contact, conductor slap, repetitive faults, apparatus failures):
<https://wildfiremitigation.tees.tamus.edu/faqs/how-power-lines-cause-wildfires>
Oh, you're in Europe. You wouldn't believe how cruddy US power distribution systems are. California is particularly bad becuase the populist state regulator has keep retail prices low at the cost of reliability, safety, and everything else.
Also keep in mind that California has conditions seen nowhere in Europe: bone dry forests with 40C temperaturees and 100Kph winds, and a power company too underfunded to keep up with tree trimming.
I think Greece also suffered a major wildfire in the 2018 that was initiated by a faulty power line. In Germany, we have some vegetation issues on train lines partially due to insufficient maintenance, but they fortunately don't trigger wildfires, only train outages. I guess most countries struggle to maintain basic infrastructure.
On 12/25/19 5:59 PM, Constantine A. Murenin wrote:
On Wed, 25 Dec 2019 at 19:32, Michael Thomas <mike@mtcc.com <mailto:mike@mtcc.com>> wrote:
On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh.
Do you have a source for this? It would seem that these power issues are rather unique to California not because of some "climate change" bogeyman, but rather because of a failed public policy at the state level.
There's no point in engaging climate deniers. This is what's happening to us right now regardless of what science you choose to call "bogeymen". We are living in fear of becoming the next Paradise. Mike
This is a very short term problem. The market is going to fill with battery storage sooner rather than later. Solar is just exploding. Your car will "house tie".6G will solve your data problem. D-- Don Gould5 Cargill PlaceRichmondChristchurch, New ZealandMobile/Telegram: + 64 21 114 0699www.bowenvale.co.nz -------- Original message --------From: Michael Thomas <mike@mtcc.com> Date: 26/12/19 2:33 PM (GMT+12:00) To: nanog@nanog.org Subject: power to the internet https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678Thi... article details some of the issues with California's "new reality" of planned blackouts. One of the big things that came to light with these blackouts is that our network infrastructure's resilience is pretty lacking. While I was (surprisingly to me) ok with my DSL connection out in the boonies, lots and lots of people with cable weren't so lucky. And I'm not sure how bad the situation is with cellular infrastructure, but I assume it's not much better than cable. And I wouldn't doubt that other DSL deployments go dark when power is down. I have no clue with fiber.So I guess what I'm wondering is what can we do about this? What should we do about this? These days IP access is not just convenience, it's the way we go about our lives, just like electricity itself. At base, it seems to me that network operators should be required to keep the lights on in blackouts just like POTS operators do now. If I have power to light my modem or charge in my phone, I should be able to get onto the net. That seems like table stakes.One of the things we learned also is that the blackouts seem to last between 2-3 days apiece. I happen to have a generator since I'm out in the boonies and our power gets cut regularly because of snow, but not everyone has that luxury. I kind of want to think that my router+modem use about 20 watts, so powering it up would take about 1.5kwh for 3 days. a quick google look shows that I'd probably need to shell out $500 or so for a battery of that capacity, and that's doesn't include your phones, laptops, tv's, etc power needs. What does that mean? That is a major expense for a lot of people.On the bright side, I hear that power generator companies stocks have gone through the roof.On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh.Mike
Unless telecom infrastructure has been diligently changing out the lead acid battery approach at all their remote terminals, powered gpon, hfc and antennae plants will never last more than minutes. If at all. A traditional car has between a 100-200amp alternator @12volts How much generating capacity can you get out of a typical hybrid? Self-isolating and re-tieing inverters. Economic household ATS systems. Do those exist? Enough independent distributed capacity and now comes the ability to create grid islands. How might that look? Electric grid shortage is likely coming to NYC, courtesy of folk of certain political persuasion and their love of stone age era living. IP decommissioning. If you have CO loop copper, keep it. Joe Don Gould wrote:
This is a very short term problem.
The market is going to fill with battery storage sooner rather than later.
Solar is just exploding.
Your car will "house tie".
6G will solve your data problem.
D
-- Don Gould 5 Cargill Place Richmond Christchurch, New Zealand Mobile/Telegram: + 64 21 114 0699 www. <http://www.tusker.net.au/>bowenvale.co.nz
-------- Original message -------- From: Michael Thomas <mike@mtcc.com> Date: 26/12/19 2:33 PM (GMT+12:00) To: nanog@nanog.org Subject: power to the internet
https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678
This article details some of the issues with California's "new reality" of planned blackouts. One of the big things that came to light with these blackouts is that our network infrastructure's resilience is pretty lacking. While I was (surprisingly to me) ok with my DSL connection out in the boonies, lots and lots of people with cable weren't so lucky. And I'm not sure how bad the situation is with cellular infrastructure, but I assume it's not much better than cable. And I wouldn't doubt that other DSL deployments go dark when power is down. I have no clue with fiber.
So I guess what I'm wondering is what can we do about this? What should we do about this? These days IP access is not just convenience, it's the way we go about our lives, just like electricity itself. At base, it seems to me that network operators should be required to keep the lights on in blackouts just like POTS operators do now. If I have power to light my modem or charge in my phone, I should be able to get onto the net. That seems like table stakes.
One of the things we learned also is that the blackouts seem to last between 2-3 days apiece. I happen to have a generator since I'm out in the boonies and our power gets cut regularly because of snow, but not everyone has that luxury. I kind of want to think that my router+modem use about 20 watts, so powering it up would take about 1.5kwh for 3 days. a quick google look shows that I'd probably need to shell out $500 or so for a battery of that capacity, and that's doesn't include your phones, laptops, tv's, etc power needs. What does that mean? That is a major expense for a lot of people.
On the bright side, I hear that power generator companies stocks have gone through the roof.
On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh.
Mike
How much generating capacity can you get out of a typical hybrid?
You’re joking right? A lot… Enough to run an entire neighborhood… The Prius makes 50,000watts alone. With the right circuitry, there is no need for power plants in the United States (save that they’re more efficient than internal combustion gas engines in the 76hp range) - existing hybrid car stock’s generating capacity exceeds the entire US supply. And it’s entirely untapped. Nissan just tested it for giggles, and found the Leaf (which has NO engine at all) can power a house for an entire week. The batteries alone are a game changer, utterly transforming grids.
Self-isolating and re-tieing inverters. Economic household ATS systems. Do those exist?
Yes I have a patent in one type of this this; it exists in numerous variants. ATS-es for partial loads of up to 6,000watts are like $400. You sound smart, but did you research any of this or just post? -Ben. -Ben Cannon CEO 6x7 Networks & 6x7 Telecom, LLC ben@6by7.net <mailto:ben@6by7.net>
On Dec 26, 2019, at 2:31 AM, Joe Maimon <jmaimon@jmaimon.com> wrote:
Unless telecom infrastructure has been diligently changing out the lead acid battery approach at all their remote terminals, powered gpon, hfc and antennae plants will never last more than minutes. If at all.
A traditional car has between a 100-200amp alternator @12volts
How much generating capacity can you get out of a typical hybrid?
Self-isolating and re-tieing inverters. Economic household ATS systems. Do those exist?
Enough independent distributed capacity and now comes the ability to create grid islands. How might that look?
Electric grid shortage is likely coming to NYC, courtesy of folk of certain political persuasion and their love of stone age era living. IP decommissioning.
If you have CO loop copper, keep it.
Joe
Don Gould wrote:
This is a very short term problem.
The market is going to fill with battery storage sooner rather than later.
Solar is just exploding.
Your car will "house tie".
6G will solve your data problem.
D
-- Don Gould 5 Cargill Place Richmond Christchurch, New Zealand Mobile/Telegram: + 64 21 114 0699 www. <http://www.tusker.net.au/>bowenvale.co.nz
-------- Original message -------- From: Michael Thomas <mike@mtcc.com> Date: 26/12/19 2:33 PM (GMT+12:00) To: nanog@nanog.org Subject: power to the internet
https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678
This article details some of the issues with California's "new reality" of planned blackouts. One of the big things that came to light with these blackouts is that our network infrastructure's resilience is pretty lacking. While I was (surprisingly to me) ok with my DSL connection out in the boonies, lots and lots of people with cable weren't so lucky. And I'm not sure how bad the situation is with cellular infrastructure, but I assume it's not much better than cable. And I wouldn't doubt that other DSL deployments go dark when power is down. I have no clue with fiber.
So I guess what I'm wondering is what can we do about this? What should we do about this? These days IP access is not just convenience, it's the way we go about our lives, just like electricity itself. At base, it seems to me that network operators should be required to keep the lights on in blackouts just like POTS operators do now. If I have power to light my modem or charge in my phone, I should be able to get onto the net. That seems like table stakes.
One of the things we learned also is that the blackouts seem to last between 2-3 days apiece. I happen to have a generator since I'm out in the boonies and our power gets cut regularly because of snow, but not everyone has that luxury. I kind of want to think that my router+modem use about 20 watts, so powering it up would take about 1.5kwh for 3 days. a quick google look shows that I'd probably need to shell out $500 or so for a battery of that capacity, and that's doesn't include your phones, laptops, tv's, etc power needs. What does that mean? That is a major expense for a lot of people.
On the bright side, I hear that power generator companies stocks have gone through the roof.
On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh.
Mike
On 12/26/19 11:00 AM, Ben Cannon wrote:
How much generating capacity can you get out of a typical hybrid?
You’re joking right? A lot… Enough to run an entire neighborhood… The Prius makes 50,000watts alone.
With the right circuitry, there is no need for power plants in the United States (save that they’re more efficient than internal combustion gas engines in the 76hp range) - existing hybrid car stock’s generating capacity exceeds the entire US supply. And it’s entirely untapped.
Nissan just tested it for giggles, and found the Leaf (which has NO engine at all) can power a house for an entire week. The batteries alone are a game changer, utterly transforming grids.
I just looked up Telsa's battery packs and they seem to be between 60-100kwh. Our daily use is about 30kwh in the fall, so it's only 2-3 days. Admittedly we can turn off the hot tub, water heater, etc to stretch it out. And of course, that means that you can't drive it... The one thing that would be for everybody's good is using them during peak hours. If you work normal hours, then that only gets part of the peak load, unfortunately. But of course this has nothing to do with the network power problem. I assume they won't be parking a Tesla next to a CMTS headend. Mike
I just looked up Telsa's battery packs and they seem to be between 60-100kwh. Our daily use is about 30kwh in the fall, so it's only 2-3 days. Admittedly we can turn off the hot tub, water heater, etc to stretch it out. And of course, that means that you can't drive it... The one thing that would be for everybody's good is using them during peak hours. If you work normal hours, then that only gets part of the peak load, unfortunately.
Just buy three of them. Two to leave in the garage as a "mobile battery pack" and one to drive around. All problems solved. -- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.
Isn't that what the Tesla Power Wall's are? I thought that was the fill measure for when the solar panels aren't generating. I have never gotten anything, but know when you look on their site for Solar, they try and pitch the batter power walls to run your house for days if needed.. --- Howard Leadmon PBW Communications, LLC http://www.pbwcomm.com On 12/26/2019 2:08 PM, Keith Medcalf wrote:
I just looked up Telsa's battery packs and they seem to be between 60-100kwh. Our daily use is about 30kwh in the fall, so it's only 2-3 days. Admittedly we can turn off the hot tub, water heater, etc to stretch it out. And of course, that means that you can't drive it... The one thing that would be for everybody's good is using them during peak hours. If you work normal hours, then that only gets part of the peak load, unfortunately. Just buy three of them. Two to leave in the garage as a "mobile battery pack" and one to drive around.
All problems solved.
That's exactly what Powerwalls are. In Vermont, Green Mountain Power had a deal where they bought 2000 Powerwalls and gave them to customers throughout the state. Customers could get up to two, paying only $1500 each for the installation and agreeing to let GMP manage them. GMP now has ~2.7 gWh of stored capacity, distributed throughout the state to minimize transmission costs. In times of high electricity spot market prices or outages, GMP draws down the Powerwalls, then refills them at night when prices are lower. The Powerwalls also act as a UPS for the whole house. When bad weather is predicted in an area, GMP makes sure your Powerwall is full. My town had a 4+ hour outage a few weeks back and I had power the whole time, the microwave clock didn't even reset. I only have one Powerwall but could easily last 2+ days with it, and it's silent, unlike a generator. --John Lightfoot -----Original Message----- From: NANOG <nanog-bounces@nanog.org> on behalf of Howard Leadmon <howard@leadmon.net> Date: Monday, December 30, 2019 at 3:09 PM To: "nanog@nanog.org" <nanog@nanog.org> Subject: Re: power to the internet Isn't that what the Tesla Power Wall's are? I thought that was the fill measure for when the solar panels aren't generating. I have never gotten anything, but know when you look on their site for Solar, they try and pitch the batter power walls to run your house for days if needed.. --- Howard Leadmon PBW Communications, LLC http://www.pbwcomm.com On 12/26/2019 2:08 PM, Keith Medcalf wrote:
I just looked up Telsa's battery packs and they seem to be between 60-100kwh. Our daily use is about 30kwh in the fall, so it's only 2-3 days. Admittedly we can turn off the hot tub, water heater, etc to stretch it out. And of course, that means that you can't drive it... The one thing that would be for everybody's good is using them during peak hours. If you work normal hours, then that only gets part of the peak load, unfortunately. Just buy three of them. Two to leave in the garage as a "mobile battery pack" and one to drive around.
All problems solved.
On Thu Dec 26, 2019 at 11:20:01AM -0800, Michael Thomas wrote:
I just looked up Telsa's battery packs and they seem to be between 60-100kwh. Our daily use is about 30kwh in the fall, so it's only 2-3 days. Admittedly we can turn off the hot tub, water heater, etc to stretch it out. And of course, that means that you can't drive it... The one thing that would be for everybody's good is using them during peak hours. If you work normal hours, then that only gets part of the peak load, unfortunately.
Many with a tesla (or three) are likely to get some local PV/wind generation to charge them (more so if the outages become a regular event). Then the base load doesn't matter and they can still drive some of them while the others charge. Some may just use their EV as a battery that has other uses when the power isn't out and not care they can't use it for both. There are plenty of projects working on this distributed storage/generation model.
But of course this has nothing to do with the network power problem. I assume they won't be parking a Tesla next to a CMTS headend.
Proliferation of local generation and storage may have a lot of impact on network expectations. Last mile that assumes when the cab loses power then the consumers have too will have to update their assumptions. The consumers running on local power are able to carry on as normal and expect network to carry on too. Architectures with lots of distributed small active plant may be harder hit trying to add generation than those with passive plant and few active nodes. brandon
Hi Brandon, I agree with lots of what you wrote, here's some more thoughts.... tl;dr Batteries from cars will fix your issues along with smart guys just wanting to get the job done. WHO AM I?! I am a coms tech. I have been on this list for ~20 years. My job is simply to look at the environment and make coms appear. My job is not to blame others, question politics or argue ethics. MY POWER, MY RULES, MY RIGHTS I have two Nissan LEAF with the 24Kwh battery, both at 10 of 12 bars of original battery capacity, roughly 8.5 years old. This gives each car roughly 110km (65 mile) of range between charges. Both are 6 months into a 5 year payment plan. "My job is simply to look at the environment and make coms appear. " So, looking at my actual environment, I know that I'm going to end up with two 15kwh batteries in about 8 years and two 15kwh generators (yes, the motors in those things put out about 15kwh under breaking). Ok, so I'm not going to personally convert these into anything at 57 (I'm 49 now), but some young up and coming (Mr 12, sitting in my living room, playing with his Christmas presents, perhaps) will. MY POWER In buying these two cars, I was wanting to address the 'demand side' of my energy question. Now I have a decent amount of demand, I'm going to put PV on my roof this year. My utility company will buy power back from me at about 15% of what I pay retail, so selling them power makes no sense. My best friend has just completed a 7kwh in solar installation to power is 210 tank gold fish facility. He's currently exporting power and now looking at storage solutions. We're 'early adoptors', so you can look to us to see where the trend is going. MY RULES My other mate asked our utility for power to his new home and they said "Yes sir, half a years wages plus a monthly contribution to the up keep for the brand new plant you're paying for up front." I'm sure it comes as no surprise, he said "no thanks" and then built a whole off grid system including a 6kwh diesel generator. (He is also a coms tech, 20 years older than I am, though not on this list). MY RIGHTS My country lets me do most of the electrical work on my own home up to the power board as long as I follow the rules. I can generate power, I can export power, I can store power. WHERE ARE WE GOING? Just like the US, my community has people who will sit and blame government, will blame the power company, will blame their local elected members, will drag us all into court, will blame the boomers. But also like the US my community also has people who are coms techs who just look at the space, see what's going on and build accordinly. TRANSITION IS A PROBLEM TO SOLVE AT THE LOCAL LEVEL As many of you are picking up, we're in a transitional time. The US utilities know this and they're scared to invest, and for good reason, some have been very burnt in the past. NOT ENOUGH COPPER!!!! "COMS WILL DIE" I remember ~20 years ago, reading posts about how there isn't enough copper to supply the growing demand in lines. Now today I'm reading about how you have abundant abandoned copper as DSLAMs are moved closer to the edge. Power is the same in my view. We're going to see local edge generation, storage and change of use (I have all LED lighting, TV's that use a fraction of the power they used to, but two electric cars). Your US power will stop browning out when you put storage in the network and can drop the peek load. But you're going to have to drive Mrs Brown to make that personal investment, and she won't until it starts to hurt. The utility will under ground lines when it can see a clearer picture for the future. It will take coms with it (FTTH). It won't until the situation smacks the political space so hard that regulation is sorted so it can be happy with investment (in NZ we had to get down this path - this is quite a good read, the incumbent - https://en.wikipedia.org/wiki/Spark_New_Zealand ). RANDOM FYI As I type this, I have over 8 layer 1 networks I can use at my house... * ENABLED - FTTH * VODAFONE - HFC * CHORUS - FTTN/POTS * VODAFONE - 4/5G * SPARK - 4G * 2DEGREES - 4G * CCC - private council network * YOURNET - my own private WISP network * OTHER WISP - there's half a dozen that I can see with a quick scan from hills about 8km away BUT THIS WAS ABOUT POWER FOR OUR EXISTING COMS, NOT THE WHOLE NEIGHBOURHOOD! Many of you are thinking in a silo. You live in a community and you're not acting like it. Fix the power problems for the community rather than just trying to care only for your little nest. As some of you have pointed out, you're lugging gas to generators, connecting trucks with inverters just to keep the coms going in case there's a 911 call required, without thinking that in the even of such, the emergency service won't be able to get to Mrs Brown because the streets are blocked by your trucks all connected to random coms gear. The irony is that people will die when Mr Brown comes running from his house, asks you to take his wife to the hospital but you won't because your job is to keep the coms up so he can call the right service provider. I read the comments here and see many of you have completely lost focus on life and our role in the coms space... WHO AM I?! I am a coms tech. I have been on this list for ~20 years. My job is simply to look at the enviornment and make coms appear. My job is not to blame others, question politics or argue ethics. ...but my job is also to be part of the community we live in. :) On 27/12/2019 12:37 pm, Brandon Butterworth wrote:
I just looked up Telsa's battery packs and they seem to be between 60-100kwh. Our daily use is about 30kwh in the fall, so it's only 2-3 days. Admittedly we can turn off the hot tub, water heater, etc to stretch it out. And of course, that means that you can't drive it... The one thing that would be for everybody's good is using them during peak hours. If you work normal hours, then that only gets part of the peak load, unfortunately. Many with a tesla (or three) are likely to get some local PV/wind generation to charge them (more so if the outages become a regular event). Then the base load doesn't matter and they can still drive some of them while the others charge. Some may just use their EV as a battery that has other uses when the power isn't out and not care
On Thu Dec 26, 2019 at 11:20:01AM -0800, Michael Thomas wrote: they can't use it for both. There are plenty of projects working on this distributed storage/generation model.
But of course this has nothing to do with the network power problem. I assume they won't be parking a Tesla next to a CMTS headend. Proliferation of local generation and storage may have a lot of impact on network expectations. Last mile that assumes when the cab loses power then the consumers have too will have to update their assumptions. The consumers running on local power are able to carry on as normal and expect network to carry on too.
Architectures with lots of distributed small active plant may be harder hit trying to add generation than those with passive plant and few active nodes.
brandon
What is wrong with lead acid battery backup? Seems to be exceedingly stable from my experience. We have all our equipment on -48V DC and have never had a power interruption at any site. The requirements here are 48 hours of backup by law. Telecom is declared to be part of emergency and defense, so they put in a requirement for resilience. Regards Baldur tor. 26. dec. 2019 11.33 skrev Joe Maimon <jmaimon@jmaimon.com>:
Unless telecom infrastructure has been diligently changing out the lead acid battery approach at all their remote terminals, powered gpon, hfc and antennae plants will never last more than minutes. If at all.
A traditional car has between a 100-200amp alternator @12volts
How much generating capacity can you get out of a typical hybrid?
Self-isolating and re-tieing inverters. Economic household ATS systems. Do those exist?
Enough independent distributed capacity and now comes the ability to create grid islands. How might that look?
Electric grid shortage is likely coming to NYC, courtesy of folk of certain political persuasion and their love of stone age era living. IP decommissioning.
If you have CO loop copper, keep it.
Joe
Don Gould wrote:
This is a very short term problem.
The market is going to fill with battery storage sooner rather than later.
Solar is just exploding.
Your car will "house tie".
6G will solve your data problem.
D
-- Don Gould 5 Cargill Place Richmond Christchurch, New Zealand Mobile/Telegram: + 64 21 114 0699 www. <http://www.tusker.net.au/>bowenvale.co.nz
-------- Original message -------- From: Michael Thomas <mike@mtcc.com> Date: 26/12/19 2:33 PM (GMT+12:00) To: nanog@nanog.org Subject: power to the internet
https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678
This article details some of the issues with California's "new reality" of planned blackouts. One of the big things that came to light with these blackouts is that our network infrastructure's resilience is pretty lacking. While I was (surprisingly to me) ok with my DSL connection out in the boonies, lots and lots of people with cable weren't so lucky. And I'm not sure how bad the situation is with cellular infrastructure, but I assume it's not much better than cable. And I wouldn't doubt that other DSL deployments go dark when power is down. I have no clue with fiber.
So I guess what I'm wondering is what can we do about this? What should we do about this? These days IP access is not just convenience, it's the way we go about our lives, just like electricity itself. At base, it seems to me that network operators should be required to keep the lights on in blackouts just like POTS operators do now. If I have power to light my modem or charge in my phone, I should be able to get onto the net. That seems like table stakes.
One of the things we learned also is that the blackouts seem to last between 2-3 days apiece. I happen to have a generator since I'm out in the boonies and our power gets cut regularly because of snow, but not everyone has that luxury. I kind of want to think that my router+modem use about 20 watts, so powering it up would take about 1.5kwh for 3 days. a quick google look shows that I'd probably need to shell out $500 or so for a battery of that capacity, and that's doesn't include your phones, laptops, tv's, etc power needs. What does that mean? That is a major expense for a lot of people.
On the bright side, I hear that power generator companies stocks have gone through the roof.
On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh.
Mike
Nothing. It is extremely cheap, extremely durable, and nearly 100% recyclable. All the things lithium is not. The only thing is lead acid is not power dense, but that is not generally a problem at sites. -Dan On Sat, 28 Dec 2019, Baldur Norddahl wrote:
What is wrong with lead acid battery backup? Seems to be exceedingly stable from my experience. We have all our equipment on -48V DC and have never had a power interruption at any site.
The requirements here are 48 hours of backup by law. Telecom is declared to be part of emergency and defense, so they put in a requirement for resilience.
Regards
Baldur
tor. 26. dec. 2019 11.33 skrev Joe Maimon <jmaimon@jmaimon.com>:
Unless telecom infrastructure has been diligently changing out the lead acid battery approach at all their remote terminals, powered gpon, hfc and antennae plants will never last more than minutes. If at all.
A traditional car has between a 100-200amp alternator @12volts
How much generating capacity can you get out of a typical hybrid?
Self-isolating and re-tieing inverters. Economic household ATS systems. Do those exist?
Enough independent distributed capacity and now comes the ability to create grid islands. How might that look?
Electric grid shortage is likely coming to NYC, courtesy of folk of certain political persuasion and their love of stone age era living. IP decommissioning.
If you have CO loop copper, keep it.
Joe
Don Gould wrote:
This is a very short term problem.
The market is going to fill with battery storage sooner rather than later.
Solar is just exploding.
Your car will "house tie".
6G will solve your data problem.
D
-- Don Gould 5 Cargill Place Richmond Christchurch, New Zealand Mobile/Telegram: + 64 21 114 0699 www. <http://www.tusker.net.au/>bowenvale.co.nz
-------- Original message -------- From: Michael Thomas <mike@mtcc.com> Date: 26/12/19 2:33 PM (GMT+12:00) To: nanog@nanog.org Subject: power to the internet
https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678
This article details some of the issues with California's "new reality" of planned blackouts. One of the big things that came to light with these blackouts is that our network infrastructure's resilience is pretty lacking. While I was (surprisingly to me) ok with my DSL connection out in the boonies, lots and lots of people with cable weren't so lucky. And I'm not sure how bad the situation is with cellular infrastructure, but I assume it's not much better than cable. And I wouldn't doubt that other DSL deployments go dark when power is down. I have no clue with fiber.
So I guess what I'm wondering is what can we do about this? What should we do about this? These days IP access is not just convenience, it's the way we go about our lives, just like electricity itself. At base, it seems to me that network operators should be required to keep the lights on in blackouts just like POTS operators do now. If I have power to light my modem or charge in my phone, I should be able to get onto the net. That seems like table stakes.
One of the things we learned also is that the blackouts seem to last between 2-3 days apiece. I happen to have a generator since I'm out in the boonies and our power gets cut regularly because of snow, but not everyone has that luxury. I kind of want to think that my router+modem use about 20 watts, so powering it up would take about 1.5kwh for 3 days. a quick google look shows that I'd probably need to shell out $500 or so for a battery of that capacity, and that's doesn't include your phones, laptops, tv's, etc power needs. What does that mean? That is a major expense for a lot of people.
On the bright side, I hear that power generator companies stocks have gone through the roof.
On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh.
Mike
To be hyper pedantic, lithium / li-on batteries are recyclable. It’s just not being done today on a substantive scale today for mostly economic reasons. On Sat, Dec 28, 2019 at 14:51 Dan Hollis <goemon@sasami.anime.net> wrote:
Nothing.
It is extremely cheap, extremely durable, and nearly 100% recyclable. All the things lithium is not.
The only thing is lead acid is not power dense, but that is not generally a problem at sites.
-Dan
On Sat, 28 Dec 2019, Baldur Norddahl wrote:
What is wrong with lead acid battery backup? Seems to be exceedingly stable from my experience. We have all our equipment on -48V DC and have never had a power interruption at any site.
The requirements here are 48 hours of backup by law. Telecom is declared to be part of emergency and defense, so they put in a requirement for resilience.
Regards
Baldur
tor. 26. dec. 2019 11.33 skrev Joe Maimon <jmaimon@jmaimon.com>:
Unless telecom infrastructure has been diligently changing out the lead acid battery approach at all their remote terminals, powered gpon, hfc and antennae plants will never last more than minutes. If at all.
A traditional car has between a 100-200amp alternator @12volts
How much generating capacity can you get out of a typical hybrid?
Self-isolating and re-tieing inverters. Economic household ATS systems. Do those exist?
Enough independent distributed capacity and now comes the ability to create grid islands. How might that look?
Electric grid shortage is likely coming to NYC, courtesy of folk of certain political persuasion and their love of stone age era living. IP decommissioning.
If you have CO loop copper, keep it.
Joe
Don Gould wrote:
This is a very short term problem.
The market is going to fill with battery storage sooner rather than later.
Solar is just exploding.
Your car will "house tie".
6G will solve your data problem.
D
-- Don Gould 5 Cargill Place Richmond Christchurch, New Zealand Mobile/Telegram: + 64 21 114 0699 www. <http://www.tusker.net.au/>bowenvale.co.nz
-------- Original message -------- From: Michael Thomas <mike@mtcc.com> Date: 26/12/19 2:33 PM (GMT+12:00) To: nanog@nanog.org Subject: power to the internet
This article details some of the issues with California's "new reality" of planned blackouts. One of the big things that came to light with these blackouts is that our network infrastructure's resilience is pretty lacking. While I was (surprisingly to me) ok with my DSL connection out in the boonies, lots and lots of people with cable weren't so lucky. And I'm not sure how bad the situation is with cellular infrastructure, but I assume it's not much better than cable. And I wouldn't doubt that other DSL deployments go dark when power is down. I have no clue with fiber.
So I guess what I'm wondering is what can we do about this? What should we do about this? These days IP access is not just convenience, it's
way we go about our lives, just like electricity itself. At base, it seems to me that network operators should be required to keep the
https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678 the lights
on in blackouts just like POTS operators do now. If I have power to light my modem or charge in my phone, I should be able to get onto the net. That seems like table stakes.
One of the things we learned also is that the blackouts seem to last between 2-3 days apiece. I happen to have a generator since I'm out in the boonies and our power gets cut regularly because of snow, but not everyone has that luxury. I kind of want to think that my router+modem use about 20 watts, so powering it up would take about 1.5kwh for 3 days. a quick google look shows that I'd probably need to shell out $500 or so for a battery of that capacity, and that's doesn't include your phones, laptops, tv's, etc power needs. What does that mean? That is a major expense for a lot of people.
On the bright side, I hear that power generator companies stocks have gone through the roof.
On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh.
Mike
-48VDC is extremely reliable, we have also never had a power incursion on our DC plant. Any of them. I’m not sure I’d consider it cheap, but it’s not horrifying expensive and it *works* when you deploy enough of it in a 2 or 3N fashion. -Ben
On Dec 28, 2019, at 9:04 AM, Baldur Norddahl <baldur.norddahl@gmail.com> wrote:
What is wrong with lead acid battery backup? Seems to be exceedingly stable from my experience. We have all our equipment on -48V DC and have never had a power interruption at any site.
The requirements here are 48 hours of backup by law. Telecom is declared to be part of emergency and defense, so they put in a requirement for resilience.
Regards
Baldur
tor. 26. dec. 2019 11.33 skrev Joe Maimon <jmaimon@jmaimon.com>:
Unless telecom infrastructure has been diligently changing out the lead acid battery approach at all their remote terminals, powered gpon, hfc and antennae plants will never last more than minutes. If at all.
A traditional car has between a 100-200amp alternator @12volts
How much generating capacity can you get out of a typical hybrid?
Self-isolating and re-tieing inverters. Economic household ATS systems. Do those exist?
Enough independent distributed capacity and now comes the ability to create grid islands. How might that look?
Electric grid shortage is likely coming to NYC, courtesy of folk of certain political persuasion and their love of stone age era living. IP decommissioning.
If you have CO loop copper, keep it.
Joe
Don Gould wrote:
This is a very short term problem.
The market is going to fill with battery storage sooner rather than later.
Solar is just exploding.
Your car will "house tie".
6G will solve your data problem.
D
-- Don Gould 5 Cargill Place Richmond Christchurch, New Zealand Mobile/Telegram: + 64 21 114 0699 www. <http://www.tusker.net.au/>bowenvale.co.nz
-------- Original message -------- From: Michael Thomas <mike@mtcc.com> Date: 26/12/19 2:33 PM (GMT+12:00) To: nanog@nanog.org Subject: power to the internet
https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678
This article details some of the issues with California's "new reality" of planned blackouts. One of the big things that came to light with these blackouts is that our network infrastructure's resilience is pretty lacking. While I was (surprisingly to me) ok with my DSL connection out in the boonies, lots and lots of people with cable weren't so lucky. And I'm not sure how bad the situation is with cellular infrastructure, but I assume it's not much better than cable. And I wouldn't doubt that other DSL deployments go dark when power is down. I have no clue with fiber.
So I guess what I'm wondering is what can we do about this? What should we do about this? These days IP access is not just convenience, it's the way we go about our lives, just like electricity itself. At base, it seems to me that network operators should be required to keep the lights on in blackouts just like POTS operators do now. If I have power to light my modem or charge in my phone, I should be able to get onto the net. That seems like table stakes.
One of the things we learned also is that the blackouts seem to last between 2-3 days apiece. I happen to have a generator since I'm out in the boonies and our power gets cut regularly because of snow, but not everyone has that luxury. I kind of want to think that my router+modem use about 20 watts, so powering it up would take about 1.5kwh for 3 days. a quick google look shows that I'd probably need to shell out $500 or so for a battery of that capacity, and that's doesn't include your phones, laptops, tv's, etc power needs. What does that mean? That is a major expense for a lot of people.
On the bright side, I hear that power generator companies stocks have gone through the roof.
On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh.
Mike
They are quite lacking in longevity and density. Which is fine if you have enough space and adequate maintenance to change them out and they are only a bridge for local generation. Remote installations do not typically have these characteristics. Pretty much every basement mux or other RT telco installation I have ever seen had the original battery setup still in place and lasted approximately 3 seconds when the lights went off. I try to put them into my batteries whenever possibly. Further, many fios/cable customer NID devices do come with batteries, but they are usually lead acid, often the telco charges to replace the battery and they are typically only there to power the POTS lines on the NID. Joe Baldur Norddahl wrote:
What is wrong with lead acid battery backup? Seems to be exceedingly stable from my experience. We have all our equipment on -48V DC and have never had a power interruption at any site.
The requirements here are 48 hours of backup by law. Telecom is declared to be part of emergency and defense, so they put in a requirement for resilience.
Regards
Baldur
tor. 26. dec. 2019 11.33 skrev Joe Maimon <jmaimon@jmaimon.com <mailto:jmaimon@jmaimon.com>>:
Unless telecom infrastructure has been diligently changing out the lead acid battery approach at all their remote terminals, powered gpon, hfc and antennae plants will never last more than minutes. If at all.
A traditional car has between a 100-200amp alternator @12volts
How much generating capacity can you get out of a typical hybrid?
Self-isolating and re-tieing inverters. Economic household ATS systems. Do those exist?
Enough independent distributed capacity and now comes the ability to create grid islands. How might that look?
Electric grid shortage is likely coming to NYC, courtesy of folk of certain political persuasion and their love of stone age era living. IP decommissioning.
If you have CO loop copper, keep it.
Joe
Don Gould wrote: > This is a very short term problem. > > The market is going to fill with battery storage sooner rather than > later. > > Solar is just exploding. > > Your car will "house tie". > > 6G will solve your data problem. > > D > > > > -- > Don Gould > 5 Cargill Place > Richmond > Christchurch, New Zealand > Mobile/Telegram: + 64 21 114 0699 > www. <http://www.tusker.net.au/>bowenvale.co.nz <http://bowenvale.co.nz> > > > > -------- Original message -------- > From: Michael Thomas <mike@mtcc.com <mailto:mike@mtcc.com>> > Date: 26/12/19 2:33 PM (GMT+12:00) > To: nanog@nanog.org <mailto:nanog@nanog.org> > Subject: power to the internet > > > https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678 > > > This article details some of the issues with California's "new reality" > of planned blackouts. One of the big things that came to light with > these blackouts is that our network infrastructure's resilience is > pretty lacking. While I was (surprisingly to me) ok with my DSL > connection out in the boonies, lots and lots of people with cable > weren't so lucky. And I'm not sure how bad the situation is with > cellular infrastructure, but I assume it's not much better than cable. > And I wouldn't doubt that other DSL deployments go dark when power is > down. I have no clue with fiber. > > So I guess what I'm wondering is what can we do about this? What should > we do about this? These days IP access is not just convenience, it's the > way we go about our lives, just like electricity itself. At base, it > seems to me that network operators should be required to keep the lights > on in blackouts just like POTS operators do now. If I have power to > light my modem or charge in my phone, I should be able to get onto the > net. That seems like table stakes. > > One of the things we learned also is that the blackouts seem to last > between 2-3 days apiece. I happen to have a generator since I'm out in > the boonies and our power gets cut regularly because of snow, but not > everyone has that luxury. I kind of want to think that my router+modem > use about 20 watts, so powering it up would take about 1.5kwh for 3 > days. a quick google look shows that I'd probably need to shell out $500 > or so for a battery of that capacity, and that's doesn't include your > phones, laptops, tv's, etc power needs. What does that mean? That is a > major expense for a lot of people. > > On the bright side, I hear that power generator companies stocks have > gone through the roof. > > On the dark side, this is probably coming to a lot more states and > countries due to climate change. Australia. Sigh. > > Mike >
Exactly. And we will build it all. The power stuff is serious people. We’ve gotten letters from the FCC over it. There is additional regulation coming down when people can’t call 911! You need at minimum 8 hours (or your CRT response time with a generator trailer, or a standby generator or two) of battery on your telecom equipment. All of it. Everywhere. Comcast is the worst about this, they never replace and often don’t even place batteries in their RTs at all - and they are going to get fined over it mark my words. -Ben Cannon CEO 6x7 Networks & 6x7 Telecom, LLC ben@6by7.net <mailto:ben@6by7.net>
On Dec 25, 2019, at 8:41 PM, Don Gould <don@bowenvale.co.nz> wrote:
This is a very short term problem.
The market is going to fill with battery storage sooner rather than later.
Solar is just exploding.
Your car will "house tie".
6G will solve your data problem.
D
-- Don Gould 5 Cargill Place Richmond Christchurch, New Zealand Mobile/Telegram: + 64 21 114 0699 www. <http://www.tusker.net.au/>bowenvale.co.nz
-------- Original message -------- From: Michael Thomas <mike@mtcc.com> Date: 26/12/19 2:33 PM (GMT+12:00) To: nanog@nanog.org Subject: power to the internet
https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678
This article details some of the issues with California's "new reality" of planned blackouts. One of the big things that came to light with these blackouts is that our network infrastructure's resilience is pretty lacking. While I was (surprisingly to me) ok with my DSL connection out in the boonies, lots and lots of people with cable weren't so lucky. And I'm not sure how bad the situation is with cellular infrastructure, but I assume it's not much better than cable. And I wouldn't doubt that other DSL deployments go dark when power is down. I have no clue with fiber.
So I guess what I'm wondering is what can we do about this? What should we do about this? These days IP access is not just convenience, it's the way we go about our lives, just like electricity itself. At base, it seems to me that network operators should be required to keep the lights on in blackouts just like POTS operators do now. If I have power to light my modem or charge in my phone, I should be able to get onto the net. That seems like table stakes.
One of the things we learned also is that the blackouts seem to last between 2-3 days apiece. I happen to have a generator since I'm out in the boonies and our power gets cut regularly because of snow, but not everyone has that luxury. I kind of want to think that my router+modem use about 20 watts, so powering it up would take about 1.5kwh for 3 days. a quick google look shows that I'd probably need to shell out $500 or so for a battery of that capacity, and that's doesn't include your phones, laptops, tv's, etc power needs. What does that mean? That is a major expense for a lot of people.
On the bright side, I hear that power generator companies stocks have gone through the roof.
On the dark side, this is probably coming to a lot more states and countries due to climate change. Australia. Sigh.
Mike
On 12/26/19 10:41 AM, Ben Cannon wrote:
Exactly. And we will build it all.
The power stuff is serious people. We’ve gotten letters from the FCC over it. There is additional regulation coming down when people can’t call 911!
You need at minimum 8 hours (or your CRT response time with a generator trailer, or a standby generator or two) of battery on your telecom equipment. All of it. Everywhere.
Comcast is the worst about this, they never replace and often don’t even place batteries in their RTs at all - and they are going to get fined over it mark my words.
Here in California, you're going to need a lot more than 8 hours. We had one that lasted 3 days, followed by about 8 hours of power, followed by 2 days of no power. If this is the new normal, and I'm afraid that it is, that's probably going to require some pretty hefty backup. Not to mention expensive. The one "good" thing that PG&E did is expose all of these vulnerabilities. Every neighborhood probably knows whether their carrier is naughty or nice now. Mike
On 12/26/19 10:55 AM, Michael Thomas wrote:
Here in California, you're going to need a lot more than 8 hours. We had one that lasted 3 days, followed by about 8 hours of power, followed by 2 days of no power. If this is the new normal, and I'm afraid that it is, that's probably going to require some pretty hefty backup. Not to mention expensive.
The one "good" thing that PG&E did is expose all of these vulnerabilities. Every neighborhood probably knows whether their carrier is naughty or nice now.
Here in Nevada, specifically at Lake Tahoe, power is less reliable because of heavy snow and sliding trucks (the power equivalent to a backhoe disconnect). One of the cell sites is on the top level of a casino parking garage. I found out about this when the casino went bankrupt, the parking garage was blocked off, and I joined the security guard crew to protect the on-site gaming equipment. Months into the project, the cell company in question begged the bankruptcy court for access -- to replace the empty propane cylinders in their shack. That's right, no mains tap at all. When the casino lost power because of bill non-payment, the cell site stayed up. A network operator will need to look at the total cost, including labor, of backing up mains power. versus using local genertion exclusively -- or using mains power as the backup! Factor in any upcoming fines for service outage, re 911. (Try to avoid piped natural gas as the fuel for onsite generation.) Longer term, review your backhauls and interconnects. Dark fiber would be preferred here, because you would be controlling backup power at both ends, and not depending on intermediate nodes.
On 12/26/19 11:18 AM, Stephen Satchell wrote:
On 12/26/19 10:55 AM, Michael Thomas wrote:
Here in California, you're going to need a lot more than 8 hours. We had one that lasted 3 days, followed by about 8 hours of power, followed by 2 days of no power. If this is the new normal, and I'm afraid that it is, that's probably going to require some pretty hefty backup. Not to mention expensive.
The one "good" thing that PG&E did is expose all of these vulnerabilities. Every neighborhood probably knows whether their carrier is naughty or nice now.
Here in Nevada, specifically at Lake Tahoe, power is less reliable because of heavy snow and sliding trucks (the power equivalent to a backhoe disconnect). One of the cell sites is on the top level of a casino parking garage. I found out about this when the casino went bankrupt, the parking garage was blocked off, and I joined the security guard crew to protect the on-site gaming equipment. Months into the project, the cell company in question begged the bankruptcy court for access -- to replace the empty propane cylinders in their shack. That's right, no mains tap at all. When the casino lost power because of bill non-payment, the cell site stayed up.
A network operator will need to look at the total cost, including labor, of backing up mains power. versus using local genertion exclusively -- or using mains power as the backup! Factor in any upcoming fines for service outage, re 911. (Try to avoid piped natural gas as the fuel for onsite generation.)
Longer term, review your backhauls and interconnects. Dark fiber would be preferred here, because you would be controlling backup power at both ends, and not depending on intermediate nodes.
One of the interesting things I found out is that POTS termination out in the field can be powered by some of the pairs back to the CO they are making redundant. It's enough power that running the DSLAM isn't a problem either. I'm not sure that that could translate for anything else, but there is probably a lot of copper sitting idle these days. Mike
On Thu, 26 Dec 2019 at 13:19, Stephen Satchell <list@satchell.net> wrote:
Longer term, review your backhauls and interconnects. Dark fiber would be preferred here, because you would be controlling backup power at both ends, and not depending on intermediate nodes.
What about the NSA taps? Do they tap the dark fibre? Would it go dark in a power outage, or do they engage some passive sort of taps? C.
AT&T land line had service trucks parked at RT’s to power them. I talked with one of the techs. He was on a 12 hour schedule and spent that time between 3 sites charging the batteries to keep the copper plant running. They plugged in to the truck inverter and ran the truck all day. He told me they ran out of generators. Talk about a waste of manpower. On Thu, Dec 26, 2019 at 10:56 AM Michael Thomas <mike@mtcc.com> wrote:
On 12/26/19 10:41 AM, Ben Cannon wrote:
Exactly. And we will build it all.
The power stuff is serious people. We’ve gotten letters from the FCC over it. There is additional regulation coming down when people can’t call 911!
You need at minimum 8 hours (or your CRT response time with a generator trailer, or a standby generator or two) of battery on your telecom equipment. All of it. Everywhere.
Comcast is the worst about this, they never replace and often don’t even place batteries in their RTs at all - and they are going to get fined over it mark my words.
Here in California, you're going to need a lot more than 8 hours. We had one that lasted 3 days, followed by about 8 hours of power, followed by 2 days of no power. If this is the new normal, and I'm afraid that it is, that's probably going to require some pretty hefty backup. Not to mention expensive.
The one "good" thing that PG&E did is expose all of these vulnerabilities. Every neighborhood probably knows whether their carrier is naughty or nice now.
Mike
--
Jason Wilson Remotely Located Providing High Speed Internet to out of the way places. 530-651-1736 530-748-9608 Cell www.remotelylocated.com
participants (23)
-
Baldur Norddahl -
Ben Cannon -
Brandon Butterworth -
Constantine A. Murenin -
Dan Hollis -
Don Gould -
Florian Weimer -
Fred Baker -
Howard Leadmon -
Jason Wilson -
Joe Maimon -
John Levine -
John Lightfoot -
John R. Levine
-
Keith Medcalf -
Mark Tinka -
Michael Loftis -
Michael Thomas -
Mike Bolitho -
Sabri Berisha -
Stephen Satchell -
tim＠pelican.org -
Tom Beecher