Comcast outages continue even in areas with PG&E power restored
The FCC asked a half-dozen carriers about their network resilience plans last month. Comcast was not one of the service providers askedd about their plans. The FCC should have looked closer at Comcast in California. While it was expected many people would loose home Internet, voice, video service when their Customer Premise Equipment lost power. The FCC no longer requires battery backup for CPE. That is now a customer responsibility. It turns out, Comcast's outside plant was woefully unprepared to handle long, i.e. 24 hour, power outages. And even when power is restored to people's homes, Comcast service is often still down. https://www.ktvu.com/news/still-no-relief-pge-shutoffs-disrupt-comcast-servi... Maybe the FCC needs to ask about cable system Outside Service Plant network reslience of Comcast. Are other cable OSP resilience similar?
On 10/11/19 4:31 PM, Sean Donelan wrote:
The FCC asked a half-dozen carriers about their network resilience plans last month. Comcast was not one of the service providers askedd about their plans.
The FCC should have looked closer at Comcast in California. While it was expected many people would loose home Internet, voice, video service when their Customer Premise Equipment lost power. The FCC no longer requires battery backup for CPE. That is now a customer responsibility.
It turns out, Comcast's outside plant was woefully unprepared to handle long, i.e. 24 hour, power outages. And even when power is restored to people's homes, Comcast service is often still down.
So I knew that telcos are required to battery backup pots, but are isp's too? I have a dinky little provider who also provides pots, but i have never been clear whether dsl stays up too in a blackout. Mike
On Fri, 11 Oct 2019, Michael Thomas wrote:
On 10/11/19 4:31 PM, Sean Donelan wrote: The FCC asked a half-dozen carriers about their network resilience plans last month. Comcast was not one of the service providers askedd about their plans.
The FCC should have looked closer at Comcast in California. While it was expected many people would loose home Internet, voice, video service when their Customer Premise Equipment lost power. The FCC no longer requires battery backup for CPE. That is now a customer responsibility.
It turns out, Comcast's outside plant was woefully unprepared to handle long, i.e. 24 hour, power outages. And even when power is restored to people's homes, Comcast service is often still down.
So I knew that telcos are required to battery backup pots, but are isp's too? I have a dinky little provider who also provides pots, but i have never been clear whether dsl stays up too in a blackout.
Mike
First of all DSL is not pots. Traditional voice services run on a subscriber loop which is a pair of copper lines running from the central office to the customer end point. This analog voice service is almost always backed up with a bank of batteries so that the service continues to run in the event of an emergency. DSL is a data service that runs on the subscriber loop at the same time as the voice service. This service is not required to be battery backed and will invariably stop working when power is cut at the customer end point. Ted
On Fri, 11 Oct 2019, Ted Hatfield wrote:
First of all DSL is not pots.
[....]
DSL is a data service that runs on the subscriber loop at the same time as the voice service. This service is not required to be battery backed and will invariably stop working when power is cut at the customer end point.
That is not why people are surprised. When the house doesn't have power, and doesn't have home generator or UPS, (most) people are less surprised their DSL or Cable modem and VOIP doesn't work anymore. The reasons I saw people angry on twitter was no Comcast service even when they had power at the house (utility, generator, UPS). Their Comcast service died quickly, even when the home had power but the Comcast outside plant didn't seem to have any backup power. DSL modems also need power at the home, but the telco providers seem to have more backup power in the outside plant or central offices. That meant DSL worked as long as the house had power (or a home generator or UPS). I know, rich people problems. Rich people can afford backup generators and got upset when their Internet and TV didn't work.
On 10/14/19 3:06 PM, Sean Donelan wrote:
That is not why people are surprised. When the house doesn't have power, and doesn't have home generator or UPS, (most) people are less surprised their DSL or Cable modem and VOIP doesn't work anymore.
The reasons I saw people angry on twitter was no Comcast service even when they had power at the house (utility, generator, UPS). Their Comcast service died quickly, even when the home had power but the Comcast outside plant didn't seem to have any backup power.
DSL modems also need power at the home, but the telco providers seem to have more backup power in the outside plant or central offices. That meant DSL worked as long as the house had power (or a home generator or UPS).
So it turns out that our local telco/isp does keep dsl running via the same mechanism as they keep pots power backed up (i'm guessing it's a diesel generator at the co, but am not sure). It seems that a lot of the pedestals terminating the local loop these days do the conversion to IP right there with sip/h.248/mgcp/rtp. I'm not sure how they get power to the pedestal, but these were all a home run to the co at one time so it probably wasn't hard to power them from the co. For all i know, that's how they're all powered all the time, with a transfer switch at the co, rather than tapping the local grid next to the pedestal. Of course this is a lot of conjecture on my part... be glad to be clued in by folks in know. Mike
On Mon, 14 Oct 2019, Michael Thomas wrote:
Of course this is a lot of conjecture on my part... be glad to be clued in by folks in know.
An old news story, but telco's usually have backup batteries in their outside plant, cell towers, etc. During power outages, they shuttle small generators between outside cabinets to re-charge the batteries. Remote Terminal Units (RTUs) use local power, i.e. look for the utility meter nearby. There is often a generator plug and battery cabinet next to the RTU. They aren't powered from the central office. Some cable systems have battery and/or generator backup on their "I-Net" cable plant serving government and major businesses, but not on their residential cable plant. I don't know Comcast's business practices. Old news story: https://www.multichannel.com/news/att-will-replace-batteries-after-fires-130... ORIGINAL: JAN 18, 2008 AT&T Will Replace Batteries After Fires City officials, long critical of the size and placement of powering cabinets needed to back up AT&T’s U-verse TV video service, now have concerns beyond aesthetics. Sometimes, the cabinets explode. AT&T acknowledged the problem and said it would replace 17,000 lithium batteries in outdoor cabinets around the country. [...] The steel cabinets house controls and backup power supplies for the video network. “They’ve been pretty cooperative,” Kesner said of AT&T. “We’re in a holding pattern” regarding the video deployment, he said. [...]
On 10/14/19 4:16 PM, Sean Donelan wrote:
On Mon, 14 Oct 2019, Michael Thomas wrote:
Of course this is a lot of conjecture on my part... be glad to be clued in by folks in know.
An old news story, but telco's usually have backup batteries in their outside plant, cell towers, etc. During power outages, they shuttle small generators between outside cabinets to re-charge the batteries. Remote Terminal Units (RTUs) use local power, i.e. look for the utility meter nearby. There is often a generator plug and battery cabinet next to the RTU. They aren't powered from the central office.
Interesting! And so primitive! So they go to all of the expense of laying fiber, but not power too? Mike
On Mon, Oct 14, 2019 at 4:26 PM Michael Thomas <mike@mtcc.com> wrote:
On 10/14/19 4:16 PM, Sean Donelan wrote:
On Mon, 14 Oct 2019, Michael Thomas wrote:
Of course this is a lot of conjecture on my part... be glad to be clued in by folks in know.
An old news story, but telco's usually have backup batteries in their outside plant, cell towers, etc. During power outages, they shuttle small generators between outside cabinets to re-charge the batteries. Remote Terminal Units (RTUs) use local power, i.e. look for the utility meter nearby. There is often a generator plug and battery cabinet next to the RTU. They aren't powered from the central office.
Interesting! And so primitive! So they go to all of the expense of laying fiber, but not power too?
Note: small local telco experience speaking below: Telco's tend to have experience with fiber, but probably not the construction and transmission of the sort of power plant that would be required to keep a bunch of 48V cabinets up and running reliably. We certainly don't. Besides, an advantage of fiber is that hopefully the copper thieves won't bother it. By definition a remote terminal/cabinet is going to be... remote. Far more simple to install commercial power, and then haul out a generator if the battery string in the cabinet appears to be in danger of dropping below about 46v. We do run some 360v DC at micro-amp levels out to equipment like ONT's and remote 12 and 48 port remote VDSLAM's. But that's over existing 24-26 ga. plant. Frequently using multiple pairs to avoid excessive voltage drop over distances. Primitive is tested and works. -- Jeff Shultz Central Office Technician -- Like us on Social Media for News, Promotions, and other information!! <https://www.facebook.com/SCTCWEB/> <https://www.instagram.com/sctc_503/> <https://www.yelp.com/biz/sctc-stayton-3> <https://www.youtube.com/c/sctcvideos> _**** This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. ****_
On 10/16/19 12:09 PM, Jeff Shultz wrote:
Interesting! And so primitive! So they go to all of the expense of laying fiber, but not power too? Note: small local telco experience speaking below:
Telco's tend to have experience with fiber, but probably not the construction and transmission of the sort of power plant that would be required to keep a bunch of 48V cabinets up and running reliably. We certainly don't. Besides, an advantage of fiber is that hopefully the copper thieves won't bother it.
By definition a remote terminal/cabinet is going to be... remote. Far more simple to install commercial power, and then haul out a generator if the battery string in the cabinet appears to be in danger of dropping below about 46v.
We do run some 360v DC at micro-amp levels out to equipment like ONT's and remote 12 and 48 port remote VDSLAM's. But that's over existing 24-26 ga. plant. Frequently using multiple pairs to avoid excessive voltage drop over distances.
Primitive is tested and works.
This is all very interesting, and thanks to everybody for giving me an education. My provider is very small as well, and spread out over a pretty large area (i'm in amador county in the mother lode). I don't know how many remote terminals they have, but i would think that it would be a lot. And if they need to be recharged every 8 hours or so, you'd be talking about a lot of people out in the field just to keep the lights on, right? And of course it takes time to recharge a battery too, so that makes it even worse. It seems that would be a pretty significant recurring cost. How many watts does a typical remote terminal draw per subscriber? Mike
After some poking around, I found this gizmo. It says that it can use between 1-8 pairs to power it from the co. If there was already a home run to the co (which is almost certainly true in my case), it seems like that would be a cheaper option? Then you just have one diesel generator at the co that charges the batteries. https://portal.adtran.com/pub/Library/Data_Sheets/International_/I61179918F1... Mike On 10/16/19 12:09 PM, Jeff Shultz wrote:
On Mon, Oct 14, 2019 at 4:26 PM Michael Thomas <mike@mtcc.com> wrote:
On 10/14/19 4:16 PM, Sean Donelan wrote:
On Mon, 14 Oct 2019, Michael Thomas wrote:
Of course this is a lot of conjecture on my part... be glad to be clued in by folks in know. An old news story, but telco's usually have backup batteries in their outside plant, cell towers, etc. During power outages, they shuttle small generators between outside cabinets to re-charge the batteries. Remote Terminal Units (RTUs) use local power, i.e. look for the utility meter nearby. There is often a generator plug and battery cabinet next to the RTU. They aren't powered from the central office. Interesting! And so primitive! So they go to all of the expense of laying fiber, but not power too?
Note: small local telco experience speaking below:
Telco's tend to have experience with fiber, but probably not the construction and transmission of the sort of power plant that would be required to keep a bunch of 48V cabinets up and running reliably. We certainly don't. Besides, an advantage of fiber is that hopefully the copper thieves won't bother it.
By definition a remote terminal/cabinet is going to be... remote. Far more simple to install commercial power, and then haul out a generator if the battery string in the cabinet appears to be in danger of dropping below about 46v.
We do run some 360v DC at micro-amp levels out to equipment like ONT's and remote 12 and 48 port remote VDSLAM's. But that's over existing 24-26 ga. plant. Frequently using multiple pairs to avoid excessive voltage drop over distances.
Primitive is tested and works.
We use 12 and 48 port VDSLAM's similar to that at some of our remote locations, and we do generally line power those. But before those came on the market we were putting out remote cabinets that could support up to 144 subscribers fed off the same sort of cards you would find in the CO. I don't know our power budget per customer, but it's not unusual to have 20 or more amps of capacity (probably overkill, likely because that was the size available) at 48V in a cabinet. Because the CO cards aren't hardened, the cabinet must be - and have some HVAC type capabilities as well - at least fans. We're now feeding line power out to some of the 12 and 48 port devices like you linked to from some of those remote cabinets. It's all about shrinking loop lengths until we can get both the time and funds to put fiber in the ground everywhere. On Wed, Oct 16, 2019 at 1:06 PM Michael Thomas <mike@mtcc.com> wrote:
After some poking around, I found this gizmo. It says that it can use between 1-8 pairs to power it from the co. If there was already a home run to the co (which is almost certainly true in my case), it seems like that would be a cheaper option? Then you just have one diesel generator at the co that charges the batteries.
https://portal.adtran.com/pub/Library/Data_Sheets/International_/I61179918F1...
Mike
On 10/16/19 12:09 PM, Jeff Shultz wrote:
On Mon, Oct 14, 2019 at 4:26 PM Michael Thomas <mike@mtcc.com> wrote:
On 10/14/19 4:16 PM, Sean Donelan wrote:
On Mon, 14 Oct 2019, Michael Thomas wrote:
Of course this is a lot of conjecture on my part... be glad to be clued in by folks in know. An old news story, but telco's usually have backup batteries in their outside plant, cell towers, etc. During power outages, they shuttle small generators between outside cabinets to re-charge the batteries. Remote Terminal Units (RTUs) use local power, i.e. look for the utility meter nearby. There is often a generator plug and battery cabinet next to the RTU. They aren't powered from the central office. Interesting! And so primitive! So they go to all of the expense of laying fiber, but not power too?
Note: small local telco experience speaking below:
Telco's tend to have experience with fiber, but probably not the construction and transmission of the sort of power plant that would be required to keep a bunch of 48V cabinets up and running reliably. We certainly don't. Besides, an advantage of fiber is that hopefully the copper thieves won't bother it.
By definition a remote terminal/cabinet is going to be... remote. Far more simple to install commercial power, and then haul out a generator if the battery string in the cabinet appears to be in danger of dropping below about 46v.
We do run some 360v DC at micro-amp levels out to equipment like ONT's and remote 12 and 48 port remote VDSLAM's. But that's over existing 24-26 ga. plant. Frequently using multiple pairs to avoid excessive voltage drop over distances.
Primitive is tested and works.
-- Jeff Shultz Central Office Technician -- Like us on Social Media for News, Promotions, and other information!! <https://www.facebook.com/SCTCWEB/> <https://www.instagram.com/sctc_503/> <https://www.yelp.com/biz/sctc-stayton-3> <https://www.youtube.com/c/sctcvideos> _**** This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. ****_
On 10/16/19 4:04 PM, Michael Thomas wrote:
After some poking around, I found this gizmo. It says that it can use between 1-8 pairs to power it from the co. If there was already a home run to the co (which is almost certainly true in my case), it seems like that would be a cheaper option? Then you just have one diesel generator at the co that charges the batteries.
Yep, things like this are a great option if you're overbuilding existing copper plant with fiber. The old copper gets relegated to duty as a power carrier, and the fiber moves the bits to the DSLAM. As another poster said, you just keep pushing these out to keep loop lengths down and get the bandwidth available to the end user up. G.FAST is the next iteration of this sort of thing. You run fiber all the way to the ped at the curb or even into the building for multi-dwelling applications then re-use the existing drop to get into the customer prem. 4-8 ports is common on these types of things. Many support either remote span power using the old copper plant or sometimes also reverse power from the customer prem which is really handy if your a pure-play fiber carrier re-using existing customer-owned copper infrastructure or if your copper plant has rotted to the point that you're loathe to put 190VDC on it for a few miles from the nearest powered RTU or CO. The actual power that's needed per port is usually pretty small. Maybe a dozen watts or so. There's obviously a base load on the unit, so the more ports you have lit the lower that per-port number will go with diminishing returns. It's low enough that, at 190VDC, you can feasibly power things over a mile or more with just a few pairs of existing 24AWG outside copper without the voltage drop or power loss and cable heating being too bad. -- Brandon Martin
On 10/14/19 6:38 PM, Michael Thomas wrote:
So it turns out that our local telco/isp does keep dsl running via the same mechanism as they keep pots power backed up (i'm guessing it's a diesel generator at the co, but am not sure). It seems that a lot of the pedestals terminating the local loop these days do the conversion to IP right there with sip/h.248/mgcp/rtp. I'm not sure how they get power to the pedestal, but these were all a home run to the co at one time so it probably wasn't hard to power them from the co. For all i know, that's how they're all powered all the time, with a transfer switch at the co, rather than tapping the local grid next to the pedestal.
Of course this is a lot of conjecture on my part... be glad to be clued in by folks in know.
Legacy carriers with existing copper plant being overbuilt by fiber often use "span power" at 48V or ~190VDC from the nearest central facility (which may be an old-school CO or RTU with utility power) to power their distributed plant that's deep into the network near the customer edge. There's usually then also some local battery as well but not much. The assumption is that the legacy copper, now being used simply for power, follows essentially the same routes as the fiber and will drop at the same time. In a cable cut situation this is often true, but in a "cable rot" situation it's obviously not, and fiber paths aren't always the same as power paths. The CO and older RTUs from the POTS and early ADSL era will have utility power as primary with somewhat extensive battery facilities. Old-school CO will have a lot more battery capacity than an RTU. An RTU will usually end up with a portable genset being delivered during an extended power outage as mentioned. These RTUs still host legacy POTS and TDM services that either have serious SLAs on them or regulatory uptime considerations whereas distributed peds are normally best-effort, non-regulated data services only in many cases precisely to keep the reliability requirements (and therefore cost) on them down. SBC/AT&T's pedestals they built for Lightspeed (U-Verse) do usually have local utility with a few hours of backup. I'm not sure why the went that route rather than span power. Might be that the early VDSL DSLAMs just used too much power for that. They seem to only have maybe a day or so of battery before they need a portable generator brought around which AT&T at least has procedures for (whether they are executed or not is another matter). They're something of a hybrid between a conventional RTU and modern distributed pedestal. All the conventional telcos are far more focused on keeping voice service alive since they get raked over the coals by the FCC if it drops due to lack of 911. That includes wireless if they are both a wireline and wireless operator. Interestingly, VoIP service delivered to the customer as such (even if there's an ATA built into the customer's "modem") often gets a pass on this since it's not considered POTS. Same goes for SLA'd business services, too especially T1s since those may host regulated voice. Coax operators have historically had less need for reliability as they were originally built purely for convenience services (cable television) and have been pressed into service for more modern data needs. Those "Alpha boxes" you may see around providing line power do usually have some batteries in them, though they're often ill-maintained and only provide maybe an hour of hold-up at most. Exclusively residential areas sometimes have zero hold-up ability at all. They'll drop at least outside node based digital services (e.g. DOCSIS) as soon as power falls along with anything being distributed into the field on AM fiber carriers. It's not unheard of for conventional linear TV to still be delivered into the field from a head-end at baseband on coax and sometimes that'll stay up longer depending on RF power/split budget as long as the local RF head end still has power. Pure-play fiber carriers, especially PON-based, get to turn what is often a curse in terms of design into a blessing, here. They usually have almost no active outside plant or, if they do, it's less distributed and can afford reasonable backup power infrastructure. While it's annoying to have no power available as you approach customer prem, that forces you to make choices such that there's no worry about backup until you can (usually) just make it the customer's responsibility right at the demarc. Your CO/head-end needs backup, of course, but that's usually a facility that can afford it. -- Brandon Martin
On 10/14/19 4:39 PM, Brandon Martin wrote:
All the conventional telcos are far more focused on keeping voice service alive since they get raked over the coals by the FCC if it drops due to lack of 911. That includes wireless if they are both a wireline and wireless operator. Interestingly, VoIP service delivered to the customer as such (even if there's an ATA built into the customer's "modem") often gets a pass on this since it's not considered POTS. Same goes for SLA'd business services, too especially T1s since those may host regulated voice.
So when we were working on this 20 years ago at Cisco, there was a tremendous amount of effort to deal with the issue of e911 and generally battery backup. I'm really surprised to hear that though we went through a lot of effort to deal with the CPE, that the cable plant was the actual problem. The cable companies should, imo, be held to the same standard as the telcos. Maybe even moreso these days since IP has taken over everything. The need for reliable e911 hasn't gone away just because the bits have turned into IP bit these days. Mike
On 10/14/19 8:26 PM, Michael Thomas wrote:
So when we were working on this 20 years ago at Cisco, there was a tremendous amount of effort to deal with the issue of e911 and generally battery backup. I'm really surprised to hear that though we went through a lot of effort to deal with the CPE, that the cable plant was the actual problem. The cable companies should, imo, be held to the same standard as the telcos. Maybe even moreso these days since IP has taken over everything. The need for reliable e911 hasn't gone away just because the bits have turned into IP bit these days.
They get around it, at least in part, by selling it as a "VoIP" service rather than "phone service". AT&T does the same with U-Verse voice. You can still buy POTS from AT&T, but it's a separate product with a completely different pricing structure from the U-Verse voice product. Voice over HFC networks is sometimes sold as a POTS-like service. I've only heard of this happening in places where the LEC and cable provider happen to end up being one-in-the-same. In those cases, yeah uptime is a big deal. I think what happens is that the standards get written and equipment designed with the assumption that everybody will be deploying all sorts of SLA'd, guaranteed services, then 99% of deployments end up being exclusively best-effort because it's so much easier and cheaper to deploy. GPON seems to be an interesting case of this since it's commonly deployed by telcos rather than cable MSOs, and, in greenfield applications, is often deployed exclusive to copper plant at all. It's pretty common to find GPON ONTs with inbuild UPS monitoring and communications as well as ATAs designed to deliver POTS-like service, but then a lot of SPs who are NOT the LEC of record just use that infrastructure to deliver VoIP-like services and push the UPS responsibility off onto the customer. -- Brandon Martin
On 10/14/19 5:58 PM, Brandon Martin wrote:
On 10/14/19 8:26 PM, Michael Thomas wrote:
So when we were working on this 20 years ago at Cisco, there was a tremendous amount of effort to deal with the issue of e911 and generally battery backup. I'm really surprised to hear that though we went through a lot of effort to deal with the CPE, that the cable plant was the actual problem. The cable companies should, imo, be held to the same standard as the telcos. Maybe even moreso these days since IP has taken over everything. The need for reliable e911 hasn't gone away just because the bits have turned into IP bit these days.
They get around it, at least in part, by selling it as a "VoIP" service rather than "phone service".
AT&T does the same with U-Verse voice. You can still buy POTS from AT&T, but it's a separate product with a completely different pricing structure from the U-Verse voice product.
Voice over HFC networks is sometimes sold as a POTS-like service. I've only heard of this happening in places where the LEC and cable provider happen to end up being one-in-the-same. In those cases, yeah uptime is a big deal.
That's what we were working on at Cisco... we partnered with Videotron up in Montreal and were trying to boil the entire telco ocean from the get-go with all of its features, guarantees, etc. The way it's turned out, nobody really cares about 90% of those features so we were wasting valuable time to get out an mvp instead. Hindsight, of course. I never did hear whether all of the elaborate QoS schemes that John Chapman and the rest of the docsis folks were working on ended up getting used, or whether they really matter much anymore since voice is so low bandwidth. Mike
On Mon, 14 Oct 2019, Michael Thomas wrote:
deal with the CPE, that the cable plant was the actual problem. The cable companies should, imo, be held to the same standard as the telcos. Maybe even moreso these days since IP has taken over everything. The need for reliable e911 hasn't gone away just because the bits have turned into IP bit these days.
Oh, but they are equal. The telco's went to the regulators and got the FCC and state PUCs to reduce or make backup power a customerresponsbility... Just like the cable companies. So now they are equal -- in the race to the bottom. Service providers must "OFFER" customers an OPTION for 8/24-hour standby backup power. The decision to puchase backup power is up to the customer. I assume you read the fine print on the back of your bill or the order terms (subject to change at anytime, without notice). The FCC is looking at standby power for cellular towers, but hasn't been paying attention to wireline and cable systems outside plant power. As I mentioned a few postings ago, cellular/wireless systems have been getting more resiliant. Wireline/cable systems have been getting less reliable.
Sean Donelan wrote: Given that providers can't supply power to mobile phones, that sending power over fiber is extremely eye unsafe and that most CPEs are routers which themselves are useless without end systems, it is reasonable that providers are not required to supply power to home. But,
The FCC is looking at standby power for cellular towers, but hasn't been paying attention to wireline and cable systems outside plant power.
why they are not equal? Masataka Ohta
I disagree with the statement that providers should not be required to backup their networks.... while I don't think it should be an FCC requirement, I do believe the providers have an obligation to do that. That's one of the reasons we generally opt for larger node sizes. Yes, it does cut down on the total bandwidth you can push out to end users, but in a disaster its much faster to recover as well as provide backup power that is up 24x7 vs hundreds of little nodes.
On 10/15/2019 09:42, Matt Hoppes wrote:
I disagree with the statement that providers should not be required to backup their networks.... while I don't think it should be an FCC requirement, I do believe the providers have an obligation to do that.
That's one of the reasons we generally opt for larger node sizes. Yes, it does cut down on the total bandwidth you can push out to end users, but in a disaster its much faster to recover as well as provide backup power that is up 24x7 vs hundreds of little nodes.
I'm generally OK with the provider pushing demarc CPE backup power off onto the customer unless they're providing "POTS-like" service. Aside from POTS, I don't think there's ever been expectation of telecommunications services working during a power outage unless the end user provides backup power. I'm not even sure residential deployments of ISDN came with power for customer-prem ATAs, etc. unless you made special arrangements, but ISDN was never popular here, so I'm not sure. If you have a T1 PRI, the LEC will probably keep it up to your prem, but you're responsible for keeping whatever you're plugging that T1 into up during a power outage. Heck, they may not even do that. I've certainly seen T1 smartjacks hosting real, honest PRIs that were line-powering repeaters without battery on them, so who knows even in that case. After all, the user has to keep the equipment they're using up, too. That's certainly not the provider's responsibility. Telcos aren't power utilities (usually, and when they are it's generally a separate operation). Yes, I understand that many users have wireless/portable in-home equipment with batteries of their own and are often using all-in-one demarc/routers provided by their provider, nowadays... I do think that providers should generally at least offer some basic guidance on how their users might accomplish this even going so far as to provide, install, and maintain such means if paid to do so, but I don't think it's something to be expected when you're buying cable TV or public Internet access service. Again, POTS-like service is different. I do feel that modern IP providers really should strive to keep their end of the network up regardless. That is, even during a reasonable power outage, providers should be able to deliver usable signal to their customers absent actual damage to plant. Exceptions made for natural disasters, fuel shortages, and widespread, extended power outages that just overcome the ability to shuttle portable gensets around fast enough to keep batteries charged. However, unless you're otherwise subject to regulation due to monopoly status, having received ample amounts of build-out subsidy, or providing historically regulated telecommunications services, I'm not sure that regulation is the best way to achieve this. -- Brandon Martin
Except I’m not talking about CPE. I agree that’s the customer’s job. I’m talking about keeping the nodes up and running.
On Oct 15, 2019, at 7:50 AM, Brandon Martin <lists.nanog@monmotha.net> wrote:
On 10/15/2019 09:42, Matt Hoppes wrote: I disagree with the statement that providers should not be required to backup their networks.... while I don't think it should be an FCC requirement, I do believe the providers have an obligation to do that. That's one of the reasons we generally opt for larger node sizes. Yes, it does cut down on the total bandwidth you can push out to end users, but in a disaster its much faster to recover as well as provide backup power that is up 24x7 vs hundreds of little nodes.
I'm generally OK with the provider pushing demarc CPE backup power off onto the customer unless they're providing "POTS-like" service. Aside from POTS, I don't think there's ever been expectation of telecommunications services working during a power outage unless the end user provides backup power. I'm not even sure residential deployments of ISDN came with power for customer-prem ATAs, etc. unless you made special arrangements, but ISDN was never popular here, so I'm not sure. If you have a T1 PRI, the LEC will probably keep it up to your prem, but you're responsible for keeping whatever you're plugging that T1 into up during a power outage. Heck, they may not even do that. I've certainly seen T1 smartjacks hosting real, honest PRIs that were line-powering repeaters without battery on them, so who knows even in that case.
After all, the user has to keep the equipment they're using up, too. That's certainly not the provider's responsibility. Telcos aren't power utilities (usually, and when they are it's generally a separate operation). Yes, I understand that many users have wireless/portable in-home equipment with batteries of their own and are often using all-in-one demarc/routers provided by their provider, nowadays...
I do think that providers should generally at least offer some basic guidance on how their users might accomplish this even going so far as to provide, install, and maintain such means if paid to do so, but I don't think it's something to be expected when you're buying cable TV or public Internet access service. Again, POTS-like service is different.
I do feel that modern IP providers really should strive to keep their end of the network up regardless. That is, even during a reasonable power outage, providers should be able to deliver usable signal to their customers absent actual damage to plant. Exceptions made for natural disasters, fuel shortages, and widespread, extended power outages that just overcome the ability to shuttle portable gensets around fast enough to keep batteries charged. However, unless you're otherwise subject to regulation due to monopoly status, having received ample amounts of build-out subsidy, or providing historically regulated telecommunications services, I'm not sure that regulation is the best way to achieve this.
-- Brandon Martin
On 10/14/19 6:11 PM, Sean Donelan wrote:
On Mon, 14 Oct 2019, Michael Thomas wrote:
deal with the CPE, that the cable plant was the actual problem. The cable companies should, imo, be held to the same standard as the telcos. Maybe even moreso these days since IP has taken over everything. The need for reliable e911 hasn't gone away just because the bits have turned into IP bit these days.
Oh, but they are equal. The telco's went to the regulators and got the FCC and state PUCs to reduce or make backup power a customerresponsbility... Just like the cable companies.
So now they are equal -- in the race to the bottom.
Service providers must "OFFER" customers an OPTION for 8/24-hour standby backup power. The decision to puchase backup power is up to the customer. I assume you read the fine print on the back of your bill or the order terms (subject to change at anytime, without notice).
Assuming that this power shutoff in california is the new normal (god help us), it would need to be a minimum of 3 days. The wind event itself usually lasts a day or two, but then they have to inspect -- completely manually from what i can tell -- the entire grid, which takes a day or two. Clearly more automation of the inspection is needed (ie, have cams, etc, deployed on the grid), and of course that would help with prevention which is where pge has fallen completely flat. It seems to me that if we have to live with unburied power lines, cameras and other sensors with some AI-like grinding on the images could detect all manner of problems and relay it back to the ops folks for a closer look, and/or roll trucks. There are folks at texas a&m who are working on something like this (not sure about the image grinding though). Sure it might be expensive, but it's probably cheaper than undergrounding, or burning down cities and roasting people alive. Mike
All true telco equipment is powered by batteries. Commerical power or generators just recharge the batteries. No switch over when commerical power is lost. Except when the generators(where equiped) switch over to recharge the batteries. Comcast and telcos do not put batteries in all remote powered terminals. I have an Enterprise grade Ethernet over coax connection. The headend it's distributed from doesn't have batteries. If it loses power, doesn't matter if I have power or a generator or ups to take over. This Internet connection goes down. For telcos(when I worked there), they usually had batteries that would last 4 to 8 hrs at remote terminals with powered equipment. And a connection for a splice crew to come out and connect their generator to it for power in case of an extended outage. Back then that was also how most cell phone towers were outfited. I also have some knowledge of the commerical power grid in my local area. It's not unheard of for the Comcast headend to lose power but my office doesn't. Lyle Giese LCR Computer Services, Inc. On 10/14/19 17:38, Michael Thomas wrote:
On 10/14/19 3:06 PM, Sean Donelan wrote:
That is not why people are surprised. When the house doesn't have power, and doesn't have home generator or UPS, (most) people are less surprised their DSL or Cable modem and VOIP doesn't work anymore.
The reasons I saw people angry on twitter was no Comcast service even when they had power at the house (utility, generator, UPS). Their Comcast service died quickly, even when the home had power but the Comcast outside plant didn't seem to have any backup power.
DSL modems also need power at the home, but the telco providers seem to have more backup power in the outside plant or central offices. That meant DSL worked as long as the house had power (or a home generator or UPS).
So it turns out that our local telco/isp does keep dsl running via the same mechanism as they keep pots power backed up (i'm guessing it's a diesel generator at the co, but am not sure). It seems that a lot of the pedestals terminating the local loop these days do the conversion to IP right there with sip/h.248/mgcp/rtp. I'm not sure how they get power to the pedestal, but these were all a home run to the co at one time so it probably wasn't hard to power them from the co. For all i know, that's how they're all powered all the time, with a transfer switch at the co, rather than tapping the local grid next to the pedestal.
Of course this is a lot of conjecture on my part... be glad to be clued in by folks in know.
Mike
On Fri, 11 Oct 2019, Michael Thomas wrote:
So I knew that telcos are required to battery backup pots, but are isp's too? I have a dinky little provider who also provides pots, but i have never been clear whether dsl stays up too in a blackout.
Of course generalizing all service providers isn't fair... From my experience tracking telecommunication during disasters for the last 20-30 years... Let me generalize (ignoring special goverment priority systems). Generally, during natural (and man-made) disasters: Telecommunication service providers historically failed in this order 1. VSAT/DTH/satellite (during weather events) 2. Cable 3. Cellular/wireless 4. Telco/Wireline 5. Broadcast radio/TV (less than 20% over-the-air stations operating) 6. Network backbone systems (inter-city and toll offices) There are too few WISPs for reliable predictions. I'd guess WISPs reliability is similar to cellular/wireless systems. Restoration order is a bit different. Telecommunications network service historically recovers in this order, assuming customer premise isn't damaged: 1. VSAT/DTH/satellite (after weather clears) 2. Network backbone (inter-city and toll offices) 3. Cellular/wireless (COWs and COLTs deployed) 4. Broadcast radio/TV (20% over-the-air stations operating) 5. Telco/Wireline 5. Cable Cable systems tend to be the first to fail, and the last to be restored. Telco systems tend to fail later, but take a long time to be restored. Network backbones can take a while to repair, but generally nothing else works until they are repaired, so they get repaired first or second. Note: During even the worst catastrophes, there is almost always one or two broadcast radio stations still operating. I set 20% radio/TV stations operating as an arbitrary minimum level. Likewise, COWs and COLTs don't provide full cellular service, but do provide minimumal cell services. In the last 10 years, cellular/wireless system resiliance has been improving while telco/wireline system resiliance has been getting noticablly worse. I assume this a flywheel affect as telco companies have been shifting infrastructure investement to wireless networks and away from wireline networks for the last 20 years.
Why you don't have Comcast service during a power outage: Throughout the state, Comcast equipment was knocked offline by PG&E’s power shutdown, Hammel said. The cable company was “only using generators in very discrete and specific cases where there’s a demonstrated need,” such as a request from the Federal Emergency Management Agency, she said.
And this is why the distributed nature of small node’s is detrimental in an extended power outage. There is no practical way to back them up with power for an extended period of time.
On Oct 11, 2019, at 8:44 PM, Sean Donelan <sean@donelan.com> wrote:
Why you don't have Comcast service during a power outage:
Throughout the state, Comcast equipment was knocked offline by PG&E’s power shutdown, Hammel said. The cable company was “only using generators in very discrete and specific cases where there’s a demonstrated need,” such as a request from the Federal Emergency Management Agency, she said.
On 10/11/19 9:43 PM, Matt Hoppes wrote:
And this is why the distributed nature of small node’s is detrimental in an extended power outage.
There is no practical way to back them up with power for an extended period of time.
How distributed is the power on a typical HFC system in practice? I'm sure I'm missing some of them, but having walked out most of a small-ish (~2000 residences) city recently for a FTTx deployment, I think I only saw 2-3 power nodes on Comcast's plant. There were several times as many fiber-coax nodes being line-powered off the coax plant though still surprisingly few (the plant is ancient and hasn't seen a lot of fiber overbuild). That's comparable to how many powered RTUs the LEC had in town and many fewer utility-powered field nodes than would have been present if it had been in AT&T Lightspeed territory. Now, I have no idea what the backup line power is in practice on that Comcast plant. I know that Bright House/Spectrum, in an another area I've supported, has very little backup on many residential-only parts of their plant. I've observed that they have, in practice, maybe 15-30 minutes of hold-up before DOCSIS nodes start dropping. -- Brandon Martin
In article <c4faaa6b-b6d0-0452-470f-cd37076a91fe@monmotha.net> you write:
On 10/11/19 9:43 PM, Matt Hoppes wrote: How distributed is the power on a typical HFC system in practice? I'm sure I'm missing some of them, but having walked out most of a small-ish (~2000 residences) city recently for a FTTx deployment, I think I only saw 2-3 power nodes on Comcast's plant.
I spend too much time looking up at the power lines while walking the dog and around here, Spectrum ex-Roadrunner, on just about every block I see something on the cable plant with an electric meter. I can't tell how many are amplifiers and how many are fiber to coax adapters. None have any evident batteries although I suppose there might be some in the cabinets. My current phone and Internet service is FTTH from the local RLEC. The box they installed here is powered from a 12v UPS and I'm reasonably sure there are no active components between here and the CO since it's only 1/4 mile away, so I'll be interested to see how it does when the power goes out.
On Oct 11, 2019, at 9:43 PM, Matt Hoppes <mattlists@rivervalleyinternet.net> wrote:
And this is why the distributed nature of small node’s is detrimental in an extended power outage.
There is no practical way to back them up with power for an extended period of time.
This is why I’m concerned about a future with micro-cell sites, we will have a few fortresses with cell service but the day-to-day service won’t look anything like the outage service and is likely to fail catastrophically as a result. - jared
Once upon a time, Sean Donelan <sean@donelan.com> said:
It turns out, Comcast's outside plant was woefully unprepared to handle long, i.e. 24 hour, power outages. And even when power is restored to people's homes, Comcast service is often still down.
When I had Comcast in Huntsville, AL, there appeared to be no backup power in their plant. Any power blink and my Internet and TV both dropped (my equipment is on UPS). -- Chris Adams <cma@cmadams.net>
It's very difficult to properly build a resilient infrastructure when those shareholders must get their value! On Fri, Oct 11, 2019 at 7:33 PM Sean Donelan <sean@donelan.com> wrote:
The FCC asked a half-dozen carriers about their network resilience plans last month. Comcast was not one of the service providers askedd about their plans.
The FCC should have looked closer at Comcast in California. While it was expected many people would loose home Internet, voice, video service when their Customer Premise Equipment lost power. The FCC no longer requires battery backup for CPE. That is now a customer responsibility.
It turns out, Comcast's outside plant was woefully unprepared to handle long, i.e. 24 hour, power outages. And even when power is restored to people's homes, Comcast service is often still down.
https://www.ktvu.com/news/still-no-relief-pge-shutoffs-disrupt-comcast-servi...
Maybe the FCC needs to ask about cable system Outside Service Plant network reslience of Comcast. Are other cable OSP resilience similar?
participants (12)
-
Brandon Martin -
Chris Adams -
Jared Mauch -
Jeff Shultz -
John Levine -
Lyle -
Masataka Ohta -
Matt Hoppes -
Michael Thomas -
Sean Donelan -
Ted Hatfield -
Tom Beecher