On 3/29/22 2:24 PM, Aaron C. de Bruyn wrote:

On Tue, Mar 29, 2022 at 12:20 PM Brie <bruns@2mbit.com <mailto:bruns@2mbit.com>> wrote:

Unifi/EdgeSwitch?

Yeah.  Unfortunately.  USW-24-250.

Oh, I have quite a few of them in service. They work great in my experience as long as you don't shove them in hot closets or cabinets and cause them to overheat. Also, firmware has gotten a bit better over the years. I have a lot of background in Ubnt stuff, so I'm a bit biased here.

Yeah, you know when 24v passive POE is turned on because it kills the port on the other end that aren't designed to handle it.  Your router would likely have a dead eth port on it.

I've never tested it with one of my routers, but a tech did accidentally test it on a UniFi AP.  I'm still not sure how he ignored the warning about sending 24 volts down the line, but the WAP didn't like it and decided to refuse to work with anyone ever again. ;)

Yup, that's normal if its a more recent AP that wasn't designed for 24V passive (pretty much anything recent AP wise). I've seen techs destroy ethernet ports on brand new laptops doing that. Very expensive lesson :) -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org

We've routinely seen where lines not even connected to the same circuit in any way (ie an OTA antenna coax line and cat5 POE) cause issues with one another. As much as we would all love to have a perfect line in the sand, there isn't. Don't rule anything out until the issue is resolved. As someone that sees this in the field and watches people simply hate on someone because there's a frustrating situation, it's worth taking a breath before too upset. On Tue, Mar 29, 2022 at 2:50 PM Joe Greco <jgreco@ns.sol.net> wrote:

...

I just got off the phone with a Comcast tech, and wanted to double-check my sanity.

Somehow in the last 6 months I've managed to reach the exact same rep twice when dealing with an outage or a degraded service event.

I asked him to remotely reboot the modem because there was high packet loss.

Both times I've talked with him, he noted the high packet loss, started to reboot the modem, and then asked me point-blank if we had any PoE switches on our network.

When I said "yes", he said I needed to disable PoE because it messes with the Comcast modems and he can see "buildups" in his graphs that show

...

is "leaking" to the Comcast modem every 24 hours.

For reference, our setup is:

Internal Network ?????? PoE Switch ?????? My Router (FreeBSD Box) ?????? Comcast Modem

I told him the Comcast modem isn't plugged into the PoE Switch, it's plugged into My Router (FreeBSD box) and My Router does not negotiate PoE+ and the switch shows PoE isn't being send to My Router's LAN port. While the switch is capable of outputting old-school 24v PoE, it must be specifically turned on for a port, and it's not enabled or used anywhere on the networks I manage.

When provided with that information, the Comcast tech still insisted that the switch was sending PoE to My Router and it was "leaking through" to

...

Comcast modem and that's why every 4-6 weeks the Comcast modem needs to be reset. The tech insisted that switches that *are* PoE-capable *always* send PoE even if the device doesn't request it or negotiate it. Attempts to explain the difference between the old 24-volt PoE and PoE+/++ were met with arguing that he's been in the industry for decades and I don't know what I'm talking about...and that all my problems would go away if I just disabled PoE everywhere on the switch.

Again, I double-checked the port and said "It's not sending PoE to my router, but even if I were, I highly doubt PoE would leak through a PCI card to the opposite side of the chassis to the on-board NIC and out to your modem".

He insisted it happened "all the time" and he had previously fried equipment by plugging it into a PoE switch. He insisted that he's also handled quite a few calls relating to this magic PoE problem over the years and Comcast has internal tools that show graphs of how much PoE power "builds up" inside their modems and he "can see a buildup in my router

On Tue, Mar 29, 2022 at 11:21:28AM -0700, Aaron de Bruyn via NANOG wrote: power the that

...

resets every 24 hours".

I didn't have the heart to tell him that I manage about 40 networks that have Comcast connections...and they *all* have identical FreeBSD boxes acting as their router, and they are *all* using the exact same PoE switches at every location with all ports set to PoE+...and we only have degraded service or outages after ~30 days at ~3 locations.

Slightly off-topic, but if I call Comcast about outages or degraded service and any *other* tech but this guy answers, they all say "you need to unplug your Comcast modem and plug it back in once every 3-4 weeks" and they act like it's normal to reboot the modems every few weeks. In fact, last week I wanted Comcast to check on a modem setting at one location and they said the modem had been up for over 127 days and it should be rebooted. I said "it's up and working fine, why would I reboot it?".

Anyways, am I insane for thinking the tech was flat-out wrong? I mean...occasionally some really bizarre stuff happens in IT...but this seems extremely far-fetched and contrary to everything I know about the PoE standard.

That's ridiculous, as you already know.

Their crappy equipment needing rebooting every few weeks, not ridiculous.

Their purchasing gear from incompetent vendors who cannot be standards compliant for PoE PD negotiation, tragically plausible.

Ethernet electronic differential signalling not being handled properly with respect to grounding or other issues, not unheard-of.

Ghosts of PoE floating around a network through other devices, causing weird problems on the far side of properly installed and standards compliant gear, ah, super-unlikely, I'll go so far as to say nah, but in this audience I am positive someone has a counterexample SOMEWHERE, cuz, you know, the world's a strange place and there's broken stuff out there.

He's got graphs showing it every 24 hours? Liar, liar, pants on fire, lazy SOB is looking for an excuse to clear you off the line. Where the heck does this "24 hour" cycle even come from? What SNMP OID is there for "ghostly PoE build-up"? What crontab is there that would clear out such buildups in the router's daily run? What capacitor would store up juice for precisely 24 hours? What's the mechanism here? CURIOUS MINDS WANT TO KNOW!

Been doing PoE everywhere for years and this is the stupidest thing I've heard this year so far in the networking category.

Next time, play dumb. People who can, do. People who can't, tech support. Worth remembering.

... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "The strain of anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that 'my ignorance is just as good as your knowledge.'"-Asimov

There's a certain manufacturer of TDD radio where the CPU clock is at the same frequency as what Verizon's enodeB will transmit. Even at miles away, it can and will cause PIM issues. Again, don't rule it out. Maybe he's just looking for a simple answer that 99% of callers will accept and it makes them happy. When a customer of mine tells me they think it's something and I know it's off, I just let them believe in their statement. There's no reason to go after this tech and insult him, all that's doing is making everyone miserable. On Tue, Mar 29, 2022 at 3:26 PM Joe Greco <jgreco@ns.sol.net> wrote:

On Tue, Mar 29, 2022 at 03:07:47PM -0400, Josh Luthman wrote:

...

We've routinely seen where lines not even connected to the same circuit in any way (ie an OTA antenna coax line and cat5 POE) cause issues with one another. As much as we would all love to have a perfect line in the sand, there isn't. Don't rule anything out until the issue is resolved.

As someone that sees this in the field and watches people simply hate on someone because there's a frustrating situation, it's worth taking a breath before too upset.

You can run cable lines next to A/C wiring and get problems too. Or ethernet lines next to A/C wiring. That does not justify wild claims about PoE such as what this tech was making, and until someone shows me a graph of "PoE buildups" observable via SNMP or whatever the cable company is using to graph trends, it seems pretty clear that this is a bogus answer.

There's a lot of difference between "we observed this very specific kind of interference related to PoE in a particular circumstance" and the crazy generalizations being made by the tech. Asking to please make sure your switch is grounded properly? That'd be good. Asking for PoE to be disabled on the port? Yeah fine. Suggesting separation of cables? Sure. Checking for proper grounding of the ground block (on the cable inlet)? Sure. There's room for things to happen.

I'm all for investigating with an open mind, but I draw the line at crazy.

Given that so much of the world works on PoE, it seems like the other potential resolution would be to note that there's an implication here by the tech that Comcast's hardware is standards noncompliant and ask them what they plan to replace their cheap CPE with.

... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "The strain of anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that 'my ignorance is just as good as your knowledge.'"-Asimov

On 3/29/22 2:19 PM, Aaron de Bruyn via NANOG wrote:

When you hear hoof beats, look for horses, not zebras.

As someone with a family member that is a zebra from a health perspective, I follow up with: Yes, /look/ for horses when you hear hoof beats, but -- and this is the important thing -- do not rule out zebras. We've run into too many people who hear hoof beats, assume horses, and proceed as if zebras are absolutely not a possibility. -- Grant. . . . unix || die

On 3/30/22 09:53, Grant Taylor via NANOG wrote:

On 3/29/22 2:19 PM, Aaron de Bruyn via NANOG wrote:

...

When you hear hoof beats, look for horses, not zebras.

As someone with a family member that is a zebra from a health perspective, I follow up with:

Yes, /look/ for horses when you hear hoof beats, but -- and this is the important thing -- do not rule out zebras.

We've run into too many people who hear hoof beats, assume horses, and proceed as if zebras are absolutely not a possibility.

https://www.nytimes.com/2021/09/30/us/zebras-escaped-maryland.html -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV

On Mar 29, 2022, at 5:46 PM, Joe Greco <jgreco@ns.sol.net> wrote:

So if you want the $100 test to eliminate PoE electrical effects, get a pair of media converters and run fiber between them. Put the CPE on the far end. Optimize as appropriate if you have SFP-capable switches.

... JG — Bingo ^^^

As a few others have said, I’d be more inclined to feel like this was a real thing if the tech had mentioned checking grounding, etc and not just said all the PoE has to go away for the modem to function correctly. If it is truly a power issue there are a bunch of ways to isolate/troubleshoot/fix it. Thanks, Matt Matt Ertle Manager - Network Operations Eastern Shore of Virginia Broadband Authority (w) 757 414.0304 (f) 757 656.7066 mertle@esvba.com

On 2022-03-29 5:46 p.m., Joe Greco wrote:

On Tue, Mar 29, 2022 at 03:42:54PM -0400, Josh Luthman wrote:

...

There's a certain manufacturer of TDD radio where the CPU clock is at the same frequency as what Verizon's enodeB will transmit. Even at miles away, it can and will cause PIM issues. Again, don't rule it out. I'm not ruling anything out, but on the flip side, here in this group of professional networkers, you'd think lots of people would have piped up by now with "me too"'s if PoE ghosts killing cable CPE on a 24 hour cycle were a common thing.

As a small DOCSIS operator, I suppose my sample set isn't large enough to be significant but I can tell you PoE has never been an issue with our modems. We've had a significantly higher failure rate due to things such as customers driving nails through the modem in an attempt to mount them to walls, and concerned citizens shooting birdshot into our overhead distribution in an attempt to curb the rodent population. K

On 3/29/22 3:46 PM, Joe Greco wrote:

So if you want the $100 test to eliminate PoE electrical effects, get a pair of media converters and run fiber between them. Put the CPE on the far end. Optimize as appropriate if you have SFP-capable switches.

I second this. I'd also be willing to get approval for the expense to procure and install the optical isolation equipment in the spirit of good client relations with many of the clients that I've had in the past. Me: $BOSS, I want to spend $200 as a troubleshooting diagnostic effort to test to see if the problem that $CLIENT is having is related. I think it's not, but I want to rule it out. Boss: Why haven't we done this yet? Please make it happen. -- Grant. . . . unix || die

On 3/29/2022 3:24 PM, Joe Greco wrote:

He's got graphs showing it every 24 hours? Liar, liar, pants on fire, lazy SOB is looking for an excuse to clear you off the line. Where the heck does this "24 hour" cycle even come from? What SNMP OID is there for "ghostly PoE build-up"? What crontab is there that would clear out such buildups in the router's daily run? What capacitor would store up juice for precisely 24 hours? What's the mechanism here? CURIOUS MINDS WANT TO KNOW!

Taken at face value, I assume the tech would be looking at historical signal graphs (we keep them for cable networks for each CM) that record stats like FEC, SNR, and signal strength. For aerial runs it's common to see some change throughout the day due to warming and cooling. These look like waves with peaks and valleys around 4PM/4AM and generally affect all customers in a service area equally. Sometimes there will be a device at a customer premise that causes interference with a CM, something like a motor or tool. These could absolutely be on a 24hr cycle (think of a programmable thermostat kicking on the blower fan in your HVAC at the same time every day). As Joe said, there's no SNMP MIB for PoE buildup. There are well documented MIBs for DOCSIS to cover standard signal level, quality, or similar. The cause of that signal strength or quality can be myriad. This Comcast tech has likely climbed the ladder of inference several steps too far.

Generally anything inside the customer premise (including wiring) is the customer’s responsibility. If your coax runs across a fluorescent light fixture, that’s on you. If your coax is RG59, it’s on you to replace with RG6 quad shield. Maybe the cable operator will work with you, maybe not. I’ve had some techs replace splitters or make custom length user cables. And yes, the RF stats I mentioned are all on the HFC interface of the cable modem (or the CMTS). Customer facing Ethernet stats may not even be tracked by Comcast. --Blake

On Mar 29, 2022, at 5:29 PM, Aaron de Bruyn <aaron@heyaaron.com> wrote:

 Thanks Blake,

As I understand it all that stuff is on the "cable provider" side of the CPE and (within reason) it's up to the provider to deal with the signals arriving on the cable side of the modem. i.e. if it was a blower or something in our suite that was causing RF interference, the provider might work with us to move the modem or the cable run.

-A

On Tue Mar 29, 2022, 09:59 PM GMT, Blake Hudson wrote:

On 3/29/2022 3:24 PM, Joe Greco wrote: He's got graphs showing it every 24 hours? Liar, liar, pants on fire, lazy SOB is looking for an excuse to clear you off the line. Where the heck does this "24 hour" cycle even come from? What SNMP OID is there for "ghostly PoE build-up"? What crontab is there that would clear out such buildups in the router's daily run? What capacitor would store up juice for precisely 24 hours? What's the mechanism here? CURIOUS MINDS WANT TO KNOW!

Taken at face value, I assume the tech would be looking at historical signal graphs (we keep them for cable networks for each CM) that record stats like FEC, SNR, and signal strength. For aerial runs it's common to see some change throughout the day due to warming and cooling. These look like waves with peaks and valleys around 4PM/4AM and generally affect all customers in a service area equally. Sometimes there will be a device at a customer premise that causes interference with a CM, something like a motor or tool. These could absolutely be on a 24hr cycle (think of a programmable thermostat kicking on the blower fan in your HVAC at the same time every day).

As Joe said, there's no SNMP MIB for PoE buildup. There are well documented MIBs for DOCSIS to cover standard signal level, quality, or similar. The cause of that signal strength or quality can be myriad. This Comcast tech has likely climbed the ladder of inference several steps too far.

On 3/30/22 10:52, Livingood, Jason via NANOG wrote:

Could well be from noise ingress - lots of work goes into finding & fixing ingress issues. Hard to say unless we look in detail at the connection in question and the neighborhood node.

Packet loss absolutely could be noise ingress. That's measurable in terms of s/n ratio and it's on the outside plant HFC side. But noise doesn't "build up", and PoE is DC, and PoE is on the LAN side, separated from Comcast's equipment by a router. Level 1 Comcast guy is simply wrong. -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV

Hi, Colleagues: 0)    I would like to share a personal experience of a different setting to offer an angle for looking into this puzzling topic. 1)    During my graduate study, I was doing microwave experiments in the laboratory. On a six foot bench, I had a series (maybe a dozen or so) of waveguide components (made of metal, either solid copper or silver plated) connected together as the test bed. With a half dozen or so equipment attached to them at various points to serve as the energy source or sink as well as detection instruments. The setup exhibited randomly and unpredictable behaviors. After awhile, my advisor brought up the topic of "Ground Loops" that could introduce the interference in mysterious manners. This is a phenomenon whereby minuscule electric current flowing along metallic parts (even though they may appear to have no "resistance" in between) that are connected to the system common potential reference (the "ground") via slightly different paths. These could be very small resistance path between two points or high resistance leakage source. The resultant electric potentials among the subsystems of interest could be significant enough to affect the outputs of sensitive instruments. 2)    After much cut-&-try, including plugging all instruments into the same electric extension strip with no avail, I finally floated the AC power cords of all instruments (using three-to-two prong adapters) but kept only the most sensitive node in the system connected to the AC power source "ground". Although this was against the electric safety code, I finally got consistent results. With clear record of the configuration, I even could take the waveguide setup apart and then reassembled days later to repeat the same results. Years later, I applied the same philosophy to a smart-meter PCB design which got a better precision than the chip manufacturer's demon board could. 3)    So, it is possible that the site with the reported "PoE induced" issues may be somehow experiencing the above related phenomena. This kind of situations are almost impossible to duplicate at another site. It has to be diagnosed with pains-taking detective efforts, such as inserting isolation subsystems suggested by one colleague. Since this phenomenon takes a month or so to show up, discipline and patience are the virtue. 4)    On the other hand, a product that can build up certain "memory" of the disturbances like the above and to the point of requiring periodical power cycling to flush clear the issue is definitely a sign of defect design, based on my old school training. Regards, Abe (2022-03-31 10:40) On 2022-03-30 15:57, Joe Greco wrote:

...

...

Their crappy equipment needing rebooting every few weeks, not ridiculous. Their purchasing gear from incompetent vendors who cannot be standards compliant for PoE PD negotiation, tragically plausible.

Many customers buy their own cable modem. You can lease an Xfinity device as well and those function pretty nicely these days but YMMV. But typically a device reboot is a way to quickly solve a few different kinds of problems, which is why techs will often recommend it as an initial step (you can generally assume that there's data behind what occurs when any one of tens of thousands of support reps suggesting something to a customer - support at scale is data-driven). No one's doubting all of that -- support is best when data-driven, scale or otherwise. But that's actually the issue here. There's no data that I know of to suggest widespread PoE ghost current buildups, and, given

On Wed, Mar 30, 2022 at 05:52:06PM +0000, Livingood, Jason wrote: the audience here, no one else has popped up with a clear "me too".

PoE is typically negotiated by modern switches, 24v Unifi special jobbies aside, so it's all DC on cables that are already handling differential signalling.

...

...

He's got graphs showing it every 24 hours? Liar, liar, pants on fire,

lazy SOB is looking for an excuse to clear you off the line.

Could well be from noise ingress - lots of work goes into finding & fixing ingress issues. Hard to say unless we look in detail at the connection in question and the neighborhood node. No doubt. There's huge amounts of room for problems to be introduced into last mile networks. But, again, this isn't about general problems. This is about a tech claiming it's due to PoE, and that he's seen it often before.

I certainly have a lot of sympathy for cable techs, but that doesn't mean I want to swallow any random garbage they want to blame issues on. Please just tell me it's the chipmunks getting feisty and nibbling on the copper if you want to feed me a line...

... JG

-- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus