Help with removing DNS shinkhole FP from Charter/Spectrum
Looking for some help/advice. Spectrum is sinkholing my company's domain, validin[.]com, to 127.0.0.54. The sinkhole responses come from their recursive DNS servers, 209.18.47.61 and 209.18.47.62, which are defaults for and in use by many of their customers and are only reachable from within the Spectrum network. I've had 4 people over the last week (think: customers, prospects, etc) who use Charter/Spectrum tell me that they have difficulty accessing my website as a result of this sinkhole behavior. This behavior is causing reputational harm to my company. I've personally confirmed this behavior from the Spectrum network (I am also a customer) using dig to test their DNS servers: ``` $ dig +short @209.18.47.61 validin.com 127.0.0.54 $ dig +short @209.18.47.62 validin.com 127.0.0.54 ``` Using Cloudflare/Google/etc works correctly: ``` $ dig +short @1.1.1.1 validin.com 137.184.54.107 157.245.112.183 $ dig +short @8.8.8.8 validin.com 157.245.112.183 137.184.54.107 ``` I suspect my domain was blocklisted last year when a threat researcher included my domain name in a blog post about a threat they were investigating and cited my company as the source for their data. Someone scraped that post, and my company's domain was accidentally added to two Alient Vault OTX pulses and at least one collection on Virus Total. I removed the domain via false positive reporting from everything I could. However, it appears that being added to Spectrum's DNS sinkhole list is effectively permanent and there's no clear path for false positive remediation. I've tried the official Spectrum support lines for months to no avail, and recently tried reaching out on Twitter, but have had no success there either. I'm clearly not able to find the right people through these routes, as none of the people I reach understand the difference between a DNS sinkhole and an IP block list and don't appear to be aware that DNS blocklisting is a separate behavior from their opt-in content filtering via Security Shield. So, if someone could please help me find the team or individual responsible for Spectrum's DNS sinkhole behavior, I would be exceptionally grateful. :-) As I mentioned, this is causing reputation harm, so switching my own DNS servers is not sufficient. People who need to reach me, can't. So, I would appreciate any other help or advice you have, Kenneth
On Sun, Apr 21, 2024 at 6:21 PM Validin Axon <axon@validin.com> wrote:
Looking for some help/advice. Spectrum is sinkholing my company's domain, validin[.]com, to 127.0.0.54.
Howdy, If you can't reach a technical POC, use the legal one. Your lawyer can find the appropriate recipient and write a cease-and-desist letter for you. After that, it's -their- lawyer's problem to track down the correct technical people. Incidentally, for folks who choose to interdict DNS: whatever your reasons, pointing the DNS to a loopback IP is bad practice. Really bad practice. Minimum good practice points it to a web site you control which provides enough information to get delisted. And provides you with a test point where you can collect information about what you've caused to be interdicted. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
It appears that William Herrin <bill@herrin.us> said:
On Sun, Apr 21, 2024 at 6:21 PM Validin Axon <axon@validin.com> wrote:
Looking for some help/advice. Spectrum is sinkholing my company's domain, validin[.]com, to 127.0.0.54.
Howdy,
If you can't reach a technical POC, use the legal one. Your lawyer can find the appropriate recipient and write a cease-and-desist letter for you. After that, it's -their- lawyer's problem to track down the correct technical people.
No, that is terrible advice. In the immortal acronym of Laura Atkins, TWSD. The only response to a letter like that is "we run our network to serve our customers and manage it the way we think is best" and you know what, they're right. It is absolutely legal to block traffic you think is malicious, even if you are wrong, and there is case law. Having said that, I suspect the least bad alternative if you can't find an out of band contact is to get some of the Spectrum customers who can't reach you to complain. They're customers, you aren't. R's, John
On Mon, Apr 22, 2024 at 4:00 PM John Levine <johnl@iecc.com> wrote:
It appears that William Herrin <bill@herrin.us> said:
If you can't reach a technical POC, use the legal one. Your lawyer can
The only response to a letter like that is "we run our network to serve our customers and manage it the way we think is best" and you know what, they're right.
Hi John, Respectfully, you're mistaken. Look up "tortious interference." Operators have considerable legal leeway to block traffic for cause, or even by mistake if corrected upon notification, but a lawyer who blows off a cease-and-desist letter without investigating it with the tech staff has committed malpractice. The lawyer doesn't want to commit malpractice. You write the lawyer via certified mail, he's going to talk to the tech staff and you're going to get a response. At that point, you have an open communication pathway to get things fixed. Which was the problem to be solved.
Having said that, I suspect the least bad alternative if you can't find an out of band contact is to get some of the Spectrum customers who can't reach you to complain. They're customers, you aren't.
My results going through the support front-door at large companies for oddball problems have been less than stellar. Has your experience truly been different? Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
“We checked the website you are trying to access for malicious and spear-phishing content and found it likely to be unsafe.” perhaps charter thinks there's a reason to not permit folks to access a possibly dangerous site? (it's also possible it just got cough up amongst some other stuff in the hosting provider's space, nothing jumps out in passive-dns lokoups.) On Mon, Apr 22, 2024 at 7:39 PM William Herrin <bill@herrin.us> wrote:
On Mon, Apr 22, 2024 at 4:00 PM John Levine <johnl@iecc.com> wrote:
It appears that William Herrin <bill@herrin.us> said:
If you can't reach a technical POC, use the legal one. Your lawyer can
The only response to a letter like that is "we run our network to serve our customers and manage it the way we think is best" and you know what, they're right.
Hi John,
Respectfully, you're mistaken. Look up "tortious interference."
Operators have considerable legal leeway to block traffic for cause, or even by mistake if corrected upon notification, but a lawyer who blows off a cease-and-desist letter without investigating it with the tech staff has committed malpractice. The lawyer doesn't want to commit malpractice. You write the lawyer via certified mail, he's going to talk to the tech staff and you're going to get a response. At that point, you have an open communication pathway to get things fixed. Which was the problem to be solved.
Having said that, I suspect the least bad alternative if you can't find an out of band contact is to get some of the Spectrum customers who can't reach you to complain. They're customers, you aren't.
My results going through the support front-door at large companies for oddball problems have been less than stellar. Has your experience truly been different?
Regards, Bill Herrin
-- William Herrin bill@herrin.us https://bill.herrin.us/
Hi Bill, I'm not sure where you saw that message, but I got this message via email after I submitted an unblock request with Spectrum Shield:
We have reviewed your request to unblock validin.com. This site was not found to be blocked by Spectrum Shield and should be accessible from your browser.
Thank you,
Spectrum
My company's domain got caught up in some lazy copy/pasting from this blog post last year that cited my company as a source for the data. Someone copy/pasted the whole page, which included my company's domain name, and that made it to a few AV OTX pulses and VT collections: https://gi7w0rm.medium.com/uncovering-ddgroup-a-long-time-threat-actor-d3b30... I've cleaned up everything I could from that botched blocklist aggregation. However, there's no correction process for Spectrum's DNS sinkhole, and I'm not even sure that's how our domain got mixed up there. The support staff I've spoken with have denied the existence of DNS sinkholing at Spectrum, and demonstrated they lack the basic technical sophistication needed to understand the concept. They've each ultimately told me that each affected customer would need to reach out to the Spectrum customer service, which would then help that customer change their DNS settings to another DNS provider. Of course, the last thing I'd want to do with a potential customer is ask them to go through that painful process. I also have no idea how many potential users or customers can't reach me and simply give up without letting me know. Lastly, I AM a Spectrum customer. My home internet service is Spectrum. If it weren't for that, I'd be truly SOL because support would just ignore me. But, they they claim the issue is resolved from their perspective because I can simply change my DNS settings. But back to the topic: someone mentioned to me that Spectrum may not be the direct providers for the DNS services they provide to their customers. If anyone knows anything about how I might discover and reach out to the people responsible, please let me know. :-) Regards, Kenneth On Mon, Apr 22, 2024 at 8:07 PM Christopher Morrow <morrowc.lists@gmail.com> wrote:
“We checked the website you are trying to access for malicious and spear-phishing content and found it likely to be unsafe.”
perhaps charter thinks there's a reason to not permit folks to access a possibly dangerous site? (it's also possible it just got cough up amongst some other stuff in the hosting provider's space, nothing jumps out in passive-dns lokoups.)
On Mon, Apr 22, 2024 at 7:39 PM William Herrin <bill@herrin.us> wrote:
On Mon, Apr 22, 2024 at 4:00 PM John Levine <johnl@iecc.com> wrote:
It appears that William Herrin <bill@herrin.us> said:
If you can't reach a technical POC, use the legal one. Your lawyer can
The only response to a letter like that is "we run our network to serve our customers and manage it the way we think is best" and you know what, they're right.
Hi John,
Respectfully, you're mistaken. Look up "tortious interference."
Operators have considerable legal leeway to block traffic for cause, or even by mistake if corrected upon notification, but a lawyer who blows off a cease-and-desist letter without investigating it with the tech staff has committed malpractice. The lawyer doesn't want to commit malpractice. You write the lawyer via certified mail, he's going to talk to the tech staff and you're going to get a response. At that point, you have an open communication pathway to get things fixed. Which was the problem to be solved.
Having said that, I suspect the least bad alternative if you can't find an out of band contact is to get some of the Spectrum customers who can't reach you to complain. They're customers, you aren't.
My results going through the support front-door at large companies for oddball problems have been less than stellar. Has your experience truly been different?
Regards, Bill Herrin
-- William Herrin bill@herrin.us https://bill.herrin.us/
On Mon, Apr 22, 2024 at 5:54 PM Validin Axon <axon@validin.com> wrote:
Hi Bill,
I'm not sure where you saw that message, but I got this message via email after I submitted an unblock request with Spectrum Shield:
Howdy, That was Christopher, not me. But you should check the talos link I sent you privately. Also https://ipcheck.proofpoint.com/. Whatever they're detecting, it didn't happen last year. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
I'm not sure where you saw that message, but I got this message via email after I submitted an unblock request with Spectrum Shield:
We have reviewed your request to unblock validin.com. This site was not found to be blocked by Spectrum Shield and should be accessible from your browser.
Sigh.
I've cleaned up everything I could from that botched blocklist aggregation. However, there's no correction process for Spectrum's DNS sinkhole, and I'm not even sure that's how our domain got mixed up there. The support staff I've spoken with have denied the existence of DNS sinkholing at Spectrum, and demonstrated they lack the basic technical sophistication needed to understand the concept.
Yeah, that's the problem. And given stuff like this link below, I wouldn't expect their legal department to be any better. Clearly there is someone somewhere who is competent because their network mostly works, but damned if I know how to find them. https://www.theverge.com/2022/7/29/23282522/charter-spectrum-customer-murder... R's, John
However, there's no correction process for Spectrum's DNS sinkhole But back to the topic: someone mentioned to me that Spectrum may not be the direct providers for the DNS services they provide to their customers. If anyone knows anything about how I might discover and reach out to the people responsible, please let me know.
I suspect what’s happened is an incorrect assumption that DNS is even the issue here. Because you mentioned Spectrum Shield, I suspect it is not. Spectrum Shield (https://www.spectrum.com/resources/internet-wifi/benefits-of-spectrum-securi...) is a customer-managed security protection service built into their gateways (I assume you can turn it off). The malware and content detection engine behind that is very likely run by CujoAI (https://cujo.com/) and it does not use DNS query/response exchanges as the control mechanism (in part to counter-act DNS-changing malware or malware using its own DoH channel for example). You should contact Charter/Spectrum to have them investigate what their system might be blocking this content. Comcast (where I work) runs a similar system (https://www.xfinity.com/support/articles/using-xfinity-xfi-advanced-security) and maintains a site to report these sorts of issues (https://www.xfinity.com/support/articles/report-blocked-website). Jason
Hi Jason,
I suspect what’s happened is an incorrect assumption that DNS is even the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
I appreciate the response and links. However, I've been told repeatedly by Spectrum that they're not blocking with Spectrum Shield. Despite these assurances, I've filled out a removal request through their published removal process several times, and the response I received stated that we're not being blocked. This check agrees with that: https://www.spectrum.net/support/forms/verify_url_security "Security Shield Is Not Blocking This Site The URL provided is not being blocked by Spectrum Security Shield The URL you entered should be accessible." Further, checking Spectrum DNS servers on the Spectrum network show that my company's main domain and all subdomains resolve to 127.0.0.54. So, if CujoAI/Spectrum Shield are not using DNS query responses to control access, then it's not CujoAI/Spectrum Shield that is responsible for the incorrect DNS response. Using a different recursive resolve, I can resolve our domains just fine. I can also resolve other domains that point to the same IPs as the sinkholed domain just fine. However, many people use the Spectrum default DNS servers and cannot access my website because of this.
You should contact Charter/Spectrum to have them investigate what their system might be blocking this content.
I have tried, for months, including spending many hours on chat and phone support, to reach someone within Spectrum support who is capable of both understanding and directing me to someone who can fix the problem, but it hasn't happened yet. I've asked to talk to someone on the DNS team and was given a flat "No." I've posted here hoping that someone in the ISP-connected world knows SOMEONE at Spectrum, Akamai, or whichever company is actually responsible for the Spectrum DNS servers who can provide a remediation path. Regards, Kenneth On Tue, Apr 23, 2024 at 12:59 PM 'Livingood, Jason' via axon < axon@validin.com> wrote:
However, there's no correction process for Spectrum's DNS sinkhole
But back to the topic: someone mentioned to me that Spectrum may not be the direct providers for the DNS services they provide to their customers. If anyone knows anything about how I might discover and reach out to the people responsible, please let me know.
I suspect what’s happened is an incorrect assumption that DNS is even the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
Spectrum Shield ( https://www.spectrum.com/resources/internet-wifi/benefits-of-spectrum-securi...) is a customer-managed security protection service built into their gateways (I assume you can turn it off). The malware and content detection engine behind that is very likely run by CujoAI (https://cujo.com/) and it does not use DNS query/response exchanges as the control mechanism (in part to counter-act DNS-changing malware or malware using its own DoH channel for example).
You should contact Charter/Spectrum to have them investigate what their system might be blocking this content.
Comcast (where I work) runs a similar system ( https://www.xfinity.com/support/articles/using-xfinity-xfi-advanced-security) and maintains a site to report these sorts of issues ( https://www.xfinity.com/support/articles/report-blocked-website).
Jason
Validin, made an interesting observation on this. I am also a Spectrum residential customer, none of their equipment, run my own DNS server (pihole). My DHCP Assigned DNS servers are 2001:1998:f00:1::1 2001:1998:f00:2::1 bash-3.2$ dig -x 2001:1998:f00:1::1 +short dns-cac-lb-01.rr.com. bash-3.2$ dig -x 2001:1998:f00:2::1 +short dns-cac-lb-02.rr.com. bash-3.2$ bash-3.2$ dig dns-cac-lb-01.rr.com +short 209.18.47.61 bash-3.2$ dig dns-cac-lb-02.rr.com +short 209.18.47.62 bash-3.2$ bash-3.2$ dig @209.18.47.61 validin.com +short 157.245.112.183 137.184.54.107 bash-3.2$ dig @209.18.47.62 validin.com +short 157.245.112.183 137.184.54.107 bash-3.2$ bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short 127.0.0.54 bash-3.2$ bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short 127.0.0.54 bash-3.2$ Same servers on V4 were returning correct info, but on V6 were not. However, a few minutes later : bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short 157.245.112.183 137.184.54.107 bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short 157.245.112.183 137.184.54.107 bash-3.2$ Deltas : bash-3.2$ dig @2001:1998:f00:1::1 validin.com ; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42329 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;validin.com. IN A ;; ANSWER SECTION: validin.com. 60 IN A 127.0.0.54 ;; Query time: 37 msec ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1) ;; WHEN: Tue Apr 23 13:50:03 EDT 2024 ;; MSG SIZE rcvd: 45 bash-3.2$ bash-3.2$ dig @2001:1998:f00:1::1 validin.com ; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9667 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;validin.com. IN A ;; ANSWER SECTION: validin.com. 600 IN A 157.245.112.183 validin.com. 600 IN A 137.184.54.107 ;; Query time: 157 msec ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1) ;; WHEN: Tue Apr 23 14:19:20 EDT 2024 ;; MSG SIZE rcvd: 72 bash-3.2$ Seems like quite possibly they are intermittently caching bunk data from something. On Tue, Apr 23, 2024 at 1:39 PM Validin Axon <axon@validin.com> wrote:
Hi Jason,
I suspect what’s happened is an incorrect assumption that DNS is even the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
I appreciate the response and links. However, I've been told repeatedly by Spectrum that they're not blocking with Spectrum Shield. Despite these assurances, I've filled out a removal request through their published removal process several times, and the response I received stated that we're not being blocked. This check agrees with that: https://www.spectrum.net/support/forms/verify_url_security
"Security Shield Is Not Blocking This Site The URL provided is not being blocked by Spectrum Security Shield The URL you entered should be accessible."
Further, checking Spectrum DNS servers on the Spectrum network show that my company's main domain and all subdomains resolve to 127.0.0.54. So, if CujoAI/Spectrum Shield are not using DNS query responses to control access, then it's not CujoAI/Spectrum Shield that is responsible for the incorrect DNS response. Using a different recursive resolve, I can resolve our domains just fine. I can also resolve other domains that point to the same IPs as the sinkholed domain just fine. However, many people use the Spectrum default DNS servers and cannot access my website because of this.
You should contact Charter/Spectrum to have them investigate what their system might be blocking this content.
I have tried, for months, including spending many hours on chat and phone support, to reach someone within Spectrum support who is capable of both understanding and directing me to someone who can fix the problem, but it hasn't happened yet. I've asked to talk to someone on the DNS team and was given a flat "No." I've posted here hoping that someone in the ISP-connected world knows SOMEONE at Spectrum, Akamai, or whichever company is actually responsible for the Spectrum DNS servers who can provide a remediation path.
Regards,
Kenneth
On Tue, Apr 23, 2024 at 12:59 PM 'Livingood, Jason' via axon < axon@validin.com> wrote:
However, there's no correction process for Spectrum's DNS sinkhole
But back to the topic: someone mentioned to me that Spectrum may not be the direct providers for the DNS services they provide to their customers. If anyone knows anything about how I might discover and reach out to the people responsible, please let me know.
I suspect what’s happened is an incorrect assumption that DNS is even the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
Spectrum Shield ( https://www.spectrum.com/resources/internet-wifi/benefits-of-spectrum-securi...) is a customer-managed security protection service built into their gateways (I assume you can turn it off). The malware and content detection engine behind that is very likely run by CujoAI (https://cujo.com/) and it does not use DNS query/response exchanges as the control mechanism (in part to counter-act DNS-changing malware or malware using its own DoH channel for example).
You should contact Charter/Spectrum to have them investigate what their system might be blocking this content.
Comcast (where I work) runs a similar system ( https://www.xfinity.com/support/articles/using-xfinity-xfi-advanced-security) and maintains a site to report these sorts of issues ( https://www.xfinity.com/support/articles/report-blocked-website).
Jason
Tom, Thank you for this! It is very interesting that the behavior is intermittent. A friend of mine who tested it this weekend saw correct answers on IPv6 and incorrect answers on IPv4. Kenneth On Tue, Apr 23, 2024 at 2:56 PM Tom Beecher <beecher@beecher.cc> wrote:
Validin, made an interesting observation on this. I am also a Spectrum residential customer, none of their equipment, run my own DNS server (pihole).
My DHCP Assigned DNS servers are
2001:1998:f00:1::1 2001:1998:f00:2::1
bash-3.2$ dig -x 2001:1998:f00:1::1 +short dns-cac-lb-01.rr.com. bash-3.2$ dig -x 2001:1998:f00:2::1 +short dns-cac-lb-02.rr.com. bash-3.2$
bash-3.2$ dig dns-cac-lb-01.rr.com +short 209.18.47.61 bash-3.2$ dig dns-cac-lb-02.rr.com +short 209.18.47.62 bash-3.2$
bash-3.2$ dig @209.18.47.61 validin.com +short 157.245.112.183 137.184.54.107 bash-3.2$ dig @209.18.47.62 validin.com +short 157.245.112.183 137.184.54.107 bash-3.2$
bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short 127.0.0.54 bash-3.2$
bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short 127.0.0.54 bash-3.2$
Same servers on V4 were returning correct info, but on V6 were not.
However, a few minutes later :
bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short 157.245.112.183 137.184.54.107 bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short 157.245.112.183 137.184.54.107 bash-3.2$
Deltas :
bash-3.2$ dig @2001:1998:f00:1::1 validin.com
; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42329 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;validin.com. IN A
;; ANSWER SECTION: validin.com. 60 IN A 127.0.0.54
;; Query time: 37 msec ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1) ;; WHEN: Tue Apr 23 13:50:03 EDT 2024 ;; MSG SIZE rcvd: 45
bash-3.2$
bash-3.2$ dig @2001:1998:f00:1::1 validin.com
; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9667 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;validin.com. IN A
;; ANSWER SECTION: validin.com. 600 IN A 157.245.112.183 validin.com. 600 IN A 137.184.54.107
;; Query time: 157 msec ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1) ;; WHEN: Tue Apr 23 14:19:20 EDT 2024 ;; MSG SIZE rcvd: 72
bash-3.2$
Seems like quite possibly they are intermittently caching bunk data from something.
On Tue, Apr 23, 2024 at 1:39 PM Validin Axon <axon@validin.com> wrote:
Hi Jason,
I suspect what’s happened is an incorrect assumption that DNS is even the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
I appreciate the response and links. However, I've been told repeatedly by Spectrum that they're not blocking with Spectrum Shield. Despite these assurances, I've filled out a removal request through their published removal process several times, and the response I received stated that we're not being blocked. This check agrees with that: https://www.spectrum.net/support/forms/verify_url_security
"Security Shield Is Not Blocking This Site The URL provided is not being blocked by Spectrum Security Shield The URL you entered should be accessible."
Further, checking Spectrum DNS servers on the Spectrum network show that my company's main domain and all subdomains resolve to 127.0.0.54. So, if CujoAI/Spectrum Shield are not using DNS query responses to control access, then it's not CujoAI/Spectrum Shield that is responsible for the incorrect DNS response. Using a different recursive resolve, I can resolve our domains just fine. I can also resolve other domains that point to the same IPs as the sinkholed domain just fine. However, many people use the Spectrum default DNS servers and cannot access my website because of this.
You should contact Charter/Spectrum to have them investigate what their system might be blocking this content.
I have tried, for months, including spending many hours on chat and phone support, to reach someone within Spectrum support who is capable of both understanding and directing me to someone who can fix the problem, but it hasn't happened yet. I've asked to talk to someone on the DNS team and was given a flat "No." I've posted here hoping that someone in the ISP-connected world knows SOMEONE at Spectrum, Akamai, or whichever company is actually responsible for the Spectrum DNS servers who can provide a remediation path.
Regards,
Kenneth
On Tue, Apr 23, 2024 at 12:59 PM 'Livingood, Jason' via axon < axon@validin.com> wrote:
However, there's no correction process for Spectrum's DNS sinkhole
But back to the topic: someone mentioned to me that Spectrum may not be the direct providers for the DNS services they provide to their customers. If anyone knows anything about how I might discover and reach out to the people responsible, please let me know.
I suspect what’s happened is an incorrect assumption that DNS is even the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
Spectrum Shield ( https://www.spectrum.com/resources/internet-wifi/benefits-of-spectrum-securi...) is a customer-managed security protection service built into their gateways (I assume you can turn it off). The malware and content detection engine behind that is very likely run by CujoAI (https://cujo.com/) and it does not use DNS query/response exchanges as the control mechanism (in part to counter-act DNS-changing malware or malware using its own DoH channel for example).
You should contact Charter/Spectrum to have them investigate what their system might be blocking this content.
Comcast (where I work) runs a similar system ( https://www.xfinity.com/support/articles/using-xfinity-xfi-advanced-security) and maintains a site to report these sorts of issues ( https://www.xfinity.com/support/articles/report-blocked-website).
Jason
Hi Kenneth, We have been working internally and with our third-party domain reputation source to get your domain removed from their malware list. Jim From: NANOG <nanog-bounces+jim.rampley=charter.com@nanog.org> on behalf of Validin Axon <axon@validin.com> Date: Tuesday, April 23, 2024 at 2:15 PM To: Tom Beecher <beecher@beecher.cc> Cc: NANOG <nanog@nanog.org> Subject: [EXTERNAL] Re: Help with removing DNS shinkhole FP from Charter/Spectrum CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Tom, Thank you for this! It is very interesting that the behavior is intermittent. A friend of mine who tested it this weekend saw correct answers on IPv6 and incorrect answers on IPv4. Kenneth On Tue, Apr 23, 2024 at 2:56 PM Tom Beecher <beecher@beecher.cc<mailto:beecher@beecher.cc>> wrote: Validin, made an interesting observation on this. I am also a Spectrum residential customer, none of their equipment, run my own DNS server (pihole). My DHCP Assigned DNS servers are 2001:1998:f00:1::1 2001:1998:f00:2::1 bash-3.2$ dig -x 2001:1998:f00:1::1 +short dns-cac-lb-01.rr.com<http://dns-cac-lb-01.rr.com>. bash-3.2$ dig -x 2001:1998:f00:2::1 +short dns-cac-lb-02.rr.com<http://dns-cac-lb-02.rr.com>. bash-3.2$ bash-3.2$ dig dns-cac-lb-01.rr.com<http://dns-cac-lb-01.rr.com> +short 209.18.47.61 bash-3.2$ dig dns-cac-lb-02.rr.com<http://dns-cac-lb-02.rr.com> +short 209.18.47.62 bash-3.2$ bash-3.2$ dig @209.18.47.61<http://209.18.47.61> validin.com<http://validin.com> +short 157.245.112.183 137.184.54.107 bash-3.2$ dig @209.18.47.62<http://209.18.47.62> validin.com<http://validin.com> +short 157.245.112.183 137.184.54.107 bash-3.2$ bash-3.2$ dig @2001:1998:f00:1::1 validin.com<http://validin.com> +short 127.0.0.54 bash-3.2$ bash-3.2$ dig @2001:1998:f00:2::1 validin.com<http://validin.com> +short 127.0.0.54 bash-3.2$ Same servers on V4 were returning correct info, but on V6 were not. However, a few minutes later : bash-3.2$ dig @2001:1998:f00:1::1 validin.com<http://validin.com> +short 157.245.112.183 137.184.54.107 bash-3.2$ dig @2001:1998:f00:2::1 validin.com<http://validin.com> +short 157.245.112.183 137.184.54.107 bash-3.2$ Deltas : bash-3.2$ dig @2001:1998:f00:1::1 validin.com<http://validin.com> ; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com<http://validin.com> ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42329 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;validin.com<http://validin.com>. IN A ;; ANSWER SECTION: validin.com<http://validin.com>. 60 IN A 127.0.0.54 ;; Query time: 37 msec ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1) ;; WHEN: Tue Apr 23 13:50:03 EDT 2024 ;; MSG SIZE rcvd: 45 bash-3.2$ bash-3.2$ dig @2001:1998:f00:1::1 validin.com<http://validin.com> ; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com<http://validin.com> ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9667 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;validin.com<http://validin.com>. IN A ;; ANSWER SECTION: validin.com<http://validin.com>. 600 IN A 157.245.112.183 validin.com<http://validin.com>. 600 IN A 137.184.54.107 ;; Query time: 157 msec ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1) ;; WHEN: Tue Apr 23 14:19:20 EDT 2024 ;; MSG SIZE rcvd: 72 bash-3.2$ Seems like quite possibly they are intermittently caching bunk data from something. On Tue, Apr 23, 2024 at 1:39 PM Validin Axon <axon@validin.com<mailto:axon@validin.com>> wrote: Hi Jason,
I suspect what’s happened is an incorrect assumption that DNS is even the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
I appreciate the response and links. However, I've been told repeatedly by Spectrum that they're not blocking with Spectrum Shield. Despite these assurances, I've filled out a removal request through their published removal process several times, and the response I received stated that we're not being blocked. This check agrees with that: https://www.spectrum.net/support/forms/verify_url_security "Security Shield Is Not Blocking This Site The URL provided is not being blocked by Spectrum Security Shield The URL you entered should be accessible." Further, checking Spectrum DNS servers on the Spectrum network show that my company's main domain and all subdomains resolve to 127.0.0.54. So, if CujoAI/Spectrum Shield are not using DNS query responses to control access, then it's not CujoAI/Spectrum Shield that is responsible for the incorrect DNS response. Using a different recursive resolve, I can resolve our domains just fine. I can also resolve other domains that point to the same IPs as the sinkholed domain just fine. However, many people use the Spectrum default DNS servers and cannot access my website because of this.
You should contact Charter/Spectrum to have them investigate what their system might be blocking this content.
I have tried, for months, including spending many hours on chat and phone support, to reach someone within Spectrum support who is capable of both understanding and directing me to someone who can fix the problem, but it hasn't happened yet. I've asked to talk to someone on the DNS team and was given a flat "No." I've posted here hoping that someone in the ISP-connected world knows SOMEONE at Spectrum, Akamai, or whichever company is actually responsible for the Spectrum DNS servers who can provide a remediation path. Regards, Kenneth On Tue, Apr 23, 2024 at 12:59 PM 'Livingood, Jason' via axon <axon@validin.com<mailto:axon@validin.com>> wrote:
However, there's no correction process for Spectrum's DNS sinkhole But back to the topic: someone mentioned to me that Spectrum may not be the direct providers for the DNS services they provide to their customers. If anyone knows anything about how I might discover and reach out to the people responsible, please let me know.
I suspect what’s happened is an incorrect assumption that DNS is even the issue here. Because you mentioned Spectrum Shield, I suspect it is not. Spectrum Shield (https://www.spectrum.com/resources/internet-wifi/benefits-of-spectrum-securi...) is a customer-managed security protection service built into their gateways (I assume you can turn it off). The malware and content detection engine behind that is very likely run by CujoAI (https://cujo.com/) and it does not use DNS query/response exchanges as the control mechanism (in part to counter-act DNS-changing malware or malware using its own DoH channel for example). You should contact Charter/Spectrum to have them investigate what their system might be blocking this content. Comcast (where I work) runs a similar system (https://www.xfinity.com/support/articles/using-xfinity-xfi-advanced-security) and maintains a site to report these sorts of issues (https://www.xfinity.com/support/articles/report-blocked-website). Jason The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Thank you, Jim. Who is the vendor responsible? Kenneth On Tue, Apr 23, 2024 at 4:24 PM Rampley, Jim F <jim.rampley@charter.com> wrote:
Hi Kenneth,
We have been working internally and with our third-party domain reputation source to get your domain removed from their malware list.
Jim
*From: *NANOG <nanog-bounces+jim.rampley=charter.com@nanog.org> on behalf of Validin Axon <axon@validin.com> *Date: *Tuesday, April 23, 2024 at 2:15 PM *To: *Tom Beecher <beecher@beecher.cc> *Cc: *NANOG <nanog@nanog.org> *Subject: *[EXTERNAL] Re: Help with removing DNS shinkhole FP from Charter/Spectrum
CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
Tom,
Thank you for this! It is very interesting that the behavior is intermittent. A friend of mine who tested it this weekend saw correct answers on IPv6 and incorrect answers on IPv4.
Kenneth
On Tue, Apr 23, 2024 at 2:56 PM Tom Beecher <beecher@beecher.cc> wrote:
Validin, made an interesting observation on this. I am also a Spectrum residential customer, none of their equipment, run my own DNS server (pihole).
My DHCP Assigned DNS servers are
2001:1998:f00:1::1 2001:1998:f00:2::1
bash-3.2$ dig -x 2001:1998:f00:1::1 +short dns-cac-lb-01.rr.com. bash-3.2$ dig -x 2001:1998:f00:2::1 +short dns-cac-lb-02.rr.com. bash-3.2$
bash-3.2$ dig dns-cac-lb-01.rr.com +short 209.18.47.61 bash-3.2$ dig dns-cac-lb-02.rr.com +short 209.18.47.62 bash-3.2$
bash-3.2$ dig @209.18.47.61 validin.com +short 157.245.112.183 137.184.54.107 bash-3.2$ dig @209.18.47.62 validin.com +short 157.245.112.183 137.184.54.107 bash-3.2$
bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short 127.0.0.54 bash-3.2$
bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short 127.0.0.54 bash-3.2$
Same servers on V4 were returning correct info, but on V6 were not.
However, a few minutes later :
bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short 157.245.112.183 137.184.54.107 bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short 157.245.112.183 137.184.54.107 bash-3.2$
Deltas :
bash-3.2$ dig @2001:1998:f00:1::1 validin.com
; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42329 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;validin.com. IN A
;; ANSWER SECTION: validin.com. 60 IN A 127.0.0.54
;; Query time: 37 msec ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1) ;; WHEN: Tue Apr 23 13:50:03 EDT 2024 ;; MSG SIZE rcvd: 45
bash-3.2$
bash-3.2$ dig @2001:1998:f00:1::1 validin.com
; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9667 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;validin.com. IN A
;; ANSWER SECTION: validin.com. 600 IN A 157.245.112.183 validin.com. 600 IN A 137.184.54.107
;; Query time: 157 msec ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1) ;; WHEN: Tue Apr 23 14:19:20 EDT 2024 ;; MSG SIZE rcvd: 72
bash-3.2$
Seems like quite possibly they are intermittently caching bunk data from something.
On Tue, Apr 23, 2024 at 1:39 PM Validin Axon <axon@validin.com> wrote:
Hi Jason,
I suspect what’s happened is an incorrect assumption that DNS is even the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
I appreciate the response and links. However, I've been told repeatedly by Spectrum that they're not blocking with Spectrum Shield. Despite these assurances, I've filled out a removal request through their published removal process several times, and the response I received stated that we're not being blocked. This check agrees with that:
https://www.spectrum.net/support/forms/verify_url_security
"Security Shield Is Not Blocking This Site
The URL provided is not being blocked by Spectrum Security Shield The URL you entered should be accessible."
Further, checking Spectrum DNS servers on the Spectrum network show that my company's main domain and all subdomains resolve to 127.0.0.54. So, if CujoAI/Spectrum Shield are not using DNS query responses to control access, then it's not CujoAI/Spectrum Shield that is responsible for the incorrect DNS response. Using a different recursive resolve, I can resolve our domains just fine. I can also resolve other domains that point to the same IPs as the sinkholed domain just fine. However, many people use the Spectrum default DNS servers and cannot access my website because of this.
You should contact Charter/Spectrum to have them investigate what their system might be blocking this content.
I have tried, for months, including spending many hours on chat and phone support, to reach someone within Spectrum support who is capable of both understanding and directing me to someone who can fix the problem, but it hasn't happened yet. I've asked to talk to someone on the DNS team and was given a flat "No." I've posted here hoping that someone in the ISP-connected world knows SOMEONE at Spectrum, Akamai, or whichever company is actually responsible for the Spectrum DNS servers who can provide a remediation path.
Regards,
Kenneth
On Tue, Apr 23, 2024 at 12:59 PM 'Livingood, Jason' via axon < axon@validin.com> wrote:
However, there's no correction process for Spectrum's DNS sinkhole
But back to the topic: someone mentioned to me that Spectrum may not be the direct providers for the DNS services they provide to their customers. If anyone knows anything about how I might discover and reach out to the people responsible, please let me know.
I suspect what’s happened is an incorrect assumption that DNS is even the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
Spectrum Shield ( https://www.spectrum.com/resources/internet-wifi/benefits-of-spectrum-securi...) is a customer-managed security protection service built into their gateways (I assume you can turn it off). The malware and content detection engine behind that is very likely run by CujoAI (https://cujo.com/) and it does not use DNS query/response exchanges as the control mechanism (in part to counter-act DNS-changing malware or malware using its own DoH channel for example).
You should contact Charter/Spectrum to have them investigate what their system might be blocking this content.
Comcast (where I work) runs a similar system ( https://www.xfinity.com/support/articles/using-xfinity-xfi-advanced-security) and maintains a site to report these sorts of issues ( https://www.xfinity.com/support/articles/report-blocked-website).
Jason
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
On Mon, 22 Apr 2024, William Herrin wrote:
Respectfully, you're mistaken. Look up "tortious interference."
I'm familiar with it. But I am also familar with many cases were spammers have sued network operators claiming that they're falsely defamed, so the operator has to deliver their mail. They have without exception lost. If you can find actual cases where a court forced an operator to deliver a third party's traffic I would like to hear about it.* 43 USC 230(c)(A) provides extremely broad protection for "good faith" blocking, which means that a complaint would have to show that the blocking was malicious rather than merited or accidental. In this case it seems probably accidental, but for all I know there might have been bad traffic to merit a block. Here's one of the cases where a spammer lost: https://jl.ly/Email/holomaxx.html https://jl.ly/Email/holo4.html And here's one where the judge rejected tortious interference: https://jl.ly/Email/spamarrest.html
My results going through the support front-door at large companies for oddball problems have been less than stellar. Has your experience truly been different?
No, it's terrible, and Spectrum is particularly bad. I am now in month three of trying to get them to route a /24 to my host that belongs to one of my users, and their responses can be summarized as very complex exegeses of "duh?" But bogus lawyer letters will just make things worse. R's, John * - let's stay away for now from the Texas and Florida social network common carrier laws which are a whole other can of s*
On Mon, Apr 22, 2024 at 5:07 PM John R. Levine <johnl@iecc.com> wrote:
a complaint would have to show that the blocking was malicious rather than merited or accidental. In this case it seems probably accidental, but for all I know there might have been bad traffic to merit a block.
Hi John, I'll try not to belabor it, but accidental that isn't corrected upon formal legal notification becomes negligent and negligent has more or less the same legal status as malicious. The spammers lost because the networks published a terms of use document that the spammers unambiguously violated. Even though it interfered with the spammer's business, the block was merited so the preponderance of the evidence fell in favor of the service provider. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
Bill is absolutely correct. The spammers lost their case because they were demonstrably spammers. We’ve had accidental black hole cases with *US* providers that removed the block once they received a C&D. If they don’t have iron clad proof in hand. (More than just a few complaints and no traffic analysis), it’s just the least risky response. That doesn’t work well with overseas providers, though, because they’re essentially immune to U.S. litigation unless the plaintiff has deep pockets. -mel On Apr 22, 2024, at 5:21 PM, William Herrin <bill@herrin.us> wrote: On Mon, Apr 22, 2024 at 5:07 PM John R. Levine <johnl@iecc.com> wrote: a complaint would have to show that the blocking was malicious rather than merited or accidental. In this case it seems probably accidental, but for all I know there might have been bad traffic to merit a block. Hi John, I'll try not to belabor it, but accidental that isn't corrected upon formal legal notification becomes negligent and negligent has more or less the same legal status as malicious. The spammers lost because the networks published a terms of use document that the spammers unambiguously violated. Even though it interfered with the spammer's business, the block was merited so the preponderance of the evidence fell in favor of the service provider. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
Bill is absolutely correct. The spammers lost their case because they were demonstrably spammers.
No, really they did not. I read the decisions. Have you? Hint: under CAN SPAM a great deal of spam is completely legal so it didn't matter.
We’ve had accidental black hole cases with *US* providers that removed the block once they received a C&D. If they don’t have iron clad proof in hand. (More than just a few complaints and no traffic analysis), it’s just the least risky response.
I will believe that there are people that cave in response to threats like this, but again, there is no case law to support it. R's, John
participants (9)
-
Christopher Morrow -
John Levine -
John R. Levine
-
Livingood, Jason -
Mel Beckman -
Rampley, Jim F -
Tom Beecher -
Validin Axon -
William Herrin