Am I correct to understand that 1.1.1.1 only does support via community forum? They had just enough interest in the service to collect user data to monetise, but 0 interest in trying to figure out how to detect and solve problems? Why not build a web form where they ask you to explain what is not working, in terms of automatically testable. Like no A record for X. Then after you submit this form, they test against all 1.1.1.1 and some 9.9.9.9 and 8.8.8.8 and if they find a difference in behaviour, the ticket is accepted and sent to someone who understands DNS? If there is no difference in behaviour, direct people to community forums. This trivial, cheap and fast to produce support channel would ensure virtually 0 trash support cases, so you wouldn't even have to hire people to support your data collection enterprise. Very obviously they selfishly had no interest in ensuring 1.1.1.1 actually works, as long as they are getting the data. I do not know how to characterise this as anything but unethical. https://community.cloudflare.com/t/1-1-1-1-wont-resolve-www-moi-gov-cy-in-lc... https://community.cloudflare.com/t/1-1-1-1-failing-to-resolve/474228 If you can't due to resources or competence support DNS, do not offer one. -- ++ytti, cake having and cake eating user
What about the zone not having a single point of failure? Both servers are covered by the same /24. % dig www.moi.gov.cy @212.31.118.19 +norec +dnssec ; <<>> DiG 9.19.11-dev <<>> www.moi.gov.cy @212.31.118.19 +norec +dnssec ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17380 ;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ; COOKIE: 6387183a6031ef182fa6ade7641ad4ff2a078213f4e24fc9 (good) ;; QUESTION SECTION: ;www.moi.gov.cy. IN A ;; ANSWER SECTION: www.moi.gov.cy. 3600 IN A 212.31.118.26 ;; AUTHORITY SECTION: moi.gov.cy. 3600 IN NS ns01.gov.cy. moi.gov.cy. 3600 IN NS ns02.gov.cy. ;; ADDITIONAL SECTION: ns02.gov.cy. 86400 IN A 212.31.118.20 ns01.gov.cy. 86400 IN A 212.31.118.19 ;; Query time: 374 msec ;; SERVER: 212.31.118.19#53(212.31.118.19) (UDP) ;; WHEN: Wed Mar 22 21:14:23 AEDT 2023 ;; MSG SIZE rcvd: 157 %
On 22 Mar 2023, at 19:36, Saku Ytti <saku@ytti.fi> wrote:
Am I correct to understand that 1.1.1.1 only does support via community forum?
They had just enough interest in the service to collect user data to monetise, but 0 interest in trying to figure out how to detect and solve problems?
Why not build a web form where they ask you to explain what is not working, in terms of automatically testable. Like no A record for X. Then after you submit this form, they test against all 1.1.1.1 and some 9.9.9.9 and 8.8.8.8 and if they find a difference in behaviour, the ticket is accepted and sent to someone who understands DNS? If there is no difference in behaviour, direct people to community forums. This trivial, cheap and fast to produce support channel would ensure virtually 0 trash support cases, so you wouldn't even have to hire people to support your data collection enterprise.
The number of times that 8.8.8.8 “works” but there is an actual error is enormous. 8.8.8.8 tolerates lots of protocol errors which ends up causing support cases for others where the result is “the servers are broken in this way”. You then try to report the issue but the report is ignored because “It works with 8.8.8.8”.
Very obviously they selfishly had no interest in ensuring 1.1.1.1 actually works, as long as they are getting the data. I do not know how to characterise this as anything but unethical.
https://community.cloudflare.com/t/1-1-1-1-wont-resolve-www-moi-gov-cy-in-lc... https://community.cloudflare.com/t/1-1-1-1-failing-to-resolve/474228
If you can't due to resources or competence support DNS, do not offer one.
-- ++ytti, cake having and cake eating user
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
If you wish to consult people on how to configure DNS, please reach out to the responsible folk. I am discussing a specific recursor in anycasted setup not resolving domain and provider offering no remediation channel. These are two entirely different classes of problem and collapsing them into a single problem is not going to help in either case. On Wed, 22 Mar 2023 at 12:25, Mark Andrews <marka@isc.org> wrote:
What about the zone not having a single point of failure? Both servers are covered by the same /24.
% dig www.moi.gov.cy @212.31.118.19 +norec +dnssec
; <<>> DiG 9.19.11-dev <<>> www.moi.gov.cy @212.31.118.19 +norec +dnssec ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17380 ;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ; COOKIE: 6387183a6031ef182fa6ade7641ad4ff2a078213f4e24fc9 (good) ;; QUESTION SECTION: ;www.moi.gov.cy. IN A
;; ANSWER SECTION: www.moi.gov.cy. 3600 IN A 212.31.118.26
;; AUTHORITY SECTION: moi.gov.cy. 3600 IN NS ns01.gov.cy. moi.gov.cy. 3600 IN NS ns02.gov.cy.
;; ADDITIONAL SECTION: ns02.gov.cy. 86400 IN A 212.31.118.20 ns01.gov.cy. 86400 IN A 212.31.118.19
;; Query time: 374 msec ;; SERVER: 212.31.118.19#53(212.31.118.19) (UDP) ;; WHEN: Wed Mar 22 21:14:23 AEDT 2023 ;; MSG SIZE rcvd: 157
%
On 22 Mar 2023, at 19:36, Saku Ytti <saku@ytti.fi> wrote:
Am I correct to understand that 1.1.1.1 only does support via community forum?
They had just enough interest in the service to collect user data to monetise, but 0 interest in trying to figure out how to detect and solve problems?
Why not build a web form where they ask you to explain what is not working, in terms of automatically testable. Like no A record for X. Then after you submit this form, they test against all 1.1.1.1 and some 9.9.9.9 and 8.8.8.8 and if they find a difference in behaviour, the ticket is accepted and sent to someone who understands DNS? If there is no difference in behaviour, direct people to community forums. This trivial, cheap and fast to produce support channel would ensure virtually 0 trash support cases, so you wouldn't even have to hire people to support your data collection enterprise.
The number of times that 8.8.8.8 “works” but there is an actual error is enormous. 8.8.8.8 tolerates lots of protocol errors which ends up causing support cases for others where the result is “the servers are broken in this way”. You then try to report the issue but the report is ignored because “It works with 8.8.8.8”.
Very obviously they selfishly had no interest in ensuring 1.1.1.1 actually works, as long as they are getting the data. I do not know how to characterise this as anything but unethical.
https://community.cloudflare.com/t/1-1-1-1-wont-resolve-www-moi-gov-cy-in-lc... https://community.cloudflare.com/t/1-1-1-1-failing-to-resolve/474228
If you can't due to resources or competence support DNS, do not offer one.
-- ++ytti, cake having and cake eating user
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- ++ytti
Why would they need it, its free, they are not being paid to be your DNS servers. Assuming the provider is 1.1.1.1 itself. YOUR ISP SHOULD NOT USE 1.1.1.1 or 8.8.8.8, you should run your OWN DNS servers. If its not within your circle of influence, don’t' risk your business on it! Dennis Burgess, Mikrotik Certified Trainer MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net Need to Automate MikroTik Backups: https://cloud.linktechs.net Create Wireless Coverage’s with www.towercoverage.com -----Original Message----- From: NANOG <nanog-bounces+dmburgess=linktechs.net@nanog.org> On Behalf Of Saku Ytti Sent: Wednesday, March 22, 2023 6:53 AM To: Mark Andrews <marka@isc.org> Cc: nanog list <nanog@nanog.org> Subject: Re: 1.1.1.1 support? If you wish to consult people on how to configure DNS, please reach out to the responsible folk. I am discussing a specific recursor in anycasted setup not resolving domain and provider offering no remediation channel. These are two entirely different classes of problem and collapsing them into a single problem is not going to help in either case. On Wed, 22 Mar 2023 at 12:25, Mark Andrews <marka@isc.org> wrote:
What about the zone not having a single point of failure? Both servers are covered by the same /24.
% dig www.moi.gov.cy @212.31.118.19 +norec +dnssec
; <<>> DiG 9.19.11-dev <<>> www.moi.gov.cy @212.31.118.19 +norec +dnssec ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17380 ;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ; COOKIE: 6387183a6031ef182fa6ade7641ad4ff2a078213f4e24fc9 (good) ;; QUESTION SECTION: ;www.moi.gov.cy. IN A
;; ANSWER SECTION: www.moi.gov.cy. 3600 IN A 212.31.118.26
;; AUTHORITY SECTION: moi.gov.cy. 3600 IN NS ns01.gov.cy. moi.gov.cy. 3600 IN NS ns02.gov.cy.
;; ADDITIONAL SECTION: ns02.gov.cy. 86400 IN A 212.31.118.20 ns01.gov.cy. 86400 IN A 212.31.118.19
;; Query time: 374 msec ;; SERVER: 212.31.118.19#53(212.31.118.19) (UDP) ;; WHEN: Wed Mar 22 21:14:23 AEDT 2023 ;; MSG SIZE rcvd: 157
%
On 22 Mar 2023, at 19:36, Saku Ytti <saku@ytti.fi> wrote:
Am I correct to understand that 1.1.1.1 only does support via community forum?
They had just enough interest in the service to collect user data to monetise, but 0 interest in trying to figure out how to detect and solve problems?
Why not build a web form where they ask you to explain what is not working, in terms of automatically testable. Like no A record for X. Then after you submit this form, they test against all 1.1.1.1 and some 9.9.9.9 and 8.8.8.8 and if they find a difference in behaviour, the ticket is accepted and sent to someone who understands DNS? If there is no difference in behaviour, direct people to community forums. This trivial, cheap and fast to produce support channel would ensure virtually 0 trash support cases, so you wouldn't even have to hire people to support your data collection enterprise.
The number of times that 8.8.8.8 “works” but there is an actual error is enormous. 8.8.8.8 tolerates lots of protocol errors which ends up causing support cases for others where the result is “the servers are broken in this way”. You then try to report the issue but the report is ignored because “It works with 8.8.8.8”.
Very obviously they selfishly had no interest in ensuring 1.1.1.1 actually works, as long as they are getting the data. I do not know how to characterise this as anything but unethical.
https://community.cloudflare.com/t/1-1-1-1-wont-resolve-www-moi-gov- cy-in-lca-235m3/487469 https://community.cloudflare.com/t/1-1-1-1-failing-to-resolve/474228
If you can't due to resources or competence support DNS, do not offer one.
-- ++ytti, cake having and cake eating user
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- ++ytti
Matt Harris VP OF INFRASTRUCTURE Follow us on LinkedIn! matt.harris@netfire.net 816-256-5446 www.netfire.com On Wed, Mar 22, 2023 at 3:36 AM Saku Ytti <saku@ytti.fi> wrote:
Am I correct to understand that 1.1.1.1 only does support via community forum?
They had just enough interest in the service to collect user data to monetise, but 0 interest in trying to figure out how to detect and solve problems?
Why not build a web form where they ask you to explain what is not working, in terms of automatically testable. Like no A record for X. Then after you submit this form, they test against all 1.1.1.1 and some 9.9.9.9 and 8.8.8.8 and if they find a difference in behaviour, the ticket is accepted and sent to someone who understands DNS? If there is no difference in behaviour, direct people to community forums. This trivial, cheap and fast to produce support channel would ensure virtually 0 trash support cases, so you wouldn't even have to hire people to support your data collection enterprise.
Very obviously they selfishly had no interest in ensuring 1.1.1.1 actually works, as long as they are getting the data. I do not know how to characterise this as anything but unethical.
https://community.cloudflare.com/t/1-1-1-1-wont-resolve-www-moi-gov-cy-in-lc... https://community.cloudflare.com/t/1-1-1-1-failing-to-resolve/474228
If you can't due to resources or competence support DNS, do not offer one.
Saku, When something is provided at no cost, I don't see how it can be unethical unless they are explicitly lying about the ways in which they use the data they gather. Ultimately, you're asking them to provide a costly service (support for end-users, the vast majority of whom will not ask informed, intelligent questions like the members of this list would be able to, but would still demand the same level of support) on top of a service they are already providing at no cost. That's both unrealistic and unnecessary. There's an exceedingly simple solution, here, after all: if you don't like their service or it isn't working for you as an end-user, don't use it. On the same token as network operators, it might be nice if cloudflare's admins were accessible to address potential issues that may actually be related to legitimate network misconfigurations or other problems on their end that result in issues resolving some folks' resources - and I suspect they may in fact be via this list or other similar ones, or other open resources that are widely available to folks who are in the know. That said, with regards to any specific case, we don't know whose end the issue lies on. It's possible that the folks managing the Cypress government resources have taken steps actively, or passively misconfigured, their systems in such a way that causes the root problem that you're pointing out. As I administer neither of the related networks, I can't speak to this, but I think it's just as likely based on a coin flip that they are responsible for the issue as it is that cloudflare is responsible for the issue. On top of that, I suspect getting technology help from a random government entity may be far less fruitful than even a public forum would be. Good luck getting a resolution to your resolution.
On Wed, 22 Mar 2023 at 15:26, Matt Harris <matt@netfire.net> wrote:
When something is provided at no cost, I don't see how it can be unethical unless they are explicitly lying about the ways in which they use the data they gather. Ultimately, you're asking them to provide a costly service (support for end-users, the vast majority of whom will not ask informed, intelligent questions like the members of this list would be able to, but would still demand the same level of support) on top of a service they are already providing at no cost. That's both unrealistic and unnecessary. There's an exceedingly simple solution, here, after all: if you don't like their service or it isn't working for you as an end-user, don't use it.
Thank you for the philosophical perspective, but currently my interest is not to debate merits or lack thereof in laissez-faire economics. The problem is, a large number of people will use 1.1.1.1, 8.8.8.8 or 9.9.9.9 despite my or your position about it. There is incentive for providers to provide it 'for free', as it adds value to their products as users are compensating providers with the data. Occasionally things don't work and when they do not, we need a way to inform the provider 'hey you have a problem'. You could be anywhere in this chain, with no ability to impact any of the decisions. I know there is a real problem, I know real users are impacted, I know almost none of them will have the ability to understand why there is a problem or remediate it. -- ++ytti
Try asking dns-operations@lists.dns-oarc.net for someone at CloudFlare. For what it's worth, it works for me. I'm in Troy, OH. C:\Users\jluthman>dig www.moi.gov.cy @1.1.1.1 +short 212.31.118.26 On Wed, Mar 22, 2023 at 9:43 AM Saku Ytti <saku@ytti.fi> wrote:
On Wed, 22 Mar 2023 at 15:26, Matt Harris <matt@netfire.net> wrote:
When something is provided at no cost, I don't see how it can be unethical unless they are explicitly lying about the ways in which they use the data they gather. Ultimately, you're asking them to provide a costly service (support for end-users, the vast majority of whom will not ask informed, intelligent questions like the members of this list would be able to, but would still demand the same level of support) on top of a service they are already providing at no cost. That's both unrealistic and unnecessary. There's an exceedingly simple solution, here, after all: if you don't like their service or it isn't working for you as an end-user, don't use it.
Thank you for the philosophical perspective, but currently my interest is not to debate merits or lack thereof in laissez-faire economics.
The problem is, a large number of people will use 1.1.1.1, 8.8.8.8 or 9.9.9.9 despite my or your position about it. There is incentive for providers to provide it 'for free', as it adds value to their products as users are compensating providers with the data.
Occasionally things don't work and when they do not, we need a way to inform the provider 'hey you have a problem'. You could be anywhere in this chain, with no ability to impact any of the decisions.
I know there is a real problem, I know real users are impacted, I know almost none of them will have the ability to understand why there is a problem or remediate it.
-- ++ytti
Yes, it works in every other CF except LCA-CF. Thank you for the additional data point. You can use `dig CHAOS TXT id.server @1.1.1.1 +nsid` to get two unicast identifiers for the server you got the response from. On Wed, 22 Mar 2023 at 15:49, Josh Luthman <josh@imaginenetworksllc.com> wrote:
Try asking dns-operations@lists.dns-oarc.net for someone at CloudFlare.
For what it's worth, it works for me. I'm in Troy, OH.
C:\Users\jluthman>dig www.moi.gov.cy @1.1.1.1 +short 212.31.118.26
On Wed, Mar 22, 2023 at 9:43 AM Saku Ytti <saku@ytti.fi> wrote:
On Wed, 22 Mar 2023 at 15:26, Matt Harris <matt@netfire.net> wrote:
When something is provided at no cost, I don't see how it can be unethical unless they are explicitly lying about the ways in which they use the data they gather. Ultimately, you're asking them to provide a costly service (support for end-users, the vast majority of whom will not ask informed, intelligent questions like the members of this list would be able to, but would still demand the same level of support) on top of a service they are already providing at no cost. That's both unrealistic and unnecessary. There's an exceedingly simple solution, here, after all: if you don't like their service or it isn't working for you as an end-user, don't use it.
Thank you for the philosophical perspective, but currently my interest is not to debate merits or lack thereof in laissez-faire economics.
The problem is, a large number of people will use 1.1.1.1, 8.8.8.8 or 9.9.9.9 despite my or your position about it. There is incentive for providers to provide it 'for free', as it adds value to their products as users are compensating providers with the data.
Occasionally things don't work and when they do not, we need a way to inform the provider 'hey you have a problem'. You could be anywhere in this chain, with no ability to impact any of the decisions.
I know there is a real problem, I know real users are impacted, I know almost none of them will have the ability to understand why there is a problem or remediate it.
-- ++ytti
-- ++ytti
On 2023-03-22 10:36:03 +0200, Saku Ytti wrote:
Am I correct to understand that 1.1.1.1 only does support via community forum?
The community forum is our preferred method of support, yes.
Why not build a web form where they ask you to explain what is not working, in terms of automatically testable. Like no A record for X. Then after you submit this form, they test against all 1.1.1.1 and some 9.9.9.9 and 8.8.8.8 and if they find a difference in behaviour, the ticket is accepted and sent to someone who understands DNS? If there is no difference in behaviour, direct people to community forums.
I'll take this feedback to our developers.
https://community.cloudflare.com/t/1-1-1-1-wont-resolve-www-moi-gov-cy-in-lc... https://community.cloudflare.com/t/1-1-1-1-failing-to-resolve/474228
I took a look at the above tickets, and it seems that one of the egress ranges from that datacenter cannot connect to the authoritative nameservers of `www.moi.gov.cy`: `ns01.gov.cy` and `ns02.gov.cy`. Here's a redacted pcap for those who like details, showing no response: IP a.b.c.d.56552 > 212.31.118.19.53: 51873+ [1au] A? www.moi.gov.cy. (55) IP a.b.c.d.51718 > 212.31.118.20.53: 31021+ [1au] A? www.moi.gov.cy. (55) TCP behaves similarly. The source prefixes having issues connecting to 212.31.118.19 and 212.31.118.20 are: 172.68.130.0/24, while a neighbouring source prefix 172.68.171.0/24 seems to connect fine. I'm filing an internal ticket right now to investigate, but I'd appreciate if you could also help us on your end for any possible solutions regarding this connectivity failure. As a general note regarding the two community posts: the straight deep dive into technical information makes it more difficult for others to interpret the request. As you said in a later post here:
I know almost none of them will have the ability to understand why there is a problem or remediate it.
Not everyone in the Community Forum (nor our company) can pull out the specific datacenter used, the specific machine(s) used, and the source ASN from the `my.ip.fi` curl. An preamble will greatly help in context. Thanks for reaching out and sorry that you had to escalate to another medium, -- alex [at] e [dot] sc alexander [at] cloudflare [dot] com
On Wed, 22 Mar 2023 at 16:04, Alexander Huynh via NANOG <nanog@nanog.org> wrote:
I'll take this feedback to our developers.
Many thanks.
I took a look at the above tickets, and it seems that one of the egress ranges from that datacenter cannot connect to the authoritative nameservers of `www.moi.gov.cy`: `ns01.gov.cy` and `ns02.gov.cy`.
Here's a redacted pcap for those who like details, showing no response:
IP a.b.c.d.56552 > 212.31.118.19.53: 51873+ [1au] A? www.moi.gov.cy. (55) IP a.b.c.d.51718 > 212.31.118.20.53: 31021+ [1au] A? www.moi.gov.cy. (55)
TCP behaves similarly.
The recursor response suggests a loop, so network problem is highly likely.
I'm filing an internal ticket right now to investigate, but I'd appreciate if you could also help us on your end for any possible solutions regarding this connectivity failure.
Sure, you might also want to look into nlnog ring, which allows a broad perspective to issues.
As a general note regarding the two community posts: the straight deep dive into technical information makes it more difficult for others to interpret the request. As you said in a later post here:
This is a very difficult subject. How to get help. If I had made it more genetic, we could refute it as it doesn't contain needed information. If I made it longer we could refute that it's not terse enough. However we submit it, we can argue it wasn't the right way. As seen in the original post, I fully appreciate almost every single case about 1.1.1.1 is incorrect and user error. But I proposed a mechanism to by-pass community forums and reach people who are able to help and understand. If there is disagreement in 1.1.1.1, 8.8.8.8 and 9.9.9.9 then let humans analyse it. The ticket volume would be trivial, if we look at community forums and see how many 1.1.1.1 complaints would bypass this filter.
Not everyone in the Community Forum (nor our company) can pull out the specific datacenter used, the specific machine(s) used, and the source ASN from the `my.ip.fi` curl.
I gave the specific unicast ID for the DNS server in addition to my IP. I cannot glean any other information. I don't think we can fairly fault either of the cases in the community forum. We must fault the process itself and look for ways to improve. -- ++ytti
participants (6)
-
Alexander Huynh -
Dennis Burgess -
Josh Luthman -
Mark Andrews -
Matt Harris -
Saku Ytti