Anyone seeing Cloudflare DNS outages or site issues? Affecting a bunch of sites in Washington and Oregon. -A
More digging shows high latency to CloudFlare DNS servers from Comcast in Washington and Oregon as well as a few other providers (Charter, ToledoTel), etc... Sites that do resolve using other DNS servers but are hosted on CloudFlare aren't loading. Sites that use CloudFlare for their DNS aren't resolving either. traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets 1 _gateway (192.168.42.254) 0.185 ms 0.109 ms 0.117 ms 2 pppoe-gw-208-70-52.toledotel.com (208.70.52.1) 1.896 ms 1.881 ms 1.903 ms 3 tuk-edge-13.inet.qwest.net (198.233.244.225) 4.158 ms 4.082 ms 4.071 ms 4 sea-brdr-03.inet.qwest.net (67.14.41.154) 8.976 ms 8.949 ms 8.903 ms 5 * * * 6 ae-1-51.ear2.Seattle1.Level3.net (4.69.203.173) 4.494 ms 4.350 ms 4.311 ms 7 4.53.154.10 (4.53.154.10) 77.622 ms 103.323 ms 103.240 ms 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 one.one.one.one (1.1.1.1) 87.515 ms * * -A On Fri, Jul 17, 2020 at 2:18 PM Aaron C. de Bruyn <aaron@heyaaron.com> wrote:
Anyone seeing Cloudflare DNS outages or site issues?
Affecting a bunch of sites in Washington and Oregon.
-A
CloudFlare updated their status page and confirmed the issue: https://www.cloudflarestatus.com/ -A On Fri, Jul 17, 2020 at 2:33 PM Aaron C. de Bruyn <aaron@heyaaron.com> wrote:
More digging shows high latency to CloudFlare DNS servers from Comcast in Washington and Oregon as well as a few other providers (Charter, ToledoTel), etc...
Sites that do resolve using other DNS servers but are hosted on CloudFlare aren't loading. Sites that use CloudFlare for their DNS aren't resolving either. traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
1 _gateway (192.168.42.254) 0.185 ms 0.109 ms 0.117 ms 2 pppoe-gw-208-70-52.toledotel.com (208.70.52.1) 1.896 ms 1.881 ms 1.903 ms 3 tuk-edge-13.inet.qwest.net (198.233.244.225) 4.158 ms 4.082 ms 4.071 ms 4 sea-brdr-03.inet.qwest.net (67.14.41.154) 8.976 ms 8.949 ms 8.903 ms 5 * * * 6 ae-1-51.ear2.Seattle1.Level3.net (4.69.203.173) 4.494 ms 4.350 ms 4.311 ms 7 4.53.154.10 (4.53.154.10) 77.622 ms 103.323 ms 103.240 ms 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 one.one.one.one (1.1.1.1) 87.515 ms * *
-A
On Fri, Jul 17, 2020 at 2:18 PM Aaron C. de Bruyn <aaron@heyaaron.com> wrote:
Anyone seeing Cloudflare DNS outages or site issues?
Affecting a bunch of sites in Washington and Oregon.
-A
We're peered with them and are having issues resolving some domains via Cloudflare right now. On Fri, Jul 17, 2020 at 2:44 PM Aaron C. de Bruyn via NANOG <nanog@nanog.org> wrote:
More digging shows high latency to CloudFlare DNS servers from Comcast in Washington and Oregon as well as a few other providers (Charter, ToledoTel), etc...
Sites that do resolve using other DNS servers but are hosted on CloudFlare aren't loading. Sites that use CloudFlare for their DNS aren't resolving either. traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
1 _gateway (192.168.42.254) 0.185 ms 0.109 ms 0.117 ms 2 pppoe-gw-208-70-52.toledotel.com (208.70.52.1) 1.896 ms 1.881 ms 1.903 ms 3 tuk-edge-13.inet.qwest.net (198.233.244.225) 4.158 ms 4.082 ms 4.071 ms 4 sea-brdr-03.inet.qwest.net (67.14.41.154) 8.976 ms 8.949 ms 8.903 ms 5 * * * 6 ae-1-51.ear2.Seattle1.Level3.net (4.69.203.173) 4.494 ms 4.350 ms 4.311 ms 7 4.53.154.10 (4.53.154.10) 77.622 ms 103.323 ms 103.240 ms 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 one.one.one.one (1.1.1.1) 87.515 ms * *
-A
On Fri, Jul 17, 2020 at 2:18 PM Aaron C. de Bruyn <aaron@heyaaron.com> wrote:
Anyone seeing Cloudflare DNS outages or site issues?
Affecting a bunch of sites in Washington and Oregon.
-A
-- <http://www.bcarlsonmedia.com/> http://www.bcarlsonmedia.com @brendancarlson <http://www.twitter.com/brendancarlson> +1 (626) 921-6503
Did anyone see any collateral damage from this outside of Cloudflare? Specifically Azure? I manage a very large site in Azure, and at the exact same time of the Cloudflare incident we saw a spike in traffic (like a DDoS or Bot), then followed by unusual hardware resource anomalies. We’re globally spread in Azure, but we only saw this in the US and Brazil. Very coincidental, but possible. -John
On Jul 17, 2020, at 5:33 PM, Aaron C. de Bruyn via NANOG <nanog@nanog.org> wrote:
More digging shows high latency to CloudFlare DNS servers from Comcast in Washington and Oregon as well as a few other providers (Charter, ToledoTel), etc...
Sites that do resolve using other DNS servers but are hosted on CloudFlare aren't loading. Sites that use CloudFlare for their DNS aren't resolving either. traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
1 _gateway (192.168.42.254) 0.185 ms 0.109 ms 0.117 ms 2 pppoe-gw-208-70-52.toledotel.com <http://pppoe-gw-208-70-52.toledotel.com/> (208.70.52.1) 1.896 ms 1.881 ms 1.903 ms 3 tuk-edge-13.inet.qwest.net <http://tuk-edge-13.inet.qwest.net/> (198.233.244.225) 4.158 ms 4.082 ms 4.071 ms 4 sea-brdr-03.inet.qwest.net <http://sea-brdr-03.inet.qwest.net/> (67.14.41.154) 8.976 ms 8.949 ms 8.903 ms 5 * * * 6 ae-1-51.ear2.Seattle1.Level3.net <http://ae-1-51.ear2.seattle1.level3.net/> (4.69.203.173) 4.494 ms 4.350 ms 4.311 ms 7 4.53.154.10 (4.53.154.10) 77.622 ms 103.323 ms 103.240 ms 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 one.one.one.one (1.1.1.1) 87.515 ms * *
-A
On Fri, Jul 17, 2020 at 2:18 PM Aaron C. de Bruyn <aaron@heyaaron.com <mailto:aaron@heyaaron.com>> wrote: Anyone seeing Cloudflare DNS outages or site issues?
Affecting a bunch of sites in Washington and Oregon.
-A
All three resolver of the big German hoster Hetzner went offline with 1.1.1.1 and were down another hour after Cloudflare was up again. I hope it was a coincidence and they are not forwarding their requests to 1.1.1.1. https://www.hetzner-status.de/ - vidister
On 18. Jul 2020, at 02:46, John Von Essen <john@essenz.com> wrote:
Did anyone see any collateral damage from this outside of Cloudflare? Specifically Azure?
I manage a very large site in Azure, and at the exact same time of the Cloudflare incident we saw a spike in traffic (like a DDoS or Bot), then followed by unusual hardware resource anomalies. We’re globally spread in Azure, but we only saw this in the US and Brazil.
Very coincidental, but possible.
-John
On Jul 17, 2020, at 5:33 PM, Aaron C. de Bruyn via NANOG <nanog@nanog.org> wrote:
More digging shows high latency to CloudFlare DNS servers from Comcast in Washington and Oregon as well as a few other providers (Charter, ToledoTel), etc...
Sites that do resolve using other DNS servers but are hosted on CloudFlare aren't loading. Sites that use CloudFlare for their DNS aren't resolving either. traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
1 _gateway (192.168.42.254) 0.185 ms 0.109 ms 0.117 ms 2 pppoe-gw-208-70-52.toledotel.com (208.70.52.1) 1.896 ms 1.881 ms 1.903 ms 3 tuk-edge-13.inet.qwest.net (198.233.244.225) 4.158 ms 4.082 ms 4.071 ms 4 sea-brdr-03.inet.qwest.net (67.14.41.154) 8.976 ms 8.949 ms 8.903 ms 5 * * * 6 ae-1-51.ear2.Seattle1.Level3.net (4.69.203.173) 4.494 ms 4.350 ms 4.311 ms 7 4.53.154.10 (4.53.154.10) 77.622 ms 103.323 ms 103.240 ms 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 one.one.one.one (1.1.1.1) 87.515 ms * *
-A
On Fri, Jul 17, 2020 at 2:18 PM Aaron C. de Bruyn <aaron@heyaaron.com> wrote: Anyone seeing Cloudflare DNS outages or site issues?
Affecting a bunch of sites in Washington and Oregon.
-A
participants (4)
-
Aaron C. de Bruyn -
Brendan Carlson -
John Von Essen -
vidister