Any Verizon techs around today? I don't know why you can't pass DNS traffic this morning, but it's the second time in as many weeks as it has been an issue, and it's rather annoying (Google is the example, but the exact same failure happens using any destination, on VZ's own or any other public DNS servers, phone support are of course, useless): C:\Users\jamie>tracert -d 71.252.0.12 Tracing route to 71.252.0.12 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 192.168.2.254 2 <1 ms <1 ms <1 ms 192.168.1.1 3 8 ms 9 ms 13 ms 96.231.199.1 4 14 ms 9 ms 9 ms 130.81.183.118 5 9 ms 9 ms 9 ms 130.81.151.232 6 9 ms 9 ms * 130.81.20.19 7 11 ms 9 ms 9 ms 71.252.0.12 Trace complete. C:\Users\jamie>nslookup www.google.com 71.252.0.12 Server: nsrest01.verizon.net Address: 71.252.0.12 DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** Request to nsrest01.verizon.net timed-out C:\Users\jamie>tracert -d 8.8.8.8 Tracing route to 8.8.8.8 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 192.168.2.254 2 <1 ms <1 ms <1 ms 192.168.1.1 3 7 ms 8 ms 9 ms 96.231.199.1 4 8 ms 9 ms 8 ms 130.81.183.118 5 9 ms 28 ms 10 ms 130.81.22.56 6 8 ms 9 ms 9 ms 152.63.36.237 7 20 ms 19 ms 19 ms 152.63.0.153 8 21 ms 18 ms 18 ms 152.63.21.73 9 41 ms 47 ms 49 ms 152.179.72.66 10 17 ms 18 ms 19 ms 209.85.255.68 11 * * * Request timed out. 12 * * * Request timed out. 13 22 ms 19 ms 19 ms 72.14.236.200 14 20 ms 31 ms 18 ms 216.239.49.145 15 18 ms 19 ms 19 ms 8.8.8.8 Trace complete. C:\Users\jamie>nslookup www.google.com 8.8.8.8 Server: google-public-dns-a.google.com Address: 8.8.8.8 DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** Request to google-public-dns-a.google.com timed-out C:\Users\jamie>
I have FIOS and I have no issues. However I do know awhile back they had issues and I was affected by the outage.... Maybe it hasn't made its way to me yet....
From: jamie@photon.com To: nanog@nanog.org Subject: VZ FiOS DNS issues: Date: Sun, 22 Jan 2012 16:10:17 +0000
Any Verizon techs around today? I don't know why you can't pass DNS traffic this morning, but it's the second time in as many weeks as it has been an issue, and it's rather annoying (Google is the example, but the exact same failure happens using any destination, on VZ's own or any other public DNS servers, phone support are of course, useless):
C:\Users\jamie>tracert -d 71.252.0.12
Tracing route to 71.252.0.12 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 192.168.2.254 2 <1 ms <1 ms <1 ms 192.168.1.1 3 8 ms 9 ms 13 ms 96.231.199.1 4 14 ms 9 ms 9 ms 130.81.183.118 5 9 ms 9 ms 9 ms 130.81.151.232 6 9 ms 9 ms * 130.81.20.19 7 11 ms 9 ms 9 ms 71.252.0.12
Trace complete.
C:\Users\jamie>nslookup www.google.com 71.252.0.12 Server: nsrest01.verizon.net Address: 71.252.0.12
DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** Request to nsrest01.verizon.net timed-out
C:\Users\jamie>tracert -d 8.8.8.8
Tracing route to 8.8.8.8 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 192.168.2.254 2 <1 ms <1 ms <1 ms 192.168.1.1 3 7 ms 8 ms 9 ms 96.231.199.1 4 8 ms 9 ms 8 ms 130.81.183.118 5 9 ms 28 ms 10 ms 130.81.22.56 6 8 ms 9 ms 9 ms 152.63.36.237 7 20 ms 19 ms 19 ms 152.63.0.153 8 21 ms 18 ms 18 ms 152.63.21.73 9 41 ms 47 ms 49 ms 152.179.72.66 10 17 ms 18 ms 19 ms 209.85.255.68 11 * * * Request timed out. 12 * * * Request timed out. 13 22 ms 19 ms 19 ms 72.14.236.200 14 20 ms 31 ms 18 ms 216.239.49.145 15 18 ms 19 ms 19 ms 8.8.8.8
Trace complete.
C:\Users\jamie>nslookup www.google.com 8.8.8.8 Server: google-public-dns-a.google.com Address: 8.8.8.8
DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** Request to google-public-dns-a.google.com timed-out
C:\Users\jamie>
On Sun, Jan 22, 2012 at 11:29 AM, Brandon Kim <brandon.kim@brandontek.com> wrote:
I have FIOS and I have no issues. However I do know awhile back they had issues and I was affected by the outage....
Maybe it hasn't made its way to me yet....
there have been instances over the time i've been a fios customer that 'upgrades' to devices in the field have caused this problem (last was ~2wks ago? in the washington, dc area). Could be you are seeing this problem affecting you :( -chris
Christopher Morrow <morrowc.lists@gmail.com> writes:
On Sun, Jan 22, 2012 at 11:29 AM, Brandon Kim <brandon.kim@brandontek.com> wrote:
I have FIOS and I have no issues. However I do know awhile back they had issues and I was affected by the outage....
Maybe it hasn't made its way to me yet....
there have been instances over the time i've been a fios customer that 'upgrades' to devices in the field have caused this problem (last was ~2wks ago? in the washington, dc area).
Could be you are seeing this problem affecting you :(
I'm a FIOS customer (LATA 246 not 236 like Chris), and haven't had any issues with the network. On the other hand, between my location and the fact that I'm on an old BPON build, perhaps the software upgrades haven't affected me. To further complicate things, ever suspicious of ISP nameservers that don't do DNSSEC validation and monetize rcode 3, and not a fan of the Actiontec boxes that Verizon hands out I run my own cacheing nameserver (hand-built openbsd+pf on embedded hardware with latest bind or unbound and isc dhcpd). Do things magically start working for you if you hard-code 8.8.8.8 or 4.2.2.1 or one of the other usual suspects? That would seem to be a quick way of narrowing it down a bit. -r
I don't care for the Actiontec boxes either, but the STB program guides and other features don't work without it, so I have mine forward all IP traffic unmolested to my own as the DMZ host (thus the dual layer of [P|N]AT you see). It's just UDP/TCP 53 traffic that's not flowing for whatever reason; it's every device in the house phones, tablets, computers, you name it, so I'm not inclined to attribute it to malware. My neighbor was also seeing it (and like last time, it seems to have magically resolved itself after ~1.5h). I'm just wondering what Verizon is DOING that they are screwing up their own DNS traffic. If they were capturing my queries and sending them to their own servers (I actually have Google's public facing servers at the top of the list handed out by DHCP) that would be one thing (irritating to be sure, but they aren't, so it's not), but when I'm explicitly hitting a name server down the street in Reston that VZ run and it's failing the same way? It makes me wonder. Jamie
-----Original Message----- From: Robert E. Seastrom [mailto:rs@seastrom.com] Sent: Monday, January 23, 2012 6:21 AM To: Christopher Morrow Cc: nanog group Subject: Re: VZ FiOS DNS issues:
Christopher Morrow <morrowc.lists@gmail.com> writes:
On Sun, Jan 22, 2012 at 11:29 AM, Brandon Kim <brandon.kim@brandontek.com> wrote:
I have FIOS and I have no issues. However I do know awhile back they
had issues and I was affected by
the outage....
Maybe it hasn't made its way to me yet....
there have been instances over the time i've been a fios customer that 'upgrades' to devices in the field have caused this problem (last was ~2wks ago? in the washington, dc area).
Could be you are seeing this problem affecting you :(
I'm a FIOS customer (LATA 246 not 236 like Chris), and haven't had any issues with the network. On the other hand, between my location and the fact that I'm on an old BPON build, perhaps the software upgrades haven't affected me. To further complicate things, ever suspicious of ISP nameservers that don't do DNSSEC validation and monetize rcode 3, and not a fan of the Actiontec boxes that Verizon hands out I run my own cacheing nameserver (hand-built openbsd+pf on embedded hardware with latest bind or unbound and isc dhcpd).
Do things magically start working for you if you hard-code 8.8.8.8 or 4.2.2.1 or one of the other usual suspects? That would seem to be a quick way of narrowing it down a bit.
-r
Jamie Bowden <jamie@photon.com> writes:
I don't care for the Actiontec boxes either, but the STB program guides and other features don't work without it, so I have mine forward all IP traffic unmolested to my own as the DMZ host
Actually this can be worked around. My config has SA, er, Cisco STBs and a Netgear MCAB1001 MOCA to Ethernet bridge. This configuration is very unsupported, which is why I keep a completely unmolested Actiontec around to plug in if I have to have the guys at Verizon take a look at it. A little magic in dhcpd.conf: subnet 192.168.1.0 netmask 255.255.255.0 { option routers 192.168.1.1 ; option domain-name-servers 71.252.0.12 ; default-lease-time 86400 ; max-lease-time 172800 ; host stb100 { hardware ethernet 0:23:be:xx:xx:xx ; fixed-address 192.168.1.100 ; } host stb101 { hardware ethernet 0:21:be:xx:xx:xx ; fixed-address 192.168.1.101 ; } host stb102 { hardware ethernet 0:25:2e:xx:xx:xx ; fixed-address 192.168.1.102 ; } host stb103 { hardware ethernet 0:21:be:xx:xx:xx ; fixed-address 192.168.1.103 ; } } and then some appropriate holes in the firewall (/etc/pf.conf): # for STBs pass in quick on $extif inet proto tcp from any to ($extif) port 35000 rdr-to 192.168.1.100 port 7547 pass in quick on $extif inet proto tcp from any to ($extif) port 35001 rdr-to 192.168.1.101 port 7547 pass in quick on $extif inet proto udp from any to ($extif) port 63145 rdr-to 192.168.1.100 port 63145 (I only have one DVR and one STB - the definitions for extra STBs came out of the Actiontek. Not sure what I'll end up needing to do if I get another DVR or STB in order to get them properly provisioned...) Guide and VOD work fine. I don't feel like playing stuff from a PC on the STBs badly enough to be willing to cram my whole life into a flat 192.168.1/24, so I give those up. I've often wondered whether the boxes care about double-hopped NAT. Perhaps one of these days I'll try putting the Actiontek and some new pf.conf rules in place of the Netgear and give that a try.
(thus the dual layer of [P|N]AT you see). It's just UDP/TCP 53 traffic that's not flowing for whatever reason; it's every device in the house phones, tablets, computers, you name it, so I'm not inclined to attribute it to malware. My neighbor was also seeing it (and like last time, it seems to have magically resolved itself after ~1.5h). I'm just wondering what Verizon is DOING that they are screwing up their own DNS traffic. If they were capturing my queries and sending them to their own servers (I actually have Google's public facing servers at the top of the list handed out by DHCP) that would be one thing (irritating to be sure, but they aren't, so it's not), but when I'm explicitly hitting a name server down the street in Reston that VZ run and it's failing the same way? It makes me wonder.
No idea, just a datapoint that we're Not Seeing That Here... but if it is failing on google's public dns servers that's troubling to say the least. -r
On Jan 22, 2012, at 8:11 AM, "Jamie Bowden" <jamie@photon.com> wrote:
Any Verizon techs around today? I don't know why you can't pass DNS traffic this morning, but it's the second time in as many weeks as it has been an issue, and it's rather annoying (Google is the example, but the exact same failure happens using any destination, on VZ's own or any other public DNS servers, phone support are of course, useless):
Have a look at: http://forums.verizon.com/t5/FiOS-Internet/DNS-issues-in-SoCal/td-p/393781/p... Are you by chance in So Cal? VZ has been having some serious pot holes on their information super highway of late. Regards, James Laszko Mythos Technology Inc
C:\Users\jamie>tracert -d 71.252.0.12
Tracing route to 71.252.0.12 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 192.168.2.254 2 <1 ms <1 ms <1 ms 192.168.1.1 3 8 ms 9 ms 13 ms 96.231.199.1 4 14 ms 9 ms 9 ms 130.81.183.118 5 9 ms 9 ms 9 ms 130.81.151.232 6 9 ms 9 ms * 130.81.20.19 7 11 ms 9 ms 9 ms 71.252.0.12
Trace complete.
C:\Users\jamie>nslookup www.google.com 71.252.0.12 Server: nsrest01.verizon.net Address: 71.252.0.12
DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** Request to nsrest01.verizon.net timed-out
C:\Users\jamie>tracert -d 8.8.8.8
Tracing route to 8.8.8.8 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 192.168.2.254 2 <1 ms <1 ms <1 ms 192.168.1.1 3 7 ms 8 ms 9 ms 96.231.199.1 4 8 ms 9 ms 8 ms 130.81.183.118 5 9 ms 28 ms 10 ms 130.81.22.56 6 8 ms 9 ms 9 ms 152.63.36.237 7 20 ms 19 ms 19 ms 152.63.0.153 8 21 ms 18 ms 18 ms 152.63.21.73 9 41 ms 47 ms 49 ms 152.179.72.66 10 17 ms 18 ms 19 ms 209.85.255.68 11 * * * Request timed out. 12 * * * Request timed out. 13 22 ms 19 ms 19 ms 72.14.236.200 14 20 ms 31 ms 18 ms 216.239.49.145 15 18 ms 19 ms 19 ms 8.8.8.8
Trace complete.
C:\Users\jamie>nslookup www.google.com 8.8.8.8 Server: google-public-dns-a.google.com Address: 8.8.8.8
DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** Request to google-public-dns-a.google.com timed-out
C:\Users\jamie>
Try a full rebind on your cpe or power cycle, whichever is easier. This seems to have worked for a few on the forums. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. James Laszko <jamesl@mythostech.com> wrote: On Jan 22, 2012, at 8:11 AM, "Jamie Bowden" <jamie@photon.com> wrote:
Any Verizon techs around today? I don't know why you can't pass DNS traffic this morning, but it's the second time in as many weeks as it has been an issue, and it's rather annoying (Google is the example, but the exact same failure happens using any destination, on VZ's own or any other public DNS servers, phone support are of course, useless):
Have a look at: http://forums.verizon.com/t5/FiOS-Internet/DNS-issues-in-SoCal/td-p/393781/p... Are you by chance in So Cal? VZ has been having some serious pot holes on their information super highway of late. Regards, James Laszko Mythos Technology Inc
C:\Users\jamie>tracert -d 71.252.0.12
Tracing route to 71.252.0.12 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 192.168.2.254 2 <1 ms <1 ms <1 ms 192.168.1.1 3 8 ms 9 ms 13 ms 96.231.199.1 4 14 ms 9 ms 9 ms 130.81.183.118 5 9 ms 9 ms 9 ms 130.81.151.232 6 9 ms 9 ms * 130.81.20.19 7 11 ms 9 ms 9 ms 71.252.0.12
Trace complete.
C:\Users\jamie>nslookup www.google.com 71.252.0.12 Server: nsrest01.verizon.net Address: 71.252.0.12
DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** Request to nsrest01.verizon.net timed-out
C:\Users\jamie>tracert -d 8.8.8.8
Tracing route to 8.8.8.8 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 192.168.2.254 2 <1 ms <1 ms <1 ms 192.168.1.1 3 7 ms 8 ms 9 ms 96.231.199.1 4 8 ms 9 ms 8 ms 130.81.183.118 5 9 ms 28 ms 10 ms 130.81.22.56 6 8 ms 9 ms 9 ms 152.63.36.237 7 20 ms 19 ms 19 ms 152.63.0.153 8 21 ms 18 ms 18 ms 152.63.21.73 9 41 ms 47 ms 49 ms 152.179.72.66 10 17 ms 18 ms 19 ms 209.85.255.68 11 * * * Request timed out. 12 * * * Request timed out. 13 22 ms 19 ms 19 ms 72.14.236.200 14 20 ms 31 ms 18 ms 216.239.49.145 15 18 ms 19 ms 19 ms 8.8.8.8
Trace complete.
C:\Users\jamie>nslookup www.google.com 8.8.8.8 Server: google-public-dns-a.google.com Address: 8.8.8.8
DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** Request to google-public-dns-a.google.com timed-out
C:\Users\jamie>
participants (6)
-
Brandon Kim
-
Christopher Morrow
-
James Laszko
-
Jamie Bowden
-
Joseph Snyder
-
Robert E. Seastrom