routing issue for verizon dsl customers in western massachusetts
Hello all, I posted this to the tech@lopsa mailing list and was advised to repost it here. If anyone can help, I would be very happy to avoid having to deal with hours more of Verizon level 1 tech support. Over the past week, we've discovered that there is an issue with the way some Verizon DSL customers are being routed in Western Massachusetts that is preventing them from reaching my employers public IPs. The problem is only limited to Verizon DSL customers, everyone else can reach these IP addresses just fine. After many hours on the phone with Verizon tech support, I finally managed to get myself and one of my coworker's home dsl connections switched from a "redback router" to a "juniper router" which resolved the issue, but only for us. I was told that everyone else in the area that is being affected by the issue have to individually call Verizon tech support, go through the same multi-hour troubleshooting steps, and if the technician is bright enough to recognize what is going on, get their issue escalated up to the central office where (in 2-4 business days) they will be switched over to the juniper router. Obviously, this is not the ideal solution. I'd really like to make the higher ups at Verizon aware of this issue and come up with a solution for all of the affected customers, but because I only have a residential account and my employer doesn't use Verizon, I've been stymied in all of my attempts so far. Does anyone here have any contacts at Verizon that I could get in touch with?
On Thu, Sep 15, 2011 at 3:34 PM, Brian Gold <bgold@simons-rock.edu> wrote:
Hello all, I posted this to the tech@lopsa mailing list and was advised to repost it here. If anyone can help, I would be very happy to avoid having to deal with hours more of Verizon level 1 tech support.
Over the past week, we've discovered that there is an issue with the way some Verizon DSL customers are being routed in Western Massachusetts that is preventing them from reaching my employers public IPs. The problem is only limited to Verizon DSL customers, everyone else can reach these IP addresses just fine. After many hours on the phone with Verizon tech support, I finally managed to get myself and one of my coworker's home dsl connections switched from a "redback router" to a "juniper router" which resolved the issue, but only for us. I was told that everyone else in the area that is being affected by the issue have to individually call Verizon tech support, go through the same multi-hour troubleshooting steps, and if the technician is bright enough to recognize what is going on, get their issue escalated up to the central office where (in 2-4 business days) they will be switched over to the juniper router. Obviously, this is not the ideal solution. I'd
actually it's not a bad solution.. if verizon is looking to lose lots of money on tech support calls... :)
really like to make the higher ups at Verizon aware of this issue and come
If you buy verizon services at your day job you can probably make noise through your sales droids better than here (sadly)... verizon likes to jump when customers have problems, if the customer is a large corporation or other 'important' customer.
up with a solution for all of the affected customers, but because I only have a residential account and my employer doesn't use Verizon, I've been stymied in all of my attempts so far. Does anyone here have any contacts at Verizon that I could get in touch with?
On Sep 15, 2011, at 3:39 PM, Christopher Morrow wrote:
On Thu, Sep 15, 2011 at 3:34 PM, Brian Gold <bgold@simons-rock.edu> wrote:
Over the past week, we've discovered that there is an issue with the way some Verizon DSL customers are being routed in Western Massachusetts that is preventing them from reaching my employers public IPs. The problem is only limited to Verizon DSL customers, everyone else can reach these IP addresses just fine. After many hours on the phone with Verizon tech support, I finally managed to get myself and one of my coworker's home dsl connections switched from a "redback router" to a "juniper router" which resolved the issue, but only for us.
[...]
If you buy verizon services at your day job you can probably make noise through your sales droids better than here (sadly)... verizon likes to jump when customers have problems, if the customer is a large corporation or other 'important' customer.
That is just the problem! The college does not buy any Verizon network stuff directly, so we don't really have any access to their support. (We have a few cell phones, but not enough to be "important".) Brian Gold (who first posted) happens to have their DSL to his house, and he was one of five who have reported the problem, so that gave him a slight in. But the only techs he could reach as an "end user" were not high enough up to fix this problem in a general way. After pressing them for literally hours, he was able to get transfered to their NOC, and get the problem resolved for his one address. But, they would not give him the NOC contact, and he had to repeat this multi- hour process to get it fixed for an other user. Verizon's DSL support suggested that we get our bandwith provider involved, and so they tried to pitch in, but they don't have any Verizon NOC contact either, especially since this issue is purely within a small corner of Verizon's DSL network, not on any of Verizon's links to our provider. This issue hits only a few Verizon DSL users in NW Mass. It does not really seem like a routing problem, because the affected users can reach many of the servers in our AS, but not some addresses. Unfortunately, the "blocked" addresses include our web server and our mail server, so our staff who live out there noticed the issue pretty quickly. Traceroutes from Brian's house show that for our blocked hosts, the users don't get beyond Verizon's NAT. The Verizon tech's "fix" of re-patching Brian's DSL line in to a different router feels to me like there is a config problem in the other router, but the tech we got is not authorized to alter the config. It would be nice if we could reach someone who could actually edit the broken config and make it right. Anyone from Verzion's NOC for Western Mass reading this? Or, does anyone else have useful contact info for them? FWIW, Simon's Rock is 208.81.88.0/21, AS 19345. Here are a failed and a good trace from Brian's house, to different servers on our campus : FAILS: Tracing route to wilbur.simons-rock.edu [208.81.88.15] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms 192.168.10.1 2 1 ms 1 ms 1 ms 192.168.1.1 3 53 ms 104 ms 116 ms 10.14.1.1 4 * * * Request timed out. 5 * * * Request timed out. 6 * * * Request timed out. 7 * * * Request timed out. WORKS: Tracing route to dev.simons-rock.edu [208.81.88.25] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms 192.168.10.1 2 1 ms 1 ms 1 ms 192.168.1.1 3 87 ms 54 ms 54 ms 10.14.1.1 4 99 ms 109 ms 103 ms at-0-3-0-1711.WMA-CORE-RTR2.verizon- gni.net [130.81.10.77] 5 16 ms 18 ms 16 ms so-7-3-1-0.NY5030-BB-RTR2.verizon- gni.net [130.81.20.6] 6 19 ms 17 ms 17 ms 0.xe-3-1-0.BR3.NYC4.ALTER.NET [152.63.2.81] 7 18 ms 21 ms 18 ms 204.255.168.194 8 108 ms 188 ms 116 ms pos5-0-2488M.cr1.BOS1.gblx.net [67.17.94.57] 9 24 ms 28 ms 23 ms pos0-0-0-155M.ar1.BOS1.gblx.net [67.17.70.162] 10 121 ms 160 ms 127 ms 64.213.79.250 11 77 ms 77 ms 78 ms 208.81.88.25 Trace complete. Anyways, thanks for any suggestions you can offer. Steve Bohrer Network Administrator ITS, Bard College at Simon's Rock 413-528-7645
On Thu, Sep 15, 2011 at 4:13 PM, Steve Bohrer <skbohrer@simons-rock.edu> wrote:
On Sep 15, 2011, at 3:39 PM, Christopher Morrow wrote:
On Thu, Sep 15, 2011 at 3:34 PM, Brian Gold <bgold@simons-rock.edu> wrote:
Over the past week, we've discovered that there is an issue with the way some Verizon DSL customers are being routed in Western Massachusetts that is preventing them from reaching my employers public IPs. The problem is only limited to Verizon DSL customers, everyone else can reach these IP addresses just fine. After many hours on the phone with Verizon tech support, I finally managed to get myself and one of my coworker's home dsl connections switched from a "redback router" to a "juniper router" which resolved the issue, but only for us.
[...]
If you buy verizon services at your day job you can probably make noise through your sales droids better than here (sadly)... verizon likes to jump when customers have problems, if the customer is a large corporation or other 'important' customer.
That is just the problem! The college does not buy any Verizon network stuff directly, so we don't really have any access to their support. (We have a few cell phones, but not enough to be "important".)
Brian Gold (who first posted) happens to have their DSL to his house, and he was one of five who have reported the problem, so that gave him a slight in. But the only techs he could reach as an "end user" were not high enough up to fix this problem in a general way. After pressing them for literally hours, he was able to get transfered to their NOC, and get the problem resolved for his one address. But, they would not give him the NOC contact, and he had to repeat this multi-hour process to get it fixed for an other user. Verizon's DSL support suggested that we get our bandwith provider involved, and so they tried to pitch in, but they don't have any Verizon NOC contact either, especially since this issue is purely within a small corner of Verizon's DSL network, not on any of Verizon's links to our provider.
This issue hits only a few Verizon DSL users in NW Mass. It does not really seem like a routing problem, because the affected users can reach many of the servers in our AS, but not some addresses. Unfortunately, the "blocked" addresses include our web server and our mail server, so our staff who live out there noticed the issue pretty quickly. Traceroutes from Brian's house show that for our blocked hosts, the users don't get beyond Verizon's NAT.
I wasn't aware verizon implemented CGN already... way to be a 'first mover' in this field verizon!
The Verizon tech's "fix" of re-patching Brian's DSL line in to a different router feels to me like there is a config problem in the other router, but the tech we got is not authorized to alter the config. It would be nice if we could reach someone who could actually edit the broken config and make it right. Anyone from Verzion's NOC for Western Mass reading this? Or, does anyone else have useful contact info for them?
you probably want someone in the NOC which is (I think) stil in Reston, va... I don't think they have separate noc's per region. The first-line tech folks you chat with on the phone really arent' able (even to login really) to fix devices in the field :( anyways, this looks crappy :( but yeah for CGN being all it's cracked up to be! -chris
FWIW, Simon's Rock is 208.81.88.0/21, AS 19345. Here are a failed and a good trace from Brian's house, to different servers on our campus :
FAILS: Tracing route to wilbur.simons-rock.edu [208.81.88.15] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 192.168.10.1 2 1 ms 1 ms 1 ms 192.168.1.1 3 53 ms 104 ms 116 ms 10.14.1.1 4 * * * Request timed out. 5 * * * Request timed out. 6 * * * Request timed out. 7 * * * Request timed out.
WORKS: Tracing route to dev.simons-rock.edu [208.81.88.25] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 192.168.10.1 2 1 ms 1 ms 1 ms 192.168.1.1 3 87 ms 54 ms 54 ms 10.14.1.1 4 99 ms 109 ms 103 ms at-0-3-0-1711.WMA-CORE-RTR2.verizon-gni.net [130.81.10.77] 5 16 ms 18 ms 16 ms so-7-3-1-0.NY5030-BB-RTR2.verizon-gni.net [130.81.20.6] 6 19 ms 17 ms 17 ms 0.xe-3-1-0.BR3.NYC4.ALTER.NET [152.63.2.81] 7 18 ms 21 ms 18 ms 204.255.168.194 8 108 ms 188 ms 116 ms pos5-0-2488M.cr1.BOS1.gblx.net [67.17.94.57] 9 24 ms 28 ms 23 ms pos0-0-0-155M.ar1.BOS1.gblx.net [67.17.70.162] 10 121 ms 160 ms 127 ms 64.213.79.250 11 77 ms 77 ms 78 ms 208.81.88.25
Trace complete.
Anyways, thanks for any suggestions you can offer.
Steve Bohrer Network Administrator ITS, Bard College at Simon's Rock 413-528-7645
On Sep 15, 2011, at 4:52 PM, Christopher Morrow wrote:
I wasn't aware verizon implemented CGN already... way to be a 'first mover' in this field verizon!
Maybe they are tying it out here in the sticks, where the glitches only hit single-digit numbers of users? (Though, I'd think if it was actually new, then they might have a higher-order tech paying attention to the glitches.) Oh well. Close enough, mostly. Steve
On Thu, Sep 15, 2011 at 20:52 UTC, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Thu, Sep 15, 2011 at 4:13 PM, Steve Bohrer <skbohrer@simons-rock.edu> wrote:
Traceroutes from Brian's house show that for our blocked hosts, the users don't get beyond Verizon's NAT.
I wasn't aware verizon implemented CGN already... way to be a 'first mover' in this field verizon!
I am betting they have not.
FAILS: Tracing route to wilbur.simons-rock.edu [208.81.88.15] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 192.168.10.1 2 1 ms 1 ms 1 ms 192.168.1.1 3 53 ms 104 ms 116 ms 10.14.1.1 4 * * * Request timed out. 5 * * * Request timed out. 6 * * * Request timed out. 7 * * * Request timed out.
Here's a trace to the same destination from a Verizon residential DSL on Maryland's Eastern Shore: Tracing route to wilbur.simons-rock.edu [208.81.88.15] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms 192.168.201.1 2 25 ms 25 ms 24 ms 10.31.8.1 3 38 ms 99 ms 78 ms at-4-3-0-1712.sal-core-rtr1.verizon-gni.net [130.81.136.122] 4 26 ms 26 ms 26 ms so-0-0-0-0.sal-core-rtr2.verizon-gni.net [130.81.18.247] 5 94 ms 31 ms 31 ms 130.81.20.238 6 32 ms 32 ms 32 ms 0.ae2.BR2.IAD8.ALTER.NET [152.63.34.73] 7 32 ms 33 ms 31 ms te2-3.ar6.DCA3.gblx.net [64.215.195.113] 8 33 ms 33 ms 32 ms xe6-2-0-10G.scr2.WDC2.gblx.net [67.16.136.197] 9 37 ms 38 ms 38 ms so2-2-0-10G.scr2.NYC1.gblx.net [67.17.95.102] 10 43 ms 44 ms 44 ms pos9-0-2488M.cr2.BOS1.gblx.net [67.17.94.157] 11 244 ms 200 ms 204 ms pos1-0-0-155M.ar1.BOS1.gblx.net [67.17.70.165] 12 50 ms 51 ms 50 ms 64.213.79.250 13 49 ms 50 ms 48 ms wilbur.simons-rock.edu [208.81.88.15] 192.168.201.1 is the router behind the bridged ADSL CPE which terminates the customer PPPoE. 10.31.8.1 is RFC 1918, but is not a NAT. I know from various "test my crappy broadband" sites that the only drain bramage on the provider side of the link is routine consumer-class port blocking (SMB networking, SQL, and of course port 80 so the mothe#@#$rs can charge extra for "business" with static IP and unblocked http). At least https works. Looking at Brian's trace above, I can't help wondering if the client is 444'd, but not due to CGN/LSN. Could both 192.168.10.1 and 192.168.1.1 be on-premises, with 192.168.1.1 terminating PPPoE? The latencies seem to confirm. It is possible it's only a single level of NAT on .1.1, with more-respectable routing by .10.1... Cheers, Dave Hart
On Thu, Sep 15, 2011 at 20:52 UTC, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Thu, Sep 15, 2011 at 4:13 PM, Steve Bohrer <skbohrer@simons-rock.edu> wrote:
Traceroutes from Brian's house show that for our blocked hosts, the users don't get beyond Verizon's NAT.
I wasn't aware verizon implemented CGN already... way to be a 'first mover' in this field verizon!
I am betting they have not.
FAILS: Tracing route to wilbur.simons-rock.edu [208.81.88.15] over a maximum of 30 hops:
 1   <1 ms   <1 ms   <1 ms  192.168.10.1  2   1 ms   1 ms   1 ms  192.168.1.1  3   53 ms  104 ms  116 ms  10.14.1.1  4   *     *     *   Request timed out.  5   *     *     *   Request timed out.  6   *     *     *   Request timed out.  7   *     *     *   Request timed out.
Here's a trace to the same destination from a Verizon residential DSL on Maryland's Eastern Shore:
Tracing route to wilbur.simons-rock.edu [208.81.88.15] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 192.168.201.1 2 25 ms 25 ms 24 ms 10.31.8.1 3 38 ms 99 ms 78 ms at-4-3-0-1712.sal-core-rtr1.verizon-gni.net [130.81.136.122] 4 26 ms 26 ms 26 ms so-0-0-0-0.sal-core-rtr2.verizon-gni.net [130.81.18.247] 5 94 ms 31 ms 31 ms 130.81.20.238 6 32 ms 32 ms 32 ms 0.ae2.BR2.IAD8.ALTER.NET [152.63.34.73] 7 32 ms 33 ms 31 ms te2-3.ar6.DCA3.gblx.net [64.215.195.113] 8 33 ms 33 ms 32 ms xe6-2-0-10G.scr2.WDC2.gblx.net [67.16.136.197] 9 37 ms 38 ms 38 ms so2-2-0-10G.scr2.NYC1.gblx.net [67.17.95.102] 10 43 ms 44 ms 44 ms pos9-0-2488M.cr2.BOS1.gblx.net [67.17.94.157] 11 244 ms 200 ms 204 ms pos1-0-0-155M.ar1.BOS1.gblx.net [67.17.70.165] 12 50 ms 51 ms 50 ms 64.213.79.250 13 49 ms 50 ms 48 ms wilbur.simons-rock.edu [208.81.88.15]
192.168.201.1 is the router behind the bridged ADSL CPE which terminates the customer PPPoE. 10.31.8.1 is RFC 1918, but is not a NAT. I know from various "test my crappy broadband" sites that the only drain bramage on the provider side of the link is routine consumer-class port blocking (SMB networking, SQL, and of course port 80 so the mothe#@#$rs can charge extra for "business" with static IP and unblocked http). At least https works.
Looking at Brian's trace above, I can't help wondering if the client is 444'd, but not due to CGN/LSN. Could both 192.168.10.1 and 192.168.1.1 be on-premises, with 192.168.1.1 terminating PPPoE? The latencies seem to confirm. It is possible it's only a single level of NAT on .1.1, with more-respectable routing by .10.1...
In my setup, 192.168.10.1 is my DD-WRT router and 192.168.1.1 is the DSL modem. When I ran a traceroute directly from the DSL modem's web interface, I got the following results: 1 57 ms 98 ms 129 ms 10.14.1.1 3 * * * Request timed out. 5 * * * Request timed out. 5 * * * Request timed out. 6 * * * Request timed out.
Cheers, Dave Hart
participants (5)
-
bgold＠simons-rock.edu -
Brian Gold
-
Christopher Morrow -
Dave Hart -
Steve Bohrer