On Thu, Sep 15, 2011 at 4:13 PM, Steve Bohrer <skbohrer@simons-rock.edu> wrote:
On Sep 15, 2011, at 3:39 PM, Christopher Morrow wrote:
On Thu, Sep 15, 2011 at 3:34 PM, Brian Gold <bgold@simons-rock.edu> wrote:
Over the past week, we've discovered that there is an issue with the way some Verizon DSL customers are being routed in Western Massachusetts that is preventing them from reaching my employers public IPs. The problem is only limited to Verizon DSL customers, everyone else can reach these IP addresses just fine. After many hours on the phone with Verizon tech support, I finally managed to get myself and one of my coworker's home dsl connections switched from a "redback router" to a "juniper router" which resolved the issue, but only for us.
[...]
If you buy verizon services at your day job you can probably make noise through your sales droids better than here (sadly)... verizon likes to jump when customers have problems, if the customer is a large corporation or other 'important' customer.
That is just the problem! The college does not buy any Verizon network stuff directly, so we don't really have any access to their support. (We have a few cell phones, but not enough to be "important".)
Brian Gold (who first posted) happens to have their DSL to his house, and he was one of five who have reported the problem, so that gave him a slight in. But the only techs he could reach as an "end user" were not high enough up to fix this problem in a general way. After pressing them for literally hours, he was able to get transfered to their NOC, and get the problem resolved for his one address. But, they would not give him the NOC contact, and he had to repeat this multi-hour process to get it fixed for an other user. Verizon's DSL support suggested that we get our bandwith provider involved, and so they tried to pitch in, but they don't have any Verizon NOC contact either, especially since this issue is purely within a small corner of Verizon's DSL network, not on any of Verizon's links to our provider.
This issue hits only a few Verizon DSL users in NW Mass. It does not really seem like a routing problem, because the affected users can reach many of the servers in our AS, but not some addresses. Unfortunately, the "blocked" addresses include our web server and our mail server, so our staff who live out there noticed the issue pretty quickly. Traceroutes from Brian's house show that for our blocked hosts, the users don't get beyond Verizon's NAT.
I wasn't aware verizon implemented CGN already... way to be a 'first mover' in this field verizon!
The Verizon tech's "fix" of re-patching Brian's DSL line in to a different router feels to me like there is a config problem in the other router, but the tech we got is not authorized to alter the config. It would be nice if we could reach someone who could actually edit the broken config and make it right. Anyone from Verzion's NOC for Western Mass reading this? Or, does anyone else have useful contact info for them?
you probably want someone in the NOC which is (I think) stil in Reston, va... I don't think they have separate noc's per region. The first-line tech folks you chat with on the phone really arent' able (even to login really) to fix devices in the field :( anyways, this looks crappy :( but yeah for CGN being all it's cracked up to be! -chris
FWIW, Simon's Rock is 208.81.88.0/21, AS 19345. Here are a failed and a good trace from Brian's house, to different servers on our campus :
FAILS: Tracing route to wilbur.simons-rock.edu [208.81.88.15] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 192.168.10.1 2 1 ms 1 ms 1 ms 192.168.1.1 3 53 ms 104 ms 116 ms 10.14.1.1 4 * * * Request timed out. 5 * * * Request timed out. 6 * * * Request timed out. 7 * * * Request timed out.
WORKS: Tracing route to dev.simons-rock.edu [208.81.88.25] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 192.168.10.1 2 1 ms 1 ms 1 ms 192.168.1.1 3 87 ms 54 ms 54 ms 10.14.1.1 4 99 ms 109 ms 103 ms at-0-3-0-1711.WMA-CORE-RTR2.verizon-gni.net [130.81.10.77] 5 16 ms 18 ms 16 ms so-7-3-1-0.NY5030-BB-RTR2.verizon-gni.net [130.81.20.6] 6 19 ms 17 ms 17 ms 0.xe-3-1-0.BR3.NYC4.ALTER.NET [152.63.2.81] 7 18 ms 21 ms 18 ms 204.255.168.194 8 108 ms 188 ms 116 ms pos5-0-2488M.cr1.BOS1.gblx.net [67.17.94.57] 9 24 ms 28 ms 23 ms pos0-0-0-155M.ar1.BOS1.gblx.net [67.17.70.162] 10 121 ms 160 ms 127 ms 64.213.79.250 11 77 ms 77 ms 78 ms 208.81.88.25
Trace complete.
Anyways, thanks for any suggestions you can offer.
Steve Bohrer Network Administrator ITS, Bard College at Simon's Rock 413-528-7645