NTT communications horrible routing, unresponsive NOC
Hi all, I've been trying to get this issue resolved for the entire day now, but NTT has been pretty unreceptive here. We're announcing a large prefix for a client across our network, and we discovered some insanely high latency. After tracking down the issue, we determined it to be something wrong with NTT at their Seattle location. We anycast this prefix, but no matter where in the world traffic is originating from, it's going to Seattle and then to Atlanta. Example: Rotterdam in the Netherlands routes from Europe -> east coast -> west coast Seattle -> los angeles -> atlanta. The gist of it is that something is seriously messed up at NTT in Seattle. We contacted our transit provider to try and carry the issue upstream, and what they told us was Sorry for delay, I've asked NTT to clear the more specific for this one as well. The problem seems to be a bug on the NTT side which keeps stale routes in the routing table for more specifics ( at random ). If you have more routes affected please notify us of the routes and I will ask them to clear the routing table for these routes. NTT is working on this with their vendor to get this resolved as soon as possible. I had spoken to a sales rep for NTT a few weeks prior, and they assured me that the NOC was top notch, and that all routes were redundant, and they guaranteed less than 50ms in the US, and all kinds of marketing. However, it looks like it's all marketing - for this entire day this router has been causing tons of issues for our clients. No-exporting it to NTT does not even solve the problem, as NTT's router in Seattle apparently just decides to keep random small prefixes in it, causing traffic to go there. At a loss as to what to do now, since their NOC isn't receptive. Anyone have someone I can contact off-list to get this issue resolved? It's especially frustrating because the problem absolutely cannot be resolved on our end, even with a no-export since NTT is keeping the routes in their router.
which route and did you look at their lookingglass during the period of time when things were/are bad to see what the LG saw? https://www.us.ntt.net/support/looking-glass/ shouting into the wind doesn't really help, information though could be helpful. On Wed, Mar 23, 2016 at 7:39 PM, Paras Jha <paras@protrafsolutions.com> wrote:
Hi all,
I've been trying to get this issue resolved for the entire day now, but NTT has been pretty unreceptive here.
We're announcing a large prefix for a client across our network, and we discovered some insanely high latency.
After tracking down the issue, we determined it to be something wrong with NTT at their Seattle location. We anycast this prefix, but no matter where in the world traffic is originating from, it's going to Seattle and then to Atlanta. Example: Rotterdam in the Netherlands routes from Europe -> east coast -> west coast Seattle -> los angeles -> atlanta. The gist of it is that something is seriously messed up at NTT in Seattle.
We contacted our transit provider to try and carry the issue upstream, and what they told us was
Sorry for delay, I've asked NTT to clear the more specific for this one as well. The problem seems to be a bug on the NTT side which keeps stale routes in the routing table for more specifics ( at random ). If you have more routes affected please notify us of the routes and I will ask them to clear the routing table for these routes. NTT is working on this with their vendor to get this resolved as soon as possible.
I had spoken to a sales rep for NTT a few weeks prior, and they assured me that the NOC was top notch, and that all routes were redundant, and they guaranteed less than 50ms in the US, and all kinds of marketing. However, it looks like it's all marketing - for this entire day this router has been causing tons of issues for our clients.
No-exporting it to NTT does not even solve the problem, as NTT's router in Seattle apparently just decides to keep random small prefixes in it, causing traffic to go there.
At a loss as to what to do now, since their NOC isn't receptive. Anyone have someone I can contact off-list to get this issue resolved? It's especially frustrating because the problem absolutely cannot be resolved on our end, even with a no-export since NTT is keeping the routes in their router.
On 23 March 2016 at 23:39, Paras Jha <paras@protrafsolutions.com> wrote:
At a loss as to what to do now, since their NOC isn't receptive. Anyone have someone I can contact off-list to get this issue resolved? It's especially frustrating because the problem absolutely cannot be resolved on our end, even with a no-export since NTT is keeping the routes in their router.
It would be helpful to show how you came to these conclusions about prefix caching and routing issues etc (share the data you gathered). If you are an NTT customer (it's sounds a bit like you're a customer of an NTT customer, it's not clear) then this should be account management 101, you should have an escalations process/matrix, kick off that process now. There are NTT dudes on this list so probably they will reach out to you off list, if not post back requesting an NTT contact. James.
Hello, Could you send me some more details in private about who you spoke with including email addresses and ticket numbers with our NOC if any? I'd like to understand what happened here as this is an uncharacteristic outcome. I know there was planned work in Seattle last night for a software upgrade combined with hardware work, but an operational issue like this should have been resolved. I'd like to understand the timeline and what broke down if anything. Thanks, Jared Mauch
On Mar 23, 2016, at 7:39 PM, Paras Jha <paras@protrafsolutions.com> wrote:
Hi all,
I've been trying to get this issue resolved for the entire day now, but NTT has been pretty unreceptive here.
We're announcing a large prefix for a client across our network, and we discovered some insanely high latency.
After tracking down the issue, we determined it to be something wrong with NTT at their Seattle location. We anycast this prefix, but no matter where in the world traffic is originating from, it's going to Seattle and then to Atlanta. Example: Rotterdam in the Netherlands routes from Europe -> east coast -> west coast Seattle -> los angeles -> atlanta. The gist of it is that something is seriously messed up at NTT in Seattle.
We contacted our transit provider to try and carry the issue upstream, and what they told us was
Sorry for delay, I've asked NTT to clear the more specific for this one as well. The problem seems to be a bug on the NTT side which keeps stale routes in the routing table for more specifics ( at random ). If you have more routes affected please notify us of the routes and I will ask them to clear the routing table for these routes. NTT is working on this with their vendor to get this resolved as soon as possible.
I had spoken to a sales rep for NTT a few weeks prior, and they assured me that the NOC was top notch, and that all routes were redundant, and they guaranteed less than 50ms in the US, and all kinds of marketing. However, it looks like it's all marketing - for this entire day this router has been causing tons of issues for our clients.
No-exporting it to NTT does not even solve the problem, as NTT's router in Seattle apparently just decides to keep random small prefixes in it, causing traffic to go there.
At a loss as to what to do now, since their NOC isn't receptive. Anyone have someone I can contact off-list to get this issue resolved? It's especially frustrating because the problem absolutely cannot be resolved on our end, even with a no-export since NTT is keeping the routes in their router.
participants (4)
-
Christopher Morrow -
James Bensley -
Jared Mauch -
Paras Jha