help with diagnosing traffic blackhole to / from select akamai ranges?
Good afternoon, we have a downstream customer originating a more specific /24 prefix, and when they do so, traffic sourced from that /24 prefix to at least a subset of akamai ranges (at minimum the 184.27.24.0/22 block at this time) are getting blackholed somewhere along the path either to or from, but i’m not sure how best to go about troubleshooting or getting assistance with diagnosing where the problem may be. from a device in the offending IP block I cannot ping or curl to www.akamai.com <http://www.akamai.com/> that resolves to 184.27.25.72, however from an IP outside the /24 specific prefix but still in their supernet range I am able to ping and get an HTTP 301 redirect response. Some other akamai prefixes like 23.73.0.0/20 seem to work without issue from what I can tell thus far. If the /24 prefix is removed, all works as expected via their covering announcement (which does return via their primary provider, as we are generally a backup path for their larger block). Any guidance the community can share as to how to go about trying to resolve I would greatly appreciate — this is the first time i’ve had to trace down a [seemingly random] reachability issue like this. Connectivity to other services seem ok from what I can gather so far, even to some other akamai ranges. from looking glass perspective it looks like the route is being accepted properly by our upstreams and other large providers like NTT, etc. I did send an email to noc@akamai.com <mailto:noc@akamai.com> but not sure that is the appropriate way to reach out for assistance or not, since we nor our downstream customer are direct customers or peers of theirs. Thanks, Will McLendon wimclend@gmail.com <mailto:wimclend@gmail.com>
It would likely be helpful if you indicate which /24 and which supernet you're talking about so people can perhaps comment if it's something around IRR, ROAs, what their views are of the block(s), etc. -- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal On Thu, Jan 9, 2020 at 10:55 AM William McLendon <wimclend@gmail.com> wrote:
Good afternoon,
we have a downstream customer originating a more specific /24 prefix, and when they do so, traffic sourced from that /24 prefix to at least a subset of akamai ranges (at minimum the 184.27.24.0/22 block at this time) are getting blackholed somewhere along the path either to or from, but i’m not sure how best to go about troubleshooting or getting assistance with diagnosing where the problem may be.
from a device in the offending IP block I cannot ping or curl to www.akamai.com that resolves to 184.27.25.72, however from an IP outside the /24 specific prefix but still in their supernet range I am able to ping and get an HTTP 301 redirect response. Some other akamai prefixes like 23.73.0.0/20 seem to work without issue from what I can tell thus far. If the /24 prefix is removed, all works as expected via their covering announcement (which does return via their primary provider, as we are generally a backup path for their larger block).
Any guidance the community can share as to how to go about trying to resolve I would greatly appreciate — this is the first time i’ve had to trace down a [seemingly random] reachability issue like this. Connectivity to other services seem ok from what I can gather so far, even to some other akamai ranges. from looking glass perspective it looks like the route is being accepted properly by our upstreams and other large providers like NTT, etc. I did send an email to noc@akamai.com but not sure that is the appropriate way to reach out for assistance or not, since we nor our downstream customer are direct customers or peers of theirs.
Thanks,
Will McLendon wimclend@gmail.com
On Thu, Jan 9, 2020 at 1:56 PM William McLendon <wimclend@gmail.com> wrote:
Good afternoon,
we have a downstream customer originating a more specific /24 prefix, and when they do so, traffic sourced from that /24 prefix to at least a subset of akamai ranges (at minimum the 184.27.24.0/22 block at this time) are getting blackholed somewhere along the path either to or from, but i’m not sure how best to go about troubleshooting or getting assistance with diagnosing where the problem may be.
from a device in the offending IP block I cannot ping or curl to www.akamai.com that resolves to 184.27.25.72, however from an IP outside the /24 specific prefix but still in their supernet range I am able to ping and get an HTTP 301 redirect response. Some other akamai prefixes like 23.73.0.0/20 seem to work without issue from what I can tell thus far. If the /24 prefix is removed, all works as expected via their covering announcement (which does return via their primary provider, as we are generally a backup path for their larger block).
The fact that when the more specific is announced through you, things change *probably* implies that the route is being accepted (and it isn't IRR filters, RPKI, etc). This sounds like it might be IP ACLs or similar, but without much more detail (like the prefix, and your AS number, etc) this is largely just shooting in the dark.... W
Any guidance the community can share as to how to go about trying to resolve I would greatly appreciate — this is the first time i’ve had to trace down a [seemingly random] reachability issue like this. Connectivity to other services seem ok from what I can gather so far, even to some other akamai ranges. from looking glass perspective it looks like the route is being accepted properly by our upstreams and other large providers like NTT, etc. I did send an email to noc@akamai.com but not sure that is the appropriate way to reach out for assistance or not, since we nor our downstream customer are direct customers or peers of theirs.
Thanks,
Will McLendon wimclend@gmail.com
-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf
thank you all for the rapid feedback and suggestions! since many have asked for more detail, the specific prefix in question is 168.8.214.0/24. it is currently being advertised; the customer just is not currently using it until we can resolve this reachability issue. As a note, our RADB irr data until Tuesday only included their supernet 168.8.208.0/21, but I have since added the more specific /24 entry as well. our AS is AS55091, and the origanating AS is AS394723 Thank you all again for feedback and offer of assistance! Thanks, Will McLendon wimclend@gmail.com <mailto:wimclend@gmail.com>
On Jan 9, 2020, at 2:12 PM, Warren Kumari <warren@kumari.net> wrote:
On Thu, Jan 9, 2020 at 1:56 PM William McLendon <wimclend@gmail.com> wrote:
Good afternoon,
we have a downstream customer originating a more specific /24 prefix, and when they do so, traffic sourced from that /24 prefix to at least a subset of akamai ranges (at minimum the 184.27.24.0/22 block at this time) are getting blackholed somewhere along the path either to or from, but i’m not sure how best to go about troubleshooting or getting assistance with diagnosing where the problem may be.
from a device in the offending IP block I cannot ping or curl to www.akamai.com that resolves to 184.27.25.72, however from an IP outside the /24 specific prefix but still in their supernet range I am able to ping and get an HTTP 301 redirect response. Some other akamai prefixes like 23.73.0.0/20 seem to work without issue from what I can tell thus far. If the /24 prefix is removed, all works as expected via their covering announcement (which does return via their primary provider, as we are generally a backup path for their larger block).
The fact that when the more specific is announced through you, things change *probably* implies that the route is being accepted (and it isn't IRR filters, RPKI, etc). This sounds like it might be IP ACLs or similar, but without much more detail (like the prefix, and your AS number, etc) this is largely just shooting in the dark....
W
Any guidance the community can share as to how to go about trying to resolve I would greatly appreciate — this is the first time i’ve had to trace down a [seemingly random] reachability issue like this. Connectivity to other services seem ok from what I can gather so far, even to some other akamai ranges. from looking glass perspective it looks like the route is being accepted properly by our upstreams and other large providers like NTT, etc. I did send an email to noc@akamai.com but not sure that is the appropriate way to reach out for assistance or not, since we nor our downstream customer are direct customers or peers of theirs.
Thanks,
Will McLendon wimclend@gmail.com
-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf
* wimclend@gmail.com (William McLendon) [Thu 09 Jan 2020, 20:18 CET]:
thank you all for the rapid feedback and suggestions! since many have asked for more detail, the specific prefix in question is 168.8.214.0/24.
The previous owner is still announcing 168.8.0.0/14. If you're shooting holes in netblocks like this you can expect routing issues. -- Niels.
participants (4)
-
Hugo Slabbert -
Niels Bakker -
Warren Kumari -
William McLendon