Re: Amazon network engineering contact? re: DDoS traffic
Zach, As mentioned before, I am open to peering (where possible) but have not received a response. My goal is to connect with someone at Amazon and work with them on a technical solution, which is why I posted asking for that here. Various factors mean that we can't just upgrade our way out of this one, and manual whac-a-mole on the sources has shown to have limited use. These attacks also impact Amazon and the networks in between the sources and targets, and they take time to handle by the abuse teams, so there are good reasons to investigate them further and find better ways to mitigate or prevent them. -John On 11/8/2018 1:12 PM, Zach Puls wrote:
Makes sense, that's understandable. Do you peer with AWS? If not, maybe opening up a peering agreement will give you a better contact, and a bit more pull when attacks occur? I know someone with a peering agreement with AWS, and they have been able to get resolutions fairly quickly when issues arise.
Other than that, I'm not sure of a solution other than more IP transit.
Thanks,
Zach Puls Network Engineer | MEF-CECP KsFiberNet
-----Original Message----- From: John Weekes <jw@nuclearfallout.net> Sent: Thursday, November 08, 2018 15:03 To: Zach Puls <zpuls@ksfiber.net> Cc: nanog@nanog.org Subject: Re: Amazon network engineering contact? re: DDoS traffic
Zach,
Yes, RTBH is used to distribute the null-routes that I mentioned.
Unfortunately, even brief saturation events lasting just 5-10 seconds (a typical amount of time to detect the loss, issue the null-route, and see the traffic start to fall off as it is distributed upstream) can cause real damage to those customers who are sensitive to latency and packet loss. So while null-routes limit the duration of the impact, they can't eliminate it entirely. And, of course, the actual target of the attack -- the now-null-routed IP address -- becomes unreachable, which was presumably the goal of the attacker.
-John
On 11/8/2018 12:54 PM, Zach Puls wrote:
No idea about an Amazon abuse contact, but do you have RTBH communities enabled with your upstream provider(s)? As a general practice, when you detect a (D)DoS attack in progress, it would help to automatically advertise that prefix to your upstream(s) with the black-hole community. This would at least help mitigate the effects of the attacks when they do occur, even if they come from a different source than AWS.
Thanks,
Zach Puls Network Engineer | MEF-CECP KsFiberNet
-----Original Message----- From: NANOG <nanog-bounces@nanog.org> On Behalf Of John Weekes Sent: Thursday, November 08, 2018 14:44 To: nanog@nanog.org Subject: Amazon network engineering contact? re: DDoS traffic
We've been seeing significant attack activity from Amazon over the last two months, involving apparently compromised instances that commonly send 1-10G of traffic per source and together generate Nx10G of total traffic. Even when our overall upstream capacity exceeds an attack's overall size, the nature of load-balancing over multiple 10G upstream links means that an individual link can be saturated by multiple large flows, forcing our systems to null-route the target to limit impact.
We've sent an abuse notification about every traffic source to Amazon, and specific sources seem to stop their involvement over time (suggesting that abuse teams are following up on them), but there is an endless parade of new attackers, and each source participates in many damaging attacks before it is shut down.
Is there anyone at Amazon who can help with an engineering solution in terms of programmatically detecting and rate-limiting attack traffic sources, to our networks or overall? Or applying the kludge of a rate-limit for all Amazon traffic to our networks? Or working with us on some other option?
At least one other large cloud provider has an automatic rate-limiting system in place that is effective in reducing the damage from repeat high-volume attacks.
Emails to the Amazon NOC, peering contacts (since that would be another possible solution), and abuse department have not connected me with anyone.
Thanks, John
Nobody should ever be forced to peer to get someone to address abusive traffic originating from networks under their control. On Thu, Nov 8, 2018 at 4:29 PM John <jw@nuclearfallout.net> wrote:
Zach,
As mentioned before, I am open to peering (where possible) but have not received a response.
My goal is to connect with someone at Amazon and work with them on a technical solution, which is why I posted asking for that here. Various factors mean that we can't just upgrade our way out of this one, and manual whac-a-mole on the sources has shown to have limited use.
These attacks also impact Amazon and the networks in between the sources and targets, and they take time to handle by the abuse teams, so there are good reasons to investigate them further and find better ways to mitigate or prevent them.
-John
Makes sense, that's understandable. Do you peer with AWS? If not, maybe opening up a peering agreement will give you a better contact, and a bit more pull when attacks occur? I know someone with a peering agreement with AWS, and they have been able to get resolutions fairly quickly when issues arise.
Other than that, I'm not sure of a solution other than more IP transit.
Thanks,
Zach Puls Network Engineer | MEF-CECP KsFiberNet
-----Original Message----- From: John Weekes <jw@nuclearfallout.net> Sent: Thursday, November 08, 2018 15:03 To: Zach Puls <zpuls@ksfiber.net> Cc: nanog@nanog.org Subject: Re: Amazon network engineering contact? re: DDoS traffic
Zach,
Yes, RTBH is used to distribute the null-routes that I mentioned.
Unfortunately, even brief saturation events lasting just 5-10 seconds (a typical amount of time to detect the loss, issue the null-route, and see
-- the now-null-routed IP address -- becomes unreachable, which was
On 11/8/2018 1:12 PM, Zach Puls wrote: the traffic start to fall off as it is distributed upstream) can cause real damage to those customers who are sensitive to latency and packet loss. So while null-routes limit the duration of the impact, they can't eliminate it entirely. And, of course, the actual target of the attack presumably the goal of the attacker.
-John
On 11/8/2018 12:54 PM, Zach Puls wrote:
No idea about an Amazon abuse contact, but do you have RTBH communities
Thanks,
Zach Puls Network Engineer | MEF-CECP KsFiberNet
-----Original Message----- From: NANOG <nanog-bounces@nanog.org> On Behalf Of John Weekes Sent: Thursday, November 08, 2018 14:44 To: nanog@nanog.org Subject: Amazon network engineering contact? re: DDoS traffic
We've been seeing significant attack activity from Amazon over the last
two months, involving apparently compromised instances that commonly send 1-10G of traffic per source and together generate Nx10G of total traffic. Even when our overall upstream capacity exceeds an attack's overall size,
We've sent an abuse notification about every traffic source to Amazon,
and specific sources seem to stop their involvement over time (suggesting
Is there anyone at Amazon who can help with an engineering solution in
terms of programmatically detecting and rate-limiting attack traffic
At least one other large cloud provider has an automatic rate-limiting
system in place that is effective in reducing the damage from repeat high-volume attacks.
Emails to the Amazon NOC, peering contacts (since that would be another
enabled with your upstream provider(s)? As a general practice, when you detect a (D)DoS attack in progress, it would help to automatically advertise that prefix to your upstream(s) with the black-hole community. This would at least help mitigate the effects of the attacks when they do occur, even if they come from a different source than AWS. the nature of load-balancing over multiple 10G upstream links means that an individual link can be saturated by multiple large flows, forcing our systems to null-route the target to limit impact. that abuse teams are following up on them), but there is an endless parade of new attackers, and each source participates in many damaging attacks before it is shut down. sources, to our networks or overall? Or applying the kludge of a rate-limit for all Amazon traffic to our networks? Or working with us on some other option? possible solution), and abuse department have not connected me with anyone.
Thanks, John
at 8:40 PM, Tom Beecher <beecher@beecher.cc> wrote:
Nobody should ever be forced to peer to get someone to address abusive traffic originating from networks under their control.
Especially considering the fact that Amazon is just a bit selective about their peers. Though, the size of our border probably had a lot to do with the fact that we didn’t get a response the first time when we requested peering for our 10s of megabits of traffic, we none the less didn’t get a response the first time we tried. Not even a “go away until you have some traffic of significance.” In the end, we had to implore a back channel to get a peering session set up.
participants (3)
-
Daniel Corbe -
John -
Tom Beecher