Large DDoS, small extortion
Apologies for the non-personal email address, but I don't want to give our attacker any additional information than I need to. I'd be happy to send personal contact/ASN information to any nanog admins or regular members of nanog if it's useful. Over the past year or so, we (a decent sized tier 2 with a nationwide US backbone) have had several large DDoS attacks from what appear to be the same person who is (we presume) going down something like the alexa list of top sites, attacking them, and asking for small amounts of money to stop. This has been going on for a long time -- almost every detail is exactly the same as what is described here: http://it.slashdot.org/story/12/11/03/1846252/ask-slashdot-how-to-deal-with-... and more recently: http://techcrunch.com/2014/03/03/meetup-suffering-significant-ddos-attack-ta... and: https://gist.github.com/dhh/9741477 And I believe attacks including vimeo, github, and others. The attacker is smarter than many random attackers, or at least has better tools. He watches when you mitigate the attack, and shifts his attack to something new. He (or his tools) also watch DNS for the thing he's attacking and the attack moves as DNS changes. We've seen UDP amplification (NTP and DNS mainly), syn flood, syn/ack flood, layer 7 cache busting (https://isc.sans.edu/forums/diary/Wordpress+Pingback+DDoS+Attacks/17801/), and others we haven't been able to fully mitigate/identify. The largest we've seen (which isn't the largest we've read about) attacks are over 50Gbit and 10s of millions of pps. He is in regular communication (via whois info and other collected contact data) asking for <$1000 USD sums to stop the attacks. While we are interested in technical means to mitigate the attacks (the syn and syn/acks are brutal, all cores pegged on multicore 10G nic servers just dealing with interrupts), what I'd really like to find out is how to help fix the problem. We've tried to engage upstream providers to help trace the attacks, but have gotten nowhere (they didn't seem to understand that the syn attacks were spoofed, and looking at source IPs didn't matter, we wanted to know the ingress points on their network.) What are the best practices for this? Are there secret code words (http://xkcd.com/806/) we can use to get to someone at our upstreams who might know what we're talking about? Is it worth the time? Is it worth talking to law enforcement? Some of these have been >500k costs to the customer, but we assume the person doing it isn't in any western country, so maybe it doesn't even matter? Thanks.
On May 22, 2014, at 12:51 AM, Beleaguered Admin <dealing.with.ddos@gmail.com> wrote:
Apologies for the non-personal email address, but I don't want to give our attacker any additional information than I need to.
I'd be happy to send personal contact/ASN information to any nanog admins or regular members of nanog if it's useful.
We've tried to engage upstream providers to help trace the attacks, but have gotten nowhere (they didn't seem to understand that the syn attacks were spoofed, and looking at source IPs didn't matter, we wanted to know the ingress points on their network.)
this sounds like a tooling issue on their part. they should be able to pick a specific set of items and trace them back and mitigate some set of spoofed packets. Some attackers are advanced and will detect when you block their spoofed packets immediately (they have telemetry/data like we all do) and move to another attack vector.
What are the best practices for this? Are there secret code words (http://xkcd.com/806/) we can use to get to someone at our upstreams who might know what we're talking about? Is it worth the time?
You need to talk to the security team in their NOC. These are usually small and sometimes difficult to reach. I know our NOC can find them quickly and works with them on customer issues often.
Is it worth talking to law enforcement?
Absolutely. Even if the "lost costs" have been just payroll which already exist, this may be related to other activity. I suggest calling your local FBI office (assuming you are in the US). They can be quite helpful. If you don't get somewhere quickly, let me know and I can try to hunt someone in a local field office for you.
Some of these have been >500k costs to the customer, but we assume the person doing it isn't in any western country, so maybe it doesn't even matter?
I'll say it does matter, because even if they are in some "unreachable" location, these folks sometimes travel to locations where they can be picked up. It may not be immediate, but can help build the case. It is sad, but I can likely guess who your upstreams are, and some are more responsive than others. I'm aware of one that puts almost no effort into tracking spoofed packets to clamp down on them. - Jared
On May 22, 2014, at 11:51 AM, Beleaguered Admin <dealing.with.ddos@gmail.com> wrote:
While we are interested in technical means to mitigate the attacks (the syn and syn/acks are brutal, all cores pegged on multicore 10G nic servers just dealing with interrupts),
Here's how to get started: <http://mailman.nanog.org/pipermail/nanog/2010-January/016747.html> Ensure you have flow telemetry enabled at all your edges; there are open-source tools like nfsen/nfdump that you can get started with quickly. ---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Equo ne credite, Teucri. -- Laocoön
On 5/22/14, 12:51 AM, "Beleaguered Admin" <dealing.with.ddos@gmail.com> wrote:
This has been going on for a long time -- almost every detail is exactly the same as what is described here: http://techcrunch.com/2014/03/03/meetup-suffering-significant-ddos-attack- taking-it-offline-for-days/
He is in regular communication (via whois info and other collected contact data) asking for <$1000 USD sums to stop the attacks.
That article said that the company didn¹t want to negotiate with criminals. As an aside I spent some time with a retired hostage negotiator on Tuesday (which was fascinating BTW). He actually said negotiation is always useful and sometimes paying a ransom demand can serve as a method to track where the money goes, to identify all the actors involved for later action (which may apply in this case). And sometimes financial demands are dropped as a result of negotiation.
Is it worth talking to law enforcement? Some of these have been >500k costs to the customer, but we assume the person doing it isn't in any western country, so maybe it doesn't even matter?
You may find the law enforcement more interested in engaging within you than you might think. Jason
On May 22, 2014, at 10:23 PM, Livingood, Jason <Jason_Livingood@cable.comcast.com> wrote:
He actually said negotiation is always useful and sometimes paying a ransom demand can serve as a method to track where the money goes, to identify all the actors involved for later action (which may apply in this case).
Bad advice for online stuff, as a) it's very, very rare that the perpetrators are caught, and b) word will get around that you're an easy mark - so, more attacks, more (and more expensive) extortion. *Never* pay extortion money to a DDoSer. ---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Equo ne credite, Teucri. -- Laocoön
Never pay extortion anyways. After you pay once, you¹ll pay again. If I were you, I¹d pay someone a few bucks to pull this kids dox and drop them on pastebin. Stainless Steel Testicles turn to itty bitty testicles when your name and phone number are sitting on the internet. Or just write back to him and tell him that if you don¹t stop being packeted, you¹re going to take his mother to a nice seafood dinner and NEVER call her again. What¹s he going to do, packet you? ;) On 5/22/14, 8:50 AM, "Roland Dobbins" <rdobbins@arbor.net> wrote:
On May 22, 2014, at 10:23 PM, Livingood, Jason <Jason_Livingood@cable.comcast.com> wrote:
He actually said negotiation is always useful and sometimes paying a ransom demand can serve as a method to track where the money goes, to identify all the actors involved for later action (which may apply in this case).
Bad advice for online stuff, as a) it's very, very rare that the perpetrators are caught, and b) word will get around that you're an easy mark - so, more attacks, more (and more expensive) extortion.
*Never* pay extortion money to a DDoSer.
---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Equo ne credite, Teucri.
-- Laocoön
On 5/22/2014 10:59 AM, Warren Bailey wrote:
Never pay extortion anyways. After you pay once, you¹ll pay again.
If I were you, I¹d pay someone a few bucks to pull this kids dox and drop them on pastebin. Stainless Steel Testicles turn to itty bitty testicles when your name and phone number are sitting on the internet. Or just write back to him and tell him that if you don¹t stop being packeted, you¹re going to take his mother to a nice seafood dinner and NEVER call her again. What¹s he going to do, packet you? ;)
World could use more opts like this.
negotiation is fine… a weakness is presuming to know what the perp wants (and many times they don;t know themselves) so engagement is good “The Cuckoo's Egg” is worth the read… /bill On 22May2014Thursday, at 8:23, Livingood, Jason <Jason_Livingood@cable.comcast.com> wrote:
On 5/22/14, 12:51 AM, "Beleaguered Admin" <dealing.with.ddos@gmail.com> wrote:
This has been going on for a long time -- almost every detail is exactly the same as what is described here: http://techcrunch.com/2014/03/03/meetup-suffering-significant-ddos-attack- taking-it-offline-for-days/
He is in regular communication (via whois info and other collected contact data) asking for <$1000 USD sums to stop the attacks.
That article said that the company didn¹t want to negotiate with criminals. As an aside I spent some time with a retired hostage negotiator on Tuesday (which was fascinating BTW). He actually said negotiation is always useful and sometimes paying a ransom demand can serve as a method to track where the money goes, to identify all the actors involved for later action (which may apply in this case). And sometimes financial demands are dropped as a result of negotiation.
Is it worth talking to law enforcement? Some of these have been >500k costs to the customer, but we assume the person doing it isn't in any western country, so maybe it doesn't even matter?
You may find the law enforcement more interested in engaging within you than you might think.
Jason
I could attribute a fair number of misdeeds to that book. ;) On 5/22/14, 10:22 AM, "manning" <bmanning@karoshi.com> wrote:
negotiation is fine… a weakness is presuming to know what the perp wants (and many times they don;t know themselves) so engagement is good “The Cuckoo's Egg” is worth the read…
/bill
On 22May2014Thursday, at 8:23, Livingood, Jason <Jason_Livingood@cable.comcast.com> wrote:
On 5/22/14, 12:51 AM, "Beleaguered Admin" <dealing.with.ddos@gmail.com> wrote:
This has been going on for a long time -- almost every detail is exactly the same as what is described here:
http://techcrunch.com/2014/03/03/meetup-suffering-significant-ddos-attac k- taking-it-offline-for-days/
He is in regular communication (via whois info and other collected contact data) asking for <$1000 USD sums to stop the attacks.
That article said that the company didn¹t want to negotiate with criminals. As an aside I spent some time with a retired hostage negotiator on Tuesday (which was fascinating BTW). He actually said negotiation is always useful and sometimes paying a ransom demand can serve as a method to track where the money goes, to identify all the actors involved for later action (which may apply in this case). And sometimes financial demands are dropped as a result of negotiation.
Is it worth talking to law enforcement? Some of these have been >500k costs to the customer, but we assume the person doing it isn't in any western country, so maybe it doesn't even matter?
You may find the law enforcement more interested in engaging within you than you might think.
Jason
Contact law enforcement -- they can combine intel from multiple cases to hopefully identify the attacker. Automate your analysis and reporting. If you send an email to the sources of abuse you can reduce the attacker's capabilities. (To set expectations: only about 10% will take action.) If you have specific customers that are being targeted, you may want to suggest they get behind a DDoS mitigation provider that can absorb large attacks (up to 500Gbps). Damian On Wed, May 21, 2014 at 9:51 PM, Beleaguered Admin < dealing.with.ddos@gmail.com> wrote:
Apologies for the non-personal email address, but I don't want to give our attacker any additional information than I need to.
I'd be happy to send personal contact/ASN information to any nanog admins or regular members of nanog if it's useful.
Over the past year or so, we (a decent sized tier 2 with a nationwide US backbone) have had several large DDoS attacks from what appear to be the same person who is (we presume) going down something like the alexa list of top sites, attacking them, and asking for small amounts of money to stop.
This has been going on for a long time -- almost every detail is exactly the same as what is described here:
http://it.slashdot.org/story/12/11/03/1846252/ask-slashdot-how-to-deal-with-...
and more recently:
http://techcrunch.com/2014/03/03/meetup-suffering-significant-ddos-attack-ta...
and:
https://gist.github.com/dhh/9741477
And I believe attacks including vimeo, github, and others.
The attacker is smarter than many random attackers, or at least has better tools. He watches when you mitigate the attack, and shifts his attack to something new. He (or his tools) also watch DNS for the thing he's attacking and the attack moves as DNS changes.
We've seen UDP amplification (NTP and DNS mainly), syn flood, syn/ack flood, layer 7 cache busting (https://isc.sans.edu/forums/diary/Wordpress+Pingback+DDoS+Attacks/17801/ ), and others we haven't been able to fully mitigate/identify.
The largest we've seen (which isn't the largest we've read about) attacks are over 50Gbit and 10s of millions of pps.
He is in regular communication (via whois info and other collected contact data) asking for <$1000 USD sums to stop the attacks.
While we are interested in technical means to mitigate the attacks (the syn and syn/acks are brutal, all cores pegged on multicore 10G nic servers just dealing with interrupts), what I'd really like to find out is how to help fix the problem.
We've tried to engage upstream providers to help trace the attacks, but have gotten nowhere (they didn't seem to understand that the syn attacks were spoofed, and looking at source IPs didn't matter, we wanted to know the ingress points on their network.)
What are the best practices for this? Are there secret code words (http://xkcd.com/806/) we can use to get to someone at our upstreams who might know what we're talking about? Is it worth the time?
Is it worth talking to law enforcement? Some of these have been >500k costs to the customer, but we assume the person doing it isn't in any western country, so maybe it doesn't even matter?
Thanks.
You know what would be nice? Some real life experience and results, case studies. I see the "common sense" and "logic" to a lot of these suggestions but that and $1.75 plus tax will get you a venti coffee of the day at Starbucks. Victim: I'd be very wary of these suggestions unless there's some good, solid reason to believe they're based on reality not just "I've simulated all of human psychology in my head and here's what I think you should do..." I think it's interesting that the guy asks for such small amounts, under US$1000. Maybe that's a lot of money for him. Maybe he thinks it won't be worth investigating such a small amount. Maybe he thinks it's not a very big crime so if he gets caught he's more likely to walk. Maybe he thinks he's poor/broke and this money is deservedly his to demand, it's such a modest demand. Note: He could be factually/legally wrong but that's why I prefaced with "maybe he thinks..." Maybe he's a sadist and gets a kick out of making you squirm and the money is just his way of keeping score, making you do something tangible, kind of like "kiss my boots!" Maybe he's insane which voids all of the above. Maybe it's some sort of penetration exercise by terrorists, a govt, etc. Maybe all I've said and $1.75 plus tax... -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*
I will use this opportunity to solicit real world experience and use cases that could be discussed at the Security Track at NANOG 61. While I've been soliciting talks in operational security specific groups, this thread also peaked my interest. Nothing beats sharing the good, the bad, the ugly and how collectively we can improve on how we mitigate against varying attacks. Please respond to me in unicast and let me know if you'd be willing to share some experiences. The Security Track is not recorded nor streamed and you do not need a formal presentation. - merike On May 22, 2014, at 1:38 PM, Barry Shein <bzs@world.std.com> wrote:
You know what would be nice? Some real life experience and results, case studies.
I see the "common sense" and "logic" to a lot of these suggestions but that and $1.75 plus tax will get you a venti coffee of the day at Starbucks.
Victim: I'd be very wary of these suggestions unless there's some good, solid reason to believe they're based on reality not just "I've simulated all of human psychology in my head and here's what I think you should do..."
I think it's interesting that the guy asks for such small amounts, under US$1000.
Maybe that's a lot of money for him.
Maybe he thinks it won't be worth investigating such a small amount.
Maybe he thinks it's not a very big crime so if he gets caught he's more likely to walk.
Maybe he thinks he's poor/broke and this money is deservedly his to demand, it's such a modest demand.
Note: He could be factually/legally wrong but that's why I prefaced with "maybe he thinks..."
Maybe he's a sadist and gets a kick out of making you squirm and the money is just his way of keeping score, making you do something tangible, kind of like "kiss my boots!"
Maybe he's insane which voids all of the above.
Maybe it's some sort of penetration exercise by terrorists, a govt, etc.
Maybe all I've said and $1.75 plus tax...
-- -Barry Shein
The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*
On May 23, 2014, at 3:38 AM, Barry Shein <bzs@world.std.com> wrote:
Some real life experience and results, case studies.
Some of us have quite a bit of real-life experience and results in these situations. ---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Equo ne credite, Teucri. -- Laocoön
Most of us wish we didn't. There are so much more productive ways to spend the day than fighting a determined and adaptive attacker. -Blake On Thu, May 22, 2014 at 10:20 PM, Roland Dobbins <rdobbins@arbor.net> wrote:
On May 23, 2014, at 3:38 AM, Barry Shein <bzs@world.std.com> wrote:
Some real life experience and results, case studies.
Some of us have quite a bit of real-life experience and results in these situations.
---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Equo ne credite, Teucri.
-- Laocoön
On May 23, 2014, at 11:22 AM, Blake Dunlap <ikiris@gmail.com> wrote:
Most of us wish we didn't.
Concur 100%. ---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Equo ne credite, Teucri. -- Laocoön
Sure, of course, many of us have. But how is $VICTIM supposed to distinguish the wheat from the chaff without reference to specific cases and results? Some reasonable-sounding suggestions could be counter-productive or even downright dangerous (depending on the nature of the attacker.) Or a waste of time. On May 22, 2014 at 23:22 ikiris@gmail.com (Blake Dunlap) wrote:
Most of us wish we didn't. There are so much more productive ways to spend the day than fighting a determined and adaptive attacker.
-Blake
On Thu, May 22, 2014 at 10:20 PM, Roland Dobbins <rdobbins@arbor.net> wrote:
On May 23, 2014, at 3:38 AM, Barry Shein <bzs@world.std.com> wrote:
Some real life experience and results, case studies.
Some of us have quite a bit of real-life experience and results in these situations.
---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Equo ne credite, Teucri.
-- Laocoön
-- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*
On May 24, 2014, at 12:13 AM, Barry Shein <bzs@world.std.com> wrote:
Some reasonable-sounding suggestions could be counter-productive or even downright dangerous (depending on the nature of the attacker.) Or a waste of time.
Sure. Every circumstance is different. But there is *one* universal rule Never pay. Never, under any circumstances, pay. Not even if you've persuaded the Men from U.N.C.L.E. to help you, and they suggest you pay because they think they can trace the money, do not pay. Why not? Because, irrespective of what happens with this one attacker, you will be swarmed by countless others. Attackers brag when they're paid; they'll exaggerate how much they received, and then you have a much bigger problem. So, yes - one's own experiences and what one did and how one did it and why one did it and how it turned out are very valuable to share. But never, under any circumstances, for any reason, no matter who advises you to do so, should you pay. ---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Equo ne credite, Teucri. -- Laocoön
On May 24, 2014 at 00:38 rdobbins@arbor.net (Roland Dobbins) wrote:
Never, under any circumstances, pay. Not even if you've persuaded the Men from U.N.C.L.E. to help you, and they suggest you pay because they think they can trace the money, do not pay.
Ok, you're recommending $VICTIM ignores or resists the advice of law enforcement authorities, right? What is this based on other than your subsequent "common sense" reasoning? (directly below)
Why not?
Because, irrespective of what happens with this one attacker, you will be swarmed by countless others. Attackers brag when they're paid; they'll exaggerate how much they received, and then you have a much bigger problem.
By "irrespective of what happens" do you include your earlier suggestion that the attacker might be traced and arrested? Tracing the money in extortion schemes is a common tactic. Obviously the likelihood of success has to be evaluated. But a lot of criminals are dumb or perhaps put better naive. DDos'ing is one thing, successfully laundering money is a different skill set. I just don't know and would suggest reliance on case studies and experienced professionals. -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*
On May 24, 2014, at 1:09 AM, Barry Shein <bzs@world.std.com> wrote:
What is this based on other than your subsequent "common sense" reasoning? (directly below)
I've been involved in helping people who've paid. It didn't turn out well (obviously, or they wouldn't need help, heh).
By "irrespective of what happens" do you include your earlier suggestion that the attacker might be traced and arrested?
Yes - it doesn't matter even if attacker #1 is traced and arrested (which doesn't happen often, and takes lots and lots of time), if you're now busy dealing with attackers #2 - N. I've never, ever heard of an LEO recommending that someone pay in these particular circumstances - i.e., DDoS extortion. I doubt one ever would. But if one did, I personally wouldn't follow that particular recommendation, and would urge others to seriously think before doing it. ---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Equo ne credite, Teucri. -- Laocoön
On Fri, May 23, 2014 at 02:09:18PM -0400, Barry Shein wrote:
I just don't know and would suggest reliance on case studies and experienced professionals.
Well, yes, but I also observe that LE's interests and your own as the operator of the site diverge, because their risk isn't the same as yours. It's worth keeping that in one's calculus. A -- Andrew Sullivan Dyn, Inc. asullivan@dyn.com v: +1 603 663 0448
On May 23, 2014 at 15:19 asullivan@dyn.com (Andrew Sullivan) wrote:
On Fri, May 23, 2014 at 02:09:18PM -0400, Barry Shein wrote:
I just don't know and would suggest reliance on case studies and experienced professionals.
Well, yes, but I also observe that LE's interests and your own as the operator of the site diverge, because their risk isn't the same as yours. It's worth keeping that in one's calculus.
Good point. There is the danger of "the operation was a success but the patient died" (i.e., they caught the perp but destroyed your business in the process.) -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*
On Fri, May 23, 2014 at 02:09:18PM -0400, Barry Shein wrote:
On May 24, 2014 at 00:38 rdobbins@arbor.net (Roland Dobbins) wrote:
Never, under any circumstances, pay. Not even if you've persuaded the Men from U.N.C.L.E. to help you, and they suggest you pay because they think they can trace the money, do not pay.
Ok, you're recommending $VICTIM ignores or resists the advice of law enforcement authorities, right?
Law enforcement and victims have different objectives. Law enforcement wants to find the criminal, gather sufficient evidence to prove their guilt, then prosecute them. More attacks helps law enforcement. The victims, in general, want the attacks to stop.
I just don't know and would suggest reliance on case studies and experienced professionals.
Agreed. But make sure the experienced professionals you talk with have interests that are aligned with yours. (Not arguing pay or don't pay here. I don't know, either. My instincts say "don't pay" but I have no data.) -- Brett
Thanks everyone. There's been a lot of great on and off list responses, and we have a much better list of contacts for the next time this happens. We are in contact with the FBI now (very impressed, particularly compared to what I expected), and have access to resources that we didn't know existed. Hopefully I'll meet some of you in bellevue next week. On Wed, May 21, 2014 at 9:51 PM, Beleaguered Admin <dealing.with.ddos@gmail.com> wrote:
Apologies for the non-personal email address, but I don't want to give our attacker any additional information than I need to.
I'd be happy to send personal contact/ASN information to any nanog admins or regular members of nanog if it's useful.
Over the past year or so, we (a decent sized tier 2 with a nationwide US backbone) have had several large DDoS attacks from what appear to be the same person who is (we presume) going down something like the alexa list of top sites, attacking them, and asking for small amounts of money to stop.
This has been going on for a long time -- almost every detail is exactly the same as what is described here:
http://it.slashdot.org/story/12/11/03/1846252/ask-slashdot-how-to-deal-with-...
and more recently:
http://techcrunch.com/2014/03/03/meetup-suffering-significant-ddos-attack-ta...
and:
https://gist.github.com/dhh/9741477
And I believe attacks including vimeo, github, and others.
The attacker is smarter than many random attackers, or at least has better tools. He watches when you mitigate the attack, and shifts his attack to something new. He (or his tools) also watch DNS for the thing he's attacking and the attack moves as DNS changes.
We've seen UDP amplification (NTP and DNS mainly), syn flood, syn/ack flood, layer 7 cache busting (https://isc.sans.edu/forums/diary/Wordpress+Pingback+DDoS+Attacks/17801/), and others we haven't been able to fully mitigate/identify.
The largest we've seen (which isn't the largest we've read about) attacks are over 50Gbit and 10s of millions of pps.
He is in regular communication (via whois info and other collected contact data) asking for <$1000 USD sums to stop the attacks.
While we are interested in technical means to mitigate the attacks (the syn and syn/acks are brutal, all cores pegged on multicore 10G nic servers just dealing with interrupts), what I'd really like to find out is how to help fix the problem.
We've tried to engage upstream providers to help trace the attacks, but have gotten nowhere (they didn't seem to understand that the syn attacks were spoofed, and looking at source IPs didn't matter, we wanted to know the ingress points on their network.)
What are the best practices for this? Are there secret code words (http://xkcd.com/806/) we can use to get to someone at our upstreams who might know what we're talking about? Is it worth the time?
Is it worth talking to law enforcement? Some of these have been >500k costs to the customer, but we assume the person doing it isn't in any western country, so maybe it doesn't even matter?
Thanks.
Thanks everyone. There's been a lot of great on and off list responses, and we have a much better list of contacts for the next time this happens.
We are in contact with the FBI now (very impressed, particularly compared to what I expected), and have access to resources that we didn't know existed.
Hopefully I'll meet some of you in bellevue next week.
seeing as the web version of the security track is content free, it would be cool if you held a little open chat. randy
On Fri, May 23, 2014 at 1:24 AM, Randy Bush <randy@psg.com> wrote:
Thanks everyone. There's been a lot of great on and off list responses, and we have a much better list of contacts for the next time this happens.
We are in contact with the FBI now (very impressed, particularly compared to what I expected), and have access to resources that we didn't know existed.
Hopefully I'll meet some of you in bellevue next week.
seeing as the web version of the security track is content free, it would be cool if you held a little open chat.
randy
I replied back privately, as the specific details I had to share weren't really best aired in an archived, public forum. I suspect others might be in a similar position, unfortunately. I can see why the decision to not webcast the security forum is a good one. Matt
On May 23, 2014, at 3:03 AM, Matthew Petach <mpetach@netflight.com> wrote:
On Fri, May 23, 2014 at 1:24 AM, Randy Bush <randy@psg.com> wrote:
Thanks everyone. There's been a lot of great on and off list responses, and we have a much better list of contacts for the next time this happens.
We are in contact with the FBI now (very impressed, particularly compared to what I expected), and have access to resources that we didn't know existed.
Hopefully I'll meet some of you in bellevue next week.
seeing as the web version of the security track is content free, it would be cool if you held a little open chat.
randy
I replied back privately, as the specific details I had to share weren't really best aired in an archived, public forum. I suspect others might be in a similar position, unfortunately. I can see why the decision to not webcast the security forum is a good one.
Whoop I may have mis-read Randy's first comment to mean there's no definitive agenda listed yet. But yes, definitely not streamed or recorded since we do want to have some open discussion on items that people don't want attributed nor shared in a public way. I've already had at least one specific request where speaker wants to give some details that they don't want to be attributed. - merike
On May 23, 2014, at 1:24 AM, Randy Bush <randy@psg.com> wrote:
Thanks everyone. There's been a lot of great on and off list responses, and we have a much better list of contacts for the next time this happens.
We are in contact with the FBI now (very impressed, particularly compared to what I expected), and have access to resources that we didn't know existed.
Hopefully I'll meet some of you in bellevue next week.
seeing as the web version of the security track is content free, it would be cool if you held a little open chat.
Am in discussions with folks who replied to my solicitation yesterday (thanks to those who did) and as soon as topics locked the content free aspect will be modified. - merike (herder of cats for committed speakers for Security Track)
participants (16)
-
Andrew Sullivan -
Barry Shein -
Beleaguered Admin -
Blake Dunlap -
Brett Frankenberger -
Damian Menscher -
Frank Doherty
-
Jared Mauch -
Livingood, Jason -
manning -
Matthew Petach -
Merike Kaeo -
Mr. Queue -
Randy Bush -
Roland Dobbins -
Warren Bailey