Are folks seeing any major DOS in progress ? Twitter seems to be under one and FB is flaky.
We are presently seeing some weird FB behavior -- timeouts and retry issues. We've had several reports from our users and just began investigating. Any info you have would be appreciated. --sjk Jorge Amodio wrote:
Are folks seeing any major DOS in progress ?
Twitter seems to be under one and FB is flaky.
Same thing for me here in Lincoln, Neb. I was having issues like this starting Thursday evening about 8 p.m. or so, and it has continued all morning. And of course with Facebook being so vital to my job.... :) I can't pin down specifics, just that it feels "flaky" I guess. Timeouts, retries, photo tagging working intermittently, and so on. -Andy Ringsmuth On Aug 6, 2009, at 10:29 AM, sjk wrote:
We are presently seeing some weird FB behavior -- timeouts and retry issues. We've had several reports from our users and just began investigating. Any info you have would be appreciated.
--sjk
Jorge Amodio wrote:
Are folks seeing any major DOS in progress ?
Twitter seems to be under one and FB is flaky.
Ditto from Canberra, Australia FB very flakey, same as Andy I guess. Thanks, Cody Appleby On 07/08/2009, at 1:36 AM, Andy Ringsmuth wrote:
Same thing for me here in Lincoln, Neb. I was having issues like this starting Thursday evening about 8 p.m. or so, and it has continued all morning.
And of course with Facebook being so vital to my job.... :)
I can't pin down specifics, just that it feels "flaky" I guess. Timeouts, retries, photo tagging working intermittently, and so on.
-Andy Ringsmuth
On Aug 6, 2009, at 10:29 AM, sjk wrote:
We are presently seeing some weird FB behavior -- timeouts and retry issues. We've had several reports from our users and just began investigating. Any info you have would be appreciated.
--sjk
Jorge Amodio wrote:
Are folks seeing any major DOS in progress ?
Twitter seems to be under one and FB is flaky.
Facebook is being really flaky here in Ireland too. http://www.irishtimes.com/newspaper/breaking/2009/0806/breaking53.htm (about Twitter) 2009/8/6 Cody Appleby <kizmet@kizmet.id.au>
Ditto from Canberra, Australia FB very flakey, same as Andy I guess.
Thanks, Cody Appleby
On 07/08/2009, at 1:36 AM, Andy Ringsmuth wrote:
Same thing for me here in Lincoln, Neb. I was having issues like this
starting Thursday evening about 8 p.m. or so, and it has continued all morning.
And of course with Facebook being so vital to my job.... :)
I can't pin down specifics, just that it feels "flaky" I guess. Timeouts, retries, photo tagging working intermittently, and so on.
-Andy Ringsmuth
On Aug 6, 2009, at 10:29 AM, sjk wrote:
We are presently seeing some weird FB behavior -- timeouts and retry
issues. We've had several reports from our users and just began investigating. Any info you have would be appreciated.
--sjk
Jorge Amodio wrote:
Are folks seeing any major DOS in progress ?
Twitter seems to be under one and FB is flaky.
Jorge Amodio wrote:
Are folks seeing any major DOS in progress ?
Twitter seems to be under one and FB is flaky.
DDoS happens hundreds of times a day. Twitter and the Internet operations security community will likely take care of it, especially as it's twitter and we all have a warm fuzzy feeling inside. Off topic, I found it hilarious how all the tweets came back to facebook and set statuses about twitter. :o) Gadi. -- Gadi Evron, ge@linuxbox.org. Blog: http://gevron.livejournal.com/
Down from Costa Rica and Ireland too... Interesting that they are starting to go for Social Networking sites now. Have they given up on online gambling sites now? It appears as though they haven't been actively attacking gambling sites for several days... 2009/8/6 Jorge Amodio <jmamodio@gmail.com>:
Are folks seeing any major DOS in progress ?
Twitter seems to be under one and FB is flaky.
check out: http://status.twitter.com/ Tells the story. Chris Gotstein Sr Network Engineer UP Logon/Computer Connection UP 500 N Stephenson Ave Iron Mountain, MI 49801 Phone: 906-774-4847 Fax: 906-774-0335 chris@uplogon.com Ken Gilmour wrote:
Down from Costa Rica and Ireland too... Interesting that they are starting to go for Social Networking sites now. Have they given up on online gambling sites now? It appears as though they haven't been actively attacking gambling sites for several days...
2009/8/6 Jorge Amodio <jmamodio@gmail.com>:
Are folks seeing any major DOS in progress ?
Twitter seems to be under one and FB is flaky.
http://status.twitter.com/ "We are defending against a denial-of-service attack, and will update status again shortly." -----Original Message----- From: Marshall Eubanks [mailto:tme@americafree.tv] Sent: 06 August 2009 16:57 To: Jorge Amodio Cc: NANOG Subject: Re: DOS in progress ? On Aug 6, 2009, at 11:25 AM, Jorge Amodio wrote:
Are folks seeing any major DOS in progress ?
Twitter seems to be under one and FB is flaky.
Twitter is very flaky & slow to load today, but that is hardly unusual. Do you have any other evidence ? Regards Marshall
http://status.twitter.com/ Ongoing denial-of-service attack<http://status.twitter.com/post/157191978/ongoing-denial-of-service-attack> 1 hour ago We are defending against a denial-of-service attack, and will update status again shortly. *Update*: the site is back up, but we are continuing to defend against and recover from this attack. On Thu, Aug 6, 2009 at 8:57 AM, Marshall Eubanks <tme@americafree.tv> wrote:
On Aug 6, 2009, at 11:25 AM, Jorge Amodio wrote:
Are folks seeing any major DOS in progress ?
Twitter seems to be under one and FB is flaky.
Twitter is very flaky & slow to load today, but that is hardly unusual.
Do you have any other evidence ?
Regards Marshall
Jorge Amodio <jmamodio@gmail.com> writes:
Are folks seeing any major DOS in progress ?
Twitter seems to be under one and FB is flaky.
From what I understand, it's quite common. I got hammered last week. It took out some routers at my upstream (it was a tcp syn flood attack, a whole lot of really small packets. 20Kpps was the peak I saw before the upstream took me out.)
Now, I've cleaned up the mess; (and for now, dropped the inexpensive upstream with the weak routers) I'm building out my monitoring infrastructure and generally preparing for next time. as far as stopping the attacks by 'finishing the job' - which is to say, blackholing the target, the way forward is pretty clear. I mean, I need to do more research and implement stuff, but I don't really need NANOG help for that. The thing is, I like my customers. I don't want to shut off people who are paying me just because they get attacked. I mean, if that's what I've got to do to keep my other paying customers up, I'll do it, but I'd really rather not. what is the 'best practice' here? I mean, most of this is scripted, so conceivably, I could get source addresses fast enough to block them upstream. (right now my provider is only allowing me to blackhole my own space, not blackhole source addresses, which while it keeps me in business, is not really what I want.) My provider does seem to be pretty responsive, so if I can bring them a tool, they might set it up for me. But yeah, I'm getting sidetracked. I guess there are two things I want to know: 1. are there people who apply pressure to ISPs to get them to shut down botnets, like maps did for spam? I've got 50 gigs of packet captures, and have been going through with perl to detect IPs who send me lots of tcp packets with 0 payloads, then manually sending abuse reports. Half the abuse reports bounce, and the other half are ignored. (most of the hosts in question are in china.) 2. is there a standard way to push a null-route on the attackers source IP upstream? I know the problem is difficult due to trust issues, but if I could null route the source, it's just a matter of detecting abusive traffic, and with this attack, that part was pretty easy. -- Luke S. Crawford http://prgmr.com/xen/ - Hosting for the technically adept http://nostarch.com/xen.htm - We don't assume you are stupid.
On Aug 8, 2009, at 11:57 AM, Luke S Crawford wrote:
2. is there a standard way to push a null-route on the attackers source IP upstream?
Sure - if you apply loose-check uRPF (and/or strict-check, when you can do so) on Cisco or Juniper routers, you can combine that with the blackhole to give you a source-based remotely-triggered blackhole, or S/RTBH. You can do this at your edges, and you *may* be able to arrange it with other networks with whom you connect (i.e., scope limited to your link with them). Combine that with the other standard architectural and hardening BCPs, along with the DNS BCPs, and you'll be much better prepared to detect, classify, traceback, and mitigate attacks. The key is to ensure you're making use of hardware-based routers which can handle high pps. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Unfortunately, inefficiency scales really well. -- Kevin Lawton
Roland Dobbins <rdobbins@arbor.net> writes:
On Aug 8, 2009, at 11:57 AM, Luke S Crawford wrote:
2. is there a standard way to push a null-route on the attackers source IP upstream?
Sure - if you apply loose-check uRPF (and/or strict-check, when you can do so) on Cisco or Juniper routers, you can combine that with the blackhole to give you a source-based remotely-triggered blackhole, or S/RTBH. You can do this at your edges, and you *may* be able to arrange it with other networks with whom you connect (i.e., scope limited to your link with them).
Ah, nice. thank you, that is exactly what I was looking for. I'll read up on it this weekend and see if I can talk my provider into letting me push that upstream. -- Luke S. Crawford http://prgmr.com/xen/ - Hosting for the technically adept http://nostarch.com/xen.htm - We don't assume you are stupid.
Some hardcore stuff on S/RTBH here: http://www.arbornetworks.com/index.php?option=com_docman&task=doc_download&g id=112 http://www.cisco.com/web/about/security/intelligence/blackhole.pdf (which appears to have replaced http://www.cisco.com/warp/public/732/Tech/security/docs/blackhole.pdf) http://www.nanog.org/meetings/nanog30/presentations/morrow.pdf http://pierky.wordpress.com/2009/05/31/gns3-lab-remote-triggered-black-holin g/ http://packetlife.net/blog/2009/jul/06/remotely-triggered-black-hole-rtbh-ro uting/ Frank -----Original Message----- From: Luke S Crawford [mailto:lsc@prgmr.com] Sent: Saturday, August 08, 2009 3:15 AM To: Roland Dobbins Cc: NANOG list Subject: Re: Botnet hunting resources (was: Re: DOS in progress ?) Roland Dobbins <rdobbins@arbor.net> writes:
On Aug 8, 2009, at 11:57 AM, Luke S Crawford wrote:
2. is there a standard way to push a null-route on the attackers source IP upstream?
Sure - if you apply loose-check uRPF (and/or strict-check, when you can do so) on Cisco or Juniper routers, you can combine that with the blackhole to give you a source-based remotely-triggered blackhole, or S/RTBH. You can do this at your edges, and you *may* be able to arrange it with other networks with whom you connect (i.e., scope limited to your link with them).
Ah, nice. thank you, that is exactly what I was looking for. I'll read up on it this weekend and see if I can talk my provider into letting me push that upstream. -- Luke S. Crawford http://prgmr.com/xen/ - Hosting for the technically adept http://nostarch.com/xen.htm - We don't assume you are stupid.
Roland Dobbins wrote:
On Aug 8, 2009, at 11:57 AM, Luke S Crawford wrote:
2. is there a standard way to push a null-route on the attackers source IP upstream?
Sure - if you apply loose-check uRPF (and/or strict-check, when you can do so) on Cisco or Juniper routers, you can combine that with the blackhole to give you a source-based remotely-triggered blackhole, or S/RTBH. You can do this at your edges, and you *may* be able to arrange it with other networks with whom you connect (i.e., scope limited to your link with them).
Warren Kumari and other collaborated on a document to describe how this is normally done: http://tools.ietf.org/html/draft-ietf-opsec-blackhole-urpf-04 Coordination with your upstreams before you need this is important.
Combine that with the other standard architectural and hardening BCPs, along with the DNS BCPs, and you'll be much better prepared to detect, classify, traceback, and mitigate attacks. The key is to ensure you're making use of hardware-based routers which can handle high pps.
----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Unfortunately, inefficiency scales really well.
-- Kevin Lawton
On Fri, 8 Aug 2009, Luke S Crawford wrote:
1. are there people who apply pressure to ISPs to get them to shut down botnets, like maps did for spam?
sadly no.
I've got 50 gigs of packet captures, and have been going through with perl to detect IPs who send me lots of tcp packets with 0 payloads, then manually sending abuse reports.
Half the abuse reports bounce, and the other half are ignored. (most of the hosts in question are in china.)
it's a big problem, especially with rogue networks like france and china. there is currently zero incentive for anyone clean up, as there are no consequences for not doing so. this will not change until there are real consequences for operating IP cesspools. -Dan
goemon@anime.net writes:
On Fri, 8 Aug 2009, Luke S Crawford wrote:
1. are there people who apply pressure to ISPs to get them to shut down botnets, like maps did for spam?
sadly no.
... Why do you think this might be? Fear of (extralegal) retaliation by botnet owners? or fear of getting sued by listed network owners? or is the idea (shunning packets from ISPs that host botnets) fundamentally unsound? If someone sufficiently trustworthy produced a BGP feed of networks that were unresponsive to abuse complaints, do you think other networks would use it to block traffic? I mean, ultimately I think that having several providers of such feeds with differing levels of aggression would be the best case, but someone has got to go first. -- Luke S. Crawford http://prgmr.com/xen/ - Hosting for the technically adept http://nostarch.com/xen.htm - We don't assume you are stupid.
On Mon, 10 Aug 2009, Luke S Crawford wrote:
goemon@anime.net writes:
On Fri, 8 Aug 2009, Luke S Crawford wrote:
1. are there people who apply pressure to ISPs to get them to shut down botnets, like maps did for spam? sadly no. ...
Why do you think this might be? Fear of (extralegal) retaliation by botnet owners? or fear of getting sued by listed network owners? or is the idea (shunning packets from ISPs that host botnets) fundamentally unsound?
such a list would include all of chinanet and france telecom. it would likely not last long. what do you do when rogue networks are state owned?
If someone sufficiently trustworthy produced a BGP feed of networks that were unresponsive to abuse complaints, do you think other networks would use it to block traffic?
no.
I mean, ultimately I think that having several providers of such feeds with differing levels of aggression would be the best case, but someone has got to go first.
consider how much time and effort it took to get intercage shut down and you'd realize it's pretty much a lost cause. -Dan
On 10/08/2009, at 8:11 PM, goemon@anime.net wrote:
such a list would include all of chinanet and france telecom. it would likely not last long.
You've mentioned France twice now. Is there a big botnet problem there? I've never heard of anything like that. I'll admit I don't follow this area of the network closely, but I'm sure there are other places higher up the list than FTE.. -- Nathan Ward
On Aug 10, 2009, at 5:34 AM, Nathan Ward <nanog@daork.net> wrote:
On 10/08/2009, at 8:11 PM, goemon@anime.net wrote:
such a list would include all of chinanet and france telecom. it would likely not last long.
You've mentioned France twice now. Is there a big botnet problem there? I've never heard of anything like that. I'll admit I don't follow this area of the network closely, but I'm sure there are other places higher up the list than FTE..
I would say the problem plagues many diverse networks. The background radiation goes undetected by most people for cost reasons. It's cheaper to pass the bits then have a human convince someone their machine is compromised. The problem will continue to be acute as transit costs get even lower. - Jared
Why do you think this might be? Fear of (extralegal) retaliation by botnet owners? or fear of getting sued by listed network owners? [TLB:] No more than any anti-spam RBL or is the idea (shunning packets from ISPs that host botnets) fundamentally unsound?
If someone sufficiently trustworthy produced a BGP feed of networks
were unresponsive to abuse complaints, do you think other networks would use it to block traffic? I mean, ultimately I think that having several providers of such feeds with differing levels of aggression would be
[TLB:] That's an ongoing raging debate. Some say, since enumerating badness cant' protect you against all threats, that you shouldn't' do it at all. My take is, if you can filter the worst actors early and fast, based on IP address, that gives you deeper packet devices more capacity, and saves you network bandwidth. It's been my experience that IP level blocking is a best practice as the second step (the first being selective availability of any service to only those it NEEDS to be, which in the case of many network operators is everywhere and everyone, and therefore a useless filter for a network operator) in a layered defense. that the
best case, but someone has got to go first.
[TLB:] <shameless plug> That's what ThreatSTOP is for. We use DNS, not BGP, because there are far more traffic management devices (think Subscriber firewalls) that can use it, and because AT&T has a patent on using BGP for block lists. </shameless plug>
Luke S Crawford wrote:
1. are there people who apply pressure to ISPs to get them to shut down botnets, like maps did for spam?
Hi, Luke! MAAWG recently published a document to help ISPs deal with infected machines in their networks. It's not the same kind of pressure, but (as we learned with open relays at MAPS) pressure isn't very effective unless there are tools available to deal with the problem. http://www.maawg.org/about/publishedDocuments/MAAWG_Bot_Mitigation_BP_2007-0... -- J.D. Falk Return Path Inc http://www.returnpath.net/
J.D. Falk wrote:
Hi, Luke! MAAWG recently published a document to help ISPs deal with infected machines in their networks. It's not the same kind of pressure, but (as we learned with open relays at MAPS) pressure isn't very effective unless there are tools available to deal with the problem.
It could also use a lot more resources? Watching traffic flows for traffic destined to known C&C addresses is nice, but including a pointer to a resource that actually gives those addresses is much more useful. For those who don't deal with it every day, the document just says they need to spend even more time with google. Jack
I surprised that nobody has mentioned the work of shadowserver.org, they are able to send reports of malware infections on your networks (see http://www.shadowserver.org/wiki/pmwiki.php/Services/Reports). The service has proved to a brilliant tool in mitigating various forms of malware such as Conficker with almost 0% false positives. Cheers Bradley -----Original Message----- From: Jack Bates [mailto:jbates@brightok.net] Sent: 11 August 2009 14:11 To: J.D. Falk Cc: NANOG Subject: Re: Botnet hunting resources J.D. Falk wrote:
Hi, Luke! MAAWG recently published a document to help ISPs deal with infected machines in their networks. It's not the same kind of pressure, but (as we learned with open relays at MAPS) pressure isn't very effective unless there are tools available to deal with the problem.
It could also use a lot more resources? Watching traffic flows for traffic destined to known C&C addresses is nice, but including a pointer to a resource that actually gives those addresses is much more useful. For those who don't deal with it every day, the document just says they need to spend even more time with google. Jack
Jack Bates wrote:
J.D. Falk wrote:
Hi, Luke! MAAWG recently published a document to help ISPs deal with infected machines in their networks. It's not the same kind of pressure, but (as we learned with open relays at MAPS) pressure isn't very effective unless there are tools available to deal with the problem.
It could also use a lot more resources? Watching traffic flows for traffic destined to known C&C addresses is nice, but including a pointer to a resource that actually gives those addresses is much more useful. For those who don't deal with it every day, the document just says they need to spend even more time with google.
I'll share your comments with the document authors. They're treating it as a living document, with updates expected regularly. -- J.D. Falk Return Path Inc http://www.returnpath.net/
participants (21)
-
Andy Ringsmuth -
Bradley Freeman -
Chris Gotstein -
Cody Appleby -
Darren -
Frank Bulk -
Gadi Evron -
goemon＠anime.net -
J.D. Falk -
Jack Bates -
Jared Mauch -
Joel Jaeggli -
Jorge Amodio -
Ken Gilmour -
Luke S Crawford -
Marshall Eubanks -
Nathan Ward -
Rachael Holt -
Roland Dobbins -
sjk -
Tomas L. Byrnes