Oh boy, what a fun night this was. After a 4 or so hours downtime, my servers are back up and running. Heres the gorey details. At about 7pm EST, we began having unusual issues with our network, the router, and several machines on the network. For the first part of the attack, we were held down for a good 30-60 minutes. Took us a while to figure out which one of our machines was being targeted. Turns out to be our NAT firewall box. We tried several things to drop the attack, but it still kept coming in strong (mind you, we don't have very much bandwidth, but we can usually ride out DoS attacks pretty well - this was an exception) Then suddenly, out of the blue it dropped. Outside connectivity was restored and things were back to normal. 20 minutes later, the relentless attack began again. This time, we were ready and waiting with tcpdump and several other handcrafted tools we use for this type of thing. The attack was coming from a single source machine, unspoofed (ballsy if you ask me), 128.186.11.215. Packets were UDP, random from 2100-2299 source and 2400-2699 dest. So, now for the fun part. Being offsite, I wasn't the one to place the calls, but my admin on site started with FSU's abuse desk. No help whatsoever. Claimed that because the abuse desk was gone, they had no authority to deal with the problem. Frustrated, annoyed, and pissed off, he tried again, and got hung up on twice. Nice people eh? Our next call was a bit later (at this point, we were very unhappy and ready to start raising hell with anyone we could find) - this time, to their upstream Qwest. After dealing with the operator, they finally sent him to one of the NOCs. Unfortunately, they sent him to the wrong NOC and not the Qwest MD NOC. Luckily, we got someone with a clue - a nice guy by the name of Richard Stein who tried to help us, but found that the other NOC was unresponsive and couldn't do anything himself to solve the problem. After hanging up with Qwest, we got a call back from FSU. After a good 20 minutes or so of talking with the net admin from FSU, things were finally set in motion. After another good 10 minutes or so, connectivity was restored and everything was back to normal. According to my guy, they yanked the whole subnet at FSU. Problem solved. So here I am, asking if anyone here has any advice on dealing with these issues in the future? Its painfully apparent noone takes these situations seriously enough. What should we do when we are put in a position like this? Just sit back and hope it goes away itself? Also, any ideas on how to deal with these attacks on lower bandwidth connections? Right now, 2mbit.com / sosdg.org is sitting on a 1.5/256 business DSL line. I really can't afford to be buying T1s or T3s just to hold up to attacks like this. As always, thanks. -------------------------- Brian Bruns The Summit Open Source Development Group Open Solutions For A Closed World / Anti-Spam Resources http://www.2mbit.com ICQ: 8077511
So here I am, asking if anyone here has any advice on dealing with these issues in the future? Its painfully apparent noone takes these situations seriously enough. What should we do when we are put in a position like this? Just sit back and hope it goes away itself?
Also, any ideas on how to deal with these attacks on lower bandwidth connections? Right now, 2mbit.com / sosdg.org is sitting on a 1.5/256 business DSL line. I really can't afford to be buying T1s or T3s just to hold up to attacks like this.
As always, thanks. -------------------------- Brian Bruns The Summit Open Source Development Group
I think I would follow two avenues next time - the direct approach with FSU (or wherever the traffic is coming from) as well as with your DSL provider. Your upstream should be able to assist in at least keeping the traffic off of your dedicated line. Whether your DSL provider has the resources to sink the traffic may be another matter -- but they are at least in a position to help you and (since you are paying them) have an interest in dealing with you. Mark Radabaugh Amplex (419) 720-3635
----- Original Message ----- From: "Mark Radabaugh" <mark@amplex.net> To: <nanog@merit.edu> Sent: Tuesday, October 07, 2003 11:56 PM Subject: Re: DoS Attacks
I think I would follow two avenues next time - the direct approach with
(or wherever the traffic is coming from) as well as with your DSL
FSU provider.
Your upstream should be able to assist in at least keeping the traffic off of your dedicated line.
Whether your DSL provider has the resources to sink the traffic may be another matter -- but they are at least in a position to help you and (since you are paying them) have an interest in dealing with you.
I hate to say this, but Ameritech/SBC is utterly useless in matters like this. I mean, at one point their redback was being nailed, and they didn't seem to care one bit. After 5pm, everyone with a clue seems to leave, and we are left with useless low level help desk techs. Our DSL service isn't bad - in fact it rarely goes down. The problem is that when we need their help with something out of our league, they are completely useless. Anyone know of a contact number for SBC/Ameritech that would be useful in a case like this? -------------------------- Brian Bruns The Summit Open Source Development Group Open Solutions For A Closed World / Anti-Spam Resources http://www.2mbit.com ICQ: 8077511
First of all, have your tools ready so that whenever DoS pounds on you, you can immediately activate them and they will give you an overview of the DoS attack such as size of the attack, source/dest (random or one/two or spoofed?), et al. Then you need to contact your upstream first to hve them deal with it, and yes I understand, most SDSL providers do not like to cooperate. Considering it takes me 1 hour of buerocracy to get an ACL put up during a DoS to my current providers, getting an ACL activated by SDSL team is.....psh.... utterly hopeless unless you have people connections :( If you can't afford T1/T3 type of circuits where you can at least call up your upstream (doesnt matter how long it takes them to put up the ACL, the point is, will they?), then I hate to say... I don't think there is much you can do :-( -hc -- Haesu C. TowardEX Technologies, Inc. Consulting, colocation, web hosting, network design and implementation http://www.towardex.com | haesu@towardex.com Cell: (978)394-2867 | Office: (978)263-3399 Ext. 170 Fax: (978)263-0033 | POC: HAESU-ARIN On Wed, Oct 08, 2003 at 12:03:19AM -0400, Brian Bruns wrote:
----- Original Message ----- From: "Mark Radabaugh" <mark@amplex.net> To: <nanog@merit.edu> Sent: Tuesday, October 07, 2003 11:56 PM Subject: Re: DoS Attacks
I think I would follow two avenues next time - the direct approach with
(or wherever the traffic is coming from) as well as with your DSL
FSU provider.
Your upstream should be able to assist in at least keeping the traffic off of your dedicated line.
Whether your DSL provider has the resources to sink the traffic may be another matter -- but they are at least in a position to help you and (since you are paying them) have an interest in dealing with you.
I hate to say this, but Ameritech/SBC is utterly useless in matters like this. I mean, at one point their redback was being nailed, and they didn't seem to care one bit. After 5pm, everyone with a clue seems to leave, and we are left with useless low level help desk techs.
Our DSL service isn't bad - in fact it rarely goes down. The problem is that when we need their help with something out of our league, they are completely useless. Anyone know of a contact number for SBC/Ameritech that would be useful in a case like this?
-------------------------- Brian Bruns The Summit Open Source Development Group Open Solutions For A Closed World / Anti-Spam Resources http://www.2mbit.com ICQ: 8077511
Can SLA's be used to cover this sort of thing. (starts to dig out his own contracts). Surely you should be able to bounce it to your upstream provider who should deal with it for you?? Just a thought. -- Martin Hepworth Senior Systems Administrator Solid State Logic Ltd tel: +44 (0)1865 842300 ********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com **********************************************************************
On Wed, 08 Oct 2003 08:44:26 +0100 Martin Hepworth <martinh@solid-state-logic.com> wrote:
Can SLA's be used to cover this sort of thing. (starts to dig out his own contracts).
Surely you should be able to bounce it to your upstream provider who should deal with it for you??
Just a thought.
-- Martin Hepworth Senior Systems Administrator Solid State Logic Ltd tel: +44 (0)1865 842300
********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager.
This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses.
www.mimesweeper.com **********************************************************************
Due to the efficiency of our upstream provider's abuse department, opening efficiently at 8 am and closing just as efficiently at 5 pm (because we all know network abuse only occurs between 8 and 5), the ISP wasn't going to be of much help with an attack that started at 6:30pm localtime. Andrew D Kirch Security Admin - Summit Open Source Development Group http://www.sosdg.org
Due to the efficiency of our upstream provider's abuse department, opening efficiently at 8 am and closing just as efficiently at 5 pm
(because we all know network abuse only occurs between 8 and 5), the ISP wasn't going to be of much help with an attack that started at 6:30pm localtime.
Andrew D Kirch Security Admin - Summit Open Source Development Group http://www.sosdg.org
* A lot of Abuse Departments are going to close at around 5pm, because the advanced staff that would know how to deal with such things goes home around that time. But most of them should have an escalation procedure, which means that there is someone(s) over the staff in the NOC or Call (Support) Center which can be called if necessary. You should insist or demand the issue be escalated if it warrants this. If they won't escalate it ask for the phone number of the supervisor... If they escalate it and you don't get a call back in 30-60 minutes ... call again (repeat until you get a call back). Be prepared to justify why you have had your issue escalated complete with emailable detailed information. Even if the guy that calls you back (semi-technical manager type) doesn't have the proper clue, chances are he has someone else on call that does. If they don't have this escalation capability ... (IMHO) get a different ISP. --- Alan Spicer (a_spicerNOSPAM@bellsouth.net) http://aspicer.homelinux.net/ Systems and Network Administration, and Telecommunications (954) 977-5245 "The wonderful thing about the Internet is that you're connected to everyone else. The terrible thing about the Internet is that you're connected to everyone else." -- Vincent Cerf (Father of the Internet) Customer: "The Internet is running too slow. Could you reboot it please?" Customer: "So that'll get me connected to the Internet, right?" Tech Support: "Yeah." Customer: "And that's the latest version of the Internet, right?" Tech Support: "Uhh...uh...uh...yeah."
<rant style="moaning and useless" context="naive"> I really don't give a ^H^H^H^H!H * !X *!X about what timeframe abuse departments operate. I just want more upstreams (or specifically my upstreams) to have a community that I can announce a /32 to null. </rant> -hc -- Haesu C. TowardEX Technologies, Inc. Consulting, colocation, web hosting, network design and implementation http://www.towardex.com | haesu@towardex.com Cell: (978)394-2867 | Office: (978)263-3399 Ext. 170 Fax: (978)263-0033 | POC: HAESU-ARIN On Wed, Oct 08, 2003 at 07:19:39PM -0400, Alan Spicer wrote:
Due to the efficiency of our upstream provider's abuse department, opening efficiently at 8 am and closing just as efficiently at 5 pm
(because we all know network abuse only occurs between 8 and 5), the ISP wasn't going to be of much help with an attack that started at 6:30pm localtime.
Andrew D Kirch Security Admin - Summit Open Source Development Group http://www.sosdg.org
* A lot of Abuse Departments are going to close at around 5pm, because the advanced staff that would know how to deal with such things goes home around that time. But most of them should have an escalation procedure, which means that there is someone(s) over the staff in the NOC or Call (Support) Center which can be called if necessary. You should insist or demand the issue be escalated if it warrants this. If they won't escalate it ask for the phone number of the supervisor... If they escalate it and you don't get a call back in 30-60 minutes ... call again (repeat until you get a call back). Be prepared to justify why you have had your issue escalated complete with emailable detailed information. Even if the guy that calls you back (semi-technical manager type) doesn't have the proper clue, chances are he has someone else on call that does.
If they don't have this escalation capability ... (IMHO) get a different ISP.
--- Alan Spicer (a_spicerNOSPAM@bellsouth.net) http://aspicer.homelinux.net/ Systems and Network Administration, and Telecommunications (954) 977-5245
"The wonderful thing about the Internet is that you're connected to everyone else. The terrible thing about the Internet is that you're connected to everyone else." -- Vincent Cerf (Father of the Internet)
Customer: "The Internet is running too slow. Could you reboot it please?"
Customer: "So that'll get me connected to the Internet, right?" Tech Support: "Yeah." Customer: "And that's the latest version of the Internet, right?" Tech Support: "Uhh...uh...uh...yeah."
Haesu wrote:
<rant style="moaning and useless" context="naive">
I really don't give a ^H^H^H^H!H * !X *!X about what timeframe abuse departments operate. I just want more upstreams (or specifically my upstreams) to have a community that I can announce a /32 to null.
</rant>
Seems like they ought to be made to offer a 1/3-the-price offering if the connectivity is only offered for 1/3 the time. No, really, I'll be alright, just me sit down here a minute.
On Tue, Oct 07, 2003 at 11:45:35PM -0400, Brian Bruns wrote:
So here I am, asking if anyone here has any advice on dealing with these issues in the future? Its painfully apparent noone takes these situations seriously enough. What should we do when we are put in a position like this? Just sit back and hope it goes away itself?
You were lucky to know where it was coming from. What you depends on what you know. You knew the sources are small and you knew where they were. You did the right thing by contacting FSU, and then their upstream. If either was unresponsive, they are being extremely neglegent. I'm one of the few who still believes that immediate 24/7 response to security problems should be a requirement for permanent internet connectivity. If you don't know the source, talking to your own upstream to see if there is anything they can do is a good first move. Most decent places will at least TRY and help you out. But realise that sometimes you just need to ride it out.
On Tue, 7 Oct 2003, Avleen Vig wrote:
You knew the sources are small and you knew where they were. You did the right thing by contacting FSU, and then their upstream. If either was unresponsive, they are being extremely neglegent.
Its generally a better idea to contact your own upstream provider first. Your own upstream knows you, and is supposedly paid to help you. Your upstream should have contacts with its BGP peers and eventually the source. A problem with calling random NOCs is they don't know you from someone trying to social engineer something. So you end up delaying effective response while they try to figure out who you are, and how your report is related to them. If you have a problem with your credit card, your own bank is in a better position to help you than calling random other banks in the world even if you did have their phone number. Other banks may care about security, but they still don't know who you are. If your own upstream won't help you, you may have no choice but to beg other NOCs to help you.
On Tue, 7 Oct 2003, Brian Bruns wrote:
So, now for the fun part. Being offsite, I wasn't the one to place the calls, but my admin on site started with FSU's abuse desk. No help whatsoever. Claimed that because the abuse desk was gone, they had no authority to deal with the problem. Frustrated, annoyed, and pissed off, he tried again, and got hung up on twice. Nice people eh?
You were speaking with one of our 2nd shift operators. Unfortunately, our 2nd and 3rd operators are not hired on the basis of cluefulness, but for their willingness to work these shifts and do so at minimal pay. They tend to take things very literally, so despite the fact that we have network admins on-call 7/24, unless it's reported as a "network problem" the operators won't call. This morning instructions have been issued (again) to treat security incidents as "network problem" and act accordingly. Additionally, your E-Mail has been hardcopied and posted in the operator's area, along with some not-for-public-consumption comments of my own.
After hanging up with Qwest, we got a call back from FSU. After a good 20 minutes or so of talking with the net admin from FSU, things were finally set in motion. After another good 10 minutes or so, connectivity was restored and everything was back to normal. According to my guy, they yanked the whole subnet at FSU. Problem solved.
The computer's MAC was blocked at the switch. Our apologies for this incident. We will continue trying to improve the responsiveness of our 2nd and 3rd shift operations staff. If all else fails, call me at home: 850-385-4725 - SLS ------------------------------------------------------------------------ Scott L. Stursa 850/644-2591 Network Security Officer stursa@acns.fsu.edu Academic Computing and Network Services Florida State University - No good deed goes unpunished -
participants (10)
-
Alan Spicer -
Andrew D Kirch -
Avleen Vig -
Brian Bruns -
Haesu -
Laurence F. Sheldon, Jr. -
Mark Radabaugh -
Martin Hepworth -
Scott Stursa -
Sean Donelan