January 2010 - Test - lists.nanog.org

D/DoS mitigation hardware/software needed.
by Rick Ernst 11 Jan '10

11 Jan '10

Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. My current system is a home-grown NetFlow parser that spits out syslog to our NOC to investigate potential attacks and manually enter them into our RTBH. Any suggestions other than Arbor? Any other mechanisms being used? My idea is to quash the immediate problem and work additional mitigation with upstreams if needed. I could probably add some automation to my NetFlow/RTBH setup, but I still need to worry about false-positives. I'd rather somebody else do the hard work of finding the various edge-cases. Thanks, Rick

16 34

Re: I don't need no stinking firewall!
by Jay Hennigan 11 Jan '10

11 Jan '10

Nenad Andric wrote: > On Tue Jan 05, 2010 at 01:04:01PM -0800, Jay Hennigan <jay(a)west.net> wrote: >> Or better: >> - Allow from anywhere port 80 to server port > 1023 established > > Adding "established" brings us back to stateful firewall! Not really. It only looks to see if the ACK or RST bits are set. This is different from a stateful firewall which memorizes each outbound packet and checks the return for a match source/destination/sequence. -- Jay Hennigan - CCIE #7880 - Network Engineering - jay(a)impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV

2 1

AT&T request...
by Matt Kelly 11 Jan '10

11 Jan '10

Can someone from AT&T please contact me off list regarding a problem with a segment of our network being blocked? Thanks. --Matt

1 0

Re: D/DoS mitigation hardware/software needed.
by Roger Marquis 10 Jan '10

10 Jan '10

Dobbins, Roland wrote: >My employer's products don't compete with firewalls, they *protect* them; >if anything, it's in my pecuniary interest to *encourage* firewall >deployments, so said firewalls will fall down and need protection, heh. Nobody's disputing that Roland, or the fact that different specialized appliances will protect against different perimeter attacks. The only thing you've said that is being disputed is the the claim that a firewall under a DDoS type of attack will fail before a server under the same type of attack. I question this claim for several reasons. * because it doesn't correlate with my 22 years of experience in systems administration and 14 years in netops (including Yahoo netsecops where I did use IXIAs to compile stats on FreeBSD and Linux packet filtering), * it doesn't correlate with experience in large networks with multiple geographically disperse data centers where we did use Arbor, Cisco and Juniper equipment, * it doesn't correlate with server and firewall hardware and software designs, and last but not least, * because you have shown no objective evidence to support the claim. > I did this kind of testing when I worked for the largest > manufacturer of firewalls in the world Where then, can we find the results of your testing? > Here's the thing; you're simply mistaken, and you hurl insults > instead of listening to the multiple people on this > thread who have vastly more large-scale Internet experience than > you do and who concur with these prescriptions. Nobody has "hurled insults" in this thread other than yourself Roland. Shame on you for such disreputable tactics. To make the case you need more than repeated dismissal of requests for evidence and repeated unsupported claims of "vast experience" with failing servers and firewalls. We just need some actual statistics. Roger Marquis

4 3

Re: D/DoS mitigation hardware/software needed.
by Roger Marquis 10 Jan '10

10 Jan '10

Dobbins, Roland wrote: >> Firewalls do have their place in DDoS mitigation scenarios, but if used as >> the "ultimate" solution you're asking for trouble. > > In my experience, their role is to fall over and die, without > exception. That hasn't been my experience but then I'm not selling anything that might have a lower ROI than firewalls, in small to mid-sized installations. > I can't imagine what possible use a stateful firewall has being > placed in front of servers under normal conditions, much less > during a DDoS attack; it just doesn't make sense. Firewalls are not designed to mitigate large scale DDoS, unlike Arbors, but they do a damn good job of mitigating small scale attacks of all kinds including DDoS. Firewalls actually do a better job for small to medium sites whereas you need an Arbor-like solution for large scale server farms. Firewalls do a good job of protecting servers, when properly configured, because they are designed exclusively for the task. Their CAM tables, realtime ASICs and low latencies are very much unlike the CPU-driven, interrupt-bound hardware and kernel-locking, multi-tasking software on a typical web server. IME it is a rare firewall that doesn't fail long, long after (that's after, not before) the hosts behind them would have otherwise gone belly-up. Rebooting a hosed firewall is also considerably easier than repairing corrupt database tables, cleaning full log partitions, identifying zombie processes, and closing their open file handles. Perhaps a rhetorical question but, does systems administration or operations staff agree with netop's assertion they 'don't need no stinking firewall'? Roger Marquis

5 6

trying to analyze vispa isp outage
by exploit dev 10 Jan '10

10 Jan '10

Hi to all, I have try to check BGP traffic behaviors related to recent VISPA ISP DDOS. For this task I have using BGplay and I need feedback about my analysis. If you are interested check http://extraexploit.blogspot.com/2010/01/trying-to-analyze-vispa-isp-outage… Thank you for your attention. --

2 1

Re: D/DoS mitigation hardware/software needed.
by Roger Marquis 10 Jan '10

10 Jan '10

Dobbins, Roland wrote: >See here for a high-profile example: ><http://files.me.com/roland.dobbins/k54qkv> Reads like a sales pitch to me. No apples to apples comparisons, nothing like an ANOVA of PPS, payload sizes, and other vectors across different types of border defenses. Your presentation makes a good case for Arbor-type defenses, against a certain type of attack, but it doesn't make the case you're referring to. What would convince me is an IXIA on a subnet with ten hosts running a db-bound LAMP stack. Plot the failure points under different loads. Then add an ASA or Netscreen and see what fails under the same loads. That would be an objective measure, unlike what has been offered as evidence in this thread so far. >Placing a stateful inspection device in a topological position where no >stateful inspection is possible due to every incoming packet being >unsolicited makes zero sense whatsoever from an architectural standpoint, >even without going into implementation-specific details. Which is basically claiming that the general purpose web server, running multiple applications, is more capable of inspecting every incoming packet than hardware specifically designed for the task and doing only the task it was designed for. Christopher Morrow wrote: >have you noticed how putting your DB and WEB server on the same hardware >is a bad plan? While often true this is entirely tangental to the thread. Roger Marquis

2 1

qwest outage no notice
by Mike 09 Jan '10

09 Jan '10

We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact. Normally we get notices in advance, even for software upgrades due to security or other important issues, so I am curious if other qwest customers had the same experience and wether this is how it's going to be from here on in? The affected platform was juniper and I'd love to know the specfic case being addressed here. Mike-

19 20

JunOS remote DoS code has been posted to FD
by Brian Keefer 09 Jan '10

09 Jan '10

I haven't tested the code myself, but no reason to think it doesn't work. Consider this your "exploits are in the wild" notice. -- bk

2 1

Re: D/DoS mitigation hardware/software needed.
by Roger Marquis 09 Jan '10

09 Jan '10

Dobbins, Roland wrote: >> Firewalls are not designed to mitigate large scale DDoS, unlike >> Arbors, but they do a damn good job of mitigating small scale >> attacks of all kinds including DDoS. > > Not been my experience at all - quite the opposite. Ok, I'll bite. What firewalls are you referring to? >> Their CAM tables, realtime ASICs and low latencies are very >> much unlike the CPU-driven, interrupt-bound hardware and >> kernel-locking, multi-tasking software on a typical web server. >> IME it is a rare firewall that doesn't fail long, long after >> (that's after, not before) the hosts behind them would have >> otherwise gone belly-up. > > Completely incorrect on all counts. So then you're talking about CPU-driven firewalls, without ASICs e.g., consumer-level gear? Well, that would explain why you think they fail before the servers behind them. >I've been a sysadmin Have you noticed how easily Drupal servers go down with corrupt MyISAM tables? How would S/RTBH and/or flow-spec protect against that? Roger Marquis

3 3