Re: D/DoS mitigation hardware/software needed.
We have such a configuration in progress, it works great without any of the issues you're proposing. Jeff On Jan 4, 2010 9:09 PM, "Dobbins, Roland" <rdobbins@arbor.net> wrote: On Jan 5, 2010, at 4:25 AM, Jeffrey Lyon wrote: > Use a robust firewall such as a Netscreen in fro... Absolutely not - the firewall will fall over due to state-table exhaustion before the mitigation system will. Firewalls (which have no place in front of servers in the first place), load-balancers, and any other stateful devices should be southbound of the mitigation system. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
On Tue, Jan 5, 2010 at 8:36 AM, Jeffrey Lyon <jeffrey.lyon@blacklotus.net> wrote:
We have such a configuration in progress, it works great without any of the issues you're proposing.
So .. this is interesting. The firewall would have to frontend your mail / web / whatever application .. and if something goes beyond the firewall's rated capacity (100k ++ - maybe nearly 150..175k connections per second for a high end firewall), the firewall falls over. And even before that, there's the risk of whatever application you're protecting getting pounded flat if your firewall passes even a small percentage of this traffic. Do you - 1. Have (say) two firewalls in HA config? 2. Back your firewall with routing based measures, S/RTBH, blackhole communities your upstream offers, etc [the standard nspsec bootcamp stuff] 3. Simply back the firewall with a netflow based device? 4. Estimate that the risk of a DDoS that exceeds your firewall's rated capacity is extremely low? [and yes, 150k ++ connections per second ddos is going to be massive, and relatively rare for most people] --srs -- Suresh Ramasubramanian (ops.lists@gmail.com)
Two more options. And for Netflow device - read that to mean Arbor or its competitors. 5 Ditch the stateful firewall and exclusively use a netflow device 6. Outsource to a hosted DDoS mitigation service (Prolexic etc) On Tue, Jan 5, 2010 at 8:43 AM, Suresh Ramasubramanian <ops.lists@gmail.com> wrote:
Do you -
1. Have (say) two firewalls in HA config?
2. Back your firewall with routing based measures, S/RTBH, blackhole communities your upstream offers, etc [the standard nspsec bootcamp stuff]
3. Simply back the firewall with a netflow based device?
4. Estimate that the risk of a DDoS that exceeds your firewall's rated capacity is extremely low? [and yes, 150k ++ connections per second ddos is going to be massive, and relatively rare for most people]
On Jan 5, 2010, at 10:18 AM, Suresh Ramasubramanian wrote:
5 Ditch the stateful firewall and exclusively use a netflow device
NetFlow analysis is very useful for network visibility, and detection/classification/traceback. There are both open-source and commercial NetFlow collection and analysis systems available (full disclosure: I work for a vendor of both NetFlow analysis and DDoS mitigation solutions); however, they don't provide mitigation, which is where S/RTBH, flow-spec, and/or IDMS come into play. PCI DSS iatrogenically *requires* that a 'Web application firewall' be placed in front of Web servers which process credit card information (PCI DSS completely ignores availability, and contains a number of recommendations which are actually harmful from an opsec standpoint). Running mod_security or its equivalent on the front-end Web servers themselves fulfills this requirement without putting a stateful DDoS chokepoint in front of the servers. It's also a really good idea to front Web servers with a tier of caching-only transparent reverse proxies; Squid is a good choice for this, as well as various commercial offerings. WCCPv2 clustering (supported by Squid and several commercial caching/proxying solutions) allows this tier to be scaled horizontally in order to meet capacity demands. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
I actually agree with most of this, but want to correct a clearly inadvertent error below, and make a couple of points. PCI DSS does not require a "Web application firewall". It requires that OR an independent audit of all code within PCI scope by a third party. If a WAF is used, it "only" need be deployed in such a manner as to protect devices inside of PCI scope (depending on design, this may or may not include everything that has public exposure). The powers that be specified additional methods by which PCI compliance could be satisfied other than just these two after the wailing of the masses. I don't know off the top of my head if those other methods will be acceptable in 2010. Personally, I believe a DOS detection/prevention system is typically going to be best placed between screening routers/switches and the next L3/L4 aware device- generally you will want it (or them) to be as close to your ingress edge as you can put it- why allow DOS traffic to go further downstresm? So I suspect Roland and I agree on that fwiw. Earlier, Roland mentions load-balancers and firewalls as both being susceptible to state-table overflows. Certainly this is possible and happened in the past, and it argues for a DOS prevention device being in front of them. At least one modern high-end load balancer handles this well and is in front of a large number if not the majority of major sites. It is possible to build equipment that can handle vast numbers of state entries and handle lookups into them in very attractive big O's. It is also possible to build systems that gracefully handle table exhaustion and enter into aggressive reaping modes. This has been empirically proven to me wrt load-balancers. Some device is going to have to handle the state and insert itself into the path- even if that is a lone webserver. I maintain that there is no reason to believe that it is not possible for a firewall to do that as well as a load-balancer. As for whether you should have a stateful firewall in front of a production web server farm, I understand Roland's point. However, I will say that there are many reasons why people put firewalls in front of webservers- to name some: * Defense in depth. You've never had a host that received external traffic ever accidentally have iptables or windows firewall turned off? Even when debugging a production outage or on accident? * Location for IDS/IDP. * Connection cleanup, re-assembling fragments, etc. * SYN flood protection, etc. * Single choke point to block incoming traffic deemed undesirable. * Single log point for inbound connections for analysis and auditing requirements. * Allows outbound traffic enforcement. * Allows conditional inbound traffic from specific approved external hosts- e.g. a partner. * Some firewalls allow programmatic modification of configurations with all the benefits/pain that brings. This is alongside traditional CLI and GUI interfaces. * In some/many cases a zone based firewall configuration can be much easier to work with than a large iptables config. Certainly the management tools are better. * Yeah, auditors like it. I'm not at all adverse to putting transparent proxies in front of your website. CDN vendors aren't either. I will say that I have had several bad experiences with WCCP not working as expected and failing non-gracefully. I am not saying its always a good idea to have a stateful firewall in front of your web servers, I'm saying that there are reasons why you might want to and in my experience it is common. --D On Mon, Jan 4, 2010 at 7:31 PM, Dobbins, Roland <rdobbins@arbor.net> wrote:
On Jan 5, 2010, at 10:18 AM, Suresh Ramasubramanian wrote:
5 Ditch the stateful firewall and exclusively use a netflow device
NetFlow analysis is very useful for network visibility, and detection/classification/traceback. There are both open-source and commercial NetFlow collection and analysis systems available (full disclosure: I work for a vendor of both NetFlow analysis and DDoS mitigation solutions); however, they don't provide mitigation, which is where S/RTBH, flow-spec, and/or IDMS come into play.
PCI DSS iatrogenically *requires* that a 'Web application firewall' be placed in front of Web servers which process credit card information (PCI DSS completely ignores availability, and contains a number of recommendations which are actually harmful from an opsec standpoint). Running mod_security or its equivalent on the front-end Web servers themselves fulfills this requirement without putting a stateful DDoS chokepoint in front of the servers.
It's also a really good idea to front Web servers with a tier of caching-only transparent reverse proxies; Squid is a good choice for this, as well as various commercial offerings. WCCPv2 clustering (supported by Squid and several commercial caching/proxying solutions) allows this tier to be scaled horizontally in order to meet capacity demands.
----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Injustice is relatively easy to bear; what stings is justice.
-- H.L. Mencken
-- -- Darren Bolding -- -- darren@bolding.org --
On Jan 5, 2010, at 2:38 PM, Darren Bolding wrote:
* Defense in depth. You've never had a host that received external traffic ever accidentally have iptables or windows firewall turned off? Even when debugging a production outage or on accident?
Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. 'Stateful inspection' where in fact there is no useful state to inspect is pointless.
* Location for IDS/IDP.
Non-sequitur, as these things have nothing to do with one another (plus, these devices are useless, anyways, heh).
* Connection cleanup, re-assembling fragments, etc.
Far, far, far better and more scalably handled by the hosts themselves and/or load-balancers.
* SYN flood protection, etc.
Firewalls simply don't handle this well, marketing claims aside. They crash and burn.
* Single choke point to block incoming traffic deemed undesirable.
Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps.
* Single log point for inbound connections for analysis and auditing requirements.
Contextless, arbitrary syslog from firewalls and other such devices is largely useless for this purpose. NetFlow combined with server/app/service logs is the answer to this requirement.
* Allows outbound traffic enforcement.
Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps.
* Allows conditional inbound traffic from specific approved external hosts- e.g. a partner.
Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps.
* Some firewalls allow programmatic modification of configurations with all the benefits/pain that brings. This is alongside traditional CLI and GUI interfaces.
Ugly, brittle, siloed, to be avoided at all costs.
* In some/many cases a zone based firewall configuration can be much easier to work with than a large iptables config. Certainly the management tools are better.
Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps.
* Yeah, auditors like it.
Education is the answer here. ;> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Comments inline. To reiterate- my entire point is that stateful firewalls are at least sometimes useful in front of large websites. Something of a veer off of the original topic, but not an at all useless exercise IMHO. On Mon, Jan 4, 2010 at 11:47 PM, Dobbins, Roland <rdobbins@arbor.net> wrote:
On Jan 5, 2010, at 2:38 PM, Darren Bolding wrote:
* Defense in depth. You've never had a host that received external traffic ever accidentally have iptables or windows firewall turned off? Even when debugging a production outage or on accident?
Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. 'Stateful inspection' where in fact there is no useful state to inspect is pointless.
Stateful policies help protect against misbehaving software, operations error, malicious internal activity, external attackers and trojans/viruses/bots making outbound connections to arbitrary ports anywhere on the Internet. Seems to have a point to me. See my previous point about blocking outbound connections. If you allow internal hosts to make connections out to any port on any ip, then perhaps this point is invalid. But lots of people _don't_ allow outbound from any to any/any, and thats a good thing in my book. As for performance, you can get .9mpps stateful firewalls for reasonable prices, and gear exists that can scale up to 15mpps.
* Location for IDS/IDP.
Non-sequitur, as these things have nothing to do with one another (plus, these devices are useless, anyways, heh).
The gear I'm talking about above supports IDP on the same platform. If I am going to deploy an IDP, why wouldn't I choose one that is flow based so it blocks attacks rather than send out panicked RST's in response to attacks, and which has a stateful firewall included (yes, any flow based IDP is essentially a stateful firewall, and I suspect that a well managed IDP with customized rules is similar in functionality to what WAF's do out of the box. In fact one vendors WAF product functionality pre-trained out of the box is snort rules)
* Connection cleanup, re-assembling fragments, etc.
Far, far, far better and more scalably handled by the hosts themselves and/or load-balancers.
Huh, I would have sworn I've seen fragment based attacks on certain linux kernels and Windows systems. I'm sure it won't ever happen again. Load-balancers may be a good place for this, but there are good reasons that you would prefer to have this taken care of as close to ingress as possible. For example, if you want to compare packet flows of a connection on either side of a load-balancer, it is easier to do that if the incoming connections have already been cleaned up.
* SYN flood protection, etc.
Firewalls simply don't handle this well, marketing claims aside. They crash and burn.
I believe that to have been true in the not so recent past, I don't think it is true of more recent equipment.
* Single choke point to block incoming traffic deemed undesirable.
Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps.
A firewall with easier to use CLI, web UI, management station and API is a much easier way to implement this and offers lots of control for time-of-day based ACL's, etc. etc. I like screening routers, and in a DOS situation I would recommend blocking as close to ingress as possible (preferably outside your own network!) but for typical blocking of nuisance connections, full featured firewalls that can be integrated with a configuration change tracking tool are a better way to go. And what about when the way of identifying undesirable traffic is something other than the source/dest ip/port such as a URI? (more of an argument for an IDP than a simply stateful firewall)
* Single log point for inbound connections for analysis and auditing requirements.
Contextless, arbitrary syslog from firewalls and other such devices is largely useless for this purpose. NetFlow combined with server/app/service logs is the answer to this requirement.
Really? How does netflow show me that a packet was dropped because of rule X? Or permitted because of rule Y? It's great at showing me data about the flow that was extant, but it doesn't tell me the context of the enforcement performed on it. I can assure you that there is a use in looking over denied packet logs. Firewall logs provide _more_ context than netflow. For DOS detection I get that netflow is probably the best tool to use- not arguing that- you made the statement that stateful firewalls are always useless in front of large websites/webservers. I am saying that they are usefull, at least some of the time, and that it is common for them to be deployed in that manner.
* Allows outbound traffic enforcement.
Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps.
So you advocate allowing any packet with ACK set into the internal network? Again, stateful firewalls make this much less of a management headache.
* Allows conditional inbound traffic from specific approved external hosts- e.g. a partner.
Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps.
See above :^)
* Some firewalls allow programmatic modification of configurations with all the benefits/pain that brings. This is alongside traditional CLI and GUI interfaces.
Ugly, brittle, siloed, to be avoided at all costs.
But doing the same thing across hundreds or thousands of web servers via automated systems = good? Personally I like being able to authorize a lower-tech-savvy member of another team to initiate a temporary configuration change via a tool rather than have to wait for the calvary to show up.
* In some/many cases a zone based firewall configuration can be much easier to work with than a large iptables config. Certainly the management tools are better.
Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps.
Huh?
* Yeah, auditors like it.
Education is the answer here.
;>
If only....
----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Injustice is relatively easy to bear; what stings is justice.
-- H.L. Mencken
-- -- Darren Bolding -- -- darren@bolding.org --
On Jan 5, 2010, at 5:04 PM, Darren Bolding wrote:
To reiterate- my entire point is that stateful firewalls are at least sometimes useful in front of large websites.
I understand completely; I simply disagree, stating my reasons for doing so in detail inline. It's my contention that under no circumstances are stateful firewalls *ever* useful in front of *any* Web server, at *any* time, in *any* deployment scenario, either public or private. ;> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
On Jan 5, 2010, at 2:38 PM, Darren Bolding wrote:
PCI DSS does not require a "Web application firewall".
<http://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1313797,00.html> Since no business is going to allow an external 'code review' (if it's even possible, given that they're likely using COTS products, the source code of which they simply don't have), this defaults to a requirement for the 'Web application firewall'. ;> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
I know of several companies, with large websites, that used code reviews as at least one way they met this DSS requirement. So, erm, empirically denied. The PCI DSS does not require code review of the software running in COTS equipment, nor of underlying OS's or applications. It requires a code review of the application code that is inside PCI scope. In general, this means the code you write to run your website is the maximum scope of this requirement. Plenty of companies allow code reviews for security and other purposes, and with good reason. There exist entire practices in IT security firms dedicated to performing code reviews, and they appear to be growing. Also, the PCI security council allows people to use automated code auditing tools (such as fortify), performing a manual "application assessment"- which plenty of firms will let you pay them to do, or even to use an automated web application security scanners. Several vendors of Vulnerability Assessment tools that meet this spec are available. I believe their is strong evidence that the use of web application firewalls to meet this DSS requirement is smaller than you might think. I would not be surprised if it was significantly less than 50%- perhaps 20%. To make the operational content clear- if someone tells you that you need to buy a Web Application Firewall to meet PCI requirements (process credit cards), be aware that is not the only option. I'd recommend you choose the option that is most likely to genuinely improve the security of your infrastructure and business, which may well be a WAF. --D On Mon, Jan 4, 2010 at 11:54 PM, Dobbins, Roland <rdobbins@arbor.net> wrote:
On Jan 5, 2010, at 2:38 PM, Darren Bolding wrote:
PCI DSS does not require a "Web application firewall".
< http://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1...
Since no business is going to allow an external 'code review' (if it's even possible, given that they're likely using COTS products, the source code of which they simply don't have), this defaults to a requirement for the 'Web application firewall'.
;>
----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Injustice is relatively easy to bear; what stings is justice.
-- H.L. Mencken
-- -- Darren Bolding -- -- darren@bolding.org --
On Jan 5, 2010, at 3:58 PM, Darren Bolding wrote:
I believe their is strong evidence that the use of web application firewalls to meet this DSS requirement is smaller than you might think. I would not be surprised if it was significantly less than 50%- perhaps 20%.
This directly contradicts my experience working for vendor of such products, FWIW. But I hope this is indeed the case, as it will lead to higher availability for organizations which go this route! ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
My basis for this is discussions with PCI assessors from multiple firms that perform large numbers of assessments per year. Next time I run into some, I'll ask to see if the usage has increased, its been a few months since I asked this of any of them. --D On Tue, Jan 5, 2010 at 1:02 AM, Dobbins, Roland <rdobbins@arbor.net> wrote:
On Jan 5, 2010, at 3:58 PM, Darren Bolding wrote:
I believe their is strong evidence that the use of web application firewalls to meet this DSS requirement is smaller than you might think. I would not be surprised if it was significantly less than 50%- perhaps 20%.
This directly contradicts my experience working for vendor of such products, FWIW.
But I hope this is indeed the case, as it will lead to higher availability for organizations which go this route!
----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Injustice is relatively easy to bear; what stings is justice.
-- H.L. Mencken
-- -- Darren Bolding -- -- darren@bolding.org --
On Jan 5, 2010, at 10:06 AM, Jeffrey Lyon wrote:
We have such a configuration in progress, it works great without any of the issues you're proposing.
Then you aren't testing it to destruction, heh. ;> If it's a stateful firewall, and state-tracking is turned on, it's quite possible to craft sufficient pathological traffic which conforms to the firewall policies and yet which leads to state-table inspection. And the stateful firewall serves no purpose in front of servers, in which *every incoming packet* is unsolicited. Far more sensible to enforce policy in stateless ACLs in ASIC-based router/switch hardware. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
On Jan 5, 2010, at 10:14 AM, Dobbins, Roland wrote:
If it's a stateful firewall, and state-tracking is turned on, it's quite possible to craft sufficient pathological traffic which conforms to the firewall policies and yet which leads to state-table inspection.
That should read 'state-table exhaustion', apologies for the typo. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
participants (4)
-
Darren Bolding -
Dobbins, Roland -
Jeffrey Lyon -
Suresh Ramasubramanian