Merits of purpose-built (appliance) vs. FreeBSD+ipfw firewalls
I am looking for comments and suggestions regarding the merits of purpose-built, appliance style firewalls (like a netscreen or Cisco PIX) vs. running ipfw on a commodity server running FreeBSD. I am interested only in packet filtering and rate limiting performance - NOT in VPNs or IPsec/crypto considerations. --- Currently, I run a FreeBSD firewall running ipfw (500 mhz celeron, 256 mags ram). This machine does nothing - runs no services but ssh, and simply sits at my network border doing packet filtering. I have a lot of hosts (four /24s - about 500 active IPs) behind this firewall, and generally push 5-7 megabits/s. Sometimes it can go as high as 12. The point is, the box is always fine and I am happy with it. Recently I have started getting more and more DoS and DDoS attacks. They range from very simple syn floods to ICMP echo floods to very odd UDP floods. The problem I am running into is simply that my firewall CPU chokes. It is not because the traffic is high - the line does not become saturdated, and sometimes total traffic can be less than 5 megabits/s - BUT the packets/s count goes way up (sometimes by a factor of 50) and because all of these packets have to go through my entire ruleset, the firewalls CPU chokes. It does not crash, it simply stops forwarding any traffic, effectively blackholing my entire network. As soon as the attack is stopped, the firewall is fine. --- I have responded by doing a ton of research, testing, reading ... and so on. Blocking obviously bad packets, rate limiting ICMP echo responses, rate limiting TCP RSTs - I admit I have a long way to go before I exhaust the bags of tricks that people have to improve their FreeBSD+ipfw firewalls. But every time I improve the ruleset, something new comes along - something that gets through the nets and once again, I have some weird attack at 12-15K packets/second traversing all 400 of my firewall rules (because they don't match anything until the end when they are allowed through) and choking my firewall up. It is very frustrating because the attacks are small in terms of bandwidth - never more than 10 megabits/s. It is simply too many packets/s and not a sophisticated enough ruleset to keep the high-rate garbage from traversing every rule. --- So my questions are as follows: 1. Am I wasting my time trying to make my FreeBSD+ipfw firewall more resilient and sophisticated ? Again, I have probably only scratched the surface, but let's say I emerge from my office 12 months from now having memorized the ipfw source code and having learned _everything_ there is to learn about this problem - will I simply conclude that FreeBSD+ipfw is not good enough and I just need to go get an appliance ? 2. I happen to like a host-based firewall (a firewall running on a normal user OS like FreeBSD) better than an appliance. You get to do anything you need with it, you have a full compliment of unix tools like grep and awk and tcpdump and expect, etc. - it seems like you have more control. Assuming (for a moment) that performance were equal, does anyone else feel this way ? Does anyone else prefer a normal system for a firewall over, say, a PIX ? 3. I am not that high profile ... but what do the high profile (shell servers like foonet and EFnet irc server operators) people use ? Would any of those people consider even for a moment using a FreeBSD+ipfw system for their packet filtering and rate shaping ? I just want to know if I should give up now and shell out a few grand for an appliance, or if it is reasonable for me to attempt to protect a network of my size. Thank you very much.
Hello all. --- Where we are --- Currently, I've got a single site, colocated on the East Coast. Currently, I've got two NetApps at that site, one serving as a mailspool, the other serving as a location for web documents. This system works via NFS to a fair number of mail and web servers, and it's running happily. --- Where I'm going --- What I seek is some help on implementing a second site, and the link between the two. The sites will be more or less the same in terms of the equipment in them, or so I hope. I want to be able to have the changes made at one site replicated to the other site transparently. That is, if I update a file at site A, I want to be able to see the changes at Site B in a reasonable period of time (i.e., short), and without having to manually move data around. I specifically want to do this for allow both sites to offer the same mailspool, so that customers can check their mail at either site. I am in the planning phase of bringing up a second site, and at this site, there will be more web servers, and more mail servers. There will also be an additional netapp for each of mail and www. Between the pair of mail netapps (and to a lesser degree, the www netapps), I want them to replicate changes to the other one. That is, if a file is removed on Mail.NetApp A, it should also disappear on Mail.NetApp B. And if a file is created on netapp B, it should also come into existance on netapp A. Bidirectional updates. My current setup consists of Linux and FreeBSD systems, and F740 NetApps. And yes, there is a lot of pressure to stay with the NetApps. Any hints, or advice will be much appreciated. Thanks in advance, Gabriel -- Gabriel Cain www.dialupusa.net Unix Systems Administrator gabriel@dialupusa.net Dialup USA, Inc. 888-460-2286 ext 208 "Your Virtual ISP Solution" "The most difficult thing in the world is to know how to do a thing and to watch someone else do it wrong without comment." -- Theodore H. White
On Thu, Jan 16, 2003 at 03:45:30PM -0800, Gabriel wrote:
Hello all.
--- Where we are ---
Currently, I've got a single site, colocated on the East Coast. Currently, I've got two NetApps at that site, one serving as a mailspool, the other serving as a location for web documents. This system works via NFS to a fair number of mail and web servers, and it's running happily.
--- Where I'm going ---
What I seek is some help on implementing a second site, and the link between the two. The sites will be more or less the same in terms of the equipment in them, or so I hope.
I want to be able to have the changes made at one site replicated to the other site transparently. That is, if I update a file at site A, I want to be able to see the changes at Site B in a reasonable period of time (i.e., short), and without having to manually move data around. I specifically want to do this for allow both sites to offer the same mailspool, so that customers can check their mail at either site.
I am in the planning phase of bringing up a second site, and at this site, there will be more web servers, and more mail servers. There will also be an additional netapp for each of mail and www.
Between the pair of mail netapps (and to a lesser degree, the www netapps), I want them to replicate changes to the other one. That is, if a file is removed on Mail.NetApp A, it should also disappear on Mail.NetApp B. And if a file is created on netapp B, it should also come into existance on netapp A. Bidirectional updates.
My current setup consists of Linux and FreeBSD systems, and F740 NetApps.
And yes, there is a lot of pressure to stay with the NetApps.
Any hints, or advice will be much appreciated.
You probally want something like snapmirror, but when a file is changed/deleted in the mailbox, you hack your pop/imap server to delete the file on the 'master' fileserver. this isn't really on-topic though, imho. - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
On Thu, 16 Jan 2003, Josh Brooks wrote:
3. I am not that high profile ... but what do the high profile (shell servers like foonet and EFnet irc server operators) people use ? Would any of those people consider even for a moment using a FreeBSD+ipfw system for their packet filtering and rate shaping ?
I have run a EFnet irc server with FreeBSD+ipfw on the irc server itself. Very few rules (like TCP syn ratelimiting, ICMP rate limiting, allow irc ports, allow ssh port, drop the rest) and that crummy old machine was able to handle a full 100megabit of spoofed SYN flooding. I am not 100% up to speed as to what people are using on EFnet/IRCnet nowadays but I am under the impression that they're still using the above, ie letting the host protect itself. Sometimes they put a capable router in front of it and let it do some of the limiting. Back then, it wasn't the host that was getting hit worst by the flooding, it was when the spoofed TCP SYNs were replied to by the machine, the upstream Catalyst 5500 with RSMs totally choked on trying to route lookup 10kpps of diverse destinations, of which some were not even in it's full routing table. The above TCP rate limiting etc (make the machine not respond to a lot of pps generated by unverified connections) did a lot of good in leveraging the upstream route lookup problem. After implementing the above I survived several large floods without much trouble and things were great for 3 months. After that the kiddies figured out that they could attack other hosts on the same network or adjacent networks and cause the RSMs to fall over and die and thus achieving their goals anyway. I have no specific suggestions to you in your specific case unfortunately, my experience with FreeBSD+ipfw is limited to the above, but I thought it might give you some insight into some of the problems I faced anyway. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Thu, Jan 16, 2003 at 03:17:44PM -0800, Josh Brooks wrote:
I am looking for comments and suggestions regarding the merits of purpose-built, appliance style firewalls (like a netscreen or Cisco PIX) vs. running ipfw on a commodity server running FreeBSD.
There is really no benefit to purchasing a vendor-built firewall when the real problem is protecting the servers' tcp/ip stacks and the applications above them, as well as all the infrastruture in between (routers, switches, whatever). Do yourself a favor and spend half as much as you would on firewalls and invest in a packet capture infrastructure to identify exactly what types of attacks you are getting. I believe the beta version of ipfilter allows you to specify bpf logic to block packets. So just configure up each *BSD host with bpf-enabled ipf filters that block the traffic you earlier identified with your packet capture infrastructure (and if you are using libpcap based tools, you are probably already using bpf to match on packets). For legitimate attacks, I suggest buying more bandwidth and scaling your infrastructure appropriately. It also helps to report your findings to others, especially the network and security communities, the places of attack origin (even when spread out), and the transit networks involved in passing along the attacks (especially your upstreams). It's also considered nice to block outgoing packets which match the attacks you've seen, even if you believe your infrastructure to be impenetrable. However, if done right or wrong, any vendor-based or commidity *BSD solution can be less or more powerful than any other solution. dre
On Thu, Jan 16, 2003 at 03:17:44PM -0800, Josh Brooks mooed:
Currently, I run a FreeBSD firewall running ipfw (500 mhz celeron, 256 mags ram). This machine does nothing - runs no services but ssh, and simply sits at my network border doing packet filtering. I have a lot of hosts (four /24s - about 500 active IPs) behind this firewall, and
The problem I am running into is simply that my firewall CPU chokes. It is not because the traffic is high - the line does not become saturdated, and sometimes total traffic can be less than 5 megabits/s - BUT the packets/s count goes way up (sometimes by a factor of 50) and because all
a) Shorten your rules. :-) b) Have you tried ipfw2, or upgraded to 5.0-DR3? (ipfw2 has some known bugs in 4.7-release, but I think it's happy in stable. test, though) c) Have you tried using polling mode for your ethernet device drivers? (options DEVICE_POLLLING, options HZ=1000) Can improve forwarding performance under heavy load/small packets, e.g. a DoS attack
So my questions are as follows:
1. Am I wasting my time trying to make my FreeBSD+ipfw firewall more resilient and sophisticated ? Again, I have probably only scratched the surface, but let's say I emerge from my office 12 months from now having memorized the ipfw source code and having learned _everything_ there is to learn about this problem - will I simply conclude that FreeBSD+ipfw is not good enough and I just need to go get an appliance ?
Not for 12Kpps. For some really sick rate, you might have to go with an (expensive!) appliance. But for what you're seeing, it should be quite feasible to handle with a host. Other questions to check on: What ethernet device are you using? If it's not de or fxp, you're shooting yourself in the foot. -Dave -- work: dga@lcs.mit.edu me: dga@pobox.com MIT Laboratory for Computer Science http://www.angio.net/ I do not accept unsolicited commercial email. Do not spam me.
On Thu, Jan 16, 2003 at 03:17:44PM -0800, user@mail.econolodgetulsa.com said:
I am looking for comments and suggestions regarding the merits of purpose-built, appliance style firewalls (like a netscreen or Cisco PIX) vs. running ipfw on a commodity server running FreeBSD. I am interested only in packet filtering and rate limiting performance - NOT in VPNs or IPsec/crypto considerations.
[snip]
The problem I am running into is simply that my firewall CPU chokes. It is not because the traffic is high - the line does not become saturdated, and sometimes total traffic can be less than 5 megabits/s - BUT the packets/s count goes way up (sometimes by a factor of 50) and because all of these packets have to go through my entire ruleset, the firewalls CPU chokes. It does not crash, it simply stops forwarding any traffic, effectively blackholing my entire network. As soon as the attack is stopped, the firewall is fine.
You may want to look into OpenBSD's new packet filter, pf(4). It's a stateful filter, which, according to pf.conf(8), is usually faster than a rule-based filter: Also, looking up states is usually faster than evaluating rules. If one has 50 rules, all of them are evaluated sequentially in O(n). Even with 50000 states, only 16 comparisons are needed to match a state, since states are stored in a binary search tree that allows searches in O(log2 n). Also, pf in -current now has filtering, NAT, queueing (rule-based bandwidth control - QoS, formerly a separate altq utility), table definitions, normalization (scrubbing), routing, macros and options to the filtering engine. Word is that some of the new queueing features are very spiffy, but that stuff is fairly bleeding edge, so you may want to test that or wait a few months for 3.3 to come out. (Sufficient testing while it's in -current will ensure it is stable enough for 3.3) I'm not a developer, just a satisfied user. I have not personally used the new pf code on anything more heavily trafficked than a home network, but several of the developers use it on fairly high-traffic areas with good results. Worth a look, at least.
2. I happen to like a host-based firewall (a firewall running on a normal user OS like FreeBSD) better than an appliance. You get to do anything you need with it, you have a full compliment of unix tools like grep and awk and tcpdump and expect, etc. - it seems like you have more control. Assuming (for a moment) that performance were equal, does anyone else feel this way ? Does anyone else prefer a normal system for a firewall over, say, a PIX ?
I'm with you on that, mainly for (a) flexibility of configuration, (b) ease/speed of upgrades/patches, and (c) price involved in purchase and maintenance. Also as you mentioned, a firewall that starts out just filtering can later be modified easily to capture packets for analysis later, run active or passive intrusion detection, etc.
3. I am not that high profile ... but what do the high profile (shell servers like foonet and EFnet irc server operators) people use ? Would any of those people consider even for a moment using a FreeBSD+ipfw system for their packet filtering and rate shaping ?
Avleen Vig may be able to give an answer from involvement with the SAFE project, or at least some interesting statistics ... :)
I just want to know if I should give up now and shell out a few grand for an appliance, or if it is reasonable for me to attempt to protect a network of my size.
When the traffic/attacks pass a certain point, my personal feeling is that it's time to distribute the load, rather than look for a more expensive single point of failure. Of course, this is not currently backed up by much personal operational experience, so take that with a grain of salt. :)
Thank you very much.
cheers, -- -= Scott Francis || darkuncle (at) darkuncle (dot) net =- GPG key CB33CCA7 has been revoked; I am now 5537F527 illum oportet crescere me autem minui
On Sat, 18 Jan 2003, Scott Francis wrote:
2. I happen to like a host-based firewall (a firewall running on a normal user OS like FreeBSD) better than an appliance. You get to do anything you need with it, you have a full compliment of unix tools like grep and awk and tcpdump and expect, etc. - it seems like you have more control. Assuming (for a moment) that performance were equal, does anyone else feel this way ? Does anyone else prefer a normal system for a firewall over, say, a PIX ? I'm with you on that, mainly for (a) flexibility of configuration, (b) ease/speed of upgrades/patches, and (c) price involved in purchase and maintenance. Also as you mentioned, a firewall that starts out just filtering can later be modified easily to capture packets for analysis later, run active or passive intrusion detection, etc.
I agree on pretty much all the points there :-)
3. I am not that high profile ... but what do the high profile (shell servers like foonet and EFnet irc server operators) people use ? Would any of those people consider even for a moment using a FreeBSD+ipfw system for their packet filtering and rate shaping ? Avleen Vig may be able to give an answer from involvement with the SAFE project, or at least some interesting statistics ... :)
:-) Thanks! (unfortauntely SAFE has hit a little snag right now and we're looking for some kind body to host our scans for us.. if anyone knows of someone willing to do this, please let me know. It's very low bandwidth / very low complaint generating). My opinion on this is that IPFW sucks for packet filtering. IPFW2 is much better - you can crunch hundreds of rules into just a handful but creating groups of IP addresses and network block. But I agree with Scott that a stateful packet filter like pf on OpenBSD or ipf on FreeBSD is much better at this task. Rate limiting using IPFW during a DoS/DDoS attack is nice if you don't want your router to get overwhelmed trying to route huge numbers of packets. I can let the following advice: On a FreeBSD router, with both IPF and IPFW compiled into the kernel, packets are passed around like this: INTERNET -> IPF -> IPFW+DUMMYNET -> Kernel -> IPF -> IPFW+DUMMYNET -> LAN LAN -> IPF -> IPFW+DUMMYNET -> Kernel -> IPF -> IPFW+DUMMYNET -> INTERNET This has the strong advantage of letting you filter off large numbers of packets before doing your rate limiting. The above combination works very well in my experience, during heavy DoS attacks. DRDoS on the other hand are more tricky. but again, rate limiting to the destination can help with this. With a stateful packet filter like pf/ipf, you can block out all packets where the connection hasn't been established, and only allow in SYN's. Then rate limit your SYN's to a very small number based on your needs.
You may want to look into OpenBSD's new packet filter, pf(4). It's a stateful filter, which, according to pf.conf(8), is usually faster than a rule-based filter:
...
But I agree with Scott that a stateful packet filter like pf on OpenBSD or ipf on FreeBSD is much better at this task.
Don't confuse "stateful" firewalls with "compiled" firewalls. Stateful just means you're maintaining state of established flows, which is behaviorly different from a non-stateful filter. Compiled is when you pre-process a normal ruleset and produce a matching engine which is better suited to doing complex lookups. Some implementations of this include Cisco's "turbo acl", Bill Fumerola's C primitive generation from ipfw rules, Juniper's internal handling of all firewalling, etc. People are trying anything, from adding a few binary trees in your lookup to making a true compiler which produces packet matching code. As I understand OpenBSD's pf (which may not be complete so feel free to point out if I'm wrong), it isn't actually doing anything to compile normal packet lookups, it just added a non-sequential lookup engine for the truely "stateful" filtering that it does. While this is nice and all, it doesn't replace the functionality of normal rule-based filtering, and it isn't the same as a true compiled filter. The closest comparison you could make for the normal readers of this list is that it is the same as speeding up acl matches by enabling the flow route-cache on a Cisco. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Sat, Jan 18, 2003 at 12:29:28PM -0500, ras@e-gerbil.net said: [snip]
As I understand OpenBSD's pf (which may not be complete so feel free to point out if I'm wrong), it isn't actually doing anything to compile normal packet lookups, it just added a non-sequential lookup engine for the truely "stateful" filtering that it does. While this is nice and all, it doesn't replace the functionality of normal rule-based filtering, and
From pf.conf(5): For each packet processed by the packet filter, the filter rules are evaluated in sequential order, from first to last. The last matching rule decides what action is taken. Does this not constitute rule-based filtering? Or am I misunderstanding you? -- -= Scott Francis || darkuncle (at) darkuncle (dot) net =- GPG key CB33CCA7 has been revoked; I am now 5537F527 illum oportet crescere me autem minui
On Sat, Jan 18, 2003 at 03:48:03PM -0800, Scott Francis wrote:
On Sat, Jan 18, 2003 at 12:29:28PM -0500, ras@e-gerbil.net said: [snip]
As I understand OpenBSD's pf (which may not be complete so feel free to point out if I'm wrong), it isn't actually doing anything to compile normal packet lookups, it just added a non-sequential lookup engine for the truely "stateful" filtering that it does. While this is nice and all, it doesn't replace the functionality of normal rule-based filtering, and
From pf.conf(5):
For each packet processed by the packet filter, the filter rules are evaluated in sequential order, from first to last. The last matching rule decides what action is taken.
Does this not constitute rule-based filtering? Or am I misunderstanding you?
Yes and no. That would prove my point, if not for the fact that they are describing the logical processing of a filter ruleset (aka "ipf-style"), not the implementation of the matching engine. But still, the stateful filtering and any lookup model it uses does not negate the need for standard rule-based filtering, and AFAIK pf still does those comparisons sequentially like any other traditional filter. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
AV> Date: Sat, 18 Jan 2003 08:41:15 -0800 (PST) AV> From: Avleen Vig AV> But I agree with Scott that a stateful packet filter like pf AV> on OpenBSD or ipf on FreeBSD is much better at this task. man ipfw /-state <ENTER> Eddy -- Brotsman & Dreger, Inc. - EverQuick Internet Division Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 (785) 865-5885 Lawrence and [inter]national Phone: +1 (316) 794-8922 Wichita ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
On Sat, 18 Jan 2003, Scott Francis wrote:
2. I happen to like a host-based firewall (a firewall running on a normal user OS like FreeBSD) better than an appliance. You get to do anything you need with it, you have a full compliment of unix tools like grep and awk and tcpdump and expect, etc. - it seems like you have more control. Assuming (for a moment) that performance were equal, does anyone else feel this way ? Does anyone else prefer a normal system for a firewall over, say, a PIX ?
I'm with you on that, mainly for (a) flexibility of configuration, (b) ease/speed of upgrades/patches, and (c) price involved in purchase and maintenance. Also as you mentioned, a firewall that starts out just filtering can later be modified easily to capture packets for analysis later, run active or passive intrusion detection, etc.
I'm in total agreement as to the untily and significant headache-reduction that a *bsd os (with real interactive editor makes -- Vi for IOS must be too challenging). However, I do see a sore spot. One area that I've not seen much attention paid to (yet?) is failover. Don't assume that I'm advocating the use of a PIX here, but has anyone yet successfully used ipf/pf to export and then import the state tables on a backup host? In my experience, doing that w/ PIXen has been quite simple. Forget all the ARP/ifconfig/heartbeat fudgery that'd be required to acheive failover on *bsd with ipf/pf -- just finding a simple way to move said state table from host to host seems interesting and challenging. How do we adress availability concerns while using comodity hardware and Os's? Are they valid concerns, even? <G> --Tk
On Sat, 18 Jan 2003, Tony Kapela wrote:
I'm in total agreement as to the untily and significant headache-reduction that a *bsd os (with real interactive editor makes -- Vi for IOS must be too challenging). However, I do see a sore spot. One area that I've not seen much attention paid to (yet?) is failover. Don't assume that I'm advocating the use of a PIX here, but has anyone yet successfully used ipf/pf to export and then import the state tables on a backup host? In my experience, doing that w/ PIXen has been quite simple.
It'd be an interesting challenge to get this working with ipf/pf.
Forget all the ARP/ifconfig/heartbeat fudgery that'd be required to acheive failover on *bsd with ipf/pf -- just finding a simple way to move said state table from host to host seems interesting and challenging.
ipf now has 'ipfs' which can dump and restore the current states table :-)
[Mail-Followup-To points to the pf list] Tony Kapela wrote/schrieb/scripsit:
Forget all the ARP/ifconfig/heartbeat fudgery that'd be required to acheive failover on *bsd with ipf/pf -- just finding a simple way to move said state table from host to host seems interesting and challenging.
OpenBSD's pf is moving there. -current now has the pfsync pseudo- interface that exposes changes to the state table as they happen. A daemon to make use of that for said purpose is expected after the 3.3 release. 'Rumor' says, a non patent-emcumbered vrrp-like mechanism will be available as well. -Stefan -- junior guru SP666-RIPE SMP@{IRC,SILC}
participants (12)
-
Avleen Vig -
David G. Andersen -
dre -
E.B. Dreger -
Gabriel -
Jared Mauch -
Josh Brooks -
Mikael Abrahamsson -
Richard A Steenbergen -
Scott Francis -
Stefan Paletta -
Tony Kapela