Dynamic routing on firewalls.
Hi, We have used dynamic routing on firewall in the old days. We did experience several severe outages due to this setup (OSPF en Cisco). As you will understand i’m not eager to go back to this solution but I am curious about your point of views. Is it advisory to so these days? Kind regards, David
On Thu, Feb 5, 2015 at 4:10 PM, David Jansen <david@nines.nl> wrote:
Hi,
We have used dynamic routing on firewall in the old days. We did experience several severe outages due to this setup (OSPF en Cisco). As you will understand i’m not eager to go back to this solution but I am curious about your point of views.
Is it advisory to so these days?
Any specific firewall in mind? As this depends from vendor to vendor. I've had some issues with OSPF and CheckPoint firewalls when the firewalls would be overloaded and started dropping packets at the interface level causing adjacencies to go down, but I solved this by using BGP instead and the routing issues went away. On Juniper things tend work OK. Other than this, make sure you don't run into asymmetric routing as connections might get dropped because the firewall does not know about them or packets arrive out of order and the firewall cannot reassemble all of them.
It all depends how much of the firewall functionality is implemented in CPU. The biggest problem is that firewalls that implement functionality in software usually saturate CPU when stressed (e.g. DOS) and routing protocols start dropping. I'm a strong believer in having a router that can do basic filtering in hardware (specifically uRPF) as the first line of defense in a DOS attack and then using a firewall behind that to reach your security policy goals. We have a pretty large network so we've expanded the concept of RTBH filtering internally and use a small ISR (currently 1841) to inject routes into our network to disable hosts using uRPF at the first router they hit. The entire thing is scripted and works very well at containing problematic hosts centrally. The other thing to watch out for IMHO is the NGFW hype. I haven't seen a NGFW that can actually do everything it claims to at the same time and meet advertised performance levels. You're much better off splitting up the workload and having a series of components architected to work with each other. On Thu, Feb 5, 2015 at 9:42 AM, Eugeniu Patrascu <eugen@imacandi.net> wrote:
On Thu, Feb 5, 2015 at 4:10 PM, David Jansen <david@nines.nl> wrote:
Hi,
We have used dynamic routing on firewall in the old days. We did experience several severe outages due to this setup (OSPF en Cisco). As you will understand i'm not eager to go back to this solution but I am curious about your point of views.
Is it advisory to so these days?
Any specific firewall in mind? As this depends from vendor to vendor.
I've had some issues with OSPF and CheckPoint firewalls when the firewalls would be overloaded and started dropping packets at the interface level causing adjacencies to go down, but I solved this by using BGP instead and the routing issues went away.
On Juniper things tend work OK.
Other than this, make sure you don't run into asymmetric routing as connections might get dropped because the firewall does not know about them or packets arrive out of order and the firewall cannot reassemble all of them.
-- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net
Hi Ray On 05 Feb 2015, at 15:51, Ray Soucy <rps@maine.edu<mailto:rps@maine.edu>> wrote: You're much better off splitting up the workload and having a series of components architected to work with each other. Especially in case of datacenter- or enterprise solutions i do agree. Thanks
Hi Eugeniu, On 05 Feb 2015, at 15:42, Eugeniu Patrascu <eugen@imacandi.net<mailto:eugen@imacandi.net>> wrote: Any specific firewall in mind? As this depends from vendor to vendor. We are using Cisco (ASA). I've had some issues with OSPF and CheckPoint firewalls when the firewalls would be overloaded and started dropping packets at the interface level causing adjacencies to go down, but I solved this by using BGP instead and the routing issues went away. The last time we were working with OSPF and Cisco was on a fwsm (cisco pix blade). Interesting to know that more vendors do have problems with OSPF on firewalls. Also good to hear that BGP seemed to have solved your problem. Kind regards, David
On 2/5/2015 9:42 AM, Eugeniu Patrascu wrote:
On Juniper things tend work OK. Other than this, make sure you don't run into asymmetric routing as connections might get dropped because the firewall does not know about them or packets arrive out of order and the firewall cannot reassemble all of them.
Agreed. Assymmetric routing is not your friend unless you plan accordingly. I use OSPF and BGP quite a bit on Juniper SRX. Works great.
Hi, We are running Juniper SRX5000 family with around 40ish routing-instances, most of them using OSPFv2 without any issues. The RIBs are not too big, just a couple of them with thousands routes. I know that some guys are testing a similar environment on Fortigates and I'm not aware of any issues with routing so far. We also have SRX's running BGP+BFD (srx240) and again no issues at all. As Eugeniu mentioned, just be careful with the asymmetric routing, then is straight forward. Hope it helps. Santiago On Thu, Feb 5, 2015 at 2:42 PM, Eugeniu Patrascu <eugen@imacandi.net> wrote:
On Thu, Feb 5, 2015 at 4:10 PM, David Jansen <david@nines.nl> wrote:
Hi,
We have used dynamic routing on firewall in the old days. We did experience several severe outages due to this setup (OSPF en Cisco). As you will understand i’m not eager to go back to this solution but I am curious about your point of views.
Is it advisory to so these days?
Any specific firewall in mind? As this depends from vendor to vendor.
I've had some issues with OSPF and CheckPoint firewalls when the firewalls would be overloaded and started dropping packets at the interface level causing adjacencies to go down, but I solved this by using BGP instead and the routing issues went away.
On Juniper things tend work OK.
Other than this, make sure you don't run into asymmetric routing as connections might get dropped because the firewall does not know about them or packets arrive out of order and the firewall cannot reassemble all of them.
Hi David, a router is a router and a firewall is a firewall. Especially a Cisco ASA is no router, period. A router in front of the firewall is my choice, it also keeps broadcasts from the firewall + can do uRPF. rm
Some Juniper models actually do a very good job of being both. In reality, a Firewall _IS_ a router, even if it's a bad one. Anything that moves packets from one interface to another is a router. Of course, the support for routing protocols is a useful feature in a router and one of the areas where firewalls often fall short. Where you want to put things (in front, behind, etc.) really depends on your topology and the problem you are trying to solve. Personally, I like to keep the firewalls as close to the end hosts as possible. This tends to greatly simplify security policies and make them MUCH easier (and more reliable) to audit. Owen
On Feb 5, 2015, at 2:49 PM, Ralph J.Mayer <rmayer@nerd-residenz.de> wrote:
Hi David,
a router is a router and a firewall is a firewall.
Especially a Cisco ASA is no router, period.
A router in front of the firewall is my choice, it also keeps broadcasts from the firewall + can do uRPF.
rm
On Feb 5, 2015, at 2:49 PM, Ralph J.Mayer <rmayer@nerd-residenz.de> wrote: a router is a router and a firewall is a firewall. Especially a Cisco ASA is no router, period.
Man-o-man did I find that out when we had to renumber our network after we got bought by the French. Oh, I'll just pop on a secondary address on this interface... What? Needed to go through fits just to get a hairpin route in the thing. The ASA series is good at what it does, just don't plan on it acting like router IOS. -- Joe Hamelin, W7COM, Tulalip, WA, 360-474-7474
On Thu, February 5, 2015 20:02, Joe Hamelin wrote:
On Feb 5, 2015, at 2:49 PM, Ralph J.Mayer <rmayer@nerd-residenz.de> wrote: a router is a router and a firewall is a firewall. Especially a Cisco ASA is no router, period.
Man-o-man did I find that out when we had to renumber our network after we got bought by the French.
Oh, I'll just pop on a secondary address on this interface... What?
Needed to go through fits just to get a hairpin route in the thing.
The ASA series is good at what it does, just don't plan on it acting like router IOS.
Sorry, but I'm with Owen. Square : Rectangle :: Firewall : Router A firewall is a router, despite how much so many security folk try to deny it. And firewalls that seem to try to intentionally be crappy routers (ie, ASAs) have no place in my network. If it can't be a decent router, then its going to suck as a firewall too, because a firewall has to be able to play nice with the rest of the network, and if they can't do that, then I have no use for them. I'll get a firewall that does. -- Jeff
Just because a cat has kittens in the oven, you don't call them biscuits. A firewall can route, but it is not a router. Both have specialized tasks. You can fix a car with a swiss army knife, but why would you want to? -- Bill Thompson billt@mahagonny.com On February 5, 2015 7:19:43 PM PST, Jeff McAdams <jeffm@iglou.com> wrote:
On Thu, February 5, 2015 20:02, Joe Hamelin wrote:
On Feb 5, 2015, at 2:49 PM, Ralph J.Mayer <rmayer@nerd-residenz.de> wrote: a router is a router and a firewall is a firewall. Especially a Cisco ASA is no router, period.
Man-o-man did I find that out when we had to renumber our network after we got bought by the French.
Oh, I'll just pop on a secondary address on this interface... What?
Needed to go through fits just to get a hairpin route in the thing.
The ASA series is good at what it does, just don't plan on it acting like router IOS.
Sorry, but I'm with Owen.
Square : Rectangle :: Firewall : Router
A firewall is a router, despite how much so many security folk try to deny it. And firewalls that seem to try to intentionally be crappy routers (ie, ASAs) have no place in my network.
If it can't be a decent router, then its going to suck as a firewall too, because a firewall has to be able to play nice with the rest of the network, and if they can't do that, then I have no use for them. I'll get a firewall that does.
A good firewall can also be a good router. Of course you can find firewalls that are crappy routers and you can find routers that are crappy firewalls, but generally, the two are not mutually exclusive. Owen
On Feb 6, 2015, at 08:39 , Bill Thompson <Billt@mahagonny.com> wrote:
Just because a cat has kittens in the oven, you don't call them biscuits. A firewall can route, but it is not a router. Both have specialized tasks. You can fix a car with a swiss army knife, but why would you want to? -- Bill Thompson billt@mahagonny.com
On February 5, 2015 7:19:43 PM PST, Jeff McAdams <jeffm@iglou.com> wrote:
On Thu, February 5, 2015 20:02, Joe Hamelin wrote:
On Feb 5, 2015, at 2:49 PM, Ralph J.Mayer <rmayer@nerd-residenz.de> wrote: a router is a router and a firewall is a firewall. Especially a Cisco ASA is no router, period.
Man-o-man did I find that out when we had to renumber our network after we got bought by the French.
Oh, I'll just pop on a secondary address on this interface... What?
Needed to go through fits just to get a hairpin route in the thing.
The ASA series is good at what it does, just don't plan on it acting like router IOS.
Sorry, but I'm with Owen.
Square : Rectangle :: Firewall : Router
A firewall is a router, despite how much so many security folk try to deny it. And firewalls that seem to try to intentionally be crappy routers (ie, ASAs) have no place in my network.
If it can't be a decent router, then its going to suck as a firewall too, because a firewall has to be able to play nice with the rest of the network, and if they can't do that, then I have no use for them. I'll get a firewall that does.
Of course you can find firewalls that are crappy routers and you can find routers that are crappy firewalls, but generally, the two are not mutually exclusive.
I completely disagree w/ such or similar statements. On the vendor datasheet it says different. On books it says different. And on real life it's different. Firewalls are firewalls. Routers are routers. Routers should do some very basic filtering (stateles, ACLs, data plane protection...) and firewalls should do basic static routing. And things should not go far beyond that. If you keep thinking like that you will soon believe an L3 switch is a firewall too. Firewalls and routers belong to different places in a serious topology. Only small networks should have both functions in the same box. It raises risks, makes different kernel tasks competing to each other for the same resources. You may run out of states, memory and CPU specially if mixing NAT & tunneling beyond firewalling and routing. A router nowadays has many tasks to accomplish, from 6to4, dual stacking, to multiple routing services (bgp, ospf, bfd). Don't add extra duties to the box. Multiple purpose systems that can act like both things (say, a Linux box), but it's just not right to have more than one critical service in the same box. They should be distributed along your network. A firewall in front of the router, a firewall after the router in front of the servers. I just had a huge problem with an engineer who decided that a router should be his CGN, and when the number of translated sessions run above the expected and planned capacity, the box just sit down unresponsive. All of this company (and it's a banking company, not an ISP who just pays some SLA debit and it's good to go) connectivity was offline due to this confusion of service profiles on the same box, and all, means servers and hosts with registered IP addresses, not only RFC1918 addresses that needed to be translated. We just split the functions, distributed firewall and CGN to different boxes and topologies in a much more logical way and the "auto DoS feature" just went away. So, please, don't insist. A firewall is a firewall. A router is a router. A translation box is another alien. Unless you are SMB or willing to pay over dimensioned boxes to mix all duties up together, which will be more expensive than distributing the services alongside the network.
Owen
On Feb 6, 2015, at 08:39 , Bill Thompson <Billt@mahagonny.com> wrote:
Just because a cat has kittens in the oven, you don't call them biscuits. A firewall can route, but it is not a router. Both have specialized tasks. You can fix a car with a swiss army knife, but why would you want to? -- Bill Thompson billt@mahagonny.com
On February 5, 2015 7:19:43 PM PST, Jeff McAdams <jeffm@iglou.com> wrote:
On Thu, February 5, 2015 20:02, Joe Hamelin wrote:
On Feb 5, 2015, at 2:49 PM, Ralph J.Mayer <rmayer@nerd-residenz.de> wrote: a router is a router and a firewall is a firewall. Especially a Cisco ASA is no router, period.
Man-o-man did I find that out when we had to renumber our network after we got bought by the French.
Oh, I'll just pop on a secondary address on this interface... What?
Needed to go through fits just to get a hairpin route in the thing.
The ASA series is good at what it does, just don't plan on it acting like router IOS.
Sorry, but I'm with Owen.
Square : Rectangle :: Firewall : Router
A firewall is a router, despite how much so many security folk try to deny it. And firewalls that seem to try to intentionally be crappy routers (ie, ASAs) have no place in my network.
If it can't be a decent router, then its going to suck as a firewall too, because a firewall has to be able to play nice with the rest of the network, and if they can't do that, then I have no use for them. I'll get a firewall that does.
You're missing the point. I would never advocate for trying to deploy a Juniper MX in the role of a firewall to provide a security boundary. I would never try to deploy a Juniper SRX to provide a huge number of GRE tunnel terminations or other sorts of aggregations of large numbers of connections or however you might describe a typical router role. But that SRX does have different IP addresses in different networks on its various interfaces, and when traffic transits through the SRX, it looks at the layer 3 addressing information to determine where the traffic needs to be sent. Ergo, it is a router. You can deny it all you want, but you're only shooting yourself in the foot by not acknowledging that and dealing with its implications. And as a router, if a vendor wants me to put it on my network, it better dern well be a well behaved router to begin with, and in my network, that means OSPF and BGP. Only once it behaves as a router can its efficacy as a firewall really be considered. I completely agree that you don't want to overload any particular device with too many functions. I've got MXes that terminate a large number of GRE tunnels, but I've also SRXes terminating a large number of IPSec tunnels that are basically acting as routers because they can handle the large quantity of crypto operations involved better than an MX. But while the SRXes that terminate the large number of IPSec tunnels do some amount of firewalling, and I only did that grudgingly because of financial reasons. The firewalling will probably be moved off to a separate set of SRXes as this project grows. -- Jeff On Sun, February 8, 2015 08:40, BPNoC Group wrote:
Of course you can find firewalls that are crappy routers and you can find routers that are crappy firewalls, but generally, the two are not mutually exclusive.
I completely disagree w/ such or similar statements. On the vendor datasheet it says different. On books it says different. And on real life it's different.
Firewalls are firewalls. Routers are routers. Routers should do some very basic filtering (stateles, ACLs, data plane protection...) and firewalls should do basic static routing. And things should not go far beyond that.
If you keep thinking like that you will soon believe an L3 switch is a firewall too.
Firewalls and routers belong to different places in a serious topology.
Only small networks should have both functions in the same box. It raises risks, makes different kernel tasks competing to each other for the same resources. You may run out of states, memory and CPU specially if mixing NAT & tunneling beyond firewalling and routing. A router nowadays has many tasks to accomplish, from 6to4, dual stacking, to multiple routing services (bgp, ospf, bfd). Don't add extra duties to the box.
Multiple purpose systems that can act like both things (say, a Linux box), but it's just not right to have more than one critical service in the same box. They should be distributed along your network. A firewall in front of the router, a firewall after the router in front of the servers.
I just had a huge problem with an engineer who decided that a router should be his CGN, and when the number of translated sessions run above the expected and planned capacity, the box just sit down unresponsive. All of this company (and it's a banking company, not an ISP who just pays some SLA debit and it's good to go) connectivity was offline due to this confusion of service profiles on the same box, and all, means servers and hosts with registered IP addresses, not only RFC1918 addresses that needed to be translated.
We just split the functions, distributed firewall and CGN to different boxes and topologies in a much more logical way and the "auto DoS feature" just went away.
So, please, don't insist. A firewall is a firewall. A router is a router. A translation box is another alien. Unless you are SMB or willing to pay over dimensioned boxes to mix all duties up together, which will be more expensive than distributing the services alongside the network.
Owen
On Feb 6, 2015, at 08:39 , Bill Thompson <Billt@mahagonny.com> wrote:
Just because a cat has kittens in the oven, you don't call them
biscuits. A firewall can route, but it is not a router. Both have specialized tasks. You can fix a car with a swiss army knife, but why would you want to?
-- Bill Thompson billt@mahagonny.com
On February 5, 2015 7:19:43 PM PST, Jeff McAdams <jeffm@iglou.com>
wrote:
On Thu, February 5, 2015 20:02, Joe Hamelin wrote:
On Feb 5, 2015, at 2:49 PM, Ralph J.Mayer <rmayer@nerd-residenz.de> wrote: a router is a router and a firewall is a firewall. Especially a Cisco ASA
is no router, period.
Man-o-man did I find that out when we had to renumber our network
after
we got bought by the French.
Oh, I'll just pop on a secondary address on this interface... What?
Needed to go through fits just to get a hairpin route in the thing.
The ASA series is good at what it does, just don't plan on it acting like router IOS.
Sorry, but I'm with Owen.
Square : Rectangle :: Firewall : Router
A firewall is a router, despite how much so many security folk try to deny it. And firewalls that seem to try to intentionally be crappy routers (ie, ASAs) have no place in my network.
If it can't be a decent router, then its going to suck as a firewall too, because a firewall has to be able to play nice with the rest of the network, and if they can't do that, then I have no use for them. I'll get a firewall that does.
-- Jeff
On Sun, Feb 8, 2015 at 12:48 PM, Jeff McAdams <jeffm@iglou.com> wrote:
You're missing the point.
I'm not missing, I'm just diverting the point. As I mentioned from a Linux box example, the fact that it can both act as a router and a firewall does not mean it should. I disagree with the simplistic idea that if a firewall L3 forwards, it's a router, or if a router has ACLs capabilities, it's a firewall. Someone just illustrated how a mission-critical placed firewall protecting a BGP router may do it bridged, without actually routing not a single extra hop.
I would never advocate for trying to deploy a Juniper MX in the role of a firewall to provide a security boundary. I would never try to deploy a Juniper SRX to provide a huge number of GRE tunnel terminations or other sorts of aggregations of large numbers of connections or however you might describe a typical router role.
So we agree! I completely agree that you don't want to overload any particular device
with too many functions. I've got MXes that terminate a large number of GRE tunnels, but I've also SRXes terminating a large number of IPSec tunnels that are basically acting as routers because they can handle the large quantity of crypto operations involved better than an MX. But while the SRXes that terminate the large number of IPSec tunnels do some amount of firewalling, and I only did that grudgingly because of financial reasons.
Yes, I understand budget restrictions sometimes takes to accumulating functions on the same box. But the notion that matters is that although a firewall *can* be, technically, implemented in the same node, it just belongs to somewhere else, in a distributed / separed box.
The firewalling will probably be moved off to a separate set of SRXes as this project grows.
Yeah, in the end we mostly agree.
-- Jeff
On Sun, February 8, 2015 08:40, BPNoC Group wrote:
Of course you can find firewalls that are crappy routers and you can find routers that are crappy firewalls, but generally, the two are not mutually exclusive.
I completely disagree w/ such or similar statements. On the vendor datasheet it says different. On books it says different. And on real life it's different.
Firewalls are firewalls. Routers are routers. Routers should do some very basic filtering (stateles, ACLs, data plane protection...) and firewalls should do basic static routing. And things should not go far beyond that.
If you keep thinking like that you will soon believe an L3 switch is a firewall too.
Firewalls and routers belong to different places in a serious topology.
Only small networks should have both functions in the same box. It raises risks, makes different kernel tasks competing to each other for the same resources. You may run out of states, memory and CPU specially if mixing NAT & tunneling beyond firewalling and routing. A router nowadays has many tasks to accomplish, from 6to4, dual stacking, to multiple routing services (bgp, ospf, bfd). Don't add extra duties to the box.
Multiple purpose systems that can act like both things (say, a Linux box), but it's just not right to have more than one critical service in the same box. They should be distributed along your network. A firewall in front of the router, a firewall after the router in front of the servers.
I just had a huge problem with an engineer who decided that a router should be his CGN, and when the number of translated sessions run above the expected and planned capacity, the box just sit down unresponsive. All of this company (and it's a banking company, not an ISP who just pays some SLA debit and it's good to go) connectivity was offline due to this confusion of service profiles on the same box, and all, means servers and hosts with registered IP addresses, not only RFC1918 addresses that needed to be translated.
We just split the functions, distributed firewall and CGN to different boxes and topologies in a much more logical way and the "auto DoS feature" just went away.
So, please, don't insist. A firewall is a firewall. A router is a router. A translation box is another alien. Unless you are SMB or willing to pay over dimensioned boxes to mix all duties up together, which will be more expensive than distributing the services alongside the network.
Owen
On Feb 6, 2015, at 08:39 , Bill Thompson <Billt@mahagonny.com> wrote:
Just because a cat has kittens in the oven, you don't call them
biscuits. A firewall can route, but it is not a router. Both have specialized tasks. You can fix a car with a swiss army knife, but why would you want to?
-- Bill Thompson billt@mahagonny.com
On February 5, 2015 7:19:43 PM PST, Jeff McAdams <jeffm@iglou.com>
wrote:
On Thu, February 5, 2015 20:02, Joe Hamelin wrote:
> On Feb 5, 2015, at 2:49 PM, Ralph J.Mayer > <rmayer@nerd-residenz.de> > wrote: > a router is a router and a firewall is a firewall. Especially a Cisco ASA
> is no router, period.
Man-o-man did I find that out when we had to renumber our network
after
we got bought by the French.
Oh, I'll just pop on a secondary address on this interface... What?
Needed to go through fits just to get a hairpin route in the thing.
The ASA series is good at what it does, just don't plan on it acting like router IOS.
Sorry, but I'm with Owen.
Square : Rectangle :: Firewall : Router
A firewall is a router, despite how much so many security folk try to deny it. And firewalls that seem to try to intentionally be crappy routers (ie, ASAs) have no place in my network.
If it can't be a decent router, then its going to suck as a firewall too, because a firewall has to be able to play nice with the rest of the network, and if they can't do that, then I have no use for them. I'll get a firewall that does.
-- Jeff
On Feb 8, 2015, at 05:40 , BPNoC Group <bpnoc.lists@gmail.com> wrote:
Of course you can find firewalls that are crappy routers and you can find routers that are crappy firewalls, but generally, the two are not mutually exclusive.
I completely disagree w/ such or similar statements. On the vendor datasheet it says different. On books it says different. And on real life it's different.
No, really it does not.
Firewalls are firewalls. Routers are routers. Routers should do some very basic filtering (stateles, ACLs, data plane protection...) and firewalls should do basic static routing. And things should not go far beyond that.
We can agree to disagree.
If you keep thinking like that you will soon believe an L3 switch is a firewall too.
An L3 switch is just another kind of router and if it’s got the ability for its switching matrix to include a packet classifier that can be preprogrammed for the appropriate firewall functions at line rate in hardware, then, yes, it’s a perfectly fine firewall, and, probably about the only solution that’s really going to work in a high line-rate scenario, actually.
Firewalls and routers belong to different places in a serious topology.
You and I apparently have very different ideas of serous topologies.
Only small networks should have both functions in the same box. It raises risks, makes different kernel tasks competing to each other for the same resources. You may run out of states, memory and CPU specially if mixing NAT & tunneling beyond firewalling and routing. A router nowadays has many tasks to accomplish, from 6to4, dual stacking, to multiple routing services (bgp, ospf, bfd). Don't add extra duties to the box.
If you are firewalling so far away from the edge that any of this matters, you have already lost and your topology is very hard to consider “serious” in my opinion.
Multiple purpose systems that can act like both things (say, a Linux box), but it's just not right to have more than one critical service in the same box. They should be distributed along your network. A firewall in front of the router, a firewall after the router in front of the servers.
I’m thinking more like a large Juniper with an ESPIC or other services interface hardware solution.
I just had a huge problem with an engineer who decided that a router should be his CGN, and when the number of translated sessions run above the expected and planned capacity, the box just sit down unresponsive. All of this company (and it's a banking company, not an ISP who just pays some SLA debit and it's good to go) connectivity was offline due to this confusion of service profiles on the same box, and all, means servers and hosts with registered IP addresses, not only RFC1918 addresses that needed to be translated.
You can always choose the wrong box for the job. I bet I can point to plenty of routers that could have handled his CGN needs just fine and had plenty of memory to hold all of his translated sessions. This is no different than if he chose an incorrect CGN box that was purpose-built. Your example is like saying “The 2514 was not adequate as a 100Mbps firewall, so all routers are inadequate as firewalls”. The 2514 was not adequate or even capable of being a 100Mbps router.
We just split the functions, distributed firewall and CGN to different boxes and topologies in a much more logical way and the "auto DoS feature" just went away.
That’s certainly one viable solution. Maybe even the right one for that particular space. However, it does not change anything I said.
So, please, don't insist. A firewall is a firewall. A router is a router. A translation box is another alien. Unless you are SMB or willing to pay over dimensioned boxes to mix all duties up together, which will be more expensive than distributing the services alongside the network.
Technically, a router is any device which takes an IP datagram on one interface and delivers it to an interface with a different network number (whether the same (hairpin) or another interface) after decrementing the TTL or Hop Count (depending on whether IPv4 or IPv6). Other than the (rather silly in virtually all circumstances) Layer 2 firewalls mentioned earlier, every firewall is technically a router. Not every router is a firewall, though there are plenty of routers that are also very capable firewalls. I will grant you that there are virtually no purpose-built firewalls that make good routers, but that’s yet another issue truly unrelated to what I said. As to translation devices, well, those also have no place in a serious topology other than dealing with limitations of an aging and hopefully soon to be deprecated protocol that should have been obsoleted years ago. Owen
Owen
On Feb 6, 2015, at 08:39 , Bill Thompson <Billt@mahagonny.com> wrote:
Just because a cat has kittens in the oven, you don't call them biscuits. A firewall can route, but it is not a router. Both have specialized tasks. You can fix a car with a swiss army knife, but why would you want to? -- Bill Thompson billt@mahagonny.com
On February 5, 2015 7:19:43 PM PST, Jeff McAdams <jeffm@iglou.com> wrote:
On Thu, February 5, 2015 20:02, Joe Hamelin wrote:
On Feb 5, 2015, at 2:49 PM, Ralph J.Mayer <rmayer@nerd-residenz.de> wrote: a router is a router and a firewall is a firewall. Especially a Cisco ASA is no router, period.
Man-o-man did I find that out when we had to renumber our network after we got bought by the French.
Oh, I'll just pop on a secondary address on this interface... What?
Needed to go through fits just to get a hairpin route in the thing.
The ASA series is good at what it does, just don't plan on it acting like router IOS.
Sorry, but I'm with Owen.
Square : Rectangle :: Firewall : Router
A firewall is a router, despite how much so many security folk try to deny it. And firewalls that seem to try to intentionally be crappy routers (ie, ASAs) have no place in my network.
If it can't be a decent router, then its going to suck as a firewall too, because a firewall has to be able to play nice with the rest of the network, and if they can't do that, then I have no use for them. I'll get a firewall that does.
On Sun, Feb 08, 2015 at 11:40:56AM -0200, BPNoC Group wrote:
Firewalls are firewalls. Routers are routers. Routers should do some very basic filtering (stateles, ACLs, data plane protection...) and firewalls should do basic static routing. And things should not go far beyond that.
This is, at a network level, an echo of the "Software Tools" philosophy that has served us exceedingly well for decades. Tools should do one thing, they should do it well, and if/when we need to do more than one thing, we should use tools in combination. There's another advantage to this: if firewalls and routers &etc are not the same system, then they can run different software on different operating systems on different architectures -- providing a significant measure of insulation against attacks unique to one particular combination. ---rsk
On Mon, Feb 9, 2015 at 10:59 AM, Rich Kulawiec <rsk@gsp.org> wrote:
On Sun, Feb 08, 2015 at 11:40:56AM -0200, BPNoC Group wrote:
Firewalls are firewalls. Routers are routers. Routers should do some very basic filtering (stateles, ACLs, data plane protection...) and firewalls should do basic static routing. And things should not go far beyond that.
This is, at a network level, an echo of the "Software Tools" philosophy that has served us exceedingly well for decades. Tools should do one thing, they should do it well, and if/when we need to do more than one thing, we should use tools in combination.
And then reality comes and disagrees with you :) I am a fan of the "use the right tool for the right job", but it is not always possible due to economical/technical/political reasons. I had situations where running dynamic routing on firewalls was the way to go to allow for geographic distribution of traffic without having to touch routers and/or firewalls when adding/deleting subnets. Devices would just learn routes and if permitted by the firewalls, traffic would pass.
There's another advantage to this: if firewalls and routers &etc are not the same system, then they can run different software on different operating systems on different architectures -- providing a significant measure of insulation against attacks unique to one particular combination.
This is a bit of a fallacy, because considering all things equal, a router looks at only Layer 3/4 headers to route a packet, whereby a firewall will look more deeper up the stack (considering a simple scenario, not considering MPLS stuff). Even if they run the same OS but with different functions enabled, a firewall having a vulnerability because it mishandles TCP packets with SYN/RST flags set, it does not mean it will be vulnerable as a router. I know companies running firewall back to back from different vendors just to make sure that they are secure if someone "hacks" one of the firewalls. My point is that: 1) you can run dynamic routing on a firewall without issues 2) it depends on the situation if it advisable to do so 3) there is no size fits all scenario whereby it is verboten to have anything else than static routes on a firewall 4) you have to consider the pros/cons about doing it or not doing it
Hello,
Some Juniper models actually do a very good job of being both.
In reality, a Firewall _IS_ a router, even if it's a bad one. Anything that moves packets from one interface to another is a router.
Technically it’s quite not a precise assumption. While routing is much likely an IP core need and OSI Layer 3 related mechanism, a firewall does not have to basic L3 forwarding. It can be a bridged/bright firewall, may fit in front of a router, protecting it, and carrying. Not routing anything. In fact in a fail-safe scenario (from availability perspective) a bridged firewall may be shut down completely and a Bypass por pair taking place will keep traffic flowing, “moving packets from one port to another” without actually ever been a router. I have recently added netmap-ipfw in front of BGP routers protecting ‘em from DDoS attacks, adding line-rate firewalling capabilities to a commodity x86/64 box or a T5 card for 10GbE/40GbE, and netmap-ipfw itself acts like mentioned, passing packets back and forth between interfaces without ever routing anything.
Of course, the support for routing protocols is a useful feature in a router and one of the areas where firewalls often fall short.
Where you want to put things (in front, behind, etc.) really depends on your topology and the problem you are trying to solve.
Personally, I like to keep the firewalls as close to the end hosts as possible. This tends to greatly simplify security policies and make them MUCH easier (and more reliable) to audit.
I agree. A firewall belongs better closer to the end hosts being protected. Maybe protection of the router is the only exception when RTBH will not fit the task (or just won’t be enough). Therefore, close to the end host usually means far from the core routers. Unless one is really considering a CPE device doing poorly jobs of “a router and a firewall”... -- Patrick Tracanelli
On Feb 8, 2015, at 06:02 , Patrick Tracanelli <eksffa@freebsdbrasil.com.br> wrote:
Hello,
Some Juniper models actually do a very good job of being both.
In reality, a Firewall _IS_ a router, even if it's a bad one. Anything that moves packets from one interface to another is a router.
Technically it’s quite not a precise assumption. While routing is much likely an IP core need and OSI Layer 3 related mechanism, a firewall does not have to basic L3 forwarding. It can be a bridged/bright firewall, may fit in front of a router, protecting it, and carrying. Not routing anything. In fact in a fail-safe scenario (from availability perspective) a bridged firewall may be shut down completely and a Bypass por pair taking place will keep traffic flowing, “moving packets from one port to another” without actually ever been a router.
Technically true, but bridged firewalls are pretty much passe these days in the real world. As a general rule, when the firewall is shut down, one usually doesn’t want the packets flowing past un-hindered. The fact that this is kind of the default of what happens with bridged firewalls is just one of the many reasons hardly anyone still uses such a thing.
I have recently added netmap-ipfw in front of BGP routers protecting ‘em from DDoS attacks, adding line-rate firewalling capabilities to a commodity x86/64 box or a T5 card for 10GbE/40GbE, and netmap-ipfw itself acts like mentioned, passing packets back and forth between interfaces without ever routing anything.
Sure, there are all kinds of things one can do. Some of them are good ideas, many of them are not. If it works in your environment and you’re OK with the failure modes and other tradeoffs, then more power to you.
Of course, the support for routing protocols is a useful feature in a router and one of the areas where firewalls often fall short.
Where you want to put things (in front, behind, etc.) really depends on your topology and the problem you are trying to solve.
Personally, I like to keep the firewalls as close to the end hosts as possible. This tends to greatly simplify security policies and make them MUCH easier (and more reliable) to audit.
I agree. A firewall belongs better closer to the end hosts being protected. Maybe protection of the router is the only exception when RTBH will not fit the task (or just won’t be enough).
DDoS mitigation on site is a questionable and usually losing proposition at best. Other than DDoS mitigation, any good router should be perfectly capable of protecting itself. For protecting a router from DDoS that it cannot properly protect itself, one needs to be able to control or alter the delivery of packets across the upstream link from the upstream side anyway. That is best done by an off-site service such as Akamai’s Prolexic.
Therefore, close to the end host usually means far from the core routers. Unless one is really considering a CPE device doing poorly jobs of “a router and a firewall”…
Depends on the nature of your network. I know lots of datacenter networks where the end hosts are not more than one or two hops removed from the core routers. I would hardly refer to those networks as a CPE device doing a poor job of “router and firewall”. Owen
On 08/02/2015, at 22:48, Owen DeLong <owen@delong.com> wrote:
On Feb 8, 2015, at 06:02 , Patrick Tracanelli <eksffa@freebsdbrasil.com.br> wrote:
Hello,
Some Juniper models actually do a very good job of being both.
In reality, a Firewall _IS_ a router, even if it's a bad one. Anything that moves packets from one interface to another is a router.
Technically it’s quite not a precise assumption. While routing is much likely an IP core need and OSI Layer 3 related mechanism, a firewall does not have to basic L3 forwarding. It can be a bridged/bright firewall, may fit in front of a router, protecting it, and carrying. Not routing anything. In fact in a fail-safe scenario (from availability perspective) a bridged firewall may be shut down completely and a Bypass por pair taking place will keep traffic flowing, “moving packets from one port to another” without actually ever been a router.
Technically true, but bridged firewalls are pretty much passe these days in the real world. As a general rule, when the firewall is shut down, one usually doesn’t want the packets flowing past un-hindered. The fact that this is kind of the default of what happens with bridged firewalls is just one of the many reasons hardly anyone still uses such a thing.
Hello Owen, I didn’t get your point. On a bridged firewall you can have the behavior you want, whatever it is. Passing packets with firewall is down, but the box still up. Dropping everything if packet is down. Combining bypass ports + STP you can have the set of availability you wish better for your scenario, from simply bypassing everything like you have no box there, if a software or power failure occurs, or having an active failover bridged together on the bypass port, allowing for full firewalling redundancy if the primary box fails. So you are no leaking options if you are doing it on L2 instead of L3. You have additional ease, redundancy won’t require VRRP or similar stuff since it’s not L3 related. To clarify, I am not saying a bridged firewall is a better option for every sort of scenario. Usually I do L3 firewall with the box being an extra hop on the topology, doing CARP for IP reduncancy, etc. But back to the point that a firewall doesn’t need to L3 forward, I like the idea of having a L2 firewall whenever an extra routing hop is not desired, and still won’t lack features and redundancy options.
I have recently added netmap-ipfw in front of BGP routers protecting ‘em from DDoS attacks, adding line-rate firewalling capabilities to a commodity x86/64 box or a T5 card for 10GbE/40GbE, and netmap-ipfw itself acts like mentioned, passing packets back and forth between interfaces without ever routing anything.
Sure, there are all kinds of things one can do. Some of them are good ideas, many of them are not. If it works in your environment and you’re OK with the failure modes and other tradeoffs, then more power to you.
I’m still missing what you are pointing as failure modes or tradeoffs. IPFW does a pretty decent job on L2 filtering, with a particular exception of “ipfw fwd” which won’t work by default under this setup. I can drop unwanted L2 traffic - in fact I like to only allow IPv4,v6 and ARP by default on LLC, cisco discovery and RARP when needed, everything else dropped on L2. Therefore whenever I decide to add it bright, I still don’t miss anything.
Of course, the support for routing protocols is a useful feature in a router and one of the areas where firewalls often fall short.
Where you want to put things (in front, behind, etc.) really depends on your topology and the problem you are trying to solve.
Personally, I like to keep the firewalls as close to the end hosts as possible. This tends to greatly simplify security policies and make them MUCH easier (and more reliable) to audit.
I agree. A firewall belongs better closer to the end hosts being protected. Maybe protection of the router is the only exception when RTBH will not fit the task (or just won’t be enough).
DDoS mitigation on site is a questionable and usually losing proposition at best. Other than DDoS mitigation, any good router should be perfectly capable of protecting itself. For protecting a router from DDoS that it cannot properly protect itself, one needs to be able to control or alter the delivery of packets across the upstream link from the upstream side anyway. That is best done by an off-site service such as Akamai’s Prolexic.
Sadly true just in theory. On real world, people still run weak and low power boxes as routers, ranging from old Cisco boxes to Routerboards, Edgerouters and x86 boxes without any special card or technologies such as DNA, DPDK, Netmap, and therefore boxes that can’t hardly protect themselves against simple DDoS, while they still can route and do their job on calm water, won’t behave well on storms. We are not always talking about big telcos and people who can rely on an efficient and well dimensioned Juniper box. Frequently, the “routers vs firewall” or “stateful vs stateles on the same box” confusion discussed on other recent threads in this group, just happens, and we get relatively big companies adding stuff like sonicwalls, fortinets, with BGP + a mix of security features on the same box, and certainly those boxes can’t hardly protect itself (but people still believe they can protect the whole network behind it). So, whether it’s caused by low power boxes or bad projects, the need for router protection is more often than it should.
Therefore, close to the end host usually means far from the core routers. Unless one is really considering a CPE device doing poorly jobs of “a router and a firewall”…
Depends on the nature of your network. I know lots of datacenter networks where the end hosts are not more than one or two hops removed from the core routers. I would hardly refer to those networks as a CPE device doing a poor job of “router and firewall”.
Yes, I may refer to a linux box, a VyOS, Endian, OpenBSD PF or a sonicwall box 2 hops after the core router a bad option for accumulating router+firewall functions if they are not protected on upper hops. Not due to any kind of feature or scalability miss (except for OpenBSD’s kernel and therefore firewall not scaling beyond CPU0). But due to their default behavior ranging from not protecting data plane from simple problems like ARP storm, up to the fact they will, by default, waste state entries for ordinary stuff like filtering. So, yes, they are usually doing poor jobs of being a router and a firewall. We see it very often, co-location setups failing due to exhaustion of resources or inability to self protect when uncalm packet flows reach their racks on data centers, but still the whole DC and previous hops still run perfectly up, available and unaffected. Sure it’s when they will earn/charge extra by adding firewall services… because customer boxes where doing their poor jobs of being both. So, back to the point, unless very well projected or engineered, they will need earlier protection. I have just recently run into a situation where a Fortigate box that should add protection and balancing on a data center colo, just died exhausted by simple memory starvation. No matter the real cause (people tend to use those boxes doing everything they can do at once, and expecting it to work under uncontrolled circumstances), for the customer it was their core node. For the datacenter it was an end host. For me it was just a box doing bad jobs of being more than it should at once, and it had to be protected. A couple months ago I have run into another scenario where a Juniper MX5 box was suffering from UDP DDoS with low packet size. It’s still not clear for me why Juniper was not sustaining the attack traffic, if it was a bad configuration or because the pps rate was close to 1GbE ports line rate limit, and the company in question didn’t want to pay for MX series upgrade to have 10G ports just for attack contention. We added a netmap-ipfw in front of the Juniper box with T5 10GbE ports filtering the profiled packet sized, and filtering other UDP patterns, which lead the Juniper box to do it’s routing job again without ever being upgraded to 10G licensed MX versions. Sure an off-site protection would do the job before upstream. But no need for that many gun powder to kill such a small sized animal :) And just like most DDoS, it didn’t least forever. And again, it was a non L3 forwarding firewall; no extra hop or relevant change in the topology. Sure L3 firewalls are more usual, it’s always been, it’s not a “nowadays” momentum. But a bright firewall still have its relevant place in the network, and it’s a firewall with no missing piece of functionality, IMHE. -- Patrick Tracanelli
On Mon, 09 Feb 2015 11:54:04 -0200, Patrick Tracanelli said:
On a bridged firewall you can have the behavior you want, whatever it is. Passing packets with firewall is down, but the box still up.
Owen's point is that passing packets if the firewall is down is really poor security-wise. If you run in that configuration, I simply DoS your firewall (probably from one set of IP addresses), and then once it has fallen over and is being bypassed, I send my *real* malicious traffic from some other IP address, totally uninspected and unhindered. Much hilarity, hijinks, and pwnage ensues.
On 09/02/2015, at 12:14, Valdis.Kletnieks@vt.edu wrote:
On Mon, 09 Feb 2015 11:54:04 -0200, Patrick Tracanelli said:
On a bridged firewall you can have the behavior you want, whatever it is. Passing packets with firewall is down, but the box still up.
Owen's point is that passing packets if the firewall is down is really poor security-wise. If you run in that configuration, I simply DoS your firewall (probably from one set of IP addresses), and then once it has fallen over and is being bypassed, I send my *real* malicious traffic from some other IP address, totally uninspected and unhindered. Much hilarity, hijinks, and pwnage ensues.
Hello Valdis, If this is really the point, I don’t know what system you are talking about, that will behave like that. If I run a closed firewall, kernel-path, and it’s unable process, and therefore “allow” the traffic, it will drop. If I run it netmap-ipfw and it’s unable to move the packet from one port to the other, it will drop. So there’s no point where a bridge implicits traffic bypass upon starvation/exaustion, unless this is your option to do so, or a default system behavior, in this case a system that should not act for this purpose. If I remember well (and I remember some effusive expressions like “L2 functions easily enabled at scale on a Junos Trio system”), on a Juniper box bridging is processed on Trio chip - even without IRB set up, as well as firewall (limited matching conditions in a bridged domain). If you can exhaust TRIO from your DoS approach (and the idea is that you can’t exhaust it without exhausting line rate first), you will have no bridging anyway, since L2 learning and forwarding will also be resource starved. But this is just all theoretical, as I mentioned you will probably reach line rate limit first if the box is not configured wrong or wrongly planned. -- Patrick Tracanelli
On Mon, 09 Feb 2015 12:56:37 -0200, Patrick Tracanelli said:
On 09/02/2015, at 12:14, Valdis.Kletnieks@vt.edu wrote: On Mon, 09 Feb 2015 11:54:04 -0200, Patrick Tracanelli said:
On a bridged firewall you can have the behavior you want, whatever it is. Passing packets with firewall is down, but the box still up.
Owen's point is that passing packets if the firewall is down is really poor security-wise. If you run in that configuration, I simply DoS your firewall (probably from one set of IP addresses), and then once it has fallen over and is being bypassed, I send my *real* malicious traffic from some other IP address, totally uninspected and unhindered. Much hilarity, hijinks, and pwnage ensues.
Hello Valdis,
If this is really the point, I don’t know what system you are talking about
The one *you* mentioned - "passing packets with firewall is down". Owen was pointing out that is a silly configuration: On 08/02/2015, at 22:48, Owen DeLong <owen@delong.com> wrote:
Technically true, but bridged firewalls are pretty much passe these days in the real world. As a general rule, when the firewall is shut down, one usually doesn’t want the packets flowing past un-hindered. The fact that this is kind of the default of what happens with bridged firewalls is just one of the many reasons hardly anyone still uses such a thing.
On 09/02/2015, at 13:25, Valdis.Kletnieks@vt.edu wrote:
On Mon, 09 Feb 2015 12:56:37 -0200, Patrick Tracanelli said:
On 09/02/2015, at 12:14, Valdis.Kletnieks@vt.edu wrote: On Mon, 09 Feb 2015 11:54:04 -0200, Patrick Tracanelli said:
On a bridged firewall you can have the behavior you want, whatever it is. Passing packets with firewall is down, but the box still up.
Owen's point is that passing packets if the firewall is down is really poor security-wise. If you run in that configuration, I simply DoS your firewall (probably from one set of IP addresses), and then once it has fallen over and is being bypassed, I send my *real* malicious traffic from some other IP address, totally uninspected and unhindered. Much hilarity, hijinks, and pwnage ensues.
Hello Valdis,
If this is really the point, I don’t know what system you are talking about
The one *you* mentioned - "passing packets with firewall is down". Owen was pointing out that is a silly configuration:
An explicit decision regarding bypass ports, as I mentioned if someone does not want a redundant approach and doesn’t want availability issues if power is down or system is overloaded. Not an inherit behavior or a must. Not related to being L2 our L3. Just a mentioned possibility. Not a limitation, not a recommendation. In the previous e-mail I mentioned “whatever option you want” upon failure, traffic still flowing, traffic bypassed, traffic dropped, L2+STP redundancy, no redundancy at all. So please don’t refer to one single option and pointing it as a failure of the methodology nature if you consider a decision/project error, and in this case just do it the other way, opting out from bypass and dropping or failing over, upon exhaustion or failure. Back to the point, doesn’t have to be different or limited from what you get in L3 firewalling.
On 08/02/2015, at 22:48, Owen DeLong <owen@delong.com> wrote:
Technically true, but bridged firewalls are pretty much passe these days in the real world. As a general rule, when the firewall is shut down, one usually doesn’t want the packets flowing past un-hindered. The fact that this is kind of the default of what happens with bridged firewalls is just one of the many reasons hardly anyone still uses such a thing.
-- Patrick Tracanelli FreeBSD Brasil LTDA. Tel.: (31) 3516-0800 316601@sip.freebsdbrasil.com.br http://www.freebsdbrasil.com.br "Long live Hanin Elias, Kim Deal!"
A router behind the firewall is nice too. It insulates the firewall from direct end-user traffic. It also makes for a cleaner cutover from one firewall to another. (Instead of the edge getting stuck ARPs their perspective of the network remains unchanged.) It also allows for stateless ACLs on both ends of the firewall. On Thu, Feb 5, 2015 at 1:49 PM, Ralph J.Mayer <rmayer@nerd-residenz.de> wrote:
Hi David,
a router is a router and a firewall is a firewall.
Especially a Cisco ASA is no router, period.
A router in front of the firewall is my choice, it also keeps broadcasts from the firewall + can do uRPF.
rm
Setup a multi tenant setup between Nexus 7K and Juniper Net screen 5400 FW using OSPF. It went OK and worked. However when under traffic load/ less than. Desirable results... OSPF peer failure / bounces etc. However using BGP with Juniper SRX FW has been working great. No issues thus far. On Feb 5, 2015 9:11 AM, "David Jansen" <david@nines.nl> wrote:
Hi,
We have used dynamic routing on firewall in the old days. We did experience several severe outages due to this setup (OSPF en Cisco). As you will understand i’m not eager to go back to this solution but I am curious about your point of views.
Is it advisory to so these days?
Kind regards, David
I have some use cases where I have Fortinet firewalls running full ospf/ospfv3/bgp and it all pretty much just works without any issues. The CLI is a bit cumbersome, but apart from that its fine. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Craig Sent: Monday, 9 February 2015 2:21 p.m. To: David Jansen Cc: nanog group Subject: Re: Dynamic routing on firewalls. Setup a multi tenant setup between Nexus 7K and Juniper Net screen 5400 FW using OSPF. It went OK and worked. However when under traffic load/ less than. Desirable results... OSPF peer failure / bounces etc. However using BGP with Juniper SRX FW has been working great. No issues thus far. On Feb 5, 2015 9:11 AM, "David Jansen" <david@nines.nl> wrote:
Hi,
We have used dynamic routing on firewall in the old days. We did experience several severe outages due to this setup (OSPF en Cisco). As you will understand i’m not eager to go back to this solution but I am curious about your point of views.
Is it advisory to so these days?
Kind regards, David
participants (18)
-
Bill Thompson
-
BPNoC Group
-
Craig
-
David Jansen
-
Doug Barton
-
Eugeniu Patrascu
-
Jeff McAdams
-
Joe Hamelin
-
ML
-
Nicholas Oas
-
Owen DeLong
-
Patrick Tracanelli
-
Ralph J.Mayer
-
Ray Soucy
-
Rich Kulawiec
-
santiago martinez
-
Tony Wicks
-
Valdis.Kletnieks@vt.edu