Firewalls in service provider environments
All, Looking for some recommendations on firewall placement in service provider environments. I'm of the school of thought that in my SP network I do as little firewalling/packet filtering as possible. As in none, leave that to my end users or offer a "managed" firewall solution where if a customer signs up for the extra service I put him in a VRF or VLAN that is "behind" a firewall and manage that solution for them. Otherwise I don't prefer to have a firewall inline in my service provider network for all customer traffic to go through. I can accomplish filtering of known bad ports on my edge routers either facing my customers or upstream providers. What is the group's thought on this? -Matt -- Matt Reath CCIE #27316 (SP) matt@mattreath.com | http://mattreath.com Twitter: http://twitter.com/mpreath
-----Original Message----- From: Matthew Reath [mailto:matt@mattreath.com] Sent: 07 February 2012 21:34 To: nanog@nanog.org Subject: Firewalls in service provider environments
All,
Looking for some recommendations on firewall placement in service provider environments. I'm of the school of thought that in my SP network I do as little firewalling/packet filtering as possible. As in none,
I had a vendor actually suggest that that ALL my customer traffic should traverse a firewall. I asked why and they said "Ahhh it the internet, must have firewall". I suppose this must have been a great firewall. So yes I would agree with you, firewall nothing for your customers unless they are paying you for a specific service. Filtering known bad ports, well, what's a known bad port? Bad for one person may be quite important for another. Whilst filtering port 25 outbound may help prevent some bots from emanating spam, it certainly does a lot to annoy other people. -- Leigh Porter ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________
-----Original Message----- From: Matthew Reath [mailto:matt@mattreath.com] Sent: 07 February 2012 21:34 To: nanog@nanog.org Subject: Firewalls in service provider environments
All,
Looking for some recommendations on firewall placement in service provider environments. I'm of the school of thought that in my SP network I do as little firewalling/packet filtering as possible. As in none,
I had a vendor actually suggest that that ALL my customer traffic should traverse a firewall. I asked why and they said "Ahhh it the internet, must have firewall". I suppose this must have been a great firewall.
So yes I would agree with you, firewall nothing for your customers unless they are paying you for a specific service. Filtering known bad ports, well, what's a known bad port? Bad for one person may be quite important for another. Whilst filtering port 25 outbound may help prevent some bots from emanating spam, it certainly does a lot to annoy other people.
-- Leigh Porter
______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________
From a filtering perspective there are some know worm ports and such that we usually have a template created for.
Here is the template we typically use (or a variant of it): <-- snippet --> access-list 102 deny ip 10.0.0.0 0.255.255.255 any access-list 102 deny ip 172.16.0.0 0.15.255.255 any access-list 102 deny ip 192.168.0.0 0.0.255.255 any access-list 102 deny ip 0.0.0.0 0.255.255.255 any access-list 102 deny ip 127.0.0.0 0.255.255.255 any access-list 102 deny ip 224.0.0.0 15.255.255.255 any access-list 102 deny ip host 255.255.255.255 any access-list 102 deny tcp any any eq 135 access-list 102 deny udp any any eq 135 access-list 102 deny udp any any eq netbios-ns access-list 102 deny tcp any any eq 139 access-list 102 deny udp any any eq netbios-ss access-list 102 deny tcp any any eq 445 access-list 102 deny tcp any any eq 593 access-list 102 deny tcp any any eq 4444 access-list 102 deny tcp any any eq 9996 access-list 102 deny tcp any any eq 5554 access-list 102 deny tcp any any eq 8888 access-list 102 deny tcp any any eq 7778 access-list 102 deny tcp any any eq 8594 access-list 102 deny tcp any any eq 8563 access-list 102 deny tcp any any eq 1434 <-- end snippet --> This blocks some common worm ports as well as traffic sourced outside of our network from reserved address space. -Matt
On Tue, Feb 7, 2012 at 4:52 PM, Matthew Reath <matt@mattreath.com> wrote:
Here is the template we typically use (or a variant of it):
<-- snippet --> access-list 102 deny ip 10.0.0.0 0.255.255.255 any access-list 102 deny ip 172.16.0.0 0.15.255.255 any access-list 102 deny ip 192.168.0.0 0.0.255.255 any access-list 102 deny ip 0.0.0.0 0.255.255.255 any access-list 102 deny ip 127.0.0.0 0.255.255.255 any access-list 102 deny ip 224.0.0.0 15.255.255.255 any access-list 102 deny ip host 255.255.255.255 any access-list 102 deny tcp any any eq 135 access-list 102 deny udp any any eq 135 access-list 102 deny udp any any eq netbios-ns access-list 102 deny tcp any any eq 139 access-list 102 deny udp any any eq netbios-ss access-list 102 deny tcp any any eq 445 access-list 102 deny tcp any any eq 593 access-list 102 deny tcp any any eq 4444 access-list 102 deny tcp any any eq 9996 access-list 102 deny tcp any any eq 5554 access-list 102 deny tcp any any eq 8888 access-list 102 deny tcp any any eq 7778 access-list 102 deny tcp any any eq 8594 access-list 102 deny tcp any any eq 8563 access-list 102 deny tcp any any eq 1434 <-- end snippet -->
One of my customers has a list like that. They can't understand why one in every hundred or so TCP connections on port 443 fails. Hint: you forgot "access-list 102 permit tcp any any established" after "access-list 102 deny ip host 255.255.255.255 any". The destination port in one direction is the source port in the other and many of those are dynamic source ports picked by Windows. Unless you restrict that filter to just packets attempting to initiate a new connection, you're shooting yourself in the foot. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Tue, Feb 7, 2012 at 4:52 PM, Matthew Reath <matt@mattreath.com> wrote:
Here is the template we typically use (or a variant of it):
<-- snippet --> access-list 102 deny ip 10.0.0.0 0.255.255.255 any access-list 102 deny ip 172.16.0.0 0.15.255.255 any access-list 102 deny ip 192.168.0.0 0.0.255.255 any access-list 102 deny ip 0.0.0.0 0.255.255.255 any access-list 102 deny ip 127.0.0.0 0.255.255.255 any access-list 102 deny ip 224.0.0.0 15.255.255.255 any access-list 102 deny ip host 255.255.255.255 any access-list 102 deny tcp any any eq 135 access-list 102 deny udp any any eq 135 access-list 102 deny udp any any eq netbios-ns access-list 102 deny tcp any any eq 139 access-list 102 deny udp any any eq netbios-ss access-list 102 deny tcp any any eq 445 access-list 102 deny tcp any any eq 593 access-list 102 deny tcp any any eq 4444 access-list 102 deny tcp any any eq 9996 access-list 102 deny tcp any any eq 5554 access-list 102 deny tcp any any eq 8888 access-list 102 deny tcp any any eq 7778 access-list 102 deny tcp any any eq 8594 access-list 102 deny tcp any any eq 8563 access-list 102 deny tcp any any eq 1434 <-- end snippet -->
One of my customers has a list like that. They can't understand why one in every hundred or so TCP connections on port 443 fails.
Hint: you forgot "access-list 102 permit tcp any any established" after "access-list 102 deny ip host 255.255.255.255 any". The destination port in one direction is the source port in the other and many of those are dynamic source ports picked by Windows. Unless you restrict that filter to just packets attempting to initiate a new connection, you're shooting yourself in the foot.
Regards, Bill Herrin
-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
Yeah agreed. The only place this gets applied is inbound on the interface facing an upstream provider. ACLs ingress from end customers are much different. In theory this could cause issues with externally initiated traffic that use lets say 8888 as its random source port. -Matt
On Tue, Feb 7, 2012 at 4:35 PM, Matthew Reath <matt@mattreath.com> wrote:
One of my customers has a list like that. They can't understand why one in every hundred or so TCP connections on port 443 fails.
Hint: you forgot "access-list 102 permit tcp any any established" after "access-list 102 deny ip host 255.255.255.255 any". The destination port in one direction is the source port in the other and many of those are dynamic source ports picked by Windows. Unless you restrict that filter to just packets attempting to initiate a new connection, you're shooting yourself in the foot.
Yeah agreed. The only place this gets applied is inbound on the interface facing an upstream provider. ACLs ingress from end customers are much different. In theory this could cause issues with externally initiated traffic that use lets say 8888 as its random source port.
If you apply the ACL you showed as an inbound ACL on your provider facing interfaces, you will be breaking any connections that exit your network with source ports from your list of bad ports. For example, you connect out from x.x.x.x:8888 to y.y.y.y:80, then the response packets coming back into your network will be from y.y.y.y:80 to x.x.x.x:8888 and will be dropped by your ACL. This seems to be a common mistake, and is often missed because it manifests as one-in-thousands failures of TCP connections. People tend to just try a second time and it works and never investigate why they had one random failure.
On Tue, Feb 7, 2012 at 4:35 PM, Matthew Reath <matt@mattreath.com> wrote:
One of my customers has a list like that. They can't understand why one in every hundred or so TCP connections on port 443 fails.
Hint: you forgot "access-list 102 permit tcp any any established" after "access-list 102 deny ip host 255.255.255.255 any". The destination port in one direction is the source port in the other and many of those are dynamic source ports picked by Windows. Unless you restrict that filter to just packets attempting to initiate a new connection, you're shooting yourself in the foot.
Yeah agreed. The only place this gets applied is inbound on the interface facing an upstream provider. ACLs ingress from end customers are much different. In theory this could cause issues with externally initiated traffic that use lets say 8888 as its random source port.
If you apply the ACL you showed as an inbound ACL on your provider facing interfaces, you will be breaking any connections that exit your network with source ports from your list of bad ports. For example, you connect out from x.x.x.x:8888 to y.y.y.y:80, then the response packets coming back into your network will be from y.y.y.y:80 to x.x.x.x:8888 and will be dropped by your ACL.
This seems to be a common mistake, and is often missed because it manifests as one-in-thousands failures of TCP connections. People tend to just try a second time and it works and never investigate why they had one random failure.
Good point. Adding in an established entry, although may open you up for TCP/SYN sort of packets is a better trade off than affecting customer traffic. -Matt
On Wed, Feb 8, 2012 at 9:25 AM, Matthew Reath <matt@mattreath.com> wrote:
Good point. Adding in an established entry, although may open you up for TCP/SYN sort of packets is a better trade off than affecting customer traffic.
'established' is explicitly NOT 'syn' ... maybe you meant 'ack flood' ? (or rst flood? or .... but certainly not syn flood)
On Wed, Feb 8, 2012 at 9:25 AM, Matthew Reath <matt@mattreath.com> wrote:
Good point. Adding in an established entry, although may open you up for TCP/SYN sort of packets is a better trade off than affecting customer traffic.
'established' is explicitly NOT 'syn' ... maybe you meant 'ack flood' ? (or rst flood? or .... but certainly not syn flood)
If I had an 'established' entry on an inbound ACL to filter traffic coming from my upstream provider wouldn't SYN ACK (2nd step in handshake) packets be allowed to pass the ACL because of this? But I see your point a connection initiation from external sources with just the SYN flag set would not be allowed. However if a session is initiated internally the returning SYN ACK from the external server would be allowed as would ACK and data packets with ACK set.
On Wed, Feb 08, 2012 at 08:25:18AM -0600, Matthew Reath wrote:
If you apply the ACL you showed as an inbound ACL on your provider facing interfaces, you will be breaking any connections that exit your network with source ports from your list of bad ports. For example, you connect out from x.x.x.x:8888 to y.y.y.y:80, then the response packets coming back into your network will be from y.y.y.y:80 to x.x.x.x:8888 and will be dropped by your ACL.
Good point. Adding in an established entry, although may open you up for TCP/SYN sort of packets is a better trade off than affecting customer traffic.
I've always thought that reflexive access lists were quite elegant, and a much better method than established, albeit for edge networks. Do they not work in the SP space? -- Henry Yen Aegis Information Systems, Inc. Senior Systems Programmer Hicksville, New York
I'm an end user but I refer to these from time to time: http://www.ietf.org/rfc/rfc3013.txt http://www.ietf.org/rfc/rfc3871.txt I suppose the salient question is what kind of customers are we talking about. Best wishes.
Here is the template we typically use (or a variant of it):
<-- snippet --> access-list 102 deny ip 10.0.0.0 0.255.255.255 any access-list 102 deny ip 172.16.0.0 0.15.255.255 any access-list 102 deny ip 192.168.0.0 0.0.255.255 any access-list 102 deny ip 0.0.0.0 0.255.255.255 any access-list 102 deny ip 127.0.0.0 0.255.255.255 any access-list 102 deny ip 224.0.0.0 15.255.255.255 any access-list 102 deny ip host 255.255.255.255 any
I typically also include traffic to/from: TCP/UDP port 0 169.254.0.0/16 192.0.2.0/24 198.51.100.0/24 203.0.113.0/24 Been wondering if I should also block 198.18.0.0/15 as well.
On Wed, Feb 8, 2012 at 4:04 AM, George Bonser <gbonser@seven.com> wrote:
I typically also include traffic to/from:
TCP/UDP port 0 169.254.0.0/16 192.0.2.0/24 198.51.100.0/24 203.0.113.0/24
Been wondering if I should also block 198.18.0.0/15 as well.
suresh@frodo 17:46:08 :~$ nslookup 1.113.0.203.bogons.cymru.com Server: 127.0.0.1 Address: 127.0.0.1#53 Non-authoritative answer: Name: 1.113.0.203.bogons.cymru.com Address: 127.0.0.2 Also available as a bgp feed, for years now. Saves you updating your martian ACLs from time to time. -- Suresh Ramasubramanian (ops.lists@gmail.com)
On 2012.02.07 20:47, Suresh Ramasubramanian wrote:
On Wed, Feb 8, 2012 at 4:04 AM, George Bonser<gbonser@seven.com> wrote:
I typically also include traffic to/from:
TCP/UDP port 0 169.254.0.0/16 192.0.2.0/24 198.51.100.0/24 203.0.113.0/24
Been wondering if I should also block 198.18.0.0/15 as well.
suresh@frodo 17:46:08 :~$ nslookup 1.113.0.203.bogons.cymru.com Server: 127.0.0.1 Address: 127.0.0.1#53
Non-authoritative answer: Name: 1.113.0.203.bogons.cymru.com Address: 127.0.0.2
Also available as a bgp feed, for years now. Saves you updating your martian ACLs from time to time.
Amen. v4 and v6 lists are available via free BGP feed (via v4 and v6 peering) from Cymru. Dynamic simplicity within community's finest standards. Works wonders for those who have s/RTBH deployed.
On Tue, Feb 7, 2012 at 4:42 PM, Leigh Porter <leigh.porter@ukbroadband.com> wrote:
-----Original Message----- From: Matthew Reath [mailto:matt@mattreath.com] Sent: 07 February 2012 21:34 To: nanog@nanog.org Subject: Firewalls in service provider environments
All,
Looking for some recommendations on firewall placement in service provider environments. I'm of the school of thought that in my SP network I do as little firewalling/packet filtering as possible. As in none,
I had a vendor actually suggest that that ALL my customer traffic should traverse a firewall. I asked why and they said "Ahhh it the internet, must have firewall". I suppose this must have been a great firewall.
'of china'! ha! hahaha.... anyway.
So yes I would agree with you, firewall nothing for your customers unless they are paying you for a specific service. Filtering known bad ports, well, what's a known bad port? Bad for one person may be quite important for another. Whilst filtering port 25 outbound may help prevent some bots from emanating spam, it certainly does a lot to annoy other people.
I think for a purely SP network, transit-provider core links sort of thing, why filter anything at all? why filter anything that's not destined to your own equipment? You can't possibly know what some customer (or set of customers) are going to do with their traffic, so you can't possibly filter it sanely/safely. for a consumer ISP, provided your TOS says it's ok, why not filter some common problems: tcp/25 ... not much else really... and REALLY you just want to send tcp/25 -> 587 on your mail-relays (or redirect to internal use addresses on the relays). If customers (in either case) want to pay you for 'security services' then rock some filters on their CPE, with the option to move that upstream to your PE if you have to (too much crap on customer link). -chris
On Tue, 7 Feb 2012, Matthew Reath wrote:
Looking for some recommendations on firewall placement in service provider environments. I'm of the school of thought that in my SP network I do as little firewalling/packet filtering as possible. As in none, leave that to my end users or offer a "managed" firewall solution where if a customer signs up for the extra service I put him in a VRF or VLAN that is "behind" a firewall and manage that solution for them. Otherwise I don't prefer to have a firewall inline in my service provider network for all customer traffic to go through. I can accomplish filtering of known bad ports on my edge routers either facing my customers or upstream providers.
I tend to agree with this, and I think you'll find that most providers agree with that as well. There are several reasons for this: 1. Firewalls present another point of failure, and SPs are generally loath to force customer traffic* through another choke point. 2. Many firewall appliances are stateful. Multihomed customers and stateful firewalls can be a major headache. Asymmetric routing through stateful firewalls is pretty much a non-starter. 3. You (the customer) know your applications and internal network better than the SP does. It makes sense for you to manage your firewalls/NAT/ internal LAN. If you can't or don't want to do this, hire a consultant to do the work for you. 4. Most SPs would not want the liability of managing firewall service. 5. Dropping packets at the SP edge could be done, but I think most SPs would only want to do so in extraordinary circumstances. * - As you mentioned, unless the SP offers, and those customers specifically pay for a firewalled service. jms
On Tue, Feb 7, 2012 at 4:31 PM, Matthew Reath <matt@mattreath.com> wrote:
Looking for some recommendations on firewall placement in service provider environments. I'm of the school of thought that in my SP network I do as little firewalling/packet filtering as possible. As in none, leave that to my end users or offer a "managed" firewall solution where if a customer signs up for the extra service I put him in a VRF or VLAN that is "behind" a firewall and manage that solution for them. Otherwise I don't prefer to have a firewall inline in my service provider network for all customer traffic to go through. I can accomplish filtering of known bad ports on my edge routers either facing my customers or upstream providers.
What is the group's thought on this?
Hi Matthew, It Depends. High end business customers (of the BGP speaking variety) generally appreciate having a remote triggered black hole facility. That's a kind of firewall. http://tools.ietf.org/html/rfc5635 Business customers in general shouldn't be filtered unless they buy a managed firewall service from you. Don't tamper with their DNS either! When you get down to the residential and Internet Cafe type users, there is some common filtering you should consider: TCP SYN to port 25 outbound from your dynamic IP customers should generally be disallowed except to your local mail servers. 99 times out of 100, connections originating to this port from dynamic IP customers will be Email Spam from an infected PC. This will hurt you. It will hurt you with spam complaints. It will hurt you with adverse action by RBL providers. It will hurt you with damage to your reputation and brand. http://www.spamhaus.org/faq/answers.lasso?section=isp%20spam%20issues#133 Blocking TCP and UDP 137, 138, 139 and 445 is not terribly unusual. These are associated with Microsoft file sharing protocols. Off the LAN and outside the enterprise anybody actually open to this traffic is generally asking to be hacked. Then a spam bot is installed and you have another problem customer who isn't paying you enough to deal with that crap. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Wed, Feb 8, 2012 at 3:52 AM, William Herrin <bill@herrin.us> wrote:
High end business customers (of the BGP speaking variety) generally appreciate having a remote triggered black hole facility. That's a kind of firewall. http://tools.ietf.org/html/rfc5635
While I 100% agree that sticking a stateful firewall into a SP environment is several kinds of dumb, I wouldn't run it wide open and unfiltered either. There are several things that a SP should definitely be looking at, that'd still describe as a firewall, and are not the "stateful firewall / IDS / IPS magic black box" half the posters in this thread are instinctively reacting to. For the record, yes, I agree those are a bad idea. But how about these - All these are going to be implemented to a greater or a lesser degree, and in different places, depending on how you define SP (selling only transit OC-48s? T1..T3 to end user corporations? Datacenter hosting?) 1. S/RTBH 2. Netflow based devices (Arbor, Tivoli TNPFA flow analyzers, etc) 3. DDoS mitigation - possibly resold as an extra service [built inhouse / provided by other vendors or your upstream tier 1] 4. Router ACLs to get rid of common worm traffic 5. Filtering both ways to prevent async routing to bypass your filters (http://irbs.net/internet/nanog/0408/0405.html and in that thread, http://irbs.net/internet/nanog/0408/0465.html for a fun example) 6. Putting different customers into different VLANs rather than packing everybody into a single VLAN - that way they don't spoof unused IPs on the same VLAN (that is, unused IPs anywhere in your IP space .. and this is, like #5, a rather old attack that I haven't seen in a while, it used to be very popular with spammers some years back, and sticking your customers into separate VLANs anyway makes a lot of sense from a management perspective, leave alone the security implications) --srs -- Suresh Ramasubramanian (ops.lists@gmail.com)
On Tue, Feb 7, 2012 at 3:31 PM, Matthew Reath <matt@mattreath.com> wrote:
traffic to go through. I can accomplish filtering of known bad ports on my edge routers either facing my customers or upstream providers.
I wouldn't want my provider breaking my connectivity in the name of security. My thought on this is: by all means filter, firewall, and mangle packets addressed From/To service provider equipment (for example, port 22 TCP packets sent to the address of the ISP router), in order to protect ISP network, But, with some minor exceptions, in your role as ISP, internal firewalls should be separate from customer networks, and, never ever filter, firewall, or mangle packets _forwarded_ by service provider equipment To/From customer networks. If you manage the downstream network, the customer has requested special filtering or restrictions, or a pattern of abuse has been detected from a certain downstream that can be temporarily mitigated with a filter, that's a different story. It is acceptable to drop packets to enforce network policy, such as no-spam rules that a customer or host has violated, for example: all IP packets to a host. It is acceptable to drop erroneous packets, such as ones with an incorrect checksum or source IP that the customer has not been assigned, or has not informed the provider that they were assigned. I would say it's in general unacceptable for an ISP to discriminate against packets addressed to or sourced from specific port numbers. That's actually breaking connectivity. There's no such thing as a "bad port number"; there are port numbers that certain applications have abused; the entire port range should be available for any host as indicated by the various RFCs that define the protocol. If packets with any valid bit pattern in the source port or destination port field are dropped, based on a valid bit pattern in that field, then that is really a breakage of proper connectivity by the provider. What is the group's thought on this?
-Matt
-- -JH
participants (13)
-
Christopher Morrow -
David Walker -
George Bonser -
Henry Yen -
Jared Mauch -
Jimmy Hess -
Justin M. Streiner -
Leigh Porter -
Matt Buford -
Matthew Reath -
Steve Bertrand -
Suresh Ramasubramanian -
William Herrin