Ingress filtering on transits, peers, and IX ports
We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how. This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet. It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it. We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include: * Permit entries for P-t-P WAN subnets on peering links * Permit entries for IP assignments to customers running multi-homed BGP * The "permit ipv4 any any" at the end :) The questions I wanted to ask the SP community are: * What traffic filtering do you do on your transits, on IX ports, and your direct peering links? * How is it accomplished? Through static ACL or some flavor of uRPF? * If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update? * How did you test your filters when they were implemented? Thanks a lot, -Brian
You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped. RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :) Here’s a Cisco brief on the topic: https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwa... I think all router vendors support this feature. Here’s a similar article by Juniper: https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/... -mel beckman On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote: We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how. This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet. It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it. We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include: * Permit entries for P-t-P WAN subnets on peering links * Permit entries for IP assignments to customers running multi-homed BGP * The "permit ipv4 any any" at the end :) The questions I wanted to ask the SP community are: * What traffic filtering do you do on your transits, on IX ports, and your direct peering links? * How is it accomplished? Through static ACL or some flavor of uRPF? * If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update? * How did you test your filters when they were implemented? Thanks a lot, -Brian
Matt Harris|Infrastructure Lead Engineer 816-256-5446|Direct Looking for something? Helpdesk Portal|Email Support|Billing Portal We build and deliver end-to-end IT solutions. On Tue, Oct 13, 2020 at 5:22 PM Mel Beckman <mel@beckman.org> wrote:
You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :
I received one of the aforementioned messages as well, and my response was that perhaps the best overall step towards protection at scale from the issue they raise would be for SPs to implement URPF facing stubby, single-homed networks. This is effectively the low-hanging fruit and doesn't require too much additional labor in terms of maintaining additional ACLs or prefix lists. In the case of multi-homed networks, things are less straight forward, but multi-homed networks make up a minority even if we exclude consumer internet connections. Take care, Matt
That’s an interesting suggestion There are 2 modes for uRPF. Loose and strict. Which one would you recommend in this scenario and why? There are many ways to solve this and definitely uRPF is one layer of defense. But, probably not the best alone. I advocate a 3 layers approach. I’m curious to hear/read which uRPF would you recommend for this particular case. Thanks Jean St-Laurent From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Mel Beckman Sent: Tuesday, October 13, 2020 6:22 PM To: Brian Knight <ml@knight-networks.com> Cc: nanog@nanog.org Subject: Re: Ingress filtering on transits, peers, and IX ports You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped. RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :) Here’s a Cisco brief on the topic: https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwa... I think all router vendors support this feature. Here’s a similar article by Juniper: https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/... -mel beckman On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org <mailto:nanog@nanog.org> > wrote: We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how. This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet. It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it. We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include: * Permit entries for P-t-P WAN subnets on peering links * Permit entries for IP assignments to customers running multi-homed BGP * The "permit ipv4 any any" at the end :) The questions I wanted to ask the SP community are: * What traffic filtering do you do on your transits, on IX ports, and your direct peering links? * How is it accomplished? Through static ACL or some flavor of uRPF? * If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update? * How did you test your filters when they were implemented? Thanks a lot, -Brian
Hi Mel, My understanding of uRPF is: * Strict mode will permit a packet only if there is a route for the source IP in the RIB, and that route points to the interface where the packet was received * Loose mode will permit a packet if there is a route for the source IP in the RIB. It does not matter where the route is pointed. Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit. Loose mode won't work for us, because all of our own prefixes are in our RIB, and thus the uRPF check on a transit would never block anything. Or am I missing something? Thanks, -Brian On 2020-10-13 17:22, Mel Beckman wrote:
You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)
Here's a Cisco brief on the topic:
https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwa...
I think all router vendors support this feature. Here's a similar article by Juniper:
https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/...
-mel beckman
On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:
We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
* Permit entries for P-t-P WAN subnets on peering links
* Permit entries for IP assignments to customers running multi-homed BGP
* The "permit ipv4 any any" at the end :)
The questions I wanted to ask the SP community are:
* What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
* How is it accomplished? Through static ACL or some flavor of uRPF?
* If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
* How did you test your filters when they were implemented?
Thanks a lot,
-Brian
Hi, Brian Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated by: RFC8704 Enhanced Feasible-Path uRPF). Ingress Access Lists require typically manual maintenance, but are the most bulletproof when done properly; typically, ingress access lists are best fit between the edge and the ISP when the configuration is not too dynamic if strict RPF is not an option, between ISPs if the number of used prefixes is low, or as an additional layer of protection Ingress filters Best Practices https://www.ripe.net/support/training/material/bgp-operations-and-security-t... *Don’t accept BOGON ASNs *Don’t accept BOGON prefixes *Don’t accept your own prefix *Don’t accept default (unless you requested it) *Don’t accept prefixes that are too specific *Don’t accept if AS Path is too long *Create filters based on Internet Routing Registries And also BGP Best Current Practices by Philip Smith http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf Regards. El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG (<nanog@nanog.org>) escribió:
Hi Mel,
My understanding of uRPF is:
* Strict mode will permit a packet only if there is a route for the source IP in the RIB, and that route points to the interface where the packet was received
* Loose mode will permit a packet if there is a route for the source IP in the RIB. It does not matter where the route is pointed.
Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.
Loose mode won't work for us, because all of our own prefixes are in our RIB, and thus the uRPF check on a transit would never block anything.
Or am I missing something?
Thanks,
-Brian
On 2020-10-13 17:22, Mel Beckman wrote:
You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)
Here's a Cisco brief on the topic:
https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwa...
I think all router vendors support this feature. Here's a similar article by Juniper:
https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/...
-mel beckman
On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:
We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
* Permit entries for P-t-P WAN subnets on peering links
* Permit entries for IP assignments to customers running multi-homed BGP
* The "permit ipv4 any any" at the end :)
The questions I wanted to ask the SP community are:
* What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
* How is it accomplished? Through static ACL or some flavor of uRPF?
* If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
* How did you test your filters when they were implemented?
Thanks a lot,
-Brian
Specifically with regards to “Don’t accept your own prefix”, this poses an interesting challenge for the original notice sent out by the security researchers. It is not uncommon today for various content networks (and others) to operate multiple “island networks” with a single ASN. For example, AS65001 operates in Los Angeles and New York, with no internal connectivity between them - only connectivity via the Internet (loop prevention disablement is done by every tier 1 provider upon request). Los Angeles advertises 192.168.10.0/24 and New York advertises 192.168.20.0/24 to the Internet, and both have relevant anti-spoofing filters (e.g. Los Angeles has an ingress filter to drop traffic with source IP of 192.168.10.0/24, and New York has an ingress filter to drop traffic with a source IP of 192.168.20.0/24). Traffic between Los Angeles and New York is not filtered as they need to legitimately pass traffic to each other over the Internet. This triggers a false positive detection by the system in question that sent the original notification email. After discussing this with the people running this project and highlighting that this will be generating false positive data as part of their research (and probably quite a lot, this practice is fairly common), the response was to establish some form of tunnel between the AS islands over the Internet. Not realistic for a bunch of the content companies who practice this design pushing tens/hundreds of Gbps over the Internet (if not more). There seemed to be no interest in the discussion that the data being generated by this system is arguably flawed. Tl;dr - definitely don’t accept your own prefix from the site it originated from, or other sites that have internal connectivity. But also don’t assume that an AS has a full-mesh of internal connectivity behind it and shouldn’t accept its own prefixes for any reason. Sent from my iPhone
On Oct 13, 2020, at 7:54 PM, Marcos Manoni <marcosmanoni@gmail.com> wrote:
Hi, Brian
Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated by: RFC8704 Enhanced Feasible-Path uRPF).
Ingress Access Lists require typically manual maintenance, but are the most bulletproof when done properly; typically, ingress access lists are best fit between the edge and the ISP when the configuration is not too dynamic if strict RPF is not an option, between ISPs if the number of used prefixes is low, or as an additional layer of protection
Ingress filters Best Practices https://www.ripe.net/support/training/material/bgp-operations-and-security-t... *Don’t accept BOGON ASNs *Don’t accept BOGON prefixes *Don’t accept your own prefix *Don’t accept default (unless you requested it) *Don’t accept prefixes that are too specific *Don’t accept if AS Path is too long *Create filters based on Internet Routing Registries
And also BGP Best Current Practices by Philip Smith http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
Regards.
El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG (<nanog@nanog.org>) escribió:
Hi Mel,
My understanding of uRPF is:
* Strict mode will permit a packet only if there is a route for the source IP in the RIB, and that route points to the interface where the packet was received
* Loose mode will permit a packet if there is a route for the source IP in the RIB. It does not matter where the route is pointed.
Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.
Loose mode won't work for us, because all of our own prefixes are in our RIB, and thus the uRPF check on a transit would never block anything.
Or am I missing something?
Thanks,
-Brian
On 2020-10-13 17:22, Mel Beckman wrote:
You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)
Here's a Cisco brief on the topic:
https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwa...
I think all router vendors support this feature. Here's a similar article by Juniper:
https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/...
-mel beckman
On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:
We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
* Permit entries for P-t-P WAN subnets on peering links
* Permit entries for IP assignments to customers running multi-homed BGP
* The "permit ipv4 any any" at the end :)
The questions I wanted to ask the SP community are:
* What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
* How is it accomplished? Through static ACL or some flavor of uRPF?
* If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
* How did you test your filters when they were implemented?
Thanks a lot,
-Brian
On 10/13/20 9:40 PM, Nikolas Geyer wrote:
Tl;dr - definitely don’t accept your own prefix from the site it originated from, or other sites that have internal connectivity. But also don’t assume that an AS has a full-mesh of internal connectivity behind it and shouldn’t accept its own prefixes for any reason.
While I can understand some reasons why people don't do it, I believe the proper thing to do in this case is have multiple ASNs - one for each island. They obviously have distinct routing policy and thus qualify at least under ARIN policy AFAIK. With AS4, we don't have any imminent shortage of ASNs and don't need to be particularly stingy about allocating them as long as a need is met. -- Brandon Martin
Hi Marcos, Thanks for your reply. But I am looking for guidance on traffic filtering, not BGP prefix filtering. I have looked at BCP 84, and it's a good overview of the methods available to an ISP. My questions are more operational in nature and are not covered by the document. Of the choices presented in BCP 84, what do folks really use? If it's an ACL, what challenges have there been with updates? etc. -Brian On 2020-10-13 18:52, Marcos Manoni wrote:
Hi, Brian
Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated by: RFC8704 Enhanced Feasible-Path uRPF).
Ingress Access Lists require typically manual maintenance, but are the most bulletproof when done properly; typically, ingress access lists are best fit between the edge and the ISP when the configuration is not too dynamic if strict RPF is not an option, between ISPs if the number of used prefixes is low, or as an additional layer of protection
Ingress filters Best Practices https://www.ripe.net/support/training/material/bgp-operations-and-security-t... *Don’t accept BOGON ASNs *Don’t accept BOGON prefixes *Don’t accept your own prefix *Don’t accept default (unless you requested it) *Don’t accept prefixes that are too specific *Don’t accept if AS Path is too long *Create filters based on Internet Routing Registries
And also BGP Best Current Practices by Philip Smith http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
Regards.
El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG (<nanog@nanog.org>) escribió:
Hi Mel,
My understanding of uRPF is:
* Strict mode will permit a packet only if there is a route for the source IP in the RIB, and that route points to the interface where the packet was received
* Loose mode will permit a packet if there is a route for the source IP in the RIB. It does not matter where the route is pointed.
Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.
Loose mode won't work for us, because all of our own prefixes are in our RIB, and thus the uRPF check on a transit would never block anything.
Or am I missing something?
Thanks,
-Brian
On 2020-10-13 17:22, Mel Beckman wrote:
You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)
Here's a Cisco brief on the topic:
https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwa...
I think all router vendors support this feature. Here's a similar article by Juniper:
https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/...
-mel beckman
On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:
We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
* Permit entries for P-t-P WAN subnets on peering links
* Permit entries for IP assignments to customers running multi-homed BGP
* The "permit ipv4 any any" at the end :)
The questions I wanted to ask the SP community are:
* What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
* How is it accomplished? Through static ACL or some flavor of uRPF?
* If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
* How did you test your filters when they were implemented?
Thanks a lot,
-Brian
So I have put together what I think is a reasonable and complete ACL. From my time in the enterprise world, I know that a good ingress ACL filters out traffic sourcing from: * Bogon blocks, like 0.0.0.0/8, 127.0.0.0/8, RFC1918 space, etc (well-documented in https://team-cymru.com/community-services/bogon-reference/bogon-reference-ht...) * RIR-assigned blocks I am announcing to the rest of the world However, I recognized a SP-specific case where we could receive legitimate traffic sourcing from our own IP blocks: customers running multi-homed BGP where we have assigned PA space to them. So I added "permit" statements for traffic sourcing from these blocks. Also, we have direct peering links that are numbered within our assigned prefixes. So we can use the same ACL with these peer interfaces and continue to have BGP work, I added "permit" statements for these point-to-point subnets. So the order of the statements is: * Permit where source is direct peer PtP networks * Permit where source is BGP customer PA prefix * Deny where source is bogon * Deny where source is our advertised prefixes * Permit all other traffic I considered BGP customer PI prefixes to be out of scope for ingress filtering, since the customer is likely to be multi-homing. Should we consider filtering them? The Team Cymru Secure IOS Template [https://www.cymru.com/Documents/secure-ios-template.html] also references an ICMP fragment drop entry on the ingress ACL. I think that's good for an enterprise network, but as an SP, I'm very hesitant to include this. Is this included in anyone else's transit / peer / IX ACL? Is there anything else that I'm not thinking of? Thanks, -Brian On 2020-10-14 09:25, Brian Knight via NANOG wrote:
Hi Marcos,
Thanks for your reply. But I am looking for guidance on traffic filtering, not BGP prefix filtering.
I have looked at BCP 84, and it's a good overview of the methods available to an ISP. My questions are more operational in nature and are not covered by the document. Of the choices presented in BCP 84, what do folks really use? If it's an ACL, what challenges have there been with updates? etc.
-Brian
On 2020-10-13 18:52, Marcos Manoni wrote:
Hi, Brian
Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated by: RFC8704 Enhanced Feasible-Path uRPF).
Ingress Access Lists require typically manual maintenance, but are the most bulletproof when done properly; typically, ingress access lists are best fit between the edge and the ISP when the configuration is not too dynamic if strict RPF is not an option, between ISPs if the number of used prefixes is low, or as an additional layer of protection
Ingress filters Best Practices https://www.ripe.net/support/training/material/bgp-operations-and-security-t... *Don’t accept BOGON ASNs *Don’t accept BOGON prefixes *Don’t accept your own prefix *Don’t accept default (unless you requested it) *Don’t accept prefixes that are too specific *Don’t accept if AS Path is too long *Create filters based on Internet Routing Registries
And also BGP Best Current Practices by Philip Smith http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
Regards.
El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG (<nanog@nanog.org>) escribió:
Hi Mel,
My understanding of uRPF is:
* Strict mode will permit a packet only if there is a route for the source IP in the RIB, and that route points to the interface where the packet was received
* Loose mode will permit a packet if there is a route for the source IP in the RIB. It does not matter where the route is pointed.
Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.
Loose mode won't work for us, because all of our own prefixes are in our RIB, and thus the uRPF check on a transit would never block anything.
Or am I missing something?
Thanks,
-Brian
On 2020-10-13 17:22, Mel Beckman wrote:
You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)
Here's a Cisco brief on the topic:
https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwa...
I think all router vendors support this feature. Here's a similar article by Juniper:
https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/...
-mel beckman
On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:
We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
* Permit entries for P-t-P WAN subnets on peering links
* Permit entries for IP assignments to customers running multi-homed BGP
* The "permit ipv4 any any" at the end :)
The questions I wanted to ask the SP community are:
* What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
* How is it accomplished? Through static ACL or some flavor of uRPF?
* If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
* How did you test your filters when they were implemented?
Thanks a lot,
-Brian
Hi Brian, "However, I recognized a SP-specific case where we could receive legitimate traffic sourcing from our own IP blocks: customers running multi-homed BGP where we have assigned PA space to them. So I added "permit" statements for traffic sourcing from these blocks." If your customers are not using your DNS in your ip space, you could deny traffic sourcing from your ip block coming from these customers based on the UDP port DST 53. Then, accept all the rest. It's just one more line in your ACL. If they are using your DNS though, this won't work. You might want to add all the amplification ports like 123, 161, etc Jean -----Original Message----- From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Brian Knight via NANOG Sent: Wednesday, October 14, 2020 6:43 PM To: nanog <nanog@nanog.org> Subject: Re: Ingress filtering on transits, peers, and IX ports So I have put together what I think is a reasonable and complete ACL. From my time in the enterprise world, I know that a good ingress ACL filters out traffic sourcing from: * Bogon blocks, like 0.0.0.0/8, 127.0.0.0/8, RFC1918 space, etc (well-documented in https://team-cymru.com/community-services/bogon-reference/bogon-reference-ht...) * RIR-assigned blocks I am announcing to the rest of the world However, I recognized a SP-specific case where we could receive legitimate traffic sourcing from our own IP blocks: customers running multi-homed BGP where we have assigned PA space to them. So I added "permit" statements for traffic sourcing from these blocks. Also, we have direct peering links that are numbered within our assigned prefixes. So we can use the same ACL with these peer interfaces and continue to have BGP work, I added "permit" statements for these point-to-point subnets. So the order of the statements is: * Permit where source is direct peer PtP networks * Permit where source is BGP customer PA prefix * Deny where source is bogon * Deny where source is our advertised prefixes * Permit all other traffic I considered BGP customer PI prefixes to be out of scope for ingress filtering, since the customer is likely to be multi-homing. Should we consider filtering them? The Team Cymru Secure IOS Template [https://www.cymru.com/Documents/secure-ios-template.html] also references an ICMP fragment drop entry on the ingress ACL. I think that's good for an enterprise network, but as an SP, I'm very hesitant to include this. Is this included in anyone else's transit / peer / IX ACL? Is there anything else that I'm not thinking of? Thanks, -Brian On 2020-10-14 09:25, Brian Knight via NANOG wrote:
Hi Marcos,
Thanks for your reply. But I am looking for guidance on traffic filtering, not BGP prefix filtering.
I have looked at BCP 84, and it's a good overview of the methods available to an ISP. My questions are more operational in nature and are not covered by the document. Of the choices presented in BCP 84, what do folks really use? If it's an ACL, what challenges have there been with updates? etc.
-Brian
On 2020-10-13 18:52, Marcos Manoni wrote:
Hi, Brian
Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated by: RFC8704 Enhanced Feasible-Path uRPF).
Ingress Access Lists require typically manual maintenance, but are the most bulletproof when done properly; typically, ingress access lists are best fit between the edge and the ISP when the configuration is not too dynamic if strict RPF is not an option, between ISPs if the number of used prefixes is low, or as an additional layer of protection
Ingress filters Best Practices https://www.ripe.net/support/training/material/bgp-operations-and-sec urity-training-course/BGP-Slides-Single.pdf *Don’t accept BOGON ASNs *Don’t accept BOGON prefixes *Don’t accept your own prefix *Don’t accept default (unless you requested it) *Don’t accept prefixes that are too specific *Don’t accept if AS Path is too long *Create filters based on Internet Routing Registries
And also BGP Best Current Practices by Philip Smith http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
Regards.
El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG (<nanog@nanog.org>) escribió:
Hi Mel,
My understanding of uRPF is:
* Strict mode will permit a packet only if there is a route for the source IP in the RIB, and that route points to the interface where the packet was received
* Loose mode will permit a packet if there is a route for the source IP in the RIB. It does not matter where the route is pointed.
Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.
Loose mode won't work for us, because all of our own prefixes are in our RIB, and thus the uRPF check on a transit would never block anything.
Or am I missing something?
Thanks,
-Brian
On 2020-10-13 17:22, Mel Beckman wrote:
You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)
Here's a Cisco brief on the topic:
https://tools.cisco.com/security/center/resources/unicast_reverse_pa th_forwarding
I think all router vendors support this feature. Here's a similar article by Juniper:
https://www.juniper.net/documentation/en_US/junos/topics/task/config uration/interfaces-configuring-unicast-rpf.html
-mel beckman
On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:
We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
* Permit entries for P-t-P WAN subnets on peering links
* Permit entries for IP assignments to customers running multi-homed BGP
* The "permit ipv4 any any" at the end :)
The questions I wanted to ask the SP community are:
* What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
* How is it accomplished? Through static ACL or some flavor of uRPF?
* If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
* How did you test your filters when they were implemented?
Thanks a lot,
-Brian
Thanks to the folks who responded to my messages on and off-list. A couple of folks have asked me to summarize the responses that I received. * Static ACL is currently the best way to protect a multi-homed network. Loose RPF may be used if bogon filtering is more important, but it does not provide anti-spoofing security. * Protect your infrastructure subnets with the ingress ACL [BCP 84 sec 3.2]. Loopbacks and point-to-point circuits can benefit from this. In the draft ACL, for example, I permit ICMP and traceroute over UDP, and block all else. * Do an egress ACL also, to prevent clutter from reaching the rest of the 'Net. Permit only your aggregate and customer prefixes going outbound. * As I worked through putting the ACLs together, I found that if one implements an egress ACL, then customer prefixes must be enumerated anyway. Once those are in an object group, it's easy to add an entry to the ingress ACL permitting traffic destined to customer PI space and aggregate space. Seems better than just permitting all traffic in. Our ACLs, both v4 and v6, now look like the following: Ingress * Deny to and from bogon networks, where bogon is either source or dest * Permit to and from WAN PtP subnets * For IPv6, also permit link-local IPs (fe80::/10) * Deny to and from multicast ranges 224.0.0.0/4 and ff00::/8 * Permit ICMP / traceroute over UDP to infrastructure * Deny all other traffic to infrastructure * Permit from customer PI / PA space * Deny from originated aggregate space * Permit all traffic to customer PI / PA space * Permit all traffic to aggregate space * Deny any any Egress * Deny to and from bogon networks * Permit to and from WAN PtP subnets * For IPv6, also permit link-local IPs * Deny to and from multicast range * Permit all traffic from customer PI / PA space * Permit all traffic from aggregate space * Deny any any We have started implementing the ACLs by blocking the bogon traffic only. The other deny rules are set up as permit rules for now with logging turned on. I'll review matching traffic before I switch the rules to deny. Future work also includes automating the updates to the object groups via IRR. BTW, Team Cymru didn't have any guidance around IPv6 bogons, so I put together the below object group based on the IANA IPv6 allocation list: https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unica.... Obviously this is only for space not yet allocated to RIRs. object-group network ipv6 IPV6-BOGON description Invalid IPV6 networks ::/3 4000::/3 6000::/3 8000::/3 a000::/3 c000::/3 e000::/4 f000::/5 f800::/6 fc00::/7 fe00::/9 fec0::/10 exit Thanks, -Brian On 2020-10-14 17:43, Brian Knight wrote:
So I have put together what I think is a reasonable and complete ACL. From my time in the enterprise world, I know that a good ingress ACL filters out traffic sourcing from:
* Bogon blocks, like 0.0.0.0/8, 127.0.0.0/8, RFC1918 space, etc (well-documented in https://team-cymru.com/community-services/bogon-reference/bogon-reference-ht...) * RIR-assigned blocks I am announcing to the rest of the world
However, I recognized a SP-specific case where we could receive legitimate traffic sourcing from our own IP blocks: customers running multi-homed BGP where we have assigned PA space to them. So I added "permit" statements for traffic sourcing from these blocks.
Also, we have direct peering links that are numbered within our assigned prefixes. So we can use the same ACL with these peer interfaces and continue to have BGP work, I added "permit" statements for these point-to-point subnets.
So the order of the statements is:
* Permit where source is direct peer PtP networks * Permit where source is BGP customer PA prefix * Deny where source is bogon * Deny where source is our advertised prefixes * Permit all other traffic
I considered BGP customer PI prefixes to be out of scope for ingress filtering, since the customer is likely to be multi-homing. Should we consider filtering them?
The Team Cymru Secure IOS Template [https://www.cymru.com/Documents/secure-ios-template.html] also references an ICMP fragment drop entry on the ingress ACL. I think that's good for an enterprise network, but as an SP, I'm very hesitant to include this. Is this included in anyone else's transit / peer / IX ACL?
Is there anything else that I'm not thinking of?
Thanks,
-Brian
On 2020-10-14 09:25, Brian Knight via NANOG wrote:
Hi Marcos,
Thanks for your reply. But I am looking for guidance on traffic filtering, not BGP prefix filtering.
I have looked at BCP 84, and it's a good overview of the methods available to an ISP. My questions are more operational in nature and are not covered by the document. Of the choices presented in BCP 84, what do folks really use? If it's an ACL, what challenges have there been with updates? etc.
-Brian
On 2020-10-13 18:52, Marcos Manoni wrote:
Hi, Brian
Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated by: RFC8704 Enhanced Feasible-Path uRPF).
Ingress Access Lists require typically manual maintenance, but are the most bulletproof when done properly; typically, ingress access lists are best fit between the edge and the ISP when the configuration is not too dynamic if strict RPF is not an option, between ISPs if the number of used prefixes is low, or as an additional layer of protection
Ingress filters Best Practices https://www.ripe.net/support/training/material/bgp-operations-and-security-t... *Don’t accept BOGON ASNs *Don’t accept BOGON prefixes *Don’t accept your own prefix *Don’t accept default (unless you requested it) *Don’t accept prefixes that are too specific *Don’t accept if AS Path is too long *Create filters based on Internet Routing Registries
And also BGP Best Current Practices by Philip Smith http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
Regards.
El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG (<nanog@nanog.org>) escribió:
Hi Mel,
My understanding of uRPF is:
* Strict mode will permit a packet only if there is a route for the source IP in the RIB, and that route points to the interface where the packet was received
* Loose mode will permit a packet if there is a route for the source IP in the RIB. It does not matter where the route is pointed.
Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.
Loose mode won't work for us, because all of our own prefixes are in our RIB, and thus the uRPF check on a transit would never block anything.
Or am I missing something?
Thanks,
-Brian
On 2020-10-13 17:22, Mel Beckman wrote:
You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)
Here's a Cisco brief on the topic:
https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwa...
I think all router vendors support this feature. Here's a similar article by Juniper:
https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/...
-mel beckman
On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:
We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
* Permit entries for P-t-P WAN subnets on peering links
* Permit entries for IP assignments to customers running multi-homed BGP
* The "permit ipv4 any any" at the end :)
The questions I wanted to ask the SP community are:
* What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
* How is it accomplished? Through static ACL or some flavor of uRPF?
* If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
* How did you test your filters when they were implemented?
Thanks a lot,
-Brian
term blocked-ports { from { protocol [ tcp udp ]; first-fragment; destination-port [ 0 sunrpc 135 netbios-ns netbios-dgm netbios-ssn 111 445 syslog 11211]; } then { sample; discard; } } and i block all external access to weak devices such as switches, pdus, ipmi, ... randy
Might filtering port 11211 like that not risk blocking random connections, where the operating system picked that port as source, which then becomes destination on the reply packets? tir. 20. okt. 2020 07.19 skrev Randy Bush <randy@psg.com>:
term blocked-ports { from { protocol [ tcp udp ]; first-fragment; destination-port [ 0 sunrpc 135 netbios-ns netbios-dgm netbios-ssn 111 445 syslog 11211]; } then { sample; discard; } }
and i block all external access to weak devices such as switches, pdus, ipmi, ...
randy
Randy, thank you for the reminder to look also at what services (L4 ports) should be generally blocked. As I was implementing a similar rule for logging purposes, I discovered an oddity with $VENDOR_C_XR ACLs. I created the following: object-group port TCPUDP-BLOCKED eq 0 eq sunrpc eq 445 range 137 139 exit ipv4 access-list IPV4-INET-IN 10 remark BCP 84 for transits, IX, and peering 101 remark *** Block bogon networks as src or dest *** 110 deny ipv4 net-group IPV4-BOGON any 111 deny ipv4 any net-group IPV4-BOGON 201 remark *** Blocked protocols PERMIT FOR NOW *** 210 permit udp any port-group TCPUDP-BLOCKED any log 211 permit udp any any port-group TCPUDP-BLOCKED log 212 permit tcp any port-group TCPUDP-BLOCKED any log 213 permit tcp any any port-group TCPUDP-BLOCKED log [snip] ipv4 access-list IPV4-INET-OUT 10 remark BCP 84 for transits, IX, and peering 101 remark *** Block bogon networks as src or dest *** 110 deny ipv4 net-group IPV4-BOGON any 111 deny ipv4 any net-group IPV4-BOGON 201 remark *** Blocked protocols PERMIT FOR NOW *** 210 permit udp any port-group TCPUDP-BLOCKED any log 211 permit udp any any port-group TCPUDP-BLOCKED log 212 permit tcp any port-group TCPUDP-BLOCKED any log 213 permit tcp any any port-group TCPUDP-BLOCKED log [snip] After I did this, logs on our syslog server started growing like crazy. It was full of entries like: 2020-10-21T01:47:17-05:00,info,RP/0/RSP1/CPU0:Oct 21 01:47:17.972 CDT: ipv4_acl_mgr[305]: %ACL-IPV4_ACL-6-IPACCESSLOGP : access-list IPV4-INET-OUT (210) permit udp on.net.ip.adr(0) -> off.net.ip.adr(0), 5 packets 2020-10-21T02:01:08-05:00,info,RP/0/RSP0/CPU0:Oct 21 02:01:08.490 CDT: ipv4_acl_mgr[263]: %ACL-IPV4_ACL-6-IPACCESSLOGP : access-list IPV4-INET-IN (210) permit udp off.net.ip.adr(0) -> on.net.ip.adr(0), 58 packets After wondering why in the world my customers were sending so much data on port 0, I found a few different sources saying that port 0 is commonly used in place of valid information when dealing with fragments. Turns out that $VENDOR_C_XR does this too. It wasn't clear why fragments would be matching that rule until I found the right vendor doc. The router will pass IP fragments with a "permit" ACL line as long as that fragment's layer 3 info matches the layer 3 information in the ACL. The router logs the packet similar the above: L4 protocol with source and dest port = 0. From the doc: ----- For an access-list entry containing Layer 3 and Layer 4 information: • The entry is applied to non-fragmented packets and initial fragments. • If the entry matches and is a permit statement, the packet or fragment is permitted. • If the entry matches and is a deny statement, the packet or fragment is denied. The entry is also applied to non-initial fragments in the following manner. Because non-initial fragments contain only Layer 3 information, only the Layer 3 portion of an access-list entry can be applied. If the Layer 3 portion of the access-list entry matches, and • If the entry is a permit statement, the non-initial fragment is permitted. • If the entry is a deny statement, the next access-list entry is processed. The deny statements are handled differently for non-initial fragments versus non-fragmented or initial fragments. ----- Since my rule's L3 info was permit any source to any destination, any IP fragment would match the rule, be passed, and be logged. The solution was to add rules explicitly permitting fragments above the layer 4 rules: ipv4 access-list IPV4-INET-IN 10 remark BCP 84 for transits, IX, and peering 101 remark *** Block bogon networks as src or dest *** 110 deny ipv4 net-group IPV4-BOGON any 111 deny ipv4 any net-group IPV4-BOGON 201 remark *** Blocked protocols PERMIT FOR NOW *** 203 permit ipv4 net-group IPV4-CUST any fragments 204 permit ipv4 net-group IPV4-BACKDOOR-HOSTS any fragments 205 permit ipv4 any net-group IPV4-BGP-AGG fragments 206 permit ipv4 any net-group IPV4-CUST fragments 210 permit udp any port-group TCPUDP-BLOCKED any log 211 permit udp any any port-group TCPUDP-BLOCKED log 212 permit tcp any port-group TCPUDP-BLOCKED any log 213 permit tcp any any port-group TCPUDP-BLOCKED log Logs are a lot calmer now in terms of new lines per minute, and far more relevant. When we switch those rules to deny statements, we can eliminate the rules specifically permitting fragments. Looks like $VENDOR_J makes things so much simpler for this task. Thanks, -Brian On 2020-10-20 00:18, Randy Bush wrote:
term blocked-ports { from { protocol [ tcp udp ]; first-fragment; destination-port [ 0 sunrpc 135 netbios-ns netbios-dgm netbios-ssn 111 445 syslog 11211]; } then { sample; discard; } }
and i block all external access to weak devices such as switches, pdus, ipmi, ...
randy
As a final update to this thread, we started blocking spoofed and invalid traffic as of early Thursday morning Nov 19th. So far, knock on wood, no reports of issues from our customer base. In addition, I've been able to verify with the security research team's test tool that we are no longer responding to the spoofed DNS requests. The ACL was implemented as follows: Ingress * Deny to and from bogon networks, where bogon is either source or dest * Deny invalid TCP and UDP ports (currently only port 0) [log] * Permit to and from transit / peer / IX connected subnets * For IPv6, also permit link-local IPs (fe80::/10) * Deny to and from multicast ranges 224.0.0.0/4 and ff00::/8 * Permit ICMP / traceroute over UDP to infrastructure * Deny all other traffic to infrastructure [log] * Permit from customer PI / PA space * Deny from originated aggregate space [log] * Permit all traffic to customer PI / PA space * Permit all traffic to aggregate space * Deny any any [log] Egress * Deny to and from bogon networks * Deny invalid ports [log] * Permit to and from transit / peer / IX connected subnets * For IPv6, also permit link-local IPs * Deny to and from multicast range * Permit all traffic from any source to customer PI / PA space * Permit all traffic from customer PI / PA space * Permit all traffic from aggregate space * Deny any any [log] Below I've included the specific $VENDOR_C config I implemented for the filtering, sans specifics on our IP blocks. I hope folks find this useful as a guide to their own efforts, and constructive criticism is always welcome. Future work includes: * Tightening the rules permitting access to/from the transit / peer / IX connected subnets, while keeping the ACL general enough for use on all Internet-facing interfaces * Automation of updates to aggregate and customer IP blocks (looking at using the irrpt project for this) Once more, to those who provided valuable input, thank you very much indeed! -Brian !------------- ! Static ACLs for Service Provider BCP 84 Compliance ! IOS XR config ! IPv4 object-group network ipv4 IPV4-BOGON description Invalid IPV4 networks 0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 127.0.0.0/8 169.254.0.0/16 172.16.0.0/12 192.0.0.0/24 192.0.2.0/24 192.168.0.0/16 198.18.0.0/15 198.51.100.0/24 203.0.113.0/24 240.0.0.0/4 exit object-group network ipv4 IPV4-TRAN-WAN description Transit WAN PtP subnets [Point to point /30's go here] exit object-group network ipv4 IPV4-IX description IX subnets [IX /24 and /23 subnets here] exit object-group network ipv4 IPV4-PEER-WAN description Direct peer WAN PtP subnets [Direct peer WAN IPs go here] exit object-group network ipv4 IPV4-BGP-AGG description ARIN IPV4 Aggregate Blocks [Aggregated IP blocks go here] exit object-group network ipv4 IPV4-INFRA description Infrastructure subnets to be protected [List of loopback blocks and backbone / core PtP /30's here] exit object-group network ipv4 IPV4-BACKDOOR-HOSTS description Hosts observed to be sending valid traffic via Internet [One-off hosts, active TCP or UDP traffic was observed during data collection] exit object-group network ipv4 IPV4-CUST [full list of all customer IP blocks] [Includes customer PI blocks, disaggregated PA from other providers,] [and PA assigned from your aggregate space] exit object-group port TCPUDP-BLOCKED eq 0 [additional ports to be generally blocked, list here] exit ipv4 access-list IPV4-INET-IN 10 remark BCP 84 for transits, IX, and peering 101 remark *** Block bogon networks as src or dest *** 110 deny ipv4 net-group IPV4-BOGON any 111 deny ipv4 any net-group IPV4-BOGON 201 remark *** Blocked protocols *** 210 deny udp any port-group TCPUDP-BLOCKED any log 211 deny udp any any port-group TCPUDP-BLOCKED log 212 deny tcp any port-group TCPUDP-BLOCKED any log 213 deny tcp any any port-group TCPUDP-BLOCKED log 301 remark *** Transit, IX, peer connected networks *** 310 permit ipv4 net-group IPV4-PEER-WAN any 311 permit ipv4 any net-group IPV4-PEER-WAN 312 permit ipv4 net-group IPV4-TRAN-WAN any 313 permit ipv4 any net-group IPV4-TRAN-WAN 314 permit ipv4 net-group IPV4-IX any 315 permit ipv4 any net-group IPV4-IX 401 remark *** Block multicast *** 410 deny ipv4 224.0.0.0/4 any 411 deny ipv4 any 224.0.0.0/4 501 remark *** Protect infrastructure subnets *** 510 deny icmp any net-group IPV4-INFRA fragments log 511 permit icmp any net-group IPV4-INFRA 512 permit udp any range 1024 65535 net-group IPV4-INFRA range 33435 33535 513 permit udp any range 33435 33535 net-group IPV4-INFRA range 1024 65535 515 deny ipv4 any net-group IPV4-INFRA 601 remark *** Customer Inet BGP Announced Prefixes *** 620 permit ipv4 net-group IPV4-CUST any 640 permit ipv4 net-group IPV4-BACKDOOR-HOSTS any 701 remark *** Block originated networks *** 710 deny ipv4 net-group IPV4-BGP-AGG any log 801 remark *** Permit traffic only to networks we announce *** 820 permit ipv4 any net-group IPV4-BGP-AGG 840 permit ipv4 any net-group IPV4-CUST 901 remark *** Deny all other traffic *** 910 deny ipv4 any any log exit ipv4 access-list IPV4-INET-OUT 10 remark BCP 84 for transits, IX, and peering 101 remark *** Block bogon networks as src or dest *** 110 deny ipv4 net-group IPV4-BOGON any 111 deny ipv4 any net-group IPV4-BOGON 201 remark *** Blocked protocols *** 210 deny udp any port-group TCPUDP-BLOCKED any log 211 deny udp any any port-group TCPUDP-BLOCKED log 212 deny tcp any port-group TCPUDP-BLOCKED any log 213 deny tcp any any port-group TCPUDP-BLOCKED log 301 remark *** Transit, IX, peer networks *** 310 permit ipv4 net-group IPV4-PEER-WAN any 311 permit ipv4 any net-group IPV4-PEER-WAN 312 permit ipv4 net-group IPV4-TRAN-WAN any 313 permit ipv4 any net-group IPV4-TRAN-WAN 314 permit ipv4 net-group IPV4-IX any 315 permit ipv4 any net-group IPV4-IX 401 remark *** Block multicast *** 410 deny ipv4 224.0.0.0/4 any 411 deny ipv4 any 224.0.0.0/4 601 remark *** Customer Inet BGP Announced Prefixes *** 620 permit ipv4 any net-group IPV4-CUST 640 permit ipv4 any net-group IPV4-BACKDOOR-HOSTS 801 remark *** Permit locally sourced traffic *** 820 permit ipv4 net-group IPV4-BGP-AGG any 840 permit ipv4 net-group IPV4-CUST any 901 remark *** Deny all other traffic *** 910 deny ipv4 any any log exit ! IPv6 object-group network ipv6 IPV6-BOGON description Invalid IPV6 networks ::/3 2001::/23 2001:2::/48 2001:10::/28 2001:db8::/32 2002::/16 3ffe::/16 4000::/3 6000::/3 8000::/3 a000::/3 c000::/3 e000::/4 f000::/5 f800::/6 fc00::/7 fe00::/9 fec0::/10 exit object-group network ipv6 IPV6-TRAN-WAN description Transit WAN PtP subnets [Point to point /126's go here] exit object-group network ipv6 IPV6-IX description IX subnets [IX /64 subnets here] exit object-group network ipv6 IPV6-PEER-WAN description Direct peer WAN PtP subnets [Direct peer WAN /126's go here] exit object-group network ipv6 IPV6-BGP-AGG description ARIN IPV6 Aggregate Blocks [Aggregated IP blocks go here] exit object-group network ipv6 IPV6-INFRA description Infrastructure subnets to be protected [List of loopback blocks and backbone / core PtP WAN IPs here] exit object-group network ipv6 IPV6-CUST [full list of all customer IP blocks] [Includes customer PI blocks, disaggregated PA from other providers,] [and PA assigned from your aggregate space] exit ipv6 access-list IPV6-INET-IN 10 remark BCP 84 for transits, IX, and peering 101 remark *** Block bogon networks as src or dest *** 110 deny ipv6 net-group IPV6-BOGON any 111 deny ipv6 any net-group IPV6-BOGON 201 remark *** Blocked protocols *** 210 deny udp any port-group TCPUDP-BLOCKED any log 211 deny udp any any port-group TCPUDP-BLOCKED log 212 deny tcp any port-group TCPUDP-BLOCKED any log 213 deny tcp any any port-group TCPUDP-BLOCKED log 301 remark *** Transit, IX, peer connected networks *** 310 permit ipv6 fe80::/10 any 311 permit ipv6 net-group IPV6-PEER-WAN any 312 permit ipv6 any net-group IPV6-PEER-WAN 313 permit ipv6 net-group IPV6-TRAN-WAN any 314 permit ipv6 any net-group IPV6-TRAN-WAN 315 permit ipv6 net-group IPV6-IX any 316 permit ipv6 any net-group IPV6-IX 401 remark *** Block multicast *** 410 deny ipv6 ff00::/8 any 411 deny ipv6 any ff00::/8 501 remark *** Protect infrastructure subnets *** 510 deny icmp any net-group IPV6-INFRA fragments log 511 permit icmp any net-group IPV6-INFRA 512 permit udp any range 1024 65535 net-group IPV6-INFRA range 33435 33535 513 permit udp any range 33435 33535 net-group IPV6-INFRA range 1024 65535 515 deny ipv6 any net-group IPV6-INFRA 601 remark *** Customer Inet BGP Announced Prefixes *** 620 permit ipv6 net-group IPV6-CUST any 701 remark *** Block networks we originate *** 710 deny ipv6 net-group IPV6-BGP-AGG any log 801 remark *** Permit traffic only to networks we announce *** 820 permit ipv6 any net-group IPV6-BGP-AGG 840 permit ipv6 any net-group IPV6-CUST 901 remark *** Deny all other traffic *** 910 deny ipv6 any any log exit ipv6 access-list IPV6-INET-OUT 10 remark BCP 84 for transits, IX, and peering 101 remark *** Block bogon networks as src or dest *** 110 deny ipv6 net-group IPV6-BOGON any 111 deny ipv6 any net-group IPV6-BOGON 201 remark *** Blocked protocols *** 210 deny udp any port-group TCPUDP-BLOCKED any log 211 deny udp any any port-group TCPUDP-BLOCKED log 212 deny tcp any port-group TCPUDP-BLOCKED any log 213 deny tcp any any port-group TCPUDP-BLOCKED log 301 remark *** Transit, IX, peer networks *** 310 permit ipv6 fe80::/10 any 311 permit ipv6 net-group IPV6-PEER-WAN any 312 permit ipv6 any net-group IPV6-PEER-WAN 313 permit ipv6 net-group IPV6-TRAN-WAN any 314 permit ipv6 any net-group IPV6-TRAN-WAN 315 permit ipv6 net-group IPV6-IX any 316 permit ipv6 any net-group IPV6-IX 401 remark *** Block multicast *** 410 deny ipv6 ff00::/8 any 411 deny ipv6 any ff00::/8 601 remark *** Customer Inet BGP Announced Prefixes *** 620 permit ipv6 any net-group IPV6-CUST 801 remark *** Permit locally sourced traffic *** 820 permit ipv6 net-group IPV6-BGP-AGG any 840 permit ipv6 net-group IPV6-CUST any 901 remark *** Deny all other traffic *** 910 deny ipv6 any any log exit ! On any transit, IX, or peer port ! interface TenGigE0/0/0/0 ipv4 access-group IPV4-INET-IN ingress ipv4 access-group IPV4-INET-OUT egress ipv6 access-group IPV6-INET-IN ingress ipv6 access-group IPV6-INET-OUT egress
Randy Bush Sent: Tuesday, October 20, 2020 6:19 AM
term blocked-ports { from { protocol [ tcp udp ]; first-fragment; destination-port [ 0 sunrpc 135 netbios-ns netbios-dgm netbios-ssn 111 445 syslog 11211]; } then { sample; discard; } }
Actually what's the latest in the net neutrality talks? Shouldn't these be just rate-limited rather than blocked? -transit traffic. (assuming ICMP is the only thing that can talk to infrastructure ranges & BGP to selected IPs with rest being dropped) adam
I took a slightly different approach for my mental exercise, expressed in IOS pigeon: object-group ip address AS65001 192.0.2.0 255.255.255.0 end object-group v6-network AS65001 2001:DB8::/32 end object-group ip address TwentyFiveGigE1/0/1 192.0.2.0 255.255.255.254 end object-group v6-network TwentyFiveGigE1/0/1 FE80::/10 2001:DB8::/127 end ip access-list extended TwentyFiveGigE1/0/1_IPV4_IN permit ip addrgroup TwentyFiveGigE1/0/1 any deny ip addrgroup AS65001 any permit ip any any end ipv6 access-list TwentyFiveGigE1/0/1_IPV6_IN permit ipv6 object-group TwentyFiveGigE1/0/1 any deny ipv6 object-group AS65001 any permit ipv6 any any end interface TwentyFiveGigE1/0/1 ip access-group TwentyFiveGigE1/0/1_IPV4_IN in ipv6 traffic-filter TwentyFiveGigE1/0/1_IPV6_IN in end I believe this is the minimum necessary to protect your AS from your netblock(s) ingress. Note: there may be technical and business reasons why you need to permit your netblock(s) ingress to your AS. I believe this concept could be used on any EBGP Inter-AS link, including peering addressed out of your own netblock. I remain unconvinced, but shrewd is the one who sees the calamity... Tim:> On Mon, Oct 19, 2020 at 8:40 PM Brian Knight via NANOG <nanog@nanog.org> wrote:
Thanks to the folks who responded to my messages on and off-list. A couple of folks have asked me to summarize the responses that I received.
* Static ACL is currently the best way to protect a multi-homed network. Loose RPF may be used if bogon filtering is more important, but it does not provide anti-spoofing security.
* Protect your infrastructure subnets with the ingress ACL [BCP 84 sec 3.2]. Loopbacks and point-to-point circuits can benefit from this. In the draft ACL, for example, I permit ICMP and traceroute over UDP, and block all else.
* Do an egress ACL also, to prevent clutter from reaching the rest of the 'Net. Permit only your aggregate and customer prefixes going outbound.
* As I worked through putting the ACLs together, I found that if one implements an egress ACL, then customer prefixes must be enumerated anyway. Once those are in an object group, it's easy to add an entry to the ingress ACL permitting traffic destined to customer PI space and aggregate space. Seems better than just permitting all traffic in.
Our ACLs, both v4 and v6, now look like the following:
Ingress
* Deny to and from bogon networks, where bogon is either source or dest * Permit to and from WAN PtP subnets * For IPv6, also permit link-local IPs (fe80::/10) * Deny to and from multicast ranges 224.0.0.0/4 and ff00::/8 * Permit ICMP / traceroute over UDP to infrastructure * Deny all other traffic to infrastructure * Permit from customer PI / PA space * Deny from originated aggregate space * Permit all traffic to customer PI / PA space * Permit all traffic to aggregate space * Deny any any
Egress
* Deny to and from bogon networks * Permit to and from WAN PtP subnets * For IPv6, also permit link-local IPs * Deny to and from multicast range * Permit all traffic from customer PI / PA space * Permit all traffic from aggregate space * Deny any any
We have started implementing the ACLs by blocking the bogon traffic only. The other deny rules are set up as permit rules for now with logging turned on. I'll review matching traffic before I switch the rules to deny.
Future work also includes automating the updates to the object groups via IRR.
BTW, Team Cymru didn't have any guidance around IPv6 bogons, so I put together the below object group based on the IANA IPv6 allocation list:
https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unica....
Obviously this is only for space not yet allocated to RIRs.
object-group network ipv6 IPV6-BOGON description Invalid IPV6 networks ::/3 4000::/3 6000::/3 8000::/3 a000::/3 c000::/3 e000::/4 f000::/5 f800::/6 fc00::/7 fe00::/9 fec0::/10 exit
Thanks,
-Brian
On 2020-10-14 17:43, Brian Knight wrote:
So I have put together what I think is a reasonable and complete ACL. From my time in the enterprise world, I know that a good ingress ACL filters out traffic sourcing from:
* Bogon blocks, like 0.0.0.0/8, 127.0.0.0/8, RFC1918 space, etc (well-documented in
https://team-cymru.com/community-services/bogon-reference/bogon-reference-ht... )
* RIR-assigned blocks I am announcing to the rest of the world
However, I recognized a SP-specific case where we could receive legitimate traffic sourcing from our own IP blocks: customers running multi-homed BGP where we have assigned PA space to them. So I added "permit" statements for traffic sourcing from these blocks.
Also, we have direct peering links that are numbered within our assigned prefixes. So we can use the same ACL with these peer interfaces and continue to have BGP work, I added "permit" statements for these point-to-point subnets.
So the order of the statements is:
* Permit where source is direct peer PtP networks * Permit where source is BGP customer PA prefix * Deny where source is bogon * Deny where source is our advertised prefixes * Permit all other traffic
I considered BGP customer PI prefixes to be out of scope for ingress filtering, since the customer is likely to be multi-homing. Should we consider filtering them?
The Team Cymru Secure IOS Template [https://www.cymru.com/Documents/secure-ios-template.html] also references an ICMP fragment drop entry on the ingress ACL. I think that's good for an enterprise network, but as an SP, I'm very hesitant to include this. Is this included in anyone else's transit / peer / IX ACL?
Is there anything else that I'm not thinking of?
Thanks,
-Brian
On 2020-10-14 09:25, Brian Knight via NANOG wrote:
Hi Marcos,
Thanks for your reply. But I am looking for guidance on traffic filtering, not BGP prefix filtering.
I have looked at BCP 84, and it's a good overview of the methods available to an ISP. My questions are more operational in nature and are not covered by the document. Of the choices presented in BCP 84, what do folks really use? If it's an ACL, what challenges have there been with updates? etc.
-Brian
On 2020-10-13 18:52, Marcos Manoni wrote:
Hi, Brian
Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated by: RFC8704 Enhanced Feasible-Path uRPF).
Ingress Access Lists require typically manual maintenance, but are the most bulletproof when done properly; typically, ingress access lists are best fit between the edge and the ISP when the configuration is not too dynamic if strict RPF is not an option, between ISPs if the number of used prefixes is low, or as an additional layer of protection
Ingress filters Best Practices
https://www.ripe.net/support/training/material/bgp-operations-and-security-t...
*Don’t accept BOGON ASNs *Don’t accept BOGON prefixes *Don’t accept your own prefix *Don’t accept default (unless you requested it) *Don’t accept prefixes that are too specific *Don’t accept if AS Path is too long *Create filters based on Internet Routing Registries
And also BGP Best Current Practices by Philip Smith http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
Regards.
El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG (<nanog@nanog.org>) escribió:
Hi Mel,
My understanding of uRPF is:
* Strict mode will permit a packet only if there is a route for the source IP in the RIB, and that route points to the interface where the packet was received
* Loose mode will permit a packet if there is a route for the source IP in the RIB. It does not matter where the route is pointed.
Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.
Loose mode won't work for us, because all of our own prefixes are in our RIB, and thus the uRPF check on a transit would never block anything.
Or am I missing something?
Thanks,
-Brian
On 2020-10-13 17:22, Mel Beckman wrote:
You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)
Here's a Cisco brief on the topic:
https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwa...
I think all router vendors support this feature. Here's a similar article by Juniper:
https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/...
-mel beckman
On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:
We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
* Permit entries for P-t-P WAN subnets on peering links
* Permit entries for IP assignments to customers running multi-homed BGP
* The "permit ipv4 any any" at the end :)
The questions I wanted to ask the SP community are:
* What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
* How is it accomplished? Through static ACL or some flavor of uRPF?
* If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
* How did you test your filters when they were implemented?
Thanks a lot,
-Brian
-- Tim:>
Hi, You are missing some IPv6 bogons prefixes. Check: http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf (page 51-53) http://bgpfilterguide.nlnog.net/guides/bogon_prefixes/#configuration-example... Regards.
On 15 Oct 2020, at 05:43, Brian Knight via NANOG <nanog@nanog.org> wrote:
I think that's good for an enterprise network, but as an SP, I'm very hesitant to include this. Is this included in anyone else's transit / peer / IX ACL?
This must not be done on peering links and on transit networks generally, as it breaks EDNS0, & everything that depends upon it, as well as some games, VoIP applications, etc. For consumer eyeball access networks only, making use of flow telemetry to analyze non-initial fragments destined for *those networks only, and excepting both one’s own recursive DNS server farms and well-known/well-operated public DNS recursive farms*, one can determine the normal rate of non-initial fragments in that very specific context and utilize QoS to police it down, leaving plenty of headroom for upticks in legitimate traffic. And with regards to disallowing one’s own address space except in special topological circumstances, it’s great to see that this initial topic has sparked renewed interest in both ingress and egress filtering. It’s highly laudable, and some good example are being posted and useful discussion taking place. It should be noted, however, that filtering one’s own prefixes at one’s peering edge in the more general cases isn’t a new concept; this has been a BCP recommendation for more than 20 years. For example, it’s referenced on p.75 of this .pdf slide deck on the topic of infrastructure self-protection, which was last revised 11 years ago: <https://app.box.com/s/osk4po8ietn1zrjjmn8b> From the way this ’new’ set of findings has been promulgated, it sounds as if the authors of the paper didn’t necessarily understand that this is a longstanding recommendation. That doesn’t in any way detract from the value of their study, mind; and perhaps they were aware, and that information simply hasn’t been communicated in this context. Also, a more proximate risk than DNS cache-poisoning or the specific, impractical, never-seen-in-the-wild DNS DDoS attack methodology cited, is for operators who aren’t filtering their own prefixes on ingres, and who’re also relying solely on iACLs to prevent off-net access to their recursive DNS server farms, thus allowing their abuse in DNS reflection/amplification attacks against their own infrastructure and access customers from outside their network. Filtering one’s own prefixes on ingress whenever possible and to the degree of granularity possible, along with making use not only of iACLs but on-server configurations to disallow off-network abuse of recursive DNS servers, are highly recommended. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
Brian Knight via NANOG wrote on 13/10/2020 23:49:
Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.
there's no "almost" about it: strict mode is unfeasible for both transit and IX ports. Nick
Great for customer-facing interfaces, though. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Nick Hilliard" <nick@foobar.org> To: "Brian Knight" <ml@knight-networks.com> Cc: nanog@nanog.org Sent: Wednesday, October 14, 2020 3:12:22 AM Subject: Re: Ingress filtering on transits, peers, and IX ports Brian Knight via NANOG wrote on 13/10/2020 23:49:
Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.
there's no "almost" about it: strict mode is unfeasible for both transit and IX ports. Nick
On Tue, Oct 13, 2020 at 05:49:42PM -0500, Brian Knight via NANOG wrote:
Hi Mel,
My understanding of uRPF is:
* Strict mode will permit a packet only if there is a route for the source IP in the RIB, and that route points to the interface where the packet was received
* Loose mode will permit a packet if there is a route for the source IP in the RIB. It does not matter where the route is pointed.
Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.
Loose mode won't work for us, because all of our own prefixes are in our RIB, and thus the uRPF check on a transit would never block anything.
You'll be surprised at the garbage you would drop that you can't return. - Jared
Aside from the BCPs currently being discussed for ingress filtering, I would be very interested in seeing what this traffic looked like from the perspective of your DNS servers' logs. I assume you're talking about customer facing recursive/caching resolvers, and not authoritative-only nameservers. Considering that one can run an instance of an anycasted recursive nameserver, under heavy load for a very large number of clients, on a $600 1U server these days... I wonder what exactly the threat model is. Reverse amplification of DNS traffic returning to the spoofed IPs for DoS purposes, such as to cause the nameserver to DoS a single /32 endpoint IP being targeted, as in common online gaming disputes? What volume of pps or Mbps would appear as spurious traffic as a result of this attack? On Tue, Oct 13, 2020 at 3:14 PM Brian Knight via NANOG <nanog@nanog.org> wrote:
We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
* Permit entries for P-t-P WAN subnets on peering links * Permit entries for IP assignments to customers running multi-homed BGP * The "permit ipv4 any any" at the end :)
The questions I wanted to ask the SP community are:
* What traffic filtering do you do on your transits, on IX ports, and your direct peering links? * How is it accomplished? Through static ACL or some flavor of uRPF? * If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update? * How did you test your filters when they were implemented?
Thanks a lot,
-Brian
Once upon a time, Eric Kuhnke <eric.kuhnke@gmail.com> said:
Considering that one can run an instance of an anycasted recursive nameserver, under heavy load for a very large number of clients, on a $600 1U server these days... I wonder what exactly the threat model is.
A customer forwarded one of these notices to us - looked like it's about recursive DNS cache poisoning. It's been a while since I looked closely, but I thought modern recursive DNS software was pretty resistant to that, and anyway, the real answer to that is DNSSEC. I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email from some group I've never heard of (and haven't AFAIK engaged the community about their "new" attack, scans, or notices)... seems more like shameless self promotion. -- Chris Adams <cma@cmadams.net>
If I had a dollar for every 'scary security alert' email received in a NOC email inbox from a 'security researcher group' that is the results of a port scan, or some small subset of trojan infected residential endpoint computers attempting outbound connections on ($common_service_port), or similar... On Tue, Oct 13, 2020 at 7:50 PM Chris Adams <cma@cmadams.net> wrote:
Once upon a time, Eric Kuhnke <eric.kuhnke@gmail.com> said:
Considering that one can run an instance of an anycasted recursive nameserver, under heavy load for a very large number of clients, on a $600 1U server these days... I wonder what exactly the threat model is.
A customer forwarded one of these notices to us - looked like it's about recursive DNS cache poisoning. It's been a while since I looked closely, but I thought modern recursive DNS software was pretty resistant to that, and anyway, the real answer to that is DNSSEC.
I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email from some group I've never heard of (and haven't AFAIK engaged the community about their "new" attack, scans, or notices)... seems more like shameless self promotion.
-- Chris Adams <cma@cmadams.net>
On 10/13/20 8:04 PM, Eric Kuhnke wrote:
If I had a dollar for every 'scary security alert' email received in a NOC email inbox from a 'security researcher group' that is the results of a port scan, or some small subset of trojan infected residential endpoint computers attempting outbound connections on ($common_service_port), or similar...
I get stupid automated "abuse" notices all the time about being an evil haxx0r, which is actually just having egress proxy enabled on GGC. The most crazy email I've had so far was saying that I "breached Section 4 of the Terms and Conditions of the domain" and that my as (the AS the GGC nodes are behind) is "to immediately cease and desist" followed by a bunch of BS about how their IP addresses are restricted and no crawl rights have been granted blah blah blah.
On Oct 13, 2020, at 8:49 PM, Chris Adams <cma@cmadams.net> wrote:
Once upon a time, Eric Kuhnke <eric.kuhnke@gmail.com> said:
Considering that one can run an instance of an anycasted recursive nameserver, under heavy load for a very large number of clients, on a $600 1U server these days... I wonder what exactly the threat model is.
A customer forwarded one of these notices to us - looked like it's about recursive DNS cache poisoning.
In part, yes. More generally, it's about allowing spoofed-source packets in your front door, appearing to be from your own network, and what a bad actor could do with them. The probes from the experiment were harmless. But if there were malicious intent, this access could facilitate cache poisoning, depending on your DNS server configuration.
It's been a while since I looked closely, but I thought modern recursive DNS software was pretty resistant to that, and anyway, the real answer to that is DNSSEC.
It is. But DNSSEC requires support both on the side of the resolver (validation enabled) and the authoritative server (zone signed). Adoption is still far from universal. There are efforts to improve that, but it can't be your only hope, in its current state. But, perhaps more importantly, cache poisoning is not the only concern. Other vulnerable DNS (for example) configurations might be exploited by a single packet being received and acted on as "trusted".
I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email
Crafting wording in an alert email such that it should both be taken seriously and it doesn't sound too dramatic is hard. We have gotten many positive responses. But we've also gotten some *meh*. In the end we made a choice about whether individual reach-out was important and worth the effort, ahead of future publication and presentation. We decided that it was. Many operators have agreed with us. But I get that not everyone will feel the same about it.
from some group I've never heard of (and haven't AFAIK engaged the community about their "new" attack, scans, or notices)
I suppose it depends on your definition of "engage the community". I think that's what we're doing right now. We're also no stranger to NANOG (though perhaps more of a lurker on the mailing list). But community is a much broader term. And anyway, there is some order to this whole thing, and broader announcements will come later. Cheers, Casey
On 15 Oct 2020, at 04:09, Casey Deccio <casey@deccio.net> wrote:
On Oct 13, 2020, at 8:49 PM, Chris Adams <cma@cmadams.net> wrote:
Once upon a time, Eric Kuhnke <eric.kuhnke@gmail.com> said:
Considering that one can run an instance of an anycasted recursive nameserver, under heavy load for a very large number of clients, on a $600 1U server these days... I wonder what exactly the threat model is.
A customer forwarded one of these notices to us - looked like it's about recursive DNS cache poisoning.
In part, yes. More generally, it's about allowing spoofed-source packets in your front door, appearing to be from your own network, and what a bad actor could do with them. The probes from the experiment were harmless. But if there were malicious intent, this access could facilitate cache poisoning, depending on your DNS server configuration.
It's been a while since I looked closely, but I thought modern recursive DNS software was pretty resistant to that, and anyway, the real answer to that is DNSSEC.
It is. But DNSSEC requires support both on the side of the resolver (validation enabled) and the authoritative server (zone signed). Adoption is still far from universal. There are efforts to improve that, but it can't be your only hope, in its current state.
But, perhaps more importantly, cache poisoning is not the only concern. Other vulnerable DNS (for example) configurations might be exploited by a single packet being received and acted on as "trusted”.
I know Casey knows this, but to the rest of you, this is why TSIG, SIG(0) and DNS COOKIE where invented. IP addresses, especially IP addresses on UDP packets, are not trustworthy. If you are not using one or more of these when sending a query you should be updating your DNS software. If you are a ISP that is purchasing CPEs you should be requiring that the CPE uses these by default. If you are purchasing other equipment it also should be using it by default. Fixing security issues requires everyone to play their part. +10% (and growing) of the worlds authoritative DNS servers support DNS COOKIE. I don’t have measurements for recursive servers. For it to be of benefit the clients also need to be sending requests with an DNS COOKIE option present. Many recursive servers send this option by default as well. The option was design to allow independent upgrade of servers and clients to occur. The only coordination required is within a anycast server cluster where all the servers need to support the option and use a common shared secret and algorithm. Clients will workaround shared secret / algorithm misconfigurations. Mark
I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email
Crafting wording in an alert email such that it should both be taken seriously and it doesn't sound too dramatic is hard. We have gotten many positive responses. But we've also gotten some *meh*. In the end we made a choice about whether individual reach-out was important and worth the effort, ahead of future publication and presentation. We decided that it was. Many operators have agreed with us. But I get that not everyone will feel the same about it.
from some group I've never heard of (and haven't AFAIK engaged the community about their "new" attack, scans, or notices)
I suppose it depends on your definition of "engage the community". I think that's what we're doing right now. We're also no stranger to NANOG (though perhaps more of a lurker on the mailing list). But community is a much broader term. And anyway, there is some order to this whole thing, and broader announcements will come later.
Cheers, Casey
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
Hi Eric, I shot a message over the folk who did the testing for more info about their test. If I'm able to find anything useful in our logs from their detail, I'll post it to the list. The message referenced DNS cache poisoning and DDOS amplification, so it seemed fairly general and more focused on whether ASes accepted spoofed traffic. They also referenced the new NXNSAttack, which I did not know about previously. Thanks, -Brian On 2020-10-13 20:49, Eric Kuhnke wrote:
Aside from the BCPs currently being discussed for ingress filtering, I would be very interested in seeing what this traffic looked like from the perspective of your DNS servers' logs.
I assume you're talking about customer facing recursive/caching resolvers, and not authoritative-only nameservers.
Considering that one can run an instance of an anycasted recursive nameserver, under heavy load for a very large number of clients, on a $600 1U server these days... I wonder what exactly the threat model is.
Reverse amplification of DNS traffic returning to the spoofed IPs for DoS purposes, such as to cause the nameserver to DoS a single /32 endpoint IP being targeted, as in common online gaming disputes?
What volume of pps or Mbps would appear as spurious traffic as a result of this attack?
On Tue, Oct 13, 2020 at 3:14 PM Brian Knight via NANOG <nanog@nanog.org> wrote:
We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
* Permit entries for P-t-P WAN subnets on peering links * Permit entries for IP assignments to customers running multi-homed BGP * The "permit ipv4 any any" at the end :)
The questions I wanted to ask the SP community are:
* What traffic filtering do you do on your transits, on IX ports, and your direct peering links? * How is it accomplished? Through static ACL or some flavor of uRPF? * If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update? * How did you test your filters when they were implemented?
Thanks a lot,
-Brian
I too would like to know more about their methodology and actual tangibles ideally in the form of PCAPs. On 10/14/20 4:56 PM, Brian Knight via NANOG wrote:
Hi Eric,
I shot a message over the folk who did the testing for more info about their test. If I'm able to find anything useful in our logs from their detail, I'll post it to the list.
The message referenced DNS cache poisoning and DDOS amplification, so it seemed fairly general and more focused on whether ASes accepted spoofed traffic. They also referenced the new NXNSAttack, which I did not know about previously.
Thanks,
-Brian
Hi Bryan,
On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net> wrote:
I too would like to know more about their methodology
We've written up our methodology and results in a paper that will be available in a few weeks. Happy to post it here if folks are interested. Obviously, no networks are individually identified; it's all aggregate. Also, we're working on a self-test tool, but it's not quite ready yet. Sorry.
and actual tangibles ideally in the form of PCAPs.
What do you mean by "tangibles in the form of PCAPs"? Casey
"What do you mean by "tangibles in the form of PCAPs”” I think Brian means packet captures, in PCAP format, so that we can observe example testing via Wireshark or some other protocol analyzer. -mel
On Oct 14, 2020, at 11:52 AM, Casey Deccio <casey@deccio.net> wrote:
Hi Bryan,
On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net> wrote:
I too would like to know more about their methodology
We've written up our methodology and results in a paper that will be available in a few weeks. Happy to post it here if folks are interested. Obviously, no networks are individually identified; it's all aggregate.
Also, we're working on a self-test tool, but it's not quite ready yet. Sorry.
and actual tangibles ideally in the form of PCAPs.
What do you mean by "tangibles in the form of PCAPs"?
Casey
I think he means packet captures from an example, voluntarily-tested recursive nameserver subject to this attack. On Wed, Oct 14, 2020 at 11:53 AM Casey Deccio <casey@deccio.net> wrote:
Hi Bryan,
On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net> wrote:
I too would like to know more about their methodology
We've written up our methodology and results in a paper that will be available in a few weeks. Happy to post it here if folks are interested. Obviously, no networks are individually identified; it's all aggregate.
Also, we're working on a self-test tool, but it's not quite ready yet. Sorry.
and actual tangibles ideally in the form of PCAPs.
What do you mean by "tangibles in the form of PCAPs"?
Casey
On Oct 14, 2020, at 3:34 PM, Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
I think he means packet captures from an example, voluntarily-tested recursive nameserver subject to this attack.
Thanks. We have updated all the report pages with a self-test tool specific to the network associated with the report. This should allow a network admin that received our report to check whether or not the condition still exists and to perform a packet capture from whatever vantage point they want in their network. A more general tool (i.e., for anyone to use) will be made available in the future. Cheers, Casey
On Wed, Oct 14, 2020 at 11:53 AM Casey Deccio <casey@deccio.net <mailto:casey@deccio.net>> wrote: Hi Bryan,
On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net <mailto:bryan@shout.net>> wrote:
I too would like to know more about their methodology
We've written up our methodology and results in a paper that will be available in a few weeks. Happy to post it here if folks are interested. Obviously, no networks are individually identified; it's all aggregate.
Also, we're working on a self-test tool, but it's not quite ready yet. Sorry.
and actual tangibles ideally in the form of PCAPs.
What do you mean by "tangibles in the form of PCAPs"?
Casey
All DNS resolvers discovered on our network belong to customers. Our own resolvers, running unbound, were not discovered. While filtering same AS on ingress could help those customers (but only one was a open relay), filtering bogons is something the customer can also do. Or the software can be fixed. Do we really expect the ISP to implement firewalls instead of customers upgrading software? I also note that apparently our own ISPs (transits) do not filter bogons either. The above is a principal question. I am going to filter bogons, it just is not very high on my long list of stuff to do. Regards Baldur ons. 14. okt. 2020 20.53 skrev Casey Deccio <casey@deccio.net>:
Hi Bryan,
On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net> wrote:
I too would like to know more about their methodology
We've written up our methodology and results in a paper that will be available in a few weeks. Happy to post it here if folks are interested. Obviously, no networks are individually identified; it's all aggregate.
Also, we're working on a self-test tool, but it's not quite ready yet. Sorry.
and actual tangibles ideally in the form of PCAPs.
What do you mean by "tangibles in the form of PCAPs"?
Casey
Simple, All stub autonomous systems should have a simple egress ACL allowing only PI of their customers and their own PAs -it’s a simple ACL at each AS-Exit points (towards transits/peers), that’s it. -not sure why this isn’t the first sentence in every BCP and “security bulletin”… adam From: NANOG <nanog-bounces+adamv0025=netconsultings.com@nanog.org> On Behalf Of Baldur Norddahl Sent: Thursday, October 15, 2020 8:38 AM To: nanog@nanog.org Subject: Re: Ingress filtering on transits, peers, and IX ports All DNS resolvers discovered on our network belong to customers. Our own resolvers, running unbound, were not discovered. While filtering same AS on ingress could help those customers (but only one was a open relay), filtering bogons is something the customer can also do. Or the software can be fixed. Do we really expect the ISP to implement firewalls instead of customers upgrading software? I also note that apparently our own ISPs (transits) do not filter bogons either. The above is a principal question. I am going to filter bogons, it just is not very high on my long list of stuff to do. Regards Baldur ons. 14. okt. 2020 20.53 skrev Casey Deccio <casey@deccio.net <mailto:casey@deccio.net> >: Hi Bryan,
On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net <mailto:bryan@shout.net> > wrote:
I too would like to know more about their methodology
We've written up our methodology and results in a paper that will be available in a few weeks. Happy to post it here if folks are interested. Obviously, no networks are individually identified; it's all aggregate. Also, we're working on a self-test tool, but it's not quite ready yet. Sorry.
and actual tangibles ideally in the form of PCAPs.
What do you mean by "tangibles in the form of PCAPs"? Casey
Hey,
All stub autonomous systems should have a simple egress ACL allowing only PI of their customers and their own PAs -it’s a simple ACL at each AS-Exit points (towards transits/peers), that’s it.
-not sure why this isn’t the first sentence in every BCP and “security bulletin”…
I will venture a guess. 1) it's very specific scenario to be stubby and have downstream PI 2) it won't address customers spoofing each other arbitrarily and customer1 spoofing as customer2 on the internet, giving large chunk of the utility of spoofing even with protection in place How do you maintain that ACL? Why doesn't that same mechanism allow ingress ACL on the customer port? Your proposal looks low utility for work needed. -- ++ytti
From: Saku Ytti <saku@ytti.fi> Sent: Thursday, October 15, 2020 11:12 AM
Hey,
Hey Saku,
All stub autonomous systems should have a simple egress ACL allowing only PI of their customers and their own PAs -it’s a simple ACL at each AS-Exit points (towards transits/peers), that’s it.
-not sure why this isn’t the first sentence in every BCP and “security bulletin”…
I will venture a guess.
1) it's very specific scenario to be stubby and have downstream PI Yes it is and in most cases it's about "I have these few /XYs from which I address my customers (eyeballs) so nothing outside of these should ever leave my AS" -as simple as that.
2) it won't address customers spoofing each other arbitrarily and customer1 spoofing as customer2 on the internet, giving large chunk of the utility of spoofing even with protection in place
Yes its not 100% effective as you pointed out, However if every stub AS implemented this egress ACL on the few of their transit/peering links, there would be a lot less inter-AS garbage floating around.
How do you maintain that ACL? Well you're gonna update the ACL every time you acquire a new PA space (every now and then), or when another customer wants you route their PI (rare, simply cause you're a stub AS with mostly eyeballs). It's one ACL you need to update on every box with AS-exit links -usually there are not that many in a stub-AS.
Why doesn't that same mechanism allow ingress ACL on the customer port? Your proposal looks low utility for work needed.
Yes one should absolutely do that, but... But considering to become a good netizen what is more work? a) Testing and the enabling uRPF on every customer facing box or setting up precise ACLs on every customer facing port, and then maintaining all that? b) Gathering all your PAs (potentially PIs) (hint: show bgp nei x.x.x.x advertised routes) crafting an ACL and apply it on several peering/transit links? One of them is couple of weeks work and one is an afternoon job. adam
On Thu, 15 Oct 2020 at 15:14, <adamv0025@netconsultings.com> wrote:
Yes one should absolutely do that, but... But considering to become a good netizen what is more work? a) Testing and the enabling uRPF on every customer facing box or setting up precise ACLs on every customer facing port, and then maintaining all that? b) Gathering all your PAs (potentially PIs) (hint: show bgp nei x.x.x.x advertised routes) crafting an ACL and apply it on several peering/transit links? One of them is couple of weeks work and one is an afternoon job.
I am not fan of uRPF, expensive for what it does. But I don't view it as an alternative here, I view it as either adding an ACE on all egresses on egress direction or adding ACE on the ingress where customer is on ingress direction. To me these options seem equally complex but the latter one seems superior. -- ++ytti
We deploy urpf strict on all customer end-host and broadband circuits. In this scenario urpf = ingress acl I don't have to think about. We deploy urpf loose on all customer multihomed DIA circuits. I dont this makes sense - ingress packet acl would be more sane. Any flavour of urpf on upstream transit or peering would be challenging. Ingress packet acl dropping source = own+customer prefix might make sense depending on your AS topology. You might argue that ingress packet acl would be operationally simpler on customer and upstream, as you could cover all scenarios. On Thu, Oct 15, 2020 at 10:05 AM Saku Ytti <saku@ytti.fi> wrote:
On Thu, 15 Oct 2020 at 15:14, <adamv0025@netconsultings.com> wrote:
Yes one should absolutely do that, but... But considering to become a good netizen what is more work? a) Testing and the enabling uRPF on every customer facing box or setting up precise ACLs on every customer facing port, and then maintaining all that? b) Gathering all your PAs (potentially PIs) (hint: show bgp nei x.x.x.x advertised routes) crafting an ACL and apply it on several peering/transit links? One of them is couple of weeks work and one is an afternoon job.
I am not fan of uRPF, expensive for what it does. But I don't view it as an alternative here, I view it as either adding an ACE on all egresses on egress direction or adding ACE on the ingress where customer is on ingress direction.
To me these options seem equally complex but the latter one seems superior.
-- ++ytti
-- Tim:>
On Thu, 15 Oct 2020 at 17:22, Tim Durack <tdurack@gmail.com> wrote:
We deploy urpf strict on all customer end-host and broadband circuits. In this scenario urpf = ingress acl I don't have to think about.
But you have to think about what prefixes a customer has. If BGP you need to generate prefix-list, if static you need to generate a static route. As you already have to know and manage this information, what is the incremental cost to also emit an ACL? -- ++ytti
Saku Ytti wrote on 15/10/2020 15:29:
But you have to think about what prefixes a customer has. If BGP you need to generate prefix-list, if static you need to generate a static route. As you already have to know and manage this information, what is the incremental cost to also emit an ACL?
the unfortunate reality is that many networks are run by CLI jockeys, so the incremental cost of this can be high. There are no good general-purpose networking sources of truth, which means that usually provisioning databases need to be highly customised, which is only worth it if the scale merits it. Nick
From: Saku Ytti <saku@ytti.fi> Sent: Thursday, October 15, 2020 3:30 PM
On Thu, 15 Oct 2020 at 17:22, Tim Durack <tdurack@gmail.com> wrote:
We deploy urpf strict on all customer end-host and broadband circuits. In this scenario urpf = ingress acl I don't have to think about.
But you have to think about what prefixes a customer has. If BGP you need to generate prefix-list, if static you need to generate a static route. As you already have to know and manage this information, what is the incremental cost to also emit an ACL?
Actually ideally there would be a feature/knob to automatically sync BGP (and static routes) with packet filters. adam
Once upon a time, adamv0025@netconsultings.com <adamv0025@netconsultings.com> said:
Actually ideally there would be a feature/knob to automatically sync BGP (and static routes) with packet filters.
Junos has prefix-lists that can be referenced in both BGP policy and firewall statements. -- Chris Adams <cma@cmadams.net>
Chris Adams Sent: Thursday, October 15, 2020 3:59 PM
Once upon a time, adamv0025@netconsultings.com <adamv0025@netconsultings.com> said:
Actually ideally there would be a feature/knob to automatically sync BGP (and static routes) with packet filters.
Junos has prefix-lists that can be referenced in both BGP policy and firewall statements.
What I mean was a firewall filter that would get updated with accordance to prefixes received/advertised via BGP, even better if this was a default behaviour. adam
On Thu, Oct 15, 2020 at 10:30 AM Saku Ytti <saku@ytti.fi> wrote:
On Thu, 15 Oct 2020 at 17:22, Tim Durack <tdurack@gmail.com> wrote:
We deploy urpf strict on all customer end-host and broadband circuits. In this scenario urpf = ingress acl I don't have to think about.
But you have to think about what prefixes a customer has. If BGP you need to generate prefix-list, if static you need to generate a static route. As you already have to know and manage this information, what is the incremental cost to also emit an ACL?
-- ++ytti
"You might argue that ingress packet acl would be operationally simpler on customer and upstream, as you could cover all scenarios." Although for a static customer urpf is hard to beat... -- Tim:>
This is about ingress ACL not egress. tor. 15. okt. 2020 12.00 skrev <adamv0025@netconsultings.com>:
Simple,
All stub autonomous systems should have a simple egress ACL allowing only PI of their customers and their own PAs -it’s a simple ACL at each AS-Exit points (towards transits/peers), that’s it.
-not sure why this isn’t the first sentence in every BCP and “security bulletin”…
adam
*From:* NANOG <nanog-bounces+adamv0025=netconsultings.com@nanog.org> *On Behalf Of *Baldur Norddahl *Sent:* Thursday, October 15, 2020 8:38 AM *To:* nanog@nanog.org *Subject:* Re: Ingress filtering on transits, peers, and IX ports
All DNS resolvers discovered on our network belong to customers. Our own resolvers, running unbound, were not discovered.
While filtering same AS on ingress could help those customers (but only one was a open relay), filtering bogons is something the customer can also do. Or the software can be fixed. Do we really expect the ISP to implement firewalls instead of customers upgrading software?
I also note that apparently our own ISPs (transits) do not filter bogons either.
The above is a principal question. I am going to filter bogons, it just is not very high on my long list of stuff to do.
Regards
Baldur
ons. 14. okt. 2020 20.53 skrev Casey Deccio <casey@deccio.net>:
Hi Bryan,
On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net> wrote:
I too would like to know more about their methodology
We've written up our methodology and results in a paper that will be available in a few weeks. Happy to post it here if folks are interested. Obviously, no networks are individually identified; it's all aggregate.
Also, we're working on a self-test tool, but it's not quite ready yet. Sorry.
and actual tangibles ideally in the form of PCAPs.
What do you mean by "tangibles in the form of PCAPs"?
Casey
Speaking as an ISP: Most of the ISP networks I manage are multi-homed, and I don't think uRPF provides the knobs to ensure legitimate traffic doesn't get dropped in some cases, so we use static ACLs at the upstream edge on ingress (and egress). These need updated any time new IP space is added to the network (not very often). Ingress ACL: Discard if source or destination is a bogon, RFC 1912, RFC1918; discard any traffic sourced from our own IP space (spoofed); discard any traffic that is not destined for our IP space (bad/mis-routed). Egress ACL: Discard if source or destination is a bogon, RFC 1912, RFC1918; discard any traffic that is not sourced from our IP space (spoofed). I believe in a policy of non-blocking and being net neutral, but if any TCP/UDP ports, IP protocols, or IP options are blocked you might add them to the above ACLs. On customer facing ports we use uRPF strict. Why? It's easy (one line to implement, zero maintenance, and it works well on our Cisco ASR1k/9k platforms)! Our customers are all single homed. For a single homed enterprises, service providers, and end-users I'd recommend uRPF strict. Why? Again, it's dead simple. I would love to see this be the default on all home, prosumer, and branch office oriented platform. Linux does this by default with the rp_filter kernel option. I suspect that networking gear based on Linux probably leaves this at the default setting (strict mode). In practice I don't know that I've ever used uRPF loose mode. ACLs have counters to verify they are working. I might have confirmed uRPF was working as intended the first time I implemented it, but otherwise I've used ACLs often enough to trust they are working as configured and trust the same for uRPF. --Blake On 10/13/2020 5:13 PM, Brian Knight via NANOG wrote:
We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
* Permit entries for P-t-P WAN subnets on peering links * Permit entries for IP assignments to customers running multi-homed BGP * The "permit ipv4 any any" at the end :)
The questions I wanted to ask the SP community are:
* What traffic filtering do you do on your transits, on IX ports, and your direct peering links? * How is it accomplished? Through static ACL or some flavor of uRPF? * If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update? * How did you test your filters when they were implemented?
Thanks a lot,
-Brian
participants (23)
-
adamv0025＠netconsultings.com -
Baldur Norddahl -
Blake Hudson -
Brandon Martin -
Brian Knight -
Bryan Holloway -
Casey Deccio -
Chris Adams -
Dobbins, Roland -
Eric Kuhnke -
Jared Mauch -
Jean St-Laurent -
Marcos Manoni -
Mark Andrews -
Matt Harris -
Mel Beckman -
Mike Hammett -
Nick Hilliard -
Nikolas Geyer -
Randy Bush -
Saku Ytti -
Seth Mattinen -
Tim Durack