For large BGP customers who service many BGP downstreams, the bottom line is that BCP 38 cannot be reasonably implemented. It's one of the weaknesses in the system.

Yes, from personal experience BCP 38 should never be implemented buy a transit provider as it will inevitably cause breakage on multi-homed downstream customers for little to no gain and a lot of customer anger. It should be implemented at the customer edge AS, so for a wholesale transit provider is more of a customer education situation. By all means use prefix lists to prevent your customer networks being received anywhere but directly from your customers to prevent them using your capacity without paying for it however.

Hi Mike

This may not exist yet, but what about a uRPF-like feature that uses RPKI, IRR, etc. instead of current BGP feed?

There is rfc8704 that extends urpf But I do not know of any commercial available solutions Brian

Hi Joel, can you please point us to the IETF draft document that describes how a "combination of ASPA and RPKI can be used to help with DDoS prevention". I was not able to find it. Thanks! -Rich On 11/8/22, 8:05 AM, "NANOG on behalf of Joel Halpern" <nanog-bounces+rich.compton=charter.com@nanog.org on behalf of jmh@joelhalpern.com> wrote: CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. There is work a tthe IETF on an addon to RPKI called ASPA. There is a draft that describes how the combiantion of ASPA and RPKI can be used to help with DDOS prevention. There is also a working group at the IETF called SAVNET that is looking at what technological additions can be made to address the shortcomings in BCP 38. In fairness, there is distinct disagreement as to what those shortcomings are, and whether the ideas being presented can help. Input from more operators would be great. (For completeness, I am a co-chair of that working group.) Yours, Joel On 11/8/2022 9:39 AM, Brian Turnbow via NANOG wrote: > Hi Mike > > > >> This may not exist yet, but what about a uRPF-like feature that uses RPKI, IRR, etc. instead of current BGP feed? > > There is rfc8704 that extends urpf > But I do not know of any commercial available solutions > > > Brian E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

----- Original Message -----

From: "Joel Halpern" <jmh@joelhalpern.com> To: "Brian Turnbow" <b.turnbow@twt.it> Cc: nanog@nanog.org Sent: Tuesday, November 8, 2022 10:03:20 AM Subject: Re: BCP38 For BGP Customers

There is work a tthe IETF on an addon to RPKI called ASPA.  There is a draft that describes how the combiantion of ASPA and RPKI can be used to help with DDOS prevention.

There is also a working group at the IETF called SAVNET that is looking at what technological additions can be made to address the shortcomings in BCP 38.  In fairness, there is distinct disagreement as to what those shortcomings are, and whether the ideas being presented can help.  Input from more operators would be great.  (For completeness, I am a co-chair of that working group.)

Wait; people are actually trying to implement BCP38, still? :-} Cheers, -- jra

On 11/8/2022 9:39 AM, Brian Turnbow via NANOG wrote:

...

...

This may not exist yet, but what about a uRPF-like feature that uses RPKI, IRR, etc. instead of current BGP feed?

There is rfc8704 that extends urpf But I do not know of any commercial available solutions

-- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

On 11/8/22 6:28 AM, Douglas Fischer wrote:

I also have this concern about Spoofing coming from Downstreams.

+1

And after a lot of struggle I can say that using uRPF in strict mode per interface doing FIB lookup is not a good idea!

Maybe it's the lack of caffeine, but would someone please remind / enlighten me as to why uRPF is a bad idea on downstream interfaces? N.B. Maybe I'm thinking more a varient of uRPF wherein if I have a route to the source from the interface in question. Thus not uRPF-strict (active route) nor uRPF-loose (route on any interface).

I've spent a lot of time wrestling with this issue, and the measurement that matters most, which are support tickets, doesn't show good results.

Will you please share a high level of what the technical problems are?

The best option I've found so far? It is to use the same Prefix-List that you use to release the routes accepted in the BGP session in a Filter Policy applied in the interface with the Downstream. Another important point to note is that you MUST NOT drop everything else that doesn't match this Prefix-List.

Please clarify if there are routes that would return the packets that would be dropped back to your router & downstream client. I don't understand why you would want to allow packets that couldn't return the same path. As for asymmetrically routed packets, I would still expect a return path to exist, even if it's not utilized.

(Yes, I know it's hard to accept that...)

#headScratching

But put a bandwidth and PPS control on what doesn't match the prefix-list, and block what exceeds. And why do it this way? Among other reasons, it is very common that Exceeded TTL responses come with source IPs for private use, or IP blocks that are for public use but not announced to the DFZ.

I'll argue -- possibly from a place of ignorance -- that there should not be any packets on the Internet at large (default free zone) to or from IPs not part of the Internet (DFZ). I view the TTL Exceeded packets as errant in their non-globally-routed source address and as such should be dropped early / often. Both "be liberal in what you accept and conservative in what you send" (Jon Postel) and "Brown M&M" (Van Halen) come to mind, as does the newer industry phrase "fail-fast". -- Grant. . . . unix || die

" Reverse path filtering literally says don't accept a packet from somewhere that isn't currently the next hop for that packet's source address." FIB or RIB? I knew of uRPF as available over an interface, per the routing table, not best path available. Or is that implementation dependent? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "William Herrin" <bill@herrin.us> To: "Grant Taylor" <gtaylor@tnetconsulting.net> Cc: nanog@nanog.org Sent: Tuesday, November 8, 2022 2:01:49 PM Subject: Re: BCP38 For BGP Customers On Tue, Nov 8, 2022 at 8:40 AM Grant Taylor via NANOG <nanog@nanog.org> wrote:

Maybe it's the lack of caffeine, but would someone please remind / enlighten me as to why uRPF is a bad idea on downstream interfaces?

Hi Grant, Two words: asymmetric routing. If the downstream network is architected in such a way that there's more than one path in and out of their network then there's no way to guarantee that any particular router believes the forward path to that network goes to a particular next hop. It can currently map to any next hop that goes in the direction of one of the valid paths. That routing is completely independent of how next hops are selected in the other direction. Packets can travel in one path and out another. Reverse path filtering literally says don't accept a packet from somewhere that isn't currently the next hop for that packet's source address. When it's possible for the forward route to point a different direction than the one from which valid packets are received, that is the wrong thing to do. In fact, it breaks the network. Useful automated reverse path filtering can ONLY be used when there is exactly ONE valid path to which and from which packets can be received. This is where strict mode uRPF actually works.

N.B. Maybe I'm thinking more a varient of uRPF wherein if I have a route to the source from the interface in question. Thus not uRPF-strict (active route) nor uRPF-loose (route on any interface).

Even if a particular router happens to have RIB entries for all the valid paths to a host (a sketchy proposition at best), only one such entry will be stored in the FIB where uRPF looks to make its filtering decision. As for loose mode, it's basically useless in a BCP38 discussion. Loose mode only filters bogons. It doesn't prevent impersonation of any IP address currently routed in the system and doesn't do anything at all on a router with a default route. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/

We’ve seen Juniper EX9ks implement uRPF in such a way that if I have two (load-balanced) BGP connections to the EX9k, and uRPF is turned on facing me, I immediately experience ~50% outbound packet loss. Methinks the EX9ks apply uRPF a little too close to the hardware and ignore the RIB. -Adam Adam Thompson Consultant, Infrastructure Services [MERLIN] 100 - 135 Innovation Drive Winnipeg, MB R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) https://www.merlin.mb.ca<https://www.merlin.mb.ca/> [cid:image002.png@01D8FE60.27B812C0]Chat with me on Teams<https://teams.microsoft.com/l/chat/0/0?users=athompson@merlin.mb.ca> From: NANOG <nanog-bounces+athompson=merlin.mb.ca@nanog.org> On Behalf Of Mike Hammett Sent: November 8, 2022 2:29 PM To: William Herrin <bill@herrin.us> Cc: nanog@nanog.org; Grant Taylor <gtaylor@tnetconsulting.net> Subject: Re: BCP38 For BGP Customers "Reverse path filtering literally says don't accept a packet from somewhere that isn't currently the next hop for that packet's source address." FIB or RIB? I knew of uRPF as available over an interface, per the routing table, not best path available. Or is that implementation dependent? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ________________________________ From: "William Herrin" <bill@herrin.us<mailto:bill@herrin.us>> To: "Grant Taylor" <gtaylor@tnetconsulting.net<mailto:gtaylor@tnetconsulting.net>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> Sent: Tuesday, November 8, 2022 2:01:49 PM Subject: Re: BCP38 For BGP Customers On Tue, Nov 8, 2022 at 8:40 AM Grant Taylor via NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> wrote:

Maybe it's the lack of caffeine, but would someone please remind / enlighten me as to why uRPF is a bad idea on downstream interfaces?

Hi Grant, Two words: asymmetric routing. If the downstream network is architected in such a way that there's more than one path in and out of their network then there's no way to guarantee that any particular router believes the forward path to that network goes to a particular next hop. It can currently map to any next hop that goes in the direction of one of the valid paths. That routing is completely independent of how next hops are selected in the other direction. Packets can travel in one path and out another. Reverse path filtering literally says don't accept a packet from somewhere that isn't currently the next hop for that packet's source address. When it's possible for the forward route to point a different direction than the one from which valid packets are received, that is the wrong thing to do. In fact, it breaks the network. Useful automated reverse path filtering can ONLY be used when there is exactly ONE valid path to which and from which packets can be received. This is where strict mode uRPF actually works.

N.B. Maybe I'm thinking more a varient of uRPF wherein if I have a route to the source from the interface in question. Thus not uRPF-strict (active route) nor uRPF-loose (route on any interface).

Even if a particular router happens to have RIB entries for all the valid paths to a host (a sketchy proposition at best), only one such entry will be stored in the FIB where uRPF looks to make its filtering decision. As for loose mode, it's basically useless in a BCP38 discussion. Loose mode only filters bogons. It doesn't prevent impersonation of any IP address currently routed in the system and doesn't do anything at all on a router with a default route. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/

On Tue, Nov 8, 2022 at 9:08 PM Grant Taylor via NANOG <nanog@nanog.org> wrote:

This thread has made me wonder if there isn't a need for a 3rd type of uRPF or comparable filtering wherein the incoming interface is a viable route in the RIB even if it's not the best route in the FIB.

Hi Grant, Two problems here: 1. uRPF reuses the existing FIB. Either the next hop matches the source (strict) or some entry in the FIB matches the source (loose). The hardware "fast path" to implement the per-packet FIB is already there; all it takes is a second lookup. Unless someone comes up with a really clever idea, that's basically all the filtering you can squeeze out of the FIB. To do something more, you'd have to implement an additional "fast path" in the router that can act on every packet. That's kinda expensive. 2. BGP is a distance-vector protocol, not a link-state protocol. The significance for filtering is that the BGP RIB on a given router does not describe the complete state of the network. It only knows about itself and its immediate neighbors. That's not algorithmically guaranteed to be enough knowledge about the network to determine that a packet with a given source address can't reasonably come from a particular interface. Consider the following scenario: A - B - C - D - E - A | F B receives a packet from A to F via C. Is the packet legitimate? B can't know because the only route to A in B's RIB is the direct connection to A. C didn't pass E's route to A back to B because it was a longer, unpreferred path. C might not even have received the route from D. So even if you go to the expense of building a fast path that can consider what's in the RIB instead of just the FIB, you don't have the information you need in the RIB either. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/

On Thu, Nov 10, 2022 at 10:08 AM Grant Taylor via NANOG <nanog@nanog.org> wrote:

I wonder if Feasible Path uRPF or Enhanced Feasible Path uRPF might help the situation. However I suspect they both suffer from the FIB != RIB problem and associated signaling.

Hi Grant, That's a fairly good way to think about it. BGP knows -a- path and sometimes it knows more than one but it simply doesn't have signal on the totality of feasible paths for a particular IP address. No distance-vector protocol can. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/

On Tue, Nov 8, 2022 at 8:44 AM Grant Taylor via NANOG <nanog@nanog.org> wrote:

[...]

I don't understand why you would want to allow packets that couldn't return the same path.

As for asymmetrically routed packets, I would still expect a return path to exist, even if it's not utilized.

Grant, You're thinking about it from the upstream perspective, where a route could be accepted but depreferenced and thus not actively used. Think about it from the downstream network's perspective, though. If you're my upstream, and I don't want to use your link for inbound traffic, but I'd like to be able to send out some traffic over the link, how can I advertise the prefix to you in a way that would both ensure that you have it in your table locally, so that uRPF is happy, but also to ensure no packets actually make *use* of that routing table entry? Sure, you could tag the routes with 'no-export', but that only prevents the prefix from propagating outward, it doesn't prevent traffic on that router from using the routing table entry. You can try adjusting your MED, and hope the upstream doesn't squash the MED back to a default value it applies to all its customers. For the most part, you're up against a wall. You don't know how your upstream's route selection process is stacked with respect to routes you advertise, so you have no certainty that if you announce a prefix to them, it won't potentially be used to carry all your inbound traffic. The only way to be sure that you won't take inbound traffic on a link is to not advertise the prefix at all across that link. Why might this be necessary, you ask? Imagine you've got links of different sizes on your network. You have a 1G link to provider A You have a 1G link to provider B You have a 10G link to cheap provider C You have a 10G link into a peering exchange Somewhere beyond provider A, someone decides they don't like one of your customers, and sends a 5Gbps DDoS flow at you. If you continue advertising that prefix to provider A, everyone going through provider A will suffer, including all their customers. You have plenty of capacity to take the inbound flow through the exchange point and through cheap provider C. So, you stop advertising the prefix under attack to provider A and provider B, to ensure the traffic doesn't saturate your 1G links. Inbound traffic happily shifts to the exchange point port and provider C's port. Life is good. Oh no! Provider A has implemented uRPF, and now all their customers are unhappy because they can't reach your websites on that prefix, because the *return* traffic is still flowing out the 1G link directly to provider A. Trying to implement a source-based routing filter to redirect all traffic *coming* from the prefix under attack destined to ISP A to instead go through ISP C is a pain in the butt. So, you grudgingly stop accepting routes from ISP A, as that's the only way to make the uRPF pain stop. Now, none of your traffic is flowing out the 1G link to ISP A; their customers are happy again, because they can reach websites on your prefix that is under attack (via ISP C, or the exchange point). At the end of the month, you look at your contracts, and realize that you had to spend a chunk of your limited engineering resources working around the upstream uRPF filter, and ultimately ended up not sending traffic across a link you were paying for. When renewal time comes, you decide it's not worth the headache to pay for a link to ISP A that you can't reliably use, and that requires manual intervention to work around whenever creative routing solutions are necessary. You don't renew your contract with ISP A. As Mr. Bush might say, "I recommend all my competitors implement this in their networks." Thanks! Matt

On Tue, Nov 8, 2022 at 5:28 AM Douglas Fischer <fischerdouglas@gmail.com> wrote:

Another important point to note is that you MUST NOT drop everything else that doesn't match this Prefix-List. But put a bandwidth and PPS control on what doesn't match the prefix-list, and block what exceeds. Among other reasons, it is very common that Exceeded TTL responses come with source IPs for private use, or IP blocks that are for public use but not announced to the DFZ.

Hi Douglas, TTL exceeded messages are not important. They're useful to diagnostics but nothing breaks if they're lost. There's no need to broadly allow misconfigured customers to originate packets from RFC1918 space nor to allow them to originate ICMP type 11 (time exceeded) messages from RFC1918 space. You should not do so. The packets you're worried about here are fragmentation needed: ICMP type 3 code 4. Fragmentation needed messages are used for Path MTU discovery. When PMTUD breaks, TCP fails. Path MTU discovery is the one place in the core TCP/IP protocol stack where the end-to-end principle was abandoned. TCP requires the ability to receive this notification from any midpoint node in order to shrink packets too large for that midpoint node's next hop. It's the most broken part of the TCP/IP design. Unfortunately, your solution of allowing ICMP type 3 messages from RFC1918 space is dysfunctional. Even if you accept the packets (and tailor your acceptance so that it only applies to ICMP type 3 messages with a source address in RFC1918 space) the packets are highly likely to be discarded elsewhere in the path. In the past, I've suggested that vendors implement a feature allowing destination unreachables generated from a privately addressed interface to be originated from a different, user-configured public IP address. So far I haven't seen any takers. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/