Dear fellow networkers, Through this tutorial-styled email I'd like to introduce the concept, usage and implementation of "selective blackholing" through the BGP protocol to the community. This email contains some python code, example router configurations references to RIPE Atlas data to demonstrate effectiveness. Selective Blackholing is a DDoS damage control mechanism which is (compared to scrubbing or conventional blackholing) cheaper and more effective from the perspective of both the Service Provider and the End-User. In today's internet DDoS attacks are common-place, and the balance is not particularly fair: very expensive on the receiver-side and very cheap for the evil-doer (think DNS-, NTP- and SNMP-amplification attacks). I propose a method which is effective for businesses with a scoped radius of interaction with their customers. To elaborate on what 'scoped' (read: selective) means: a Dutch herring-shop owner will see most legitimate visitors come from within the Netherlands; a Dallas, TX based Automotive Bulletin Board will see most conversions from visitor to part-buyer come from visitors within the United States. If we look at today's DDoS mitigation solutions (scrubbers: expensive hardware or subscription services, or BGP blackhole: all or nothing reachability)... it seems intuitive that a lot of business-owners would care more about traffic from visitors in the surrounding 1000 kilometers (or from the same country), than say (for the duration of the attack) traffic sourced from other continents. In this document we sequentially number any line of router config or python code, to allow for easy references. Let's focus on the implementation following four features: * discard traffic sourced outside 'this' country (5580:664) * discard traffic sourced outside 'this' continent (5580:660) * discard traffic sourced outside a 1000 km radius from 'here' (5580:663) * discard traffic sourced outside a 2500 km radius from 'here' (5580:662) The 'this' and 'here' designators refer to the point of interconnection with any given customer: 'this' and 'here' for a customer connecting to AS5580 in Amsterdam, would respectable be "Netherlands, Europe, 'most of West Europe', 'most of West + Central Europe'". It is important to understand that all proposed mechanisms are designed to perform uniformly in terms of intended usage regardless of location of interconnection, so for a Dutch customer 5580:664 means 'discard any traffic for my prefix received on PEs outside the Netherlands' but for an Dallas, TX based customer BGP community '5580:663' would mean 'discard traffic destined to my prefix on routers located outside a 1000 km radius around Dallas'. The 1000 km and 2500 km distances referenced throughout this document mean the distance between AS 5580 network devices as calculated with a haversine formula with the GPS coordinates of devices as input. The haversine formula is an equation giving great-circle distances between two points on a sphere from their longitudes and latitudes. Actual length of datapath or optical paths is not taken into consideration, nor is a peering partner's home country a factor. To create the above described features we need a simplistic CMDB database, some python code, static inbound iBGP route-map and semi-dynamic customer facing inbound route-map. For illustration purposes we assume a fictional ISP with a POP in the following cities: Tokyo, San Jose, Dallas, New York, London, Amsterdam and Stockholm. +-----------+-----------+----------+-------+---------------------+ | router | continent | country | metro | latitude, longitude | | name | id | ISO31661 | id | | +-----------+-----------+----------+-------+---------------------+ | r1.tky.jp | 3 | 392 | 46 | 35.65671, 139.80342 | | r1.sjo.us | 1 | 840 | 29 | 37.44569,-122.16111 | | r1.dal.us | 1 | 840 | 33 | 32.80096, -96.81962 | | r1.nyc.us | 1 | 840 | 26 | 40.71780, -74.00885 | | r1.lon.uk | 2 | 276 | 23 | 51.51173, -0.00197 | | r1.ams.nl | 2 | 528 | 20 | 52.35600, 4.95068 | | r1.sto.se | 2 | 752 | 22 | 59.36264, 17.95560 | +-----------+-----------+----------+-------+---------------------+ Table 1. Simplistic CMDB Based on the above CMDB table we can construct the following device specific configuration snippets. Within these snippets we adhere to the concept that edge devices are smart and core devices dumb. At the edge of the network we accept BGP Communities from customers, which we translate through a process of testing & branching from 'something the customer understands' to 'something the network understands'. r1.tky.jp: 001. ip community-list THIS:METRO seq 5 permit 65123:30046 002. ip community-list THIS:COUNTRY seq 5 permit 65123:392 003. ip community-list THIS:CONTINENT seq 5 permit 65123:3000 r1.sjo.us: 004. ip community-list THIS:METRO seq 5 permit 65123:10029 005. ip community-list THIS:COUNTRY seq 5 permit 65123:840 006. ip community-list THIS:CONTINENT seq 5 permit 65123:1000 r1.dal.us: 007. ip community-list THIS:METRO seq 5 permit 65123:10033 008. ip community-list THIS:COUNTRY seq 5 permit 65123:840 009. ip community-list THIS:CONTINENT seq 5 permit 65123:1000 r1.nyc.us: 010. ip community-list THIS:METRO seq 5 permit 65123:10026 011. ip community-list THIS:COUNTRY seq 5 permit 65123:840 012. ip community-list THIS:CONTINENT seq 5 permit 65123:1000 r1.lon.uk: 013. ip community-list THIS:METRO seq 5 permit 65123:20023 014. ip community-list THIS:COUNTRY seq 5 permit 65123:276 015. ip community-list THIS:CONTINENT seq 5 permit 65123:2000 r1.ams.nl: 016. ip community-list THIS:METRO seq 5 permit 65123:20020 017. ip community-list THIS:COUNTRY seq 5 permit 65123:528 018. ip community-list THIS:CONTINENT seq 5 permit 65123:2000 r1.sto.us: 019. ip community-list THIS:METRO seq 5 permit 65123:20022 020. ip community-list THIS:COUNTRY seq 5 permit 65123:725 021. ip community-list THIS:CONTINENT seq 5 permit 65123:2000 all devices (common references for more humane route-map definitions): 022. ! communities which are exposed in public documentation for customers 023. ip community-list OUTSIDE:THIS:CONTINENT:DISCARD seq 5 permit 5580:660 024. ip community-list OUTSIDE:2500KM:RADIUS:DISCARD seq 5 permit 5580:662 025. ip community-list OUTSIDE:1000KM:RADIUS:DISCARD seq 5 permit 5580:663 026. ip community-list OUTSIDE:THIS:COUNTRY:DISCARD seq 5 permit 5580:664 027. ip community-list GLOBAL:BLACKHOLE seq 5 permit 5580:666 028. ip community-list SCOPED:ACTION seq 5 permit 5580:660 029. ip community-list SCOPED:ACTION seq 5 permit 5580:662 030. ip community-list SCOPED:ACTION seq 5 permit 5580:663 031. ip community-list SCOPED:ACTION seq 5 permit 5580:664 032. ip route 10.0.0.1/32 null0 all devices (static inbound iBGP route-map): 033. route-map RX:iBGP permit 100 034. match community GLOBAL:BLACKHOLE 035. continue 1100 036. route-map RX:iBGP permit 100 037. match community OUTSIDE:THIS:COUNTRY:DISCARD OUTSIDE:THIS:CONTINENT:DISCARD 038. continue 1100 039. route-map RX:iBGP permit 110 040. match community OUTSIDE:1000KM:RADIUS:DISCARD OUTSIDE:2500KM:RADIUS:DISCARD 041. continue 1100 042. route-map RX:iBGP permit 1000 043 route-map RX:iBGP permit 1100 044. match community THIS:METRO THIS:COUNTRY THIS:CONTINENT 045. route-map RX:iBGP permit 1101 046. set community no-export additive 047. set ip next-hop 10.0.0.1 048. set local-preference 1000000 The above route-map allows BGP speakers in our routing domain to apply different actions depending on the (geographically significant) communities. Let's use three example input and run them through the route-map to illustrate what happens: evaluation takes place on: router r1.lon.uk input: communities: none result: prefix accepted without modifications why: The first match is 'permit 1000' at line 042, because there is no match statement all prefixes will be accepted and the route-map is not evaulated any further. evaluation takes place on: router r1.lon.uk input: communities: "5580:662 65123:10033 65123:10029" (for example when a customer attached to r1.sjo.us sets "5580:662 - discard outside 2500km") result: traffic for the prefix will be discarded why: The prefix matches with the clause 110 (line 039) because 5580:662 matches OUTSIDE:2500KM:RADIUS:DISCARD (line 040). Because a match is found on line 041 we jump to line 043 for further matching. Because the community string cannot match any of the communities on line 044 (in the case of r1.lon.uk 65123:20023 OR 65123:276 OR 65123:2000), the prefix is further evaluated at line 045. There is no match statement in the 'permit 1101' clause so for any prefix that arrives at this particular clause in the route-map the next-hop will be 10.0.0.1 (discard). Because the prefix matches with clause 1101 the route-map will not be evaluated any further. evaluation takes place on: router r1.dal.us input: communities: "5580:662 65123:10033 65123:10029" (for example when a customer attached to r1.sjo.us sets "5580:662 - discard outside 2500km") result: prefix will be accepted with no further modifications why: The prefix matches with the clause 110 (line 039) because 5580:662 matches OUTSIDE:2500KM:RADIUS:DISCARD (line 040). Because a match is found, the continue statement (line 041) is followed and we jump to line 043 for further matching. Because the community string matches one of the communities on line 044 (in the case of r1.dal.us 65123:10033 OR 65123:840 OR 65123:1000), further evaluation of the route-map is halted, and the clause which actually makes it a blackhole is never reached. The attentive reader will now probably think "How come you are inputting "5580:662 65123:10033 65123:10029", and not just 5580:662?! You promised simple end-user configuration, no customer will want to invest time to figure out magic values such as 65123:10033 65123:10029! Now we get down the the secret sauce: extensive community rewriting.... Based on our CMDB database, we can calculate sets of magic values that will cause the network to behave according to the desired feature. I've made a sample python program available that will do these calculations. This is the output: 049. Eleanor:~ job$ wget -q http://noc.as5580.net/~job/example_community_calculator.py 050. Eleanor:~ job$ python example_community_calculator.py 051. r1.lon.uk - rewrite targets: 052. 1000 km: 65123:20020 65123:276 053. 2500 km: 65123:2000 054. 055. r1.dal.us - rewrite targets: 056. 1000 km: 65123:10033 057. 2500 km: 65123:840 058. 059. r1.sjo.us - rewrite targets: 060. 1000 km: 65123:10029 061. 2500 km: 65123:10033 65123:10029 062. 063. r1.nyc.us - rewrite targets: 064. 1000 km: 65123:10026 065. 2500 km: 65123:10026 65123:10033 066. 067. r1.tky.jp - rewrite targets: 068. 1000 km: 65123:30046 069. 2500 km: 65123:30046 070. 071. r1.sto.se - rewrite targets: 072. 1000 km: 65123:20022 073. 2500 km: 65123:2000 074. 075. r1.ams.nl - rewrite targets: 076. 1000 km: 65123:20020 65123:276 077. 2500 km: 65123:2000 078. Eleanor:~ job$ Now we can use the above to create the route-map for a customer connected to r1.sjo.us (San Jose, US): 079. ip prefix-list 15562:PREFIX seq 5 permit 194.33.96.0/24 080. ip prefix-list 15562:PREFIX:BLACKHOLE seq 5 permit 194.33.96.0/24 le 32 081. route-map IMPORT:FROM:15562:CUSTOMER:IPv4 permit 200 082. match ip address prefix-list 15562:PREFIX:BLACKHOLE 083. match community GLOBAL:BLACKHOLE 084. set community 5580:26220 no-export additive 085. set ip next-hop 10.0.0.1 086. set local-preference 1000000 087. route-map IMPORT:15562:CUSTOMER:IPv4 permit 300 088. match ip address prefix-list 15562:PREFIX:BLACKHOLE 089. match community SCOPED:ACTION 090. set community 5580:26220 no-export additive 091. set local-preference 650 092. continue 600 093. route-map IMPORT:15562:CUSTOMER:IPv4 permit 400 094. match ip address prefix-list 15562:PREFIX 095. set community 5580:26220 additive 096. set local-preference 650 097. route-map IMPORT:15562:CUSTOMER:IPv4 deny 500 098. route-map IMPORT:15562:CUSTOMER:IPv4 permit 600 099. match community OUTSIDE:1000KM:RADIUS:DISCARD 100. set community 65123:10029 additive 101. continue 1200 102. route-map IMPORT:15562:CUSTOMER:IPv4 permit 700 103. match community OUTSIDE:2500KM:RADIUS:DISCARD 104. set community 65123:10033 65123:10029 additive 105. continue 1200 106. route-map IMPORT:15562:CUSTOMER:IPv4 permit 900 107. match community OUTSIDE:THIS:COUNTRY:DISCARD 108. set community 65123:840 additive 109. continue 1200 110. route-map IMPORT:15562:CUSTOMER:IPv4 permit 1100 111. match community OUTSIDE:THIS:CONTINENT:DISCARD 112. set community 65123:1000 additive 113. route-map IMPORT:15562:CUSTOMER:IPv4 permit 1200 Various rewrites need to happen to offer the four scoped features we promised at the beginning of the document, the rewrites come either directly from the CMDB (line 108 and 112) or from the python program (line 100 and 104). Any network that wants to implement this should run software such as the python program every time a new POP is added, and update all IMPORT route-maps on all devices accordingly. Lines 081 to 113 bring us back to the second two examples where I illustrated what happens with various community strings if they are parsed through the RX:iBGP route-map. The customer in San Jose just sets 5580:662 (matched at line 102), but the IMPORT route-map will rewrite that to "5580:662 65123:10033 65123:10029" (line 104). Et voila, we have accomplished selective blackholing based on distance. :-) I ran some measurements with the RIPE Atlas system to test which probes can reach a given prefixes when a selective blackhole community is attached, the results are available here: http://noc.as5580.net/~job/selective_blackholing/ Disclaimer: Seeming route-map inefficiencies might be more related to the particular vendor on which I implemented this rather than my logic capabilities. If this document contains any errors, they most likely originated in the process of converting real world configuration to something of educational value. :-) I'd like to thank Saku Ytti Torsten Blum and Peter van Dijk for their gracious inspiration. If not for free and unhindered dialogue between organisations, we would never be able to create mechanisms like these. If you have any questions, please approach me through email or in person at NANOG60 in Atlanta! Kind regards, Job