On 30/Jul/20 12:00, Simon Leinen wrote:
As Nick mentions, the hostnames are from the BGP hostname extension.
I should have noticed that, but we use "BGP unnumbered"[1][2], which uses RAs to discover the peer's IPv6 link-local address, and then builds an IPv6 BGP session (that uses RFC 5549 to transfer IPv4 NLRIs as well).
Here are some excerpts of the configuration on such a leaf router.
General BGP boilerplate:
------------------------------ router bgp 65111 bgp router-id 10.1.1.46 bgp bestpath as-path multipath-relax bgp bestpath compare-routerid ! address-family ipv4 unicast network 10.1.1.46/32 redistribute connected redistribute static exit-address-family ! address-family ipv6 unicast network 2001:db8:1234:101::46/128 redistribute connected redistribute static exit-address-family ------------------------------
Leaf switch <-> server connection: (we use a 802.1q tagged subinterface for the BGP peering and L3 server traffic; the untagged interface is used only for netbooting the servers when (re)installing the OS. Here, servers just get IPv4+IPv6 default routes, and each server will only announce a single IPv4+IPv6 (loopback) address, i.e. the leaf/server links are also "unnumbered". Very simple redundant setup without any LACP/MLAG protocols... it's all just BGP+IPv6 ND. You can basically connect any server to any switch port and things will "just work" without special inter-switch links etc.)
------------------------------ interface swp1s0 description s0001.s1.scloud.switch.ch p8p1 ! interface swp1s0.3 description s0001.s1.scloud.switch.ch p8p1 ipv6 nd ra-interval 3 no ipv6 nd suppress-ra ! [...] router bgp 65111 neighbor servers peer-group neighbor servers remote-as external neighbor servers capability extended-nexthop neighbor swp1s0.3 interface peer-group servers ! address-family ipv4 unicast neighbor servers default-originate neighbor servers soft-reconfiguration inbound neighbor servers prefix-list DEFAULTV4-PERMIT out exit-address-family ! address-family ipv6 unicast neighbor servers activate neighbor servers default-originate neighbor servers soft-reconfiguration inbound neighbor servers prefix-list DEFAULTV6-PERMIT out exit-address-family ! ip prefix-list DEFAULT-PERMIT permit 0.0.0.0/0 ! ipv6 prefix-list DEFAULTV6-PERMIT permit ::/0 ------------------------------
Leaf <-> spine:
------------------------------ interface swp16 description sw-o port 22 ipv6 nd ra-interval 3 no ipv6 nd suppress-ra ! [...] router bgp 65111 neighbor fabric peer-group neighbor fabric remote-as external neighbor fabric capability extended-nexthop neighbor swp16 interface peer-group fabric ! address-family ipv4 unicast neighbor fabric soft-reconfiguration inbound ! address-family ipv6 unicast neighbor fabric activate neighbor fabric soft-reconfiguration inbound ------------------------------
Note the "remote-as external" - this will accept any AS other than the router's own AS. AS numbering in this DC setup is a bit weird if you're used to BGP... each leaf switch has its own AS, all spine switches should have the same AS number (for reasons...), and all servers have the same AS because who cares. (We are talking about three disjoint sets of AS numbers for leaves/spines/servers though.)
Interesting. Data centre bits are, interesting :-). Thanks for sharing. Mark.