(still here, just been really busy at work today; will try to finish sending the notes out tonight. --Matt) 2006.06.06 MPLS TE tutorial Pete Templin, Nextlink [slides are at: http://www.nanog.org/mtg-0606/pdf/pete-templin.pdf http://www.nanog.org/mtg-0606/pdf/pete-templin-exercise.pdf He works in a Cisco shop, no JunOs experience Operator perspective, no logos Traffic engineering before MPLS --the "fish" problem. two parallel paths, one entry router, one exit router, you end up with all traffic taking one path, not using the other path. IGP metric adjustments can lead to routing loops hard to split traffic No redundancy left over if both paths filled, but can be good for using 2 out of 3 paths. MPLS TE fundamentals Packets are forwarded based on FIB or LFIB FIB/LFIBS built based on RIB TE tunnels; TE tunnel interface is a unidirectional logical link from one router to another. Once the tunnel is configured, a label is assigned for the tunnel that corresponds to the path through the MPLS network (LSP) TE tunnel basics Once traffic is routed onto the tunnel, the traffic flows through the tunnel based on the path. Return traffic could be placed onto a tunnel going the opposite direction, or simply routed by IGP Key terms for TE Headend router on which the tunnel is configured Tail destination address of tunnel Midpoint router(s) along the path along the tunnel LSP Basic TE config Global: mpls traffic-eng tunnels IGP: must be OSPF or IS-IS mpls traffic-eng rouer-id Loopback0 mpls traffic-eng <area X | level2> physical interfaces mpls ip mpls traffic-eng tunnels tells IGP to share TE info with other TE nodes interface TunnelX ip unnumbered loopback0 borrow the loopbak's address so we can forward traffic down the tunnel tunnel mode mpls traffic-eng tunnel destination <a.b.c.d> tunnel tail tunnel mpls traffic-eng path-option 10 dynamic find a dynamic path through network best path with sufficient bandwidth will discuss path selection in a bit Where are we at? Tunnels go from headend to tail end through midpoint routers over a deterministic path we know what commands go on a router for the global physical interface tunnel interface commands TE and bandwidth Physical interfaces can be told how much bandwidth can be reserved (used) ip rsvp bandwidth X X TE tunenls can be configured with how much bandwidth they need: tun mpls traff bandw Y Tunnels will reserve Y bw on outbound interfaces, and find a path across the network wth X(unused)>Y BW. This prevents oversubscription, or at least helps control it. You can allow for burst room, but for now we'll stick with static, non-oversubscribed links. TE BW operators can adjust the tunnel bandwidth values over time to match changes in traffic. If tunnels are dynamically placed, the tunnels will dynamically find a path through the network with sufficient bandwidth, or will go down. TE auto-bandwidth magic Tunnels can be configured to watch their actual traffic as in "shw int <blah>| inc rate" every five minutes, and update their reservation to match, at periodic intervals. Dynamic reservations to match the live network Bandwidth is 'reserved' using RSVP but not "saved" for TE Often buys enough time to identify the surge, see where the traffic is coming/going. The number is only a number in control plane; no actual impact on data plane, no shaping, no control on real data flows. tunnel mpls traffic-eng auto-bw frequency Y each auto-bw tunnel does "sh int" to capture its rate every 300* seconds each auto-bw tunnel updates "tunn mpls traff bandwidth X" every Y seconds The config actually changes; this will impact your RANCID tracking. It uses highest measured rate during the interval Y May want to tweak your load-interval, since it's a decaying function over time; 5 minute is a fairly smooth value. May need to tweak config check-in system to avoid getting flooded with bandwidth adjustments. Covered: TE tunnel basics router config basics general concepts about TE and bandwidth In this case, the shortest path that has X BW available for reservation actually, bw X at or below priority Y, but that's later. SPF calculations step 0: create a PATH list and a TENT list step 1: put "self" on PATH list. step 2: step 3: put PATH nodes' neighbors on TENT list step 4: if TENT list is empty, stop. step 5: jump back to step 2: Example exercise -- calculate router A's best path to router D using the handout. CSPF notes No load sharing is performed within a tunnel; as soon as a path is found, it wins CSPF tiebreakers: lowest IGP cost largest minimum available bandwidth lowest hop count top node on the PATH list Creating paths -- can be created dynamically, or statically via static paths. Dynamic: tunnel mpls traff path-option X dynamic Explicit paths paths can be crated manually by explicitly creating a path "ip explicit-path name <name?>" next-address X next-address Y tunnel mpls traff path-option X explicit name blah Paths can be created manually by explicity configuring a path that excludes an address; use any link EXCEPT this one. for backup links, oob links, etc. can't combine exclude-address and next-address on the same explicit path. Q: if all other paths go away, will an excluded path still be used? A: only if you have a fallback option of "dynamic" can't both include and exclude on same path. A TE tunnel can have multiple path options. lowest cost path option is attempted higher-cost paths attempted sequentially until a path can successfully established, or failure Usually best to have a dynamic option as the highest-cost option, to ensure you don't fall back to IGP (and lose traffic matrix accounting!) CSPF checks tunnel path options in sequence for one that has sufficient information. OK, we've got tunnels now--how do we share info around the network? TE info is shared around using IGP available bw per interface eight priority levels high priority tunnels can push low priority tunnels out of the way some dynamics as far as tunnel vs interface sizing administrative weight (TE specific IGP metric Affinity (customizable) You can configure different interfaces with bits that indicate territorial restrictions, or high-latency links, etc. Use the interface affinity bits, and match/exclude tunnel affinity bits to include or exclude certain links. This information is distributed immediately for "significant" changes periodically for "insignificant" changes immediately, if a change causes an error Significant changes occur when available bandwidth on an interface passes preset thresholds customizable with 16 thresholds. (based on percentages) Key points where information is passed from device to device. Insignificant changes flooded every 3 minutes, significant flooded immediately, by default. If a new path (dynamic or static) appears that's better than the current one re-optimize periodically, every hour from tunnel bringup by default manually "mpls traffic-eng reoptimize" event-driven when a link comes up optional: requires "mpls traff reo events link-up" not so good with flapping links, though. If you have a flapping link, TE tunnels will stay off that link for about an hour; you have flap dampening in your network. Up next, how to put IP traffic on tunnels static routes "ip route x.x.x.x y.y.y.y tuZ" policy-based routing route-map PBR permit 20 match ip addr ACL set ip next-hop tunX Autoroute Autoroute treat this tunnel as though it's a logically directly connected link to the tunnel tail updates the local router's RIB/FIB with "tunnelX" in place of the IGP next-hop; preserves the IGP cost all the way to the tail. One line of config per tunnel; it updates the LOCAL router's RIB/FIB; prior routers not made aware of this router as a next-hop for the tunnel tail. This is supported for IS-IS, but can be more difficult; you increase number of links in your IGP pretty quickly. tunn mpls traff autoroute announce autoroute and load-sharing parallel tunnels will load-share inversely proportional to their configured bandwidth auto-bandwith can really muck with these values! load-sharing can be tuned separately with "tunn mpls traff load-share X" Limited to CEF 16 buckets, depending on when it measures, can end with values drifting apart. If you use same "X" on two tunnels, they will load share equally. IGPs can load-share over equal-cost paths BUT TE tunnel cannot load-share over multiple physical interfaces TE diagnostics show ip route X sh run int tuX sh ip rsvp reservation show mpls traff tun suboptimal constr none shows headend tunnels taking suboptimal paths to the tunnel tail (eg, different from IGP best path) show mpls traff tun detailed info for all tunnel headends bandwidth info (auto-bw) MPLS labels, hop by hop path show mpls traff tun role <head|middle|remote|tail> remote is non-headend (not originating or ending on this router) show ip rsvp interfaces shows max allocated bw on an interface. MPLS VPNs if a tunnel tail is not the egress PE, add "mpls ip" to the tunnel configuration PE-P-P-PE-PE --adds another label onto the stack. If the last router VPN isn't on penultimate router, the TE label won't be recognized, it'll be dropped. This adds the TE label back on, keeps the LDP label as well as the VPN label still intact Add "mpls ldp discovery directed-hello accept" to config If you have unidirectional tunnels, that way when you're bringing up tunnels LDP info can be exchanged as you're going. Multicast: RPF calculations are normally based on unicast RIB Unidirectional TE tunnels cause RPF failures add "mpls traffic-engineering multicast-intact" to IGP config Bases RPF checks on RIB BEFORE TE tunnels are substituted Questions? templin at templin.org Swap to looking at operational network for some troubleshooting views. About 150-200Mb traffic aggregate, 4 pops, DAL and Houston, four parallel DS3s, 2 between core1's and 2 between core2's 30mbit IP RSVP reservation; TE kicks in, and moves traffic. Houston intercore links tends to not have traffic unless TE has kicked in. Aggressive 15 minute auto recalculations, since surprises can kick up fairly quickly. Room heads for cookies at 1526 hours Pacific Time