The differential delay is most likely caused by the T1's in the MFR bundle riding physically diverse paths across the TDM network. The carrier would need to validate their CLR/DLR to see what paths/DS3's the individual T1's follow to verify they are on the same circuit. Unfortunately there are those that try and sell MFR as "redundancy" and have the T1's ride diverse paths that can sometimes be pretty huge in difference of loop distance etc.., when they should really just be selling MFR for the bandwidth. - Jim P. -----Original Message----- From: R. Benjamin Kessler [mailto:rbk@mnsginc.com] Sent: Thursday, February 18, 2010 10:55 PM To: nanog@nanog.org Subject: MLFR Differential Delay Problems Hello NANOGers - I'm working on a project to migrate a customer from one "Tier 1" provider to another at 50+ locations (all domestic US sites). Most of these connections are 4xT1 multi-link bundles. The old router configuration was MLPPP which was rock-solid for 3 years (save for the typical "last-mile" circuit issues, fiber-cuts, etc.). The new carrier uses FRF.16 multi-link Frame Relay vs. MLPPP. We've completed the migration on 10+ sites and all of them are now reporting errors like the following: Feb 17 21:01:39 /kernel: MFR bundle ls-0/0/0:0 link t1-1/0/0 differential 91.7 ms over yellow differential delay 75 ms Feb 17 21:01:50 /kernel: MFR bundle ls-0/0/0:0 link t1-1/0/0 differential 115.9 ms over yellow differential delay 75 ms Feb 17 21:01:50 /kernel: MFR bundle ls-0/0/0:0 link t1-1/0/1 differential 79.0 ms over yellow differential delay 75 ms Feb 17 21:01:50 /kernel: MFR bundle ls-0/0/0:0 link t1-2/0/1 differential 79.1 ms over yellow differential delay 75 ms Feb 17 21:01:50 /kernel: MFR bundle ls-0/0/0:0 link t1-1/0/1 differential 97.4 ms over yellow differential delay 75 ms Feb 17 21:01:50 /kernel: MFR bundle ls-0/0/0:0 link t1-2/0/0 differential 97.5 ms over yellow differential delay 75 ms Feb 17 21:01:50 /kernel: MFR bundle ls-0/0/0:0 link t1-2/0/1 differential 97.5 ms over yellow differential delay 75 ms Feb 17 21:01:52 /kernel: MFR bundle ls-0/0/0:0 link t1-1/0/1 differential 97.4 ms over yellow differential delay 75 ms Feb 17 21:01:52 /kernel: MFR bundle ls-0/0/0:0 link t1-2/0/0 differential 97.5 ms over yellow differential delay 75 ms Feb 17 21:01:52 /kernel: MFR bundle ls-0/0/0:0 link t1-2/0/1 differential 97.5 ms over yellow differential delay 75 ms Feb 17 21:01:53 /kernel: MFR bundle ls-0/0/0:0 link t1-1/0/1 differential 90.0 ms over yellow differential delay 75 ms Feb 17 21:01:53 /kernel: MFR bundle ls-0/0/0:0 link t1-2/0/1 differential 100.0 ms over yellow differential delay 75 ms The customer routers are all Juniper J6350; I believe the Carrier's routers are all Cisco GSRs. Advanced JTAC says that our configurations are solid and that there are no known bugs that would exhibit behavior like this. The carrier is insisting on performing physical-level tests of the circuits (even though they're running error free) before they'll engage higher-level engineers so I'm currently in a holding pattern awaiting those results. My Google-foo is failing me and I'm not able to find any documents that help explain what may be causing this and how to troubleshoot and find an eventual solution. I would really appreciate any tips or suggestions from anyone on the list that may have seen issues like this in the past. Thanks, Ben