On Fri, May 31, 2013 at 03:25:22PM -0700, Mike wrote:
Gang,
In the interest of sharing 'the weird stuff' which makes the job of being an operator ... uh, fun? is that the right word?..., I would like to present the following two smokeping latency/packetloss plots, which are by far the weirdest I have ever seen.
These plots are from our smokeping host out to a customer location. The customer is connected via DSL and they run PPPoE over it to connect with our access concentrator. There is about 5 physical insfastructure hops between the host and customer; The switch, the BRAS, the Switch again, and then directly to the DSLAM and then customer on the end.
The 10 day plot: http://picpaste.com/10_Day_graph-YV3IdvRV.png
The 30 hour plot: http://picpaste.com/30_hour_graph-DrwzfhYJ.png
How can you possibly have consistent increase in latency like that? I'd love to hear theories (or offers of beer, your choice!).
Theory: There's a stateful device (firewall, NAT, something else) in the path that is creating state for every ICMP Echo Request it forwards and (possibly) searching that state when forwarding the ICMP Echo Reply responses, and never destroying that state, and either the create operation or the search operation (or both) takes an amount of time that is a linear function of the number of state entries. -- Brett