From: Blake Dunlap [mailto:ikiris@gmail.com] While any provider will attempt to fix peer / upstream issues as they can, any SLA you would have is between two points on their private network, not from point A to point Z that they have no control over across multiple peers and the public internet itself.

makes sense- thanks for confirming

The much more common design is using a single provider for each thread between sites. Then at least you have an end-to- end SLA in effect, as well as a single entity that is responsible for the entire link in question.

This sounds like you're trying to achieve private link IGP / FRR level site to site failover/convergence across the public internet. Perhaps you should rethink your goals here or your design?

Kind of- I can actually tolerate the blips, but I want to be able to measure and track them in such a way that I know where the loss is occurring. If a particular path is reconverging more often than should be reasonably expected I want to be able to prove it within reason. We also have a customer who happens to host at DC B with the same connectivity. Every time there is one of these blips their alerting fires off a thousand messages and they open a ticket with us. I'd like to be able to show them some good data on the path during the blip so we back a discussion along the lines of "live with it, or pay to privately connect to us". -andy

-Blake

On Mon, Jul 15, 2013 at 4:18 PM, Andy Litzinger <Andy.Litzinger@theplatform.com> wrote: Hi,

Does anyone have any recommendations on how to pinpoint and react to packet loss across the internet?  preferably in an automated fashion.  For detection I'm currently looking at trying smoketrace to run from inside my network, but I'd love to be able to run traceroutes from my edge routers triggered during periods of loss.  I have Juniper MX80s on one end- which I'm hopeful I'll be able to cobble together some combo of RPM and event scripting to kick off a traceroute.  We have Cisco4900Ms on the other end and maybe the same thing is possible but I'm not so sure.

I'd love to hear other suggestions and experience for detection and also for options on what I might be able to do when loss is detected on a path.

In my specific situation I control equipment on both ends of the path that I care about with details below.

we are a hosted service company and we currently have two data centers, DC A and DC B.  DC A uses juniper MX routers, advertises our own IP space and takes full BGP feeds from two providers, ISPs A1 and A2.  At DC B we have a smaller installation and instead take redundant drops (and IP space) from a single provider, ISP B1, who then peers upstream with two providers, B2 and B3

We have a fairly consistent bi-directional stream of traffic between DC A and DC B.  Both of ISP A1 and A2 have good peering with ISP B2 so under normal network conditions traffic flows across ISP B1 to B2 and then to either ISP A1 or A2

oversimplified ascii pic showing only the normal best paths:

              -- ISP A1----------------------ISP B2-- DC A-- |                                                                 |---  ISP B1 ----- DC B              -- ISP A2----------------------ISP B2--

with increasing frequency we've been experiencing packet loss along the path from DC A to DC B.  Usually the periods of loss are brief,  30 seconds to a minute, but they are total blackouts.

  I'd like to be able to collect enough relevant data to pinpoint the trouble spot as much as possible so I can take it to the ISPs and request a solution.  The blackouts are so quick that it's impossible to log in and get a trace- hence the desire to automate it.

I can provide more details off list if helpful- I'm trying not to vilify anyone- especially without copious amounts of data points.

As a side question, what should my expectation be regarding packet loss when sending packets from point A to point B across multiple providers across the internet?  Is 30 seconds to a minute of blackout between two destinations every couple of weeks par for the course?  My directly connected ISPs offer me an SLA, but what should I reasonably expect from them when one of their upstream peers (or a peer of their peers) has issues?  If this turns out to be BGP reconvergence or similar do I have any options?

many thanks, -andy