On Dec 2, 2013, at 12:19 AM, joel jaeggli <joelja@bogus.com> wrote:

Given a measurement target on the customer side and smokeping instance on your side you can actively measure the availability/latency/loss rates between them.

I think he's actually the end-customer, and he's saying that his upstream transit ISP won't accept non-RF-specific diags . . . ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton

I'm actually halfway through trying to setup a smokeping appliance. Sent from my BlackBerry wireless device from MTN -----Original Message----- From: joel jaeggli <joelja@bogus.com> Date: Sun, 01 Dec 2013 09:38:39 To: Dobbins, Roland<rdobbins@arbor.net>; nanog@nanog.org list<nanog@nanog.org> Subject: Re: Is there a method or tool(s) to prove network outages? On 12/1/13, 9:23 AM, Dobbins, Roland wrote:

On Dec 2, 2013, at 12:19 AM, joel jaeggli <joelja@bogus.com> wrote:

...

Given a measurement target on the customer side and smokeping instance on your side you can actively measure the availability/latency/loss rates between them.

I think he's actually the end-customer, and he's saying that his upstream transit ISP won't accept non-RF-specific diags . . .

and if you don't control any of the air interfaces you don't get that.

----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>

Luck is the residue of opportunity and design.

-- John Milton

Hmm. Great points. Didn't think of that. Sent from my BlackBerry wireless device from MTN -----Original Message----- From: Matt Palmer <mpalmer@hezmatt.org> Date: Mon, 2 Dec 2013 06:50:31 To: <nanog@nanog.org> Subject: Re: Is there a method or tool(s) to prove network outages? On Sun, Dec 01, 2013 at 05:56:51PM +0100, Notify Me wrote:

Please I have a very problematic radio link which goes out and back on again every few hours. The only way I know this is happening is from my gateway device: a Sophos UTM that sends email anytime there's been an outage.

The ISP refuses to accept this as outage/instability proof, and I'm wondering if there's something I can run behind the gateway UTM that can provide output information over time.

I'm surprised nobody's mentioned the root question to answer before you go off spending time setting up anything in particular: what *will* the ISP accept (or be forced to accept) as outage/instability proof? Contracts are your first line of defence, but it's nigh-on universal that they don't cover these sorts of situations well enough. So you probably need to have a discussion, as a follow-on from being told that your UTM's e-mails *aren't* sufficient, to determine what *is* sufficient. Once you've got that, only then can you evaluate appropriate methods of gathering the necessary data to support a claim of an outage. I like the *idea* of smokeping, but when gathering data on complete service loss (which was my use case for it as well) I found its methods of collecting and displaying that data to be very suboptimal and counter-intuitive. For something small and once-off like this, I'd probably just break out my text editor and script up something that would collect the relevant data and process it into the acceptable form. - Matt

On Dec 1, 2013, at 11:50 AM, Matt Palmer <mpalmer@hezmatt.org> wrote:

I'm surprised nobody's mentioned the root question to answer before you go off spending time setting up anything in particular: what *will* the ISP accept (or be forced to accept) as outage/instability proof? Contracts are your first line of defence, but it's nigh-on universal that they don't cover these sorts of situations well enough. So you probably need to have a discussion, as a follow-on from being told that your UTM's e-mails *aren't* sufficient, to determine what *is* sufficient.

This. They may not cooperate, in which case, you have to force proof down their throats. I would go with the Zenoss (or Zabbix, or...) option - a free to use, professionally supported, professional grade commonly used monitoring package that would meet anyone's basic "credible tool" definition plus neat GUI to send a snapshot of the results. Use it to perform various tests of the net - pings, http gets of some small target, starting pings with the next hop outside your premise and working outwards to the outside world. Don't overwhelm your net with tests, but test as often as needed to demonstrate an issue. -george william herbert george.herbert@gmail.com Sent from Kangphone

On Sun, 1 Dec 2013 20:25:36 +0000, Sina Owolabi <notify.sina@gmail.com> said: > Its cyclical, but I have not tried to graph/measure its > repetition before now... Body of tidal water..could be This is speculation until you have measurements, but if this is the case I'd wager you are having reflected signal interference off of the water. The water acts like a mirror and as it moves up and down the reflected signal will move in and out of phase with the main signal. At certain points you'll get near complete cancellation and the link will fail. See section 4 here for some explanations, fig 5 and 6 for what you could expect the graphs of signal strength, time, link capactity to look like: http://homepages.inf.ed.ac.uk/mmarina/papers/mobicom_winsdr08.pdf But not having access to the RF part you can't measure this directly. If you can get tide tables for a nearby location, what you could do is say that signal strength is 1 if the link is working and 0 if it is not. Measure for a while then scatterplot that against the level of the tide. If the measurements of 0 group tightly together in a few spots then you know definitely what is happening. Perhaps that plot together with a pointer to a nice academic paper would be enough to convince the provider of what is happening. What could you do about this? If you are lucky and the interference does not complete a full cycle from destructive to constructive and back with the largest amplitude of the tides that you experience in that place, you could try moving the antenna up or down. How much depends on the frequency and distances involved but I'd try 25cm increments up to a couple of meters if you can. You'll still get degradation but can hopefully avoid the deep nulls that take the link out completely. If you are able and willing to replace the end-site radios or antennas with your own, and the link uses some sort of 2xN MIMO, you could arrange vertical spacing between the antennas so that you have a good signal at one antenna when the other one is experiencing a null. This should get you on average half the best-case throughput the equipment is capable of but it should get you that consistently. The actual spacing depends on the distances and heights involved. -w

On Dec 2, 2013, at 6:26 AM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:

I would hold off on considering Multipath as a problem until you see the RSL.

Concur. It could also be related to precipitation or other adverse conditions. Or, in fact, it could be related to the 'UTM' box and/or something else on the endpoint network. It could be a periodic DDoS attack, or traffic causing an availability hit as an unintended consequence. It's difficult to say without data. Since the OP has the ability to gather IP-level data on his own network, he should utilize whatever instrumentation and telemetry he can set up in order to diagnose the issue as accurately as possible. And the OP should dig out his SLA and see what it says about the obligations of his upstream. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton

Thanks a lot for the in-depth insights, all. Ill be doing a lot of "sleuthing" in the next few days based on all this information. Sent from my BlackBerry wireless device from MTN -----Original Message----- From: Warren Bailey <wbailey@satelliteintelligencegroup.com> Date: Mon, 2 Dec 2013 03:09:13 To: Dobbins, Roland<rdobbins@arbor.net>; nanog@nanog.org<nanog@nanog.org> Subject: Re: Is there a method or tool(s) to prove network outages? Keep in mind that inter web traffic has nothing to do with the overall health of the radio link. In RF land, we really don¹t care what is going over that link - just that we have enough RSL hitting the receiver to be above threshold thus allowing the box to demodulate that signal. If your radio is sitting at a threshold RSL of -108 and you¹re coming in at -105, big trouble in little China (3dB fade murdered your link). Stop thinking like a network engineer.. If your DS-1 was taking hits, an ICMP request (or lack thereof) would mean little (read: zero) to me as an RF Engineer. I want to see the BER/PER of the circuit over time so I can correlate possible trouble with real world issues. With that being said.. the tidal issue comes up a lot, and more times than not I see someone who said ³Point that dish over there² and when it magically works they have earned the title of ³Best RF Engineer in History² until the tide rolls in and their link suddenly has ³issues². The invention of cheap wireless has caused many people to believe they have in depth wireless experience, and that is usually not the case. Not trying to preach, but I¹ve spent a *TON* of time and other people¹s money in multi path land.. If someone was responsible for the proper design of the link multi path would not be a factor as it would be addressed early on in the link. You are not going to gain much traction with a wireless company when you call and tell them your pings aren¹t working.. They are kind of like parents.. They just don¹t understand. ;) //warren Ps - I welcome any replies on or off list.. I know how frustrating it can be to have a link that seems to work well until you look at it, so I probably have a bit more compassion than others when talking about broken Microwave/Satellite hops. On 12/1/13, 5:40 PM, "Dobbins, Roland" <rdobbins@arbor.net> wrote:

On Dec 2, 2013, at 6:26 AM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:

...

I would hold off on considering Multipath as a problem until you see the RSL.

Concur. It could also be related to precipitation or other adverse conditions.

Or, in fact, it could be related to the 'UTM' box and/or something else on the endpoint network. It could be a periodic DDoS attack, or traffic causing an availability hit as an unintended consequence.

It's difficult to say without data. Since the OP has the ability to gather IP-level data on his own network, he should utilize whatever instrumentation and telemetry he can set up in order to diagnose the issue as accurately as possible.

And the OP should dig out his SLA and see what it says about the obligations of his upstream.

----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>

Luck is the residue of opportunity and design.

-- John Milton