Best practices for sending network maintenance notifications
All, We recently, at $dayjob, had one of our peers (at Symantec) send out a network maint notification, putting 70 addresses in the "To:" field, rather than using BCC or the exchange's mailing list. Naturally, when you mail 30 addresses, of the forms peering@ and noc@ various organizations, you're likely to hit at least a few autoresponders and ticket systems... And at least one or two of those autoresponders are of course brainded and configured to reply-all. (In this case, Verizon's ServiceNow setup was such a stupid responder). And that made things fun in our own ticket system, as our RT setup happily created a bunch of tickets. My question for the group -- does anyone know if there's a "best practices" for sending maint notifications like this? An RFC sort of thing? While it would define a social protocol, rather than a truly technical one, if there's not such a document, it seems like it could useful. And once such a thing exists, exchanges could of course helpfully point their members AT it (for both their humans, and ticket systems, to follow). -Dan --
I think there was a BCP being worked on. I seem to recall it was being discussed as a Facebook group. But there's no RFC, at least that I know of. Regards, Hal Ponton Senior Network Engineer Buzcom / FibreWiFi Tel: 07429 979 217 Email: hal@buzcom.net
On 6 Apr 2016, at 19:56, Dan Mahoney, System Admin <danm@prime.gushi.org> wrote:
All,
We recently, at $dayjob, had one of our peers (at Symantec) send out a network maint notification, putting 70 addresses in the "To:" field, rather than using BCC or the exchange's mailing list.
Naturally, when you mail 30 addresses, of the forms peering@ and noc@ various organizations, you're likely to hit at least a few autoresponders and ticket systems...
And at least one or two of those autoresponders are of course brainded and configured to reply-all. (In this case, Verizon's ServiceNow setup was such a stupid responder). And that made things fun in our own ticket system, as our RT setup happily created a bunch of tickets.
My question for the group -- does anyone know if there's a "best practices" for sending maint notifications like this? An RFC sort of thing?
While it would define a social protocol, rather than a truly technical one, if there's not such a document, it seems like it could useful. And once such a thing exists, exchanges could of course helpfully point their members AT it (for both their humans, and ticket systems, to follow).
-Dan
--
I think there was a BCP being worked on. I seem to recall it was being discussed as a Facebook group. But there's no RFC, at least that I know of.
And additionally, putting the recipients in the To: line sounds like a really bad idea. Sharing PII without permission and stuff like that. Make absolutely certain that all the SPF, DKIM and DMARC stuff is perfect. Make sure that any links are to the corporate domain... always. You want a neutral, paranoid 3rd party who is receiving the notice to be absolutely convinced of its Bona Fides. Do not suppose that your abundance of Sincerity excuses sloppiness. It won't. :(
Regards,
Hal Ponton
Senior Network Engineer
Buzcom / FibreWiFi
Tel: 07429 979 217 Email: hal@buzcom.net
On 6 Apr 2016, at 19:56, Dan Mahoney, System Admin <danm@prime.gushi.org> wrote:
All,
We recently, at $dayjob, had one of our peers (at Symantec) send out a network maint notification, putting 70 addresses in the "To:" field, rather than using BCC or the exchange's mailing list.
Naturally, when you mail 30 addresses, of the forms peering@ and noc@ various organizations, you're likely to hit at least a few autoresponders and ticket systems...
And at least one or two of those autoresponders are of course brainded and configured to reply-all. (In this case, Verizon's ServiceNow setup was such a stupid responder). And that made things fun in our own ticket system, as our RT setup happily created a bunch of tickets.
My question for the group -- does anyone know if there's a "best practices" for sending maint notifications like this? An RFC sort of thing?
While it would define a social protocol, rather than a truly technical one, if there's not such a document, it seems like it could useful. And once such a thing exists, exchanges could of course helpfully point their members AT it (for both their humans, and ticket systems, to follow).
-Dan
--
Aloha mai Nai`a. -- " So this is how Liberty dies ... http://kapu.net/~mjwise/ " To Thunderous Applause.
On Wed, Apr 6, 2016 at 3:05 PM, Hal Ponton <hal@buzcom.net> wrote:
I think there was a BCP being worked on. I seem to recall it was being discussed as a Facebook group.
True. https://www.facebook.com/groups/maintnote/ Currently under development, but fairly far along... Cheers, ~Chris -- @ChrisGrundemann http://chrisgrundemann.com
On 4/6/16 3:56 PM, Dan Mahoney, System Admin wrote:
All,
We recently, at $dayjob, had one of our peers (at Symantec) send out a network maint notification, putting 70 addresses in the "To:" field, rather than using BCC or the exchange's mailing list.
Naturally, when you mail 30 addresses, of the forms peering@ and noc@ various organizations, you're likely to hit at least a few autoresponders and ticket systems...
And at least one or two of those autoresponders are of course brainded and configured to reply-all. (In this case, Verizon's ServiceNow setup was such a stupid responder). And that made things fun in our own ticket system, as our RT setup happily created a bunch of tickets.
My question for the group -- does anyone know if there's a "best practices" for sending maint notifications like this? An RFC sort of thing?
In general I'd push for a little automation for the sending of notifications as reducing the likelihood of mishap. Targeting bcc is nice, but so does simply generating a message for each peer precludes this. we store contact information which bgp neighbor parameters in our config generation.
While it would define a social protocol, rather than a truly technical one, if there's not such a document, it seems like it could useful. And once such a thing exists, exchanges could of course helpfully point their members AT it (for both their humans, and ticket systems, to follow).
-Dan
On Wed, 6 Apr 2016, Dan Mahoney, System Admin wrote:
My question for the group -- does anyone know if there's a "best practices" for sending maint notifications like this? An RFC sort of thing?
It falls in the category of "Doctor, it hurts when I do this. Don't do that." Even the most dense CSR managers figure it out after a few attempts. The other "don't do that" is never configure Music on Hold for any NOC/SOC lines. Few things are more annoying than a eight hour trouble shooting conference bridge, and one of the dozen NOC/SOCs on the bridge hits the Hold button.
"The other "don't do that" is never configure Music on Hold for any NOC/SOC lines. Few things are more annoying than a eight hour trouble shooting conference bridge, and one of the dozen NOC/SOCs on the bridge hits the Hold button." Now that you've said it it seems so obvious. But, honestly I'd never thought it until right now. Thanks! Regards, Ray Orsini – CEO Orsini IT, LLC – Technology Consultants VOICE DATA BANDWIDTH SECURITY SUPPORT P: 305.967.6756 x1009 E: ray@orsiniit.com TF: 844.OIT.VOIP 7900 NW 155th Street, Suite 103, Miami Lakes, FL 33016 http://www.orsiniit.com | View My Calendar | View/Pay Your Invoices | View Your Tickets -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Sean Donelan Sent: Wednesday, April 6, 2016 4:53 PM To: nanog@nanog.org Subject: Re: Best practices for sending network maintenance notifications On Wed, 6 Apr 2016, Dan Mahoney, System Admin wrote:
My question for the group -- does anyone know if there's a "best practices" for sending maint notifications like this? An RFC sort of thing?
It falls in the category of "Doctor, it hurts when I do this. Don't do that." Even the most dense CSR managers figure it out after a few attempts. The other "don't do that" is never configure Music on Hold for any NOC/SOC lines. Few things are more annoying than a eight hour trouble shooting conference bridge, and one of the dozen NOC/SOCs on the bridge hits the Hold button.
And some genius at an ISP's NOC has put rick_asley_never_gonna_give_you_up.mp3 in the their hold queue music rotation list. On Wed, Apr 6, 2016 at 1:56 PM, Ray Orsini <ray@orsiniit.com> wrote:
"The other "don't do that" is never configure Music on Hold for any NOC/SOC lines. Few things are more annoying than a eight hour trouble shooting conference bridge, and one of the dozen NOC/SOCs on the bridge hits the Hold button."
Now that you've said it it seems so obvious. But, honestly I'd never thought it until right now. Thanks!
Regards, Ray Orsini – CEO Orsini IT, LLC – Technology Consultants VOICE DATA BANDWIDTH SECURITY SUPPORT P: 305.967.6756 x1009 E: ray@orsiniit.com TF: 844.OIT.VOIP 7900 NW 155th Street, Suite 103, Miami Lakes, FL 33016 http://www.orsiniit.com | View My Calendar | View/Pay Your Invoices | View Your Tickets
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Sean Donelan Sent: Wednesday, April 6, 2016 4:53 PM To: nanog@nanog.org Subject: Re: Best practices for sending network maintenance notifications
On Wed, 6 Apr 2016, Dan Mahoney, System Admin wrote:
My question for the group -- does anyone know if there's a "best practices" for sending maint notifications like this? An RFC sort of thing?
It falls in the category of "Doctor, it hurts when I do this. Don't do that." Even the most dense CSR managers figure it out after a few attempts.
The other "don't do that" is never configure Music on Hold for any NOC/SOC lines. Few things are more annoying than a eight hour trouble shooting conference bridge, and one of the dozen NOC/SOCs on the bridge hits the Hold button.
I've been on hold a few times with some companies that had great 80's music. I almost asked them to put me back on hold when they finally took me off. Sometimes it's a party when one of the people on the call hits the hold button, it depends on how bad the outage is :) On 4/6/2016 4:56 PM, Ray Orsini wrote:
"The other "don't do that" is never configure Music on Hold for any NOC/SOC lines. Few things are more annoying than a eight hour trouble shooting conference bridge, and one of the dozen NOC/SOCs on the bridge hits the Hold button."
Now that you've said it it seems so obvious. But, honestly I'd never thought it until right now. Thanks!
Regards, Ray Orsini – CEO Orsini IT, LLC – Technology Consultants VOICE DATA BANDWIDTH SECURITY SUPPORT P: 305.967.6756 x1009 E: ray@orsiniit.com TF: 844.OIT.VOIP 7900 NW 155th Street, Suite 103, Miami Lakes, FL 33016 http://www.orsiniit.com | View My Calendar | View/Pay Your Invoices | View Your Tickets
participants (9)
-
Chris Grundemann -
Dan Mahoney, System Admin -
Eric Kuhnke -
Hal Ponton -
joel jaeggli -
Michael J Wise -
Ray Orsini -
Robert Drake -
Sean Donelan