We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact. Normally we get notices in advance, even for software upgrades due to security or other important issues, so I am curious if other qwest customers had the same experience and wether this is how it's going to be from here on in? The affected platform was juniper and I'd love to know the specfic case being addressed here. Mike-
Any other specifics? Got a trouble ticket ID? I'm located in the NW (Talent, Oregon, just over CA border..) and we have a few customers on Qwest T1's and the like. We also have a customer who gets MPLS directly from Q. We've yet to hear of any outages for our customers - but I suppose the night is still young... Any other information you got might be helpful.. Regards, Steve On 1/7/2010 2:04 AM, Mike wrote:
We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact. Normally we get notices in advance, even for software upgrades due to security or other important issues, so I am curious if other qwest customers had the same experience and wether this is how it's going to be from here on in? The affected platform was juniper and I'd love to know the specfic case being addressed here.
Mike-
Give the nature of the issue from juniper I am guessing that a large number of companies were doing upgrades over the last few days as fast as they could. http://j.mp/8XaReK has the details I know of now. On Jan 7, 2010, at 5:14 AM, Steve Ryan wrote:
Any other specifics? Got a trouble ticket ID?
I'm located in the NW (Talent, Oregon, just over CA border..) and we have a few customers on Qwest T1's and the like. We also have a customer who gets MPLS directly from Q.
We've yet to hear of any outages for our customers - but I suppose the night is still young...
Any other information you got might be helpful..
Regards,
Steve
On 1/7/2010 2:04 AM, Mike wrote:
We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact. Normally we get notices in advance, even for software upgrades due to security or other important issues, so I am curious if other qwest customers had the same experience and wether this is how it's going to be from here on in? The affected platform was juniper and I'd love to know the specfic case being addressed here.
Mike-
We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact. Normally we get notices in advance, even for software upgrades due to security or other important issues, so I am curious if other qwest customers had the same experience and wether this is how it's going to be from here on in? The affected platform was juniper and I'd love to know the specfic case being addressed here.
We received 7 Juniper Security Advisories today. My guess is that this is the reason for the Qwest outage you've seen. Steinar Haug, Nethelp consulting, sthaug@nethelp.no
On Thu, Jan 7, 2010 at 4:06 AM, <sthaug@nethelp.no> wrote:
We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact. Normally we get notices in advance, even for software upgrades due to security or other important issues, so I am curious if other qwest customers had the same experience and wether this is how it's going to be from here on in? The affected platform was juniper and I'd love to know the specfic case being addressed here.
We received 7 Juniper Security Advisories today. My guess is that this is the reason for the Qwest outage you've seen.
My QWest account manager called three different people at my business 7hrs before the maintenance. Also mentioned the Juniper Security Advisories. -- Later, Joe
We also got email notifications about 'emergency maintenance' on our Qwest circuits, from their notice: Reason For Maintenance: EMERGENCY MAINTENANCE TO IMPLEMENT A SOFTWARE PATCH FOR NETWORK RELIABILITY Sure sounds like it's all related to the Juniper advisory to me. Ken Matlock Network Analyst Exempla Healthcare (303) 467-4671 matlockk@exempla.org -----Original Message----- From: JoeSox [mailto:joesox@gmail.com] Sent: Thursday, January 07, 2010 8:25 AM To: nanog@nanog.org Subject: Re: qwest outage no notice My QWest account manager called three different people at my business 7hrs before the maintenance. Also mentioned the Juniper Security Advisories. -- Later, Joe
sthaug@nethelp.no wrote:
We received 7 Juniper Security Advisories today. My guess is that this is the reason for the Qwest outage you've seen.
Yeah, they refused to notify due to security concerns from what they told me last night. Notification was performed after maintenance was complete. Jack
Yeah, they refused to notify due to security concerns from what they told me last night. Notification was performed after maintenance was complete.
Ok, so the next question is, what harm would a simple advance notice of 'emergency maintenance' caused, vs the very real hassle and inconvenience that DS3-down in the middle of the night caused for operations staff? I personally was yanked out of bed by my network monitoring systems and had to spend at least 5 minutes looking at logs and perf monitors and so forth so I had enough information before making the call to qwest support, thinking we took a hit on our fiber. Some others reported getting 7 or more hours of advance notice - this would have been enough for me, even tho of course I don't like it - and I suspect it could have saved many many others from a similar fate of responding to the event so late. Just saying "we need to do this", does not give the bad guys any ammo with which to attack, and would go a long ways twords preventing needless heroics such as middle of the night investigations by senior staff. The follow up email we subsequently received was fine and we appreciated learning it really was a necessary upgrade and seems justified now with that knowledge, I would simply have appreciated not having to engage my emergency processes for something that was planned. Mike-
That should read... Yeah, they refused to notify due to [marketing] concerns from what they Richey -----Original Message----- From: Mike [mailto:mike-nanog@tiedyenetworks.com] Sent: Thursday, January 07, 2010 12:04 PM Cc: nanog@nanog.org Subject: Re: qwest outage no notice
Yeah, they refused to notify due to security concerns from what they told me last night. Notification was performed after maintenance was complete.
Ok, so the next question is, what harm would a simple advance notice of 'emergency maintenance' caused, vs the very real hassle and inconvenience that DS3-down in the middle of the night caused for operations staff? I personally was yanked out of bed by my network monitoring systems and had to spend at least 5 minutes looking at logs and perf monitors and so forth so I had enough information before making the call to qwest support, thinking we took a hit on our fiber. Some others reported getting 7 or more hours of advance notice - this would have been enough for me, even tho of course I don't like it - and I suspect it could have saved many many others from a similar fate of responding to the event so late. Just saying "we need to do this", does not give the bad guys any ammo with which to attack, and would go a long ways twords preventing needless heroics such as middle of the night investigations by senior staff. The follow up email we subsequently received was fine and we appreciated learning it really was a necessary upgrade and seems justified now with that knowledge, I would simply have appreciated not having to engage my emergency processes for something that was planned. Mike-
Same thing for us in Minnesota. Brief outage and emergency outage notification came after the outage. The outage window was for 6:00-12:00GMT, and the outage came at 6:15GMT. We didn't get the notice until 10:30GMT. Qwest had a major outage over the Xmas weekend in MN. I wonder if this is related. They told me it was a bad switch, but that sounded funny. Dylan Ebner -----Original Message----- From: Jack Bates [mailto:jbates@brightok.net] Sent: Thursday, January 07, 2010 10:25 AM To: sthaug@nethelp.no Cc: nanog@nanog.org Subject: Re: qwest outage no notice sthaug@nethelp.no wrote:
We received 7 Juniper Security Advisories today. My guess is that this is the reason for the Qwest outage you've seen.
Yeah, they refused to notify due to security concerns from what they told me last night. Notification was performed after maintenance was complete. Jack
Notices were left at the discretion of Qwest account teams. There was no mass notification. Jason -----Original Message----- From: Mike [mailto:mike-nanog@tiedyenetworks.com] Sent: Thursday, January 07, 2010 4:04 AM To: NANOG list Subject: qwest outage no notice We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact. Normally we get notices in advance, even for software upgrades due to security or other important issues, so I am curious if other qwest customers had the same experience and wether this is how it's going to be from here on in? The affected platform was juniper and I'd love to know the specfic case being addressed here. Mike- *** NOTICE--The attached communication contains privileged and confidential information. If you are not the intended recipient, DO NOT read, copy, or disseminate this communication. Non-intended recipients are hereby placed on notice that any unauthorized disclosure, duplication, distribution, or taking of any action in reliance on the contents of these materials is expressly prohibited. If you have received this communication in error, please delete this information in its entirety and contact the Amedisys Privacy Hotline at 1-866-518-6684. Also, please immediately notify the sender via e-mail that you have received this communication in error. ***
Once upon a time, Mike <mike-nanog@tiedyenetworks.com> said:
We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact. Normally we get notices in advance, even for software upgrades due to security or other important issues, so I am curious if other qwest customers had the same experience and wether this is how it's going to be from here on in? The affected platform was juniper and I'd love to know the specfic case being addressed here.
I got 3 notices about the outage related to our 1 Qwest OC-3. As for the Juniper security issues, see juniper-nsp archives. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jan 7, 2010, at 4:04 AM, Mike wrote:
We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact. Normally we get notices in advance, even for software upgrades due to security or other important issues, so I am curious if other qwest customers had the same experience and wether this is how it's going to be from here on in? The affected platform was juniper and I'd love to know the specfic case being addressed here.
Mike-
We experienced the outage, but so far have not received any notifications. Steve - -- - --------------------------------------------------------------- Steven Saner <ssaner@hubris.net> Director of Network Operations Hubris Communications -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAktF+xsACgkQvgCxUpg3pZPDrwCgo9IPUGExaOlsmAkoctnpzzkR SpYAniXz5P8Y/2YiCZMjD/t3Bx4/SmC0 =sqgK -----END PGP SIGNATURE-----
Mike wrote:
We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact.
Hmm - I got notice in advance. I'll have to go search for the email, but it was sometime before Christmas, since I had it on my calendar. -C
This is one reason why companies should use twitter...great for those impromptu emergency messages. Our electrical utility company uses twitter and I have to admit that it's nice to know what is going on ahead of time even if it is a short message! On Thu, 2010-01-07 at 10:31 -0700, Chris De Young wrote:
Mike wrote:
We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact.
Hmm - I got notice in advance. I'll have to go search for the email, but it was sometime before Christmas, since I had it on my calendar.
-C
Probably related to this: http://www.theregister.co.uk/2010/01/07/juniper_critical_router_bug/ -----Original Message----- From: Chris De Young [mailto:chd@arizona.edu] Sent: Thursday, January 07, 2010 9:32 AM To: Mike Cc: NANOG list Subject: Re: qwest outage no notice Mike wrote:
We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact.
Hmm - I got notice in advance. I'll have to go search for the email, but it was sometime before Christmas, since I had it on my calendar. -C
On Thu, Jan 7, 2010 at 5:04 AM, Mike <mike-nanog@tiedyenetworks.com> wrote:
We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact.
That's not a maintenance, that's an outage. I hope everybody impacted on this list is claiming SLA. Drive Slow, much like the M40, Paul Wall
Paul Wall wrote:
On Thu, Jan 7, 2010 at 5:04 AM, Mike <mike-nanog@tiedyenetworks.com> wrote:
We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact.
That's not a maintenance, that's an outage.
I hope everybody impacted on this list is claiming SLA.
Qwest NEVER EVER provides SLA adjustments, no longer how long it's down or what their own role in it being down is. They toss it from department to department and then hand down judgments that 'we're not providing credits' and that's that. So in the three years we've had qwest we've had probably 10 major service impacting failures and at least 3 over 6 hours each, and a 9 month period where the (ssshhh!! secret!!) mpls tunnel they put our 45mbs 'clear channel ds3' on was oversubscribed with another user, resulting on 600+ms latencies at random times and they refused to accept our mrtg and smokeping traces showing the problem. No adjustments at all, just pay up, and oh we'll get around to that someday when it suits us but you have nothing to say about it that we care about. </rant>
On Sat, 09 Jan 2010 07:00:42 -0800, Mike wrote:
Qwest NEVER EVER provides SLA adjustments, no longer how long it's down or what their own role in it being down is. They toss it from department
If they honored every SLA adjustment they would not be able to pay the current stockholders a 6.8% yield! At 20.63% over their 200day and 12.6% over their 50day moving average their stock is only 4.52% below their 52 week high, and about 50% down from its 2007 high, makes their stock a better investment than their service. Well that is assuming you do not look back to 2000 when it was a $50 stock before the roller coaster ride to a low of a little over $2 in oct 2009 and is going back up again due to that extra little bit of cash they can squeeze for those damb customers. If it was not for the customers, this would be a good business investment. Back under my rock .... The other Bob
On Sat, Jan 9, 2010 at 9:37 AM, Paul Wall <pauldotwall@gmail.com> wrote:
On Thu, Jan 7, 2010 at 5:04 AM, Mike <mike-nanog@tiedyenetworks.com> wrote:
We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact.
That's not a maintenance, that's an outage.
I hope everybody impacted on this list is claiming SLA.
Drive Slow, much like the M40, Paul Wall
SLA for what? < 2m of outage time related to an emergency maintenance event? I don't think so. Most agreement language covers this kind of event. You'll be lucky if you can badger your account team into a free dinner and/or some free beer for it. -M< -- Martin Hannigan martin@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Martin Hannigan wrote:
On Sat, Jan 9, 2010 at 9:37 AM, Paul Wall <pauldotwall@gmail.com> wrote:
On Thu, Jan 7, 2010 at 5:04 AM, Mike <mike-nanog@tiedyenetworks.com> wrote:
We just had a qwest outage of about 2 mins at 1:41am pst. When I called to report it I was told it was a 200+ emergency software upgrade due to a security concern, and that we will get a notice later after the fact. That's not a maintenance, that's an outage.
I hope everybody impacted on this list is claiming SLA.
Drive Slow, much like the M40, Paul Wall
SLA for what? < 2m of outage time related to an emergency maintenance event? I don't think so. Most agreement language covers this kind of event.
I think it comes down to disclosure policy. I cannot imagine qwest was the only provider who was hit by this emergency patch upgrade. I don't think general public is really keen in knowing the exact details of the vulnerabilities as much as getting some sort of heads up about emergency maintenance window which IMHO should have been issued. Given that I'm a staunch believer in openness when it comes down to outages related to critical infrastructure :-) regards, /virendra
You'll be lucky if you can badger your account team into a free dinner and/or some free beer for it.
-M<
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFLSVTkpbZvCIJx1bcRAvX2AJsEmwKOBOs9Ynl8VaHTnS0SLP+SlACgykj3 nEmbjBDWmFj6XUI3k7Kv7fw= =yGz+ -----END PGP SIGNATURE-----
participants (19)
-
Bob Bradlee -
Bret Clark -
Chris Adams -
Chris De Young -
Dylan Ebner -
Greg Olson -
Jack Bates -
Jason Shearer -
Jeremy Rossi -
JoeSox -
Martin Hannigan -
Matlock, Kenneth L -
Mike -
Paul Wall -
Richey -
Steve Ryan -
Steven Saner -
sthaug＠nethelp.no -
virendra rode