Seems like QWEST doesn't have any edge ACL's in place to deal with this lovely worm issue. Count Source Prexix, rounded up to a /16 144 208.46.0.0 199 65.114.0.0 347 208.45.0.0 462 65.118.0.0 486 65.119.0.0 702 208.44.0.0 ---- 2340 TOTAL Packets out of 2500 for 2 seconds This is ICMP and TCP MS bad traffic for a 2500 packet capture on a DS1 that is directly connected to Qwest. Ergo, Qwest is the transit provider. Capture period was about 2 seconds. ICK According to Qwest Tech/Noc people they can't leave filters up for more than 1 day. Given that this worm has lasted more than 1 day, I'd think its reasonable to leave filters up for say more than one day ???? The other thing I learned from QWEST IP-NOC was that it seems managment decided *NOT TO* filter packets related to this worm issue at the edge...... john brown AS 10480 and others
Not sure how many places you intend to post this or related messages, but if you've got a problem vote with your money. Whining to NANOG and a slew of other mailing lists and still giving money to Qwest seems silly to me... Likewise, the Qwest folks likely aren't quite as clueless as you've attempted to portray them over the last few days, silly policies (policies that are clearly in place for a reason) can be fixed -- and I assure you, above all else, money talks... -danny On Thursday, August 28, 2003, at 09:25 PM, John Brown wrote:
Seems like QWEST doesn't have any edge ACL's in place to deal with this lovely worm issue.
Count Source Prexix, rounded up to a /16
144 208.46.0.0 199 65.114.0.0 347 208.45.0.0 462 65.118.0.0 486 65.119.0.0 702 208.44.0.0 ---- 2340 TOTAL Packets out of 2500 for 2 seconds
This is ICMP and TCP MS bad traffic for a 2500 packet capture on a DS1 that is directly connected to Qwest. Ergo, Qwest is the transit provider. Capture period was about 2 seconds. ICK
According to Qwest Tech/Noc people they can't leave filters up for more than 1 day.
Given that this worm has lasted more than 1 day, I'd think its reasonable to leave filters up for say more than one day ????
The other thing I learned from QWEST IP-NOC was that it seems managment decided *NOT TO* filter packets related to this worm issue at the edge......
john brown AS 10480 and others
Sorry to those that may be on other lists. Given general operational nature, I posted to NANOG, so that: 1. money can talk, others will see one view of this provider 2. operationally maybe something will get done 3. policy wise maybe this provider will change its policy 4. Qwest said their people had installed the ACL's properly my evidence is to the contrary. The customer that was impacted is certainly considering their options. I suspect they will vote with their checkbook. PS: Slew == 1 Private email list, 1, Well known public list 1 Local Public-ish list. Slew != as large as it may have sounded... Policies are sometimes in place for good reasons, sometimes because the makers of said policy are void clue. To assume they are inplace for good reason is a leap imho. Some Qwest people I've worked with on this issue are rich with clue, others (ergo via the nice normal paths) are not. My thanks to those that have clue, and my suggestion to management that they help those without clue. On Thu, Aug 28, 2003 at 09:36:37PM -0600, Danny McPherson wrote:
Not sure how many places you intend to post this or related messages, but if you've got a problem vote with your money. Whining to NANOG and a slew of other mailing lists and still giving money to Qwest seems silly to me...
Likewise, the Qwest folks likely aren't quite as clueless as you've attempted to portray them over the last few days, silly policies (policies that are clearly in place for a reason) can be fixed -- and I assure you, above all else, money talks...
-danny
On Thursday, August 28, 2003, at 09:25 PM, John Brown wrote:
Seems like QWEST doesn't have any edge ACL's in place to deal with this lovely worm issue.
Count Source Prexix, rounded up to a /16
144 208.46.0.0 199 65.114.0.0 347 208.45.0.0 462 65.118.0.0 486 65.119.0.0 702 208.44.0.0 ---- 2340 TOTAL Packets out of 2500 for 2 seconds
This is ICMP and TCP MS bad traffic for a 2500 packet capture on a DS1 that is directly connected to Qwest. Ergo, Qwest is the transit provider. Capture period was about 2 seconds. ICK
According to Qwest Tech/Noc people they can't leave filters up for more than 1 day.
Given that this worm has lasted more than 1 day, I'd think its reasonable to leave filters up for say more than one day ????
The other thing I learned from QWEST IP-NOC was that it seems managment decided *NOT TO* filter packets related to this worm issue at the edge......
john brown AS 10480 and others
On Thursday, August 28, 2003, at 09:51 PM, John Brown wrote:
Given general operational nature, I posted to NANOG, so that: 1. money can talk, others will see one view of this provider
Don't talk with other peoples money, talk with your own. If you plan to post to NANOG, it'd be a wise assumption that a significant subset of the folks here reside on other lists you post to as well.
2. operationally maybe something will get done
Perhaps. Though if/when it does, it'll be Qwest and you that will be involved, no one here.
3. policy wise maybe this provider will change its policy
Perhaps, though given the discussions on this and a hundred other lists in the last three weeks, I'm not sure providers know what to do. As Sean points out, every other email contradicts the previous. If I filter, I'm responsive, clueful & saving the Internet. When something breaks as a result, I'm clueless and trying to play netpolice, violating my SLA, plain suck, and need to just worry about delivering packets.
4. Qwest said their people had installed the ACL's properly my evidence is to the contrary.
Hence the need to further engage with Qwest, folks here will be of little benefit at the end of the day.
The customer that was impacted is certainly considering their options. I suspect they will vote with their checkbook.
PS: Slew == 1 Private email list, 1, Well known public list 1 Local Public-ish list.
Slew != as large as it may have sounded...
Correct me if I'm wrong, but I seem to recall a strikingly similar message posted to several mailing lists regarding very similar topics and the same provider within the past .. 4 days (no, it was 2 days)? Had it not been for that I wouldn't have bothered posting. One attempt to humiliate your provider in order to trigger some action is perhaps arguable, two or more is just plain annoying.
Policies are sometimes in place for good reasons, sometimes because the makers of said policy are void clue. To assume they are inplace for good reason is a leap imho.
So providers should play netpolice or Internet-Firewall-provider some amount of time, depending on _your gauge of the activity of a given incident? Folks need to realize that if large networks didn't have policies of this sort in place they'd be blocking pretty much every port on every interface by now.. You can't have it both ways... -danny
At 11:36 PM 8/28/2003, Danny McPherson wrote:
Not sure how many places you intend to post this or related messages, but if you've got a problem vote with your money. Whining to NANOG and a slew of other mailing lists and still giving money to Qwest seems silly to me...
Agreed...
Likewise, the Qwest folks likely aren't quite as clueless as you've attempted to portray them over the last few days, silly policies (policies that are clearly in place for a reason) can be fixed -- and I assure you, above all else, money talks...
I dunno... in my experience, Qwest is pretty clue-free. Of course money talks, but it takes a LOT of defections to make a significant impact.
On Fri, 29 Aug 2003, Randy Bush wrote:
when folk want to pay $50/mb, how much clue do we think isps can pay for, especially to deal with peak clue loads such as this last week or two?
yes, money talks. but in many ways.
Doesn't work this way. It is much better to have one clueful guy than to keep three clueless ones. Costs the same, the results are strikingly different. --vadim
Anyone that works for Qwest (Spirit of Service.....HA HA HA HA HA) and can actually stop having your clueless NOC personnel from calling me at the flipping early hours of the morning because your non working proactive monitoring system keeps opening pro active tickets. No one has yet to verify that at any of the countless times (yes this little ordeal has been going on for months now) that your so called pro active monitoring system opens a ticket that it has ever been right. Ever heard of false positives???????? Funny that your pro active ticket has never really detected an actual issue, because when these do happen it takes over a couple of hours to get anyone to begin the troubleshooting process. Is it customary for Qwest to call customers at 2, 3, 4, or 5 AM to tell them that they have a ticket opened by their pro active system? Here is a concept....get the proactive ticket, pull the interface, or look at the circuit before calling your customers...now that would be a Spirit of Service. What you are doing now is the spirit of laziness........ Gerardo A. Gregory Manager Network Administration and Security 402-970-1463 (Direct) 402-850-4008 (Cell) ------------------------------------------------ Affinitas - Latin for "Relationship" Helping Businesses Acquire, Retain, and Cultivate Customers Visit us at http://www.affinitas.net
I apologize to the list for including a subject line in all caps regarding my attempt to contact someone at Qwest to fix this "pro active monitoring" issue I have. I hope that someone from that network contacts me since all other normal channels of communication that they provide to their customers has not provided a solution in the months that this issue has been going on. So far, opening tickets, calling the NOC, escalating to managers, and the local Qwest team have provided no solution to these erroneous alarms. I am just given the ol' "We took care of it" until the next 2 AM pro active ticket gets opened, and once again am roused from my sleep because of a false alarm that they could not bother veryfing first. My apologies for the All Caps subject line. Rico Gerardo Gregory writes:
Anyone that works for Qwest (Spirit of Service.....HA HA HA HA HA) and can actually stop having your clueless NOC personnel from calling me at the flipping early hours of the morning because your non working proactive monitoring system keeps opening pro active tickets. No one has yet to verify that at any of the countless times (yes this little ordeal has been going on for months now) that your so called pro active monitoring system opens a ticket that it has ever been right. Ever heard of false positives???????? Funny that your pro active ticket has never really detected an actual issue, because when these do happen it takes over a couple of hours to get anyone to begin the troubleshooting process. Is it customary for Qwest to call customers at 2, 3, 4, or 5 AM to tell them that they have a ticket opened by their pro active system? Here is a concept....get the proactive ticket, pull the interface, or look at the circuit before calling your customers...now that would be a Spirit of Service. What you are doing now is the spirit of laziness........
Gerardo A. Gregory Manager Network Administration and Security 402-970-1463 (Direct) 402-850-4008 (Cell) ------------------------------------------------ Affinitas - Latin for "Relationship" Helping Businesses Acquire, Retain, and Cultivate Customers Visit us at http://www.affinitas.net
Gerardo A. Gregory Manager Network Administration and Security 402-970-1463 (Direct) 402-850-4008 (Cell) ------------------------------------------------ Affinitas - Latin for "Relationship" Helping Businesses Acquire, Retain, and Cultivate Customers Visit us at http://www.affinitas.net
The other thing I learned from QWEST IP-NOC was that it seems managment decided *NOT TO* filter packets related to this worm issue at the edge......
an isp of any non-trivial size, has one or more customers who are either in the security business or in security research. also ip behavior business or research. or ... the job of isps is to deliver packets, not to alter or drop them. if a custumer wishes there traffic shaped, dropped, mangled, ... at the edge, that's a nice [sellable] extra service. randy, who is right now trying to chase down what and why an upstream has done to stop some traffic i was measuring, harumph!
participants (7)
-
bdragon＠gweep.net -
Danny McPherson -
Dave Stewart -
Gerardo Gregory -
John Brown -
Randy Bush -
Vadim Antonov