Pakistan government orders ISP service level agreement
Pakistan has been suffering a 40 day DOS attack, disrupting most of the International Internet service in the country. The Pakistan Government, Minister for IT & Telecom, has "directed" the Pakistan Telecommunications Corporation to sign service level agreements to ensure 99% Internet availability. Over 200 official(government?) websites have been inaccessible for over three weeks. http://www.paknews.com/main.php?id=5&date1=2003-05-05 "Within a period of five days, the body has been assigned to formulate a concrete strategy to stem the incidence of DoS attacks.. The committee will work on urgent basis and suggest short-term measures within two day."
On Mon, 5 May 2003, Christopher L. Morrow wrote:
I suppose contacting upstreams for traceback and filtration isn't something they want to do??
I may not be on the correct lists to hear about the local Pakistan ISP scene, but I didn't find much technical information about what was happening. The attacks APPEAR to be originating from India and a virus called YAHA which targets certain Pakistan government web sites, .PK top-level domain name servers and the primary Internet exchange in Pakistan. Most of the Pakistan Internet traffic is funneled through a single exchange point and official provider PTCL. Good for intelligence agencies, bad for reliability and DDOS survivability. YAHA variants have been circulating since last June. The infected computers are inside and outside of Pakistan, and continuing to increase So you need to solve 1. Viruses 2. Spam 3. DDOS I suppose you could renumber/rename Pakaistan. But when the next variant of Yaha was released, it would probably just include the updated information. One of the government ministers suggested moving some of the official sites to the colocation in the US. It might help in the short term, since the US has excess bandwidth and can absorb the attacks longer.
On Monday, May 5, 2003, at 14:58 Canada/Eastern, Sean Donelan wrote:
Two (of three) of the PK servers are in the US (in AS701). If the attacks were happening due to some generally-distributed windows worm, and were targetting the PK servers with any degree of ferocity, you'd think this would be more than just a South Asian problem. It's possible, of course, that the extremely different scales of infrastructure deployment in South Asia compared with the 701 backbone could cause the same attack traffic to be highly disruptive to the former, and yet barely noticable by the latter. Joe
Another example of how lack of splay allows DDOS to devastate a network. Paul Vixie's presentation on anycast F root server provided good points about splay. They need more peer points and sites, not a move from point A to point B. On Monday, May 5, 2003, at 01:58 PM, Sean Donelan wrote:
-- Joseph T. Klein VP/CTO and bottle washer Titania Corporation, Inc. PSTN: +1 415 462 1534 Mobile: +1 414 628 3380
"Joseph T. Klein" <jtk@titania.net> writes:
Another example of how lack of splay allows DDOS to devastate a network.
Give it up on the splay argument please. Maybe in poorly engineered networks you need a high splay factor (high being user defined). The point is all moot, considering how easy it is to make a router go explody today. High splay or not, it isn't going to do jack when your boxes crash. /vijay
The Pakistan Telecommunications Company Ltd has aquired a firewall to solve the DDOS situation impacting Internet service in the country. An unnamed security advisor asserted the proper use of a firewall would control the DDOS attacks and prevent hacking. I can understand the Pakistan government minister's frustration, and desire to get things fixed. Unfortunately, it seems like security incidents also attract security snake oil consultants. Buy my tonic to cure your ills.
SD> Date: Tue, 6 May 2003 19:28:48 -0400 (EDT) SD> From: Sean Donelan SD> The Pakistan Telecommunications Company Ltd has aquired a SD> firewall to solve the DDOS situation impacting Internet SD> service in the country. An unnamed security advisor asserted SD> the proper use of a firewall would control the DDOS attacks SD> and prevent hacking. Now the DDoS melts the pipes _and_ the firewall. I'd like to know if said "consultant" ever considered recommending the PTC contact their upstreams for help with backtrace/blocking. Anyone with a modicum of clue (or Google access) should figure out that one... Eddy -- Brotsman & Dreger, Inc. - EverQuick Internet Division Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 (785) 865-5885 Lawrence and [inter]national Phone: +1 (316) 794-8922 Wichita ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
Hi, NANOGers. ] Now the DDoS melts the pipes _and_ the firewall. Bonus prize: A DDoS that wouldn't fill the pipe melts the firewall. On the bright side the firewall will likely fail long before the attack causes any noticeable pain to the components it is in place to protect. :| <http://www.kb.cert.org/vuls/id/539363> <http://www.qorbit.net/documents/maximizing-firewall-availability.pdf> <http://www.cymru.com/SteveGill/maximizing-firewall-availability.pdf> Been there, somewhat survived that, Rob. -- Rob Thomas http://www.cymru.com ASSERT(coffee != empty);
On 5/6/03 7:51 PM, "E.B. Dreger" <eddy+public+spam@noc.everquick.net> wrote:
Not every upstream is as clueful as Uunet, and not every noc employee is as clueful as Chris and Brian at UUnet. It has been my experience that most upstreams have no concept that they CAN backtrace, and generally have no interest in helping you do it. I'm not mudslinging here, so I won't say who my experience is with, but a few transitless/near transitless upstreams I've dealt with were most unhelpful, either because they didn't know how to help, or worse, they did know how to help and didn't care. And, depending on the nature of the DDoS attack, perhaps it isn't related to saturation, but rather to overloading router processors, or something else that can effectively be filtered customer-side? Our policy as of late has just been to make sure we have equipment on our side fast enough to filter at wire speed, and get enough capacity to our upstreams that it is signifigantly unlikely that anyone could generate enough traffic to saturate it (in which case, we would have no choice but to ask carriers to filter, and backtrace). --Phil ISPrime
Unless you actually call UUnet and your not a customer, God help you then. Some companies are very very good at dealing with DDOS, Internap being one and UUNET if you are a customer another. Even a post here although maybe not exactly proper will get you responses from people like Chris and so on who can and will be helpful. ----- Original Message ----- From: "Phil Rosenthal" <pr@isprime.com> To: "E.B. Dreger" <eddy+public+spam@noc.everquick.net>; <nanog@merit.edu> Sent: Tuesday, May 06, 2003 5:02 PM Subject: Re: We have a firewall (was Re: Pakistan government orders ISPservice level agreement)
On 5/6/03 7:51 PM, "E.B. Dreger" <eddy+public+spam@noc.everquick.net>
wrote:
as
--On Tuesday, May 6, 2003 17:25 -0700 Scott Granados <scott@wworks.net> wrote:
Unless you actually call UUnet and your not a customer, God help you then.
I don't know why people keep saying that. I've spoken to UUnet many times on security issues, sometimes as a customer, sometimes not, and every time I've gotten exactly the response I wanted and support above and beyond what I needed. Hell, I even got a lot of support when it was "my friend over here claims to be under attack". Am I lucky, or are others just repeating what they want to hear?
Hi, John. ] Am I lucky, or are others just repeating what they want to hear? Nope, it's not luck. UUNET has always been extremely responsive when I've called them. I've called them countless times, and I think only one time was as an official customer. :) Thanks, Rob. -- Rob Thomas http://www.cymru.com ASSERT(coffee != empty);
On that note, I've heard the responses you get when using an ip phone with that inoc project are excelent as well. ----- Original Message ----- From: "Rob Thomas" <robt@cymru.com> To: "NANOG" <nanog@merit.edu> Sent: Tuesday, May 06, 2003 7:53 PM Subject: Re: We have a firewall (was Re: Pakistan government orders ISPservice level agreement)
## Manager ## ## Customer Router Security Engineering Team ## ## (W)703-886-3823 (C)703-338-7319 ## ####################################################### On Tue, 6 May 2003, Scott Granados wrote:
Well I'll say that I got great help from Chris on the nanog list on more than one occasion even when it wasn't a security issue he pointed me in the proper location. Turned out to not even be a UU problem but they helped beat up on a closer transit provider anyway. However the noc I called to start the process was just awful. Litterally told me "If you wanted help you should have bought servic from us". Actually had me thinking for a while of just buying a ds0 or something from them and giving it to my kids or something for a goof just so I could say Yes, I'm a customer! ----- Original Message ----- From: "John Payne" <john@sackheads.org> To: <nanog@merit.edu> Sent: Tuesday, May 06, 2003 7:43 PM Subject: Re: We have a firewall (was Re: Pakistan government orders ISPservice level agreement)
then.
what
On Tue, 6 May 2003, John Payne wrote:
I've had the exact opposite experience when calling UUnet. I was told in no uncertain terms that they WOULD NOT let me speak with ANYONE if I was not a customer, despite 10s of megabits of DDoS coming through their network to mine. Maybe you called the right people, but UUnet's main NOC line certainly had no interest in helping us. And when our upstream who is a UUnet customer called them, they refused to even perform a backtrace without a subpoena in hand for the results of that backtrace. Tim -- Tim Wilde twilde@dyndns.org Systems Administrator Dynamic DNS Network Services http://www.dyndns.org/
On Wed, 7 May 2003, Tim Wilde wrote:
I've had the exact opposite experience when calling UUnet. I was told in
your upstream, InterNap I believe, called on your behalf, I believe I also spoke directly with you or someone from dyndns... in the particular case I am thinking of, about 2 weeks ago perhaps, we did trace the flood 3 times the same day. This information was provided to your upstream provider. Calling the NOC, as I said before (which you most likely actually called the customer service number which isn't the NOC), is not productive because no one in the NOC (or customer service group) has anyway to authenticate that Tim is Tim from dyndns and not Tim from Savvis... or Tim from UltraDns now trying to social engineer some 'outage' for their good friends at DynDns :( (of course the names used are fictional and the companies are used as a convenience for the example, nothing more)
Yes, your upstream, as I recall, Internap did call and we did help them to the best of our ability, given the attack I recall... I can't remember the specifics and for that I apologize... :(
as I said, for the attack I recall this was not the case. If the attack was perhaps all UDP and not spoofed we don't bother tracing since its not spoofed... perhaps that was the case?
And I think teh bottom line from all this is to use some of the numbers You provided which will help us get better results. I certainly wrote them down:). ----- Original Message ----- From: "Christopher L. Morrow" <chris@UU.NET> To: "Tim Wilde" <twilde@dyndns.org> Cc: "John Payne" <john@sackheads.org>; <nanog@merit.edu> Sent: Tuesday, May 06, 2003 10:37 PM Subject: Re: We have a firewall (was Re: Pakistan government orders ISPservice level agreement)
On Wed, 7 May 2003, Tim Wilde wrote:
I've had the exact opposite experience when calling UUnet. I was told
in
NOC
backtrace
In a message written on Wed, May 07, 2003 at 05:37:18AM +0000, Christopher L. Morrow wrote:
This is not a knock on UUNet specifically, but does get to the real problem. With many large providers it's not that the abuse/security group is unresponsive, it's that you can't figure out how to contact them, and the catch-all published numbers don't work. This is doubly true when the company has gone to an IVR system, almost none of which have the "I'm not a customer but I want to alert you to something that's real important" option. I think all companies that have separated their customer/peer facing support into multiple groups need more training on how to redirect the call to the right group when the wrong group receives it in the first place. Most often the person answering the phone doesn't know the right place to redirect the call, so it appears to just be an unhelpful support system. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
On Wed, 7 May 2003, Leo Bicknell wrote:
There is the issue of what to do with this data also :( And filtering out the 'kook' calls (as the abuse team calls them) from 'real' calls. :( This is a significant nut to crack, in a smaller ISP where 1-5 (or some 'manageable number') does 'all that is important' things are quite different than in a multinational multithousand person company. Also, 'important' takes on different meanings in this scale also.
This is, at UUNET, a continuing education process, as people come/go/reorg the messages get repeated up and down the pike... Sometimes we (me) forget to get my important messages out :( So, for 'security' at UUNET I suppose blame me, mostly.
On Wed, 7 May 2003, Tim Wilde wrote:
I think theres a need to be a bit savvy and to know from experience the best route to talk to someone helpful.. to be fair if every large ISP made access to their expensive NOC resources too simple and too easy to find they'd be bogged down with the less clueful folks. But if you are one of the less clueful folks you can always go via your upstream (I assume you have a decent upstream with someone in the NOC with a bit of know-how!) Steve
On Tue, 6 May 2003, Scott Granados wrote:
Unless you actually call UUnet and your not a customer, God help you then.
The problem is that ALL isp's (large ones atleast) are setup to handle direct customers only. They expect downstreams of downstreams to call the downstream first :( There is authentication information setup and ready to figure out that Scott from Internap is in fact Scott from Internap and not Scott from wworks :( This impedes the process for some situations, like attacks. It also protects the direct customer and the customer's customer from social engineering attacks.
There are other ways to get in touch with me or brian or with other ISP's. In the last few months some outside folks have started getting together some cross provider contact methods. These are making contact much easier for things of this sort. Apparently the Gov't gotten onto the tip that there is little if any interprovider communications :( (Atleast for security) So, the long and the short of it is things are getting better...
<disclaimer> I don't work for UUNET (uumci? mcinet?...), although I know Chris and others who do work there, and have a biased, very good opinion of their skills and approaches </disclaimer> --- Scott Granados <scott@wworks.net> wrote:
Unless you actually call UUnet and your not a customer, God help you then.
Well, I don't have a whole lot of sympathy for this - how many (non-networking) companies will do things which don't benefit their customers on behalf of someone who is not a customer (and shows no sign of becoming one)? I can't think of any offhand, and I don't think that a whole lot would show up in an exhaustive search. The chain should be: if under attack (or whatever) - you call YOUR UPSTREAM. They should call their upstream, etc. Basically, you should call providers with whom you have a relationship, and not call those with whom you don't. If you DO call a provider with whom you don't have a relationship, don't expect the highest quality of service! To me, this is pretty obvious: UUNet's policies are geared to help their customers. Duh...
Here you are dramatically benefiting from the altruism and general nice-guy-ness of Chris, etc.. on the UU Security team. I'm glad that they help non-customer, non-peers, but I'd have to call it going "above and beyond". just my $.0158 (adjusted for inflation) ===== David Barak -fully RFC 1925 compliant- __________________________________ Do you Yahoo!? The New Yahoo! Search - Faster. Easier. Bingo. http://search.yahoo.com
--- Scott Granados <scott@wworks.net> wrote:
Unless you actually call UUnet and your not a customer, God help you then.
* thegameiam@yahoo.com (David Barak) [Wed 07 May 2003, 15:24 CEST]:
I'd have thought having a customer *not* waste all their outgoing bandwidth on useless data such as participating in a DoS attack would make for a happier customer. If you're one of those believers in only your own bottom line, perhaps the liability stick is a good on to wave in your general direction in cases like this? (not stating that you are negligent when advised of DoS attacks in progress, of course) Regards, -- Niels.
On Wed, 7 May 2003, Niels Bakker wrote:
This is, of course, true, and happier customers are a good thing. Unfortunately, there are MANY customers that just don't know that they are the source of someone else's troubles :( Not to mention customers with 'lots' of bandwidth who don't even notice 100mbps of 'extra' traffic :( It sounds whacky, but it is true, sadly. This also only matters if you can pin the traffic down to a far end customer, which is not always the case with spoofed attacks for instance... (from the attackee perspective that is)
Hmm, as with any large carrier (I think) UUNET (mci/ex-wcom/whomever-we-are-for-now but UUNET works for me) will always attempt to do the right thing with respect to the customer being attacked. We do hope that customers ATTACKING folks will do the right thing also and stop the pain on themselves and others. We have on many occasions contacted these folks and requested their help in stopping the pain... If we do trace traffic back we always filter there if possible, why bother transitting the traffic if we are just going to drop it on the far side? The sad reality here is that not all customers are reachable all the time, not all are interested in stopping the traffic, and not all know how to stop the traffic :(
--- Niels Bakker <niels=nanog@bakker.net> wrote:
All I'm saying is that it should be expected that customers receive a much higher quality of service (better response time, etc) than non-customers. I've always been surprised that this is an issue - perhaps network people expect a very high altruism quotient from each other? ===== David Barak -fully RFC 1925 compliant- __________________________________ Do you Yahoo!? The New Yahoo! Search - Faster. Easier. Bingo. http://search.yahoo.com
On Tue, 6 May 2003, Phil Rosenthal wrote:
oh oh... there are quite a few folks who work, including the NOC here, to make Brian look good :) (and me, but mostly brian looks good)
Unfortunately this is the case at times, I will name some good names though, C&W/Qwest/Verio/Sprint/ATT among the larger carriers I've recently dealt with, in the US. Francetelecom and SwissComm atleast external to the US are also quite helpful these days. It seems that atleast all of these folks have been active in stopping many recent attacks. There are some others that don't seem quite as helpful, but that number is getting smaller.
There is a fine balance that has to be struck... killing a provider side router and N customers or degraded service for a single customer who can still filter their side of the link :( sometimes people aren't happy with the response.
On Tue, 6 May 2003, Sean Donelan wrote:
I know we're all entertained by this quite unbelievable situation but is anyone actually helping these folks out? Someone on here must have a link to the Telecoms company concerned and theres plenty folks on here who can help get this cleaned up nice and quick.. ? Perhaps anyone with a contact over there should make themselves known and solicit for a helpful techie (to respond offlist, I dont want flaming for starting a consulting agency here :) Steve
--On 07 May 2003 12:05 +0100 "Stephen J. Wilcox" <steve@telecomplete.co.uk> wrote:
They seem to have issued an ITT if anyone is interested: http://www.ptcl.com.pk/tenders/tender_may_02_2003.html BTW, the firewall may be more than a response to DDOS, it seems that Pakistan doesn't have a fully liberalised telecomms infrastructure yet: http://www.ptcl.com.pk/news/apr_30_b_2003.html -- Rob.
On Mon, 5 May 2003 at 9:31pm Vijay Gill wrote:
Looky here, it is a good enough term for our beloved Bush of Mesopotamia, I don't see why it's not good enough for Mr. Golding. We look to Our President for guidance in these matters...isn't that what Presidents are for? -- Joseph F. Noonan
participants (19)
-
Christopher L. Morrow -
Daniel Golding -
David Barak -
E.B. Dreger -
Eric Gauthier -
Joe Abley -
John Payne -
Joseph Noonan -
Joseph T. Klein -
Leo Bicknell -
Niels Bakker -
Phil Rosenthal -
Rob Pickering -
Rob Thomas -
Scott Granados -
Sean Donelan -
Stephen J. Wilcox -
Tim Wilde -
Vijay Gill