DoS attacks, NSPs unresponsiveness
Hi, This e-mail comes to describe a common problem among a large number of ISPs, mostly foreign, when dealing with US network service providers. I don't want to talk about anyone I don't know of, so I will limit this initial e-mail to talking about UUnet. As most of you know, some ISPs run irc servers, and provide an IRC service to the community. The service is free, and maintenance and cost of networking/hardware/human hours is on the ISPs expense. Irc tends to be a volatile medium, like interpersonal relationships in real life. Thus, many times arguements turn into heated disputes, and sometimes, some people pick up arms, and attack. The attacks usually take out whole ISPs for hours, or days. The problem is that when trying to get help from the upstream provider (UUnet in this example), you either receive a negative answer, or you're just ignored completely. Thus, by terrorism, people get what they want, and hold you at a threat of force, without any ability to defend yourself. Smurfing, icmp attacks, udp attacks, tcp synflooding (spoofed sources) are just a number of these weapons. The problem with alot of networking entities, be it ISPs, enterprises, and such, is that they allow spoofed packets to leave their network (i.e. do not check if the packets originate from within their netblocks before letting them leave their routers). The question is, how can we defend ourselves, and why do the large NSPs turn a blind eye, and act as if it's not their concern ? Is there a chance that by helping one another, and by implementing Internet RFCs corrctly (rfc 1918 for example), we can contribute to the elimination of this kind of electronic terrorism ? Any chance a UUnet person might answer ? best regards, --Ariel -- Ariel Biener e-mail: ariel@post.tau.ac.il Work phone: 03-6406086 fingerprint = 07 D1 E5 3E EF 6D E5 82 0B E9 21 D4 3C 7D 8B BC
On Thu, 2 Nov 2000, Ariel Biener wrote: Hi, One addendum to my mail. I am well aware of how some of you react to the combination of the "I" "R" "C" letters. That is a nice way of avoiding the problem. There are other examples. For example, it seems that lately, Israeli sites are attacked by nameless individuals, taking down whole networks on their way, just because they hate Israel. In the last Ten-155 meeting kast month, I also heard from them, who run the whole european academic internetworking backbone, that UUnet is totally unresponsive to their abuse requests as well. So, I plea you not to discard my e-mail, and the issues it brings up with the usual "irc is bad for you" excuse, just because I gave that example to begin with. --Ariel -- Ariel Biener e-mail: ariel@post.tau.ac.il Work phone: 03-6406086 fingerprint = 07 D1 E5 3E EF 6D E5 82 0B E9 21 D4 3C 7D 8B BC
On Thu, 2 Nov 2000, Ariel Biener wrote:
As most of you know, some ISPs run irc servers, and provide an IRC service to the community. The service is free, and maintenance and cost of networking/hardware/human hours is on the ISPs expense.
This begs to question: Why do they still do it? (Put the targets....er IRC servers on their networks?)
sometimes, some people pick up arms, and attack. The attacks usually take out whole ISPs for hours, or days.
Why do people set their network up as a target? I just don't understand.
The problem is that when trying to get help from the upstream provider (UUnet in this example), you either receive a negative answer, or you're just ignored completely. Thus, by terrorism, people get what they want, and hold you at a threat of force, without any ability to defend yourself.
While I agree that it is unprofessional for your contact at a provider to ignore or be disrespectful of you regarding a DoS against an IRC server, it is just a fact of life that attacks against commercial entities will be treated with much higher priority than attacks against a non-revenue producing "service." Quite frankly, the pizza man comes in WAY above an IRC server in my book.
Smurfing, icmp attacks, udp attacks, tcp synflooding (spoofed sources) are just a number of these weapons. The problem with alot of networking entities, be it ISPs, enterprises, and such, is that they allow spoofed packets to leave their network (i.e. do not check if the packets originate from within their netblocks before letting them leave their routers).
Filtering scales best to ingress vs egress. I agree that filtering should be in place. "Sanity checking" traffic from your downstream customers is a lot smarter than simply hoping they're cluefull enough to block bogons leaving their network though.
The question is, how can we defend ourselves, and why do the large NSPs turn a blind eye, and act as if it's not their concern ?
Quite frankly, unless the source of the attack lives on their network, they bear no responsibility, period, the end. They're providing transit. It's 1's and 0's with no discrimination.
Is there a chance that by helping one another, and by implementing Internet RFCs corrctly (rfc 1918 for example), we can contribute to the elimination of this kind of electronic terrorism ?
RFC1918 specifically addresses filtering routing information. Not spoofed addresses. It states "routing information about private networks shall not be propagated on inter-enterprise links, and packets with private source or destination addresses should not be forwarded across such links." Notice the placement of "shall" and "should." I'm not saying that you don't have a valid point. Just that the RFC doesn't specifically prohibit forwarding the packets. Only routing information about RFC1918 address space. Now, in specific response to your question about eliminating electronic terrorism, it is doubtful. Doubtful that you'll ever: #1 spread enough clue around. #2 get everyone to cooperate. --- John Fraizer EnterZone, Inc
On Wed, 1 Nov 2000, John Fraizer wrote:
While I agree that it is unprofessional for your contact at a provider to ignore or be disrespectful of you regarding a DoS against an IRC server, it is just a fact of life that attacks against commercial entities will be treated with much higher priority than attacks against a non-revenue producing "service."Quite frankly, the pizza man comes in WAY above an IRC server in my book.
That is not true, since the attacks take down the whole ISP, and this is a commercial damage to the said ISP, which is counted in $$.
Quite frankly, unless the source of the attack lives on their network, they bear no responsibility, period, the end.They're providing transit.It's 1's and 0's with no discrimination.
Yes, but it seems others (AT&T for example) do not take the above point of view, and actually do provide a real service, and help track attackers that use their network. Even without filtering, backtracing an attack to it's source, with responsive abuse teams, can take about 10 hours, counting all the networking entities in the path of the attack. If said NSP would have been responsive, they'd see where the attack is coming from, and either conact the next in-line network entity on the path of the attack, or at least supply the details to the attacked ISP.
RFC1918 specifically addresses filtering routing information.Not spoofed addresses.It states "routing information about private networks shall not be propagated on inter-enterprise links, and packets with private source or destination addresses should not be forwarded across such links."Notice the placement of "shall" and "should."
Yes. But while it says should, it's obviously a good idea to implement. About spoofing, Cisco (and others) have simple cook book solutions of a few lines on your router's conf, that stop that. It just takes some selfless acts, and caring about others, to do that when you configure your own edge routers, being a good Netizen.
Now, in specific response to your question about eliminating electronic terrorism,it is doubtful.Doubtful that you'll ever: #1 spread enough clue around. #2 get everyone to cooperate.
Well, about #2, that is what I am hoping to achieve. Even if not EVERYONE will cooperate, then enough to have the big players in the game, to get something going. Otherwise, what you're saying, YES, terrorism is here to stay, BUT, we don't give a damn. Now, judging at what you see in the world today, while not ALL countries cooperate on terrorism issues (real life terror - bombs and stuff), there are a large number of countries that do, and this helps make it much harder on extremists to compromise your security. I guess you do like not having to look behind your shoulder when you go to work, right ? :) --Ariel
--- John Fraizer EnterZone, Inc
-- Ariel Biener e-mail: ariel@post.tau.ac.il Work phone: 03-6406086 fingerprint = 07 D1 E5 3E EF 6D E5 82 0B E9 21 D4 3C 7D 8B BC
On Wed, 1 Nov 2000, John Fraizer wrote:
On Thu, 2 Nov 2000, Ariel Biener wrote:
As most of you know, some ISPs run irc servers, and provide an IRC service to the community. The service is free, and maintenance and cost of networking/hardware/human hours is on the ISPs expense.
This begs to question: Why do they still do it? (Put the targets....er IRC servers on their networks?)
Where should they put them? You also have to take into account, a large portion of crippling attacks aren't directed at the servers themselves, but at the actual users. This makes the problem a bit larger than just one of those that should only concern owners of IRC servers.
The problem is that when trying to get help from the upstream provider (UUnet in this example), you either receive a negative answer, or you're just ignored completely. Thus, by terrorism, people get what they want, and hold you at a threat of force, without any ability to defend yourself.
While I agree that it is unprofessional for your contact at a provider to ignore or be disrespectful of you regarding a DoS against an IRC server, it is just a fact of life that attacks against commercial entities will be treated with much higher priority than attacks against a non-revenue producing "service." Quite frankly, the pizza man comes in WAY above an IRC server in my book.
That's the biggest fscking cop-out I've seen recently. Is profit or profit-capability the new litmus test for whether or not attacks should be dealt with or rectified? Let's take efnet as an example. Most of the efnet servers are hosted by commercial entities. Does it matter less that the attack is against their IRC server than it would if it were against their web server? Both are certainly a theft of resources, and you can certainly put a price on bandwidth consumption. IRC servers are a benefit of services. If they weren't, then AOL wouldn't have one. Don't even remind me that aol.com is currently k-lined off most of efnet, as well as most of the other IRC networks. I'd love to see someone take this type of case to civil court. I'd be very interested to see how hard it would be to prove negligence, though I could imagine it could get complicated if it's a matter of international law.
Smurfing, icmp attacks, udp attacks, tcp synflooding (spoofed sources) are just a number of these weapons. The problem with alot of networking entities, be it ISPs, enterprises, and such, is that they allow spoofed packets to leave their network (i.e. do not check if the packets originate from within their netblocks before letting them leave their routers).
Filtering scales best to ingress vs egress. I agree that filtering should be in place. "Sanity checking" traffic from your downstream customers is a lot smarter than simply hoping they're cluefull enough to block bogons leaving their network though.
How many ISP/NSP's do ingress filtering beyond anything other than routing announcements from their customers/peers and the standard RFC1918 space and Martian filters? I bet you it's a lot less than those who don't, considering the amount of clue available to the internet connected companies. Furthermore, if you don't specifically ingress filter your customers, who's to say they don't just dump traffic on you? I've never tried it personally, but it seems possible. Really, large NSP's which should be clue enabled should be the least of our worries. How hard would it be to have a router automatically filter IP packets heading out an interface if they aren't being advertised via a routing protocol across that interface or statically routed to the connected interface? I'm trying to think of an instance where it's desired to not announce a network, but still forward traffic for it, and I haven't come up with one yet unless there are specific reasons why asynchronous routing is desired. It certainly would seem more secure than the way routers handle traffic now.
The question is, how can we defend ourselves, and why do the large NSPs turn a blind eye, and act as if it's not their concern ?
Quite frankly, unless the source of the attack lives on their network, they bear no responsibility, period, the end. They're providing transit. It's 1's and 0's with no discrimination.
Please clarify what you mean by 'their network.' It seems that you're suggesting that if an NSP's customers are spoofing traffic it's not the NSP's responsibility to do any sort of validation or filtering, since it's transit of "1's and 0's with no discrimination." That goes against what you said earlier about ingress filtering, but it's what I'm gathering in the context of that paragraph. It's late, so I may be misunderstanding what you're trying to say. If you are just talking about Company A's upstream provider not helping them during an attack, I think it should definitely be a concern if the victim is a metered service customer. How hard is it for an NSP to route attack traffic to Null0 on the router they connect to? I'd certailny stay away from any NSP that chose to ignore me while I'm under attack, but I find myself auditing large networks these days instead of running them, so that's probably a non-issue.
Now, in specific response to your question about eliminating electronic terrorism, it is doubtful. Doubtful that you'll ever: #1 spread enough clue around. #2 get everyone to cooperate.
The real problem is that the IPv4 protocol suite as a whole is really, really, really insecure in a lot of ways, and has outlived it's usefulness. Even with the security features of IPv6 retrofitted to it, IPv4 still has lots of issues. It amazes me that companies are actually relying on public IP services for critical business systems and revenue generation. If we had packet accountability from end to end then most, if not all of the network-based DoS attacks of the last 5 years would have been a non-issue. And tracking down the DDoS kiddies would have been relatively simple and easy. I'm all for privacy and a certain amount of anonymity, but total anonymity seems to turn normal people into crazed morons who will do things they would not think of doing in the physical world. -- Joseph W. Shaw Sr. Network Security Specialist for Big Company not to be named because I don't speak for them here. I have public opinions, and they don't.
On Thu, 2 Nov 2000, Joe Shaw wrote:
On Wed, 1 Nov 2000, John Fraizer wrote:
Where should they put them? You also have to take into account, a large
IMHO, in a museum.
portion of crippling attacks aren't directed at the servers themselves, but at the actual users. This makes the problem a bit larger than just one of those that should only concern owners of IRC servers.
OK. I will grant you that there are attacks aimed at individuals. And I will also grant you that some of these script kiddies like to "kill a fly with a hammer" IE; 100Mb/s attack launched against some poor schmuck on a 28.8 modem. The script kiddies aren't terribly stupid though. They don't generally continue to attack once their objective has been achieved. To do so increases the chances that the source of the attack will be identified. One of the keys to winning a war is to choose your battles wisely and attempt to limit casualties in the battles you do fight.. Don't throw 1Gb/s of capacity to a server that is only going to use 20Mb/s but is highly likely to attract 600Mb/s of "hate" from the script kiddies. Go find someone with a legacy /24 they're not using (there are TONS of them) and convince them to sell it to you. Put the "target" on that /24. If you're under attack, retract the announcement. Now, the "hate" stays on the originating network. I know you're probably going to say "You expect them to disconnect just because some punk downloaded the latest attack script?" I'll answer that with a question for you. Some idiot is shooting a shotgun at you and your family. Do you just stand there hoping that the police will arrest the idiot for violating the law or do you try to limit your exposure to this attack?
While I agree that it is unprofessional for your contact at a provider to ignore or be disrespectful of you regarding a DoS against an IRC server, it is just a fact of life that attacks against commercial entities will be treated with much higher priority than attacks against a non-revenue producing "service." Quite frankly, the pizza man comes in WAY above an IRC server in my book.
That's the biggest fscking cop-out I've seen recently. Is profit or profit-capability the new litmus test for whether or not attacks should be dealt with or rectified? Let's take efnet as an example. Most of the efnet servers are hosted by commercial entities. Does it matter less that the attack is against their IRC server than it would if it were against their web server? Both are certainly a theft of resources, and you can certainly put a price on bandwidth consumption.
Sir, if you're going to curse at me, at least spell the foul language correctly. As for my opinion being a cop-out, I suppose you're entitled to your opinion. I'll take your EFnet example. Is it a responsible action for these "commercial entities" to exponentially increase the chances of being attacked by hosting an EFnet server? As you have stated, these attacks often cripple their entire network. Is this benefitial to their stockholders? How about to their customers? You see, the vast majority of us are in business to make money. When something that does NOT generate revenue impacts your revenue generating services, it's not a good thing. If running an IRC server was a multi-billion dollar revenue producing venture, you could argue "the lesser of two evils." It is doubtful that you will ever convince me that the PROs of running an IRC service outweigh the CONs.
IRC servers are a benefit of services. If they weren't, then AOL wouldn't have one. Don't even remind me that aol.com is currently k-lined off most of efnet, as well as most of the other IRC networks.
You picked the wrong company to cite to try to convince me that running an IRC server is wise business practice. I really appreciate all of the "replacement DVD cases" they send me. I enjoy the laughter they bring me when there is a power outage at their Columbus POP. It's cool to see row after row after row of cabinets filles with Total Controls go DARK because they aren't on UPS or 48V DC. You answered your own question as to why they have an IRC server.
I'd love to see someone take this type of case to civil court. I'd be very interested to see how hard it would be to prove negligence, though I could imagine it could get complicated if it's a matter of international law.
Give me a break. "Yes your honor. We incurred $200K of extra bandwidth costs as a result of this attack. What? Why didn't we retract the network announcement and try to limit our liability? Duh.... That never occured to us! Um, yes. This is the third time this year that this has happened to us. Why do we still host a server that only generates revenue for the entity who sells us bandwidth? We think it's cool to be able to say that we host one. I suppose it is stupid now that we think about it. Is it possible for us to plead to a lesser charge? Felony Stupidity carries such stiff minimum sentences!"
How many ISP/NSP's do ingress filtering beyond anything other than routing announcements from their customers/peers and the standard RFC1918 space and Martian filters? I bet you it's a lot less than those who don't,
Um, lets see. If they're filtering out martians and bogons, and only accepting traffic from their customers assigned space, what seems to be the problem?
customers, who's to say they don't just dump traffic on you? I've never tried it personally, but it seems possible. Really, large NSP's which should be clue enabled should be the least of our worries.
Clueless ISPs/NSPs will continue to be abused in this manner. As for lerge NSPs being the least of your worries, I disagree. Sure, they "should" be clue enabled. The last time I checked, noone was enforcing the "minumum" clue standard though and there are pleanty of entities with big pipe and little clue.
How hard would it be to have a router automatically filter IP packets heading out an interface if they aren't being advertised via a routing protocol across that interface or statically routed to the connected interface? I'm trying to think of an instance where it's desired to not announce a network, but still forward traffic for it, and I haven't come up with one yet unless there are specific reasons why asynchronous routing is desired. It certainly would seem more secure than the way routers handle traffic now.
It is possible to do this now. As for asymetric (I believe that's the word you were grasping for. Don't worry about it. My wife would have made the same mistake.) being routing being desired, in many cased, it IS desired. There are also cases where you are providing transit to a customer who, for whatever reason, is NOT announcing routes to you.
Quite frankly, unless the source of the attack lives on their network, they bear no responsibility, period, the end. They're providing transit. It's 1's and 0's with no discrimination.
Please clarify what you mean by 'their network.' It seems that you're suggesting that if an NSP's customers are spoofing traffic it's not the NSP's responsibility to do any sort of validation or filtering, since it's transit of "1's and 0's with no discrimination." That goes against what you said earlier about ingress filtering, but it's what I'm gathering in the context of that paragraph. It's late, so I may be misunderstanding what you're trying to say.
As a NSP, tier 1, whatever you want to call it, many of us sell transit to other transit providers. Filtering based on source/dest IP address does not scale well for either router CPU, config size, or management. You filter end users, not the middlemen. IE; tier 2's filter tier 3's, etc based on source address. Don't get me wrong. We do prefix-list and as-path filter our BGP speaking customers and set up source address filters on static customers border routers.
If you are just talking about Company A's upstream provider not helping them during an attack, I think it should definitely be a concern if the victim is a metered service customer. How hard is it for an NSP to route attack traffic to Null0 on the router they connect to? I'd certailny stay away from any NSP that chose to ignore me while I'm under attack, but I find myself auditing large networks these days instead of running them, so that's probably a non-issue.
OK. So, you expect me to accept and possibly pay for traffic that is destined for your network but, instead of passing it to you and billing you for it, I should route it to Null0 and just eat the cost? Sounds like a great deal for you. Not so good a deal for me.
The real problem is that the IPv4 protocol suite as a whole is really, really, really insecure in a lot of ways, and has outlived it's usefulness. Even with the security features of IPv6 retrofitted to it, IPv4 still has lots of issues. It amazes me that companies are actually relying on public IP services for critical business systems and revenue generation. If we had packet accountability from end to end then most, if
I don't agree that v4 has outlived it's usefullness. v6 has nice features and we do speak v6. v4 still has a lot of kick left in it though.
not all of the network-based DoS attacks of the last 5 years would have been a non-issue. And tracking down the DDoS kiddies would have been relatively simple and easy. I'm all for privacy and a certain amount of
Is it just me or is it not pretty simple to set up monitoring on your network that would squeal real loud when your bandwidth usage goes outside of it's expected "profile?" We do it. It works very well. When the source of the traffic knows immediately about it, the nasty traffic goes away quicker.
anonymity, but total anonymity seems to turn normal people into crazed morons who will do things they would not think of doing in the physical world.
I think that's human nature.
-- Joseph W. Shaw Sr. Network Security Specialist for Big Company not to be named because I don't speak for them here. I have public opinions, and they don't.
--- John Fraizer Head Nerd in Charge EnterZone, Inc And I *do* speak for my company.
John, Regretfully, I must say, portraying a world where there are absolutely no values except money is a rather morbid picture. While I agree that the companies involved with the Internet want to earn money, they also want to provide a reliable service to their customers, and also entertainment, and alot of other stuff that brings people closer together. Chosing a completely economical view and completely disregarding any sociologic features in this global spanning network is, as I see it, a mistake. We all want to make money, but not at all costs, and we also have some values. I know NSPs who take special pride in their professionalism, and their ability to solve problems, and also to be curtious and helpful towards their customers. Those are not the largest NSPs (maybe one or two). Keeping such staff is expensive, and surely not very smart, economically speaking. Their clients are very happy, but still, the NSP doesn't become a HUGE multi-billion dollar company. Some of us take pride in what we do, and try to do it the best we can, and usually, past working hours, and sometimes at our own time's expense. With your purely economical view, this would be ... stupid, but, yet, we do it. No need to lose all that makes one human because of the shine of a silver dollar. --Ariel -- Ariel Biener e-mail: ariel@post.tau.ac.il Work phone: 03-6406086 fingerprint = 07 D1 E5 3E EF 6D E5 82 0B E9 21 D4 3C 7D 8B BC
On Thu, 2 Nov 2000, John Fraizer wrote:
One of the keys to winning a war is to choose your battles wisely and attempt to limit casualties in the battles you do fight.. Don't throw 1Gb/s of capacity to a server that is only going to use 20Mb/s but is highly likely to attract 600Mb/s of "hate" from the script kiddies.
My war is to increase Internet security. It's generally impossible with the current implementation, and I'm not exactly sure how much better IPv6 is going to be if we ever get around to deploying it Internet-wide.
Go find someone with a legacy /24 they're not using (there are TONS of them) and convince them to sell it to you. Put the "target" on that /24. If you're under attack, retract the announcement. Now, the "hate" stays on the originating network.
Actually, that's not a bad idea in and of itself if you have the ability to do so. But people generally filter at /19 or /20 advertisements, and what happens when it's more than just some moron taking down an IRC server? What happens when it's a customer doing business that someone has decided to take issue with? What happens when it's a political web site being attacked by someone who does not agree with that point of view? What happens when it's some bored kid who's rooted a couple hundred Solaris or Linux boxes and is using them to take businesses off the net for no particular reason. Retracting routes is a band-aid solution, and in some instances it's certanly workable. But you're entirely fixated on IRC and 'making yourself a target,' which still doesn't fix the problem as a whole. More importantly, having to remove advertised routes for the service effectively accomplishes what the DoS set out to do. The solution is making it virtually impossible for the offensive acts to be carried out.
Some idiot is shooting a shotgun at you and your family. Do you just stand there hoping that the police will arrest the idiot for violating the law or do you try to limit your exposure to this attack?
This is a poor analogy. In Texas, I'm able to carry a concealed weapon because I've got no criminal history, can pass a test that shows I know the state laws concerning concealed carry and gun laws, and can pass another test that shows I can shoot accurately. Furthermore, in your analogy, I would be quite aware of who was attacking me, it would be matter of life and death and not network resources or money, and there would be certain steps I could take to actively defend myself, my family, and anyone else who might be around if the police were not there to handle the situation because of the information in the previous paragraph. I'm not advocating net vigilantism, or vigilantism of any kind for that matter.
Sir, if you're going to curse at me, at least spell the foul language correctly.
Sigh... I can spell. It was intentional. I trust no 'Head Geek' that does not get the reference.
As for my opinion being a cop-out, I suppose you're entitled to your opinion. I'll take your EFnet example. Is it a responsible action for these "commercial entities" to exponentially increase the chances of being attacked by hosting an EFnet server? As you have stated, these attacks often cripple their entire network. Is this benefitial to their stockholders? How about to their customers?
Any Internet connected server can be subjected to DoS attacks. Think E-bay, buy.com, and all the others that were taken down this last February by the new DDoS attacks. Most of those businesses were solely based upon web traffic. I'll admit your argument holds water if we're only talking about IRC servers. But it stops making sense when you refer to any other service.
You see, the vast majority of us are in business to make money. When something that does NOT generate revenue impacts your revenue generating services, it's not a good thing. If running an IRC server was a multi-billion dollar revenue producing venture, you could argue "the lesser of two evils." It is doubtful that you will ever convince me that the PROs of running an IRC service outweigh the CONs.
You don't have to explain money to me. I work for a multi-billion dollar corporation in the top half of the Fortune 50. The bottom line is generally always their primary concern, and for good reason. But, they also take the security of their network and it's resources very seriously. Even with no Return On Investment, they're still willing to spend large sums on security software and hardware.
IRC servers are a benefit of services. If they weren't, then AOL wouldn't have one. Don't even remind me that aol.com is currently k-lined off most of efnet, as well as most of the other IRC networks.
You picked the wrong company to cite to try to convince me that running an IRC server is wise business practice. I really appreciate all of the "replacement DVD cases" they send me.
Me too, actually. If only I could gracefully transition those pesky cardboard cases into them. However, this has absolutely nothing to do with the topic of this discussion.
I'd love to see someone take this type of case to civil court. I'd be very interested to see how hard it would be to prove negligence, though I could imagine it could get complicated if it's a matter of international law.
Give me a break. "Yes your honor. We incurred $200K of extra bandwidth costs as a result of this attack. What? Why didn't we retract the network announcement and try to limit our liability?
That would never be asked in a US court of law by a judge, regardless of the fact if it were a mail, web, or any other type of server. Here is why. "Your { web | news | SMTP | POP3 | <INSERT SERVICE HERE>} server was under attack? Why didn't you take it down?" "Because doing so would be a more effective and pervasive denial of service attack than the actual attack they were performing by flooding."
Duh.... That never occured to us! Um, yes. This is the third time this year that this has happened to us. Why do we still host a server that only generates revenue for the entity who sells us bandwidth? We think it's cool to be able to say that we host one. I suppose it is stupid now that we think about it. Is it possible for us to plead to a lesser charge? Felony Stupidity carries such stiff minimum sentences!"
Why is it more acceptable if they attack an IRC server than if they attack a webserver of a web retailer? So far, your only reasoning has been solely based around money. That's sad, because if it's only money that's driving the ".com economy," then we're all going to be screwed when it's not in anyone's best interest to deal with any other entity unless there's some monetary gain or loss involved.
How many ISP/NSP's do ingress filtering beyond anything other than routing announcements from their customers/peers and the standard RFC1918 space and Martian filters? I bet you it's a lot less than those who don't,
Um, lets see. If they're filtering out martians and bogons, and only accepting traffic from their customers assigned space, what seems to be the problem?
There isn't one, except the last time I looked, the classic martian filters had nothing to do with customer space. They were strictly blocking 'odd' things to from RFC1918 space, packets with a source or destination of 255.255.255.255 or 0.0.0.0, 127.0.0.1 of improperly configured UNIX hosts, etc. How many of them are actually doing more than this was the question. Furthermore, how many of them are doing it at all? The big guys certainly don't seem to be interested in answering, even though we know they are on the list.
Clueless ISPs/NSPs will continue to be abused in this manner. As for lerge NSPs being the least of your worries, I disagree. Sure, they "should" be clue enabled. The last time I checked, noone was enforcing the "minumum" clue standard though and there are pleanty of entities with big pipe and little clue.
And eventually, someone will take this matter to court. Then we'll see how things play out and if the clue disabled can be forced to act properly if negligence can be proven.
It is possible to do this now. As for asymetric (I believe that's the word you were grasping for. Don't worry about it. My wife would have made the same mistake.)
You are correct, though I wonder what your wife has to do with this. My apologies for using the incorrect term. My intention was not to be vague or misleading, but it was late.
routing being desired, in many cased, it IS desired.
I'm looking for actual examples. If you have some, I'd love to here them. There has only been one time in the past where I actually wanted asymetrical routing, and it certainly took some work to make traffic flow that way. I'm not saying don't allow it to happen, just make it the default not to allow traffic you're not specifically routing.
There are also cases where you are providing transit to a customer who, for whatever reason, is NOT announcing routes to you.
How can you possibly have transit customers who you are not announcing any type of routes for? Has the meaning of a transit network, which transit customers generally buy access to for connectivity, changed? Transit networks used to mean networks used to transit traffic between two other different networks, generally a Tier 2 ISP/NSP. Is this not the case anymore? This would include things like address space that you are advertising as part of a larger aggregate.
OK. So, you expect me to accept and possibly pay for traffic that is destined for your network but, instead of passing it to you and billing you for it, I should route it to Null0 and just eat the cost? Sounds like a great deal for you. Not so good a deal for me.
If you're a Tier 1, and bandwidth costs you little or nothing, then I certainly do. In your case, I'd expect you to do the same thing with your upstream Tier-1 providers if it were causing a problem with you. In my years or working with various upstreams, some of whom have been swallowed by the Worldcom communications giant or sold off so that Worldcom could buy other parts of the companies, no Tier1 provider I've bought access from has ever had a problem with not charging me for flood traffic. Maybe as a Tier 2 your budget is a bit tighter and your bottom line a bit lower, but unless your upstream is charging you for the bandwidth, and in my experience with what used to be the big 3 (UUNet, MCI, and Sprint) as well as C&W and Savvis, they never did when it was obvious it was a DoS attack, then you shouldn't be charge and you shouldn't charge the victim.
I don't agree that v4 has outlived it's usefullness. v6 has nice features and we do speak v6. v4 still has a lot of kick left in it though.
Opinions do vary. While I consider v4 to be functional, I do not consider it adequate, especially when it comes to security. It was not designed with security/accountability in mind. It was designed to be functional first. If it were secure, we wouldn't have many of the problems we're facing today. There are certainly pockets of v6 speakers tunneling over v4, but until v6 is widely deployed and spoken by everyone, it's really not even a concern. But, seeing how most people can't even take basic security measures, moving to an entirely new version of IP could be asking too much.
Is it just me or is it not pretty simple to set up monitoring on your network that would squeal real loud when your bandwidth usage goes outside of it's expected "profile?" We do it. It works very well. When the source of the traffic knows immediately about it, the nasty traffic goes away quicker.
It's just as easy to protect against letting DoS/spoofing attacks occur. We rate-limit ICMP and certain types of UDP, enable TCP-intercept, utilize process scheduling, apply both egress and ingress compiled ACL's, disable TCP and UDP small-servers, turn off source routing, disable IP directed broadcasts, and other proven methods to pro-actively keep bad things from happening. You know, an ounce of prevention is worth a pound of cure and all that stuff. All this, on top of 24x7x365 manned monitoring of the routers, as well as firewalls and strategically placed network IDS devices. Our abuse notifications are even answered by real people in times of minutes or hours, not days, which so far has been better than any other organization I've had to deal with in hunting down abuse from other networks. If a security notification goes more than a few hours before we investigate it, then managment starts getting notified that we're not doing our job. So far, that's never happened. What's it going to take to get people to cooperate in an effective manner? Even with what some might perceive as rancor or disagreement between you and I, I would certainly handle any security issues from my network with the same urgency as I do with any of the other people or entities I deal with, becuase it's the right thing to do as a responsible Internet entity. You certanly won't get the "Please learn how to read your BlackIce firewall output and understand that 5 UDP packets from a server does not constitute an attack, neither does traceroute" letter, unless that's the information you send me. -- Joseph W. Shaw Sr. Network Security Specialist for Big Company not to be named because I don't speak for them here. I have public opinions, and they don't.
On Thu, 2 Nov 2000, Joe Shaw wrote:
On Thu, 2 Nov 2000, John Fraizer wrote:
One of the keys to winning a war is to choose your battles wisely and attempt to limit casualties in the battles you do fight.. Don't throw 1Gb/s of capacity to a server that is only going to use 20Mb/s but is highly likely to attract 600Mb/s of "hate" from the script kiddies.
My war is to increase Internet security. It's generally impossible with the current implementation, and I'm not exactly sure how much better IPv6 is going to be if we ever get around to deploying it Internet-wide.
You can ALWAYS increase security. You may not be able to do it the way you want but, you can always do it in some shape or form.
Go find someone with a legacy /24 they're not using (there are TONS of them) and convince them to sell it to you. Put the "target" on that /24. If you're under attack, retract the announcement. Now, the "hate" stays on the originating network.
Actually, that's not a bad idea in and of itself if you have the ability to do so. But people generally filter at /19 or /20 advertisements, and what happens when it's more than just some moron taking down an IRC server? What happens when it's a customer doing
If they're filtering on the /19 or /20 boundry on legacy space, they're VERY misconfigured and breaking a whole bunch of connectivity. The rest of your paragraph was full of what-ifs. There is a solution for every problem. It is not always painless and sometimes involves shooting some moron who IS the problem square between the eyes. There IS a solution to every problem though. I have provided two solutions to attacks on IRC servers. #1, don't run one -- IE; limit your worthiness as a target. #2, give yourself the ability to "dissappear" as is outlined above. #3 - #10,000 I am reserving for paying customers.
I'm looking for actual examples. If you have some, I'd love to here them. There has only been one time in the past where I actually wanted asymetrical routing, and it certainly took some work to make traffic flow that way. I'm not saying don't allow it to happen, just make it the default not to allow traffic you're not specifically routing.
There are also cases where you are providing transit to a customer who, for whatever reason, is NOT announcing routes to you.
How can you possibly have transit customers who you are not announcing any type of routes for? Has the meaning of a transit network, which transit customers generally buy access to for connectivity, changed? Transit networks used to mean networks used to transit traffic between two
OK. Try this one on. You're announcing 89K prefixes to customer X. They're seeing the same 89K prefixes from another provider too. They don't want ANY incoming traffic via their connection from you. They do however preference routes to ASX, ASY and ASZ via your connection. What's the best way for them to do this? Don't announce to you and route-map those routes to X Y and Z to be preferred. Asymetric routing. --- John Fraizer EnterZone, Inc
On Fri, 3 Nov 2000, John Fraizer wrote:
You can ALWAYS increase security. You may not be able to do it the way you want but, you can always do it in some shape or form.
I wholeheartedly agree. And you have to balance functionality against security, but I believe a happy compromise is generally available.
If they're filtering on the /19 or /20 boundry on legacy space, they're VERY misconfigured and breaking a whole bunch of connectivity.
I thought filtering at /19-/20 space was still considered best common practice by some of the older Tier 1's who were interested in keeping the routing tables small. Maybe with less legacy equipment deployed this isn't an issue anymore.
OK. Try this one on. You're announcing 89K prefixes to customer X. They're seeing the same 89K prefixes from another provider too. They don't want ANY incoming traffic via their connection from you. They do however preference routes to ASX, ASY and ASZ via your connection. What's the best way for them to do this? Don't announce to you and route-map those routes to X Y and Z to be preferred.
Wouldn't it be better, at least from an engineering standpoint, to still announce their routes with AS padding to increase the AS-path so in the event their other connection(s) goes down they still have some type of inbound connectivity? It seems like your example would work in a best case scenario, but customer X would drop off of the planet in the event of a partial outage without some manual reconfiguration. I did something similar to what you are suggesting, but we still announced the routes, with padding, so that in the event of a failure the network could still function. The link did fail eventually (would you believe me if I mentioned there was a backhoe and a contractor involved?), and while the network was certainly slower than normal, it continued to function adequately so that there was no perceivable outage seen by our customers. -- Joseph W. Shaw Sr. Network Security Specialist for Big Company not to be named because I don't speak for them here. I have public opinions, and they don't.
Joe Shaw wrote:
Wouldn't it be better, at least from an engineering standpoint, to still announce their routes with AS padding to increase the AS-path so in the event their other connection(s) goes down they still have some type of inbound connectivity? It seems like your example would work in a best case scenario, but customer X would drop off of the planet in the event of a partial outage without some manual reconfiguration. I did something similar to what you are suggesting, but we still announced the routes, with padding, so that in the event of a failure the network could still function. The link did fail eventually (would you believe me if I mentioned there was a backhoe and a contractor involved?), and while the network was certainly slower than normal, it continued to function adequately so that there was no perceivable outage seen by our customers.
In theory, yes, but in practice, I've found that no matter how much you use AS prepending on routing announcements for a specific link, there is some minimum amount of traffic that you will always receive through it, and that amount may be more than you're willing to accept for anything other than an emergency situation. The length of the AS path is not the first thing that a router looks at when deciding which path offers the best route. I've used certain circuits as "manually operated backups" in the past because the minimum amount of traffic I'd have pulled in on that pipe was larger than the pipe itself. During an emergency, though, slow and bad connectivity is better than no connectivity at all. (In case of contractor, break glass.) My BGP wish list includes some better way of selecting a path, which gives the initial advertiser more control over inbound traffic. Mark
On Fri, 3 Nov 2000, Joe Shaw wrote:
If they're filtering on the /19 or /20 boundry on legacy space, they're VERY misconfigured and breaking a whole bunch of connectivity.
I thought filtering at /19-/20 space was still considered best common practice by some of the older Tier 1's who were interested in keeping the routing tables small. Maybe with less legacy equipment deployed this isn't an issue anymore.
I don't think it's a matter of legacy equipment and many networks do filter on the /19-/20 boundry for address space that is assigned as shorter than /21. I don't know of anyone who is _purposely_ filtering out legacy /24's though. By legacy /24's, I am referring to address space that was allocated by the RIR as /24's. The chances of those /24's being aggregated into /20 or shorter prefixes are pretty slim.
Wouldn't it be better, at least from an engineering standpoint, to still announce their routes with AS padding to increase the AS-path so in the event their other connection(s) goes down they still have some type of inbound connectivity? It seems like your example would work in a best case scenario, but customer X would drop off of the planet in the event of a partial outage without some manual reconfiguration. I did something similar to what you are suggesting, but we still announced the routes, with padding, so that in the event of a failure the network could still function. The link did fail eventually (would you believe me if I mentioned there was a backhoe and a contractor involved?), and while the network was certainly slower than normal, it continued to function adequately so that there was no perceivable outage seen by our customers.
I agree that it would be better. I know of several instances where people are doing it as I described though, for whatever reasons. --- John Fraizer EnterZone, Inc
On Thu, 2 Nov 2000, Joe Shaw wrote:
routing being desired, in many cased, it IS desired.
I'm looking for actual examples. If you have some, I'd love to here them. There has only been one time in the past where I actually wanted asymetrical routing, and it certainly took some work to make traffic flow that way. I'm not saying don't allow it to happen, just make it the default not to allow traffic you're not specifically routing.
90-something percent of our network is asymetrical routing, we provide satellite bandwidth to around 100 ISPs in Australia and New Zealand (and our own direct customers). A small ISP might have a 128k leased wire to a local telco, the buy a dish and a computer with a PCI card and we can provide them with a few hundred kb/s of bandwith. The network is advertised at our US uplink site, transmitted and received by them, outgoing packets go via their local telco. They may advertise their network only for Australian traffic or even not at all. You can even do the same for home customers. There are at least 3 other largish providers doing the same in the Australian market, perhaps totalling 20-40% (rough guess) of total bandwidth into the country. When costs of circuits (the charge change per megabyte of traffic on those circuits) are high you have to do various tricks to keep expenses to a minimum. It's not like in the US where you have a dozen providers you can just order random OCn's from. -- Simon Lyall. | Newsmaster | Work: simon.lyall@ihug.co.nz Senior Network/System Admin | | Home: simon@darkmere.gen.nz ihug, Auckland, NZ | Asst Doorman | Web: http://www.darkmere.gen.nz
John Fraizer wrote:
Is there a chance that by helping one another, and by implementing Internet RFCs corrctly (rfc 1918 for example), we can contribute to the elimination of this kind of electronic terrorism ?
RFC1918 specifically addresses filtering routing information. Not spoofed addresses. It states "routing information about private networks shall not be propagated on inter-enterprise links, and packets with private source or destination addresses should not be forwarded across such links." Notice the placement of "shall" and "should."
Although 1918 was given only as an example, substituting the number 1918 for 2827 is a common mistake. RFC 2827 addresses spoofing and is a BCP. You can't argue that widespread implementation of RFC 2827's concepts wouldn't benefit the Internet.
Now, in specific response to your question about eliminating electronic terrorism, it is doubtful. Doubtful that you'll ever: #1 spread enough clue around. #2 get everyone to cooperate.
This can't go on forever. I'd like to spread the clue about ingress filtering, and am willing to commit time to the cause. Is anyone with me? Mark
I fully agree with Mark, These simple things in themselves will not stop some people from doing certain things, but it will prevent the typical 'script kiddies' and it will make the Internet a better place for all, Gone are the days when we can sit back and say 'not my problem' The main reason the Internet will carry on growing is to do with QoS, and this, although very small is the first step in making it work... just my 2p worth Richard Smith Firstnet Leeds -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Mark Mentovai Sent: 02 November 2000 14:59 To: nanog@merit.edu Subject: Re: DoS attacks, NSPs unresponsiveness John Fraizer wrote:
Is there a chance that by helping one another, and by implementing Internet RFCs corrctly (rfc 1918 for example), we can contribute to the elimination of this kind of electronic terrorism ?
RFC1918 specifically addresses filtering routing information. Not spoofed addresses. It states "routing information about private networks shall not be propagated on inter-enterprise links, and packets with private source or destination addresses should not be forwarded across such links." Notice the placement of "shall" and "should."
Although 1918 was given only as an example, substituting the number 1918 for 2827 is a common mistake. RFC 2827 addresses spoofing and is a BCP. You can't argue that widespread implementation of RFC 2827's concepts wouldn't benefit the Internet.
Now, in specific response to your question about eliminating electronic terrorism, it is doubtful. Doubtful that you'll ever: #1 spread enough clue around. #2 get everyone to cooperate.
This can't go on forever. I'd like to spread the clue about ingress filtering, and am willing to commit time to the cause. Is anyone with me? Mark
On Thu, 02 Nov 2000 09:59:04 EST, Mark Mentovai <mark-list@mentovai.com> said:
This can't go on forever. I'd like to spread the clue about ingress filtering, and am willing to commit time to the cause. Is anyone with me?
The problem is that for many ISPs, I fear the only way to get them to implement 2827-style filtering is for their upstreams to implement a policy of fascist-mode ingress filtering - "We see a bogon packet that your site should have filtered, we pull the plug on your link till you fix it". Time alone won't be enough. Bring a baseball bat. And a spare bat. -- Valdis Kletnieks Operating Systems Analyst Virginia Tech
On Thu, 2 Nov 2000 Valdis.Kletnieks@vt.edu wrote:
The problem is that for many ISPs, I fear the only way to get them to implement 2827-style filtering is for their upstreams to implement a policy of fascist-mode ingress filtering - "We see a bogon packet that your site should have filtered, we pull the plug on your link till you fix it".
Wonderful. The problem has been identified. But, other than foot-stomping, I haven't seen any solutions to correct it. The "we'll pull the plug" attitude won't work unless absence of said filtering violates that ISP's upstream AUP or contract. Some problems: ISPs should be doing ingress filtering and aren't. There [may] exist ISPs that [may] know that such filtering needs to be done and don't possess the information/wherewithall/incentive to determine a resolution for implementation. Some suggestions: 1) Develop a group of technical contacts, one each company, for each Tier 1 provider. 2) Create a document with configuration examples for various routers 3) Request that each technical contact of these Tier 1 providers coordinate with its respective internal customer service reps to handle dissemination of said document to its ISP customers. or 4) Disseminate the document through other appropriate mailing lists or newsgroups. It's completely pointless to identify a problem without also identifying possible solutions or working toward correcting the problem.
On Thu, 02 Nov 2000 09:58:25 CST, J Bacher said:
Wonderful. The problem has been identified. But, other than foot-stomping, I haven't seen any solutions to correct it.
The "we'll pull the plug" attitude won't work unless absence of said filtering violates that ISP's upstream AUP or contract.
Yes, but sometimes...
4) Disseminate the document through other appropriate mailing lists or newsgroups.
It's completely pointless to identify a problem without also identifying possible solutions or working toward correcting the problem.
Myself and some co-workers have been involved in this.. http://www.sans.org/dosstep/index.htm http://www.sans.org/topten.htm Both go down to the "Install *this* patch" or "use *this* command" level. And it's amazing how many sites simply *WILL NOT* do it unless forced to. Enlightened altruism still exists in scattered pockets of the Internet, but vast wastelands are managed by people and organizations that either don't care or are unable to budget resources to act until the bits actually stop moving from place to place. -- Valdis Kletnieks Operating Systems Analyst Virginia Tech
Both go down to the "Install *this* patch" or "use *this* command" level.
Exactly my point. I think that you will have to reach that level unfortunately.
And it's amazing how many sites simply *WILL NOT* do it unless forced to.
Ignorance breeds fear.
Enlightened altruism still exists in scattered pockets of the Internet, but vast wastelands are managed by people and organizations that either don't care or are unable to budget resources to act until the bits actually stop moving from place to place.
Altruism gags me.
J Bacher wrote:
Some suggestions:
A response I get is that they won't do it because it has a negative performance impact on their routers. They blame the router vendors. Suggestion 5), someone calculate what performance penalty there is for typical router configurations when these filters are applied. Show some performance numbers that make the case for or against filtering. John
Some suggestions:
A response I get is that they won't do it because it has a negative performance impact on their routers. They blame the router vendors. Suggestion 5), someone calculate what performance penalty there is for typical router configurations when these filters are applied. Show some performance numbers that make the case for or against filtering.
I've seen the same concerns. Point is, if an individual is concerned enough to identify what is perceived as a serious problem, then the indivdiual also needs to identify potential solutions. If those solutions require sufficient data for justification -- fine. If not, then what is the point of complaining?
On Thu, 2 Nov 2000, John Kristoff wrote:
J Bacher wrote:
Some suggestions:
A response I get is that they won't do it because it has a negative performance impact on their routers. They blame the router vendors. Suggestion 5), someone calculate what performance penalty there is for typical router configurations when these filters are applied. Show some performance numbers that make the case for or against filtering.
You'll need to take into account the type of hardware being used, as well as the platforms ability to take 'compiled' filter lists, like what Cisco calls Turbo ACL's. Turbo ACL's require 12.0(1)T and up on the 7200-12000 platforms, and supposedly require the same time to process whether the ACL is 1 line or 100. Are there any Tier 1 providers who are using hardware less powerful than the 7200 series? Where I'm currently at, we don't use anything less powerful than the 7200VXR series, with the majority of our hardware made up of GSR's, but we're not a Tier1 provider either. Is there a significant amount of legacy gear deployed out in the Tier-1 networks? I still think my idea of having the router core logic work in a "I don't advertise network x.x.x.x, so I don't pass traffic from network x.x.x.x" manner is an ideal solution, and should work as fast as route table lookups. So far, no one has presented a case where opposite behavior is desirable. -- Joseph W. Shaw Sr. Network Security Specialist for Big Company not to be named because I don't speak for them here. I have public opinions, and they don't.
On Thu, 2 Nov 2000, J Bacher wrote:
Wonderful.The problem has been identified. But, other than foot-stomping, I haven't seen any solutions to correct it.
You hit the nail on it's head, that is exactly the feeling I have.
The "we'll pull the plug" attitude won't work unless absence of said filtering violates that ISP's upstream AUP or contract.
Naturally, people still want to do business, and we're not looking for an intimidation atmosphere, but ... Reading the below, what I see is a Internet Draft that defines how to properly protect your own network and other networks, using known RFCs are techniques that are common practice among cluefull people nowadays. This draft could be come the cook-book of connecting your enterprise/company/ISP/entity to an upstream provider, and be used a a reference guide for the said purpose. Of course, if it will be updated with contemporan knowledge and methods from time to time, it will really be best. The tier1 (and even tier2) providers should insist, as part of the AUP or "getting-connected to us" policy that their downstream implement. This mailing list (not only) has a handful of very knowledgeable and experienced inter-networking experts, as well as people who have grown into management throughout the years. I think it's well within the capability of this forum (or any other better suited forum, if one will be suggested) to create and maintain such a draft. Thoughts ? --Ariel
Some suggestions:
1) Develop a group of technical contacts, one each company, for each Tier 1 provider. 2) Create a document with configuration examples for various routers 3) Request that each technical contact of these Tier 1 providers coordinate with its respective internal customer service reps to handle dissemination of said document to its ISP customers.
or
4) Disseminate the document through other appropriate mailing lists or newsgroups.
It's completely pointless to identify a problem without also identifying possible solutions or working toward correcting the problem.
-- Ariel Biener e-mail: ariel@post.tau.ac.il Work phone: 03-6406086 fingerprint = 07 D1 E5 3E EF 6D E5 82 0B E9 21 D4 3C 7D 8B BC
Well since everyone else is stating their opinions, I'll join in as well. First off I think pulling the plug is a great idea ( =] ). Anyways the point comes down to this. Who should be doing the ingress filtering? Tier-2's, Tier-1's, the actual customer? I know this whole idea sounds very pretty and nice, however, when it comes down to it there are many real problems with this idea. One, the hardware on most ISP's backbones cannot realistically do ingress filtering. I'm sorry to say but a GSR is not able to do ingress filtering on 5 Channelized OC-12's that hold 400+ Customers a piece. It just does not work, I don't care what Cisco claims, it just does not work. What about other vendors? I have no experience with Bay or Lucent, however, Juniper (which I do have experience with) has the ability due to the hardware based filtering available but that brings up a whole set of other questions. How will ingress filtering from an ISP level effect downstream customers that do asymmetrical routing? How about the management overhead that comes into play when you are a Tier-1 or a large Tier-2 with tens of thousands of customers? What is comes down to is that customers need to be doing egress filtering, it's the only scalable solution, however this just is not happening. Don't blame the ISPs only, it's their customers that are really the problem. Lack of security/knowledge on the customer's end leads to hacked boxes, which in turn lead to DoS attacks. It really comes down to not the responsibility of the ISP, but in fact the responsibility of the customers! Maybe we all should thinkg about that before we point fingers. On Thu, 2 Nov 2000 Valdis.Kletnieks@vt.edu wrote:
On Thu, 02 Nov 2000 09:59:04 EST, Mark Mentovai <mark-list@mentovai.com> said:
This can't go on forever. I'd like to spread the clue about ingress filtering, and am willing to commit time to the cause. Is anyone with me?
The problem is that for many ISPs, I fear the only way to get them to implement 2827-style filtering is for their upstreams to implement a policy of fascist-mode ingress filtering - "We see a bogon packet that your site should have filtered, we pull the plug on your link till you fix it".
Time alone won't be enough. Bring a baseball bat. And a spare bat.
-- Valdis Kletnieks Operating Systems Analyst Virginia Tech
dies wrote:
Well since everyone else is stating their opinions, I'll join in as well. First off I think pulling the plug is a great idea ( =] ).
Actually, adding the requirement for ingress filtering to the terms of service for backbones (and smaller) networks is a better idea. Start with the legal stuff, then enforce it.
Anyways the point comes down to this. Who should be doing the ingress filtering? Tier-2's, Tier-1's, the actual customer? I know this whole idea sounds very pretty and nice, however, when it comes down to it there are many real problems with this idea.
One, the hardware on most ISP's backbones cannot realistically do ingress filtering.
It's a political issue. When we first wrote the drafts which became RFC 2267 and later RFC 2827, we had this exact argument from several folks... "our equipment can't handle it" or "it'll melt our routers." That was 4 years ago. And they were right, the routers of that era would melt down, and we knew that. Most of those folks have replaced their router and switch gear since then. Do you suppose they added wire-speed ingress filtering to their requirements list when shopping for new product? Some did, others chose to ignore it. Vendors will build what you ask for. Equipment DOES exist which can filter at wire speed up to and including gigabit Ethernet. The question is whether there's the will or desire to actually deploy such things. I think you're going to see larger end-user customers selecting ISPs based on who has a commitment to requiring ingress filtering of all customers.
I'm sorry to say but a GSR is not able to do ingress filtering on 5 Channelized OC-12's that hold 400+ Customers a piece. It just does not work, I don't care what Cisco claims, it just does not work.
So you have a problem with one of your vendors. Beat 'em up, throw their gear out, or whatever.
What about other vendors? I have no experience with Bay or Lucent, however, Juniper (which I do have experience with) has the ability due to the hardware based filtering available but that brings up a whole set of other questions. How will ingress filtering from an ISP level effect downstream customers that do asymmetrical routing?
It is entirely possible to develop a product which can handle ingress filtering based on BGP tables, and do it all automatically. Cisco has not done this, and it's not clear their gear can do it in their present designs. I don't know if anyone else is doing it either. That's not because it's impossible, just that the capability wasn't on a list of requirements when the hardware and software were designed. If this is important to you, make sure your vendors know this. Tell them in writing, so the message sticks around.
How about the management overhead that comes into play when you are a Tier-1 or a large Tier-2 with tens of thousands of customers?
Tier-1 and Tier-2 providers seem to handle management of IP address space delegation, routing topology, BGP route filtering and lots of other per-customer tasks without trouble. It's a commitment thing. If providing ingress filtering becomes a standard part of the suite of functions, then it'll get integrated into the rest and it'll be supportable. Have you asked those you buy from if they'll do this? Did you suggest you'd switch to a tier-1 who does? Lacking incentives, status quo will rule.
What is comes down to is that customers need to be doing egress filtering, it's the only scalable solution, however this just is not happening.
Egress filtering is ingress filtering being done on the other end of the pipe. The ISP has no control, and thus will not be aware of whether it's really taking place. For ISPs who manage the routers at customer locations, this is clearly a no-brainer, but how many do implement the filters? Pushing the problem out and saying it's really the end-user's issue is a dodge. The ISPs have to be responsible for their own networks too. I'm waiting to see how many negligence lawsuits are filed by the various targets of the DoS attacks last spring. Those who didn't implement ingress filtering (and end users who don't implement egress filtering) may well be found liable for failing to do so, and thereby contributing to damage of others' systems.
Don't blame the ISPs only, it's their customers that are really the problem. Lack of security/knowledge on the customer's end leads to hacked boxes, which in turn lead to DoS attacks. It really comes down to not the responsibility of the ISP, but in fact the responsibility of the customers! Maybe we all should thinkg about that before we point fingers.
Maybe those who should have the clue (the ISPs) should take on the responsibility and do the work. Let's face it, sales staff at the ISPs are busy trying to sell lines to every business, small and large, and those businesses aren't going to be networking experts. The ISPs are in the appropriate position to understand the problem and implement the solution. The solution from an ISP's perspective could be a variety of things, ranging from a Terms of Service item that requires filtering at the customer router, to actually implementing filtering at the access router/server/switch where the customers attach.
On Thu, 2 Nov 2000 Valdis.Kletnieks@vt.edu wrote:
On Thu, 02 Nov 2000 09:59:04 EST, Mark Mentovai <mark-list@mentovai.com> said:
This can't go on forever. I'd like to spread the clue about ingress filtering, and am willing to commit time to the cause. Is anyone with me?
The problem is that for many ISPs, I fear the only way to get them to implement 2827-style filtering is for their upstreams to implement a policy of fascist-mode ingress filtering - "We see a bogon packet that your site should have filtered, we pull the plug on your link till you fix it".
Time alone won't be enough. Bring a baseball bat. And a spare bat.
-- Valdis Kletnieks Operating Systems Analyst Virginia Tech
-- ----------------------------------------------------------------- Daniel Senie dts@senie.com Amaranth Networks Inc. http://www.amaranth.com
On Thu, 2 Nov 2000, dies wrote: Actually, I was thinking of taking a slightly different path. 1). ISPs (downstreams) responsibilities: 1a). Active and clueful NOC. 1b). Proper implementation of Internet RFCs (private networks and spoofing). 1c). Proper BGP filtering towards upstream. 1d). Running routing software adequate to today's challenges (for example, something that can deal properly with small fragments...). 2). NSPs (upstreams, tier1/2) responsibilities: 2a). Active and clueful abuse department. 2b). Monitoring tools that will allow tracking down a stream throghout their own backbone, and cooperation with their peers to get to the source. With a handful of cluefull people, and a set of autmatic tools, this IMHO is something at hand. 2c). Proper BGP filtering of their customers, to not allow their customers to fuck-up Internet routing (Sprintlink incidents...). The main idea is that downstreams make sure they are not very susceptable to abuse. Then, if attacked, the upstream provider should cooperate in detecting the source, by monitoring their own network, and cooperating with it's peers. If for any reason, it takes more than X hours, the upstream provider should try to protect it's customers. Usually, it wont, and the attacker will be indentified, and shut down at it's source. There is no requirement that tier1/2 NSPs tie up their equipment into an ACL monster. What is needed is cooperation. With proper cooperation, identifying a stream's source is much easier that you may think. Do you find the above so hard to do ? All it requires is some professional attitude, and a bit more cooperation and consideration from NSPs and ISPs as one, what happens to someone you don't know today may happen to you tomorrow ! Thoughts ? --Ariel
Well since everyone else is stating their opinions, I'll join in as well.First off I think pulling the plug is a great idea ( =] ). Anyways the point comes down to this.Who should be doing the ingress filtering?Tier-2's, Tier-1's, the actual customer?I know this whole idea sounds very pretty and nice, however, when it comes down to it there are many real problems with this idea.One, the hardware on most ISP's backbones cannot realistically do ingress filtering.I'm sorry to say but a GSR is not able to do ingress filtering on 5 Channelized OC-12's that hold 400+ Customers a piece.It just does not work, I don't care what Cisco claims, it just does not work.What about other vendors? I have no experience with Bay or Lucent, however, Juniper (which I do have experience with) has the ability due to the hardware based filtering available but that brings up a whole set of other questions.How will ingress filtering from an ISP level effect downstream customers that do asymmetrical routing?How about the management overhead that comes into play when you are a Tier-1 or a large Tier-2 with tens of thousands of customers?What is comes down to is that customers need to be doing egress filtering, it's the only scalable solution, however this just is not happening.Don't blame the ISPs only, it's their customers that are really the problem.Lack of security/knowledge on the customer's end leads to hacked boxes, which in turn lead to DoS attacks.It really comes down to not the responsibility of the ISP, but in fact the responsibility of the customers!Maybe we all should thinkg about that before we point fingers.
On Thu, 2 Nov 2000 Valdis.Kletnieks@vt.edu wrote:
On Thu, 02 Nov 2000 09:59:04 EST, MarkMentovai <mark-list@mentovai.com>said:
This can't go on forever.I'd like to spread the clue about ingress filtering, and am willing to commit time to the cause.Is anyone with me?
The problem is that for many ISPs, I fear the only way to get them to implement 2827-style filtering is for their upstreams to implement a policy of fascist-mode ingress filtering - "We see a bogon packet that your site should have filtered, we pull the plug on your link till you fix it".
Time alone won't be enough.Bring a baseball bat. And a spare bat.
-- Valdis Kletnieks Operating Systems Analyst Virginia Tech
-- Ariel Biener e-mail: ariel@post.tau.ac.il Work phone: 03-6406086 fingerprint = 07 D1 E5 3E EF 6D E5 82 0B E9 21 D4 3C 7D 8B BC
I agree that getting help from Tier1/2 NSPs is not a outlandish request/idea. However, in my personal experience UUNet has been the most responsive out of all the Tier-1 NSPs (Assuming they are contacted correctly). Tracking the attack, however, is not always a trivial task. If you are a large NSP like UUNet, Sprint, and C&W, where your backbone spans multiple POPs or even countries with their own POPs it is time consuming and you run a real risk of crashing transit routers if the flood is heavy (read high packets per second). At that point the question becomes, "Is it worth tracing an attack on a shell server/irc server?" I'm not trying to say running an IRC server or shell server is wrong, however, some (if not most) NSPs/ISPs consider them attractive nuisances. This topic is continously covered (as you can tell) so I'm not sure where to go from here. I've been looking into starting a BGP speaker that announces the top 4000 smurf amplifiers from my list and the top 2000 from netscan, which in turn can be pushed to null0 or discard. This would prevent one from actually attacking from a network participating in this BGP session. I've tested it and it's working on a couple smaller ISPs as we speak. My major problem is getting some Tier-1's to go for it...Anyways I guess larger attacks than last Feb. will have to happen to get something done? Let's hope not... On Fri, 3 Nov 2000, Ariel Biener wrote:
On Thu, 2 Nov 2000, dies wrote:
Actually, I was thinking of taking a slightly different path.
1). ISPs (downstreams) responsibilities:
1a). Active and clueful NOC. 1b). Proper implementation of Internet RFCs (private networks and spoofing). 1c). Proper BGP filtering towards upstream. 1d). Running routing software adequate to today's challenges (for example, something that can deal properly with small fragments...).
2). NSPs (upstreams, tier1/2) responsibilities:
2a). Active and clueful abuse department. 2b). Monitoring tools that will allow tracking down a stream throghout their own backbone, and cooperation with their peers to get to the source. With a handful of cluefull people, and a set of autmatic tools, this IMHO is something at hand. 2c). Proper BGP filtering of their customers, to not allow their customers to fuck-up Internet routing (Sprintlink incidents...).
The main idea is that downstreams make sure they are not very susceptable to abuse. Then, if attacked, the upstream provider should cooperate in detecting the source, by monitoring their own network, and cooperating with it's peers. If for any reason, it takes more than X hours, the upstream provider should try to protect it's customers. Usually, it wont, and the attacker will be indentified, and shut down at it's source.
There is no requirement that tier1/2 NSPs tie up their equipment into an ACL monster. What is needed is cooperation. With proper cooperation, identifying a stream's source is much easier that you may think.
Do you find the above so hard to do ? All it requires is some professional attitude, and a bit more cooperation and consideration from NSPs and ISPs as one, what happens to someone you don't know today may happen to you tomorrow !
Thoughts ?
--Ariel
Well since everyone else is stating their opinions, I'll join in as well.First off I think pulling the plug is a great idea ( =] ). Anyways the point comes down to this.Who should be doing the ingress filtering?Tier-2's, Tier-1's, the actual customer?I know this whole idea sounds very pretty and nice, however, when it comes down to it there are many real problems with this idea.One, the hardware on most ISP's backbones cannot realistically do ingress filtering.I'm sorry to say but a GSR is not able to do ingress filtering on 5 Channelized OC-12's that hold 400+ Customers a piece.It just does not work, I don't care what Cisco claims, it just does not work.What about other vendors? I have no experience with Bay or Lucent, however, Juniper (which I do have experience with) has the ability due to the hardware based filtering available but that brings up a whole set of other questions.How will ingress filtering from an ISP level effect downstream customers that do asymmetrical routing?How about the management overhead that comes into play when you are a Tier-1 or a large Tier-2 with tens of thousands of customers?What is comes down to is that customers need to be doing egress filtering, it's the only scalable solution, however this just is not happening.Don't blame the ISPs only, it's their customers that are really the problem.Lack of security/knowledge on the customer's end leads to hacked boxes, which in turn lead to DoS attacks.It really comes down to not the responsibility of the ISP, but in fact the responsibility of the customers!Maybe we all should thinkg about that before we point fingers.
On Thu, 2 Nov 2000 Valdis.Kletnieks@vt.edu wrote:
On Thu, 02 Nov 2000 09:59:04 EST, MarkMentovai <mark-list@mentovai.com>said:
This can't go on forever.I'd like to spread the clue about ingress filtering, and am willing to commit time to the cause.Is anyone with me?
The problem is that for many ISPs, I fear the only way to get them to implement 2827-style filtering is for their upstreams to implement a policy of fascist-mode ingress filtering - "We see a bogon packet that your site should have filtered, we pull the plug on your link till you fix it".
Time alone won't be enough.Bring a baseball bat. And a spare bat.
-- Valdis Kletnieks Operating Systems Analyst Virginia Tech
-- Ariel Biener e-mail: ariel@post.tau.ac.il Work phone: 03-6406086 fingerprint = 07 D1 E5 3E EF 6D E5 82 0B E9 21 D4 3C 7D 8B BC
On Thu, 2 Nov 2000, dies wrote: Still, the basic rules of connecting to a network provider should be implemented correctly (if you can come close to eliminating spoofed attacks - identifying where an attack comes from becomes alot easier, don't you agree ?). My goal was not just to mutter. I thank UUnet for providing the details, and I will make sure these phone numbers are properly used. My goal was to create a framework that will define how to properly define your networking equipment when you connect to a NSP, and what the NSPs should do in order to protect themselves and their customers. As I said, there are experienced and knowledgeable people on this list. I am anticipating your responses, and insights. thanks, --Ariel
I agree that getting help from Tier1/2 NSPs is not a outlandish request/idea.However, in my personal experience UUNet has been the most responsive out of all the Tier-1 NSPs (Assuming they are contacted correctly).Tracking the attack, however, is not always a trivial task.If you are a large NSP like UUNet, Sprint, and C&W, where your backbone spans multiple POPs or even countries with their own POPs it is time consuming and you run a real risk of crashing transit routers if the flood isheavy (read high packets per second).At that point the question becomes, "Is it worth tracing an attack on a shell server/irc server?"I'm not trying to say running an IRC server or shell server is wrong, however, some (if not most) NSPs/ISPs consider them attractive nuisances.
This topic is continously covered (as you can tell) so I'm not sure where to go from here.I've been looking into starting a BGP speaker that announces the top 4000 smurf amplifiers from my list and the top 2000 from netscan, which in turn can be pushed to null0 or discard.This would prevent one from actually attacking from a network participating in this BGP session.I've tested it and it's working on a couple smaller ISPs as we speak.My major problem is getting some Tier-1's to go for it...Anyways I guess larger attacks than last Feb. will have to happen to get something done?Let's hope not...
On Fri, 3 Nov 2000, Ariel Biener wrote:
On Thu, 2 Nov 2000, dies wrote:
Actually, I was thinking of taking a slightly different path.
1). ISPs (downstreams) responsibilities:
1a). Active and clueful NOC. 1b). Proper implementation of Internet RFCs (private networks and spoofing). 1c). Proper BGP filtering towards upstream. 1d). Running routing software adequate to today's challenges (for example, something that can deal properly with small fragments...).
2). NSPs (upstreams, tier1/2) responsibilities:
2a). Active and clueful abuse department. 2b). Monitoring tools that will allow tracking down a stream throghout their own backbone, and cooperation with their peers to get to the source. With a handful of cluefull people, and a set of autmatic tools, this IMHO is something at hand. 2c). Proper BGP filtering of their customers, to not allow their customers to fuck-up Internet routing (Sprintlink incidents...).
The main idea is that downstreams make sure they are not very susceptable to abuse. Then, if attacked, the upstream provider should cooperate in detecting the source, by monitoring their own network, and cooperating with it's peers. If for any reason, it takes more than X hours, the upstream provider should try toprotect it's customers. Usually, it wont, and the attacker will be indentified, and shut down at it's source.
There is no requirement that tier1/2 NSPs tie up their equipment into an ACL monster. What is needed is cooperation. With proper cooperation, identifying a stream's source is much easier that you may think.
Do you find the above so hard to do ?All it requires is some professional attitude, and a bit more cooperation and consideration from NSPs and ISPs as one, what happens to someone you don't know today may happen to you tomorrow !
Thoughts ?
--Ariel
Well since everyone else is stating their opinions, I'll join in as well.First off I think pulling the plug is a great idea ( =] ). Anyways the point comes down to this.Who should be doing the ingress filtering?Tier-2's, Tier-1's, the actual customer?I know this whole idea sounds very pretty and nice, however, when it comes down to it there are many real problems with this idea.One, the hardware on most ISP's backbones cannot realistically do ingress filtering.I'm sorry to say but a GSR is not able to do ingress filtering on 5 Channelized OC-12's that hold 400+ Customers a piece.It just does not work, I don't care what Cisco claims, it just does not work.What about other vendors?I have no experience with Bay or Lucent, however, Juniper (which I do have experience with) has the ability due to the hardware based filtering available but that brings up a whole set of other questions.How will ingress filtering from an ISP level effect downstream customers that do asymmetrical routing?How about the management overhead that comes into play when you are a Tier-1 or a large Tier-2 with tens of thousands of customers?What is comes down to is that customers need to be doing egress filtering, it's the only scalable solution, however this just is not happening.Don't blame the ISPs only, it's their customers that are really the problem.Lack of security/knowledge on the customer's end leads to hacked boxes, which in turn lead to DoS attacks.It really comes down to not the responsibility of the ISP, but in fact the responsibility of the customers!Maybe we all should thinkg about that before we point fingers.
On Thu, 2 Nov 2000 Valdis.Kletnieks@vt.edu wrote:
On Thu, 02 Nov 2000 09:59:04 EST, MarkMentovai <mark-list@mentovai.com>said:
This can'tgo on forever.I'd like to spread the clue about ingress filtering, and am willing to commit time to the cause.Is anyone with me?
The problem is that for many ISPs, I fear the only way to get them to implement 2827-style filtering is for their upstreams to implement a policy of fascist-mode ingress filtering - "We see a bogon packet that your site should have filtered, we pull the plug on your link till you fix it".
Time alone won't be enough.Bring a baseball bat.And a spare bat.
-- Valdis Kletnieks Operating Systems Analyst Virginia Tech
-- Ariel Biener e-mail: ariel@post.tau.ac.il Work phone: 03-6406086 fingerprint = 07 D1 E5 3E EF 6D E5 82 0B E9 21 D4 3C 7D 8B BC
-- Ariel Biener e-mail: ariel@post.tau.ac.il Work phone: 03-6406086 fingerprint = 07 D1 E5 3E EF 6D E5 82 0B E9 21 D4 3C 7D 8B BC
On Thu, Nov 02, 2000 at 10:29:51PM -0500, dies wrote:
This topic is continously covered (as you can tell) so I'm not sure where to go from here. I've been looking into starting a BGP speaker that announces the top 4000 smurf amplifiers from my list and the top 2000 from netscan, which in turn can be pushed to null0 or discard. This would prevent one from actually attacking from a network participating in this BGP session. I've tested it and it's working on a couple smaller ISPs as we speak. My major problem is getting some Tier-1's to go for it...Anyways I guess larger attacks than last Feb. will have to happen to get something done? Let's hope not...
Oh, do tell... how does the fact that you null route smurf amps stop those amps attacking you? -- John Payne http://www.sackheads.org/jpayne/ john@sackheads.org http://www.sackheads.org/uce/ Fax: +44 870 0547954 To send me mail, use the address in the From: header
prevent one from actually attacking from a network participating in
Hmmm... This would this
BGP session.
That is from my email...It stops the attack from happening...It will not stop the attack if it is ongoing...Was there some confusion in how I stated that? On Fri, 3 Nov 2000, John Payne wrote:
On Thu, Nov 02, 2000 at 10:29:51PM -0500, dies wrote:
This topic is continously covered (as you can tell) so I'm not sure where to go from here. I've been looking into starting a BGP speaker that announces the top 4000 smurf amplifiers from my list and the top 2000 from netscan, which in turn can be pushed to null0 or discard. This would prevent one from actually attacking from a network participating in this BGP session. I've tested it and it's working on a couple smaller ISPs as we speak. My major problem is getting some Tier-1's to go for it...Anyways I guess larger attacks than last Feb. will have to happen to get something done? Let's hope not...
Oh, do tell... how does the fact that you null route smurf amps stop those amps attacking you?
-- John Payne http://www.sackheads.org/jpayne/ john@sackheads.org http://www.sackheads.org/uce/ Fax: +44 870 0547954 To send me mail, use the address in the From: header
On Wed, Nov 01, 2000 at 11:39:45PM -0500, John Fraizer put this into my mailbox:
This begs to question: Why do they still do it? (Put the targets....er IRC servers on their networks?)
[...]
Why do people set their network up as a target? I just don't understand.
[...] Any high-profile site is a target. How about you ask that same question of Yahoo, eBay, CNN, or any of the other sites that were massively attacked early this year? How about Slashdot, which seems to get attacked regularly? Maybe they'll realize that they're setting themselves up as targets by being so popular and will shut down simply to protect the networks that host them.
While I agree that it is unprofessional for your contact at a provider to ignore or be disrespectful of you regarding a DoS against an IRC server, it is just a fact of life that attacks against commercial entities will be treated with much higher priority than attacks against a non-revenue producing "service." Quite frankly, the pizza man comes in WAY above an IRC server in my book.
Something I've found in my time doing security work is that IRC provides an extremely useful 'early warning system'. What attacks and exploits get tried against IRC networks/servers today are the ones that are used against the internet at large tomorrow. This maxim seems to have been coming true with DoS attacks in general now. Apparently these sorts of tools are being used between Israeli and Palestinian web sites, and have been in use against web sites of and in various eastern european countries, not to mention the various attacks discussed above. In my time working at a particular ISP, I helped deal with several attacks against customers that simply hosted web sites. Last I checked none of these folks had anything at all to do with IRC. I would strongly recommend that instead of berating people for 'setting themselves up as targets' you concentrate your efforts on curing the disease -- not the symptom. If for whatever reason some script kiddie decides to attack someone on your network, you won't be able to say "But I'm not running an IRC server!" and expect the attack to go away. You'll have to deal with it, the same way us folks who participate in the 'early warning system' have had to for quite some time now. (Yes, DALnet still mails administrators of open broadcast networks; yes, we still mail admins of hacked boxes; yes, we still try and do whatever we can to help secure the internet. It would be nice if we had help instead of people who think we deserve it for running 'targets', who sit back while we do their work for them.) -Sven -- Dalvenjah FoxFire (aka Sven Nielsen) I bet living in a nudist colony takes Founder, the DALnet IRC Network all the fun out of Halloween. e-mail: dalvenjah@dal.net WWW: http://www.dal.net/~dalvenjah/ whois: SN90 Try DALnet! http://www.dal.net/
Greetings. Stoned koala bears drooled eucalyptus spit in awe as Sven Nielsen exclaimed:
On Wed, Nov 01, 2000 at 11:39:45PM -0500, John Fraizer put this into my mailbox:
This begs to question: Why do they still do it? (Put the targets....er IRC servers on their networks?)
Internet chat in it's various forms is what brings a lot of people to the net to begin with, and for most of these applications to work, *somebody* has to run a server, right? This ain't U*IX talk(1). For a lot of ISPs, IRC is another means of getting the exposure and "brand recognition" that brings both residential and commercial customers to them. It is extremely narrow-minded to only look at internet services that directly make you money. So you're a web hosting provider? Then screw all the dialup (l)users, right? I don't make money directly off them. It doesn't matter that the majority of the people that hit my web farm are on dialup lines, I don't ever see any of the money that they pay their providers, so screw them, screw them all, right? Is this the way we should look at things?
Any high-profile site is a target. How about you ask that same question of Yahoo, eBay, CNN, or any of the other sites that were massively attacked early this year? How about Slashdot, which seems to get attacked regularly? Maybe they'll realize that they're setting themselves up as targets by being so popular and will shut down simply to protect the networks that host them.
Yes. While we are shutting down all these evil, bandwidth-eating IRC servers, let's shut down all of these 5kr1p+-k1dd13-attracting web sites as well. Let's make the internet a hell of a lot less useful to millions of people. Let's see if we still have jobs tomorrow when this happens.
While I agree that it is unprofessional for your contact at a provider to ignore or be disrespectful of you regarding a DoS against an IRC server, it is just a fact of life that attacks against commercial entities will be treated with much higher priority than attacks against a non-revenue producing "service." Quite frankly, the pizza man comes in WAY above an IRC server in my book.
Mr. Fraizer, how do you react when some dialup customer on one of your networks pisses off some script kiddie on IRC and they start sending 100Mbit of garbage at/through you? Do you tell the customer "Well, don't use IRC then?" I do network security for a very large Tier 1 provider and I get calls all the time from customers who are under attack for whatever reason. Lately, the popular way to do it seems to be to send tons of ICMP garbage to the IP of the terminal server that the victim is behind. I can just see it now: Customer: My circuit is being saturated by tons of ICMP garbage. Can you please do something to stop it before it gets to my pipe? I don't know what they did, but apparently one of my dialup users has pissed somebody off, for these attacks are aimed at my modem pool. Me: Nope, can't help you, sorry. Customer: Excuse me? Me: It's your own fault that you're getting attacked because you your customers use IRC. Now sit back and take your spanking. If you wish, I can have a sales representative contact you tomorrow if you feel you need more bandwidth. Customer: You've got to be kidding! Me: Nope, thank you, drive through. *click* See me sitting in the unemployment office the next day.
Something I've found in my time doing security work is that IRC provides an extremely useful 'early warning system'. What attacks and exploits get tried against IRC networks/servers today are the ones that are used against the internet at large tomorrow.
Yeah, you'd think that people would learn wouldn't you? But no, they don't attempt to fix something until it is directly affecting them. Of course, I was already aware of the Trinoo/etc attacks before CNN/etc got hit, thanks to BugTraq and IRC, and had already gotten the tools necessary to monitor my (then relatively small) network to ensure that such attacks didn't originate from me. If the rest of the internet had taken such measures, then the damage wouldn't have been anywhere near as bad as what it was.
I would strongly recommend that instead of berating people for 'setting themselves up as targets' you concentrate your efforts on curing the disease -- not the symptom. If for whatever reason some script kiddie decides to attack someone on your network, you won't be able to say "But I'm not running an IRC server!" and expect the attack to go away. You'll have to deal with it, the same way us folks who participate in the 'early warning system' have had to for quite some time now.
Well Sven, I think the days of the internet being self-policing are long gone. Remember back when if you sent out a complaint about somebody probing your machine, you actually got a human reply? Not anymore. I send out lots of these every month and most of the time, I don't even get an autoreply, much less a human response. Attempting to contact somebody via telephone usually proves to be an exercise in futility as well. Somebody mentioned in this thread that the Government needed to get involved to regulate the industry. Is this really what you want? Personally, I prefer it if the (US) government kept it's hands out of my business as much as possible. I feel that if/when the government *does* step in, we will all find the internet to be a much less useful (and therefore less profitable) place to be. Not to mention the difficulty the government would have enforcing such laws on an international medium. Ingress-egress filtering would be a major step forward, but also, cooperation between providers so we can nip the problem "in the bud." After all, the less malicious (l)users we have on the net, the less likely we will have malicious packets crossing our routers that we must filter. If I send an email saying "somebody from $ip_address at $time, $timezone was doing $malicious_activity" and in fact $ip_address is under your control or under your customers control, I expect you to do something about it, and I expect you to inform me of what you have done. Unless a court order is involved, I don't expect to ever learn the identity of the problem user, just that he/she has been dealt with appropriately. Do I get this? Nowhere near as often as I should. Why not? It's simple, more likely than not, the ISP in question is not making any money from me, and therefore feels as though that they need not listen to me or deal with my complaint. But, on the same note, I guarantee that if somebody from *my* network does the same exact thing to *his* machine or *his* customer, I would be hearing about it. I am thinking about putting up a "Wall of Shame" that lists those ISPs (especially large ones) who ignore abuse complaints that come from my company. I may even start posting this to NANOG just so we know who's lame and who isn't. Jeff -- "For competitive reasons we can't tell you the location of our fiber." -- An anonymous representative of a very large telco "For competitive reasons we can't tell you the location of our backhoe." -- An anonymous representative of a contractor.
On Sat, 4 Nov 2000, Jeff Workman wrote:
I am thinking about putting up a "Wall of Shame" that lists those ISPs (especially large ones) who ignore abuse complaints that come from my company. I may even start posting this to NANOG just so we know who's lame and who isn't.
Why not just get a domain. Theres fuckedcompany.com, how about fuckednetwork.com - "The tier1 deadpool". -Dan
Stoned koala bears drooled eucalyptus spit in awe as Dan Hollis exclaimed:
Why not just get a domain. Theres fuckedcompany.com, how about fuckednetwork.com - "The tier1 deadpool".
Actually I was thinking more along the lines of "dotnetdeadbeats.org" :). Jeff -- "For competitive reasons we can't tell you the location of our fiber." -- An anonymous representative of a very large telco "For competitive reasons we can't tell you the location of our backhoe." -- An anonymous representative of a contractor.
On Sat, Nov 04, 2000 at 06:00:38AM -0500, Jeff Workman wrote:
Customer: My circuit is being saturated by tons of ICMP garbage. Can you please do something to stop it before it gets to my pipe? I don't know what they did, but apparently one of my dialup users has pissed somebody off, for these attacks are aimed at my modem pool.
Me: Nope, can't help you, sorry.
Customer: Excuse me?
Me: It's your own fault that you're getting attacked because you your customers use IRC. Now sit back and take your spanking. If you wish, I can have a sales representative contact you tomorrow if you feel you need more bandwidth.
Customer: You've got to be kidding!
Me: Nope, thank you, drive through. *click*
A large IP provider (and construction/mining company) essentially told me the above. That was after they turned down my circuit to "stop" the attack. -- Bill Fumerola - Network Architect, BOFH / billf@FreeBSD.org
Stoned koala bears drooled eucalyptus spit in awe as Bill Fumerola exclaimed:
On Sat, Nov 04, 2000 at 06:00:38AM -0500, Jeff Workman wrote:
<dialog snipped>
Customer: You've got to be kidding!
Me: Nope, thank you, drive through. *click*
A large IP provider (and construction/mining company) essentially told me the above. That was after they turned down my circuit to "stop" the attack.
This is the point where I would start questioning whether my business relationship with this provider is such a wise idea, and also whether or not this behavior is the provider's policy or just the network engineer on duty's policy. I pay for bandwith, and for my upstream to at *least* do a little bit to protect my investment. I understand that if I need more than an ACL on my router interface that they will gladly (most likely) sell me a managed firewall package but even without that, I expect them to get the garbage off my pipe when I notice it. I don't even expect them to take proactive measures to keep it off my wire, but when I see it, I expect something to be done about it. If not, then I will let my money do the talking elsewhere. Jeff Representing me, myself, and I. -- "For competitive reasons we can't tell you the location of our fiber." -- An anonymous representative of a very large telco "For competitive reasons we can't tell you the location of our backhoe." -- An anonymous representative of a contractor.
On Wed, Nov 08, 2000 at 02:09:42AM -0500, Jeff Workman wrote:
This is the point where I would start questioning whether my business relationship with this provider is such a wise idea, and also whether or not this behavior is the provider's policy or just the network engineer on duty's policy. I pay for bandwith, and for my upstream to at *least* do a little bit to protect my investment. I understand that if I need more than an ACL on my router interface that they will gladly (most likely) sell me a managed firewall package but even without that, I expect them to get the garbage off my pipe when I notice it. I don't even expect them to take proactive measures to keep it off my wire, but when I see it, I expect something to be done about it. If not, then I will let my money do the talking elsewhere.
Turning off the circuit was the executive opinion, not the technical opinion, of the provider. Simply put: this customer who was shut off paid less then other people in the building, so cutting their circuit made life easier for the provider. -- Bill Fumerola - Lame Duck, BOFH / Chimes, Inc. billf@chimesnet.com / billf@FreeBSD.org
participants (17)
-
Ariel Biener -
Bill Fumerola -
Dan Hollis -
Daniel Senie -
dies -
J Bacher -
Jeff Workman -
Joe Shaw -
John Fraizer -
John Kristoff -
John Payne -
Mark Mentovai -
Randy Bush -
rick -
Simon Lyall -
Sven Nielsen -
Valdis.Kletnieks＠vt.edu