Owen DeLong wrote: Again, I just don't see where an ISP can or should be held liable for

forwarding what appears to be a correctly formatted datagram with a valid destination address. This is the desired behavior and without it, the internet stops working. The problem is systems with consistent and persistent vulnerabilities. One software company is responsible for most of these, and, that would be the best place to concentrate any litigation aimed at fixing the problem through liquidated damages.

Most dDOS's come from bots. Bots are installed on all operating systems and all architectures. I'd be surprised if the packets are all spoofed. In most dDOS cases these days, they are real IP's and just a high number of bots. The person responsible is the bot maintainer. Finding the controller medium (probably irc) is the hard part, but once done, monitoring who controls the bots isn't near as hard. Tracking them down can be abit of fun, but usually they get lazy about covering tracks at that point. A few media enriched prison sentences would be good. -Jack

Jack Bates wrote:

Owen DeLong wrote: Again, I just don't see where an ISP can or should be held liable for

...

forwarding what appears to be a correctly formatted datagram with a valid destination address. This is the desired behavior and without it, the internet stops working. The problem is systems with consistent and persistent vulnerabilities. One software company is responsible for most of these, and, that would be the best place to concentrate any litigation aimed at fixing the problem through liquidated damages.

Most dDOS's come from bots. Bots are installed on all operating systems and all architectures. I'd be surprised if the packets are all spoofed. In most dDOS cases these days, they are real IP's and just a high number of bots.

From the traffic I've seen there are alot of bots - possibly a couple of 1000, however there are 2 distinct traffic types - spoofed and non spoofed. The non spoofed is a pain, but easy to stop. I have been phoning ISPs NOCs getting them shutdown one by one. The spoofed is the problem and when we are seeing 300k SYN pps coming from addresses 0.0.0.4 through 40.0.0.0 in .4 increments I think the traffic is spoofed rather than there being 10's of thousands of machines (especially from hosts like 0.0.0.4!)

The person responsible is the bot maintainer. Finding the controller medium (probably irc) is the hard part, but once done, monitoring who controls the bots isn't near as hard. Tracking them down can be abit of fun, but usually they get lazy about covering tracks at that point. A few media enriched prison sentences would be good.

Granted, however if we could get rid of most of the spoof attacks it would be a damn site easier to track them - I can tell you that the source is coming vi the SprintLink - Telecom NZ international link - however I cannot find out more on the other side of the pond without speaking to Sprint and I know noone there. Yours Mat

On Sat, 30 Aug 2003, Terry Baranski wrote:

Owen DeLong wrote:

...

The ISPs aren't who should be sued. The people running vulnerable systems generating the DDOS traffic and the company providing the Exploding Pinto should be sued. An ISPs job is to forward IP traffic on a best effort basis to the destination address contained in the header of the datagram. Any other behavior can be construed as a breach of contract. Sure, blocking spoofed traffic in the limited cases where it is feasible at the edge would be a good thing, but, I don't see failure to do so as negligent.

In what instances is blocking spoofed traffic at the edge not feasible? ("Spoofed" as in not sourced from one of the customer's netblocks.)

...

Where exactly do you think that the duty to care in this matter would come from for said ISP?

Isn't the edge by far the easiest and most logical place to filter spoofed packets? What are the good reasons not to do so?

As I'v said many times (so have a few others, more now than before) you have to define the 'edge' first... My definition is: "as close to the end system as possible". For instance the LAN segment seems like the ideal place, its where there is the most CPU per packet, with the most simple routing config and most predictable traffic patterns/requirements.

such packets from ever getting past their edge routers. If edge filtering isn't considered a "reasonably simple" thing to do, I'd like to hear the reasons why.

its not tough, you just have to define the edge in the right way.

--On Sunday, August 31, 2003 7:28 AM -0400 Matthew Crocker <matthew@crocker.com> wrote:

...

As I'v said many times (so have a few others, more now than before) you have to define the 'edge' first... My definition is: "as close to the end system as possible". For instance the LAN segment seems like the ideal place, its where there is the most CPU per packet, with the most simple routing config and most predictable traffic patterns/requirements.

The 'edge' is the last piece of equipment on your network. It is what connects you to your customer and what connects you to your upstreams. Every ISP should put Anti spoofing filters on ALL edge interfaces. My entire customer edge (dialup,ISDN,DSL, T1, FR, ATM, Wireless, colo) is defined in LDAP/RADIUS. When a session is established my edge equipment configures itself over RADIUS. It isn't hard to use that information to build a customer specific filter for the session. For example, Every dialup (PPP) or DSL (PPPoE) session should have a filter which *only* allows packets sourced from the customer IP in. It should also deny packets coming from the customer out to the customer. It is pretty simple to do this but you do need to maintain proper customer records. Your customer edge is his equipment and they should also put anti-spoof filters in line. Security is not a single point on a map. Security must be established on every interface. Most people say that you can't filter an OC-48 at line speeds, or that it will increase the latency too much. If filtering increases latency by 5% but decreases junk traffic by 20% don't you think you and the network are better off? For true redundancy for dual-homed sites the links shouldn't be running above 40% capacity anyway. If your router can't filter at 40% line speed you need another router. I know in the core it gets much more complex but when I connected my Verio link I had to make sure all of my IRR entries were correct. They already filter my BGP prefixes I would assume they filter my IP as well. I know I filter my outbound to make sure it is only coming from me.

What you are saying works only so long as none of your edge connections represent a significant portion of the internet. How do you anti-spoof, for example, a peering link with SPRINT or UUNET? It's not realistic to think that you know which addresses could or could not legitimately come from them. You might be able to validate that SOME of your addresses should not come from them, but, that would assume that your customers don't multihome with them. It gets even more sticky when you have edge connections to more than one provider near the top of the food chain. As to latency, trying to filter an AS of any significant size across an OC-48 would require a rule for each prefix said AS advertises to you. If said AS can advertise, say, 10,000 prefixes, then you need an ACL on the edge with 10,000 rules. Most routers won't accept a 10,000 line ACL at all. Even if they do, I suspect on an OC48 the increase in latency would be much more than 5%. Probably more than 20% for any packet that had to traverse the entire list. Yes, VERIO filters your BGP connection because you don't provide a large number of prefixes and have a small number of ASs behind you. I bet they don't filter their connection to SPRINT, UUNET, or AOL. Of course, I also bet your connection to VERIO is probably OC3 or smaller.

The edge is everywhere and the more specific you get the more specific your filters can be. In the core you can't be very specific. We have a bunch of routes that we announce (/16, 2 x /21, 3 x /24). It wouldn't be hard for my upstreams to filter my traffic. I already have to notify them (via IRR) when I have a new announcement. They can update my filter when they update the prefix-list

Yes. Your situation is simple. Your situation isn't everyone's situation. Owen

Owen DeLong wrote:

...

The ISPs aren't who should be sued. The people running vulnerable systems generating the DDOS traffic and the company providing the Exploding Pinto should be sued. An ISPs job is to forward IP traffic on a best effort basis to the destination address contained in the header of the datagram. Any other behavior can be construed as a breach of contract. Sure, blocking spoofed traffic in the limited cases where it is feasible at the edge would be a good thing, but, I don't see failure to do so as negligent.

In what instances is blocking spoofed traffic at the edge not feasible? ("Spoofed" as in not sourced from one of the customer's netblocks.)

That depends on your definition of edge, I suppose. I define it as the port on one of my routers where the other end of the link is connected to a machine I don't control. In those terms, edge filtering makes sense in some cases and not in others. If it's a dial-up or T1 customer which is a single business, it makes sense. If it's an ISP with a few fortune 500 customers, it doesn't work out as well.

...

Where exactly do you think that the duty to care in this matter would come from for said ISP?

Isn't the edge by far the easiest and most logical place to filter spoofed packets? What are the good reasons not to do so?

Again, where "edge" is a single end-customer, yes. Where edge is simply the connection of two border routers among ISPs, it's alot harder vs. minimal gain. While I agree that "edge" filtering is good practice anywhere it makes sense, I still don't think that legislating it through liability is a good precedent to set. I'm already far enough off topic for today that won't go into the details of the legal slippery slope it creates.

...

Again, I just don't see where an ISP can or should be held liable for forwarding what appears to be a correctly formatted datagram with a valid destination address.

I guess "correctly formatted" is a relative term. When *isn't* a packet with a spoofed source IP address guaranteed to be illegitimate? Maybe such packets shouldn't be considered "correct".

I carefully chose the term "correctly formatted" instead of "valid" for exactly that reason. If the datagram contents conform to the RFC definitions of what an IP datagram should contain and in the correct order and relative octet positions, then, the packet is a "correctly formatted" packet. If an ISP has a way to feasibly filter a link for spoofed addresses without risk of creating false matches, then, it is good practice to do so. However, there are many links where this is not feasible.

...

This is the desired behavior and without it, the internet stops working.

The Internet stops working when legitimate packets aren't forwarded. Spoofed packets don't fall into this category.

Agreed. However, there are a limited number of places where this distinction can be reliably made in software. In those locations, it makes sense to discard what can reliably be discarded. More agressive proposals represent damage.

...

The problem is systems with consistent and persistent vulnerabilities. One software company is responsible for most of these, and, that would be the best place to concentrate any litigation aimed at fixing the problem through liquidated damages.

I don't think it's appropriate to point the finger at one entity here. Lots of folks can play a part in helping out with this problem. That spoofed packets often originate from compromised hosts running Microsoft software doesn't justify ISPs standing around with their hands in their pockets if there are reasonably simple measures they can take to prevent such packets from ever getting past their edge routers. If edge filtering isn't considered a "reasonably simple" thing to do, I'd like to hear the reasons why.

I think it is appropriate to point the finger at root cause and focus resolution on the root cause. The root cause is a software company which has systematically engineered vulnerabilities into their software and aggressively propogated these vulnerabilities to as many systems as they can. However, that having been said, I'm not saying that ISPs should stand around with their hands in their pockets. Where reasonably simple measures which do not create collateral damage can be taken, they should. As to edge filtering, I suspect you are restricting the term to a different definition of edge than mine. As such, I think I have explained the parts of the edge where I consider it unreasonable. I also think that ISPs should take the relatively simple precaution of including in their AUP that if the customer starts sending attack traffic, regardless of reason, the ISP has the right to filter, block, rate limit, or otherwise disconnect the customer until customer resolves the issue. Then, I think ISPs should be more agressive about actually doing so. However, I'm very tired of the idea that everyone else should go to elaborate lengths to engineer around broken software because it's too popular and too hard to get it fixed. At some point, we're going to have to recognize that broken software (at this level, at least) is unacceptable and as much pressure as possible to resolve that issue _MUST_ be brought to bear on the responsible party. This is inherently the biggest disadvantage to closed-source software. Owen

Subject: RE: On the back of other 'security' posts.... Date: Sat, Aug 30, 2003 at 11:51:02PM -0700 Quoting Owen DeLong (owen@delong.com):

...

That depends on your definition of edge, I suppose. I define it as the port on one of my routers where the other end of the link is connected to a machine I don't control. In those terms, edge filtering makes sense in some cases and not in others. If it's a dial-up or T1 customer which is a single business, it makes sense. If it's an ISP with a few fortune 500 customers, it doesn't work out as well.

I'd go with Chris view here. Let me try to define why I think so: A device[0] on the network should: * Protect themselves against external[1] threat. * Enforce sense and order in what they allow. * Only try protecting others when they have full knowledge of what they are protecting. This leads to: * Only trust authenticated logins, do as much as possible away with using the network address as a authenticator, except for trivial stuff like perhaps printing. * Stop spoofing by filtering routing. - It is not rocket science to put spoofing filters on CPEs. - More complex in backbones or in multi homed setups. - Enforce some kind of prefix/AS path checks on peerings. Routers know this, and excel at routing or not. They sometimes suck at dropping packets (at least in a controlled fashion). * Filter on the host, where knowledge is maximal (Which hosts do I want to talk to, and by which means?) and collateral damage is minimal (no other activities on other hosts are blocked) * Do not impose general blocks over large user bases. The resulting productivity hit, coupled with the mess of exceptions to be managed will cause more trouble than is won by blocking. * Be prepared to reevaluate in crisis situations. -- Måns Nilsson Systems Specialist +46 70 681 7204 KTHNOC MN1334-RIPE I just remembered something about a TOAD! [0] Any IP-speaking box, be it router, switch, host. [1] meaning anything not in my box, coming from LAN or console.

the rest of the paper is also germane to this thread. just fya, we keep rehashing the UNimportant part of this argument, and never progressing. (from this, i deduce that we must be humans.)

Ok, so we seem to have a general agreement that anti-spoof & BGP prefix filtering on all standard customer edge links is a worthwhile practice. Now what? Is there any hope of this ever happening on a very large scale without somehow being mandated? (Not that it necessarily should be mandated.) How much success have Barry Green and co. had? Is there something the rest of us could be doing?

Ok, so we seem to have a general agreement that anti-spoof & BGP prefix filtering on all standard customer edge links is a worthwhile practice.

actually, we don't. what we've achieved is that gray area / middle ground where the people who don't think it's important are mostly afraid to speak out against it. while this is an important milestone, it's not nearly the same as general agreement.

Now what? Is there any hope of this ever happening on a very large scale without somehow being mandated? (Not that it necessarily should be mandated.)

there is no way to mandate it. even if it were somehow a full standard in the ietf, network owners who didn't want to do it wouldn't have to do it.

How much success have Barry Green and co. had? Is there something the rest of us could be doing?

i'm thinking we may need some kind of branding campaign, so that rfp authors can refer to a set of "good practices" like terminating spammers, not writing "pink contracts", not hosting spamvertised web sites, publishing in the radb, filtering customer routes by rir, running full uprf on customer-facing links, and so on down the line. i'm not sure that we (isc) would be the best people to run an isp branding/certification programme, so i'm hoping someone else steps up, like maybe the rirs or isp/c or maps or whatever. but once the sales people inside isp's have to contend with this as a checklist item in incoming rfp's, it'll see fast deployment even in bankrupt high-inertia "backbone" networks like uunet. -- Paul Vixie

On Sat, 30 Aug 2003, Terry Baranski wrote:

...

Sure, blocking spoofed traffic in the limited cases where it is feasible at the edge would be a good thing, but, I don't see failure to do so as negligent.

In what instances is blocking spoofed traffic at the edge not feasible? ("Spoofed" as in not sourced from one of the customer's netblocks.)

Where the customer is not a basic end user.. an ISP for example who may be transiting traffic from netblocks that are not theirs. We still have the other problem where a lot of large networks are using RFC1918 addresses that do not get NAT'd thus filtering will break pMTU.. this is an issue in my experience mainly for those who host websites, altho many of those are filtering their own packets anyway and have broken sites!

...

Where exactly do you think that the duty to care in this matter would come from for said ISP?

Isn't the edge by far the easiest and most logical place to filter spoofed packets? What are the good reasons not to do so?

It is the only place, virtually any source and destination address could pass across our network core, there is no way to filter at any part of the Internet backbone

...

Again, I just don't see where an ISP can or should be held liable for forwarding what appears to be a correctly formatted datagram with a valid destination address.

I guess "correctly formatted" is a relative term. When *isn't* a packet with a spoofed source IP address guaranteed to be illegitimate? Maybe such packets shouldn't be considered "correct".

Edge filtering ok. But in general it is not the job of the IP router to look into the higher level protocols to determine what the packet does. Look up papers on why this would be bad, RFC3439 by Randy and Dave Meyer is a good read! Steve

Owen DeLong wrote:

...

Yet more spoofed traffic aimed at the SORBS nameservers - this time enough to crash a core router of my upstream... Hopefully the commercial damage now may insite people getting damaged by these DDoSes to start proceedings against those ISPs whom continue to show a lack of respobsibility and allow unfiltered spoofed DDoS traffic from their networks. Certainly I have been told to talk to various US authorities about the problem, and will be doing so as soon as I have the nessesary information.

The ISPs aren't who should be sued.

Any irresponsible party should be in the firing line.

The people running vulnerable systems generating the DDOS traffic and the company providing the Exploding Pinto should be sued. An ISPs job is to forward IP traffic on a best effort basis to the destination address contained in the header of the datagram.

I disagree, they should forward valid traffic

Any other behavior can be construed as a breach of contract.

The depends squarely on their contract, and do contracts say 'we will forward all your traffic including that which is designed to force others off the net'?

Sure, blocking spoofed traffic in the limited cases where it is feasible at the edge would be a good thing, but, I don't see failure to do so as negligent. Where exactly do you think that the duty to care in this matter would come from for said ISP?

In the fact that if an ISP has a customer (a single peered ISP or or enduser) that is sending traffic from addresses that they are not permitted to use publically, they should be blocked and told not to do it again... I remember a _long_ time ago (1991) when I was signed up with my first ISP in the UK a friend and I were experimenting with SYN, UDP and ICMP spoofed traffic, flooding each other (on different ISPs) I got a mail from ISP security telling me to stop within 24 hours, none of the traffic got off their network. Following that, my friend dialed into his account on the same ISP as me, and we continued the tests and next thing I know I got booted for having been warned --- security didn't care that we were doing it to each other with permission etc..... Now I know the net has grown a lot since then, however my ISP did have more than 17k customers at the time... However if they blocked all traffic that didn't originate from the customer back then how come 12 years later some ISPs still continue to allow spoofed traffic from their customers knowing full well that traffic is invalid and likely attacking something.

...

In the mean time a plea to people on this list in all countries - watch for the DDoS attacks (particually against 203.15.51.33, 203.15.51.35, 203.15.51.44 & 203.101.254.254) and stop the damn traffic before you are held responsible for your customers actions. There is still a 10k pps SYN flood occuring 8 hours later - this is being rate limited upstream.

Again, I just don't see where an ISP can or should be held liable for forwarding what appears to be a correctly formatted datagram with a valid destination address.

Where the packet is sourced from 1 network and is addressed as from another network I do not consider that as 'correctly formatted'

This is the desired behavior and without it, the internet stops working.

Bull

The problem is systems with consistent and persistent vulnerabilities. One software company is responsible for most of these, and, that would be the best place to concentrate any litigation aimed at fixing the problem through liquidated damages.

Suing M$ will not solve the problem. Look at the remidies the DoJ set - M$ just completely ignored them and carried on. M$ is too big now, they will continue to do as they see fit and there is noone powerful enough in the market to stop them. This is not about M$ bashing though - its about non carriers originating spoofed traffic and not caring. / Mat