On Mon, 9 Sep 1996, Avi Freedman wrote: ==> ==>I will make time to start running the route aggregator at routes.netaxs.com ==>again; we've been fighting a random-src-address-SYN-attacker for the last ==>week or two. I may have some comments on THAT for NANOG re: inter-provider ==>cooperation shortly. ==> ==>Avi ==> A friend of mine gave me a photocopy of a page in the latest 2600 magazine. It was the source code for a SYN flooder on Linux, with a description of what it does and a notice on how it can really cause denial-of-service attacks. I can't remember if it also supplied the source for the source-spoof kernel patch or not, but it does mention that you should use the source-spoof patch to hide your identity. So, what does this say? Look for more 13-year-olds causing denial-of-service attacks for the hell of it. It seems a lot of the providers SYN flooders like to attack are the ones which have IRC servers, but the flooders attack the more traditional services of those providers, too. /cah ---- Craig A. Huegen CCIE || || Network Analyst, IS-Network/Telecom || || cisco Systems, Inc., 250 West Tasman Drive |||| |||| San Jose, CA 95134, (408) 526-8104 ..:||||||:..:||||||:.. email: chuegen@cisco.com c i s c o S y s t e m s

Re: SYN floods PANIX, a large public access provider in New York, was badly hit with SYN flood attacks from random source addresses over the last few days. It nearly wrecked them. I think its time for the larger providers to start filtering packets coming from customers so that they only accept packets with the customer's network number on it. Yes, its a load on routers. Yes, its nasty for the mobile IP weenies. Unfortunately, the only known way to stop this. Many TCPs go belly up as soon as they get SYN flooded -- its a defect in the protocol design, and other than Karn style anti-clogging tokens ("cookies") being put into a TCP++ and mass implemented worldwide soon, the only reasonable way to stop this sort of terrorism is provider filtering. Perry

BTW, Alexis Rosen at Panix could use some help tracking down the person(s) attacking his machines -- he's more or less being shut down by this. He's having some trouble finding the right person at Sprint (one of his two providers) to talk to. If the right person could get in touch with me, I'll hook the two of you up.

Hopefully, with a little inter-provider cooperation, the guy will get caught and arrested soon.

Perry

I'll post more a bit later (the attack is under way now). MCI was very cooperative, but Sprint said they didn't have time or energy (even though Panix is a Sprint customer) to help to find out where on Sprint's network the packets are entering. (Panix has a t1 to MCI and a t1 to Sprintlink. In fact, Panix was Sprintlink's first ISP customer, (used to be on sl-dc-1-s0)). For a while, the attacker was using a constant seq # (though random ports and src addresses). We hacked the kernel to filter out that seq # in tcp input routines. While how to fix kernels so they're not as vulnerable to huge syn storms is not a NANOG topic, finding the <expletives deleted regretfully> who do this is. More later, Avi

And let's stop fooling ourselves with all those firewalls and other security toys - what we really need is cooperation among ISPs and world peace. Cheers Dima Michael Dillon writes:

On Mon, 9 Sep 1996, Perry E. Metzger wrote:

...

PANIX, a large public access provider in New York, was badly hit with SYN flood attacks from random source addresses over the last few days. It nearly wrecked them.

I think its time for the larger providers to start filtering packets coming from customers so that they only accept packets with the customer's network number on it.

I disagree. A better way to do this would be for providers to cooperate to track down the people who are doing it and make sure to flood the media with press releases when the culprits are arrested. If the cracker wannabe's realize that source-spoofed SYN attacks can still be quickly traced, they will stop doing it.

And the cooperation would do the net some good; maybe lead to more cooperation down the line.

Michael Dillon - ISP & Internet Consulting Memra Software Inc. - Fax: +1-604-546-3049 http://www.memra.com - E-mail: michael@memra.com

...

I think its time for the larger providers to start filtering packets coming from customers so that they only accept packets with the customer's network number on it.

This is VERY important as well. Better to limit the trouble to a local network.

I disagree. A better way to do this would be for providers to cooperate to track down the people who are doing it and make sure to flood the media with press releases when the culprits are arrested. If the cracker wannabe's realize that source-spoofed SYN attacks can still be quickly traced, they will stop doing it.

And the cooperation would do the net some good; maybe lead to more cooperation down the line.

Michael Dillon - ISP & Internet Consulting Memra Software Inc. - Fax: +1-604-546-3049 http://www.memra.com - E-mail: michael@memra.com

Avi

On my private network I can send 600 or more SYN packets to my telnet port (w/faked, unreachable source addresses + random seq numbers), yet the port doesn't seem to be flooded.

It's a linux box.

The telnet daemon seems to be able to tell the difference between a faked packet and a real one. Even when spoofing from localhost, it reports a connection from unknown.

Obviously, there seems to be a solution to this problem. ??

-- Billy Biggs Ottawa, Canada

Nope; it's just that when the kernel on your linux box responds to the SYN, the machine you're doing it from says "RST" and the SYN leaves the "incompleted-connections" listen queue for the socket you're attacking. If you forge random IP source addresses, those packets won't go away and whatever you're pounding on will be hosed until a) 75 seconds (or whatever the timer is set to) expires, or b) you kill and restart the service in question. Avi

In message <199609091719.NAA24855@jekyll.piermont.com>, "Perry E. Metzger" writ es:

Re: SYN floods

PANIX, a large public access provider in New York, was badly hit with SYN flood attacks from random source addresses over the last few days. It nearly wrecked them.

I think its time for the larger providers to start filtering packets coming from customers so that they only accept packets with the customer's network number on it.

Yes, its a load on routers. Yes, its nasty for the mobile IP weenies. Unfortunately, the only known way to stop this. Many TCPs go belly up as soon as they get SYN flooded -- its a defect in the protocol design, and other than Karn style anti-clogging tokens ("cookies") being put into a TCP++ and mass implemented worldwide soon, the only reasonable way to stop this sort of terrorism is provider filtering.

Perry

Perry, I'm not arguing against your point. There are some feasibility issues right now that mean we can't just "turn this on". Packet filtering on prefixes is only feasible as slower speeds with current routers and even then it can give smaller routers a tough time. There is also a problem with packet filtering due to assymetric routing. You can legitimately end up with packets coming from addresses other than those that you route towards and should not black hole that traffic. Given the practical limits, this is a good idea for single homed customers behind a router that can handle it. The problem is what can large providers do about other providers that don't even want to keep track of who their customers are and exchanging the list of prefixes with other providers in support of reliable routing. If routing were symetric, you could look up the destination and only take packets whose source address is reached through that path (given influence on the forwarding code). This was actually considered in the NSFNET days. Routing often isn't symmetric so this would present a black hole problem and so it wasn't done. An alternate is to keep stats on the source/dest prefix pairs (not host pairs). If stats on all of your routers are collected and forged source flooding attacks occur, you need to look at all of the routers for the src/dst pairs reported by the target of the attack. You have then traced the problem back to the physical entry point to your own network. This *was* done in the NSFNET days. If the commonly used routers supported this sort of stat collection providers could quite easily trace an attack back to the source (or at least back to a provider in the path not collecting stats). We've made this request (the ability to collect these sort of stats) to both Bay and Cisco (neither has said "no" but they are busy and haven't got to this yet). I think providing the means to trace back these attacks represents the best large provider level solution to the forged source SYN flooding attacks. At the single homed connection a router option to reverse the sense of the forwarding table on a specific interface (look up the source in the forwarding table and only accept if the source is reachable through that next hop) seems to be a effective preventative that could be easily just "switched on". Otherwise prefix filters have to be configured, which does represent work for the ISP and so won't get done. We have not yet made this request of Bay or Cisco (unless they are reading right now). This would help, except for the providers whose routers don't even support CIDR yet (and are unlikely to pick this up quickly) from whom some of these attacks may originate. Curtis

circuit, so thats not too bad a problem there.

...

At the single homed connection a router option to reverse the sense of the forwarding table on a specific interface (look up the source in the forwarding table and only accept if the source is reachable through that next hop) seems to be a effective preventative that could be easily just "switched on".

A very good idea. If CISCO'll hear it -:)!

Perry

--- Aleksei Roudnev, Network Operations Center, Relcom, Moscow (+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 239-10-10, N 13729 (pager) (+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)

>   -->>   > the forwarding table and only accept if the source is reachable
>   -->>   > through that next hop) seems to be a effective preventative that could
>   -->>   > be easily just "switched on".
>   -->>
>   -->>   A very good idea.
>   -->If CISCO'll hear it -:)!
>   -->
>   -->
>   -->
>   -->>
>   -->>   Perry
>   -->>
>   That sounded like a good idea until I considered asymetric routing. You
>   are assuming the router always knows how to get back to its source, but
Did you read me and Antonov carefully? We have spoken about BORDER
interfaces with the CUSTOMERS. If -
- the default behaviour of CISCO would be _filter out packets with SRC addresses
not from the routing table for this interface_,
- it'll work on the CUSTOMER's interfaces for the single-home customers,
- I should install this behaviour on the part of my interfaces

it'll protect us against more than 90% of this attackes.

Of cource it's not possible to use this for internetwork interfaces in the
big network; it's difficult to use this for inter-network interfaces in case
of multihoming.

Now I have 2 kinds of interfaces there:

1) Strictly controled interfaces for the customers. I have to use exact list
for the network numbers I receive from this interfaces (even in case of BGP I
check not only AS-es but Networks too), and so on  - it's because I don't
trust this users.

2) Peering interfaces - when I excahneg routing with other ISP I trrust them and
am controlling AS pathes only.

Usially I have assymmetrical routing on the interfaces of 2'th type (but this
routing is usially the sighn of _something wrong in this world_). And I do
not want assymmetric routing on the interfaces of the 1'th kind.

>   Traffic is already slow enough when a router is unstable because it may
>   not know how to get to the destination, but if you throw in the
>   requirement that it has to know how to get to the source as well, didn't
>   you just help the hacker by shutting down service for lots of people?
How? I can't understand how this helps the hackers.

Through you are right in case of Universities (and it's not secret just universities
are the motherland of the hackers -:)).
--- 
Aleksei Roudnev, Network Operations Center, Relcom, Moscow
(+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 239-10-10, N 13729 (pager)
(+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)

"Perry E. Metzger" <perry@piermont.com> is alleged to have said: | BTW, I would suggest that for a variety of applications, hardware | assisted filtering boxes that simply take in IP one end and put out | processed IP on the other end would be of use -- not just for this, | but also for helping in doing packet traces through high traffic | areas, for implementing firewalls, and for all sorts of other | things. Vendors, are you listening? Listening? Um, we make such a product, it's been shipping for some time. Our network address translator, product name Private Internet Exchange, can do what you ask, and with speed to spare, too. It seems to be sort of an SR-71 for packet-filtering-- our engineers haven't been able to tell me just what the upper performance bounds are because they seem to have trouble finding them. Right now we offer Fast Ethernet and Ethernet interfaces; I'm sure if there's enough market interest we'd look into doing FDDI or perhaps ATM OC3c. FWIW, last estimates I heard were that the box should scale to around 70K flows or so, which would be enough to handle a NAP connection, I should think. This is all at full line rate. More info on our web site, see http://www.cisco.com/warp/public/751/pix/index.html I've suggesteed to the PIX engineers that they look into whether it is possible to have the PIX reserve enough data structures in the IP queue to "stay ahead" of line rate flooding of SYN packets. In other words, you'd always have a connection being torn down even as more bogons came in. That would let good packets through, too, along with the evil ones. No word yet; I suspect that due to the long timeout needed for bogus source addresses that this won't be doable, but it'd sure be a nice way to pull the teeth on a SYN flood. FWIW, Cheers, Paul Paul "Corwin" Frommeyer Work Internet Engineer, CCIE Play ISP Systems Engineer Network Sorcerer At Large Cisco Systems, Inc. Paul's Fone Company pfrommey@cisco.com corwin@palas.com *** Speaking solely for myself unless otherwise noted ***

In reply to your message of Thu, 19 Sep 1996 15:22:35 EDT: | I am sure a question most of us has is, what kind of latency does your | filtering box add? Doing something at line rate is fine, but latency is | rather important at line speed. Very low, on the order of tens of microseconds, if I remember correctly (the code itself is very small, only a couple hundred K). The PIX operates by switching on flows rather than routing, so latency is comparable to a switch. However, a word on latency since this urban myth seems to keep creeping back: While a device with large latency, on the order hundreds of milliseconds or even seconds, would obviously contribute some detriment to the data path, ultimately the largest latency lies in the transmission media and the processing overhead on the end stations, and not the network nodes themselves. This is an old issue that goes 'way back, and it just won't seem to die. I never like trying to address the issue of latency in a network device, because invariably it isn't the real contributor to latency on a network. In fact, many of the unwashed in the end user community confuse latency with response time, and they are not the same nor are they necessarily related. Seconds-long response times due to congestion do not mean that forwarding latency is at issue in any network devices, just like a traffic jam at a major turnpike does not mean that the speed limits have been reduced or the road surface degraded to where travel beyond a moderate speed is impossible. There is just simply more traffic than the device can handle, and things are going to back up-- but the packets are still being forwarded through the device at the same rate. Back to the PIX, since it filters and forwards at line rate, packets go out as fast as they come in, eliminating the issue of congestion. And I've already touched on the estimated latency for completeness. Hope this helps, Cheers, Paul Paul "Corwin" Frommeyer Work Internet Engineer, CCIE Play ISP Systems Engineer Network Sorcerer At Large Cisco Systems, Inc. Paul's Fone Company pfrommey@cisco.com corwin@palas.com *** Speaking solely for myself unless otherwise noted ***

So, what does this say? Look for more 13-year-olds causing denial-of-service attacks for the hell of it. It seems a lot of the providers SYN flooders like to attack are the ones which have IRC servers, but the flooders attack the more traditional services of those providers, too.

My outbound filter blocks packets not from an address in my space. Am I wrong in thinking this is the right thing for non-transit networks to do? -- Dick St.Peters, Gatekeeper, Pearly Gateway, Ballston Spa, NY stpeters@NetHeaven.com Owner, NetHeaven 518-885-1295/800-910-6671 Albany/Saratoga/Glens Falls/North Creek/Lake Placid/Blue Mountain Lake First Internet service based in the 518 area code

"Dick St.Peters" writes:

My outbound filter blocks packets not from an address in my space. Am I wrong in thinking this is the right thing for non-transit networks to do?

Yes, it is. If everyone on earth did this, it would make the problem largely go away. I higly recommend that people implement it on a wide scale. Perry