Hi, Just a general note on the UDP 80 style DoS attacks. I'm not entirely certain that UDP 80 attacks are always related to the gameserver bug that you're citing below. We have seen in the wild php scripts that are hard coded to use UDP 80 to deliver DoS attacks towards their targets. Basically it's just GET /script.php?ip.of.victim and instant UDP 80 flood, I've also seen perl versions of the same script.. most notably UDP.PL What would be interesting is to see just how much UDP 80 traffic exists on the Internet at any given moment. I don't know if Arbor's ATLAS really tracks traffic in that way but it would be interesting to get a semi-global view of just how many PPS/BPS are being wasted on these types of floods. Maybe even as a research paper =) -Drew -----Original Message----- From: Fredrik Holmqvist / I2B [mailto:fredrik@i2b.se] Sent: Sunday, February 05, 2012 6:47 PM To: nanog@nanog.org Subject: Re: UDP port 80 DDoS attack Hi. We had a customer that was attacked by the same "game server feature". We received aprox 10 Gbit of traffic against the customer. The attacker sends spoofed packets to the game server with the target IP as "source", the gameserver sends replies back via UDP to the target host. The attacker sends a couple of hundred packets per second and thus generating a 10 Mbit UDP flood. There is fixes/workarounds for the game servers, just a matter of the admin taking care of it. See: http://rankgamehosting.ru/index.php?showtopic=1320 The "attacking" IPs aren't spoofed, so just compile a list and send e-mails to each provider. We had 1000+ IPs gathered and sent 100+ abuse e-mails, only received reply from less than 20%. Sad that people care so little about mitigating DDoS/UDP/ICMP floods. On Sun, 5 Feb 2012 18:36:13 -0500, Ray Gasnick III <rgasnick@milestechnologies.com> wrote:

We just saw a huge flux of traffic occur this morning that spiked one of our upstream ISPs gear and killed the layer 2 link on another becuase of a DDoS attack on UDP port 80.

Wireshark shows this appears to be from a compromised game server (call of duty) with source IPs in a variety of different prefixes.

Only solution thus far was to dump the victim IP address in our block into the BGP Black hole community with one of our 2 providers and completely stop advertising to the other.

Anybody see this recently and have any tips on mitigation, reply on or off list.

Thank You,

Ray Gasnick III CISSP, Technology Specialist: Network Security & Infrastructure Miles Technologies www.milestechnologies.com<http://www.milestechnologies.com/>

Phone: (856) 439-0999 x127 Direct: (856) 793-3821 How am I doing? Email my manager at itmanager@milestechnologies.com<mailto:itmanager@milestechnologies.com

...

Computer Networking – IT Support – Business Software – Website Design – Online Marketing & PR

-- Fredrik Holmqvist I2B (Internet 2 Business) 070-740 5033

On Feb 6, 2012, at 7:21 AM, Keegan Holley wrote:

There aren't very many ways to combat DDOS.

Start with the various infrastructure/host/service BCPs, and S/RTBH, as outlined in this preso: <https://files.me.com/roland.dobbins/dweagy> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> The basis of optimism is sheer terror. -- Oscar Wilde

On Feb 6, 2012, at 8:10 AM, Keegan Holley wrote:

An entire power point just to recommend ACL's, uRPF, CPP, DHCP snooping, and RTBH?

Actually, no, that isn't the focus of the preso.

The first four will not work against a DDOS attack

This is incorrect - suggest you read the preso.

and the last one just kills the patient so he does not infect other patients.

S/RTBH - as opposed to D/RTBH - doesn't kill the patient. Again, suggest you read the preso. There's been a lot of discussion on this topic on NANOG, suggest you take a look through the archives. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> The basis of optimism is sheer terror. -- Oscar Wilde

2012/2/5 Dobbins, Roland <rdobbins@arbor.net>

On Feb 6, 2012, at 8:10 AM, Keegan Holley wrote:

...

An entire power point just to recommend ACL's, uRPF, CPP, DHCP snooping, and RTBH?

Actually, no, that isn't the focus of the preso.

...

The first four will not work against a DDOS attack

This is incorrect - suggest you read the preso.

The ACL's are configured on the routers belonging to the victim AS which will not save their access pipe if it's overrun unless I'm missing something. uRPF may help with spoofed traffic, but sometimes causes problems with multi-homing and is often more harmful than helpful depending on the network design.

...

and the last one just kills the patient so he does not infect other patients.

S/RTBH - as opposed to D/RTBH - doesn't kill the patient. Again, suggest you read the preso.

Source RTBH often falls victim to rapidly changing or spoofed source IP"s. It also isn't as widely supported as it should be. I never said DDOS was hopeless, there just aren't a wealth of defenses against it.

2012/2/5 Dobbins, Roland <rdobbins@arbor.net>

On Feb 6, 2012, at 8:37 AM, Keegan Holley wrote:

...

Source RTBH often falls victim to rapidly changing or spoofed source IP"s.

S/RTBH can be rapidly shifted in order to deal with changing purported source IPs, and it isn't limited to /32s. It's widely supported on Cisco and Juniper gear (flowspec is a better choice on Juniper gear).

I was referring to support from ISP's not from hardware vendors.

If folks don't want to read the presos or search through the archives,

that's fine, of course. The fact is that there are quite a few things that operators can and should do in order to mitigate DDoS attacks; and making the perfect the enemy of the merely good only helps the attackers, doesn't it?

Yes but assuming everything discussed at a conference is instantly adopted by the entire industry gives one false hope no?

Roland, On Mon, Feb 6, 2012 at 2:43 AM, Dobbins, Roland <rdobbins@arbor.net> wrote:

S/RTBH can be rapidly shifted in order to deal with changing purported source IPs, and it isn't limited to /32s.

The big drawback with S/RTBH is that it is a DoS method in itself. Say eyeball provider X has implemented automated S/RTBH, and I have a grudge against them. I would simply DoS a couple of the subscribers with spoofed source IP addresses from google, youtube, netflow and hulu. The automated S/RTBH drops all packets coming from those IP addresses. Presto; many angry consumers call the ISP's helpdesk. The same goes for hosting networks, I just need to identify what kind of service my intended victim is dependent on. (i.e. paypal). Then DoS any part of the hosters network with spoofed source addresses of paypal, the automated S/RTBH makes sure the entire hosting network is not able to reach paypal anymore. Presto, mission achieved. Bas

2012/2/8 George Bonser <gbonser@seven.com>

...

-----Original Message----- From: bas Sent: Tuesday, February 07, 2012 11:56 PM To: Dobbins, Roland; nanog Subject: Re: UDP port 80 DDoS attack

Say eyeball provider X has implemented automated S/RTBH, and I have a grudge against them. I would simply DoS a couple of the subscribers *with spoofed source IP* addresses from google, youtube, netflow and hulu. The automated S/RTBH drops all packets coming from those IP addresses. Presto; many angry consumers call the ISP's helpdesk.

Comes back to providers allowing "spoofed" traffic into their networks from customers. That seems to me to be the low-hanging fruit here.

How do you stop it? Granted, traffic from 10/8 or 127.0.0.1 coming in via an upstream is obvious, but that's about it. There's nothing in a packet that will tell you where it came from compared to the source IP field in the IP header. uRPF is a problem for anyone who's sufficiently multihomed since it causes asymmetric routing.

It works in theory, but to get every ISP and hosting provider to ACL their edges and maintain those ACL's for every customer no matter how large might be a bit difficult. Also, what about non-BGP customers or customers that just accept a default route? Or even customers that just want return traffic to come in a different link for some reason. ISP's would suddenly become giant traffic registries. 2012/2/8 George Bonser <gbonser@seven.com>

...

From: Keegan Holley

...

How do you stop it?

A provider knows what destination IP traffic they route TO a customer, don't they? That should be the only source IPs they accept FROM a customer.

If you don't route it TO the customer, you shouldn't accept it FROM the customer unless you have made special arrangements with them and verified they are entitled to source the traffic from the desired IPs.

No, we have registries to act as registries, the ISPs should be checking them, and double checking. It isn't something that is going to change every day or every week. Once you get it set up, it is going to be stable for a while. Sure, it means a little more work in setting up a customer, but it also means that if all your neighbors do the same thing, you field many fewer calls dealing with stupid DoS crap.

I'll put it another way. Any provider that does not police their customer traffic has no business whining about DoS problems.

77% of all networks seem to think so. http://spoofer.csail.mit.edu/summary.php

And it would be the remaining 23% that really need to understand how difficult they are making life for the rest of the Internet.

However the remaining networks allow spoofed traffic to egress their networks.

When that traffic enters my network, I have no method whatsoever to differentiate it from any other traffic.

I'm not really thinking about traffic coming from the Internet. I'm thinking about its originating location. Correct, once it gets into the Internet, you really have no way to tell.

I could ask my upstream where they see it coming from, which will be quite hard if they do not have pretty fancy systems.

At that point the game is really hard, agreed. And if it is distributed, it could be coming from any number of places or from every single one of their upstreams.

But if they receive it from a peer, I am as good as lost in trying to find the culprit.

Agreed. That's why it is important to stop it at the source.

Bas

Stop paying transit providers for delivering spoofed packets to the edge of your network and they will very quickly develop methods of proving that the traffic isn't spoofed, or block it altogether. =) -Drew -----Original Message----- From: George Bonser [mailto:gbonser@seven.com] Sent: Wednesday, February 08, 2012 1:27 PM To: bas; nanog Subject: RE: UDP port 80 DDoS attack

77% of all networks seem to think so. http://spoofer.csail.mit.edu/summary.php

And it would be the remaining 23% that really need to understand how difficult they are making life for the rest of the Internet.

However the remaining networks allow spoofed traffic to egress their networks.

When that traffic enters my network, I have no method whatsoever to differentiate it from any other traffic.

I'm not really thinking about traffic coming from the Internet. I'm thinking about its originating location. Correct, once it gets into the Internet, you really have no way to tell.

I could ask my upstream where they see it coming from, which will be quite hard if they do not have pretty fancy systems.

At that point the game is really hard, agreed. And if it is distributed, it could be coming from any number of places or from every single one of their upstreams.

But if they receive it from a peer, I am as good as lost in trying to find the culprit.

Agreed. That's why it is important to stop it at the source.

Bas

2012/2/8 Steve Bertrand <steve.bertrand@gmail.com>

On 2012.02.08 14:23, Drew Weaver wrote:

...

Stop paying transit providers for delivering spoofed packets to the edge of your network and they will very quickly develop methods of proving that the traffic isn't spoofed, or block it altogether. =)

I firmly believe in this recourse, amongst others...

How do you tell the spoofed packets from the non-spoofed ones? Especially if you have more than one provider.

If you know that your provider allows spoofed traffic, let the community know about it.

According to a company wide NDA I'm only allowed to disclose that to the best of my knowledge my upstreams permits packets sent from users or other NSP's who may or may not permit or generate packets. The source IP addresses are checked to be valid 32 bit numbers before being sent to my routers. My upstreams to the best of their knowledge have never sent me a single spoofed packet and will refrain from doing so unless they receive written consent from me, in triplicate. ;)

In all aspects of life, a problem must be 'fixed' at the source. All of the small-medium size ops have to connect to the big-boys somewhere, and what I've seen in this industry is that the big-boys are generally compliant.

As long as compliant means completely indifferent to your concerns and unwilling to change or compromise in any meaningful while sucking money away faster than the government. They are all very very compliant and a pleasure to do business with.

Providers don't even check the registries for bgp advertisements. See the thread on hijacked routes for proof. Not to mention how do you handle a small transit AS? Do you trust that they have the correct filters as well? Do you start reading their AS paths and try to filter based on the registry for folks down stream? Then there's the RLDRAM issue. Most edge boxes will just run out if ACL's. Lastly there's no contractual obligation to play traffic cop for the entire Internet so providers would be dropping traffic that they can legitimately bill for. Sent from my iPhone On Feb 8, 2012, at 4:56 AM, George Bonser <gbonser@seven.com> wrote:

...

No, we have registries to act as registries, the ISPs should be checking them, and double checking. It isn't something that is going to change every day or every week. Once you get it set up, it is going to be stable for a while. Sure, it means a little more work in setting up a customer, but it also means that if all your neighbors do the same thing, you field many fewer calls dealing with stupid DoS crap.

I'll put it another way. Any provider that does not police their customer traffic has no business whining about DoS problems.

-----Original Message----- From: christopher.morrow

to be fair: "Some Providers do not check registries for 'right to use' information about prefixes their customers wish to announce to them over BGP."

Maybe not but I would think that in practice it would be something like: 1. Provider initially filters traffic based on the address range they have issued to the customer. 2. If the customer brings their own IP addresses, the provider does a quick check to see if those have been SWIPed to the customer 3. If the customer wants the filtration opened up to include additional IPs, the do the same as #2 4. If the customer has no record of having control of those IPs, a quick call to the listed assignee of those numbers would verify that the customer is mutual and is properly sourcing traffic in that IP range and filters are adjusted accordingly. In about 99% of cases that would be the end of the story and everything runs merrily along after that. Sure, there are going to be corner cases but if someone starts playing whack-a-mole with IP address assignments and is asking for frequent changes, that might be a tip-off that they might be trouble. It *does* involve maintaining some record of the configuration settings someplace in case of equipment changes/failures, etc. but that would be a small price to pay for reducing the amount of time spent chasing DoS complaints. It has to be a community effort with a set of best practices developed and applied by the community.

On Feb 8, 2012, at 4:51 AM, George Bonser <gbonser@seven.com> wrote:

...

From: Keegan Holley Subject: Re: UDP port 80 DDoS attack

...

It works in theory, but to get every ISP and hosting provider to ACL their edges and maintain those ACL's for every customer no matter how large might be a bit difficult.

You don't have to ACL in most cases. RPF works for most. There will be a few, relatively darned few, that you will need to ACL, but RPF takes care of a large number of them.

RPF on the whole Internet would pretty much lead to an instant outage. What happens when you have two upstreams and one has an incoming route to you but your out going route for which ever of their customers talking to you doesn't agree? Instant outage. Multiply that by the entire table and then add churn. I'd give it a week before everyone turned it off, if you could even get them to enable it to begin with.

...

Also, what about non-BGP customers or customers that just accept a default route?

I don't follow. The ISP still knows what traffic gets routed TO them. You only accept FROM them what you route TO them, even if you hand them a default route.

...

Or even customers that just want return traffic to come in a different link for some reason.

Still don't follow. I am not talking about having a firewall that is stateful. I am talking packet by packet. If you don't route it to them, you don't accept it from them unless you have made arrangements otherwise, which will be a small percentage of your customers. Sure, some might be multihomed but it is easy enough to verify that they have the networks in question SWIPed to them or a call to the other provider can clear that up in a few minutes. It isn't THAT hard.

...

ISP's would suddenly become giant traffic registries.

No, we have registries to act as registries, the ISPs should be checking them, and double checking. It isn't something that is going to change every day or every week. Once you get it set up, it is going to be stable for a while. Sure, it means a little more work in setting up a customer, but it also means that if all your neighbors do the same thing, you field many fewer calls dealing with stupid DoS crap.

On Wed, Feb 8, 2012 at 9:29 AM, Dobbins, Roland <rdobbins@arbor.net> wrote:

On Feb 8, 2012, at 2:56 PM, bas wrote:

...

The big drawback with S/RTBH is that it is a DoS method in itself.

I'm not an advocate of *automated* S/RTBH, and I am an advocate of whitelisting various well-known 'golden networks/IPs'

So I would need to find out which networks you would have classified as "golden" and use those as sources for my DDoS. Either I can achieve DoS with S/RTBH, or I can abuse the "golden networks" to circumvent S/RTBH. As far as I see it S/RTBH is in no way a solution against smart attackers, of course it does help against all the kiddie attacks out there. Bas

2012/2/8 Dobbins, Roland <rdobbins@arbor.net>

On Feb 8, 2012, at 8:07 PM, bas wrote:

...

As far as I see it S/RTBH is in no way a solution against smart attackers, of course it does help against all the kiddie attacks out there.

Once again, I've used S/RTBH myself and helped others use it many, many times, including to defend against attacks with shifting purported source IPs. flowspec, IDMS and other tools are very useful as well, but S/RTBH is supported on a lot of hardware, if operators choose to configure it.

It is not a panacea. It is one tool in the toolbox.

Folks can either choose to make use of it or choose not to do so; it is operationally proven, it does work, and it's certainly better than nothing. YMMV.

I agree. I think RTBH is a broadsword not a scalpel. It's a tool in the tool box and there is a danger of dropping legitimate traffic with both S/RTBH and D/RTBH. BGP isn't a security protocol. It's not even that great of a routing protocol.

2012/2/5 Steve Bertrand <steve.bertrand@gmail.com>

On 2012.02.05 20:37, Keegan Holley wrote:

...

2012/2/5 Dobbins, Roland<rdobbins@arbor.net>

S/RTBH - as opposed to D/RTBH - doesn't kill the patient. Again, suggest

...

...

you read the preso.

Source RTBH often falls victim to rapidly changing or spoofed source IP"s. It also isn't as widely supported as it should be. I never said DDOS was hopeless, there just aren't a wealth of defenses against it.

This is so very easily automated. Even if you don't actually want to trigger the routes automatically, finding the sources you want to blackhole is as simple as a monitor port, tcpdump and some basic Perl.

This is still vulnerable to spoofing which could cause you to filter legitimate traffic and make the problem worse. Not saying that S/RTBH is a bad idea. RTBH is effective and a great idea just not very elegant.

...and as far as this not having been deployed in many ISPs (per your next message)... their mitigation strategies should be asked up front, and if they don't have any (or don't know what you speak of), find a new ISP.

You sometimes have to weigh the pro's and cons. You can't always pick the guys with the coolest knobs.

On Sun, Feb 5, 2012 at 10:08 PM, Steve Bertrand <steve.bertrand@gmail.com> wrote:

This is so very easily automated. Even if you don't actually want to trigger the routes automatically, finding the sources you want to blackhole is as

What transit providers are doing flow-spec, or otherwise, to allow their downstreams to block malicious traffic by SOURCE address? -- Jeff S Wheeler <jsw@inconcepts.biz> Sr Network Operator  /  Innovative Network Concepts

...

It also isn't as widely supported as it should be. I never said DDOS was hopeless, there just aren't a wealth of defenses against it.

there is a fix for it, it's called "putting a fuckton of ram in -most- routers on the internet" and keeping statistics for each destination ip:destination port:outgoing interface so that none of them individually can (entirely/procentually compared to other traffic) flood the outgoing interface on that router... end result, if enough routers are structured like that, is that ddos attacks will be come completely useless. keyword here, is terabytes of ram. statistic tables on very basic ipv4 stuff alone would already take multiple 100's of gigabytes... (keep in mind, this won't be "cpu work", its just using the table entry size * dest ip as an offset and reading it out ;) the good news is, ram doesn't cost a flying fuck anymore... it just requires a complete re-think of router design, but then again, with the prices that cisco and juniper charge we expect a bit more than a 1960's telephone exchange look and feel device, or we might as well use 1 linux box/blade per 20gbit/s throughput and consider that whole thing a "hotpluggable interface". cisco and juniper, at the moment, sell faster versions of their crap from the 1990s, but not much effort went into completely re-designing the stuff to be suitable for the internet as it is today, they still forward all packets they can get their hands on, and besides rather simple stuff like QoS, not much effort went into inteligently spreading the capacity available on the outgoing interface (at least for that individual router) over all the possible targets. now, if an outgoing interface is 10ge, and 10mbit traffic should go to 1.2.3.4 and 20ge (ddos) traffic should go to 4.3.2.1, i'd say, a router should give 1.2.3.4 the full 10ge, as it is available, and lower volume should basically just get a higher priority. we have not quite worked out the formula yet... but it should be something along those lines (simply to say, any destination ip can never fill more than half of the outgoing interface, doesn't quite cover it, it needs some "intelligent adjustment of the percentage").. basically... table: destip destport packetcounter every so and so many packets/timeslices,whatever that packetcounter is decreased by 1 every packet, the packet counter is incremented after inactivity for that ip for timeperiod x, the packet counter is cleared. with ipv4, this "destip" entry is such a small (32 bit) integer that its better to just not store the ip itself but rather just throw more ram at it and use the ip address number as the offset in the array (for faster lookups, preferably in hardware logic). the same thing could be applied to pretty much every other concept still done with slow lookups nowadays (arp, routes, etc)... throw more ram at it and use the destination as the offset, who cares if 99.9% of the ram is not actually used. for the price of a juniper, you can buy a truck full of ram chips ;).. it's faster that way, and it allows us to do a lot of things, like not passing potential ddos floods in the first place, and intelligently allowing everyone, not an equal share of the traffic capacity, but the part they need. if you don't mind wasting 50% of the available capacity the formula to determine wether to forward a packet or not is quite simple, if you also want to give a destip 100% of the traffic in situations where there is no other traffic, it becomes a bit more complicated.. as stated before, we haven't quite worked it out in full yet, but in any case... this would require for at least 30% of the routers that currently are on the internet and are 110 kg heavy "1960s telephone exchange models" to be replaced with 2012 style hardware, not "just faster cisco 7200 like things".

2012/2/6 Jeff Wheeler <jsw@inconcepts.biz>

On Mon, Feb 6, 2012 at 8:43 PM, Sven Olaf Kamphuis <sven@cb3rob.net> wrote:

...

there is a fix for it, it's called "putting a fuckton of ram in -most- routers on the internet" and keeping statistics for each destination ip:destination port:outgoing interface so that none of them individually can (entirely/procentually compared to other traffic) flood the outgoing interface on that router... end result, if enough routers are structured like that, is that ddos attacks will be come completely useless.

There are two obvious problems with your approach.

First, adding the policers you suggest, at the scale needed, is a little harder than you imagine. It's not a simple matter of the cost of RAM but also power/heat density per port.

Since when are policers implemented in ram? You're talking FPGA if you want to be able to make forwarding/filtering decisions assuming it's possible which it isn't you're 1 million dollar boxes suddenly become hundred million dollar boxes. Then there's v6 info..

Second, if you re-engineer every router on the Internet to prevent an interface from being congested by malicious flow(s) destined for one particular destination IP:port, then DDoS attacks will simply target multiple ports or multiple destination IP addresses that are likely to traverse a link they are able to congest.

Not to mention that the routers themselves become an attack vector. This cache will have a finite limit because there's no such thing as an infinite amount of cache among other flaws. When that limit is reached it will do something no one want's it to do and that will become the new DDOS attack.

If you want to dramatically increase the cost of routers in order to solve the problem of DDoS with one deft (and expensive) move, you have to imagine that the people behind DDoS attacks aren't complete idiots, and will actually spend some time thinking about how to defeat your system.

Not to mention cost? You're not going to get a tier I ISP to upgrade to

this new super router (assuming it's possible to build such a things) without an act of congress or at least the FCC. They won't even spend enough on fiber to bring broadband into rural areas.

Of course it's not possible ... if you use a crummy design. It's trivial to come up with non-completely-crummy designs. For example, adding a front-end where you take a hash of source-ip/dest-ip and run it through a smallish hash table, you can use that as a filter to eliminate a lot of traffic that's just normal and non-interesting. You want to take a closer look at the traffic that's heaviest (read: most hits) or new and significant (read: diff against an hour ago). You probably don't want to do this just per-IP, but likely also per- network.

I think one of the problems is that with modern bot-nets, traffic can be generated by a huge number of hosts and assuming your DoS traffic is coming from a source that can be blocked might be an unreasonable expectation. You can't assume that you are going to get a flood of traffic from some source that you can pin down and block. You might get flood of traffic from tens of thousands of source IPs from all over the world with each one sending only a very small amount AND the source IPs constantly changing. They might even be sending traffic that looks perfectly legitimate on the surface and might need to be profiled/fingerprinted in some manner at layer 4. It isn't as easy as just handling it at the router. And there is no guarantee the source IP of the traffic is really where it is coming from since there are still a good number of providers out there who don't install packet filters on their customer links. They accept any traffic their customer sends them even if the source IP isn't within the customer's network range. So that is part of the game, too. If you have 10,000 hosts sending packets with spoofed IP addresses where the goal is to get you to block the apparent source network, as soon as you block those source addresses, the DoS has succeeded.

And you probably don't want to use just this one technique, you want to combine it with others.

I think "probably" is the wrong word here. The word "certainly" leaps to mind.

And you probably need to consider the types of attacks that are known, likely, etc., and design accordingly, because this one little example I've provided is just one part of a comprehensive solution, but it is capable of dealing with any amount of traffic and it would be a very useful filter to start pulling out potentially interesting stuff.

The problem is that you have a game of cat and mouse with what amounts to an infinite supply of mice. It takes cooperation between the source and the provider networks. The "eyeball" heavy networks need to ensure they can't source bogus traffic. Having gear these days where the ACLs are in hardware has greatly reduced the CPU expense of filtering on edge ports but the human resource expense of maintaining those is still high unless automation is brought into the mix so that those filters are changed when the addresses served by a port change.

This stuff isn't *easy*. Fine. But it certainly *is* possible.

Of course it isn't easy. It is designed to be difficult. But there is plenty of "low hanging fruit" out there still.