This evening all of my servers lost NTP sync, stating that our on-site NTP servers hadn't synced in too long. Reference time noted by the local NTP servers: Fri, Jan 31 2014 19:11:29.725 Apparently since then, NTP has been unable to traverse the circuit. Our other provider is shuffling NTP packets just fine, and after finding an NTP peer that return routed in that direction, I was able to get NTP back in shape. Spot checking various NTP peers configured on my end with various looking glasses close to the far-end confirm that anytime the return route is through AS11351, we never get the responses. Outbound routes almost always take the shorter route through our other provider. Is anyone else seeing this, or am I lucky enough to have it localized to my region (Northern NY)? I've created a ticket with the provider, although with it being the weekend, I have doubts it'll be a quick resolution. I'm sure its a strange knee-jerk response to the monlist garbage. Still, stopping time without warning is Uncool, Man. -- Jonathan Towne
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 While I do not profess to know the cause of your particular NTP sync problem, this *might* be due to knee-jerk reactions to the NTP reflection/amplification DDoS attacks that have been quite an annoyance and operational issue lately. suspect that some operators have found that perhaps they harbored some device inside their own networks are being used (or might be used) to facilitate these attacks: https://www.us-cert.gov/ncas/current-activity/2014/01/10/Network-Time-Protoc... See also: http://openntpproject.org/ - - ferg On 2/1/2014 8:03 PM, Jonathan Towne wrote:
This evening all of my servers lost NTP sync, stating that our on-site NTP servers hadn't synced in too long.
Reference time noted by the local NTP servers: Fri, Jan 31 2014 19:11:29.725
Apparently since then, NTP has been unable to traverse the circuit. Our other provider is shuffling NTP packets just fine, and after finding an NTP peer that return routed in that direction, I was able to get NTP back in shape.
Spot checking various NTP peers configured on my end with various looking glasses close to the far-end confirm that anytime the return route is through AS11351, we never get the responses. Outbound routes almost always take the shorter route through our other provider.
Is anyone else seeing this, or am I lucky enough to have it localized to my region (Northern NY)?
I've created a ticket with the provider, although with it being the weekend, I have doubts it'll be a quick resolution. I'm sure its a strange knee-jerk response to the monlist garbage. Still, stopping time without warning is Uncool, Man.
-- Jonathan Towne
- -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlLubvMACgkQKJasdVTchbK8mwD9HDHJ2YSDciN8k6YkRDu4MbxS r0zEU/8ofP8HaK8YoEYBANhDP+VIhC3Cz/cKc4TI8WeGHqX1ZWN1OwnxLihR3sjx =KEeR -----END PGP SIGNATURE-----
The provider has kindly acknowledged that there is an issue, and are working on a resolution. Heads up, it may be more than just my region. -- Jonathan Towne On Sat, Feb 01, 2014 at 11:03:19PM -0500, Jonathan Towne scribbled: # This evening all of my servers lost NTP sync, stating that our on-site NTP # servers hadn't synced in too long. # # Reference time noted by the local NTP servers: # Fri, Jan 31 2014 19:11:29.725 # # Apparently since then, NTP has been unable to traverse the circuit. Our # other provider is shuffling NTP packets just fine, and after finding an # NTP peer that return routed in that direction, I was able to get NTP back # in shape. # # Spot checking various NTP peers configured on my end with various looking # glasses close to the far-end confirm that anytime the return route is # through AS11351, we never get the responses. Outbound routes almost always # take the shorter route through our other provider. # # Is anyone else seeing this, or am I lucky enough to have it localized to # my region (Northern NY)? # # I've created a ticket with the provider, although with it being the weekend, # I have doubts it'll be a quick resolution. I'm sure its a strange knee-jerk # response to the monlist garbage. Still, stopping time without warning is # Uncool, Man. # # -- Jonathan Towne #
In article <20140202163313.GF24634@hijacked.us> you write:
The provider has kindly acknowledged that there is an issue, and are working on a resolution. Heads up, it may be more than just my region.
I'm a Time-Warner cable customer in the Syracuse region, and both of the NTP servers on my home LAN are happily syncing with outside peers. My real servers are hosted in Ithaca, with T-W being one of the upstreams and they're also OK. They were recruited into an NTP DDoS last month (while I was at a meeting working on anti-DDoS best practice, which was a little embarassing) but they're upgraded and locked down now. R's, John
The recently publicized mechanism to leverage NTP servers for amplified DoS attacks is seriously effective. I had a friend who had a local ISP affected by this Thursday and also another case where just two asterisk servers saturated a 100mbps link to the point of unusability. Once more - this exploit is seriously effective at using bandwidth by reflection. From a provider point of view, given the choices between contacting the end-users vs. mitigating the problem, if I were in TW position if I was unable to immediately contact the numerous downstream customers that were affected by this, I would take the option to block NTP on a case-by-case basis (perhaps even taking a broad brush) rather than allow it to continue and cause disruptions elsewhere. - Mike On Feb 2, 2014, at 12:44 PM, John Levine <johnl@iecc.com> wrote:
In article <20140202163313.GF24634@hijacked.us> you write:
The provider has kindly acknowledged that there is an issue, and are working on a resolution. Heads up, it may be more than just my region.
I'm a Time-Warner cable customer in the Syracuse region, and both of the NTP servers on my home LAN are happily syncing with outside peers.
My real servers are hosted in Ithaca, with T-W being one of the upstreams and they're also OK. They were recruited into an NTP DDoS last month (while I was at a meeting working on anti-DDoS best practice, which was a little embarassing) but they're upgraded and locked down now.
R's, John
On Feb 3, 2014, at 12:45 PM, Michael DeMan <nanog@deman.com> wrote:
From a provider point of view, given the choices between contacting the end-users vs. mitigating the problem, if I were in TW position if I was unable to immediately contact the numerous downstream customers that were affected by this, I would take the option to block NTP on a case-by-case basis (perhaps even taking a broad brush) rather than allow it to continue and cause disruptions elsewhere.
Per my previous post in this thread, there are ways to do this without blocking client access to ntp servers; in point of fact, unless the ISP in question isn't performing antispoofing at their customer aggregation edge, blocking client access to ntp servers does nothing to address (pardon the pun) the issue of ntp reflection/amplification DDoS attacks. All that broadband access operators need to do is to a) enforce antispoofing as close to their customers as possible, and b) enforce their AUPs (most broadband operators prohibit operating servers) by blocking *inbound* UDP/123 traffic towards their customers at the customer aggregation edge (same for DNS, chargen, and SNMP). ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Feb 3, 2014, at 1:02 PM, Dobbins, Roland <rdobbins@arbor.net> wrote:
b) enforce their AUPs (most broadband operators prohibit operating servers) by blocking *inbound* UDP/123 traffic towards their customers at the customer aggregation edge
Actually, this can cause problems for ntpds operating in symmetric mode, where both the source and destination ports are UDP/123. Allowing inbound UDP/123 - UDP/123 and then rate-limiting it would be one approach; another would be to block outbound UDP/123 emanating from customers based upon packet size, if one's hardware allows matching on size in ACLs. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Feb 2, 2014, at 10:02 PM, Dobbins, Roland <rdobbins@arbor.net> wrote:
On Feb 3, 2014, at 12:45 PM, Michael DeMan <nanog@deman.com> wrote:
From a provider point of view, given the choices between contacting the end-users vs. mitigating the problem, if I were in TW position if I was unable to immediately contact the numerous downstream customers that were affected by this, I would take the option to block NTP on a case-by-case basis (perhaps even taking a broad brush) rather than allow it to continue and cause disruptions elsewhere.
Per my previous post in this thread, there are ways to do this without blocking client access to ntp servers; in point of fact, unless the ISP in question isn't performing antispoofing at their customer aggregation edge, blocking client access to ntp servers does nothing to address (pardon the pun) the issue of ntp reflection/amplification DDoS attacks.
Agreed, and I was not trying to get into arguments about saying whether 'blocking' is appropriate or not. I was simply suggesting that a provider, if they found themselves in a position where this was causing lots of troubles and impacting things in a large, might have had taken a 'broad brush' approach to stabilize things while they work on a more proper solution.
All that broadband access operators need to do is to a) enforce antispoofing as close to their customers as possible, and b) enforce their AUPs (most broadband operators prohibit operating servers) by blocking *inbound* UDP/123 traffic towards their customers at the customer aggregation edge (same for DNS, chargen, and SNMP).
I certainly would not want to provide as part the AUP (as seller or buyer), a policy that fundamentals like NTP are 'blocked' to customers. Seems like too much of a slippery slope for my taste. In regards to anti-spoofing measures - I think there a couple of vectors about the latest NTP attack where more rigorous client-side anti-spoofing could help but will not solve it overall. Trying to be fair and practical (from my perspective) - it is a lot easier and quicker to patch/workaround IPv4 problems and address proper solutions via IPv6 and associated RFCs? - Michael DeMan
----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Luck is the residue of opportunity and design.
-- John Milton
On Feb 3, 2014, at 1:54 PM, Michael DeMan <nanog@deman.com> wrote:
I certainly would not want to provide as part the AUP (as seller or buyer), a policy that fundamentals like NTP are 'blocked' to customers. Seems like too much of a slippery slope for my taste.
The idea is to block traffic to misconfigured ntpds on broadband customer access networks, not to limit their choice of which ntp servers to use.
In regards to anti-spoofing measures - I think there a couple of vectors about the latest NTP attack where more rigorous client-side anti-spoofing could help but will not solve it overall.
Rigorous antispoofing would solve the problem of all reflection/amplification DDoS attacks. My hunch is that most spoofed traffic involved in these attacks actually emanates from compromised/abused servers on IDC networks (including so-called 'bulletproof' miscreant-friendly networks), but I've no data to support that, yet.
Trying to be fair and practical (from my perspective) - it is a lot easier and quicker to patch/workaround IPv4 problems and address proper solutions via IPv6 and associated RFCs?
There's nothing in IPv6 which makes any difference. The ultimate solution is antispoofing at the customer edge. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Mon, 3 Feb 2014 07:08:25 +0000 "Dobbins, Roland" <rdobbins@arbor.net> wrote:
There's nothing in IPv6 which makes any difference. The ultimate solution is antispoofing at the customer edge.
There is at least one small thing that may change some part of this and similar problems. If the threat vector were only accessible on IPv6 and that service on those systems is not easily discoverable then it will probably reduce the total population of systems being abused. I do realize in practice there are ways to discover systems, but the change in address architecture could change things, not perfectly, but I'll venture to suggest noticeably in some not so difficult to imagine scenarios. John
On Feb 4, 2014, at 5:38 AM, John Kristoff <jtk@cymru.com> wrote:
I do realize in practice there are ways to discover systems, but the change in address architecture could change things, not perfectly, but I'll venture to suggest noticeably in some not so difficult to imagine scenarios.
I know you're aware of the work in this area, so I'll just note for the record that the 'IPv6 makes scanning impossible!' has already been exploded. ;> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
In regards to anti-spoofing measures - I think there a couple of vectors about the latest NTP attack where more rigorous client-side anti-spoofing could help but will not solve it overall.
Most NTP servers only send legitimate traffic to a handful of masters, often in the ntp.org pool, and to peers and clients on their own network. I know that when I adjusted my NTP config to stop responding to traffic other than its ntp.org masters and the local LAN, the outbound DDoS traffic stopped. It took a while for the bad guys to notice, so I added some packet filters to limit the load on the NTP daemon. It seems thata hosts sending large amounts of NTP traffic over the public Internet can be safely filtered if you don't already know that it's one of the handful that's in the ntp.org pools or another well known NTP master. R's, John
On 03 Feb 2014 18:23:31 +0000, "John Levine" said:
It seems thata hosts sending large amounts of NTP traffic over the public Internet can be safely filtered if you don't already know that it's one of the handful that's in the ntp.org pools or another well known NTP master.
You have that backwards - you can (possibly) safely filter it if you can establish *for certain* that it *isn't* in the ntp.org pools.
On 2/3/14, 1:02 AM, "Dobbins, Roland" <rdobbins@arbor.net> wrote:
All that broadband access operators need to do is to a) enforce antispoofing as close to their customers as possible,
Many probably do so already. But as well all know, it only takes a few that don¹t to create a large scale issue. IMHO if we want to change this we need to (1) measure where and how anti-spoofing is not implemented, (2) contact networks that have not implemented with recommendations on how to do so (a la bcp38.info), and (3) failing reasonable response to #2 to start a public running list of networks that have inadequate levels of anti-spoofing (according to some reasonable standard, and probably more nuanced than a binary result). Jason
On Feb 3, 2014, at 12:45 AM, Michael DeMan <nanog@deman.com> wrote:
The recently publicized mechanism to leverage NTP servers for amplified DoS attacks is seriously effective. I had a friend who had a local ISP affected by this Thursday and also another case where just two asterisk servers saturated a 100mbps link to the point of unusability. Once more - this exploit is seriously effective at using bandwidth by reflection.
The challenge I see is there's some hosts like this one: [jared@nowherelikehome ]$ ntpq -c rv 111.107.252.142 associd=0 status=06f4 leap_none, sync_ntp, 15 events, freq_mode, version="ntpd 4.2.0-r Fri Jul 22 09:50:16 JST 2011 (1)", processor="seil5", system="NetBSD/3.1_STABLE", leap=00, stratum=5, precision=-18, rootdelay=9.138, rootdispersion=132.247, peer=58012, refid=172.22.203.213, reftime=d685a094.9c806290 Sun, Jan 19 2014 0:53:40.611, poll=10, clock=d69a5d3c.c6b1a2a4 Mon, Feb 3 2014 18:23:56.776, state=4, offset=-0.598, frequency=-1.463, jitter=0.229, stability=0.042 This host will happily generate 100GB response to a single packet. They even have advisories posted: http://www.seil.jp/support/security/a01411.html Getting the information into the admin is hard. Time zones, language barriers, folks understanding why having unmaintained NTP hosts out there can be a significant issue. We found many ILO/IPMI interfaces that have NTP you can't do anything about (no filters, etc) - let alone patch .. Through ACL (hopefully not) or folks fixing hosts the following trend is observable in # of unique hosts that respond to NTP packets: 1529866 2014-01-10 1402569 2014-01-17 803156 2014-01-24 564027 2014-01-31 I will say that an awful lot of "firewall" operators out there seem to now be saying "NTP BAD" and generating panic'ed emails about NTP traffic. - Jared
On Feb 2, 2014 8:35 AM, "Jonathan Towne" <jtowne@slic.com> wrote:
The provider has kindly acknowledged that there is an issue, and are working on a resolution. Heads up, it may be more than just my region.
And not just your provider, everyone is dealing with UDP amp attacks. These UDP based amp attacks are off the charts. Wholesale blocking of traffic at the protocol level to mitigate 10s to 100s of gigs of ddos traffic is not "knee jerk", it is the right thing to do in a world where bcp 38 is far from universal and open dns servers, ntp, chargen, and whatever udp 172 is run wild. People who run networks know what it takes to restore service. And increasingly, that will be clamping ipv4 UDP in the plumbing, both reactively and proactively. And, i agree bcp38 would help but that was published 14 years ago. CB
-- Jonathan Towne
On Sat, Feb 01, 2014 at 11:03:19PM -0500, Jonathan Towne scribbled: # This evening all of my servers lost NTP sync, stating that our on-site NTP # servers hadn't synced in too long. # # Reference time noted by the local NTP servers: # Fri, Jan 31 2014 19:11:29.725 # # Apparently since then, NTP has been unable to traverse the circuit. Our # other provider is shuffling NTP packets just fine, and after finding an # NTP peer that return routed in that direction, I was able to get NTP back # in shape. # # Spot checking various NTP peers configured on my end with various looking # glasses close to the far-end confirm that anytime the return route is # through AS11351, we never get the responses. Outbound routes almost always # take the shorter route through our other provider. # # Is anyone else seeing this, or am I lucky enough to have it localized to # my region (Northern NY)? # # I've created a ticket with the provider, although with it being the weekend, # I have doubts it'll be a quick resolution. I'm sure its a strange knee-jerk # response to the monlist garbage. Still, stopping time without warning is # Uncool, Man. # # -- Jonathan Towne #
On Sun, Feb 2, 2014 at 2:17 PM, Cb B <cb.list6@gmail.com> wrote:
On Feb 2, 2014 8:35 AM, "Jonathan Towne" <jtowne@slic.com> wrote:
The provider has kindly acknowledged that there is an issue, and are working on a resolution. Heads up, it may be more than just my region.
And not just your provider, everyone is dealing with UDP amp attacks.
These UDP based amp attacks are off the charts. Wholesale blocking of traffic at the protocol level to mitigate 10s to 100s of gigs of ddos traffic is not "knee jerk", it is the right thing to do in a world where bcp 38 is far from universal and open dns servers, ntp, chargen, and whatever udp 172 is run wild.
People who run networks know what it takes to restore service. And increasingly, that will be clamping ipv4 UDP in the plumbing, both reactively and proactively.
Please note that it's not that UDP is at fault here; it's applications that are structured to respond to small input packets with large responses. If NTP responded to a single query with a single equivalently sized response, its effectiveness as a DDoS attack would be zero; with zero amplification, the volume of attack traffic would be exactly equivalent to the volume of spoofed traffic the originator could send out in the first place. I agree the source obfuscation aspect of UDP can be annoying, from the spoofing perspective, but that really needs to be recognized to be separate from the volume amplification aspect, which is an application level issue, not a protocol level issue. Thanks! Matt PS--yes, I know it would completely change the dynamics of the internet as we know it today to shift to a 1:1 correspondence between input requests and output replies...but it *would* have a nice side effect of balancing out traffic ratios in many places, altering the settlement landscape in the process. ;)
On Feb 2, 2014 2:54 PM, "Matthew Petach" <mpetach@netflight.com> wrote:
On Sun, Feb 2, 2014 at 2:17 PM, Cb B <cb.list6@gmail.com> wrote:
On Feb 2, 2014 8:35 AM, "Jonathan Towne" <jtowne@slic.com> wrote:
The provider has kindly acknowledged that there is an issue, and are working on a resolution. Heads up, it may be more than just my
region.
And not just your provider, everyone is dealing with UDP amp attacks.
These UDP based amp attacks are off the charts. Wholesale blocking of traffic at the protocol level to mitigate 10s to 100s of gigs of ddos traffic is not "knee jerk", it is the right thing to do in a world where bcp 38 is far from universal and open dns servers, ntp, chargen, and whatever udp 172 is run wild.
People who run networks know what it takes to restore service. And increasingly, that will be clamping ipv4 UDP in the plumbing, both reactively and proactively.
Please note that it's not that UDP is at fault here; it's applications that are structured to respond to small input packets with large responses.
I dont want to go into fault, there is plenty of that to go around.
If NTP responded to a single query with a single equivalently sized response, its effectiveness as a DDoS attack would be zero; with zero amplification, the volume of attack traffic would be exactly equivalent to the volume of spoofed traffic the originator could send out in the first place.
I agree the source obfuscation aspect of UDP can be annoying, from the spoofing perspective, but that really needs to be recognized to be separate from the volume amplification aspect, which is an application level issue, not a protocol level issue.
Source obfuscation is not annoying. Combined with amplification, it is the perfect storm for shutting down networks... whereby the only solution is to shutdown ipv4 udp. Or wave the magic wand that makes bcp38 universal, patches boxes, and so on. My point is: dont expect these abbused services on UDP to last. We have experience in access networks on how to deal with abused protocols. Here is one reference http://customer.comcast.com/help-and-support/internet/list-of-blocked-ports/ My crystal ball says all of UDP will show up soon. CB
Thanks!
Matt PS--yes, I know it would completely change the dynamics of the internet as we know it today to shift to a 1:1 correspondence between input requests and output replies...but it *would* have a nice side effect of balancing out traffic ratios in many places, altering the settlement landscape in the process. ;)
I'd hate to think that NetOps would be so heavy handed in blocking all of UDP, as this would essentially halt quite a bit of audio/video traffic. That being said, there's still quite the need for protocol improvement when making use of UDP, but blocking UDP as a whole is definitely not a resolution, and simply creating a wall that not only keeps the abusive traffic out, but keeps legitimate traffic from flowing freely as it should. Sent on the TELUS Mobility network with BlackBerry -----Original Message----- From: Cb B <cb.list6@gmail.com> Date: Sun, 2 Feb 2014 15:09:55 To: Matthew Petach<mpetach@netflight.com> Cc: <nanog@nanog.org> Subject: Re: TWC (AS11351) blocking all NTP? On Feb 2, 2014 2:54 PM, "Matthew Petach" <mpetach@netflight.com> wrote:
On Sun, Feb 2, 2014 at 2:17 PM, Cb B <cb.list6@gmail.com> wrote:
On Feb 2, 2014 8:35 AM, "Jonathan Towne" <jtowne@slic.com> wrote:
The provider has kindly acknowledged that there is an issue, and are working on a resolution. Heads up, it may be more than just my
region.
And not just your provider, everyone is dealing with UDP amp attacks.
These UDP based amp attacks are off the charts. Wholesale blocking of traffic at the protocol level to mitigate 10s to 100s of gigs of ddos traffic is not "knee jerk", it is the right thing to do in a world where bcp 38 is far from universal and open dns servers, ntp, chargen, and whatever udp 172 is run wild.
People who run networks know what it takes to restore service. And increasingly, that will be clamping ipv4 UDP in the plumbing, both reactively and proactively.
Please note that it's not that UDP is at fault here; it's applications that are structured to respond to small input packets with large responses.
I dont want to go into fault, there is plenty of that to go around.
If NTP responded to a single query with a single equivalently sized response, its effectiveness as a DDoS attack would be zero; with zero amplification, the volume of attack traffic would be exactly equivalent to the volume of spoofed traffic the originator could send out in the first place.
I agree the source obfuscation aspect of UDP can be annoying, from the spoofing perspective, but that really needs to be recognized to be separate from the volume amplification aspect, which is an application level issue, not a protocol level issue.
Source obfuscation is not annoying. Combined with amplification, it is the perfect storm for shutting down networks... whereby the only solution is to shutdown ipv4 udp. Or wave the magic wand that makes bcp38 universal, patches boxes, and so on. My point is: dont expect these abbused services on UDP to last. We have experience in access networks on how to deal with abused protocols. Here is one reference http://customer.comcast.com/help-and-support/internet/list-of-blocked-ports/ My crystal ball says all of UDP will show up soon. CB
Thanks!
Matt PS--yes, I know it would completely change the dynamics of the internet as we know it today to shift to a 1:1 correspondence between input requests and output replies...but it *would* have a nice side effect of balancing out traffic ratios in many places, altering the settlement landscape in the process. ;)
How about this - I have proposed to NIST we start filtering - realize that the NIST ITS program itself was setup to run NTP in an open access mode - we host a dozen or so of those systems and so we get hit all the time. The solution is actually not running timing services across UDP because of the hand shaking over the open Internet - and that obviously isnt going to happen. My suggestion is that for those that need access we set up VLAN trunked private networking models to your ISP MPOE as it were in a digital context. If this interests you contact me off list. Todd Glassey - USTiming.ORG On 2/2/2014 7:17 PM, ryangard@gmail.com wrote:
I'd hate to think that NetOps would be so heavy handed in blocking all of UDP, as this would essentially halt quite a bit of audio/video traffic. That being said, there's still quite the need for protocol improvement when making use of UDP, but blocking UDP as a whole is definitely not a resolution, and simply creating a wall that not only keeps the abusive traffic out, but keeps legitimate traffic from flowing freely as it should. Sent on the TELUS Mobility network with BlackBerry
-----Original Message----- From: Cb B <cb.list6@gmail.com> Date: Sun, 2 Feb 2014 15:09:55 To: Matthew Petach<mpetach@netflight.com> Cc: <nanog@nanog.org> Subject: Re: TWC (AS11351) blocking all NTP?
On Feb 2, 2014 2:54 PM, "Matthew Petach" <mpetach@netflight.com> wrote:
On Sun, Feb 2, 2014 at 2:17 PM, Cb B <cb.list6@gmail.com> wrote:
On Feb 2, 2014 8:35 AM, "Jonathan Towne" <jtowne@slic.com> wrote:
The provider has kindly acknowledged that there is an issue, and are working on a resolution. Heads up, it may be more than just my region. And not just your provider, everyone is dealing with UDP amp attacks.
These UDP based amp attacks are off the charts. Wholesale blocking of traffic at the protocol level to mitigate 10s to 100s of gigs of ddos traffic is not "knee jerk", it is the right thing to do in a world where bcp 38 is far from universal and open dns servers, ntp, chargen, and whatever udp 172 is run wild.
People who run networks know what it takes to restore service. And increasingly, that will be clamping ipv4 UDP in the plumbing, both reactively and proactively.
Please note that it's not that UDP is at fault here; it's applications that are structured to respond to small input packets with large responses.
I dont want to go into fault, there is plenty of that to go around.
If NTP responded to a single query with a single equivalently sized response, its effectiveness as a DDoS attack would be zero; with zero amplification, the volume of attack traffic would be exactly equivalent to the volume of spoofed traffic the originator could send out in the first place.
I agree the source obfuscation aspect of UDP can be annoying, from the spoofing perspective, but that really needs to be recognized to be separate from the volume amplification aspect, which is an application level issue, not a protocol level issue.
Source obfuscation is not annoying. Combined with amplification, it is the perfect storm for shutting down networks... whereby the only solution is to shutdown ipv4 udp. Or wave the magic wand that makes bcp38 universal, patches boxes, and so on.
My point is: dont expect these abbused services on UDP to last. We have experience in access networks on how to deal with abused protocols. Here is one reference
http://customer.comcast.com/help-and-support/internet/list-of-blocked-ports/
My crystal ball says all of UDP will show up soon.
CB
Thanks!
Matt PS--yes, I know it would completely change the dynamics of the internet as we know it today to shift to a 1:1 correspondence between input requests and output replies...but it *would* have a nice side effect of balancing out traffic ratios in many places, altering the settlement landscape in the process. ;)
-- ------------- Personal Email - Disclaimers Apply
Or a whole bunch of small ones Vladis - and yes we are capable of handling the loads. :-) Todd On 2/3/2014 6:34 AM, Valdis.Kletnieks@vt.edu wrote:
On Mon, 03 Feb 2014 06:14:30 -0800, TGLASSEY said:
My suggestion is that for those that need access we set up VLAN trunked private networking models to your ISP MPOE as it were in a digital context. That's going to be one big VLAN.
/me makes popcorn.
-- ------------- Personal Email - Disclaimers Apply
Huh? The issue with NTP relates to the monlist command (and a few others). These are management queries, and are not critical to the operation of a NTP server. You can disable these quite easily, and still run a NTP server that provides accurate time services. On 2/3/2014 9:14 AM, TGLASSEY wrote:
How about this - I have proposed to NIST we start filtering - realize that the NIST ITS program itself was setup to run NTP in an open access mode - we host a dozen or so of those systems and so we get hit all the time.
The solution is actually not running timing services across UDP because of the hand shaking over the open Internet - and that obviously isnt going to happen.
My suggestion is that for those that need access we set up VLAN trunked private networking models to your ISP MPOE as it were in a digital context.
If this interests you contact me off list.
Todd Glassey - USTiming.ORG
On 2/2/2014 7:17 PM, ryangard@gmail.com wrote:
I'd hate to think that NetOps would be so heavy handed in blocking all of UDP, as this would essentially halt quite a bit of audio/video traffic. That being said, there's still quite the need for protocol improvement when making use of UDP, but blocking UDP as a whole is definitely not a resolution, and simply creating a wall that not only keeps the abusive traffic out, but keeps legitimate traffic from flowing freely as it should. Sent on the TELUS Mobility network with BlackBerry
-----Original Message----- From: Cb B <cb.list6@gmail.com> Date: Sun, 2 Feb 2014 15:09:55 To: Matthew Petach<mpetach@netflight.com> Cc: <nanog@nanog.org> Subject: Re: TWC (AS11351) blocking all NTP?
On Feb 2, 2014 2:54 PM, "Matthew Petach" <mpetach@netflight.com> wrote:
On Sun, Feb 2, 2014 at 2:17 PM, Cb B <cb.list6@gmail.com> wrote:
On Feb 2, 2014 8:35 AM, "Jonathan Towne" <jtowne@slic.com> wrote:
The provider has kindly acknowledged that there is an issue, and are working on a resolution. Heads up, it may be more than just my region. And not just your provider, everyone is dealing with UDP amp attacks.
These UDP based amp attacks are off the charts. Wholesale blocking of traffic at the protocol level to mitigate 10s to 100s of gigs of ddos traffic is not "knee jerk", it is the right thing to do in a world where bcp 38 is far from universal and open dns servers, ntp, chargen, and whatever udp 172 is run wild.
People who run networks know what it takes to restore service. And increasingly, that will be clamping ipv4 UDP in the plumbing, both reactively and proactively.
Please note that it's not that UDP is at fault here; it's applications that are structured to respond to small input packets with large responses.
I dont want to go into fault, there is plenty of that to go around.
If NTP responded to a single query with a single equivalently sized response, its effectiveness as a DDoS attack would be zero; with zero amplification, the volume of attack traffic would be exactly equivalent to the volume of spoofed traffic the originator could send out in the first place.
I agree the source obfuscation aspect of UDP can be annoying, from the spoofing perspective, but that really needs to be recognized to be separate from the volume amplification aspect, which is an application level issue, not a protocol level issue.
Source obfuscation is not annoying. Combined with amplification, it is the perfect storm for shutting down networks... whereby the only solution is to shutdown ipv4 udp. Or wave the magic wand that makes bcp38 universal, patches boxes, and so on.
My point is: dont expect these abbused services on UDP to last. We have experience in access networks on how to deal with abused protocols. Here is one reference
http://customer.comcast.com/help-and-support/internet/list-of-blocked-ports/
My crystal ball says all of UDP will show up soon.
CB
Thanks!
Matt PS--yes, I know it would completely change the dynamics of the internet as we know it today to shift to a 1:1 correspondence between input requests and output replies...but it *would* have a nice side effect of balancing out traffic ratios in many places, altering the settlement landscape in the process. ;)
On Feb 4, 2014, at 12:11 AM, Brian Rak <brak@gameservers.com> wrote:
You can disable these quite easily, and still run a NTP server that provides accurate time services.
Concur 100% - although it should be noted that 1:1 reflection without any amplification is also quite useful to attackers. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On 2/3/2014 2:46 PM, Dobbins, Roland wrote:
On Feb 4, 2014, at 12:11 AM, Brian Rak <brak@gameservers.com> wrote:
You can disable these quite easily, and still run a NTP server that provides accurate time services. Concur 100% - although it should be noted that 1:1 reflection without any amplification is also quite useful to attackers.
That's true, but there are countless services out there that could be abused in such a way. It's pretty much the same issue with DNS, even authoritative-only servers can be abused for reflection. Securing everything that could possibly be used for reflection is going to be a long and painful process, preventing this specific amplification attack is pretty easy. NTP clients have a long history of poor implementations, so the server already has rate limiting built in. While rate limiting outgoing replies isn't a perfect solution, it's significantly better then no rate limiting (for the curious, add 'limited' to your 'restrict default' lines to enable rate limiting. This doesn't help with the current amplification issues, but will help should someone just be abusing NTP servers for reflection).
On Feb 4, 2014, at 3:09 AM, Brian Rak <brak@gameservers.com> wrote:
It's pretty much the same issue with DNS, even authoritative-only servers can be abused for reflection.
They are, every minute of every day - and they provide amplification, too. ;> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Sun, Feb 02, 2014 at 02:49:49PM -0800, Matthew Petach <mpetach@netflight.com> wrote a message of 49 lines which said:
If NTP responded to a single query with a single equivalently sized response, its effectiveness as a DDoS attack would be zero; with zero amplification, the volume of attack traffic would be exactly equivalent to the volume of spoofed traffic the originator could send out in the first place.
It is a bit more complicated. Reflection with amplification is certainly much less useful for an attacker but it has still some advantages: the attack traffic coming to the victim's AS will be distributed differently (entering via different peers), making tracking the attacker through Netflow/Ipfix more difficult.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2/2/2014 2:17 PM, Cb B wrote:
And, i agree bcp38 would help but that was published 14 years ago.
But what? Are you somehow implying that because BCP38 was "...published 14 years ago" (RFC2267 was initially published in 1998, and it was subsequently replaced by RFC2827)? I hope not, because BCP38 filtering would still help stop spoofed traffic now perpetuating these attacks, 14 years after BCP38 was published, because spoofing is at the root of this problem (reflection/amplification attacks). This horse is not dead, and still deserves a lot of kicking. $.02, - - ferg (co-author of BCP38) - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlLv3ocACgkQKJasdVTchbLhowEAuO9DSQiRswVeqpHSccHo060h cqmIB8XlaNkzEPQw1w0A/0G6cjvtWBiJfwWbWoTY7X3RRMHeN36RkYR+2TonyNBi =W2wU -----END PGP SIGNATURE-----
On Feb 3, 2014 10:23 AM, "Paul Ferguson" <fergdawgster@mykolab.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2/2/2014 2:17 PM, Cb B wrote:
And, i agree bcp38 would help but that was published 14 years ago.
But what? Are you somehow implying that because BCP38 was "...published 14 years ago" (RFC2267 was initially published in 1998, and it was subsequently replaced by RFC2827)?
I hope not, because BCP38 filtering would still help stop spoofed traffic now perpetuating these attacks, 14 years after BCP38 was published, because spoofing is at the root of this problem (reflection/amplification attacks).
This horse is not dead, and still deserves a lot of kicking.
$.02,
- - ferg (co-author of BCP38)
I completely agree. My sphere of influence is bcp38 compliant. And, networks that fail to support some form of bcp38 are nothing short of negligent. That said, i spend too much time taking defensive action against ipv4 amp udp attacks. And wishing others would deploy bcp38 does not solve today's ddos attacks. CB
- -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iF4EAREIAAYFAlLv3ocACgkQKJasdVTchbLhowEAuO9DSQiRswVeqpHSccHo060h cqmIB8XlaNkzEPQw1w0A/0G6cjvtWBiJfwWbWoTY7X3RRMHeN36RkYR+2TonyNBi =W2wU -----END PGP SIGNATURE-----
----- Original Message -----
From: "Cb B" <cb.list6@gmail.com>
I completely agree. My sphere of influence is bcp38 compliant. And, networks that fail to support some form of bcp38 are nothing short of negligent.
That said, i spend too much time taking defensive action against ipv4 amp udp attacks. And wishing others would deploy bcp38 does not solve today's ddos attacks.
Nope. But providing a bigger, better tuned hammer to apply to people's heads may. So any contributions you can make to http://www.bcp38.info will be cheerfully accepted. :-) Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
Don't fight it. It's clear that implementation on a per-packet basis of RFC4824 (datagrams over Semaphore Flag Signaling System) would have prevented this entire situation. Refer to sections 3.3 and 3.4. -j On Mon, Feb 3, 2014 at 12:23 PM, Paul Ferguson <fergdawgster@mykolab.com> wrote:
On 2/2/2014 2:17 PM, Cb B wrote:
And, i agree bcp38 would help but that was published 14 years ago.
But what? Are you somehow implying that because BCP38 was "...published 14 years ago" (RFC2267 was initially published in 1998, and it was subsequently replaced by RFC2827)?
On Sun, Feb 2, 2014 at 5:17 PM, Cb B <cb.list6@gmail.com> wrote:
And, i agree bcp38 would help but that was published 14 years ago.
Howdy, If just three of the transit-free networks rewrote their peering contracts such that there was a $10k per day penalty for sending packets with source addresses the peer should reasonably have known were forged, this problem would go away in a matter of weeks. Granted it would also be helpful to have a BGP extension signifying allowed-source-but-don't-route so that RP filtering would work even when multihomed. Still, even without automatic RP filtering we're capable of preventing spoofed packets if financially incentivized. Thing is, they can't be the source of the solution until they stop being part of the problem. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Feb 4, 2014, at 11:04 AM, William Herrin <bill@herrin.us> wrote:
On Sun, Feb 2, 2014 at 5:17 PM, Cb B <cb.list6@gmail.com> wrote:
And, i agree bcp38 would help but that was published 14 years ago.
Howdy,
If just three of the transit-free networks rewrote their peering contracts such that there was a $10k per day penalty for sending packets with source addresses the peer should reasonably have known were forged, this problem would go away in a matter of weeks. Granted it would also be helpful to have a BGP extension signifying allowed-source-but-don't-route so that RP filtering would work even when multihomed. Still, even without automatic RP filtering we're capable of preventing spoofed packets if financially incentivized.
Thing is, they can't be the source of the solution until they stop being part of the problem.
I’ve seen similar comments in other forums. We are all generally paid for moving packets, not filtering them. The speed at which you can forward packets can often cause increased $$. Using these features also impacts performance, so the cost may actually be 2x in capex+opex to provision ports due to reduced line-rate capability. Even if you take a RPSL-IRR approach to building filters, and even if the router can handle such long ACLs bug-free, you have some objects that expand to cover 50-90% of the internet. They may be someones backup route at some point because of ‘something’. Clearly putting the filters as close to the source is helpful but detecting the actual spoofed packet is hard. Take the thread from last-week about how I can detect folks that are allowed to “spoof”, or “forward”/“rewrite” my packet destination. Even if it goes over GRE to somewhere, that IP should only be sourced from *my* host. At some point the rest of the trust comes into play that the IP is correct. Too many devices are generous in what they accept and allows these types of attack surfaces to be abused. Until you find yourself on the receiving end of these types of things, you may not ask for or pay for DDoS protection services, or advanced filtering, or even ask your vendor to support these features. I have to wait months for fixes in the features because no support from others in the industry on the platform, etc. Those that are up in arms about this stuff seem to not be the ones asking the vendors for features and fixes. - Jared
On Tue, Feb 4, 2014 at 11:23 AM, Jared Mauch <jared@puck.nether.net> wrote:
On Feb 4, 2014, at 11:04 AM, William Herrin <bill@herrin.us> wrote:
If just three of the transit-free networks rewrote their peering contracts such that there was a $10k per day penalty for sending packets with source addresses the peer should reasonably have known were forged, this problem would go away in a matter of weeks.
I've seen similar comments in other forums. We are all generally paid for moving packets, not filtering them. The speed at which you can forward packets can often cause increased $$. Using these features also impacts performance, so the cost may actually be 2x in capex+opex to provision ports due to reduced line-rate capability.
Hi Jared, You're gonna need a bigger TCAM, but even so I think you're overstating the case.
Even if you take a RPSL-IRR approach to building filters, and even if the router can handle such long ACLs bug-free, you have some objects that expand to cover 50-90% of the internet. They may be someones backup route at some point because of 'something'.
Yes, but that's OK. In order to make sure that they're aren't originating from the penalizing 10%, your peers will have to implement similar filtering downstream... where the breadth isn't 90%.
Clearly putting the filters as close to the source is helpful but detecting the actual spoofed packet is hard.
At the customer boundary it's trivial: they'll tell you what they originate, and that's what you'll allow. If your customer lies, pass the penalty forward. At the peering boundary, you don't have to detect the forged packets. You can wait until someone complains, confirm it, and then apply the penalty. Packets coming from your peers won't go to your other peers, only to your customers. That's how you rigged your routing. More, evidence that the downstream was authorized to send those packets refutes the penalty.
Until you find yourself on the receiving end of these types of things, you may not ask for or pay for DDoS protection services, or advanced filtering, or even ask your vendor to support these features. I have to wait months for fixes in the features because no support from others in the industry on the platform, etc.
DDoS is a bigger problem than spoofing and amplification. My suggestion only addresses spoofing and amplification, not botnets in general.
Those that are up in arms about this stuff seem to not be the ones asking the vendors for features and fixes.
Like I said, the "tier 1's" can't be the source of the solution until they stop being part of the problem. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Feb 4, 2014, at 11:52 AM, William Herrin <bill@herrin.us> wrote:
Those that are up in arms about this stuff seem to not be the ones asking the vendors for features and fixes.
Like I said, the "tier 1's" can't be the source of the solution until they stop being part of the problem.
This is the attitude that I've seen elsewhere that is devoid of any meat. As I said before, we hit a big preventing the ability to do this even if we wanted to. The impact is drop all traffic or permit all in that case. If you were my customer which would you decide? Ask your vendors for these features. Ask them to fix the bugs. The ball rolls uphill here and it's in their lap. Blaming the carriers is wrongheaded and putting it where it doesn't belong in many cases. Happy to discuss offline. - Jared
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2/4/2014 10:03 AM, Jared Mauch wrote:
Ask your vendors for these features. Ask them to fix the bugs. The ball rolls uphill here and it's in their lap. Blaming the carriers is wrongheaded and putting it where it doesn't belong in many cases. Happy to discuss offline.
I'd like to echo Jared's sentiment here -- collectively speaking, service providers need to figure out a way to deal with this issue, before some congresscritters start to try to introduce legislation that will force you to to do it in a way that no one will like. $.02, - - ferg - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlLxLL4ACgkQKJasdVTchbJ95AEAm5GcMZUKvy5WDjycH8f4C4Dq 7t1inFCPmGhbmo/456kBAKpUxaf/y7eXVnsxo9/IvULsGEbrf8zdapuAOSUdJrem =v0jF -----END PGP SIGNATURE-----
On Tue, 04 Feb 2014 10:09:02 -0800, Paul Ferguson said:
I'd like to echo Jared's sentiment here -- collectively speaking, service providers need to figure out a way to deal with this issue, before some congresscritters start to try to introduce legislation that will force you to to do it in a way that no one will like.
Can somebody explain to me why those who run eyeball networks are able to block outbound packets when the customer hasn't paid their bill, but can't seem to block packets that shouldn't be coming from that cablemodem? (And yes, I know that in the first case, it urges the customer to cough up the bucks, and in the second case, it's usually not a revenue generator)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2/4/2014 10:47 AM, Valdis.Kletnieks@vt.edu wrote:
On Tue, 04 Feb 2014 10:09:02 -0800, Paul Ferguson said:
I'd like to echo Jared's sentiment here -- collectively speaking, service providers need to figure out a way to deal with this issue, before some congresscritters start to try to introduce legislation that will force you to to do it in a way that no one will like.
Can somebody explain to me why those who run eyeball networks are able to block outbound packets when the customer hasn't paid their bill, but can't seem to block packets that shouldn't be coming from that cablemodem?
(And yes, I know that in the first case, it urges the customer to cough up the bucks, and in the second case, it's usually not a revenue generator)
It's a dichotomy that is... unexplainable for me personally. - - ferg - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlLxNz8ACgkQKJasdVTchbJq6AEApzaaZ9PpPX30kUYAxsGZFzeV WR98y6VBxlocQE2oQFkBANSa4m0/JOGv+PDQovI4xSkjaE/Ru0V8woagAs1hS1C0 =KAL8 -----END PGP SIGNATURE-----
----- Original Message -----
From: "Paul Ferguson" <fergdawgster@mykolab.com>
(And yes, I know that in the first case, it urges the customer to cough up the bucks, and in the second case, it's usually not a revenue generator)
It's a dichotomy that is... unexplainable for me personally.
Nope: it's easy to explain; you merely have to be a cynical bastard: Attack traffic takes up bandwidth. Providers sell bandwidth. It *is in their commercial best interest (read: maximizing shareholder value) *NOT* to filter out DOS, DDOS, and spam traffic until their hand is forced -- it's actually their fiduciary duty not to. *THIS* is the problem we have to fix. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
On 04/02/14 11:35, Jay Ashworth wrote:
It *is in their commercial best interest (read: maximizing shareholder value) *NOT* to filter out DOS, DDOS, and spam traffic until their hand is forced -- it's actually their fiduciary duty not to.
That's short-sighted, but I agree in that that's what happens. Not filtering doesn't prevent them to operate.
*THIS* is the problem we have to fix.
Source-based routing when going back to the backbone, at least on IPv6. It allows end-user multihoming with no BGP, and routers could be programmed to, by default, drop packages that don't know how to source-route, hence, automatically source filtering for those that don't care enough. Difficult to do. Will take years to develop and adopt... if at all.
In message <977303.7242.1391542533531.JavaMail.root@benjamin.baylink.com>, Jay Ashworth writes:
----- Original Message -----
From: "Paul Ferguson" <fergdawgster@mykolab.com>
(And yes, I know that in the first case, it urges the customer to cough up the bucks, and in the second case, it's usually not a revenue generator)
It's a dichotomy that is... unexplainable for me personally.
Nope: it's easy to explain; you merely have to be a cynical bastard:
Attack traffic takes up bandwidth.
Providers sell bandwidth.
It *is in their commercial best interest (read: maximizing shareholder value) *NOT* to filter out DOS, DDOS, and spam traffic until their hand is forced -- it's actually their fiduciary duty not to.
Then the need to be made criminally liable for the damage that it causes. Yes, the directors of these companies need to serve gaol time.
*THIS* is the problem we have to fix.
Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.co m Designer The Things I Think RFC 210 0 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DI I St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 127 4
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
Then the need to be made criminally liable for the damage that it causes. Yes, the directors of these companies need to serve gaol time.
why not just have god send down lightning bolts? quicker and cheaper. or maybe they will just drown as the level of hyperbole keeps rising. randy
On 2/4/2014 5:00 PM, Mark Andrews wrote:
Nope: it's easy to explain; you merely have to be a cynical bastard:
Attack traffic takes up bandwidth.
Providers sell bandwidth.
It *is in their commercial best interest (read: maximizing shareholder value) *NOT* to filter out DOS, DDOS, and spam traffic until their hand is forced -- it's actually their fiduciary duty not to. Then the need to be made criminally liable for the damage that it causes. Yes, the directors of these companies need to serve gaol time.
That would never fly, because it would put the politicians at odds with the telecom buddies that make huge political donations. Hard to throw someone in jail then hit them up for campaign money. What will probably happen is the same thing we do with everything else that might be used for evil purposes but where we don't want to tackle the real underlying problem -- just write a law banning something and hope the problem goes away. Make it illegal to posses a device capable of bandwith greater than 33.6Kbps without a special license, and BAM -- no more problems, overnight. For added political-style points, tack on a catchy moniker, like "Immoral Bandwidth Prohibition", "The War on DDOS", or "High-Capacity Digital Assault Bandwidth" to help sell it to the public. The public will be OK with their funny cat videos taking 19 hours to load if they know they're preventing bad guys from doing something evil. After all, it's worked flawlessly for alcohol, drugs and guns, so it MUST work for networks... and it's much easier than those silly, so-called "solutions" y'all are talking about! :p - Pete (P.S. Dear politicians: in case you're reading this, the above was satire and should not be construed as anything resembling a good idea.)
In message <52F17931.40604@alter3d.ca>, Peter Kristolaitis writes:
On 2/4/2014 5:00 PM, Mark Andrews wrote:
Nope: it's easy to explain; you merely have to be a cynical bastard:
Attack traffic takes up bandwidth.
Providers sell bandwidth.
It *is in their commercial best interest (read: maximizing shareholder value) *NOT* to filter out DOS, DDOS, and spam traffic until their hand is forced -- it's actually their fiduciary duty not to. Then the need to be made criminally liable for the damage that it causes. Yes, the directors of these companies need to serve gaol time.
That would never fly, because it would put the politicians at odds with the telecom buddies that make huge political donations. Hard to throw someone in jail then hit them up for campaign money. What will probably happen is the same thing we do with everything else that might be used for evil purposes but where we don't want to tackle the real underlying problem -- just write a law banning something and hope the problem goes away.
No, you write a law requiring something, e.g. BCP 38 filtering by ISPs, and you audit it. You also make the ISPs directors liable for the impact that results from spoofed traffic from them. Making it law puts all the ISP's in the country on a equal footing with respect to implementation costs.
Make it illegal to posses a device capable of bandwith greater than 33.6Kbps without a special license, and BAM -- no more problems, overnight. For added political-style points, tack on a catchy moniker, like "Immoral Bandwidth Prohibition", "The War on DDOS", or "High-Capacity Digital Assault Bandwidth" to help sell it to the public. The public will be OK with their funny cat videos taking 19 hours to load if they know they're preventing bad guys from doing something evil.
If you have millions of compromised customers it doesn't matter what bandwidth limits they have. You can still launch a amplifying reflection DDoS from hosts behind 300 baud links.
After all, it's worked flawlessly for alcohol, drugs and guns, so it MUST work for networks... and it's much easier than those silly, so-called "solutions" y'all are talking about! :p
Regulation and audits works well enough for butchers, resturants etc. Remember once BCP 38 is implemented it is relatively easy to continue. The big step is getting it turned on in the first place which requires having the right equipment. Now if we could get equipement vendors to stop shipping models without the necessary support it would help but that also may require government intervention.
- Pete
(P.S. Dear politicians: in case you're reading this, the above was satire and should not be construed as anything resembling a good idea.)
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
No, you write a law requiring something, e.g. BCP 38 filtering by ISPs, and you audit it. You also make the ISPs directors liable for the impact that results from spoofed traffic from them.
Making it law puts all the ISP's in the country on a equal footing with respect to implementation costs.
you have done this in your network, the isp for which you are an engineer? randy
On Wed, 05 Feb 2014 12:18:54 +1100, Mark Andrews said:
Regulation and audits works well enough for butchers, resturants etc. Remember once BCP 38 is implemented it is relatively easy to continue. The big step is getting it turned on in the first place which requires having the right equipment.
Now if we could get equipement vendors to stop shipping models without the necessary support it would help but that also may require government intervention.
Time to name-and-shame. It's 2014. Who's still shipping gear that can't manage eyeball-facing BCP38?
On Tue, Feb 4, 2014 at 10:01 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Wed, 05 Feb 2014 12:18:54 +1100, Mark Andrews said:
Now if we could get equipement vendors to stop shipping models without the necessary support it would help but that also may require government intervention.
A good start would be to get BCP38 revised to router the Host requirements RFCs, to indicate that ingress filtering should be considered mandatory on site-facing interfaces. If the standards documents still just call it a best practice.... what hope is there of having governments require it of the service providers that their networks are connected to, anyways?
Time to name-and-shame. It's 2014. Who's still shipping gear that can't manage eyeball-facing BCP38?
-- -JH
On Feb 5, 2014, at 2:12 AM, Jimmy Hess <mysidia@gmail.com> wrote:
On Wed, 05 Feb 2014 12:18:54 +1100, Mark Andrews said:
Now if we could get equipement vendors to stop shipping models without the necessary support it would help but that also may require government intervention. ...
A good start would be to get BCP38 revised to router the Host requirements RFCs, to indicate that ingress filtering should be considered mandatory on site-facing interfaces. ...
It's also true that if a sizable group of network operators were to actually deploy source address validation (thus proving that it really is a reasonable approach and doesn't carry too much operational or vendor implications), then it would be quite reasonable for those operators to bring the results to NANOG and get it recognized as a best current operating practice for networks of similar design/purpose.
If the standards documents still just call it a best practice.... what hope is there of having governments require it of the service providers that their networks are connected to, anyways?
There is a significant difference between a "best current practice" (BCP) document from the IETF (a technical standards body) versus one which actually reflects the well-considered best practices of a large network operator forum. The latter would be of some interest to governments (and groups of governments) when they ask for any options that might help with their growing spam and DDoS concerns... FYI, /John
On Feb 8, 2014, at 3:37 AM, John Curran <jcurran@arin.net> wrote:
It's also true that if a sizable group of network operators were to actually deploy source address validation (thus proving that it really is a reasonable approach and doesn't carry too much operational or vendor implications), then it would be quite reasonable for those operators to bring the results to NANOG and get it recognized as a best current operating practice for networks of similar design/purpose.
Many already do - including operators of very large networks. There are operational, vendor, and topological considerations which mean that it's achieved utilizing various mechanisms in different scenarios. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Fri, Feb 7, 2014 at 2:07 PM, Dobbins, Roland <rdobbins@arbor.net> wrote:
On Feb 8, 2014, at 3:37 AM, John Curran <jcurran@arin.net> wrote:
It's also true that if a sizable group of network operators were to actually deploy source address validation (thus proving that it really is a reasonable approach and doesn't carry too much operational or vendor implications), then it would be quite reasonable for those operators to bring the results to NANOG and get it recognized as a best current operating practice for networks of similar design/purpose.
Many already do - including operators of very large networks. There are operational, vendor, and topological considerations which mean that it's achieved utilizing various mechanisms in different scenarios.
Documenting those various mechanisms which are actually utilized is the key here. =) $0.02 ~Chris -- @ChrisGrundemann http://chrisgrundemann.com
On Feb 8, 2014, at 4:25 AM, Chris Grundemann <cgrundemann@gmail.com> wrote:
Documenting those various mechanisms which are actually utilized is the key here. =)
Yes, as well as the various limitations and caveats, like the wholesale/retail issue (i.e., customers of my customer). ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
----- Original Message -----
From: "Roland Dobbins" <rdobbins@arbor.net>
On Feb 8, 2014, at 4:25 AM, Chris Grundemann <cgrundemann@gmail.com> wrote:
Documenting those various mechanisms which are actually utilized is the key here. =)
Yes, as well as the various limitations and caveats, like the wholesale/retail issue (i.e., customers of my customer).
And anyone who has factual data on that topic is invited to contribute it to (stop me if you've heard this one)... http://www.bcp38.info Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
On (2014-02-04 23:01 -0500), Valdis.Kletnieks@vt.edu wrote:
Regulation and audits works well enough for butchers, resturants etc. Remember once BCP 38 is implemented it is relatively easy to continue. The big step is getting it turned on in the first place which requires having the right equipment.
Now if we could get equipement vendors to stop shipping models without the necessary support it would help but that also may require government intervention.
Time to name-and-shame. It's 2014. Who's still shipping gear that can't manage eyeball-facing BCP38?
If we keep thinking this problem as last-mile port problem, it won't be solved in next 20 years. Because lot of those ports really can't do RPF and even if they can do it, they are on autopilot and next change is market forced fork-lift change. Company may not even employ technical personnel, only buy consulting when making changes. If we focus on transit borders, we can make spoofed DoS completely impractical very rapidly, as spoofing is then restricted inside domain, and if target isn't in same domain, you can't benefit from it. And as attacks are from distributed botnets, you'll simply generate more attack traffic by not spooffing, as you're not restricted inside spooffing domain. However transit border doing ACL is something that seems to very controversial, there is no universal consensus that it even should be done and quite few seem to do it. I feel we need to change this, and make community at large agree it is the BCP and solve the challenges presented. -- ++ytti
On Wed, Feb 5, 2014 at 2:46 AM, Saku Ytti <saku@ytti.fi> wrote:
If we keep thinking this problem as last-mile port problem, it won't be solved
in next 20 years. Because lot of those ports really can't do RPF and even
if
[snip] The last-mile ports don't necessarily need RPF; a simple inbound access list on the ISP side....... Or even outbound on CPE side, with all valid source addresses allowed, and nothing else: is just perfect. In essence; it is a last-mile problem, and that is part of the challenge. The last-mile is the best possible place to filter, without breaking things. As for the idea, that the world can take a shortcut, and filter in some manner at transit services is tantalizing, but also: is not quite adequate, and that's probably not going to happen either.
[snip]
However transit border doing ACL is something that seems to very
controversial, there is no universal consensus that it even should be
Anything that is likely to blackhole legitimate traffic is going to be controversial. IP source based filtering on transit links may very well fall into that category of greatly increasing that risk in many cases. Restricting the source IP address range in from transit links is a bad idea, unless you can be certain that no other source IPs will show up legitimately, which you cannot necessarily be. If i am a transit provider, and I connect with a peer network buying transit from me, then they get to route traffic over that link: according to the routes my network announced to their router. If my router discards any of that traffic based on source, then the route I propagated to my peer was dishonest --- that is, it would mean my route announcement was a lie: the filtering would in essence make some routes blackhole routes, and I am disrupting the connectivity for the unexpected source addresses, just by turning up that link. Or I am at risk of disrupting connectivity in the future, to any network that my downstream peer later interconnects with, if they will also provide transit in that relationship, and also... it would be a common practice on many networks to turn up such interconnections at a date before I or any other transit upstreams are informed. It is likely from time to time, that many transit downstreams will obtain additional address allocations, or that they will make additional network connections: especially, if in fact, my downstream peer is multihomed, possibly with numerous providers, and they may themselves be a transit provider. At a certain level; "RPF" does not work, because: by design, routing IN and OUT can very well be asymmetric traffic flows for networks that are multihomed. Not announcing the source to a specific network, doesn't make it OKAY for the adjacent transit network to drop traffic from that source.
done and quite few seem to do it. I feel we need to change this, and make community at large agree it is the BCP and solve the challenges presented.
-- -JH
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2/5/2014 7:06 PM, Jimmy Hess wrote:
The last-mile is the best possible place to filter, without breaking things.
I could not agree more. :-) - - ferg - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlLy/5gACgkQKJasdVTchbKXbAEAqFswL2qtqIgRcALVMLdQA0H/ dRLvmDhCoXEmRtOB2B8BAMbH489q8lB/qdiEofYviAAnNA6aqpT38ASXDzBUKO/K =xjIk -----END PGP SIGNATURE-----
In message <52F2FF98.2030507@mykolab.com>, Paul Ferguson writes:
On 2/5/2014 7:06 PM, Jimmy Hess wrote:
The last-mile is the best possible place to filter, without breaking things.
I could not agree more. :-)
- - ferg
Remember "last mile" includes "datacenter" and "noc". Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2/5/2014 7:35 PM, Mark Andrews wrote:
In message <52F2FF98.2030507@mykolab.com>, Paul Ferguson writes:
On 2/5/2014 7:06 PM, Jimmy Hess wrote:
The last-mile is the best possible place to filter, without breaking things.
I could not agree more. :-)
- - ferg
Remember "last mile" includes "datacenter" and "noc".
Whatever gets it done. I'm just sick of hearing why people can't do it, instead of reasons why they can. - - ferg - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlLzBE8ACgkQKJasdVTchbL6mQEAo6p/QQxyEdiQkf1/91nteK1u z3zyR9QbO7dtDuEBftkBANAlfy+zqbMuOp03rIiPu/pk3Ixt+mo58Yjs3fOHfqUG =9wRN -----END PGP SIGNATURE-----
The last-mile is the best possible place to filter, without breaking things. I could not agree more. :-)
very large consumer populations are on metro-ether-like things. and it gets kinkier from there, don't eat before looking at what ntt-east has done with ngn. i fear we really have most of the easy big deployments and all of the cool kids. we're down to statistically small stubborn do-nothings and some folk with equipment that will take years to be pushed off net. randy
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2/5/2014 7:43 PM, Randy Bush wrote:
The last-mile is the best possible place to filter, without breaking things. I could not agree more. :-)
very large consumer populations are on metro-ether-like things. and it gets kinkier from there, don't eat before looking at what ntt-east has done with ngn.
i fear we really have most of the easy big deployments and all of the cool kids. we're down to statistically small stubborn do-nothings and some folk with equipment that will take years to be pushed off net.
Maybe. Maybe not. I think it really depends how we approach the problem -- apparently our approaches up until now have been failures to a certain degree. At least 20-30% failure, if you believe the Spoofer Project numbers. I'd like to think (and I am not happy smiley person as you well know) that perhaps we can motivate some younger, brighter, ingenious people who have not been tilting at this for 15 years to consider new ways to approach this problem. :-) <-- Smiley! - - ferg - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlLzBggACgkQKJasdVTchbL8hwEAwXbejfCFaOQnqYz6v8xcXfb7 uTmSIWZj+kuiGh976lUA/A5gGGrrAzaVyp3SqX57p5AR8w9kfMQEEbVMLCn7il4R =FE9f -----END PGP SIGNATURE-----
I'd like to think (and I am not happy smiley person as you well know) that perhaps we can motivate some younger, brighter, ingenious people who have not been tilting at this for 15 years to consider new ways to approach this problem. :-) <-- Smiley!
we should definitely scream at them and threaten legal action and lightning strikes from the gods. randy
On Feb 5, 2014, at 2:46 AM, Saku Ytti <saku@ytti.fi> wrote:
If we keep thinking this problem as last-mile port problem, it won't be solved in next 20 years. Because lot of those ports really can't do RPF and even if they can do it, they are on autopilot and next change is market forced fork-lift change. Company may not even employ technical personnel, only buy consulting when making changes.
It can be solved, but not by NANOG. Imagine if Cable labs required all DOCSIS compliant cable modems to default to doing source address verification in the next version of DOCSIS? It would (eventually) get rolled out, and it would solve the problem. Even if it doesn't default to on, requiring the hardware to be capable would be a nice step. The consumer last mile is actually simpler in that there are a few organizations who "control" the standards. Efforts need to focus on getting the BCP38 stuff into those standards, ideally as mandatory defaults. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
----- Original Message -----
From: "Valdis Kletnieks" <Valdis.Kletnieks@vt.edu>
Time to name-and-shame. It's 2014. Who's still shipping gear that can't manage eyeball-facing BCP38?
It sure is. ==== POLL: If you run "eyeball" equipment -- edge concentrators/routers/CMTSen, would you please post, without employer attribution, what make & model it is, and which firmware rev it's running, and whether it has a knob for unicast-strict-RPF or an equivalent automatic filtering method which is compatible with "flip the switch" BCP38 deployment, and preferably runs on the relevant line cards, not CPU. At your option, you can mention whether it's already on, whether you had to look it up, and which network it is. :-) PLEASE RESPOND to jra+bcp38@baylink.com, not the list; I will aggregate. I do not plan to mention any people in the results, but if I'm told the names of networks in sufficient specificity to avoid confusion, I will include those. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
----- Original Message -----
From: "Valdis Kletnieks" <Valdis.Kletnieks@vt.edu>
Can somebody explain to me why those who run eyeball networks are able to block outbound packets when the customer hasn't paid their bill, but can't seem to block packets that shouldn't be coming from that cablemodem?
The purported argument is "our edge concentrators don't have that knob/ enough horsepower to do it manually and stay on the line card". I'm not sure how accurate that argument is any more and (as I noted in another reply just now[1]), I'm officially not buying it anymore. Cheers, -- jra [1] http://www.bcp38.info/index.php/Information_for_equipment_manufacturers -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
On Tue, 4 Feb 2014, Valdis.Kletnieks@vt.edu wrote:
On Tue, 04 Feb 2014 10:09:02 -0800, Paul Ferguson said:
I'd like to echo Jared's sentiment here -- collectively speaking, service providers need to figure out a way to deal with this issue, before some congresscritters start to try to introduce legislation that will force you to to do it in a way that no one will like.
Can somebody explain to me why those who run eyeball networks are able to block outbound packets when the customer hasn't paid their bill, but can't seem to block packets that shouldn't be coming from that cablemodem?
(And yes, I know that in the first case, it urges the customer to cough up the bucks, and in the second case, it's usually not a revenue generator)
The only way this is going to get fixed is to make it more expensive to originate abuse than it is to block it. The only thing management is going to pay attention to is their pocketbooks. -Dan
On Tue, Feb 4, 2014 at 1:47 PM, <Valdis.Kletnieks@vt.edu> wrote:
Can somebody explain to me why those who run eyeball networks are able to block outbound packets when the customer hasn't paid their bill, but can't seem to block packets that shouldn't be coming from that cablemodem?
The DOCSIS spec has source address verification (as I understand it, for about a decade.) It is deployed within at least one large cable access provider network I am familiar with (though I don't personally work on the DOCSIS side of things). Why don't enterprises, hosting and cloud providers do it? (I don't know that they don't, but I figured I'd just keep with the tone.) Enterprises know what prefixes they have so should drop outbound packets with source IPs other than those, right? Likewise hosting providers ought to put in some safeguards. What about cloud providers who also provide virtual OSes and other software? Are those VMs and their third-party software kept patched? All those folks also provide access at the network edge. Tony
Can somebody explain to me why those who run eyeball networks are able to block outbound packets when the customer hasn't paid their bill, but can't seem to block packets that shouldn't be coming from that cablemodem?
i suspect the non-payment case is solved at a layer below three randy
Can somebody explain to me why those who run eyeball networks are able to block outbound packets when the customer hasn't paid their bill, but can't seem to block packets that shouldn't be coming from that cablemodem?
i suspect the non-payment case is solved at a layer below three
In a DOCSIS network the source address verification (as Tony said) is typically done on the CMTS IIRC. Turning a customer off for non-payment is done in an accounts management / billing system. While I am sure continuing to agree with each other that spoofing is bad, we lack actionable data. ;-) As I said in another thread, I think someone / some group needs to invest to collect actual data and share the results openly. So any volunteers out there? I¹m sure there are lots of ways to underwrite independent research on the subject (contact me off-list). Jason
On 04/02/14 16:31, Livingood, Jason wrote:
Can somebody explain to me why those who run eyeball networks are able to block outbound packets when the customer hasn't paid their bill, but can't seem to block packets that shouldn't be coming from that cablemodem?
i suspect the non-payment case is solved at a layer below three
In a DOCSIS network the source address verification (as Tony said) is typically done on the CMTS IIRC. Turning a customer off for non-payment is done in an accounts management / billing system.
While I am sure continuing to agree with each other that spoofing is bad, we lack actionable data. ;-) As I said in another thread, I think someone / some group needs to invest to collect actual data and share the results openly.
So any volunteers out there? I¹m sure there are lots of ways to underwrite independent research on the subject (contact me off-list).
Maybe I'm oversimplifying things but I'm really curious to know why can't the nearest-to-end-user ACL-enabled router simply have an ACL to only allows packets from end-users that has a valid source-address from the network segment they provide service to. What I'm failing to understand, and again, please excuse me if I'm oversimplifying, is what data do you need from researchers, specifically. What specific actionable data would be helpful? Why does the lack of the data prevent you from applying an ACL.
On 2/4/14, 7:48 PM, "Octavio Alvarez" <alvarezp@alvarezp.ods.org> wrote:
What I'm failing to understand, and again, please excuse me if I'm oversimplifying, is what data do you need from researchers, specifically. What specific actionable data would be helpful? Why does the lack of the data prevent you from applying an ACL.
What I am suggesting is that the community at large needs measurement data, rather than individual network operators (which already know if they do or do not implement BCP38). Only with a long list of operators that DO prevent spoofing and a list of those that DO NOT, backed up with a decent data set (enough measurement points, enough measurement tests, across enough time, with an openly shared methodology), can the community start to encourage non-adopters to change their position. Just my two cents though... Jason
Here's such a report: http://spoofer.cmand.org/summary.php Frank -----Original Message----- From: Livingood, Jason [mailto:Jason_Livingood@cable.comcast.com] Sent: Tuesday, February 04, 2014 6:53 PM To: Octavio Alvarez; North American Network Operators' Group Subject: Re: BCP38 [Was: Re: TWC (AS11351) blocking all NTP?] On 2/4/14, 7:48 PM, "Octavio Alvarez" <alvarezp@alvarezp.ods.org> wrote:
What I'm failing to understand, and again, please excuse me if I'm oversimplifying, is what data do you need from researchers, specifically. What specific actionable data would be helpful? Why does the lack of the data prevent you from applying an ACL.
What I am suggesting is that the community at large needs measurement data, rather than individual network operators (which already know if they do or do not implement BCP38). Only with a long list of operators that DO prevent spoofing and a list of those that DO NOT, backed up with a decent data set (enough measurement points, enough measurement tests, across enough time, with an openly shared methodology), can the community start to encourage non-adopters to change their position. Just my two cents though... Jason
Cool, thanks for the pointed. Now if we could get the data by ASN and publish it on a site like bcp38.info, that would be awesome. On 2/4/14, 11:03 PM, "Frank Bulk" <frnkblk@iname.com> wrote:
Here's such a report:
http://spoofer.cmand.org/summary.php
Frank
-----Original Message----- From: Livingood, Jason [mailto:Jason_Livingood@cable.comcast.com] Sent: Tuesday, February 04, 2014 6:53 PM To: Octavio Alvarez; North American Network Operators' Group Subject: Re: BCP38 [Was: Re: TWC (AS11351) blocking all NTP?]
On 2/4/14, 7:48 PM, "Octavio Alvarez" <alvarezp@alvarezp.ods.org> wrote:
What I'm failing to understand, and again, please excuse me if I'm oversimplifying, is what data do you need from researchers, specifically. What specific actionable data would be helpful? Why does the lack of the data prevent you from applying an ACL.
What I am suggesting is that the community at large needs measurement data, rather than individual network operators (which already know if they do or do not implement BCP38). Only with a long list of operators that DO prevent spoofing and a list of those that DO NOT, backed up with a decent data set (enough measurement points, enough measurement tests, across enough time, with an openly shared methodology), can the community start to encourage non-adopters to change their position. Just my two cents though...
Jason
I here tell the spoofer project people are looking to improve their data and stats... And reporting. On Feb 5, 2014 1:08 PM, "Livingood, Jason" < Jason_Livingood@cable.comcast.com> wrote:
Cool, thanks for the pointed. Now if we could get the data by ASN and publish it on a site like bcp38.info, that would be awesome.
On 2/4/14, 11:03 PM, "Frank Bulk" <frnkblk@iname.com> wrote:
Here's such a report:
http://spoofer.cmand.org/summary.php
Frank
-----Original Message----- From: Livingood, Jason [mailto:Jason_Livingood@cable.comcast.com] Sent: Tuesday, February 04, 2014 6:53 PM To: Octavio Alvarez; North American Network Operators' Group Subject: Re: BCP38 [Was: Re: TWC (AS11351) blocking all NTP?]
On 2/4/14, 7:48 PM, "Octavio Alvarez" <alvarezp@alvarezp.ods.org> wrote:
What I'm failing to understand, and again, please excuse me if I'm oversimplifying, is what data do you need from researchers, specifically. What specific actionable data would be helpful? Why does the lack of the data prevent you from applying an ACL.
What I am suggesting is that the community at large needs measurement data, rather than individual network operators (which already know if they do or do not implement BCP38). Only with a long list of operators that DO prevent spoofing and a list of those that DO NOT, backed up with a decent data set (enough measurement points, enough measurement tests, across enough time, with an openly shared methodology), can the community start to encourage non-adopters to change their position. Just my two cents though...
Jason
On 2/5/2014 1:20 PM, Christopher Morrow wrote:
I here tell the spoofer project people are looking to improve their data and stats... And reporting. I know it's not possible due to the limitations of javascript sandboxing, but this really needs to be browser based so it can be like DNSSEC or MX or IPV6 testing. Users (and reddit) can be coerced into clicking a link if it shows a happy green sign when they pass the test. Asking them to download an executable is too much for most of them.
I'd also love a way as a network administrator that I could audit my own network. Even with all the correct knobs tweaked I occasionally find a site where someone turned up an interface and forgot some template commands, or in the case of gear that doesn't support it there might not be a filter on an upstream device even though there should be. It'd be nice to have a CM profile that would attempt to spoof something to a control server then alert if it works.
----- Original Message -----
From: "Octavio Alvarez" <alvarezp@alvarezp.ods.org>
Maybe I'm oversimplifying things but I'm really curious to know why can't the nearest-to-end-user ACL-enabled router simply have an ACL to only allows packets from end-users that has a valid source-address from the network segment they provide service to.
The common answer, Octavio, at least *used to* be "our line cards aren't smart enough to implement strict-unicast-RPF, and our boxes don't have enough horsepower to handle every packet through the CPU". As I've noted, I'm not sure I believe that's true of current generation gear, and if it *is*, then it should cost manufacturers business. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
On 2/5/14, 1:24 PM, Jay Ashworth wrote:
----- Original Message -----
From: "Octavio Alvarez" <alvarezp@alvarezp.ods.org>
Maybe I'm oversimplifying things but I'm really curious to know why can't the nearest-to-end-user ACL-enabled router simply have an ACL to only allows packets from end-users that has a valid source-address from the network segment they provide service to.
The common answer, Octavio, at least *used to* be "our line cards aren't smart enough to implement strict-unicast-RPF, and our boxes don't have enough horsepower to handle every packet through the CPU".
As I've noted, I'm not sure I believe that's true of current generation gear, and if it *is*, then it should cost manufacturers business.
There are boxes that haven't aged out of the network yet where that's an issue, some are more datacenter-centric than others. force10 e1200 was one platform that had this limitation for example.
Cheers, -- jra
----- Original Message -----
From: "joel jaeggli" <joelja@bogus.com>
As I've noted, I'm not sure I believe that's true of current generation gear, and if it *is*, then it should cost manufacturers business.
There are boxes that haven't aged out of the network yet where that's an issue, some are more datacenter-centric than others. force10 e1200 was one platform that had this limitation for example.
So making sure manufacturers are producing gear that's BCP38-compliant, and buyers have it on their tick-list, is still a productive goal, too. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
On 2/5/14, 1:46 PM, Jay Ashworth wrote:
----- Original Message -----
From: "joel jaeggli" <joelja@bogus.com>
As I've noted, I'm not sure I believe that's true of current generation gear, and if it *is*, then it should cost manufacturers business.
There are boxes that haven't aged out of the network yet where that's an issue, some are more datacenter-centric than others. force10 e1200 was one platform that had this limitation for example.
So making sure manufacturers are producing gear that's BCP38-compliant, and buyers have it on their tick-list, is still a productive goal, too.
it is... The products are probably close to the end of their sales life, but they'll likely be around for a while.
Cheers, -- jra
On Wed, Feb 5, 2014 at 4:46 PM, Jay Ashworth <jra@baylink.com> wrote:
----- Original Message -----
From: "joel jaeggli" <joelja@bogus.com>
As I've noted, I'm not sure I believe that's true of current generation gear, and if it *is*, then it should cost manufacturers business.
There are boxes that haven't aged out of the network yet where that's an issue, some are more datacenter-centric than others. force10 e1200 was one platform that had this limitation for example.
So making sure manufacturers are producing gear that's BCP38-compliant, and buyers have it on their tick-list, is still a productive goal, too.
but, if it's a datacenter deployment there are mitigations you can perform aside from uRPF... right? you COULD just use a simple acl on the interface: "my local network is..." which you could even automate. you COULD do dhcp-snooping/mac-locking/etc and ensure that the end-host is only using the one address(es) it's permitted to use. (potentially harder to do on some gear) you COULD clamp the outbound path from edge-L3 box -> code with the right acl, since you konw what traffic should come out of the local L3 edge piece. the answer doesn't' have to be uRPF.
On 2/5/14, 13:24, Jay Ashworth wrote:
The common answer, Octavio, at least*used to* be "our line cards aren't smart enough to implement strict-unicast-RPF, and our boxes don't have enough horsepower to handle every packet through the CPU".
As I've noted, I'm not sure I believe that's true of current generation gear, and if it*is*, then it should cost manufacturers business.
In Cisco 6500 land - which were very popular - Earl7 uRPF is limited to one of strict or loose (no mixing modes) for IPv4 only. Otherwise you have to rely on ACLs and the possibility of running out of TCAM space for them depending on density. The Sup2T (Earl8) does fix these limitations: uRPF is configurable per-interface basis and independent of IPv4/IPv6, and can be a mix of loose or strict mode. But Sup2T only came out in 2011. ~Seth
On Wednesday, February 05, 2014 11:24:42 PM Jay Ashworth wrote:
As I've noted, I'm not sure I believe that's true of current generation gear, and if it *is*, then it should cost manufacturers business.
But only matters if you're refreshing or just starting out. A lot of operators have a large installed base of such kit, and given horsepower is still plenty, getting that swapped out is a tall ask. Mark.
Sure. Part of the data collection task. Making sure all the current new gear knows how, still a good idea. On February 5, 2014 11:32:26 PM EST, Mark Tinka <mark.tinka@seacom.mu> wrote:
On Wednesday, February 05, 2014 11:24:42 PM Jay Ashworth wrote:
As I've noted, I'm not sure I believe that's true of current generation gear, and if it *is*, then it should cost manufacturers business.
But only matters if you're refreshing or just starting out.
A lot of operators have a large installed base of such kit, and given horsepower is still plenty, getting that swapped out is a tall ask.
Mark.
-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
On Thursday, February 06, 2014 06:34:16 AM Jay Ashworth wrote:
Sure. Part of the data collection task. Making sure all the current new gear knows how, still a good idea.
Yep - like Joel said; current kit supports it (well, the ones I buy, anyway), and certainly a good idea for operators to make sure their favorite vendor can support this when they run their next purchase cycle. Mark.
I'm going to be somewhat of a pain in everybody's ass this year, pounding on the drum whenever the topic pops up. :-) On February 5, 2014 11:38:08 PM EST, Mark Tinka <mark.tinka@seacom.mu> wrote:
On Thursday, February 06, 2014 06:34:16 AM Jay Ashworth wrote:
Sure. Part of the data collection task. Making sure all the current new gear knows how, still a good idea.
Yep - like Joel said; current kit supports it (well, the ones I buy, anyway), and certainly a good idea for operators to make sure their favorite vendor can support this when they run their next purchase cycle.
Mark.
-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
On Tue, Feb 4, 2014 at 1:03 PM, Jared Mauch <jared@puck.nether.net> wrote:
On Feb 4, 2014, at 11:52 AM, William Herrin <bill@herrin.us> wrote:
Those that are up in arms about this stuff seem to not be the ones asking the vendors for features and fixes.
Like I said, the "tier 1's" can't be the source of the solution until they stop being part of the problem.
This is the attitude that I've seen elsewhere that is devoid of any meat. As I said before, we hit a big preventing the ability to do this even if we wanted to. The impact is drop all traffic or permit all in that case.
Hi Jared, I'm not confident you caught the implications of what I said. At the reciprocal peering link, you don't drop the spoofed traffic. You let it flow. You then charge a penalty when it turns out the peering traffic includes spoofed packets. The impact isn't drop or permit. It's dollars. Those who can't or won't control their customer links (where they trivially know what addresses are allowed) start to pay large amounts of money where they peer. More money than it takes to to properly implement customer-link filters so that they don't send spoofed packets to the peer. No new tech. No blocking. Just cashflow. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
Please let us know your results. Jared Mauch
On Feb 4, 2014, at 1:55 PM, William Herrin <bill@herrin.us> wrote:
On Tue, Feb 4, 2014 at 1:03 PM, Jared Mauch <jared@puck.nether.net> wrote:
On Feb 4, 2014, at 11:52 AM, William Herrin <bill@herrin.us> wrote: Those that are up in arms about this stuff seem to not be the ones asking the vendors for features and fixes.
Like I said, the "tier 1's" can't be the source of the solution until they stop being part of the problem.
This is the attitude that I've seen elsewhere that is devoid of any meat. As I said before, we hit a big preventing the ability to do this even if we wanted to. The impact is drop all traffic or permit all in that case.
Hi Jared,
I'm not confident you caught the implications of what I said. At the reciprocal peering link, you don't drop the spoofed traffic. You let it flow. You then charge a penalty when it turns out the peering traffic includes spoofed packets. The impact isn't drop or permit. It's dollars. Those who can't or won't control their customer links (where they trivially know what addresses are allowed) start to pay large amounts of money where they peer. More money than it takes to to properly implement customer-link filters so that they don't send spoofed packets to the peer.
No new tech. No blocking. Just cashflow.
Regards, Bill Herrin
-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
----- Original Message -----
From: "Jared Mauch" <jared@puck.nether.net>
Ask your vendors for these features. Ask them to fix the bugs. The ball rolls uphill here and it's in their lap. Blaming the carriers is wrongheaded and putting it where it doesn't belong in many cases. Happy to discuss offline.
I phrased that a bit more stridently here: http://www.bcp38.info/index.php/Information_for_equipment_manufacturers Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
On Feb 4, 2014, at 8:52 AM, William Herrin <bill@herrin.us> wrote:
On Tue, Feb 4, 2014 at 11:23 AM, Jared Mauch <jared@puck.nether.net> wrote:
On Feb 4, 2014, at 11:04 AM, William Herrin <bill@herrin.us> wrote:
If just three of the transit-free networks rewrote their peering contracts such that there was a $10k per day penalty for sending packets with source addresses the peer should reasonably have known were forged, this problem would go away in a matter of weeks.
I've seen similar comments in other forums. We are all generally paid for moving packets, not filtering them. The speed at which you can forward packets can often cause increased $$. Using these features also impacts performance, so the cost may actually be 2x in capex+opex to provision ports due to reduced line-rate capability.
Hi Jared,
You're gonna need a bigger TCAM, but even so I think you're overstating the case.
No, he's not. The intelligence required to analyze packets is in addition to the intelligence required to move them. More packets, more cost.
Even if you take a RPSL-IRR approach to building filters, and even if the router can handle such long ACLs bug-free, you have some objects that expand to cover 50-90% of the internet. They may be someones backup route at some point because of 'something'.
Yes, but that's OK. In order to make sure that they're aren't originating from the penalizing 10%, your peers will have to implement similar filtering downstream... where the breadth isn't 90%.
So who determines this break point? Who is responsible for a full-table Tier-1 to Tier-1 peering link? Who polices it? Who arbitrates disputes?
Clearly putting the filters as close to the source is helpful but detecting the actual spoofed packet is hard.
At the customer boundary it's trivial: they'll tell you what they originate, and that's what you'll allow. If your customer lies, pass the penalty forward.
At the peering boundary, you don't have to detect the forged packets. You can wait until someone complains, confirm it, and then apply the penalty. Packets coming from your peers won't go to your other peers, only to your customers. That's how you rigged your routing. More, evidence that the downstream was authorized to send those packets refutes the penalty.
You know this is completely unworkable at scale right?
Until you find yourself on the receiving end of these types of things, you may not ask for or pay for DDoS protection services, or advanced filtering, or even ask your vendor to support these features. I have to wait months for fixes in the features because no support from others in the industry on the platform, etc.
DDoS is a bigger problem than spoofing and amplification. My suggestion only addresses spoofing and amplification, not botnets in general.
But they have the same economic inputs, yes? As Jared said, providers get paid by the bit. Many (most?) Bad Actors get paid by the bit, Vendors get paid by the bit, mitigation vendors get paid by the bit. That's a lot of dollars for a lot of bits and they increase together.
Those that are up in arms about this stuff seem to not be the ones asking the vendors for features and fixes.
Like I said, the "tier 1's" can't be the source of the solution until they stop being part of the problem.
You are asking the guys who build and maintain the highways to be responsible for checking every car on the road to see if it's carrying illegal drugs. How can that possibly work? Mike
On 02/04/2014 08:04 AM, William Herrin wrote:
On Sun, Feb 2, 2014 at 5:17 PM, Cb B <cb.list6@gmail.com> wrote:
And, i agree bcp38 would help but that was published 14 years ago.
Howdy,
If just three of the transit-free networks rewrote their peering contracts such that there was a $10k per day penalty for sending packets with source addresses the peer should reasonably have known were forged, this problem would go away in a matter of weeks. Granted it would also be helpful to have a BGP extension signifying allowed-source-but-don't-route so that RP filtering would work even when multihomed. Still, even without automatic RP filtering we're capable of preventing spoofed packets if financially incentivized.
Thing is, they can't be the source of the solution until they stop being part of the problem.
Won't work because no one will sign that contract. The answer is lawsuits. People who are damaged by DDOS need to file suit against the networks that allowed the spoofed packets. Once it becomes more expensive to allow the spoofing (due to both damages and legal bills) than it is to prevent it, people will work harder to prevent it. Doug
On Tue, Feb 4, 2014 at 2:08 PM, Doug Barton <dougb@dougbarton.us> wrote:
On 02/04/2014 08:04 AM, William Herrin wrote:
If just three of the transit-free networks rewrote their peering contracts such that there was a $10k per day penalty for sending packets with source addresses the peer should reasonably have known were forged, this problem would go away in a matter of weeks.
Won't work because no one will sign that contract.
Hi Doug, Verizon Business is willing to do settlement-free peering with you but you won't agree to a reciprocal penalty if either allows its customers to forge packets? I call that a weed-out factor. Weed out the bad actors because anyone else would consider that peering arrangement too valuable to pass up. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Tue, Feb 4, 2014 at 2:28 PM, William Herrin <bill@herrin.us> wrote:
On Tue, Feb 4, 2014 at 2:08 PM, Doug Barton <dougb@dougbarton.us> wrote:
On 02/04/2014 08:04 AM, William Herrin wrote:
If just three of the transit-free networks rewrote their peering contracts such that there was a $10k per day penalty for sending packets with source addresses the peer should reasonably have known were forged, this problem would go away in a matter of weeks.
Won't work because no one will sign that contract.
Hi Doug,
Verizon Business is willing to do settlement-free peering with you but
you forgot an IF there, right? All of these 'get N tierM networks to peer and agree to penalties amongst eachother in the case of Y happening' discussions sound a LOT like longdistance settlement regimes. There's a nice fellow in tcpm/iccrwg in the ietf that's happy to talk a lot about 'red packets' and 'black packets' and congestion and cost shifting for this sort of thing. which frankly sounds almost exactly like the conversation about spoofed packets. In a world where folk connect to a peering fabric and default-route toward a peer, or never send routes to a peer yet prefer paths across that peer... or hell, do this with their ISP network connections. How does one tell that 'ISPX sent me a packet that is spoofed' ? how does that hold up in court? (which will happen eventually when the billing dispute goes south... and will happen months after the event in question.) It's a laudable goal, to do some enforcement of bcp38-like functions, but doing at SFP links is frankly impactical and bound to fail. Instead, concentrate on the customer edge of the problem and solve things there, eh? -chris
On Tue, Feb 04, 2014 at 02:28:22PM -0500, William Herrin wrote:
Verizon Business is willing to do settlement-free peering with you but you won't agree to a reciprocal penalty if either allows its customers to forge packets? I call that a weed-out factor. Weed out the bad actors because anyone else would consider that peering arrangement too valuable to pass up.
Bill, Are you willing to warrant the source, destination and lawful purpose of every single frame exiting your network? (Insert usual encouraging of competition to do same, etc., etc.) --msa
On Tue, Feb 4, 2014 at 2:48 PM, Majdi S. Abbas <msa@latt.net> wrote:
Are you willing to warrant the source, destination and lawful purpose of every single frame exiting your network?
Yes, no and no respectively. As a BGP leaf node, I can guarantee that no packets leave my network from a source address that isn't mine. The destination I can't control -- my software will respond to any apparently legitimate external query. As for legal, even if I'm fortunate enough to never have a customer hijacked into a botnet, there are a vast number of legal systems in the world and no practical way to be sure that a given communication is legitimate in all of them. But by golly I have total control over the source address in packets leaving my network. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
If just three of the transit-free networks rewrote their peering contracts such that there was a $10k per day penalty for sending packets with source addresses the peer should reasonably have known were forged, this problem would go away in a matter of weeks.
Won't work because no one will sign that contract.
Oh, right, how hard can it be to put a bell on that pesky cat? I was at a conference with people from some Very Large ISPs. They told me that many of their large customers absolutely will not let them do BCP38 filtering. ("If you don't want our business, we can find someone else who does.") The usual problem is that they have PA space from two providers and for various reasons, not all of which are stupid, traffic with provider A's addresses sometimes goes out through provider B. Adding to the excitement, some of these customers are medium sized ISPs with multihomed customers of their own. I don't know BGP well enough to know if it's possible to send out announcements for this situtation, this address range is us, but don't route traffic to it. Even if it is, not all of the customers do BGP, some are just stub networks. If we could figure out a reasonable way (i.e., one that the customers might be willing to implement) to handle this, it'll make BCP38 a lot more doable. R's, John
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2/4/2014 2:18 PM, John Levine wrote:
If just three of the transit-free networks rewrote their peering contracts such that there was a $10k per day penalty for sending packets with source addresses the peer should reasonably have known were forged, this problem would go away in a matter of weeks.
Won't work because no one will sign that contract.
Oh, right, how hard can it be to put a bell on that pesky cat?
I was at a conference with people from some Very Large ISPs. They told me that many of their large customers absolutely will not let them do BCP38 filtering. ("If you don't want our business, we can find someone else who does.") The usual problem is that they have PA space from two providers and for various reasons, not all of which are stupid, traffic with provider A's addresses sometimes goes out through provider B. Adding to the excitement, some of these customers are medium sized ISPs with multihomed customers of their own.
I don't know BGP well enough to know if it's possible to send out announcements for this situtation, this address range is us, but don't route traffic to it. Even if it is, not all of the customers do BGP, some are just stub networks.
If we could figure out a reasonable way (i.e., one that the customers might be willing to implement) to handle this, it'll make BCP38 a lot more doable.
BCP84? :-) - - ferg - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlLxaWoACgkQKJasdVTchbIy9AD/eILZC1RBKpcnSGfYvmWhkmiF L1egq0XmR2EqlG9ta5ABALrHWUwaV0COd5I6Mz6vZL2Zoa2AkO1w7DC6hvcGAIkM =R7VB -----END PGP SIGNATURE-----
On Tue, Feb 04, 2014 at 10:18:21PM -0000, John Levine wrote:
I was at a conference with people from some Very Large ISPs. They told me that many of their large customers absolutely will not let them do BCP38 filtering. ("If you don't want our business, we can find someone else who does.") The usual problem is that they have PA space from two providers and for various reasons, not all of which are stupid, traffic with provider A's addresses sometimes goes out through provider B. Adding to the excitement, some of these customers are medium sized ISPs with multihomed customers of their own.
I don't know BGP well enough to know if it's possible to send out announcements for this situtation, this address range is us, but don't route traffic to it. Even if it is, not all of the customers do BGP, some are just stub networks.
If we could figure out a reasonable way (i.e., one that the customers might be willing to implement) to handle this, it'll make BCP38 a lot more doable.
No-Advertise or No-Export community?
On Tue, Feb 4, 2014 at 5:18 PM, John Levine <johnl@iecc.com> wrote:
I was at a conference with people from some Very Large ISPs. They told me that many of their large customers absolutely will not let them do BCP38 filtering. ("If you don't want our business, we can find someone else who does.") The usual problem is that they have PA space from two providers and for various reasons, not all of which are stupid, traffic with provider A's addresses sometimes goes out through provider B.
Then: (A) It isn't spoofed traffic. The relevant block of ISP A's addresses should be permitted in ISP B's filter. It shouldn't even need much in the way of verification: confirm that the requested block is either relatively small and not obviously registered to someone else in rwhois, or confirm that it is registered to the customer in rwhois. (B) When it comes time to apply a penalty up at the peering sessions, those packets aren't eligible. The penalty can be refuted and, if based on those particular source addresses, dropped.
I don't know BGP well enough to know if it's possible to send out announcements for this situtation, this address range is us, but don't route traffic to it.
No. A BGP option could be added to support this, but in many cases the blocks in question are smaller than /24. The advertisements would end up filtered anyway. There really isn't a good technical solution to automated filtering at the reciprocal peering level. That part only works at the customer edge. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On 04/02/14 14:18, John Levine wrote:
I was at a conference with people from some Very Large ISPs. They told me that many of their large customers absolutely will not let them do BCP38 filtering. ("If you don't want our business, we can find someone else who does.") The usual problem is that they have PA space from two providers and for various reasons, not all of which are stupid, traffic with provider A's addresses sometimes goes out through provider B. Adding to the excitement, some of these customers are medium sized ISPs with multihomed customers of their own.
I haven't read it all, but section 3 says:
However, by restricting transit traffic which originates from a downstream network to known, and intentionally advertised, prefix(es), the problem of source address spoofing can be virtually eliminated in this attack scenario.
If ISP has customer A with multiple *known* valid networks --doesn't matter if ISP allocated them to customer or not-- and ISP lets them all out, but filters everything else, ISP is still complying with BCP 38. Here it's not a matter of blocking "just because". It's blocking unknown addresses. It doesn't either mean that ISP should not open the filters if a new prefix is requested by the customer.
If ISP has customer A with multiple *known* valid networks --doesn't matter if ISP allocated them to customer or not-- and ISP lets them all out, but filters everything else, ISP is still complying with BCP 38.
Of course. The question is how the ISP knows what the customer's address ranges are, without demanding vast amounts of nitpicky manual work on both sides. Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Please consider the environment before reading this e-mail. http://jl.ly
On Tue, Feb 4, 2014 at 6:24 PM, John R. Levine <johnl@iecc.com> wrote:
If ISP has customer A with multiple *known* valid networks --doesn't matter if ISP allocated them to customer or not-- and ISP lets them all out, but filters everything else, ISP is still complying with BCP 38.
Of course. The question is how the ISP knows what the customer's address ranges are, without demanding vast amounts of nitpicky manual work on both sides.
Hi John, "whois 1.2.3.4" Why does it have to be hard? Restricting the filter to addresses which (A) the customer asserts are theirs and (B) aren't obviously registered to someone else would more than solve the problem. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
Why does it have to be hard? Restricting the filter to addresses which (A) the customer asserts are theirs
How does the customer do that in a way that scales? I don't think any of this is rocket science, but it apparently is a real block to BCP38/84 implementatin. R's, John
In message <20140205002905.57856.qmail@joyce.lan>, "John Levine" writes:
Why does it have to be hard? Restricting the filter to addresses which (A) the customer asserts are theirs
How does the customer do that in a way that scales?
You implement SIDR to the extent where you issue your multi homed customers CERTs for the address space you allocated to them. The customer can then just send signed requests to a automated service at the other ISPs along with the CERT which then builds the filters based on that information after verifying the CERTs authenticity. Now all of the above is completely automatable including the CERT generation. Yes, it requires someone to write a implementation and integrate it with the existing systems.
I don't think any of this is rocket science, but it apparently is a real block to BCP38/84 implementatin.
No, this isn't rocket science. It just requires a little co-ordination.
R's, John -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
How does the customer do that in a way that scales? You implement SIDR to the extent where you issue your multi homed customers CERTs for the address space you allocated to them. The customer can then just send signed requests to a automated service at the other ISPs along with the CERT which then builds the filters based on that information after verifying the CERTs authenticity.
you have done this in your network, the isp for which you are an engineer? randy
----- Original Message -----
From: "John Levine" <johnl@iecc.com>
Subject: Re: BCP38 is hard, was TWC (AS11351) blocking all NTP?
Why does it have to be hard? Restricting the filter to addresses which (A) the customer asserts are theirs
How does the customer do that in a way that scales?
I don't think any of this is rocket science, but it apparently is a real block to BCP38/84 implementatin.
Well, there are two issues: how many exceptions at the transit layer will actually be needed, in practice... and how much trouble will there still be there if we can get appreciable uptake at the edge? Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
On (2014-02-05 00:29 -0000), John Levine wrote:
Why does it have to be hard? Restricting the filter to addresses which (A) the customer asserts are theirs
How does the customer do that in a way that scales?
I don't think any of this is rocket science, but it apparently is a real block to BCP38/84 implementatin.
Transit provider can do ACL, in some platforms it can be 100% same object as used for BGP. Then setup ultimate rule to allow and log. Then cooperate with customer to weed through the unexpected, until none remain and flip the allow to deny. But I guess no one is saying it cannot be done, more that there is no pay-off in it. Transit provider is compensated for bits transferred, spending money to receive less money may not appeal to people in charge. You also wrote:
I was at a conference with people from some Very Large ISPs. They told me that many of their large customers absolutely will not let them do BCP38 filtering. ("If you don't want our business, we can find someone else who does.") The usual problem is that they have PA space from two providers and for various reasons, not all of which are stupid, traffic with provider A's addresses sometimes goes out through provider B. Adding to the excitement, some of these customers are medium sized ISPs with multihomed customers of their own.
Someone who worked for such ISP, told they don't accept BCP38, because their business is to sell services to instances who want to spoof for what ever reason. The official reasons told to upstreams are different. He didn't appreciate the business and no longer works for said ISP. If what you say was actual reason, it could be solved by logging ACL. We the community, could produce tooling to automate this in few popular platforms. Automatically builds the ACL, web interface for humans to classify the logged/unknown. When classified by human as legit source, automatically create route object for it. Recreate ACL from route-objects, submit to router. Repeat until human operator is confident no further classification is needed, and ask tool to swap log+permit + deny. Probably takes like maybe 50h development work. -- ++ytti, not it
On Feb 5, 2014, at 3:35 AM, Saku Ytti <saku@ytti.fi> wrote:
If what you say was actual reason, it could be solved by logging ACL.
We the community, could produce tooling to automate this in few popular platforms. Automatically builds the ACL, web interface for humans to classify the logged/unknown. When classified by human as legit source, automatically create route object for it. Recreate ACL from route-objects, submit to router.
The problem is many of these can compile to larger than the physical amount of space in the router/LC have to handle it. I’ve done presentations to vendors about what percentage (in bytes and per-line) of the configuration is of what component. 90%+ tends to be customer-specific prefix-list/set/filter lines. These can easily reach many megabytes of configuration and tens or hundreds of thousands of lines. Asking someone to duplicate that to also have an ingress ACL of equivalent size, and *assuming* the router can handle that ACL and compile it properly is a challenge to say the least.
Repeat until human operator is confident no further classification is needed, and ask tool to swap log+permit + deny.
Similar to the above, doing the log permit, etc.. is all dependent on the platform and what scale is feasible. Some devices you can’t do things like log-input and capture the ingress MAC that originated the packet as it’s been stripped off before it gets to that part of the engine. Similar to Randys previous comments, I would like to see another operator talk about their efforts here that has actually implemented something and is willing to share. Right now, I’ve seen a lot of people say what others should do with “their” network, and limited data about what they have done to help solve this problem. It’s harder than it seems, and even those that invite regulation and other things, the technology isn’t capable because it’s not something folks “ask for”.
Probably takes like maybe 50h development work.
Let me know how that goes. I’ve found estimates for this stuff can be off by as much as 10x + once all the details are chased down. my wife has regularly been very patient with me when i say “10 minutes” and it’s closer to 2+ hours. I know we can do better than what the state is today, but there’s only so much that one network can do. - Jared
On (2014-02-05 11:15 -0500), Jared Mauch wrote:
The problem is many of these can compile to larger than the physical amount of space in the router/LC have to handle it. I’ve done presentations to vendors about what percentage (in bytes and per-line) of the configuration is of what component. 90%+ tends to be customer-specific prefix-list/set/filter lines.
Absolutely. But the good thing is, we don't need to have it comprehensively deployed in transit scenarios, just as long as spoofing domains are sufficiently fragmented DoS attack gets get better pay off from not spoofing. -- ++ytti
On 04/02/14 15:24, John R. Levine wrote:
If ISP has customer A with multiple *known* valid networks --doesn't matter if ISP allocated them to customer or not-- and ISP lets them all out, but filters everything else, ISP is still complying with BCP 38.
Of course. The question is how the ISP knows what the customer's address ranges are, without demanding vast amounts of nitpicky manual work on both sides.
The same way BGP outbound prefix filters are applied nowadays, would be a good start. Some have BGP filtering but don't have ACLs.
In message <52F17102.2000505@alvarezp.ods.org>, Octavio Alvarez writes:
On 04/02/14 14:18, John Levine wrote:
I was at a conference with people from some Very Large ISPs. They told me that many of their large customers absolutely will not let them do BCP38 filtering. ("If you don't want our business, we can find someone else who does.") The usual problem is that they have PA space from two providers and for various reasons, not all of which are stupid, traffic with provider A's addresses sometimes goes out through provider B. Adding to the excitement, some of these customers are medium sized ISPs with multihomed customers of their own.
I haven't read it all, but section 3 says:
However, by restricting transit traffic which originates from a downstream network to known, and intentionally advertised, prefix(es), the problem of source address spoofing can be virtually eliminated in this attack scenario.
If ISP has customer A with multiple *known* valid networks --doesn't matter if ISP allocated them to customer or not-- and ISP lets them all out, but filters everything else, ISP is still complying with BCP 38.
Here it's not a matter of blocking "just because". It's blocking unknown addresses. It doesn't either mean that ISP should not open the filters if a new prefix is requested by the customer.
Or downstream customers. SIDR provides provides the crypotographic glue that can be used to automate this. The end customer has a CERT which authenticates their use of the address. The second ISP supplies a CERT which the end customer signs saying they can source this range. Repeat until you reach a big enough ISP that you just have a agreement that no unverified traffic will injected down the link. These CERTS can then be used to build perfect input and output filters. Initially you may have to have manual inputs but with increasing use of SIDR the amount of manual intervention will drop. I.e. If you multi-home you need to have provable use of the address space. This isn't significantly different to the regular use of SIDR. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Tue, Feb 4, 2014 at 11:08 AM, Doug Barton <dougb@dougbarton.us> wrote:
The answer is lawsuits. People who are damaged by DDOS need to file suit against the networks that allowed the spoofed packets. Once it becomes more expensive to allow the spoofing (due to both damages and legal bills) than it is to prevent it, people will work harder to prevent it.
+1 for this. While lawsuits rarely improve a situation, I agree it's probably the only way to shift costs back to the bad networks. But then the problem shifts to one of detection and tracing. The bad networks can only be identified if the transit providers have netflow. When I ask transit providers to trace spoofed packets they either don't respond or claim their netflow was temporarily broken. It's not just transit providers, though -- many spoofed attacks come through IXPs. To help, the IXPs need to provide sflow that shows which peers traffic is coming from. I've seen some basic functionality at AMS-IX for this, but unfortunately it's just rrd graphs, not full data. Still, they're better than most. And then the IXPs need to have a policy forbidding spoofed packets. Damian
participants (42)
-
Brian Rak -
Cb B -
Chris Grundemann -
Christopher Morrow -
Christopher Morrow -
Chuck Anderson -
Damian Menscher -
Dobbins, Roland -
Doug Barton -
Frank Bulk -
goemon＠anime.net -
jamie rishaw -
Jared Mauch -
Jay Ashworth -
Jimmy Hess -
joel jaeggli -
John Curran -
John Kristoff -
John Levine -
John R. Levine
-
Jonathan Towne -
Leo Bicknell -
Livingood, Jason -
Majdi S. Abbas -
Mark Andrews -
Mark Tinka -
Matthew Petach -
Michael DeMan -
Michael Smith -
Octavio Alvarez -
Paul Ferguson -
Peter Kristolaitis -
Randy Bush -
Robert Drake -
ryangard＠gmail.com -
Saku Ytti -
Seth Mattinen -
Stephane Bortzmeyer -
TGLASSEY -
Tony Tauber -
Valdis.Kletnieks＠vt.edu -
William Herrin