Tier1 blackholing policy?
Greetings, I know Tier1s are blackholing traffic all the time :) (de-peering, congestion etc.) but did it became a new role for Tier1s to go from transit provider to transit blocker? We received recently customer complaints stating they can't reach certain websites. Investigation showed that the sites were not reachable via Tier1-T, but fine via Tier1-L. I contacted Tier1-T and the answer was something like "yeah, this is a known phishing site and to protect our customers we blackhole that IP" (btw - it was 2 ASes away from Tier1-T). Huh? If I want to block something there, it should me my decision or that of my country's legal entities by court order and not being decided by some Tier1's intransparent security department. (Not even mentioning words like 'CGN', 'legal', 'net neutrality' or 'censorship') This might be an acceptable policy for a cable provider but not for a Tier1. Haven't seen something like this in many years. Did I miss a pardigm-shift here and has this become a common "service" at Tier1s? Thomas
On 4/30/2013 10:31 AM, Thomas Schmid wrote:
Greetings,
I know Tier1s are blackholing traffic all the time :) (de-peering, congestion etc.) but did it became a new role for Tier1s to go from transit provider to transit blocker?
We received recently customer complaints stating they can't reach certain websites. Investigation showed that the sites were not reachable via Tier1-T, but fine via Tier1-L. I contacted Tier1-T and the answer was something like "yeah, this is a known phishing site and to protect our customers we blackhole that IP" (btw - it was 2 ASes away from Tier1-T).
Huh? If I want to block something there, it should me my decision or that of my country's legal entities by court order and not being decided by some Tier1's intransparent security department. (Not even mentioning words like 'CGN', 'legal', 'net neutrality' or 'censorship') This might be an acceptable policy for a cable provider but not for a Tier1.
Haven't seen something like this in many years. Did I miss a pardigm-shift here and has this become a common "service" at Tier1s?
Thomas
Ideally what should a Tier 1 or default-free network do in this situation[1]? 1) Do nothing - They're supposed deliver any and all bits (Disregarding a DoS or similiar situation which impedes said network) 2) Prefix filter - Don't be a party (at least in one direction) to the bad actors traffic. 3) ? [1] Assuming there is some sort of security and/or wrongdoing event that isn't getting resolved via contact with their peer.
On Tue, 2013-04-30 at 10:59 -0400, ML wrote:
1) Do nothing - They're supposed deliver any and all bits (Disregarding a DoS or similiar situation which impedes said network) 2) Prefix filter - Don't be a party (at least in one direction) to the bad actors traffic.
3 - Deliver all packets unless I've signed up for an enhanced security offering? --Chris
Sounds like a no win situation. Either you let the bad guys do things or get complaints you blocked the bad guys. Jared Mauch On Apr 30, 2013, at 11:07 AM, Chris Boyd <cboyd@gizmopartners.com> wrote:
On Tue, 2013-04-30 at 10:59 -0400, ML wrote:
1) Do nothing - They're supposed deliver any and all bits (Disregarding a DoS or similiar situation which impedes said network) 2) Prefix filter - Don't be a party (at least in one direction) to the bad actors traffic.
3 - Deliver all packets unless I've signed up for an enhanced security offering?
--Chris
I think blocking phishing sites vs blocking ddos require a different approach. -- Tassos Jared Mauch wrote on 30/04/2013 18:11:
Sounds like a no win situation. Either you let the bad guys do things or get complaints you blocked the bad guys.
Jared Mauch
On Apr 30, 2013, at 11:07 AM, Chris Boyd <cboyd@gizmopartners.com> wrote:
On Tue, 2013-04-30 at 10:59 -0400, ML wrote:
1) Do nothing - They're supposed deliver any and all bits (Disregarding a DoS or similiar situation which impedes said network) 2) Prefix filter - Don't be a party (at least in one direction) to the bad actors traffic. 3 - Deliver all packets unless I've signed up for an enhanced security offering?
--Chris
On Tue, Apr 30, 2013 at 11:22 AM, Tassos Chatzithomaoglou <achatz@forthnetgroup.gr> wrote:
I think blocking phishing sites vs blocking ddos require a different approach.
I think I agree with this, and I think it can help draw a useful line. Large DDoS attacks can and do directly affect the service that the "tier 1" is providing to its customers (namely, moving their bits), so filtering such attacks seems like a reasonably agreeable thing by really anyone I think. Phishing on the other hand will not really stop bits from moving (except perhaps through rather long chain of unlikely things that'd have to happen). The last-mile consumer ISPs don't just "move bits" for their customers really, its more about providing "internet" (which is a different concept to normal users) -- and this is where filtering phishing sites and blocking port 25 and such makes much more sense, because these users will have a highly degraded experience if they become a botnet drone or some such thing. Granted, as Patrick says, "tier 1" isn't really a thing, and they have a mix of customers, but I think its safe to say that these "tier 1" providers should apply different policies for different types of customers, because they are providering different services (even if the underlying technology is the same/similar). -- Darius Jahandarie
On Apr 30, 2013, at 12:43 PM, Darius Jahandarie <djahandarie@gmail.com> wrote:
I think I agree with this, and I think it can help draw a useful line.
Large DDoS attacks can and do directly affect the service that the "tier 1" is providing to its customers (namely, moving their bits), so filtering such attacks seems like a reasonably agreeable thing by really anyone I think.
Phishing on the other hand will not really stop bits from moving (except perhaps through rather long chain of unlikely things that'd have to happen).
The last-mile consumer ISPs don't just "move bits" for their customers really, its more about providing "internet" (which is a different concept to normal users) -- and this is where filtering phishing sites and blocking port 25 and such makes much more sense, because these users will have a highly degraded experience if they become a botnet drone or some such thing.
If the phishing attack is against an enterprise that is also an ISP, surely you can imagine a case where they might block traffic to prevent folks from being phished. i think it's great that someone is blocking folks from being infected with either malware or giving up their private details improperly. Typically these sites are hacked anyways or something else. I think that keeping the broadest set of people from being phished or compromised is a good thing(tm). Typically a site is cleaned up in a few hours or day or two without trouble. If your communication is that urgent, there are other methods like phone to communicate with the other party. not ideal, but they do exist. - jared
On Tue, Apr 30, 2013 at 12:47:40PM -0400, Jared Mauch wrote:
If the phishing attack is against an enterprise that is also an ISP, surely you can imagine a case where they might block traffic to prevent folks from being phished.
This is not an effective anti-phishing tactic, any more than "user education" is an effective anti-phishing tactic. (Let me quote Marcus Ranum on the latter: "if it was going to work, it would have worked by now." And let me observe: it's never worked; it's not working; it's never going to work.)
i think it's great that someone is blocking folks from being infected with either malware or giving up their private details improperly.
One person's "malware" is merely an interesting collection of inert bits to someone else, just as "email virus" has no operational meaning to anyone clueful enough to run a sensible mail client on a sensible operating system. Thus one undesirable effect of such blocking is that it denies access to researchers who are at nearly zero risk of negative consequences *and* who might be the very people in a position to understand the threat (phishing, malware, etc.) and figure out how to mitigate it. Another is that it presents a false sense of security to the ignorant, the lazy, and the careless. While in the short term that may seem benevolent and useful, I think in the long term it has a deleterious effect on security as a whole. And if we've arrived at a point in time where people are actually considering making routing decisions based on longstanding design and implementation defects in consumer operating systems and applications, then I think "long term" equates to "right now". ---rsk
On May 1, 2013, at 7:44 AM, Rich Kulawiec <rsk@gsp.org> wrote:
On Tue, Apr 30, 2013 at 12:47:40PM -0400, Jared Mauch wrote:
If the phishing attack is against an enterprise that is also an ISP, surely you can imagine a case where they might block traffic to prevent folks from being phished.
This is not an effective anti-phishing tactic, any more than "user education" is an effective anti-phishing tactic. (Let me quote Marcus Ranum on the latter: "if it was going to work, it would have worked by now." And let me observe: it's never worked; it's not working; it's never going to work.)
We're talking about denying access to what is typically a compromised end-host which is in violation of an AUP. Speaking about my employer, we typically don't see something null0'ed for more than a few hours until we have confirmed the host is offline being repaired. I don't know about other networks practices which is what started the thread.
i think it's great that someone is blocking folks from being infected with either malware or giving up their private details improperly.
One person's "malware" is merely an interesting collection of inert bits to someone else, just as "email virus" has no operational meaning to anyone clueful enough to run a sensible mail client on a sensible operating system.
Thus one undesirable effect of such blocking is that it denies access to researchers who are at nearly zero risk of negative consequences *and* who might be the very people in a position to understand the threat (phishing, malware, etc.) and figure out how to mitigate it. Another is that it presents a false sense of security to the ignorant, the lazy, and the careless. While in the short term that may seem benevolent and useful, I think in the long term it has a deleterious effect on security as a whole. And if we've arrived at a point in time where people are actually considering making routing decisions based on longstanding design and implementation defects in consumer operating systems and applications, then I think "long term" equates to "right now".
I think many people understand these risks and tradeoffs. We could stop mitigating DDoS attacks or responding to security complaints as well with this line of reasoning as it could be interfering with law-enforcement actions, or a researcher. Just because the house has been broken into, doesn't mean as the provider of the roads that we're going to let everyone visit it until the owner has a chance to secure it properly. I don't like that role, but it becomes necessary at times. What you are suggesting is a slippery slope to no mitigation of any badness which will lead to a lack of trust and confidence in the market. That to me is a plain and simple reason to do the right thing, even if it causes a problem for a few hours or a day or two. - Jared
On Apr 30, 2013, at 11:07 , Chris Boyd <cboyd@gizmopartners.com> wrote:
On Tue, 2013-04-30 at 10:59 -0400, ML wrote:
1) Do nothing - They're supposed deliver any and all bits (Disregarding a DoS or similiar situation which impedes said network) 2) Prefix filter - Don't be a party (at least in one direction) to the bad actors traffic.
3 - Deliver all packets unless I've signed up for an enhanced security offering?
While I like that plan, there are a LOT more people who will scream about not being "protected" than those who will bitch they can't get to a phishing site. Since networks are for-profit companies, they'll lower their costs (e.g. support calls), as long as it lowers their cost more than the "cost" of losing a customer or two (and let's be honest, that is about all they _might_ lose) who are religious about the whole "transit means everywhere" thing. -- TTFN, patrick
On Apr 30, 2013, at 10:07 PM, Chris Boyd wrote:
3 - Deliver all packets unless I've signed up for an enhanced security offering?
Even if said packets from an obviously compromised server on a high-speed link are attack packets causing problems for the ISP itself as well as for its customers? Trust me, large transit ISPs don't *want* to be in the blackholing business. They only do so when they're forced into it by necessity (operational, legal, regulatory). Also note that in the case of the server(s) you can't access, they may well be on shared hosting with thousands of sites/accounts on a single IP, one or more of which may be compromised. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On 30.04.2013 17:07, Chris Boyd wrote:
On Tue, 2013-04-30 at 10:59 -0400, ML wrote:
1) Do nothing - They're supposed deliver any and all bits (Disregarding a DoS or similiar situation which impedes said network) 2) Prefix filter - Don't be a party (at least in one direction) to the bad actors traffic.
3 - Deliver all packets unless I've signed up for an enhanced security offering?
right - I see this really as something that should be decided at the edge of the internet (Tier2+) and not in the core.
On Apr 30, 2013, at 11:23 , Thomas Schmid <schmid@dfn.de> wrote:
On 30.04.2013 17:07, Chris Boyd wrote:
On Tue, 2013-04-30 at 10:59 -0400, ML wrote:
1) Do nothing - They're supposed deliver any and all bits (Disregarding a DoS or similiar situation which impedes said network) 2) Prefix filter - Don't be a party (at least in one direction) to the bad actors traffic.
3 - Deliver all packets unless I've signed up for an enhanced security offering?
right - I see this really as something that should be decided at the edge of the internet (Tier2+) and not in the core.
"Core"? Seriously? Which of these statements are true: A) Is it impossible for an end user or business (i.e. non-ISP) to get a direct connection to a "Tier 1" (whatever the hell that means) provider. B) Most traffic on the Internet traverses Tier 1s today. C) A Tier 1 has a different profit motive than a Tier 2 (whatever the hell that means) providers. D) All Tier 1 providers are larger than all Tier 2 providers. We'll just skip over the E) all of the above. -- TTFN, patrick P.S. Hint: If you answered A, B, C, or D, you aren't paying attention.
Am 30.04.2013 17:53, schrieb Patrick W. Gilmore:
"Core"? Seriously? Which of these statements are true: A) Is it impossible for an end user or business (i.e. non-ISP) to get a direct connection to a "Tier 1" (whatever the hell that means) provider. B) Most traffic on the Internet traverses Tier 1s today. C) A Tier 1 has a different profit motive than a Tier 2 (whatever the hell that means) providers. D) All Tier 1 providers are larger than all Tier 2 providers. We'll just skip over the E) all of the above.
agree - I oversimplified, but I think you got the idea ... Thomas
Composed on a virtual keyboard, please forgive typos. On Apr 30, 2013, at 12:32, Thomas Schmid <schmid@dfn.de> wrote:
Am 30.04.2013 17:53, schrieb Patrick W. Gilmore:
"Core"? Seriously? Which of these statements are true: A) Is it impossible for an end user or business (i.e. non-ISP) to get a direct connection to a "Tier 1" (whatever the hell that means) provider. B) Most traffic on the Internet traverses Tier 1s today. C) A Tier 1 has a different profit motive than a Tier 2 (whatever the hell that means) providers. D) All Tier 1 providers are larger than all Tier 2 providers. We'll just skip over the E) all of the above.
agree - I oversimplified, but I think you got the idea ...
No, I did not get the point. I am not trolling. I just do not understand what you meant. Probably because there is no "core", so your statement did not make sense. -- TTFN, patrick
Am 30.04.2013 18:41, schrieb Patrick W. Gilmore:
Composed on a virtual keyboard, please forgive typos.
On Apr 30, 2013, at 12:32, Thomas Schmid <schmid@dfn.de> wrote:
Am 30.04.2013 17:53, schrieb Patrick W. Gilmore:
"Core"? Seriously? Which of these statements are true: A) Is it impossible for an end user or business (i.e. non-ISP) to get a direct connection to a "Tier 1" (whatever the hell that means) provider. B) Most traffic on the Internet traverses Tier 1s today. C) A Tier 1 has a different profit motive than a Tier 2 (whatever the hell that means) providers. D) All Tier 1 providers are larger than all Tier 2 providers. We'll just skip over the E) all of the above. agree - I oversimplified, but I think you got the idea ... No, I did not get the point.
I am not trolling. I just do not understand what you meant. Probably because there is no "core", so your statement did not make sense.
Patrick, what I mean is that someone that I pay money for providing me access to the guys I don't peer with, decides for me what's good (according to his criteria) for me and my customers or even my customer's customers etc. If one of my peers blackholes his customers, it's his business and not mine and I don't care. While I eventually could vote with my wallet if I don't like that policy, my question was more, if that behavior is already that common at 'Tier1s' (definition omitted) that it would not make a difference anyway. Thomas
On 30.04.2013 17:07, Chris Boyd wrote:
On Tue, 2013-04-30 at 10:59 -0400, ML wrote:
1) Do nothing - They're supposed deliver any and all bits (Disregarding a DoS or similiar situation which impedes said network) 2) Prefix filter - Don't be a party (at least in one direction) to the bad actors traffic.
3 - Deliver all packets unless I've signed up for an enhanced security offering?
right - I see this really as something that should be decided at the edge of the internet (Tier2+) and not in the core. You seem to have odd ideas about what it means to be a settlement free
On 4/30/13 8:23 AM, Thomas Schmid wrote: provider. Most of their customers are not smaller internet service providers.
Joel, Am 30.04.2013 18:00, schrieb joel jaeggli:
On 30.04.2013 17:07, Chris Boyd wrote:
On Tue, 2013-04-30 at 10:59 -0400, ML wrote:
1) Do nothing - They're supposed deliver any and all bits (Disregarding a DoS or similiar situation which impedes said network) 2) Prefix filter - Don't be a party (at least in one direction) to the bad actors traffic.
3 - Deliver all packets unless I've signed up for an enhanced security offering?
right - I see this really as something that should be decided at the edge of the internet (Tier2+) and not in the core. You seem to have odd ideas about what it means to be a settlement free
On 4/30/13 8:23 AM, Thomas Schmid wrote: provider. Most of their customers are not smaller internet service providers.
I know what it means to be a customer of $LargeGlobalISPthatsellsTransittootherISPs since 1995 and I have *never* seen one of these guys blackholing single IPs on their own (and I'm not talking about RTB, botnet controllers that threaten to kill the internet etc.). Now since a few weeks we get regular complaints about this. So something has changed. The sensitive approach would really be to make this an opt-in service for their customers and not a default service without opt-out option. In times of CGN and hundrets or thousands of websites behind one IP, blocking addresses is not the right answer to the phishing problem. Thomas
On May 1, 2013, at 4:40 PM, Thomas Schmid wrote:
Now since a few weeks we get regular complaints about this. So something has changed.
Yes, things have changed. There are reasons that some of the transit ISPs are performing this blocking. They aren't doing it for kicks. For example, there are non-insignificant numbers of servers/accounts which have been compromised and used to launch large-scale, high-impact DDoS attacks. The negative impact of allowing these servers to emit attack traffic far outweighs the inconvenience experienced by a few end-customers trying to access these servers (which are compromised, anyways, and therefore it isn't a good idea to try and access them in the first place). Suggest you ask the transit ISPs in question directly. You aren't likely to get an authoritative answer on a public email list. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On 05/01/2013 05:40 AM, Thomas Schmid wrote:
Joel,
Am 30.04.2013 18:00, schrieb joel jaeggli:
On 4/30/13 8:23 AM, Thomas Schmid wrote:
On 30.04.2013 17:07, Chris Boyd wrote:
On Tue, 2013-04-30 at 10:59 -0400, ML wrote:
1) Do nothing - They're supposed deliver any and all bits (Disregarding a DoS or similiar situation which impedes said network) 2) Prefix filter - Don't be a party (at least in one direction) to the bad actors traffic.
3 - Deliver all packets unless I've signed up for an enhanced security offering?
right - I see this really as something that should be decided at the edge of the internet (Tier2+) and not in the core. You seem to have odd ideas about what it means to be a settlement free provider. Most of their customers are not smaller internet service providers.
I know what it means to be a customer of $LargeGlobalISPthatsellsTransittootherISPs since 1995 and I have *never* seen one of these guys blackholing single IPs on their own (and I'm not talking about RTB, botnet controllers that threaten to kill the internet etc.). Now since a few weeks we get regular complaints about this. So something has changed.
The sensitive approach would really be to make this an opt-in service for their customers and not a default service without opt-out option. In times of CGN and hundrets or thousands of websites behind one IP, blocking addresses is not the right answer to the phishing problem.
... or perhaps on an internet where many network owners block / police / throttle packets by source or destination, implementing CGN or stacking thousands of websites behind one IP address are poor solutions to the connectivity problem. My only issue is the lack of information provided when blocks go into place. I would love to see networks provide information publicly that shows what is being blocked along with a description of why. A history that extends for a few days would be a bonus. -DMM
Le 01/05/2013 14:46, David Miller a écrit :
On 05/01/2013 05:40 AM, Thomas Schmid wrote:
Joel,
On 4/30/13 8:23 AM, Thomas Schmid wrote:
On 30.04.2013 17:07, Chris Boyd wrote:
On Tue, 2013-04-30 at 10:59 -0400, ML wrote:
1) Do nothing - They're supposed deliver any and all bits (Disregarding a DoS or similiar situation which impedes said network) 2) Prefix filter - Don't be a party (at least in one direction) to the bad actors traffic. 3 - Deliver all packets unless I've signed up for an enhanced security offering?
right - I see this really as something that should be decided at the edge of the internet (Tier2+) and not in the core. You seem to have odd ideas about what it means to be a settlement free provider. Most of their customers are not smaller internet service providers. I know what it means to be a customer of $LargeGlobalISPthatsellsTransittootherISPs since 1995 and I have *never* seen one of these guys blackholing single IPs on their own (and I'm not talking about RTB, botnet controllers that threaten to kill
Am 30.04.2013 18:00, schrieb joel jaeggli: the internet etc.). Now since a few weeks we get regular complaints about this. So something has changed.
The sensitive approach would really be to make this an opt-in service for their customers and not a default service without opt-out option. In times of CGN and hundrets or thousands of websites behind one IP, blocking addresses is not the right answer to the phishing problem.
... or perhaps on an internet where many network owners block / police / throttle packets by source or destination, implementing CGN or stacking thousands of websites behind one IP address are poor solutions to the connectivity problem.
My only issue is the lack of information provided when blocks go into place. I would love to see networks provide information publicly that shows what is being blocked along with a description of why. A history that extends for a few days would be a bonus.
I agree with that. While some blocking and policing may be judged "good thing" there is a well-known potential for "other kinds" of policing... Cheers, mh
-DMM
On Tue, 30 Apr 2013, Thomas Schmid wrote:
I know Tier1s are blackholing traffic all the time :) (de-peering, congestion etc.) but did it became a new role for Tier1s to go from transit provider to transit blocker?
We received recently customer complaints stating they can't reach certain websites. Investigation showed that the sites were not reachable via Tier1-T, but fine via Tier1-L. I contacted Tier1-T and the answer was something like "yeah, this is a known phishing site and to protect our customers we blackhole that IP" (btw - it was 2 ASes away from Tier1-T).
Huh? If I want to block something there, it should me my decision or that of my country's legal entities by court order and not being decided by some Tier1's intransparent security department. (Not even mentioning words like 'CGN', 'legal', 'net neutrality' or 'censorship') This might be an acceptable policy for a cable provider but not for a Tier1.
Haven't seen something like this in many years. Did I miss a pardigm-shift here and has this become a common "service" at Tier1s?
I vaguely recall having the same sort of problem many years ago with Above.net transit. IIRC, the sentiment back then was similarly that this was inappropriate behavior for a Tier1/2 transit provider. If you're going to propagate the routes, deliver the traffic. I suppose an argument could be made though that if there's phishing or malicious traffic targeting your customers from a single IP, it could be appropriate to blackhole the IP rather than reject the advertisement for an entire CIDR. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On Tue, Apr 30, 2013 at 10:31 AM, Thomas Schmid <schmid@dfn.de> wrote:
We received recently customer complaints stating they can't reach certain websites. Investigation showed that the sites were not reachable via Tier1-T, but fine via Tier1-L. I contacted Tier1-T and the answer was something like "yeah, this is a known phishing site and to protect our customers we blackhole that IP" (btw - it was 2 ASes away from Tier1-T).
Hi Thomas, On the one hand, companies providing Internet transit are not generally compelled by law to pass packets for any other given company on the Internet. On the other hand, announcing via BGP that you will carry particular packets and then intentionally dropping them on the floor could easily be construed as tortious interference. The middle ground... propagating a BGP announcement but blocking a small piece within it... I think I'd want to cover my backside by setting a BGP community on that route which advised my peers that a portion of it is dead-routed within my network so that they may discard or deprioritize it if they choose. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
participants (14)
-
Chris Boyd -
Darius Jahandarie -
David Miller -
Dobbins, Roland -
Jared Mauch -
joel jaeggli -
Jon Lewis -
Michael Hallgren -
ML -
Patrick W. Gilmore -
Rich Kulawiec -
Tassos Chatzithomaoglou -
Thomas Schmid -
William Herrin