PlayStationNetwork blocking of CGNAT public addresses
All, We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers. We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address. Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves. Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it? Many thanks in advance, Simon
A network that doesn't support IPv6, yet discriminates against CGNAT? That seems like a promising future. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Simon Lockhart" <simon@slimey.org> To: nanog@nanog.org Sent: Friday, September 16, 2016 8:12:46 AM Subject: PlayStationNetwork blocking of CGNAT public addresses All, We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers. We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address. Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves. Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it? Many thanks in advance, Simon
On 16 Sep 2016, at 20:12, Simon Lockhart wrote:
Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it?
I'm pretty sure that at least part of it has to do with DDoS-related activity. The best bet is to try and identify and engage with the relevant operational personnel with clue. Going the customer-service route isn't fruitful, as you indicate. Another aspect is ensuring that one has the ability to detect, classify, traceback, and mitigate outbound badness southbound of the CGN. This sort of thing has always been a problem with NAT; as CGN becomes more prevalent on wireline broadband networks, it's only going to get worse. AFAIK, PSN doesn't support IPv6. That would be another topic of discussion with the operational folks. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
On Fri Sep 16, 2016 at 08:32:12PM +0700, Roland Dobbins wrote:
Another aspect is ensuring that one has the ability to detect, classify, traceback, and mitigate outbound badness southbound of the CGN.
Unless PSN can tell us what traffic they consider bad, how can we detect and classify it? We certainly have the ability to traceback and mitigate, once we know what we're looking for. My understanding of the issue is that there are infected PCs on our network, which are being used as part of a distributed attack, but at the application layer, rather than network layer - distributed password brute-force, or similar. Unless we know what to look for, it's hard to detect and stop it. Simon
On 16 Sep 2016, at 20:38, Simon Lockhart wrote:
Unless we know what to look for, it's hard to detect and stop it.
It's not just application-layer stuff - they're subject to all sorts of attacks. Screening out the obvious stuff would certainly help. The main issue is a dearth of engagement of clueful folks in the global operational community. Some gaming-oriented networks are well-represented; others are not, sadly. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
Another aspect, for those users that need to go the PSN network but experience issues via the CGNAT, an opt-out solution (giving them public IPv4) may should mitigate the problem, that PSN network does not support IPv6. After all what percentage of your total subscribers that uses PSN and are gamers 2-3% ? Which might be relatively small amount to give public IPv4. Michalis -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Roland Dobbins Sent: Friday, September 16, 2016 4:32 PM To: nanog@nanog.org Subject: Re: PlayStationNetwork blocking of CGNAT public addresses On 16 Sep 2016, at 20:12, Simon Lockhart wrote:
Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it?
I'm pretty sure that at least part of it has to do with DDoS-related activity. The best bet is to try and identify and engage with the relevant operational personnel with clue. Going the customer-service route isn't fruitful, as you indicate. Another aspect is ensuring that one has the ability to detect, classify, traceback, and mitigate outbound badness southbound of the CGN. This sort of thing has always been a problem with NAT; as CGN becomes more prevalent on wireline broadband networks, it's only going to get worse. AFAIK, PSN doesn't support IPv6. That would be another topic of discussion with the operational folks. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
Hi, as others have said, need to engage with one of their other units to get this sorted out - as a network provider, their customers are relying on YOU to access their service, PSN should care. technically, you could start looking at netflows to the PSN and see if anyone is engaged in DDoS via that route...and , if you offer IPv6 native service to end users, ask PSN when they are going to be offer an IPv6 service to their users - so this CGNAT stuff can go ;-) alan
On Friday, September 16, 2016, Simon Lockhart <simon@slimey.org> wrote:
All,
We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers.
We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address.
Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves.
Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it?
Many thanks in advance,
Simon
Here is a picture of what you are experiencing http://test-ipv6.com/faq_avoids_ipv6.html Sometimes people need pictures to understand why IPv6 is important
So the pain has finally flowed down to other parts of the world. (APNIC ran out of IP's a long time ago, so CGN has been in use here for a lot longer) This issue is one I have been dealing with for the last four years. Only with Sony, no other company has caused such a headache in regard to CGNAT. I will not go into the long and painful saga of dealing with the constant issue of Sony putting blocks on random pool addresses, refusing to supply sufficient information to identify rouge users (timestamp, source IP, destination IP and port) then telling our customers it is a problem at the ISP end, but... Something happened about three months ago that Proves that if the Sony technical people want to get off their asses they are perfectly capable of supplying adequate information to identify a rogue user for the ISP to deal with. One of the local Sony PSN helpline managers actually managed to convince one of their technical people to supply a spreadsheet that magically contained sufficient information to allow us to identify a couple of users that did indeed have multiple infections. Great I thought, now if we can just get them to automate/regularly sent this info we will have a way forward. Alas, it appears it was a one off and we are back to the start. I will quote below what the Sony Network guy said when explaining why they can't send detailed information every time - " From: SNEI-NOC-Abuse [mailto:SNEI-NOC-Abuse@am.sony.com] Sent: Thursday, 11 August 2016 8:38 AM To: ##me## Cc: ##helpful Sony guy## Subject: RE: PSN / Flip Network blocks Hello, There is quite a bit of extra computing power required to produce the CSV file with timestamps and destination IP addresses. We send out over 6000 emails per day which already takes a significant amount of resources and time. We tend to get around 20-30 responses. Instead of wasting the resources on all those emails we generate CSV files for those who respond. We hope you understand. Thank you for taking action on these." So there you go, Sony can indeed solve this issue, but apparently a company that makes computers has insufficient computing power and staff to do so. Oh and after this, despite being asked many times they have never responded to requests for the CSV or similar detailed info. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Simon Lockhart Sent: Saturday, 17 September 2016 1:13 AM To: nanog@nanog.org Subject: PlayStationNetwork blocking of CGNAT public addresses All, We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers. We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address. Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves. Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it? Many thanks in advance, Simon
Simon Lockhart wrote:
Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it?
The best solution is to have a common practice on a set of public port numbers assigned to a host behind NAT. For example, with a practice that, if a port in a range between N*8 and N*8+7 is assigned to a host, other ports in the range is not assigned to other hosts, service providers can block packets based on IP addresses and ranges, especially if correspondence between hosts and ranges are rather stable. But, it may be too late to make such practice common, I'm afraid. Or, wait for a while until service providers receive enough amount of feedback from innocent users. To accelerate it, you can make correspondence between hosts and public addresses not so stable, which makes almost all your IP addresses marked bad quickly, which may make you loss some customer, unless other ISPs also do so. Masataka Ohta
Hi Simon, as other responders have said it is an inherent issue with NAT in general, on workaround is to limit the ratio of actual users to an external IPv4 address, the other thing we have seen from our Abuse contact emails from PSN, is that malicious activity towards the PSN is often accompanied by other malicious activities such as SSH brute force outbound and spaming... I would suggest that 1) limit the ratio of users to an external ipv4 address as much as possible (which would reduce the impact of one compromised customer bringing down play time for other clients behind the same nat 2)do some "canary in the mine" monitoring for obviously malicious traffic (loads of SMTP traffic outbound) and lots of connection requests to SSH servers ... if you see that traffic from behind your CGNAT device .. just temporarily block the internal ip of the user until they clean up their devices. this is the pain with NAT you have to do extra work in order prevent infected users interrupting internet connectivity for other innocent users... I think you can use simple firewall rules on your edge router to identify multiple connections to SMTP and SSH in a short period of time.. If you do the minimum to detect that abuse then you cant be accused of invading peoples privacy... (bear in mind obvious false positives) (Monitoring systems etc) ... Hope this helps, On Fri, Sep 16, 2016 at 2:12 PM, Simon Lockhart <simon@slimey.org> wrote:
All,
We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers.
We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address.
Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves.
Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it?
Many thanks in advance,
Simon
-- Kindest regards, Tom Smyth Mobile: +353 87 6193172 --------------------------------- PLEASE CONSIDER THE ENVIRONMENT BEFORE YOU PRINT THIS E-MAIL This email contains information which may be confidential or privileged. The information is intended solely for the use of the individual or entity named above. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you have received this electronic transmission in error, please notify me by telephone or by electronic mail immediately. Any opinions expressed are those of the author, not the company's .This email does not constitute either offer or acceptance of any contractually binding agreement. Such offer or acceptance must be communicated in writing. You are requested to carry out your own virus check before opening any attachment. Thomas Smyth accepts no liability for any loss or damage which may be caused by malicious software or attachments.
On Sun, Sep 18, 2016 at 01:30:52PM +0100, Tom Smyth wrote:
2)do some "canary in the mine" monitoring for obviously malicious traffic (loads of SMTP traffic outbound) and lots of connection requests to SSH servers ... if you see that traffic from behind your CGNAT device .. just temporarily block the internal ip of the user until they clean up their devices.
Seconded. This is something I've recommended for years (decades, I suppose by now). Simple measurements of what's "normal" for your operation in terms of connection rates, types, etc., are easy to make. That in turn enables measurements of what's abnormal and that in turn enables manual or automatic actions. For example: if the average number of outbound SSH connections established per hour per host across all hosts behind CGNAT is 3.2, and you see a host making 1100/hour: that's a problem. It might be someone who botched a Perl script; or it might be a botted host trying to brute-force its way into something. These kinds of measurements are relatively easy to make and don't require invading user privacy. They won't catch everything, of course, but they're not intended to. They may catch enough to solve the problem in front of you at the moment *and*, if they do that, they may reduce the scope/scale of the rest of the problems to make them more tractable via other techniques. ---rsk
* Rich Kulawiec:
For example: if the average number of outbound SSH connections established per hour per host across all hosts behind CGNAT is 3.2, and you see a host making 1100/hour: that's a problem. It might be someone who botched a Perl script; or it might be a botted host trying to brute-force its way into something.
If you do this, you break Github. (If I guess Simon's network correctly, then I've seen reports which suggest that they might already be doing this.)
On Sun, Sep 18, 2016 at 03:56:30PM +0200, Florian Weimer wrote:
* Rich Kulawiec:
For example: if the average number of outbound SSH connections established per hour per host across all hosts behind CGNAT is 3.2, and you see a host making 1100/hour: that's a problem. It might be someone who botched a Perl script; or it might be a botted host trying to brute-force its way into something.
If you do this, you break Github.
1. I didn't know that: *how* does this break Github? 2. This is just an *example* of how to use the technique. It's not meant to be literal. The general approach of determining the statistical characteristics of "normal" and then flagging things that are "way outside normal" works -- but of course it requires sufficient knowledge to account for things like Github usage and/or infrequent events and/or usage spikes triggered by real-world events, etc. The more you do it, and the longer you do it, the better you'll get at it. (But of course the false positive rate will never be zero. That's why the question of what to do when anomalies happen isn't easy: poke a human? throttle? block? further analysis?) ---rsk
* Rich Kulawiec:
On Sun, Sep 18, 2016 at 03:56:30PM +0200, Florian Weimer wrote:
* Rich Kulawiec:
For example: if the average number of outbound SSH connections established per hour per host across all hosts behind CGNAT is 3.2, and you see a host making 1100/hour: that's a problem. It might be someone who botched a Perl script; or it might be a botted host trying to brute-force its way into something.
If you do this, you break Github.
1. I didn't know that: *how* does this break Github?
Github users create several orders of magnitude more SSH connections than average users because the most convenient way to set up read/write access is to use SSH. Depending on how you use Github, you might update lots and lots of local repositories from Github at certain times of the day.
2. This is just an *example* of how to use the technique. It's not meant to be literal. The general approach of determining the statistical characteristics of "normal" and then flagging things that are "way outside normal" works -- but of course it requires sufficient knowledge to account for things like Github usage and/or infrequent events and/or usage spikes triggered by real-world events, etc.
Sure, and people already do this, and are not very flexible about it. Support staff isn't briefed, and claim they do such stochastic behavior adjustment across all (server) products, which I find difficult to believe. I'm worried that this leads to a future where tunnelling everything over HTTP(S) is no longer sufficient. You have to make it look like a web server or browser, too. Everything else risks triggering automated countermeasures. That's the anti-thesis of good protocol design.
On Mon, Sep 19, 2016 at 09:55:56PM +0200, Florian Weimer wrote:
Github users create several orders of magnitude more SSH connections [snip]
Ah. I didn't know that. Thanks!
Sure, and people already do this, and are not very flexible about it. Support staff isn't briefed, and claim they do such stochastic behavior adjustment across all (server) products, which I find difficult to believe.
You're right: those are serious drawbacks. If folks are going to do this, then they need to do it right, which means making sure everyone is in the loop and making sure that support staff are clueful/diligent enough to investigate -- or at least hand off to someone who'll investigate. This stuff works but only if you're adaptive/flexible and willing to learn and adjust on an ongoing basis.
I'm worried that this leads to a future where tunnelling everything over HTTP(S) is no longer sufficient. You have to make it look like a web server or browser, too. Everything else risks triggering automated countermeasures.
And as someone who constantly beats the "Internet != web" drum, I second this. Marginalizing other protocols doesn't serve us well in short term (it breaks things) or the long term (it stifles innovation). ---rsk
This is, as many things are, a huge problem in communication. Sony tells ISP 'Hey, you have customers abusing us. Fix it!'. ISP says 'Oh crap, sorry, what's going on? We'll run it down.' Sony says nothing. Let's just stop here for a second. This is fundamentally no different then the 'I have a problem, it's the network! complaints we've all dealt with forever. You spend days/weeks/months working on it. Maybe you ultimately find a goofy switchport, or maybe you discover that the server HDDs were crapping the bed and the problem server was chugging because of that. But you had to spend tons of time working on it because you couldn't get the info you need because the reporter was CONVINCED they KNEW what it was. Why should Simon have to spend hours of engineering time fishing through traffic captures and logs when he doesn't even know what he's LOOKING for? What does PSN consider 'abuse' here? Does Simon have customers infected with botnets that are targeting PSN at times? Or does PSN assume nobody will ever have more than a couple Playstations in a house, so if they see more than N connections to PSN from the same IP, it's malicious, since CGN is likely not something they considered? ( If anyone wants to place beer wagers, I'm picking the later. ) I spend about 8 weeks this year going back and forth with a Very Large Website Network who had blocked a /17 of IP space from accessing ANY of their sites because of 'malicious traffic' from a specific /23. 5 of those weeks, their responses consisted of 'it's malicious, you go find it, should be obvious', 'you clearly don't know what you're doing, we're wasting our time', etc. Week 5, I was able to extract that it was a specific web crawler that they said was knocking their databases over. After a conversation with their CIO the following week, they came back and admitted that a junior system admin made some PHP changes on a bunch of servers that he didn't think was in production,and when we crawled THOSE servers, Bad Things Happened for them. We were doing nothing wrong ; they just refused to look, and found it easier to blame us. Simon's getting screwed because he's not being given any information to try and solve the problem, and because his customers are likely blaming him because he's their ISP. Sony needs to stand up and work with him here. On Sun, Sep 18, 2016 at 8:30 AM, Tom Smyth <tom.smyth@wirelessconnect.eu> wrote:
Hi Simon,
as other responders have said it is an inherent issue with NAT in general, on workaround is to limit the ratio of actual users to an external IPv4 address, the other thing we have seen from our Abuse contact emails from PSN, is that malicious activity towards the PSN is often accompanied by other malicious activities such as SSH brute force outbound and spaming...
I would suggest that
1) limit the ratio of users to an external ipv4 address as much as possible (which would reduce the impact of one compromised customer bringing down play time for other clients behind the same nat
2)do some "canary in the mine" monitoring for obviously malicious traffic (loads of SMTP traffic outbound) and lots of connection requests to SSH servers ... if you see that traffic from behind your CGNAT device .. just temporarily block the internal ip of the user until they clean up their devices.
this is the pain with NAT you have to do extra work in order prevent infected users interrupting internet connectivity for other innocent users... I think you can use simple firewall rules on your edge router to identify multiple connections to SMTP and SSH in a short period of time..
If you do the minimum to detect that abuse then you cant be accused of invading peoples privacy... (bear in mind obvious false positives) (Monitoring systems etc) ...
Hope this helps,
On Fri, Sep 16, 2016 at 2:12 PM, Simon Lockhart <simon@slimey.org> wrote:
All,
We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers.
We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address.
Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves.
Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it?
Many thanks in advance,
Simon
-- Kindest regards, Tom Smyth
Mobile: +353 87 6193172 --------------------------------- PLEASE CONSIDER THE ENVIRONMENT BEFORE YOU PRINT THIS E-MAIL This email contains information which may be confidential or privileged. The information is intended solely for the use of the individual or entity named above. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you have received this electronic transmission in error, please notify me by telephone or by electronic mail immediately. Any opinions expressed are those of the author, not the company's .This email does not constitute either offer or acceptance of any contractually binding agreement. Such offer or acceptance must be communicated in writing. You are requested to carry out your own virus check before opening any attachment. Thomas Smyth accepts no liability for any loss or damage which may be caused by malicious software or attachments.
People love to hate incumbent telcos because of their arrogance (and frankly it's deserved), but people forget that big content can be just as arrogant and just as deserving of hatred. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Tom Beecher" <beecher@beecher.cc> To: "Tom Smyth" <tom.smyth@wirelessconnect.eu> Cc: "NANOG" <nanog@nanog.org> Sent: Sunday, September 18, 2016 8:15:08 AM Subject: Re: PlayStationNetwork blocking of CGNAT public addresses This is, as many things are, a huge problem in communication. Sony tells ISP 'Hey, you have customers abusing us. Fix it!'. ISP says 'Oh crap, sorry, what's going on? We'll run it down.' Sony says nothing. Let's just stop here for a second. This is fundamentally no different then the 'I have a problem, it's the network! complaints we've all dealt with forever. You spend days/weeks/months working on it. Maybe you ultimately find a goofy switchport, or maybe you discover that the server HDDs were crapping the bed and the problem server was chugging because of that. But you had to spend tons of time working on it because you couldn't get the info you need because the reporter was CONVINCED they KNEW what it was. Why should Simon have to spend hours of engineering time fishing through traffic captures and logs when he doesn't even know what he's LOOKING for? What does PSN consider 'abuse' here? Does Simon have customers infected with botnets that are targeting PSN at times? Or does PSN assume nobody will ever have more than a couple Playstations in a house, so if they see more than N connections to PSN from the same IP, it's malicious, since CGN is likely not something they considered? ( If anyone wants to place beer wagers, I'm picking the later. ) I spend about 8 weeks this year going back and forth with a Very Large Website Network who had blocked a /17 of IP space from accessing ANY of their sites because of 'malicious traffic' from a specific /23. 5 of those weeks, their responses consisted of 'it's malicious, you go find it, should be obvious', 'you clearly don't know what you're doing, we're wasting our time', etc. Week 5, I was able to extract that it was a specific web crawler that they said was knocking their databases over. After a conversation with their CIO the following week, they came back and admitted that a junior system admin made some PHP changes on a bunch of servers that he didn't think was in production,and when we crawled THOSE servers, Bad Things Happened for them. We were doing nothing wrong ; they just refused to look, and found it easier to blame us. Simon's getting screwed because he's not being given any information to try and solve the problem, and because his customers are likely blaming him because he's their ISP. Sony needs to stand up and work with him here. On Sun, Sep 18, 2016 at 8:30 AM, Tom Smyth <tom.smyth@wirelessconnect.eu> wrote:
Hi Simon,
as other responders have said it is an inherent issue with NAT in general, on workaround is to limit the ratio of actual users to an external IPv4 address, the other thing we have seen from our Abuse contact emails from PSN, is that malicious activity towards the PSN is often accompanied by other malicious activities such as SSH brute force outbound and spaming...
I would suggest that
1) limit the ratio of users to an external ipv4 address as much as possible (which would reduce the impact of one compromised customer bringing down play time for other clients behind the same nat
2)do some "canary in the mine" monitoring for obviously malicious traffic (loads of SMTP traffic outbound) and lots of connection requests to SSH servers ... if you see that traffic from behind your CGNAT device .. just temporarily block the internal ip of the user until they clean up their devices.
this is the pain with NAT you have to do extra work in order prevent infected users interrupting internet connectivity for other innocent users... I think you can use simple firewall rules on your edge router to identify multiple connections to SMTP and SSH in a short period of time..
If you do the minimum to detect that abuse then you cant be accused of invading peoples privacy... (bear in mind obvious false positives) (Monitoring systems etc) ...
Hope this helps,
On Fri, Sep 16, 2016 at 2:12 PM, Simon Lockhart <simon@slimey.org> wrote:
All,
We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers.
We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address.
Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves.
Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it?
Many thanks in advance,
Simon
-- Kindest regards, Tom Smyth
Mobile: +353 87 6193172 --------------------------------- PLEASE CONSIDER THE ENVIRONMENT BEFORE YOU PRINT THIS E-MAIL This email contains information which may be confidential or privileged. The information is intended solely for the use of the individual or entity named above. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you have received this electronic transmission in error, please notify me by telephone or by electronic mail immediately. Any opinions expressed are those of the author, not the company's .This email does not constitute either offer or acceptance of any contractually binding agreement. Such offer or acceptance must be communicated in writing. You are requested to carry out your own virus check before opening any attachment. Thomas Smyth accepts no liability for any loss or damage which may be caused by malicious software or attachments.
* Tom Beecher:
Simon's getting screwed because he's not being given any information to try and solve the problem, and because his customers are likely blaming him because he's their ISP.
We don't know that for sure. Another potential issue is that the ISP just cannot afford to notify its compromised customers, even if they were able to detect them.
On Sun Sep 18, 2016 at 03:58:57PM +0200, Florian Weimer wrote:
* Tom Beecher:
Simon's getting screwed because he's not being given any information to try and solve the problem, and because his customers are likely blaming him because he's their ISP.
We don't know that for sure. Another potential issue is that the ISP just cannot afford to notify its compromised customers, even if they were able to detect them.
I'd like to think that we're pretty responsive to taking our users offline when they're compromised and we're made aware of it - either through our own tools, or through 3rd party notifications. The process with Sony goes something like: - User reports they can't reach PSN - We report the Sony/PSN, they say "Yes, it's blocked because that IP attacked us" - We say "Okay, that's a CGNAT public IP, can you help us identify the which inside user that is - (timestamp,ip,port) logs, or some way to identify the bad traffic so we can look for it ourselves" - Sony say no, either through silence, or explicitly. - We have unhappy user(s), who blame us. Simon
* Simon Lockhart:
On Sun Sep 18, 2016 at 03:58:57PM +0200, Florian Weimer wrote:
* Tom Beecher:
Simon's getting screwed because he's not being given any information to try and solve the problem, and because his customers are likely blaming him because he's their ISP.
We don't know that for sure. Another potential issue is that the ISP just cannot afford to notify its compromised customers, even if they were able to detect them.
I'd like to think that we're pretty responsive to taking our users offline when they're compromised and we're made aware of it - either through our own tools, or through 3rd party notifications.
Okay, then perhaps my guess of the ISP involved is wrong.
The process with Sony goes something like:
- User reports they can't reach PSN - We report the Sony/PSN, they say "Yes, it's blocked because that IP attacked us" - We say "Okay, that's a CGNAT public IP, can you help us identify the which inside user that is - (timestamp,ip,port) logs, or some way to identify the bad traffic so we can look for it ourselves" - Sony say no, either through silence, or explicitly. - We have unhappy user(s), who blame us.
Yes, that's not very constructive. Out of curiosity, how common is end-to-end reporting of source/destination port information (in addition to source IP addresses and destination IP addresses)? Have the anti-abuse mechanisms finalyl caught on with CGNAT, or is it possible that the PSN operator themselves do not have such detailed data?
On Sun Sep 18, 2016 at 05:17:33PM +0200, Florian Weimer wrote:
Okay, then perhaps my guess of the ISP involved is wrong.
It's not hard to find out who I work for :)
Out of curiosity, how common is end-to-end reporting of source/destination port information (in addition to source IP addresses and destination IP addresses)? Have the anti-abuse mechanisms finalyl caught on with CGNAT, or is it possible that the PSN operator themselves do not have such detailed data?
99.99% of abuse reports we receive contain the information, but that's because 99.99% of abuse reports we receive are from the 'copyright police', and their tools capture and include it in the reports. Once you discard that 99.99%, and are left with the stuff that is worthy of manual investigation, I'd say that almost all of it only contains timestamp and source IP. Sometimes it'll also contain destination IP (so we can take a best guess based on netflow data), and very occasionally it'll also contain source port information. I'd say the same also applies to requests for information that we receive from law enforcement agencies. In most cases, they're working from weblogs, and I'd be tempted to say that most webservers' 'out of the box' configuration does not log source port, only source IP in the web access logs. Simon
On Mon, 19 Sep 2016 10:41:59 +1200, "Tony Wicks" said:
Interestingly, Sony (SNEI-NOC-Abuse <SNEI-NOC-Abuse@am. sony dot com) jut replied to being forwarded back one of their notification blocks requesting more detailed information with a csv file in under an hour!
So I guess name-and-shame *does* work? :)
So I should try again to get them to tell me what an "Account Takeover Attempt" is? They ignored my last request. It's easy to explain DMCA or spam to an end-user, but it's difficult to explain to some soccer mom that her kids are doing something to make Sony mad, when I can't explain to them what Sony is mad about. On Sun, Sep 18, 2016 at 5:58 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Mon, 19 Sep 2016 10:41:59 +1200, "Tony Wicks" said:
Interestingly, Sony (SNEI-NOC-Abuse <SNEI-NOC-Abuse@am. sony dot com) jut replied to being forwarded back one of their notification blocks requesting more detailed information with a csv file in under an hour!
So I guess name-and-shame *does* work? :)
So the last one we successfully managed to isolate, our customer they had more than one PC with multiple infections. It’s not Playstation’s, but Windows machines that are infected with I assume some malware that is trying to log into PSN. cheers From: Jason Baugher [mailto:jason@thebaughers.com] Sent: Monday, 19 September 2016 12:09 PM To: Valdis.Kletnieks@vt.edu Cc: Tony Wicks <tony@wicks.co.nz>; NANOG <nanog@nanog.org> Subject: Re: PlayStationNetwork blocking of CGNAT public addresses So I should try again to get them to tell me what an "Account Takeover Attempt" is? They ignored my last request. It's easy to explain DMCA or spam to an end-user, but it's difficult to explain to some soccer mom that her kids are doing something to make Sony mad, when I can't explain to them what Sony is mad about. On Sun, Sep 18, 2016 at 5:58 PM, <Valdis.Kletnieks@vt.edu <mailto:Valdis.Kletnieks@vt.edu> > wrote: On Mon, 19 Sep 2016 10:41:59 +1200, "Tony Wicks" said:
Interestingly, Sony (SNEI-NOC-Abuse <SNEI-NOC-Abuse@am. <mailto:SNEI-NOC-Abuse@am. %20 sony%20dot%20com)%20jut%0b> sony dot com) jut replied to being forwarded back one of their notification blocks requesting more detailed information with a csv file in under an hour!
So I guess name-and-shame *does* work? :)
An email to a user notifying them they're likely compromised costs basically nothing. An email to their entire subscriber base also costs nothing. If you find me an ISP that can't afford to notify users, I'll show you one that shouldn't be in business anyways. There's this presumption of guilt here, that Sony is right, and Simon's subscribers are doing something malicious, yet they won't provide any evidence of that. Even if they didn't know what it was, come back with 'We're seeing weird bursts of [traffic characteristics] aimed at PSN during these times. We're not quite sure what it is, but it's causing [problem X].' It would still be a question of maliciousness or not, but it would be something to work with. Providing nothing just perpetuates this finger pointing game, and nothing gets solved. On Sun, Sep 18, 2016 at 9:58 AM, Florian Weimer <fw@deneb.enyo.de> wrote:
* Tom Beecher:
Simon's getting screwed because he's not being given any information to try and solve the problem, and because his customers are likely blaming him because he's their ISP.
We don't know that for sure. Another potential issue is that the ISP just cannot afford to notify its compromised customers, even if they were able to detect them.
* Tom Beecher:
An email to a user notifying them they're likely compromised costs basically nothing.
If this increases the probability that the customer contacts customer support, in some markets, there is a risk that the account will never turn profitable during the current contract period. (Granted, my information may be woefully out of date, but my impression is that price-based competition is still pretty much cut-throat over here.)
If you find me an ISP that can't afford to notify users, I'll show you one that shouldn't be in business anyways.
I'm not blaming the ISP. (I may have done so in the past.) If we end up in such a situation, it's hardly the fault of one single ISP.
There's this presumption of guilt here, that Sony is right, and Simon's subscribers are doing something malicious, yet they won't provide any evidence of that. Even if they didn't know what it was, come back with 'We're seeing weird bursts of [traffic characteristics] aimed at PSN during these times. We're not quite sure what it is, but it's causing [problem X].' It would still be a question of maliciousness or not, but it would be something to work with. Providing nothing just perpetuates this finger pointing game, and nothing gets solved.
Yes, indeed. Resolving most networking problems needs cooperation, because at the most basic level, the Internet is still about connecting otherwise unrelated networks.
Something similar happened to a local FantasyConon I was helping set up, we had only two PS4 machines there and accounts provided by Blizzard for Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in about 8h. There was no other traffic other then those two accounts playing Overwatch so my guess is that they have some too aggressive checks. I've managed to convince our ISP there to change the outside IP of the link so we got them working the next day but it happened again in 8h. -- *blap* On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhart <simon@slimey.org> wrote:
All,
We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers.
We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address.
Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves.
Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it?
Many thanks in advance,
Simon
PSN is one reason I am not a fan of CGNAT. All they see are tons of connections from the same IP. This results in them banning folks. Due to them being hacked so many times getting them to actually communicate is almost impossible. My .02 is just get the gamers a true public if at all possible. Justin Wilson j2sw@mtin.net --- http://www.mtin.net Owner/CEO xISP Solutions- Consulting – Data Centers - Bandwidth http://www.midwest-ix.com COO/Chairman Internet Exchange - Peering - Distributed Fabric
On Sep 20, 2016, at 8:24 AM, Danijel Starman <theghost101@gmail.com> wrote:
Something similar happened to a local FantasyConon I was helping set up, we had only two PS4 machines there and accounts provided by Blizzard for Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in about 8h. There was no other traffic other then those two accounts playing Overwatch so my guess is that they have some too aggressive checks. I've managed to convince our ISP there to change the outside IP of the link so we got them working the next day but it happened again in 8h.
-- *blap*
On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhart <simon@slimey.org> wrote:
All,
We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers.
We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address.
Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves.
Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it?
Many thanks in advance,
Simon
In message <09342130-874F-4FA4-B410-B7B66A75FA4D@mtin.net>, Justin Wilson write s:
PSN is one reason I am not a fan of CGNAT. All they see are tons of connections from the same IP. This results in them banning folks. Due to them being hacked so many times getting them to actually communicate is almost impossible. My .02 is just get the gamers a true public if at all possible.
Justin Wilson j2sw@mtin.net
What we need is business tech reporters to continually report on these failures of content providers to deliver their services over IPv6. 20 years lead time should be enough for any service. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Wed, 21 Sep 2016 11:29:49 +1000, Mark Andrews said:
What we need is business tech reporters to continually report on these failures of content providers to deliver their services over IPv6. 20 years lead time should be enough for any service.
Interestingly enough, the Playstation 4 has at least rudimentary IPv6 support - it will DHCPv6 and answer pings. Threw me for a loop first time I saw it, I couldn't figure out what unaccounted-for gear I had that was grabbing an IPv6 address... :)
I have a hard time accepting that service providers should re-engineer their networks because other companies cannot properly engineer their abuse tooling. On Tue, Sep 20, 2016 at 11:33 AM, Justin Wilson <lists@mtin.net> wrote:
PSN is one reason I am not a fan of CGNAT. All they see are tons of connections from the same IP. This results in them banning folks. Due to them being hacked so many times getting them to actually communicate is almost impossible. My .02 is just get the gamers a true public if at all possible.
Justin Wilson j2sw@mtin.net
--- http://www.mtin.net Owner/CEO xISP Solutions- Consulting – Data Centers - Bandwidth
http://www.midwest-ix.com COO/Chairman Internet Exchange - Peering - Distributed Fabric
On Sep 20, 2016, at 8:24 AM, Danijel Starman <theghost101@gmail.com> wrote:
Something similar happened to a local FantasyConon I was helping set up, we had only two PS4 machines there and accounts provided by Blizzard for Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in about 8h. There was no other traffic other then those two accounts playing Overwatch so my guess is that they have some too aggressive checks. I've managed to convince our ISP there to change the outside IP of the link so we got them working the next day but it happened again in 8h.
-- *blap*
On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhart <simon@slimey.org> wrote:
All,
We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers.
We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address.
Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves.
Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it?
Many thanks in advance,
Simon
Hi We have the opposite problem with PSN: Sometimes they will send abuse reports with several of our IP addresses listed. The problem with that is that we can not give data about one customer to another customer. By listing multiple IP addresses we are prevented from forwarding the email to the customer. Which means we may ignore it instead. Regards, Baldur
On 21 Sep 2016, at 15:37, Baldur Norddahl wrote:
Which means we may ignore it instead.
. . . copy/paste or awk/sed or whatever isn't an option? If not, have you requested a) separate notifications per source and/or b) a more textual-manipulation-friendly format? Unless they're sending .gifs or something, surely this might be possible, yes? It seems within the realm of possibility this sort of response - or lack thereof - could result in some gaming network operators becoming a bit jaded. And perhaps some customers, too. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
participants (18)
-
A.L.M.Buxey＠lboro.ac.uk -
Baldur Norddahl -
Ca By -
Danijel Starman -
Florian Weimer -
Jason Baugher -
Justin Wilson -
Mark Andrews -
Masataka Ohta -
michalis.bersimis＠hq.cyta.gr -
Mike Hammett -
Rich Kulawiec -
Roland Dobbins -
Simon Lockhart -
Tom Beecher -
Tom Smyth -
Tony Wicks -
Valdis.Kletnieks＠vt.edu