Reaching out to Sony NOC, resolving DDoS Issues - Need POC
Hi all, We've been trying to get in contact with Sony and/or Akamai to resolve an IP blacklisting issue. Support is not useful, and our customers are complaining. If anyone has a POC for somebody over at Sony or PSN who can help us resolve these issues, it would be much appreciated! We're facing some reflected DDoS attacks, where the source address is spoofed to appear to be our IPs, and as a result getting blacklisted. Sony's support has told us to "change IPs", which just isn't realistic, so we're looking for someone actually at the NOC. Already been in touch with SNEI-NOC, which has completely ignored our emails. They only seem to respond to the normal "Account Takeover" requests, or whitelist requests. All of my IP's are already whitelisted, but there seems to be a DDoS Attack that will ban anyone's IP from Sony/PlayStation Network. Akamai's also been giving us the runaround, and told us it's purely a Sony problem, so we're trying to get to the right POC to get this issue resolved Thanks in advance!
Went through this last year. They simply didn't do anything productive. You have to change IPs if you want a quick resolution. They should email the POC for the IP (I think towards the end of the day) as to what happened and I believe a time frame when it will get resolved. Hopefully someone with more than basic (or no) technical capacity can contact you and fix this stupid problem, but I would not count on it =( It is a Sony problem, not Akamai. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Mon, Jan 6, 2020 at 1:27 PM Octolus Development <admin@octolus.net> wrote:
Hi all,
We've been trying to get in contact with Sony and/or Akamai to resolve an IP blacklisting issue.
Support is not useful, and our customers are complaining.
If anyone has a POC for somebody over at Sony or PSN who can help us resolve these issues, it would be much appreciated!
We're facing some reflected DDoS attacks, where the source address is spoofed to appear to be our IPs, and as a result getting blacklisted. Sony's support has told us to "change IPs", which just isn't realistic, so we're looking for someone actually at the NOC.
Already been in touch with SNEI-NOC, which has completely ignored our emails. They only seem to respond to the normal "Account Takeover" requests, or whitelist requests. All of my IP's are already whitelisted, but there seems to be a DDoS Attack that will ban anyone's IP from Sony/PlayStation Network.
Akamai's also been giving us the runaround, and told us it's purely a Sony problem, so we're trying to get to the right POC to get this issue resolved
Thanks in advance!
Good luck! I’ve dealt with such PSN IP blocking issues for several years and have found that Sony is the absolute worst possible gaming/content provider I’ve ever dealt with. One company I worked at had to threaten legal action as PSN would block CGN IPv4 addresses on their network and then tell customers to contact the ISP as it’s the ISP’s fault. At the same time PSN would provide no useful or actionable information to the ISP as to why they has implemented almost random IP blocking on their network. Not once was any issue raised in regard to their biggest competitor. The sheer arrogance shown by their management at the time was infuriating, it was a you are not big enough to waste our precious time on attitude. On Mon, Jan 6, 2020 at 1:27 PM Octolus Development <admin@octolus.net <mailto:admin@octolus.net> > wrote: Hi all, We've been trying to get in contact with Sony and/or Akamai to resolve an IP blacklisting issue. Support is not useful, and our customers are complaining. If anyone has a POC for somebody over at Sony or PSN who can help us resolve these issues, it would be much appreciated!
To be fair they do contact you. It's an automated process that's done daily and it has a light amount of information. The rest is totally accurate - the Playstation network stuff is an absolute joke (think back to how they were down for MONTHS). Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Jan 7, 2020 at 1:14 PM Tony Wicks <tony@wicks.co.nz> wrote:
Good luck! I’ve dealt with such PSN IP blocking issues for several years and have found that Sony is the absolute worst possible gaming/content provider I’ve ever dealt with. One company I worked at had to threaten legal action as PSN would block CGN IPv4 addresses on their network and then tell customers to contact the ISP as it’s the ISP’s fault. At the same time PSN would provide no useful or actionable information to the ISP as to why they has implemented almost random IP blocking on their network. Not once was any issue raised in regard to their biggest competitor. The sheer arrogance shown by their management at the time was infuriating, it was a you are not big enough to waste our precious time on attitude.
On Mon, Jan 6, 2020 at 1:27 PM Octolus Development <admin@octolus.net> wrote:
Hi all,
We've been trying to get in contact with Sony and/or Akamai to resolve an IP blacklisting issue.
Support is not useful, and our customers are complaining.
If anyone has a POC for somebody over at Sony or PSN who can help us resolve these issues, it would be much appreciated!
No, that's only for "Account Takeover".. And those problems we've solved. That was false reports, and we got whitelisted. However with this issue? They decide to completely ignore the emails, it seems like we're being either spoofed or people are attacking us with Sony's IP space. What happens, is really difficult to determine without some contact with Sony to confirm what's the real problem. But that seems impossible now. On 07.01.2020 19:16:10, Josh Luthman <josh@imaginenetworksllc.com> wrote: To be fair they do contact you. It's an automated process that's done daily and it has a light amount of information. The rest is totally accurate - the Playstation network stuff is an absolute joke (think back to how they were down for MONTHS). Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Jan 7, 2020 at 1:14 PM Tony Wicks <tony@wicks.co.nz [mailto:tony@wicks.co.nz]> wrote: Good luck! I’ve dealt with such PSN IP blocking issues for several years and have found that Sony is the absolute worst possible gaming/content provider I’ve ever dealt with. One company I worked at had to threaten legal action as PSN would block CGN IPv4 addresses on their network and then tell customers to contact the ISP as it’s the ISP’s fault. At the same time PSN would provide no useful or actionable information to the ISP as to why they has implemented almost random IP blocking on their network. Not once was any issue raised in regard to their biggest competitor. The sheer arrogance shown by their management at the time was infuriating, it was a you are not big enough to waste our precious time on attitude. On Mon, Jan 6, 2020 at 1:27 PM Octolus Development <admin@octolus.net [mailto:admin@octolus.net]> wrote: Hi all, We've been trying to get in contact with Sony and/or Akamai to resolve an IP blacklisting issue. Support is not useful, and our customers are complaining. If anyone has a POC for somebody over at Sony or PSN who can help us resolve these issues, it would be much appreciated!
Peace, On Mon, Jan 6, 2020, 9:27 PM Octolus Development <admin@octolus.net> wrote:
We're facing some reflected DDoS attacks, where the source address is spoofed to appear to be our IPs, and as a result getting blacklisted. Sony's support has told us to "change IPs"
Wait, are they blacklisting spoofed IP(v4?) addresses? If so, this is hilarious. When at some point they will finally blacklist the whole 0/0, the problem will be solved by itself. Still, are you completely sure this is the accurate description of what they are doing? -- Töma
And you're sure that you are the reflection target not the reflection vector? As in it's definitely the case that you are the *target* here (your IP addresses are being spoofed, and the reflection attack is hitting you) rather than that someone is abusing endpoints in your network, i.e. reflecting off of your endpoints with Sony's addresses as the spoofed source such that Sony is getting targeted? If the former: How is Sony involved there? Are people spoofing your source addresses and trying to reflect off of Sony? Or how else did Sony catch wind of it? -- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal On Tue, Jan 7, 2020 at 9:58 AM Töma Gavrichenkov <ximaera@gmail.com> wrote:
Peace,
On Mon, Jan 6, 2020, 9:27 PM Octolus Development <admin@octolus.net> wrote:
We're facing some reflected DDoS attacks, where the source address is spoofed to appear to be our IPs, and as a result getting blacklisted. Sony's support has told us to "change IPs"
Wait, are they blacklisting spoofed IP(v4?) addresses? If so, this is hilarious. When at some point they will finally blacklist the whole 0/0, the problem will be solved by itself.
Still, are you completely sure this is the accurate description of what they are doing?
-- Töma
Peace, On Tue, Jan 7, 2020, 9:10 PM Hugo Slabbert <hugo@slabnet.com> wrote:
And you're sure that you are the reflection target not the reflection vector?
Well, in almost any* case blacklisting reflection vectors by IP is an insanely bad practice. * — I can *think* of a use case when this could be an appropriate solution (I recall Netscout/Arbor once had such a use case), but in the overwhelming majority of incidents it is absolutely not, and you need to be one hundred percent sure you know what you're doing. -- Töma
Peace, On Tue, Jan 7, 2020 at 9:10 PM Hugo Slabbert <hugo@slabnet.com> wrote:
And you're sure that you are the reflection target not the reflection vector?
NB: I have just checked the IP addresses the OP has provided me with (offlist) against our database of known reflection sources, and I confirm that none of those seem to ever host UDP software vulnerable to amplification. -- Töma
Well, in almost any* case blacklisting reflection vectors by IP is an insanely bad practice. * — I can *think* of a use case when this could be an appropriate solution (I recall Netscout/Arbor once had such a use case), but in the overwhelming majority of incidents it is absolutely not, and you need to be one hundred percent sure you know what you're doing.
Agreed; drop the vector not the address, but was looking to just clarify the direction of things a bit. NB: I have just checked the IP addresses the OP has provided me with
(offlist) against our database of known reflection sources, and I confirm that none of those seem to ever host UDP software vulnerable to amplification
ty; good to know. They decide to completely ignore the emails, it seems like we're being
either spoofed *or people are attacking us with Sony's IP space*.
So you're getting inbound traffic that has Sony IP space source addresses in it? That does start to sound more like people trying to reflect off of you to Sony. What's the protocol and destination ports on the traffic you're receiving with Sony source addresses (and the source ports for good measure, if they're fairly consistent)? -- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal On Tue, Jan 7, 2020 at 10:54 AM Töma Gavrichenkov <ximaera@gmail.com> wrote:
Peace,
On Tue, Jan 7, 2020 at 9:10 PM Hugo Slabbert <hugo@slabnet.com> wrote:
And you're sure that you are the reflection target not the reflection vector?
NB: I have just checked the IP addresses the OP has provided me with (offlist) against our database of known reflection sources, and I confirm that none of those seem to ever host UDP software vulnerable to amplification.
-- Töma
Tracked it down. Sony are using "Imperva" which is former Incapsula. The IP's that was attacked by this DDoS Attack, have been added to their threatradar, their phone support (Imperva) literally hangs up the call when you try to question if they can provide more information about why the IP's are blocked. They said since I am not Sony, I can not request information. But here's the funny part, when connecting to their own website imperva.com from those IP's -- we are getting the exactly same error code that Sony are returning. Indicating that Imperva is the main problem here, they seem to block spoofed IP's. On 07.01.2020 21:20:08, Hugo Slabbert <hugo@slabnet.com> wrote: Well, in almost any* case blacklisting reflection vectors by IP is an insanely bad practice. * — I can *think* of a use case when this could be an appropriate solution (I recall Netscout/Arbor once had such a use case), but in the overwhelming majority of incidents it is absolutely not, and you need to be one hundred percent sure you know what you're doing. Agreed; drop the vector not the address, but was looking to just clarify the direction of things a bit. NB: I have just checked the IP addresses the OP has provided me with (offlist) against our database of known reflection sources, and I confirm that none of those seem to ever host UDP software vulnerable to amplification ty; good to know. They decide to completely ignore the emails, it seems like we're being either spoofed or people are attacking us with Sony's IP space. So you're getting inbound traffic that has Sony IP space source addresses in it? That does start to sound more like people trying to reflect off of you to Sony. What's the protocol and destination ports on the traffic you're receiving with Sony source addresses (and the source ports for good measure, if they're fairly consistent)? -- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com [mailto:hugo@slabnet.com] pgp key: B178313E | also on Signal On Tue, Jan 7, 2020 at 10:54 AM Töma Gavrichenkov <ximaera@gmail.com [mailto:ximaera@gmail.com]> wrote: Peace, On Tue, Jan 7, 2020 at 9:10 PM Hugo Slabbert <hugo@slabnet.com [mailto:hugo@slabnet.com]> wrote:
And you're sure that you are the reflection target not the reflection vector?
NB: I have just checked the IP addresses the OP has provided me with (offlist) against our database of known reflection sources, and I confirm that none of those seem to ever host UDP software vulnerable to amplification. -- Töma
Hello, On Wed, 8 Jan 2020 at 16:53, Octolus Development <admin@octolus.net> wrote:
But here's the funny part, when connecting to their own website imperva.com from those IP's -- we are getting the exactly same error code that Sony are returning.
And what error code / full error is that *exactly*? I assumed we where talking about a complete IP block here (no response at all), apparently that is not the case. Lukas
The error it displays on both Sony, and Imperva (and whatever websites who uses their protection). So this problem is not with Sony, but rather Imperva blocking IP's wildly. The IP's are not blocks, it's a single IP and the block/blacklist lifts after 7 days. Error that appears on those websites, including imperva themself: This page can't be displayed. Contact support for additional information. The incident ID is: N/A. On 08.01.2020 18:22:09, Lukas Tribus <lists@ltri.eu> wrote: Hello, On Wed, 8 Jan 2020 at 16:53, Octolus Development wrote:
But here's the funny part, when connecting to their own website imperva.com from those IP's -- we are getting the exactly same error code that Sony are returning.
And what error code / full error is that *exactly*? I assumed we where talking about a complete IP block here (no response at all), apparently that is not the case. Lukas
Hello, On Wed, 8 Jan 2020 at 18:26, Octolus Development <admin@octolus.net> wrote:
The error it displays on both Sony, and Imperva (and whatever websites who uses their protection). So this problem is not with Sony, but rather Imperva blocking IP's wildly.
The IP's are not blocks, it's a single IP and the block/blacklist lifts after 7 days.
Error that appears on those websites, including imperva themself: This page can't be displayed. Contact support for additional information. The incident ID is: N/A.
That looks like a WAF, so reflection/spoofing is probably *not* the reason your IPs ended up on those lists. I assume what you see looks similar to what this returns (a request that looks like a sql injection): https://www.imperva.com/bla%20OR%201=1 A few of those hits, or crossing a certain threshold per IP (very easy for CGN IPs), and your IP probably ends up on those lists I guess. And of course those endpoints are not IPv6 enabled, so behind CGN the end customers shares his luck with it's neighbors even if everything is IPv6 enabled. Imperva, is that the "cybersecurity firm" that was breached 6 months ago? https://krebsonsecurity.com/2019/08/cybersecurity-firm-imperva-discloses-bre... Lukas
The thing is. I can buy a brand new IP. It works fine on the websites. The moment it's hit by a DDoS Attack (TCP-AMP) .. Only 24-48 hours later, it's banned from all Inculpsa's aka Imperva's websites :) so something is horrible done wrong on their end and they're not interested in helping.. neither is Sony. On 08.01.2020 20:26:14, Lukas Tribus <lists@ltri.eu> wrote: Hello, On Wed, 8 Jan 2020 at 18:26, Octolus Development wrote:
The error it displays on both Sony, and Imperva (and whatever websites who uses their protection). So this problem is not with Sony, but rather Imperva blocking IP's wildly.
The IP's are not blocks, it's a single IP and the block/blacklist lifts after 7 days.
Error that appears on those websites, including imperva themself: This page can't be displayed. Contact support for additional information. The incident ID is: N/A.
That looks like a WAF, so reflection/spoofing is probably *not* the reason your IPs ended up on those lists. I assume what you see looks similar to what this returns (a request that looks like a sql injection): https://www.imperva.com/bla%20OR%201=1 A few of those hits, or crossing a certain threshold per IP (very easy for CGN IPs), and your IP probably ends up on those lists I guess. And of course those endpoints are not IPv6 enabled, so behind CGN the end customers shares his luck with it's neighbors even if everything is IPv6 enabled. Imperva, is that the "cybersecurity firm" that was breached 6 months ago? https://krebsonsecurity.com/2019/08/cybersecurity-firm-imperva-discloses-bre... Lukas
You're getting hit with something reported as "TCP-AMP" (I'm assuming TCP amplification; not sure what's classifying this for you) on your IP address, and then shortly thereafter that IP address is blocked from Imperva's services? Are the source IP addresses in those "TCP-AMP" attacks Sony IP addresses? That does start to sound like someone is bouncing TCP off of you (send you a SYN with spoofed Sony source IP address; have your devices respond with TCP SYN+ACK). It would still be unwise of Imperva to flag the address, but that could be the mechanism here, perhaps? -- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal On Wed, Jan 8, 2020 at 1:06 PM Octolus Development <admin@octolus.net> wrote:
The thing is.
I can buy a brand new IP. It works fine on the websites.
The moment it's hit by a DDoS Attack (TCP-AMP) .. Only 24-48 hours later, it's banned from all Inculpsa's aka Imperva's websites :) so something is horrible done wrong on their end and they're not interested in helping.. neither is Sony.
On 08.01.2020 20:26:14, Lukas Tribus <lists@ltri.eu> wrote: Hello,
On Wed, 8 Jan 2020 at 18:26, Octolus Development wrote:
The error it displays on both Sony, and Imperva (and whatever websites
who uses their protection). So this problem is not with Sony, but rather Imperva blocking IP's wildly.
The IP's are not blocks, it's a single IP and the block/blacklist lifts
after 7 days.
Error that appears on those websites, including imperva themself: This page can't be displayed. Contact support for additional
information.
The incident ID is: N/A.
That looks like a WAF, so reflection/spoofing is probably *not* the reason your IPs ended up on those lists.
I assume what you see looks similar to what this returns (a request that looks like a sql injection):
https://www.imperva.com/bla%20OR%201=1
A few of those hits, or crossing a certain threshold per IP (very easy for CGN IPs), and your IP probably ends up on those lists I guess. And of course those endpoints are not IPv6 enabled, so behind CGN the end customers shares his luck with it's neighbors even if everything is IPv6 enabled.
Imperva, is that the "cybersecurity firm" that was breached 6 months ago?
https://krebsonsecurity.com/2019/08/cybersecurity-firm-imperva-discloses-bre...
Lukas
Peace, Hey, your website says you're the developer of OctoVPN which is a VPN solution. *This* might be effectively the reason of blocking, not a DDoS. Gaming and streaming services typically discourage VPN traffic because a) VPNs help to circumvent regional restrictions, b) miscreants use VPNs to hide while breaking into systems, c) other reasons. Imperva is a Web app firewall solution much more than it is a DDoS protection device after all. -- Töma On Wed, Jan 8, 2020, 11:56 PM Octolus Development <admin@octolus.net> wrote:
The error it displays on both Sony, and Imperva (and whatever websites who uses their protection). So this problem is not with Sony, but rather Imperva blocking IP's wildly.
The IP's are not blocks, it's a single IP and the block/blacklist lifts after 7 days.
Error that appears on those websites, including imperva themself: This page can't be displayed. Contact support for additional information. The incident ID is: N/A.
No, that is not why. We deployed a brand new IP, and it was banned 24-48 hours after the DDoS Attack was hit. The other IP that was never attacked, never got banned. We've tracked down the issue and confirmed it is the DDoS Attack coming from Akamai and Imperva's IP's that are banning us from their network somehow. Sony are currently "looking into it" but they do not seem to care much. I am a customer of Sony, I own PlayStation consoles and I am not able to access their service. They tell me to change my IP instead of solving the actual problem with this exploit. On 08.01.2020 22:29:12, Töma Gavrichenkov <ximaera@gmail.com> wrote: Peace, Hey, your website says you're the developer of OctoVPN which is a VPN solution. *This* might be effectively the reason of blocking, not a DDoS. Gaming and streaming services typically discourage VPN traffic because a) VPNs help to circumvent regional restrictions, b) miscreants use VPNs to hide while breaking into systems, c) other reasons. Imperva is a Web app firewall solution much more than it is a DDoS protection device after all. -- Töma On Wed, Jan 8, 2020, 11:56 PM Octolus Development <admin@octolus.net [mailto:admin@octolus.net]> wrote: The error it displays on both Sony, and Imperva (and whatever websites who uses their protection). So this problem is not with Sony, but rather Imperva blocking IP's wildly. The IP's are not blocks, it's a single IP and the block/blacklist lifts after 7 days. Error that appears on those websites, including imperva themself: This page can't be displayed. Contact support for additional information. The incident ID is: N/A.
On Wednesday, 8 January, 2020 14:35. Octolus Development <admin@octolus.net> wrote:
Sony are currently "looking into it" but they do not seem to care much. I am a customer of Sony, I own PlayStation consoles and I am not able to access their service. They tell me to change my IP instead of solving the actual problem with this exploit.
Stop doing business with Criminal Organizations (SONY). Problem solved. -- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.
On Thu, Jan 9, 2020, at 00:05, Keith Medcalf wrote:
On Wednesday, 8 January, 2020 14:35. Octolus Development <admin@octolus.net> wrote:
Stop doing business with Criminal Organizations (SONY). Problem solved.
You (as a provider) may not do any business with them, but your customers may, and will yell at you if/when it doesn't work.
Exactly that. I run a VPN Business dedicated to protecting clients from DDoS Attacks that happens "all day long" on PlayStation Network. We need our VPN to work on PSN, all our customers uses their service. They are still investigating the problem, let's see what the results will be. On 10.01.2020 10:20:08, Radu-Adrian Feurdean <nanog@radu-adrian.feurdean.net> wrote: On Thu, Jan 9, 2020, at 00:05, Keith Medcalf wrote:
On Wednesday, 8 January, 2020 14:35. Octolus Development wrote:
Stop doing business with Criminal Organizations (SONY). Problem solved.
You (as a provider) may not do any business with them, but your customers may, and will yell at you if/when it doesn't work.
On Fri, 10 Jan 2020, Octolus Development wrote:
I run a VPN Business dedicated to protecting clients from DDoS Attacks that happens "all day long" on PlayStation Network. We need our VPN to work on PSN, all our customers uses their service.
They are still investigating the problem, let's see what the results will be.
Does your VPN provide what Sony cares about, which I do not know but might include things like only exiting CH customers via CH end-points / proxies so that non-CH (e.g., UK) only content can be blocked -- if not you may never gain traction with them and even if you do it might be quite hard to prove to their satisfaction. /mark
It's not about that this thread is about, nor why it is blacklisted. There is an exploit (DDoS) that will ban even home connections from their networks. On 10.01.2020 19:51:10, Mark Milhollan <mlm@pixelgate.net> wrote: On Fri, 10 Jan 2020, Octolus Development wrote:
I run a VPN Business dedicated to protecting clients from DDoS Attacks that happens "all day long" on PlayStation Network. We need our VPN to work on PSN, all our customers uses their service.
They are still investigating the problem, let's see what the results will be.
Does your VPN provide what Sony cares about, which I do not know but might include things like only exiting CH customers via CH end-points / proxies so that non-CH (e.g., UK) only content can be blocked -- if not you may never gain traction with them and even if you do it might be quite hard to prove to their satisfaction. /mark
Hey everyone, decided to do a small update for those who are interested. - Sony reached out to me, they whitelisted our IP's temporarily but then removed them. We have not heard from them since (10th January) - We tracked down the cause of the blacklist, it is happening because we are a victim of a TCP-AMP DDoS Attack. The TCP-AMP Attack works like this; - The attacker spoofs our server's ip, to thousands of services running a web server on port 80. - These web services, then respond back to our server - thinking we're the one that made a request. It seems like hundreds of these web servers that are receiving those spoofed requests from our IP, runs CSF or some kind of firewall system that automatically detects many connections to their web server. And automatically reports it to multiple different services, which ends up in us getting blacklisted. Imperva, which is what Sony uses are importing blacklists from multiple different trusted databases.. Which is how we're getting banned by Sony. Which uses Imperva on all their services, as their web firewall. The solution? There isn't really any. We are the victim here, the attackers are spoofing attacks from our IP's - and the services that are reflecting back to us, are reporting us for "attacking" them even though the requests are fully spoofed. On 10.01.2020 19:51:10, Mark Milhollan <mlm@pixelgate.net> wrote: On Fri, 10 Jan 2020, Octolus Development wrote:
I run a VPN Business dedicated to protecting clients from DDoS Attacks that happens "all day long" on PlayStation Network. We need our VPN to work on PSN, all our customers uses their service.
They are still investigating the problem, let's see what the results will be.
Does your VPN provide what Sony cares about, which I do not know but might include things like only exiting CH customers via CH end-points / proxies so that non-CH (e.g., UK) only content can be blocked -- if not you may never gain traction with them and even if you do it might be quite hard to prove to their satisfaction. /mark
One approach would be to trace the true origin of the spoofed packets, and get it filtered by their upstream. To that end, can you share some details of a recent tcp-amp attack? Eg, the victim IP and a timestamp? Damian On Mon, Jan 27, 2020 at 12:06 PM Octolus Development <admin@octolus.net> wrote:
Hey everyone, decided to do a small update for those who are interested.
- Sony reached out to me, they whitelisted our IP's temporarily but then removed them. We have not heard from them since (10th January) - We tracked down the cause of the blacklist, it is happening because we are a victim of a TCP-AMP DDoS Attack.
The TCP-AMP Attack works like this; - The attacker spoofs our server's ip, to thousands of services running a web server on port 80. - These web services, then respond back to our server - thinking we're the one that made a request.
It seems like hundreds of these web servers that are receiving those spoofed requests from our IP, runs CSF or some kind of firewall system that automatically detects many connections to their web server. And automatically reports it to multiple different services, which ends up in us getting blacklisted.
Imperva, which is what Sony uses are importing blacklists from multiple different trusted databases.. Which is how we're getting banned by Sony. Which uses Imperva on all their services, as their web firewall.
The solution? There isn't really any. We are the victim here, the attackers are spoofing attacks from our IP's - and the services that are reflecting back to us, are reporting us for "attacking" them even though the requests are fully spoofed.
On 10.01.2020 19:51:10, Mark Milhollan <mlm@pixelgate.net> wrote: On Fri, 10 Jan 2020, Octolus Development wrote:
I run a VPN Business dedicated to protecting clients from DDoS Attacks that happens "all day long" on PlayStation Network. We need our VPN to work on PSN, all our customers uses their service.
They are still investigating the problem, let's see what the results will be.
Does your VPN provide what Sony cares about, which I do not know but might include things like only exiting CH customers via CH end-points / proxies so that non-CH (e.g., UK) only content can be blocked -- if not you may never gain traction with them and even if you do it might be quite hard to prove to their satisfaction.
/mark
It is impossible to find the true origin of where the spoofed attacks are coming from. I don't have an exact timestamp, because the attacks are really difficult to see as well. As I said, you can block the IP from accessing internet completely. Yet, some services will flag our IP as "port flooding" their service - despite the fact it's fully spoofed. We received multiple flags at OVH of port flooding. This was one of the reports we got: tcp: 51.81.119.7:30364 -> 209.208.32.250:80 (SYN_RECV) tcp: 51.81.119.7:41535 -> 209.208.32.250:80 (SYN_RECV) tcp: 51.81.119.7:1089 -> 209.208.32.250:80 (SYN_RECV) tcp: 51.81.119.7:4433 -> 209.208.32.250:80 (SYN_RECV) On 27.01.2020 21:29:11, Damian Menscher <damian@google.com> wrote: One approach would be to trace the true origin of the spoofed packets, and get it filtered by their upstream. To that end, can you share some details of a recent tcp-amp attack? Eg, the victim IP and a timestamp? Damian On Mon, Jan 27, 2020 at 12:06 PM Octolus Development <admin@octolus.net [mailto:admin@octolus.net]> wrote: Hey everyone, decided to do a small update for those who are interested. - Sony reached out to me, they whitelisted our IP's temporarily but then removed them. We have not heard from them since (10th January) - We tracked down the cause of the blacklist, it is happening because we are a victim of a TCP-AMP DDoS Attack. The TCP-AMP Attack works like this; - The attacker spoofs our server's ip, to thousands of services running a web server on port 80. - These web services, then respond back to our server - thinking we're the one that made a request. It seems like hundreds of these web servers that are receiving those spoofed requests from our IP, runs CSF or some kind of firewall system that automatically detects many connections to their web server. And automatically reports it to multiple different services, which ends up in us getting blacklisted. Imperva, which is what Sony uses are importing blacklists from multiple different trusted databases.. Which is how we're getting banned by Sony. Which uses Imperva on all their services, as their web firewall. The solution? There isn't really any. We are the victim here, the attackers are spoofing attacks from our IP's - and the services that are reflecting back to us, are reporting us for "attacking" them even though the requests are fully spoofed. On 10.01.2020 19:51:10, Mark Milhollan <mlm@pixelgate.net [mailto:mlm@pixelgate.net]> wrote: On Fri, 10 Jan 2020, Octolus Development wrote:
I run a VPN Business dedicated to protecting clients from DDoS Attacks that happens "all day long" on PlayStation Network. We need our VPN to work on PSN, all our customers uses their service.
They are still investigating the problem, let's see what the results will be.
Does your VPN provide what Sony cares about, which I do not know but might include things like only exiting CH customers via CH end-points / proxies so that non-CH (e.g., UK) only content can be blocked -- if not you may never gain traction with them and even if you do it might be quite hard to prove to their satisfaction. /mark
On Jan 28, 2020, at 04:12, Octolus Development <admin@octolus.net> wrote: It is impossible to find the true origin of where the spoofed attacks are coming from. This is demonstrably untrue. If you provide the requisite information to operators, they can look through their flow telemetry collection/analysis systems in order to determine whether the spoofed traffic traversed their network; if it did so, they will see where it ingressed their network. With enough participants who have this capability, it's possible to trace the spoofed traffic back to its origin network, or at least some network or networks topologically proximate to the origin network. That's what Damian is suggesting. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
If someone is being spoofed, they aren't receiving the spoofed packets. How are they supposed to collect anything on the attack? Offending host pretending to be Octolus -> Sony -> Real Octolus. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Roland Dobbins" <Roland.Dobbins@netscout.com> To: "Octolus Development" <admin@octolus.net> Cc: "Heather Schiller via NANOG" <nanog@nanog.org> Sent: Monday, January 27, 2020 6:29:16 PM Subject: Re: Reaching out to Sony NOC, resolving DDoS Issues - Need POC On Jan 28, 2020, at 04:12, Octolus Development <admin@octolus.net> wrote: <blockquote> It is impossible to find the true origin of where the spoofed attacks are coming from. </blockquote> This is demonstrably untrue. If you provide the requisite information to operators, they can look through their flow telemetry collection/analysis systems in order to determine whether the spoofed traffic traversed their network; if it did so, they will see where it ingressed their network. With enough participants who have this capability, it's possible to trace the spoofed traffic back to its origin network, or at least some network or networks topologically proximate to the origin network. That's what Damian is suggesting. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
Transit carriers could work the flows backwards. -Ben Cannon CEO 6x7 Networks & 6x7 Telecom, LLC ben@6by7.net <mailto:ben@6by7.net>
On Jan 27, 2020, at 4:39 PM, Mike Hammett <nanog@ics-il.net> wrote:
If someone is being spoofed, they aren't receiving the spoofed packets. How are they supposed to collect anything on the attack?
Offending host pretending to be Octolus -> Sony -> Real Octolus.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com <http://www.ics-il.com/>
Midwest-IX http://www.midwest-ix.com <http://www.midwest-ix.com/>
From: "Roland Dobbins" <Roland.Dobbins@netscout.com <mailto:Roland.Dobbins@netscout.com>> To: "Octolus Development" <admin@octolus.net <mailto:admin@octolus.net>> Cc: "Heather Schiller via NANOG" <nanog@nanog.org <mailto:nanog@nanog.org>> Sent: Monday, January 27, 2020 6:29:16 PM Subject: Re: Reaching out to Sony NOC, resolving DDoS Issues - Need POC
On Jan 28, 2020, at 04:12, Octolus Development <admin@octolus.net <mailto:admin@octolus.net>> wrote:
It is impossible to find the true origin of where the spoofed attacks are coming from.
This is demonstrably untrue.
If you provide the requisite information to operators, they can look through their flow telemetry collection/analysis systems in order to determine whether the spoofed traffic traversed their network; if it did so, they will see where it ingressed their network.
With enough participants who have this capability, it's possible to trace the spoofed traffic back to its origin network, or at least some network or networks topologically proximate to the origin network.
That's what Damian is suggesting.
-------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com <mailto:roland.dobbins@netscout.com>>
How would they know what to look for? I'm assuming Sony isn't cooperating. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Ben Cannon" <ben@6by7.net> To: "Mike Hammett" <nanog@ics-il.net> Cc: "Roland Dobbins" <Roland.Dobbins@netscout.com>, "NANOG Operators' Group" <nanog@nanog.org> Sent: Monday, January 27, 2020 6:40:25 PM Subject: Re: Reaching out to Sony NOC, resolving DDoS Issues - Need POC Transit carriers could work the flows backwards. -Ben Cannon CEO 6x7 Networks & 6x7 Telecom, LLC ben@6by7.net On Jan 27, 2020, at 4:39 PM, Mike Hammett < nanog@ics-il.net > wrote: If someone is being spoofed, they aren't receiving the spoofed packets. How are they supposed to collect anything on the attack? Offending host pretending to be Octolus -> Sony -> Real Octolus. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Roland Dobbins" < Roland.Dobbins@netscout.com > To: "Octolus Development" < admin@octolus.net > Cc: "Heather Schiller via NANOG" < nanog@nanog.org > Sent: Monday, January 27, 2020 6:29:16 PM Subject: Re: Reaching out to Sony NOC, resolving DDoS Issues - Need POC <blockquote> On Jan 28, 2020, at 04:12, Octolus Development < admin@octolus.net > wrote: <blockquote> It is impossible to find the true origin of where the spoofed attacks are coming from. </blockquote> This is demonstrably untrue. If you provide the requisite information to operators, they can look through their flow telemetry collection/analysis systems in order to determine whether the spoofed traffic traversed their network; if it did so, they will see where it ingressed their network. With enough participants who have this capability, it's possible to trace the spoofed traffic back to its origin network, or at least some network or networks topologically proximate to the origin network. That's what Damian is suggesting. -------------------------------------------- Roland Dobbins < roland.dobbins@netscout.com > </blockquote>
The victim already posted the signature to this thread: - source IP: 51.81.119.7 - protocol: 6 (tcp) - tcp_flags: 2 (syn) That alone is sufficient for Level3/CenturyLink/etc to identify the source of this abuse and apply filters, if they choose. For a more detailed explanation of how to trace and filter spoofed attacks, see my talk at NANOG last year: https://pc.nanog.org/static/published/meetings//NANOG76/daily/day_2.html#tal... Damian On Mon, Jan 27, 2020 at 4:57 PM Mike Hammett <nanog@ics-il.net> wrote:
How would they know what to look for?
I'm assuming Sony isn't cooperating.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
------------------------------ *From: *"Ben Cannon" <ben@6by7.net> *To: *"Mike Hammett" <nanog@ics-il.net> *Cc: *"Roland Dobbins" <Roland.Dobbins@netscout.com>, "NANOG Operators' Group" <nanog@nanog.org> *Sent: *Monday, January 27, 2020 6:40:25 PM *Subject: *Re: Reaching out to Sony NOC, resolving DDoS Issues - Need POC
Transit carriers could work the flows backwards.
-Ben Cannon CEO 6x7 Networks & 6x7 Telecom, LLC ben@6by7.net
On Jan 27, 2020, at 4:39 PM, Mike Hammett <nanog@ics-il.net> wrote:
If someone is being spoofed, they aren't receiving the spoofed packets. How are they supposed to collect anything on the attack?
Offending host pretending to be Octolus -> Sony -> Real Octolus.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
------------------------------ *From: *"Roland Dobbins" <Roland.Dobbins@netscout.com> *To: *"Octolus Development" <admin@octolus.net> *Cc: *"Heather Schiller via NANOG" <nanog@nanog.org> *Sent: *Monday, January 27, 2020 6:29:16 PM *Subject: *Re: Reaching out to Sony NOC, resolving DDoS Issues - Need POC
On Jan 28, 2020, at 04:12, Octolus Development <admin@octolus.net> wrote:
It is impossible to find the true origin of where the spoofed attacks are coming from.
This is demonstrably untrue.
If you provide the requisite information to operators, they can look through their flow telemetry collection/analysis systems in order to determine whether the spoofed traffic traversed their network; if it did so, they will see where it ingressed their network.
With enough participants who have this capability, it's possible to trace the spoofed traffic back to its origin network, or at least some network or networks topologically proximate to the origin network.
That's what Damian is suggesting.
-------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
Peace, On Tue, Jan 28, 2020, 4:02 AM Damian Menscher via NANOG <nanog@nanog.org> wrote:
The victim already posted the signature to this thread: - source IP: 51.81.119.7 - protocol: 6 (tcp) - tcp_flags: 2 (syn)
That alone is sufficient for Level3/CenturyLink/etc to identify the source of this abuse and apply filters, if they choose.
If this endpoint doesn't connect to anything outside of their network, then yes. If it does though, the design of the filter might become more complicated. If the OP posted their IPv4 addresses and networks to the list, it could've been easier though (however the concerns about the administrative processing procedures outlined before still apply). -- Töma
On Mon, Jan 27, 2020 at 5:10 PM Töma Gavrichenkov <ximaera@gmail.com> wrote:
On Tue, Jan 28, 2020, 4:02 AM Damian Menscher via NANOG <nanog@nanog.org> wrote:
The victim already posted the signature to this thread: - source IP: 51.81.119.7 - protocol: 6 (tcp) - tcp_flags: 2 (syn)
That alone is sufficient for Level3/CenturyLink/etc to identify the source of this abuse and apply filters, if they choose.
If this endpoint doesn't connect to anything outside of their network, then yes. If it does though, the design of the filter might become more complicated.
Not really... just requires sorting by volume. Turns out most legitimate hosts don't send high-volume syn packets. ;) The same could be said of high-volume UDP packets destined to known amplification ports. If the OP posted their IPv4 addresses and networks to the list, it could've
been easier though (however the concerns about the administrative processing procedures outlined before still apply).
The victim info is only really needed if you are focused on a particular case. A motivated person at a transit provider could likely identify all sources of spoofing (from their customers) with a day's work. Multiple transit providers would need to work together to address all cases, as the source might be a customer of only one of them. If anyone at a transit provider wants to attempt this feel free to contact me off-list for tips. Damian
Peace, On Tue, Jan 28, 2020, 4:32 AM Damian Menscher <damian@google.com> wrote:
On Mon, Jan 27, 2020 at 5:10 PM Töma Gavrichenkov <ximaera@gmail.com> wrote:
If this endpoint doesn't connect to anything outside of their network, then yes. If it does though, the design of the filter might become more complicated.
Not really... just requires sorting by volume. Turns out most legitimate hosts don't send high-volume syn packets. ;)
This is a good *detection* technique, but you cannot filter by volume in transit if the set of destinations is large (and random) enough, and you don't have a time machine. Not sure if this is the case but might as well be. As for the detection of the real source, everything is technically possible but you need certain bargaining power which a medium-sized (at best) VPN service probably doesn't have. -- Töma
Peace, On Tue, Jan 28, 2020, 4:42 AM Töma Gavrichenkov <ximaera@gmail.com> wrote:
As for the detection of the real source, everything is technically possible but you need certain bargaining power which a medium-sized (at best) VPN service probably doesn't have.
...because if they *did* have some, they could've rather convinced Sony (or the respective security providers) not to apply braindead filtering techniques (assuming they do, as the OP states). -- Töma
On Mon, Jan 27, 2020 at 5:43 PM Töma Gavrichenkov <ximaera@gmail.com> wrote:
On Tue, Jan 28, 2020, 4:32 AM Damian Menscher <damian@google.com> wrote:
On Mon, Jan 27, 2020 at 5:10 PM Töma Gavrichenkov <ximaera@gmail.com> wrote:
If this endpoint doesn't connect to anything outside of their network, then yes. If it does though, the design of the filter might become more complicated.
Not really... just requires sorting by volume. Turns out most legitimate hosts don't send high-volume syn packets. ;)
This is a good *detection* technique, but you cannot filter by volume in transit if the set of destinations is large (and random) enough, and you don't have a time machine. Not sure if this is the case but might as well be.
They don't need to filter by destination. Once a problem customer has been identified, they can apply an ACL restricting them to only originate IPs they own. This was all covered in my talk at NANOG last year: https://pc.nanog.org/static/published/meetings//NANOG76/daily/day_2.html#tal... As for the detection of the real source, everything is technically possible
but you need certain bargaining power which a medium-sized (at best) VPN service probably doesn't have.
True, but there are ways around that, including public shaming (here), or involving law enforcement. Damian
Peace, On Tue, Jan 28, 2020, 4:49 AM Damian Menscher <damian@google.com> wrote:
They don't need to filter by destination. Once a problem customer has been identified, they can apply an ACL restricting them to only originate IPs they own.
[..]
there are ways around that, including public shaming (here), or involving
law enforcement.
Well, don't we see how well public shaming has worked out with Sony so far. -- Töma
Maybe we're looking at the wrong place when dealing with TCP amp. I believe there is a much easier way to solve this. @OP: can you post the tcp flags of the SYN/CK you are receiving from Sony? Thanks Jean On 2020-01-27 20:49, Damian Menscher via NANOG wrote:
On Mon, Jan 27, 2020 at 5:43 PM Töma Gavrichenkov <ximaera@gmail.com <mailto:ximaera@gmail.com>> wrote:
On Tue, Jan 28, 2020, 4:32 AM Damian Menscher <damian@google.com <mailto:damian@google.com>> wrote:
On Mon, Jan 27, 2020 at 5:10 PM Töma Gavrichenkov <ximaera@gmail.com <mailto:ximaera@gmail.com>> wrote:
If this endpoint doesn't connect to anything outside of their network, then yes. If it does though, the design of the filter might become more complicated.
Not really... just requires sorting by volume. Turns out most legitimate hosts don't send high-volume syn packets. ;)
This is a good *detection* technique, but you cannot filter by volume in transit if the set of destinations is large (and random) enough, and you don't have a time machine. Not sure if this is the case but might as well be.
They don't need to filter by destination. Once a problem customer has been identified, they can apply an ACL restricting them to only originate IPs they own. This was all covered in my talk at NANOG last year: https://pc.nanog.org/static/published/meetings//NANOG76/daily/day_2.html#tal...
As for the detection of the real source, everything is technically possible but you need certain bargaining power which a medium-sized (at best) VPN service probably doesn't have.
True, but there are ways around that, including public shaming (here), or involving law enforcement.
Damian
You will still see backscatter which will let you know your space is being spoofed as well. This will show in your telemetry. Sent from my iCar
On Jan 27, 2020, at 7:57 PM, Mike Hammett <nanog@ics-il.net> wrote:
How would they know what to look for?
I'm assuming Sony isn't cooperating.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
From: "Ben Cannon" <ben@6by7.net> To: "Mike Hammett" <nanog@ics-il.net> Cc: "Roland Dobbins" <Roland.Dobbins@netscout.com>, "NANOG Operators' Group" <nanog@nanog.org> Sent: Monday, January 27, 2020 6:40:25 PM Subject: Re: Reaching out to Sony NOC, resolving DDoS Issues - Need POC
Transit carriers could work the flows backwards.
-Ben Cannon CEO 6x7 Networks & 6x7 Telecom, LLC ben@6by7.net
On Jan 27, 2020, at 4:39 PM, Mike Hammett <nanog@ics-il.net> wrote:
If someone is being spoofed, they aren't receiving the spoofed packets. How are they supposed to collect anything on the attack?
Offending host pretending to be Octolus -> Sony -> Real Octolus.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
From: "Roland Dobbins" <Roland.Dobbins@netscout.com> To: "Octolus Development" <admin@octolus.net> Cc: "Heather Schiller via NANOG" <nanog@nanog.org> Sent: Monday, January 27, 2020 6:29:16 PM Subject: Re: Reaching out to Sony NOC, resolving DDoS Issues - Need POC
On Jan 28, 2020, at 04:12, Octolus Development <admin@octolus.net> wrote:
It is impossible to find the true origin of where the spoofed attacks are coming from.
This is demonstrably untrue.
If you provide the requisite information to operators, they can look through their flow telemetry collection/analysis systems in order to determine whether the spoofed traffic traversed their network; if it did so, they will see where it ingressed their network.
With enough participants who have this capability, it's possible to trace the spoofed traffic back to its origin network, or at least some network or networks topologically proximate to the origin network.
That's what Damian is suggesting.
-------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
Peace, On Tue, Jan 28, 2020, 3:43 AM Ben Cannon <ben@6by7.net> wrote:
Transit carriers could work the flows backwards.
And if the stars align, some of them might even do that for you once even though you are not their direct customer. Next you're going to convince them to talk to the (probably abuse resistant) data center where this garbage is hosted in order to shut the offender down. This even happened before in the past, but not frequently enough I'd say. -- Töma
On Jan 28, 2020, at 07:39, Mike Hammett <nanog@ics-il.net> wrote: If someone is being spoofed, they aren't receiving the spoofed packets. How are they supposed to collect anything on the attack? OP stated that *his own network* was being packeted with a TCP reflection/amplification attack. This means that if he's collecting flow telemetry from his edge routers, he sees the details of the resultant attack traffic, & since that attack traffic isn't spoofed from his perspective, he can ask the networks on which the abused reflectors/amplifiers reside, & their peers/transits he can infer, to perform traceback, & work it network-by-network. And even if his network weren't on the receiving end of a reflection/amplification attack, OP could still see backscatter, as Jared indicated. Instrumenting one's network in order to achieve visibility into one's traffic is quite beneficial. It's easy & inexpensive to get started with open-source tools. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
On Jan 28, 2020, at 11:40, Dobbins, Roland <Roland.Dobbins@netscout.com> wrote: And even if his network weren't on the receiving end of a reflection/amplification attack, OP could still see backscatter, as Jared indicated. In point of fact, if the traffic was low-volume, this might in fact be what he was seeing. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
The problem is that they are spoofing our IP, to millions of IP's running port 80. Making upstream providers filter it is quite difficult, i don't know all the upstream providers are used. The main problem is honestly services that reports SYN_RECV as Port Flood, but there isn't much one can do about misconfigured firewalls.I am sure there is a decent amount of honeypots on the internet acting the same way, resulting us (the victims of the attack) getting blacklisted for 'sending' attacks. On 28.01.2020 05:50:14, "Dobbins, Roland" <roland.dobbins@netscout.com> wrote: On Jan 28, 2020, at 11:40, Dobbins, Roland <Roland.Dobbins@netscout.com> wrote: And even if his network weren't on the receiving end of a reflection/amplification attack, OP could still see backscatter, as Jared indicated. In point of fact, if the traffic was low-volume, this might in fact be what he was seeing. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
On 28 Jan 2020, at 18:15, Octolus Development wrote:
The problem is that they are spoofing our IP, to millions of IP's running port 80.
So that does in fact sound like a TCP reflection/amplification attack. If you have the relevant information, as it seems that you do, you can ask operators to perform traceback (they'll likely need timestamps). Hopefully, some operators will read this thread and begin looking into it, as well. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
Trying to summarize here, this convo has been a bit disjointed. Is this an accurate summary? - The malicious traffic with spoofed sources is targeting multiple different destinations. - The aggregate of all those flows is causing Impervia to flag your IP range as a bad actor. - Sony uses Impervia blacklists, and since Impervia has flagged your space as bad, Sony is blocking you. If that is true, my advice would be to go right to Impervia. Explain the situation, and ask for their assistance in identifying and or/reaching out to the networks that they are detecting this spoofed traffic coming from. The backscatter, as Jared said earlier, could probably help you a bit too, but Impervia should be willing to assist. It's in their best interests to not have false positives, but who knows. On Tue, Jan 28, 2020 at 6:17 AM Octolus Development <admin@octolus.net> wrote:
The problem is that they are spoofing our IP, to millions of IP's running port 80. Making upstream providers filter it is quite difficult, i don't know all the upstream providers are used.
The main problem is honestly services that reports SYN_RECV as Port Flood, but there isn't much one can do about misconfigured firewalls.I am sure there is a decent amount of honeypots on the internet acting the same way, resulting us (the victims of the attack) getting blacklisted for 'sending' attacks.
On 28.01.2020 05:50:14, "Dobbins, Roland" <roland.dobbins@netscout.com> wrote:
On Jan 28, 2020, at 11:40, Dobbins, Roland <Roland.Dobbins@netscout.com> wrote:
And even if his network weren't on the receiving end of a reflection/amplification attack, OP could still see backscatter, as Jared indicated.
In point of fact, if the traffic was low-volume, this might in fact be what he was seeing.
--------------------------------------------
Roland Dobbins <roland.dobbins@netscout.com>
I have tried numerous of times to reach out to Imperva. Imperva said Sony have to contact them & said they cannot help me because I am not a customer of theirs. Something Sony will not do. Sony simply stopped responding my emails after some time. But yes you are right. My IP's are being spoofed, spoofing SYN requests to hundreds of thousands of web servers. Which then results in a blacklist, that Imperva uses.. which prevents me and my clients from accessing Sony's services.. because they use Imperva. On 28.01.2020 17:29:12, Tom Beecher <beecher@beecher.cc> wrote: Trying to summarize here, this convo has been a bit disjointed. Is this an accurate summary? - The malicious traffic with spoofed sources is targeting multiple different destinations. - The aggregate of all those flows is causing Impervia to flag your IP range as a bad actor. - Sony uses Impervia blacklists, and since Impervia has flagged your space as bad, Sony is blocking you. If that is true, my advice would be to go right to Impervia. Explain the situation, and ask for their assistance in identifying and or/reaching out to the networks that they are detecting this spoofed traffic coming from. The backscatter, as Jared said earlier, could probably help you a bit too, but Impervia should be willing to assist. It's in their best interests to not have false positives, but who knows. On Tue, Jan 28, 2020 at 6:17 AM Octolus Development <admin@octolus.net [mailto:admin@octolus.net]> wrote: The problem is that they are spoofing our IP, to millions of IP's running port 80. Making upstream providers filter it is quite difficult, i don't know all the upstream providers are used. The main problem is honestly services that reports SYN_RECV as Port Flood, but there isn't much one can do about misconfigured firewalls.I am sure there is a decent amount of honeypots on the internet acting the same way, resulting us (the victims of the attack) getting blacklisted for 'sending' attacks. On 28.01.2020 05:50:14, "Dobbins, Roland" <roland.dobbins@netscout.com [mailto:roland.dobbins@netscout.com]> wrote: On Jan 28, 2020, at 11:40, Dobbins, Roland <Roland.Dobbins@netscout.com [mailto:Roland.Dobbins@netscout.com]> wrote: And even if his network weren't on the receiving end of a reflection/amplification attack, OP could still see backscatter, as Jared indicated. In point of fact, if the traffic was low-volume, this might in fact be what he was seeing. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com [mailto:roland.dobbins@netscout.com]>
But you do receive the SYN/ACK? The way to open a TCP socket is the 3 way handshake. Sorry to write that here... I feel it's useless. 1. SYN 2. SYN/ACK 3. ACK Step 1: So hackers spoof the original SYN with your source IP of your network. Step 2: You should then receive those SYN/ACK packets with your network as the dst ip and SONY as the src ip. Can you catch a few and post the TCP flags that you see please? (This is step 2) You don't need sony or imperva for that. Just a sniffer at the right place in your network. You won't block anything, but we should see something very interesting that will help you fix this. If it is happening like you are describing, you should see those packets and you should be able to capture them. No worries if you can't. Jean On 2020-01-28 11:31, Octolus Development wrote:
I have tried numerous of times to reach out to Imperva.
Imperva said Sony have to contact them & said they cannot help me because I am not a customer of theirs. Something Sony will not do. Sony simply stopped responding my emails after some time.
But yes you are right.
My IP's are being spoofed, spoofing SYN requests to hundreds of thousands of web servers. Which then results in a blacklist, that Imperva uses.. which prevents me and my clients from accessing Sony's services.. because they use Imperva.
On 28.01.2020 17:29:12, Tom Beecher <beecher@beecher.cc> wrote:
Trying to summarize here, this convo has been a bit disjointed.
Is this an accurate summary?
- The malicious traffic with spoofed sources is targeting multiple different destinations. - The aggregate of all those flows is causing Impervia to flag your IP range as a bad actor. - Sony uses Impervia blacklists, and since Impervia has flagged your space as bad, Sony is blocking you.
If that is true, my advice would be to go right to Impervia. Explain the situation, and ask for their assistance in identifying and or/reaching out to the networks that they are detecting this spoofed traffic coming from. The backscatter, as Jared said earlier, could probably help you a bit too, but Impervia should be willing to assist. It's in their best interests to not have false positives, but who knows.
On Tue, Jan 28, 2020 at 6:17 AM Octolus Development <admin@octolus.net <mailto:admin@octolus.net>> wrote:
The problem is that they are spoofing our IP, to millions of IP's running port 80. Making upstream providers filter it is quite difficult, i don't know all the upstream providers are used.
The main problem is honestly services that reports SYN_RECV as Port Flood, but there isn't much one can do about misconfigured firewalls.I am sure there is a decent amount of honeypots on the internet acting the same way, resulting us (the victims of the attack) getting blacklisted for 'sending' attacks.
On 28.01.2020 05:50:14, "Dobbins, Roland" <roland.dobbins@netscout.com <mailto:roland.dobbins@netscout.com>> wrote:
On Jan 28, 2020, at 11:40, Dobbins, Roland <Roland.Dobbins@netscout.com <mailto:Roland.Dobbins@netscout.com>> wrote:
And even if his network weren't on the receiving end of a reflection/amplification attack, OP could still see backscatter, as Jared indicated.
In point of fact, if the traffic was low-volume, this might in fact be what he was seeing.
--------------------------------------------
Roland Dobbins <roland.dobbins@netscout.com <mailto:roland.dobbins@netscout.com>>
Yes, my server would then respond with RST. Screenshot: https://i.imgur.com/ZVti2yY.png We've blocked outgoing RST, 136.244.67.19 was our test server. But even if the ip is not even exposed to the internet, services will blacklist us. Even if we don't respond, and block every request from the internet incoming & outgoing. On 28.01.2020 22:36:18, "Jean | ddostest.me via NANOG" <nanog@nanog.org> wrote: But you do receive the SYN/ACK? The way to open a TCP socket is the 3 way handshake. Sorry to write that here... I feel it's useless. 1. SYN 2. SYN/ACK 3. ACK Step 1: So hackers spoof the original SYN with your source IP of your network. Step 2: You should then receive those SYN/ACK packets with your network as the dst ip and SONY as the src ip. Can you catch a few and post the TCP flags that you see please? (This is step 2) You don't need sony or imperva for that. Just a sniffer at the right place in your network. You won't block anything, but we should see something very interesting that will help you fix this. If it is happening like you are describing, you should see those packets and you should be able to capture them. No worries if you can't. Jean On 2020-01-28 11:31, Octolus Development wrote: I have tried numerous of times to reach out to Imperva. Imperva said Sony have to contact them & said they cannot help me because I am not a customer of theirs. Something Sony will not do. Sony simply stopped responding my emails after some time. But yes you are right. My IP's are being spoofed, spoofing SYN requests to hundreds of thousands of web servers. Which then results in a blacklist, that Imperva uses.. which prevents me and my clients from accessing Sony's services.. because they use Imperva. On 28.01.2020 17:29:12, Tom Beecher <beecher@beecher.cc> [mailto:beecher@beecher.cc] wrote: Trying to summarize here, this convo has been a bit disjointed. Is this an accurate summary? - The malicious traffic with spoofed sources is targeting multiple different destinations. - The aggregate of all those flows is causing Impervia to flag your IP range as a bad actor. - Sony uses Impervia blacklists, and since Impervia has flagged your space as bad, Sony is blocking you. If that is true, my advice would be to go right to Impervia. Explain the situation, and ask for their assistance in identifying and or/reaching out to the networks that they are detecting this spoofed traffic coming from. The backscatter, as Jared said earlier, could probably help you a bit too, but Impervia should be willing to assist. It's in their best interests to not have false positives, but who knows. On Tue, Jan 28, 2020 at 6:17 AM Octolus Development <admin@octolus.net [mailto:admin@octolus.net]> wrote: The problem is that they are spoofing our IP, to millions of IP's running port 80. Making upstream providers filter it is quite difficult, i don't know all the upstream providers are used. The main problem is honestly services that reports SYN_RECV as Port Flood, but there isn't much one can do about misconfigured firewalls.I am sure there is a decent amount of honeypots on the internet acting the same way, resulting us (the victims of the attack) getting blacklisted for 'sending' attacks. On 28.01.2020 05:50:14, "Dobbins, Roland" <roland.dobbins@netscout.com [mailto:roland.dobbins@netscout.com]> wrote: On Jan 28, 2020, at 11:40, Dobbins, Roland <Roland.Dobbins@netscout.com [mailto:Roland.Dobbins@netscout.com]> wrote: And even if his network weren't on the receiving end of a reflection/amplification attack, OP could still see backscatter, as Jared indicated. In point of fact, if the traffic was low-volume, this might in fact be what he was seeing. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com [mailto:roland.dobbins@netscout.com]>
I recommend you *not* block the outgoing RST packets, as blocking them will only make matters worse: - it leaves the webservers being abused for reflection in the half-open SYN_RECV state, which may attract more attention (and blacklisting) - retries from those servers will increase the load to your network Damian On Tue, Jan 28, 2020 at 1:42 PM Octolus Development <admin@octolus.net> wrote:
Yes, my server would then respond with RST.
Screenshot: https://i.imgur.com/ZVti2yY.png
We've blocked outgoing RST, 136.244.67.19 was our test server.
But even if the ip is not even exposed to the internet, services will blacklist us. Even if we don't respond, and block every request from the internet incoming & outgoing.
On 28.01.2020 22:36:18, "Jean | ddostest.me via NANOG" <nanog@nanog.org> wrote:
But you do receive the SYN/ACK?
The way to open a TCP socket is the 3 way handshake. Sorry to write that here... I feel it's useless.
1. SYN
2. SYN/ACK
3. ACK
Step 1: So hackers spoof the original SYN with your source IP of your network.
Step 2: You should then receive those SYN/ACK packets with your network as the dst ip and SONY as the src ip. Can you catch a few and post the TCP flags that you see please? (This is step 2)
You don't need sony or imperva for that. Just a sniffer at the right place in your network. You won't block anything, but we should see something very interesting that will help you fix this.
If it is happening like you are describing, you should see those packets and you should be able to capture them.
No worries if you can't.
Jean On 2020-01-28 11:31, Octolus Development wrote:
I have tried numerous of times to reach out to Imperva.
Imperva said Sony have to contact them & said they cannot help me because I am not a customer of theirs. Something Sony will not do. Sony simply stopped responding my emails after some time.
But yes you are right.
My IP's are being spoofed, spoofing SYN requests to hundreds of thousands of web servers. Which then results in a blacklist, that Imperva uses.. which prevents me and my clients from accessing Sony's services.. because they use Imperva.
On 28.01.2020 17:29:12, Tom Beecher <beecher@beecher.cc> <beecher@beecher.cc> wrote: Trying to summarize here, this convo has been a bit disjointed.
Is this an accurate summary?
- The malicious traffic with spoofed sources is targeting multiple different destinations. - The aggregate of all those flows is causing Impervia to flag your IP range as a bad actor. - Sony uses Impervia blacklists, and since Impervia has flagged your space as bad, Sony is blocking you.
If that is true, my advice would be to go right to Impervia. Explain the situation, and ask for their assistance in identifying and or/reaching out to the networks that they are detecting this spoofed traffic coming from. The backscatter, as Jared said earlier, could probably help you a bit too, but Impervia should be willing to assist. It's in their best interests to not have false positives, but who knows.
On Tue, Jan 28, 2020 at 6:17 AM Octolus Development <admin@octolus.net> wrote:
The problem is that they are spoofing our IP, to millions of IP's running port 80. Making upstream providers filter it is quite difficult, i don't know all the upstream providers are used.
The main problem is honestly services that reports SYN_RECV as Port Flood, but there isn't much one can do about misconfigured firewalls.I am sure there is a decent amount of honeypots on the internet acting the same way, resulting us (the victims of the attack) getting blacklisted for 'sending' attacks.
On 28.01.2020 05:50:14, "Dobbins, Roland" <roland.dobbins@netscout.com> wrote:
On Jan 28, 2020, at 11:40, Dobbins, Roland <Roland.Dobbins@netscout.com> wrote:
And even if his network weren't on the receiving end of a reflection/amplification attack, OP could still see backscatter, as Jared indicated.
In point of fact, if the traffic was low-volume, this might in fact be what he was seeing.
--------------------------------------------
Roland Dobbins <roland.dobbins@netscout.com>
Not blocking them will drain my outgoing bandwidth. ---- On Wed, 29 Jan 2020 01:18:32 +0100 damian@google.com wrote ---- I recommend you *not* block the outgoing RST packets, as blocking them will only make matters worse: - it leaves the webservers being abused for reflection in the half-open SYN_RECV state, which may attract more attention (and blacklisting) - retries from those servers will increase the load to your network Damian On Tue, Jan 28, 2020 at 1:42 PM Octolus Development <admin@octolus.net> wrote: Yes, my server would then respond with RST. Screenshot: https://i.imgur.com/ZVti2yY.png We've blocked outgoing RST, 136.244.67.19 was our test server. But even if the ip is not even exposed to the internet, services will blacklist us. Even if we don't respond, and block every request from the internet incoming & outgoing. On 28.01.2020 22:36:18, "Jean | ddostest.me via NANOG" <nanog@nanog.org> wrote: But you do receive the SYN/ACK? The way to open a TCP socket is the 3 way handshake. Sorry to write that here... I feel it's useless. 1. SYN 2. SYN/ACK 3. ACK Step 1: So hackers spoof the original SYN with your source IP of your network. Step 2: You should then receive those SYN/ACK packets with your network as the dst ip and SONY as the src ip. Can you catch a few and post the TCP flags that you see please? (This is step 2) You don't need sony or imperva for that. Just a sniffer at the right place in your network. You won't block anything, but we should see something very interesting that will help you fix this. If it is happening like you are describing, you should see those packets and you should be able to capture them. No worries if you can't. Jean On 2020-01-28 11:31, Octolus Development wrote: I have tried numerous of times to reach out to Imperva. Imperva said Sony have to contact them & said they cannot help me because I am not a customer of theirs. Something Sony will not do. Sony simply stopped responding my emails after some time. But yes you are right. My IP's are being spoofed, spoofing SYN requests to hundreds of thousands of web servers. Which then results in a blacklist, that Imperva uses.. which prevents me and my clients from accessing Sony's services.. because they use Imperva. On 28.01.2020 17:29:12, Tom Beecher <beecher@beecher.cc> wrote: Trying to summarize here, this convo has been a bit disjointed. Is this an accurate summary? - The malicious traffic with spoofed sources is targeting multiple different destinations. - The aggregate of all those flows is causing Impervia to flag your IP range as a bad actor. - Sony uses Impervia blacklists, and since Impervia has flagged your space as bad, Sony is blocking you. If that is true, my advice would be to go right to Impervia. Explain the situation, and ask for their assistance in identifying and or/reaching out to the networks that they are detecting this spoofed traffic coming from. The backscatter, as Jared said earlier, could probably help you a bit too, but Impervia should be willing to assist. It's in their best interests to not have false positives, but who knows. On Tue, Jan 28, 2020 at 6:17 AM Octolus Development <admin@octolus.net> wrote: The problem is that they are spoofing our IP, to millions of IP's running port 80. Making upstream providers filter it is quite difficult, i don't know all the upstream providers are used. The main problem is honestly services that reports SYN_RECV as Port Flood, but there isn't much one can do about misconfigured firewalls.I am sure there is a decent amount of honeypots on the internet acting the same way, resulting us (the victims of the attack) getting blacklisted for 'sending' attacks. On 28.01.2020 05:50:14, "Dobbins, Roland" <roland.dobbins@netscout.com> wrote: On Jan 28, 2020, at 11:40, Dobbins, Roland <Roland.Dobbins@netscout.com> wrote: And even if his network weren't on the receiving end of a reflection/amplification attack, OP could still see backscatter, as Jared indicated. In point of fact, if the traffic was low-volume, this might in fact be what he was seeing. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
I'm a bit confused as I thought it was the other way around. No big deal though. So these SYN don't have options which is not normal today. It was in the previous millenium. You should see more options. What you can do is filter SYN based on packet length. 54 bytes is your signature here. The hacker is using hping3 or some basic rudimentary tools. Cheers Jean On 2020-01-28 16:41, Octolus Development wrote:
Yes, my server would then respond with RST.
Screenshot: https://i.imgur.com/ZVti2yY.png
We've blocked outgoing RST, 136.244.67.19 was our test server.
But even if the ip is not even exposed to the internet, services will blacklist us. Even if we don't respond, and block every request from the internet incoming & outgoing.
On 28.01.2020 22:36:18, "Jean | ddostest.me via NANOG" <nanog@nanog.org> wrote:
But you do receive the SYN/ACK?
The way to open a TCP socket is the 3 way handshake. Sorry to write that here... I feel it's useless.
1. SYN
2. SYN/ACK
3. ACK
Step 1: So hackers spoof the original SYN with your source IP of your network.
Step 2: You should then receive those SYN/ACK packets with your network as the dst ip and SONY as the src ip. Can you catch a few and post the TCP flags that you see please? (This is step 2)
You don't need sony or imperva for that. Just a sniffer at the right place in your network. You won't block anything, but we should see something very interesting that will help you fix this.
If it is happening like you are describing, you should see those packets and you should be able to capture them.
No worries if you can't.
Jean
On 2020-01-28 11:31, Octolus Development wrote:
I have tried numerous of times to reach out to Imperva.
Imperva said Sony have to contact them & said they cannot help me because I am not a customer of theirs. Something Sony will not do. Sony simply stopped responding my emails after some time.
But yes you are right.
My IP's are being spoofed, spoofing SYN requests to hundreds of thousands of web servers. Which then results in a blacklist, that Imperva uses.. which prevents me and my clients from accessing Sony's services.. because they use Imperva.
On 28.01.2020 17:29:12, Tom Beecher <beecher@beecher.cc> wrote:
Trying to summarize here, this convo has been a bit disjointed.
Is this an accurate summary?
- The malicious traffic with spoofed sources is targeting multiple different destinations. - The aggregate of all those flows is causing Impervia to flag your IP range as a bad actor. - Sony uses Impervia blacklists, and since Impervia has flagged your space as bad, Sony is blocking you.
If that is true, my advice would be to go right to Impervia. Explain the situation, and ask for their assistance in identifying and or/reaching out to the networks that they are detecting this spoofed traffic coming from. The backscatter, as Jared said earlier, could probably help you a bit too, but Impervia should be willing to assist. It's in their best interests to not have false positives, but who knows.
On Tue, Jan 28, 2020 at 6:17 AM Octolus Development <admin@octolus.net <mailto:admin@octolus.net>> wrote:
The problem is that they are spoofing our IP, to millions of IP's running port 80. Making upstream providers filter it is quite difficult, i don't know all the upstream providers are used.
The main problem is honestly services that reports SYN_RECV as Port Flood, but there isn't much one can do about misconfigured firewalls.I am sure there is a decent amount of honeypots on the internet acting the same way, resulting us (the victims of the attack) getting blacklisted for 'sending' attacks.
On 28.01.2020 05:50:14, "Dobbins, Roland" <roland.dobbins@netscout.com <mailto:roland.dobbins@netscout.com>> wrote:
On Jan 28, 2020, at 11:40, Dobbins, Roland <Roland.Dobbins@netscout.com <mailto:Roland.Dobbins@netscout.com>> wrote:
And even if his network weren't on the receiving end of a reflection/amplification attack, OP could still see backscatter, as Jared indicated.
In point of fact, if the traffic was low-volume, this might in fact be what he was seeing.
--------------------------------------------
Roland Dobbins <roland.dobbins@netscout.com <mailto:roland.dobbins@netscout.com>>
On Jan 28, 2020, at 04:12, Octolus Development <admin@octolus.net> wrote: I don't have an exact timestamp, because the attacks are really difficult to see as well. If you implement an open-source flow telemetry collection system & export flow telemetry from your edge routers to it, this becomes trivial. See this .pdf preso (it's my standard telemetry preso): <https://app.box.com/s/mnshn99c13uekrggy99b> [Full disclosure: I work for a commercial vendor of such systems.] -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
participants (17)
-
Ben Cannon -
Damian Menscher -
Dobbins, Roland -
Hugo Slabbert -
Jared Mauch -
Jean | ddostest.me -
Josh Luthman -
Keith Medcalf -
Lukas Tribus -
Mark Milhollan -
Mike Hammett -
Octolus Development -
Radu-Adrian Feurdean -
Tom Beecher -
Tony Wicks -
Töma Gavrichenkov -
xanonyws