Hello, We found out recently that we are on the HostingProviderIPList (found here https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-group...) at AWS and it's affecting our customers' access to various websites. We are a datacenter, and a hosting provider, but we have plenty of enterprise customers with eyeballs. We're finding it difficult to find a technical contact that we can reach since we're not an AWS customer. Does anyone have a contact or advice on a solution? Thank you, Justin H.
Justin H. wrote:
Hello,
We found out recently that we are on the HostingProviderIPList (found here https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-group...) at AWS and it's affecting our customers' access to various websites. We are a datacenter, and a hosting provider, but we have plenty of enterprise customers with eyeballs.
We're finding it difficult to find a technical contact that we can reach since we're not an AWS customer. Does anyone have a contact or advice on a solution? Sadly we're not getting any traction from standard AWS support, and end users of the WAF list like Reddit and Eventbrite are refusing to whitelist anyone. Does anyone have any AWS contacts that might be able to assist? Our enterprise customers are becoming more and more impacted.
Justin H.
The whole situation with these WAF as a service setups is a nightmare for the affected (afflicted) parties. I saw this problem from both sides when I was at Akamai. It’s not great from the service provider side, but it’s an absolute shit show for anyone on the wrong side of a block. There’s no accountability or process for redress of errors whatsoever. The impacted party isn’t a customer of the WAF publisher, so they cant get any traction there. The WAF subscriber blindly applies the WAF and it’s virtually impossible to track down anyone there who even knows that they subscribe to such a thing, let alone get them to take useful action. Best of luck. The only thing I saw that worked while I was at Akamai was a few entities subscribed to the WAF service and then complained about getting blocked from their own web sites. Since they were then Akamai WAF customers, they could get Akamai to take action. Crazy. Owen
On Feb 16, 2024, at 09:19, Justin H. <justindh.ml@gmail.com> wrote:
Justin H. wrote:
Hello,
We found out recently that we are on the HostingProviderIPList (found here https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-group...) at AWS and it's affecting our customers' access to various websites. We are a datacenter, and a hosting provider, but we have plenty of enterprise customers with eyeballs.
We're finding it difficult to find a technical contact that we can reach since we're not an AWS customer. Does anyone have a contact or advice on a solution? Sadly we're not getting any traction from standard AWS support, and end users of the WAF list like Reddit and Eventbrite are refusing to whitelist anyone. Does anyone have any AWS contacts that might be able to assist? Our enterprise customers are becoming more and more impacted.
Justin H.
That matches my experience with these types of problems in the past. Especially when the end-users don't have a process for white-listing. We actually got a response from one WAF user to "connect to another network to log in, then you should be able to use the site, because it's just the login page that's protected". I am working with someone off-list, so I have hope this can be resolved without account gymnastics. :) Justin H. Owen DeLong wrote:
The whole situation with these WAF as a service setups is a nightmare for the affected (afflicted) parties.
I saw this problem from both sides when I was at Akamai. It’s not great from the service provider side, but it’s an absolute shit show for anyone on the wrong side of a block. There’s no accountability or process for redress of errors whatsoever. The impacted party isn’t a customer of the WAF publisher, so they cant get any traction there. The WAF subscriber blindly applies the WAF and it’s virtually impossible to track down anyone there who even knows that they subscribe to such a thing, let alone get them to take useful action.
Best of luck. The only thing I saw that worked while I was at Akamai was a few entities subscribed to the WAF service and then complained about getting blocked from their own web sites. Since they were then Akamai WAF customers, they could get Akamai to take action.
Crazy.
Owen
On Feb 16, 2024, at 09:19, Justin H. <justindh.ml@gmail.com> wrote:
Justin H. wrote:
Hello,
We found out recently that we are on the HostingProviderIPList (found here https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-group...) at AWS and it's affecting our customers' access to various websites. We are a datacenter, and a hosting provider, but we have plenty of enterprise customers with eyeballs.
We're finding it difficult to find a technical contact that we can reach since we're not an AWS customer. Does anyone have a contact or advice on a solution? Sadly we're not getting any traction from standard AWS support, and end users of the WAF list like Reddit and Eventbrite are refusing to whitelist anyone. Does anyone have any AWS contacts that might be able to assist? Our enterprise customers are becoming more and more impacted.
Justin H.
This is terrible advice, but you might need another netblock for the eyeballs. Possibly a small one with enterprise NAT, but something outside the AWS list ranges... -George On Mon, Feb 19, 2024 at 7:35 PM Justin H. <justindh.ml@gmail.com> wrote:
That matches my experience with these types of problems in the past. Especially when the end-users don't have a process for white-listing. We actually got a response from one WAF user to "connect to another network to log in, then you should be able to use the site, because it's just the login page that's protected".
I am working with someone off-list, so I have hope this can be resolved without account gymnastics. :)
Justin H.
The whole situation with these WAF as a service setups is a nightmare for the affected (afflicted) parties.
I saw this problem from both sides when I was at Akamai. It’s not great from the service provider side, but it’s an absolute shit show for anyone on the wrong side of a block. There’s no accountability or process for redress of errors whatsoever. The impacted party isn’t a customer of the WAF publisher, so they cant get any traction there. The WAF subscriber blindly applies the WAF and it’s virtually impossible to track down anyone
Owen DeLong wrote: there who even knows that they subscribe to such a thing, let alone get them to take useful action.
Best of luck. The only thing I saw that worked while I was at Akamai
was a few entities subscribed to the WAF service and then complained about getting blocked from their own web sites. Since they were then Akamai WAF customers, they could get Akamai to take action.
Crazy.
Owen
On Feb 16, 2024, at 09:19, Justin H. <justindh.ml@gmail.com> wrote:
Justin H. wrote:
Hello,
We found out recently that we are on the HostingProviderIPList (found
here https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-group...) at AWS and it's affecting our customers' access to various websites. We are a datacenter, and a hosting provider, but we have plenty of enterprise customers with eyeballs.
We're finding it difficult to find a technical contact that we can
reach since we're not an AWS customer. Does anyone have a contact or advice on a solution? Sadly we're not getting any traction from standard AWS support, and end users of the WAF list like Reddit and Eventbrite are refusing to whitelist anyone. Does anyone have any AWS contacts that might be able to assist? Our enterprise customers are becoming more and more impacted.
Justin H.
-- -george william herbert george.herbert@gmail.com
There are other WAF lists available on AWS besides their native one. Ones that have support.
On Feb 20, 2024, at 16:18, George Herbert <george.herbert@gmail.com> wrote:
This is terrible advice, but you might need another netblock for the eyeballs. Possibly a small one with enterprise NAT, but something outside the AWS list ranges...
-George
On Mon, Feb 19, 2024 at 7:35 PM Justin H. <justindh.ml@gmail.com <mailto:justindh.ml@gmail.com>> wrote:
That matches my experience with these types of problems in the past. Especially when the end-users don't have a process for white-listing. We actually got a response from one WAF user to "connect to another network to log in, then you should be able to use the site, because it's just the login page that's protected".
I am working with someone off-list, so I have hope this can be resolved without account gymnastics. :)
Justin H.
Owen DeLong wrote:
The whole situation with these WAF as a service setups is a nightmare for the affected (afflicted) parties.
I saw this problem from both sides when I was at Akamai. It’s not great from the service provider side, but it’s an absolute shit show for anyone on the wrong side of a block. There’s no accountability or process for redress of errors whatsoever. The impacted party isn’t a customer of the WAF publisher, so they cant get any traction there. The WAF subscriber blindly applies the WAF and it’s virtually impossible to track down anyone there who even knows that they subscribe to such a thing, let alone get them to take useful action.
Best of luck. The only thing I saw that worked while I was at Akamai was a few entities subscribed to the WAF service and then complained about getting blocked from their own web sites. Since they were then Akamai WAF customers, they could get Akamai to take action.
Crazy.
Owen
On Feb 16, 2024, at 09:19, Justin H. <justindh.ml@gmail.com <mailto:justindh.ml@gmail.com>> wrote:
Justin H. wrote:
Hello,
We found out recently that we are on the HostingProviderIPList (found here https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-group...) at AWS and it's affecting our customers' access to various websites. We are a datacenter, and a hosting provider, but we have plenty of enterprise customers with eyeballs.
We're finding it difficult to find a technical contact that we can reach since we're not an AWS customer. Does anyone have a contact or advice on a solution? Sadly we're not getting any traction from standard AWS support, and end users of the WAF list like Reddit and Eventbrite are refusing to whitelist anyone. Does anyone have any AWS contacts that might be able to assist? Our enterprise customers are becoming more and more impacted.
Justin H.
-- -george william herbert george.herbert@gmail.com <mailto:george.herbert@gmail.com>
Unfortunately, the victim doesn’t chose the WAF list, the web site that is causing the victim grief chooses the WAF list. Owen
On Feb 20, 2024, at 14:15, joel@joelesler.net wrote:
There are other WAF lists available on AWS besides their native one. Ones that have support.
On Feb 20, 2024, at 16:18, George Herbert <george.herbert@gmail.com> wrote:
This is terrible advice, but you might need another netblock for the eyeballs. Possibly a small one with enterprise NAT, but something outside the AWS list ranges...
-George
On Mon, Feb 19, 2024 at 7:35 PM Justin H. <justindh.ml@gmail.com <mailto:justindh.ml@gmail.com>> wrote:
That matches my experience with these types of problems in the past. Especially when the end-users don't have a process for white-listing. We actually got a response from one WAF user to "connect to another network to log in, then you should be able to use the site, because it's just the login page that's protected".
I am working with someone off-list, so I have hope this can be resolved without account gymnastics. :)
Justin H.
Owen DeLong wrote:
The whole situation with these WAF as a service setups is a nightmare for the affected (afflicted) parties.
I saw this problem from both sides when I was at Akamai. It’s not great from the service provider side, but it’s an absolute shit show for anyone on the wrong side of a block. There’s no accountability or process for redress of errors whatsoever. The impacted party isn’t a customer of the WAF publisher, so they cant get any traction there. The WAF subscriber blindly applies the WAF and it’s virtually impossible to track down anyone there who even knows that they subscribe to such a thing, let alone get them to take useful action.
Best of luck. The only thing I saw that worked while I was at Akamai was a few entities subscribed to the WAF service and then complained about getting blocked from their own web sites. Since they were then Akamai WAF customers, they could get Akamai to take action.
Crazy.
Owen
On Feb 16, 2024, at 09:19, Justin H. <justindh.ml@gmail.com <mailto:justindh.ml@gmail.com>> wrote:
Justin H. wrote:
Hello,
We found out recently that we are on the HostingProviderIPList (found here https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-group...) at AWS and it's affecting our customers' access to various websites. We are a datacenter, and a hosting provider, but we have plenty of enterprise customers with eyeballs.
We're finding it difficult to find a technical contact that we can reach since we're not an AWS customer. Does anyone have a contact or advice on a solution? Sadly we're not getting any traction from standard AWS support, and end users of the WAF list like Reddit and Eventbrite are refusing to whitelist anyone. Does anyone have any AWS contacts that might be able to assist? Our enterprise customers are becoming more and more impacted.
Justin H.
-- -george william herbert george.herbert@gmail.com <mailto:george.herbert@gmail.com>
There must be a reason why the web site chooses the WAF list to block out the victim? If so why not the victim to contact the website to request them to talk to the waf list provider to remove victim ip block? Edy From: NANOG <nanog-bounces+email=edylie.net@nanog.org> On Behalf Of Owen DeLong via NANOG Sent: Wednesday, 21 February 2024 7:04 am To: joel@joelesler.net Cc: NANOG <nanog@nanog.org> Subject: Re: AWS WAF list Unfortunately, the victim doesn’t chose the WAF list, the web site that is causing the victim grief chooses the WAF list. Owen On Feb 20, 2024, at 14:15, joel@joelesler.net <mailto:joel@joelesler.net> wrote: There are other WAF lists available on AWS besides their native one. Ones that have support. On Feb 20, 2024, at 16:18, George Herbert <george.herbert@gmail.com <mailto:george.herbert@gmail.com> > wrote: This is terrible advice, but you might need another netblock for the eyeballs. Possibly a small one with enterprise NAT, but something outside the AWS list ranges... -George On Mon, Feb 19, 2024 at 7:35 PM Justin H. <justindh.ml@gmail.com <mailto:justindh.ml@gmail.com> > wrote: That matches my experience with these types of problems in the past. Especially when the end-users don't have a process for white-listing. We actually got a response from one WAF user to "connect to another network to log in, then you should be able to use the site, because it's just the login page that's protected". I am working with someone off-list, so I have hope this can be resolved without account gymnastics. :) Justin H. Owen DeLong wrote:
The whole situation with these WAF as a service setups is a nightmare for the affected (afflicted) parties.
I saw this problem from both sides when I was at Akamai. It’s not great from the service provider side, but it’s an absolute shit show for anyone on the wrong side of a block. There’s no accountability or process for redress of errors whatsoever. The impacted party isn’t a customer of the WAF publisher, so they cant get any traction there. The WAF subscriber blindly applies the WAF and it’s virtually impossible to track down anyone there who even knows that they subscribe to such a thing, let alone get them to take useful action.
Best of luck. The only thing I saw that worked while I was at Akamai was a few entities subscribed to the WAF service and then complained about getting blocked from their own web sites. Since they were then Akamai WAF customers, they could get Akamai to take action.
Crazy.
Owen
On Feb 16, 2024, at 09:19, Justin H. <justindh.ml@gmail.com <mailto:justindh.ml@gmail.com> > wrote:
Justin H. wrote:
Hello,
We found out recently that we are on the HostingProviderIPList (found here https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-group...) at AWS and it's affecting our customers' access to various websites. We are a datacenter, and a hosting provider, but we have plenty of enterprise customers with eyeballs.
We're finding it difficult to find a technical contact that we can reach since we're not an AWS customer. Does anyone have a contact or advice on a solution? Sadly we're not getting any traction from standard AWS support, and end users of the WAF list like Reddit and Eventbrite are refusing to whitelist anyone. Does anyone have any AWS contacts that might be able to assist? Our enterprise customers are becoming more and more impacted.
Justin H.
-- -george william herbert george.herbert@gmail.com <mailto:george.herbert@gmail.com>
and it's affecting our customers' access to various ===>> websites.<<===
On Tue, Feb 20, 2024 at 6:15 PM Pui Ee Luun Edylie <email@edylie.net> wrote:
There must be a reason why the web site chooses the WAF list to block out the victim? If so why not the victim to contact the website to request them to talk to the waf list provider to remove victim ip block?
Edy
*From:* NANOG <nanog-bounces+email=edylie.net@nanog.org> *On Behalf Of *Owen DeLong via NANOG *Sent:* Wednesday, 21 February 2024 7:04 am *To:* joel@joelesler.net *Cc:* NANOG <nanog@nanog.org> *Subject:* Re: AWS WAF list
Unfortunately, the victim doesn’t chose the WAF list, the web site that is causing the victim grief chooses the WAF list.
Owen
On Feb 20, 2024, at 14:15, joel@joelesler.net wrote:
There are other WAF lists available on AWS besides their native one. Ones that have support.
On Feb 20, 2024, at 16:18, George Herbert <george.herbert@gmail.com> wrote:
This is terrible advice, but you might need another netblock for the eyeballs. Possibly a small one with enterprise NAT, but something outside the AWS list ranges...
-George
On Mon, Feb 19, 2024 at 7:35 PM Justin H. <justindh.ml@gmail.com> wrote:
That matches my experience with these types of problems in the past. Especially when the end-users don't have a process for white-listing. We actually got a response from one WAF user to "connect to another network to log in, then you should be able to use the site, because it's just the login page that's protected".
I am working with someone off-list, so I have hope this can be resolved without account gymnastics. :)
Justin H.
The whole situation with these WAF as a service setups is a nightmare for the affected (afflicted) parties.
I saw this problem from both sides when I was at Akamai. It’s not great from the service provider side, but it’s an absolute shit show for anyone on the wrong side of a block. There’s no accountability or process for redress of errors whatsoever. The impacted party isn’t a customer of the WAF publisher, so they cant get any traction there. The WAF subscriber blindly applies the WAF and it’s virtually impossible to track down anyone
Owen DeLong wrote: there who even knows that they subscribe to such a thing, let alone get them to take useful action.
Best of luck. The only thing I saw that worked while I was at Akamai
was a few entities subscribed to the WAF service and then complained about getting blocked from their own web sites. Since they were then Akamai WAF customers, they could get Akamai to take action.
Crazy.
Owen
On Feb 16, 2024, at 09:19, Justin H. <justindh.ml@gmail.com> wrote:
Justin H. wrote:
Hello,
We found out recently that we are on the HostingProviderIPList (found
here https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-group...) at AWS and it's affecting our customers' access to various websites. We are a datacenter, and a hosting provider, but we have plenty of enterprise customers with eyeballs.
We're finding it difficult to find a technical contact that we can
reach since we're not an AWS customer. Does anyone have a contact or advice on a solution? Sadly we're not getting any traction from standard AWS support, and end users of the WAF list like Reddit and Eventbrite are refusing to whitelist anyone. Does anyone have any AWS contacts that might be able to assist? Our enterprise customers are becoming more and more impacted.
Justin H.
--
-george william herbert george.herbert@gmail.com
Here’s the usual problem: Victim is a customer Q of ISP A. WAF provided by provider X is chosen by website Y. A has no business relationship with X or Y. A’s requests to X are rebuffed because A is not a customer of X. A’s requests to Y are rebuffed because A is not a customer of Y. A tells Q to reach out to Y directly and attempts to explain the situation to Q. Q does not understand half of the words coming out of A’s mouth and asks “can’t you just fix this?” Sucks for everyone involved. Sucks most for Q, second most for A because A often loses customer Q over the issue (where Q is often multiplied to several customers). Sucks for Y (to a much lesser degree) because some fraction of customer Qs will be smart enough to be angry with Y. OK, maybe it doesn’t really suck much for X, but they do get a bunch of calls they have to blow off, so it still costs them a little bit of money. Owen
On Feb 20, 2024, at 16:05, Tom Beecher <beecher@beecher.cc> wrote:
and it's affecting our customers' access to various ===>> websites.<<===
On Tue, Feb 20, 2024 at 6:15 PM Pui Ee Luun Edylie <email@edylie.net <mailto:email@edylie.net>> wrote:
There must be a reason why the web site chooses the WAF list to block out the victim? If so why not the victim to contact the website to request them to talk to the waf list provider to remove victim ip block?
Edy
From: NANOG <nanog-bounces+email=edylie.net@nanog.org <mailto:edylie.net@nanog.org>> On Behalf Of Owen DeLong via NANOG Sent: Wednesday, 21 February 2024 7:04 am To: joel@joelesler.net <mailto:joel@joelesler.net> Cc: NANOG <nanog@nanog.org <mailto:nanog@nanog.org>> Subject: Re: AWS WAF list
Unfortunately, the victim doesn’t chose the WAF list, the web site that is causing the victim grief chooses the WAF list.
Owen
On Feb 20, 2024, at 14:15, joel@joelesler.net <mailto:joel@joelesler.net> wrote:
There are other WAF lists available on AWS besides their native one. Ones that have support.
On Feb 20, 2024, at 16:18, George Herbert <george.herbert@gmail.com <mailto:george.herbert@gmail.com>> wrote:
This is terrible advice, but you might need another netblock for the eyeballs. Possibly a small one with enterprise NAT, but something outside the AWS list ranges...
-George
On Mon, Feb 19, 2024 at 7:35 PM Justin H. <justindh.ml@gmail.com <mailto:justindh.ml@gmail.com>> wrote:
That matches my experience with these types of problems in the past. Especially when the end-users don't have a process for white-listing. We actually got a response from one WAF user to "connect to another network to log in, then you should be able to use the site, because it's just the login page that's protected".
I am working with someone off-list, so I have hope this can be resolved without account gymnastics. :)
Justin H.
Owen DeLong wrote:
The whole situation with these WAF as a service setups is a nightmare for the affected (afflicted) parties.
I saw this problem from both sides when I was at Akamai. It’s not great from the service provider side, but it’s an absolute shit show for anyone on the wrong side of a block. There’s no accountability or process for redress of errors whatsoever. The impacted party isn’t a customer of the WAF publisher, so they cant get any traction there. The WAF subscriber blindly applies the WAF and it’s virtually impossible to track down anyone there who even knows that they subscribe to such a thing, let alone get them to take useful action.
Best of luck. The only thing I saw that worked while I was at Akamai was a few entities subscribed to the WAF service and then complained about getting blocked from their own web sites. Since they were then Akamai WAF customers, they could get Akamai to take action.
Crazy.
Owen
On Feb 16, 2024, at 09:19, Justin H. <justindh.ml@gmail.com <mailto:justindh.ml@gmail.com>> wrote:
Justin H. wrote:
Hello,
We found out recently that we are on the HostingProviderIPList (found here https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-group...) at AWS and it's affecting our customers' access to various websites. We are a datacenter, and a hosting provider, but we have plenty of enterprise customers with eyeballs.
We're finding it difficult to find a technical contact that we can reach since we're not an AWS customer. Does anyone have a contact or advice on a solution? Sadly we're not getting any traction from standard AWS support, and end users of the WAF list like Reddit and Eventbrite are refusing to whitelist anyone. Does anyone have any AWS contacts that might be able to assist? Our enterprise customers are becoming more and more impacted.
Justin H.
--
-george william herbert george.herbert@gmail.com <mailto:george.herbert@gmail.com>
participants (6)
-
George Herbert -
joel＠joelesler.net -
Justin H. -
Owen DeLong -
Pui Ee Luun Edylie -
Tom Beecher