Having Geo-IP issues in the past week or so? Digital Elements wet the bed…
If any eyeball networks are having geo-ip issues that started a week or so ago for their customers trying to use HBO, Hulu, Netflix or any others that use Digital Elements for their geo-ip, be sure you make Digital Elements aware. Digital Elements ran a process update last week that screwed up a bunch of netblocks. See ticket verbatim below. And to top it off, they are holding my customers hostage until a week from today when they run the corrective update. Hi Mike, While refining our processes to improve accuracy, some of our changes inadvertently over-corrected existing updates we had in place. We have noticed these and are correcting them currently. This unfortunately was one of the overridden updates. Thank you. Mike Lyon Aug 22, 2025, 11:00 AM EDT Do we know why it changed? On Aug 22, 2025, at 05:03, Data Management <ip-data@digitalenvoy.net> wrote: *redacted* (Digital Element) Aug 22, 2025, 8:03 AM EDT Hi Mike, We apologize for the inconvenience. We will update this IP range back to the US for our release next Friday, August 29. Thank you. ———————— Fun times… -Mike
Thanks for closing the loop. I'm curious why it takes a week to adjust one IP block. It's frustrating as a provider to have no ability to fix it and get all of the angry phone calls. On Fri, Aug 22, 2025, 6:14 PM Mike Lyon via NANOG <nanog@lists.nanog.org> wrote:
If any eyeball networks are having geo-ip issues that started a week or so ago for their customers trying to use HBO, Hulu, Netflix or any others that use Digital Elements for their geo-ip, be sure you make Digital Elements aware.
Digital Elements ran a process update last week that screwed up a bunch of netblocks. See ticket verbatim below.
And to top it off, they are holding my customers hostage until a week from today when they run the corrective update.
Hi Mike,
While refining our processes to improve accuracy, some of our changes inadvertently over-corrected existing updates we had in place. We have noticed these and are correcting them currently. This unfortunately was one of the overridden updates.
Thank you. Mike Lyon Aug 22, 2025, 11:00 AM EDT
Do we know why it changed?
On Aug 22, 2025, at 05:03, Data Management <ip-data@digitalenvoy.net> wrote:
*redacted* (Digital Element) Aug 22, 2025, 8:03 AM EDT
Hi Mike,
We apologize for the inconvenience. We will update this IP range back to the US for our release next Friday, August 29.
Thank you.
————————
Fun times… -Mike
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/UORMFGBQ...
I agree with you, Josh. Only had one complaint today for Amazon Prime, my office happens to be in the same subnet and it worked fine. I also haven’t heard from any other customers, very unusual. Mike, thanks for bringing this to our attention, Sir!
On Aug 22, 2025, at 18:42, Josh Luthman via NANOG <nanog@lists.nanog.org> wrote:
Thanks for closing the loop.
I'm curious why it takes a week to adjust one IP block. It's frustrating as a provider to have no ability to fix it and get all of the angry phone calls.
On Fri, Aug 22, 2025, 6:14 PM Mike Lyon via NANOG <nanog@lists.nanog.org> wrote:
If any eyeball networks are having geo-ip issues that started a week or so ago for their customers trying to use HBO, Hulu, Netflix or any others that use Digital Elements for their geo-ip, be sure you make Digital Elements aware.
Digital Elements ran a process update last week that screwed up a bunch of netblocks. See ticket verbatim below.
And to top it off, they are holding my customers hostage until a week from today when they run the corrective update.
Hi Mike,
While refining our processes to improve accuracy, some of our changes inadvertently over-corrected existing updates we had in place. We have noticed these and are correcting them currently. This unfortunately was one of the overridden updates.
Thank you. Mike Lyon Aug 22, 2025, 11:00 AM EDT
Do we know why it changed?
On Aug 22, 2025, at 05:03, Data Management <ip-data@digitalenvoy.net> wrote:
*redacted* (Digital Element) Aug 22, 2025, 8:03 AM EDT
Hi Mike,
We apologize for the inconvenience. We will update this IP range back to the US for our release next Friday, August 29.
Thank you.
————————
Fun times… -Mike
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/UORMFGBQ...
NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/XUCZBEUN...
I’ve had a handful of customers complain this week. Have had to move them to a different subnet temporarily… -Mike
On Aug 22, 2025, at 19:24, Jason Canady via NANOG <nanog@lists.nanog.org> wrote:
I agree with you, Josh. Only had one complaint today for Amazon Prime, my office happens to be in the same subnet and it worked fine. I also haven’t heard from any other customers, very unusual.
Mike, thanks for bringing this to our attention, Sir!
On Aug 22, 2025, at 18:42, Josh Luthman via NANOG <nanog@lists.nanog.org> wrote:
Thanks for closing the loop.
I'm curious why it takes a week to adjust one IP block. It's frustrating as a provider to have no ability to fix it and get all of the angry phone calls.
On Fri, Aug 22, 2025, 6:14 PM Mike Lyon via NANOG <nanog@lists.nanog.org> wrote:
If any eyeball networks are having geo-ip issues that started a week or so ago for their customers trying to use HBO, Hulu, Netflix or any others that use Digital Elements for their geo-ip, be sure you make Digital Elements aware.
Digital Elements ran a process update last week that screwed up a bunch of netblocks. See ticket verbatim below.
And to top it off, they are holding my customers hostage until a week from today when they run the corrective update.
Hi Mike,
While refining our processes to improve accuracy, some of our changes inadvertently over-corrected existing updates we had in place. We have noticed these and are correcting them currently. This unfortunately was one of the overridden updates.
Thank you. Mike Lyon Aug 22, 2025, 11:00 AM EDT
Do we know why it changed?
On Aug 22, 2025, at 05:03, Data Management <ip-data@digitalenvoy.net> wrote:
*redacted* (Digital Element) Aug 22, 2025, 8:03 AM EDT
Hi Mike,
We apologize for the inconvenience. We will update this IP range back to the US for our release next Friday, August 29.
Thank you.
————————
Fun times… -Mike
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/UORMFGBQ...
NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/XUCZBEUN...
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/EH2KZ3FO...
I'm curious why it takes a week to adjust one IP block.
Content Provider pays the Geolocation Provider for access to their database. If the Content Provider is fine with the Geolocation Provider only updating their database weekly, then that's what they will do. The Geo Provider has no incentive to do updates quicker than the people paying them require them. By extension, the Content Provider only cares if enough users are impacted by the problem that they NEED to care. The entire geo-ip ecosystem is hot garbage. On Fri, Aug 22, 2025 at 6:42 PM Josh Luthman via NANOG < nanog@lists.nanog.org> wrote:
Thanks for closing the loop.
I'm curious why it takes a week to adjust one IP block. It's frustrating as a provider to have no ability to fix it and get all of the angry phone calls.
On Fri, Aug 22, 2025, 6:14 PM Mike Lyon via NANOG <nanog@lists.nanog.org> wrote:
If any eyeball networks are having geo-ip issues that started a week or so ago for their customers trying to use HBO, Hulu, Netflix or any others that use Digital Elements for their geo-ip, be sure you make Digital Elements aware.
Digital Elements ran a process update last week that screwed up a bunch of netblocks. See ticket verbatim below.
And to top it off, they are holding my customers hostage until a week from today when they run the corrective update.
Hi Mike,
While refining our processes to improve accuracy, some of our changes inadvertently over-corrected existing updates we had in place. We have noticed these and are correcting them currently. This unfortunately was one of the overridden updates.
Thank you. Mike Lyon Aug 22, 2025, 11:00 AM EDT
Do we know why it changed?
On Aug 22, 2025, at 05:03, Data Management <ip-data@digitalenvoy.net> wrote:
*redacted* (Digital Element) Aug 22, 2025, 8:03 AM EDT
Hi Mike,
We apologize for the inconvenience. We will update this IP range back to the US for our release next Friday, August 29.
Thank you.
————————
Fun times… -Mike
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/UORMFGBQ... _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/XUCZBEUN...
On Aug 23, 2025, at 17:25, Tom Beecher via NANOG <nanog@lists.nanog.org> wrote:
I'm curious why it takes a week to adjust one IP block.
Content Provider pays the Geolocation Provider for access to their database. If the Content Provider is fine with the Geolocation Provider only updating their database weekly, then that's what they will do. The Geo Provider has no incentive to do updates quicker than the people paying them require them. By extension, the Content Provider only cares if enough users are impacted by the problem that they NEED to care.
The entire geo-ip ecosystem is hot garbage.
This… 100%!! Hot piece of shit! -Mike
On Fri, Aug 22, 2025 at 6:42 PM Josh Luthman via NANOG < nanog@lists.nanog.org> wrote:
Thanks for closing the loop.
I'm curious why it takes a week to adjust one IP block. It's frustrating as a provider to have no ability to fix it and get all of the angry phone calls.
On Fri, Aug 22, 2025, 6:14 PM Mike Lyon via NANOG <nanog@lists.nanog.org> wrote:
If any eyeball networks are having geo-ip issues that started a week or so ago for their customers trying to use HBO, Hulu, Netflix or any others that use Digital Elements for their geo-ip, be sure you make Digital Elements aware.
Digital Elements ran a process update last week that screwed up a bunch of netblocks. See ticket verbatim below.
And to top it off, they are holding my customers hostage until a week from today when they run the corrective update.
Hi Mike,
While refining our processes to improve accuracy, some of our changes inadvertently over-corrected existing updates we had in place. We have noticed these and are correcting them currently. This unfortunately was one of the overridden updates.
Thank you. Mike Lyon Aug 22, 2025, 11:00 AM EDT
Do we know why it changed?
On Aug 22, 2025, at 05:03, Data Management <ip-data@digitalenvoy.net> wrote:
*redacted* (Digital Element) Aug 22, 2025, 8:03 AM EDT
Hi Mike,
We apologize for the inconvenience. We will update this IP range back to the US for our release next Friday, August 29.
Thank you.
————————
Fun times… -Mike
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/UORMFGBQ... _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/XUCZBEUN...
NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/BGDX2K5P...
participants (4)
-
Jason Canady -
Josh Luthman -
Mike Lyon -
Tom Beecher