Unusually High traffic from Akamai/Oracle - public-yum.oracle.com
Hi, Since I'm not a customer of either organization, I'm reaching out to NANOG for a contact and perhaps others may also be experiencing similar symptoms over the past 3-4 weeks. The situation appears to be that customers of ours have Oracle Linux and when they attempt to download updates, their traffic goes through the roof for hours on end. While researching this phenomenon, I found this discussion which coincides with the traffic I've seen, however there is no mention of excessive traffic resulting from this "corruption" nor have their been any additional reports: https://community.oracle.com/thread/4138810 Currently, I have two customer environments which are hitting about ~2Gb/s when normally their traffic levels are nearly zero. At first I thought it was an isolated incident but then we observed the same issue with another customer. All of this traffic is coming from 23.35.204.188:80, which belongs to Akamai. Since that's somewhat of a dead end, we examined the hosts which are requesting the data from Akamai and found that they are all Oracle Linux boxes and it's a yum process on Oracle Linux which appears to be repeatedly downloading the same content for hours on end: [root@xyzzy noc]# netstat -plutan | grep :80 tcp 0 0 172.16.122.112:14272 23.35.204.188:80 ESTABLISHED 58880/python [root@xyzzy noc]# ps auxww | grep python root 41015 0.0 0.3 401940 52044 ? S Apr30 0:02 /usr/bin/python2 /usr/share/system-config-lvm/system-config-lvm.py root 58880 59.7 1.0 479680 164140 ? R 18:24 27:18 /usr/bin/python /usr/share/PackageKit/helpers/yum/yumBackend.py get-updates none I can only assume that the data being downloaded is corrupt as this multiple hour download does not consume any disk space and because the file(s) are repeatedly downloaded, the logic behind the yum routines are also at fault for 1TB of I don't expect anyone at Akamai to reach out to me since they are simply the middle man here, but I'm hoping that someone at Oracle will because the cost to Oracle for Akamai to deliver this junk traffic is not zero and I have a hard time seeing how this issue is isolated to our network. I'd also be interested to hear from anyone else who has been seeing traffic spikes from public-yum.oracle.com. -James Stahr
latest yum oracle linux applied ok today fine, 153mb Have not seen looking content either Colin
On 9 May 2018, at 20:48, James Stahr <stahr@mailbag.com> wrote:
Hi,
Since I'm not a customer of either organization, I'm reaching out to NANOG for a contact and perhaps others may also be experiencing similar symptoms over the past 3-4 weeks. The situation appears to be that customers of ours have Oracle Linux and when they attempt to download updates, their traffic goes through the roof for hours on end. While researching this phenomenon, I found this discussion which coincides with the traffic I've seen, however there is no mention of excessive traffic resulting from this "corruption" nor have their been any additional reports:
https://community.oracle.com/thread/4138810
Currently, I have two customer environments which are hitting about ~2Gb/s when normally their traffic levels are nearly zero. At first I thought it was an isolated incident but then we observed the same issue with another customer. All of this traffic is coming from 23.35.204.188:80, which belongs to Akamai. Since that's somewhat of a dead end, we examined the hosts which are requesting the data from Akamai and found that they are all Oracle Linux boxes and it's a yum process on Oracle Linux which appears to be repeatedly downloading the same content for hours on end:
[root@xyzzy noc]# netstat -plutan | grep :80 tcp 0 0 172.16.122.112:14272 23.35.204.188:80 ESTABLISHED 58880/python [root@xyzzy noc]# ps auxww | grep python root 41015 0.0 0.3 401940 52044 ? S Apr30 0:02 /usr/bin/python2 /usr/share/system-config-lvm/system-config-lvm.py root 58880 59.7 1.0 479680 164140 ? R 18:24 27:18 /usr/bin/python /usr/share/PackageKit/helpers/yum/yumBackend.py get-updates none
I can only assume that the data being downloaded is corrupt as this multiple hour download does not consume any disk space and because the file(s) are repeatedly downloaded, the logic behind the yum routines are also at fault for 1TB of
I don't expect anyone at Akamai to reach out to me since they are simply the middle man here, but I'm hoping that someone at Oracle will because the cost to Oracle for Akamai to deliver this junk traffic is not zero and I have a hard time seeing how this issue is isolated to our network. I'd also be interested to hear from anyone else who has been seeing traffic spikes from public-yum.oracle.com.
-James Stahr
Hi James, I've forwarded your email to the yum.oracle.com team internally; they've acknowledged receipt and asked me to let the list know. Apologies for the delay - I only noticed this thread today. Kind regards, Jon On 09/05/18 20:48, James Stahr wrote:
Hi,
Since I'm not a customer of either organization, I'm reaching out to NANOG for a contact and perhaps others may also be experiencing similar symptoms over the past 3-4 weeks. The situation appears to be that customers of ours have Oracle Linux and when they attempt to download updates, their traffic goes through the roof for hours on end. While researching this phenomenon, I found this discussion which coincides with the traffic I've seen, however there is no mention of excessive traffic resulting from this "corruption" nor have their been any additional reports:
https://community.oracle.com/thread/4138810
Currently, I have two customer environments which are hitting about ~2Gb/s when normally their traffic levels are nearly zero. At first I thought it was an isolated incident but then we observed the same issue with another customer. All of this traffic is coming from 23.35.204.188:80, which belongs to Akamai. Since that's somewhat of a dead end, we examined the hosts which are requesting the data from Akamai and found that they are all Oracle Linux boxes and it's a yum process on Oracle Linux which appears to be repeatedly downloading the same content for hours on end:
[root@xyzzy noc]# netstat -plutan | grep :80 tcp 0 0 172.16.122.112:14272 23.35.204.188:80 ESTABLISHED 58880/python [root@xyzzy noc]# ps auxww | grep python root 41015 0.0 0.3 401940 52044 ? S Apr30 0:02 /usr/bin/python2 /usr/share/system-config-lvm/system-config-lvm.py root 58880 59.7 1.0 479680 164140 ? R 18:24 27:18 /usr/bin/python /usr/share/PackageKit/helpers/yum/yumBackend.py get-updates none
I can only assume that the data being downloaded is corrupt as this multiple hour download does not consume any disk space and because the file(s) are repeatedly downloaded, the logic behind the yum routines are also at fault for 1TB of
I don't expect anyone at Akamai to reach out to me since they are simply the middle man here, but I'm hoping that someone at Oracle will because the cost to Oracle for Akamai to deliver this junk traffic is not zero and I have a hard time seeing how this issue is isolated to our network. I'd also be interested to hear from anyone else who has been seeing traffic spikes from public-yum.oracle.com.
-James Stahr
--
participants (3)
-
Colin Johnston -
James Stahr -
Jonathan Roach