Looking for an Akamai contact, strange DoS traffic sourcing from Akamai sources
I'm looking for an Akamai contact to try and address a strange situation. We have multiple sites across the country that aggregate 56k dialup customers. Different sites are randomly experiencing inbound traffic spikes that are overwhelming the uplinks to our carriers, causing DoS situations. These spikes far exceed the bandwidth that could possibly be used by the number of dialup customers connected. We've been able to trace the source of the traffic to Akamai boxes, but so far have been unable to reach anyone at Akamai to discuss the situation. We're attempting to get payload information, but the traffic volume is making it slow going setting up packet captures at these sites remotely. Thanks in advance, Tom -- Thomas Beecher II Senior Network Administrator LocalNet Corp. CoreComm Internet Services tbeecher at localnet.com
I've received a couple of responses off list, and am now in touch with Akamai directly. I appreciate everyone's assistance. On 1/20/2011 4:04 PM, Tom Beecher wrote:
I'm looking for an Akamai contact to try and address a strange situation.
We have multiple sites across the country that aggregate 56k dialup customers. Different sites are randomly experiencing inbound traffic spikes that are overwhelming the uplinks to our carriers, causing DoS situations. These spikes far exceed the bandwidth that could possibly be used by the number of dialup customers connected. We've been able to trace the source of the traffic to Akamai boxes, but so far have been unable to reach anyone at Akamai to discuss the situation. We're attempting to get payload information, but the traffic volume is making it slow going setting up packet captures at these sites remotely.
Thanks in advance,
Tom
-- Thomas Beecher II Senior Network Administrator LocalNet Corp. CoreComm Internet Services tbeecher at localnet.com
I have a customer reporting the same thing. The traffic flood goes to offline modem bank IPs. So far, Akamai hasn't actually grasped what the problem is and says everything is fine. :( Luckily, most of the traffic (not all) is coming from my local cluster, so it's easier to monitor what's going on. Packet captures have shown the same packet being sent over and over, usually over 1400 bytes in size. Different floods may have different packets, but within a flood it's identical. I wouldn't think you'd have data prior to the 3-way, so I'm curious how the 3-way is being completed for the data to be sent. Jack On 1/20/2011 4:46 PM, Tom Beecher wrote:
I've received a couple of responses off list, and am now in touch with Akamai directly.
I appreciate everyone's assistance.
On 1/20/2011 4:04 PM, Tom Beecher wrote:
I'm looking for an Akamai contact to try and address a strange situation.
We have multiple sites across the country that aggregate 56k dialup customers. Different sites are randomly experiencing inbound traffic spikes that are overwhelming the uplinks to our carriers, causing DoS situations. These spikes far exceed the bandwidth that could possibly be used by the number of dialup customers connected. We've been able to trace the source of the traffic to Akamai boxes, but so far have been unable to reach anyone at Akamai to discuss the situation. We're attempting to get payload information, but the traffic volume is making it slow going setting up packet captures at these sites remotely.
Thanks in advance,
Tom
Once upon a time, Jack Bates <jbates@brightok.net> said:
I have a customer reporting the same thing. The traffic flood goes to offline modem bank IPs. So far, Akamai hasn't actually grasped what the problem is and says everything is fine. :(
<aol>me too</aol> I hadn't captured the traffic during one of the floods yet, but now that you mention it, I'm seeing spikes on my Akamai graphs at the same time as the spikes on the dialup graphs. I wonder if some Microsoft PPP update triggered an Akamai bug or some such (why else would it just be hitting dialups)? -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Jack- This is exactly what we're seeing. The Akamai server starts a retransmission flood aimed at a specific address randomly. We're seeing thousands of retransmissions of the same packet over and over again, same sequence/ack numbers, all 1460 bytes. In the last capture I have, it was all JPEG data, although we weren't capturing entire packets. There is a slight difference in the capture payloads, two bytes each time. I had another dial-up provider contact me off list, and he's seeing the same thing. I'm wondering if this is actually more widespread, but only dial-up providers are really seeing the effects since a 3-5Mbps burst is most noticeable for us on our smaller upstream links. // On 1/21/2011 8:45 AM, Jack Bates wrote:
I have a customer reporting the same thing. The traffic flood goes to offline modem bank IPs. So far, Akamai hasn't actually grasped what the problem is and says everything is fine. :(
Luckily, most of the traffic (not all) is coming from my local cluster, so it's easier to monitor what's going on. Packet captures have shown the same packet being sent over and over, usually over 1400 bytes in size. Different floods may have different packets, but within a flood it's identical. I wouldn't think you'd have data prior to the 3-way, so I'm curious how the 3-way is being completed for the data to be sent.
Jack
On 1/20/2011 4:46 PM, Tom Beecher wrote:
I've received a couple of responses off list, and am now in touch with Akamai directly.
I appreciate everyone's assistance.
On 1/20/2011 4:04 PM, Tom Beecher wrote:
I'm looking for an Akamai contact to try and address a strange situation.
We have multiple sites across the country that aggregate 56k dialup customers. Different sites are randomly experiencing inbound traffic spikes that are overwhelming the uplinks to our carriers, causing DoS situations. These spikes far exceed the bandwidth that could possibly be used by the number of dialup customers connected. We've been able to trace the source of the traffic to Akamai boxes, but so far have been unable to reach anyone at Akamai to discuss the situation. We're attempting to get payload information, but the traffic volume is making it slow going setting up packet captures at these sites remotely.
Thanks in advance,
Tom
-- Thomas Beecher II Senior Network Administrator LocalNet Corp. CoreComm Internet Services tbeecher at localnet.com
On 1/21/2011 8:38 AM, Tom Beecher wrote:
Jack-
This is exactly what we're seeing. The Akamai server starts a retransmission flood aimed at a specific address randomly. We're seeing thousands of retransmissions of the same packet over and over again, same sequence/ack numbers, all 1460 bytes. In the last capture I have, it was all JPEG data, although we weren't capturing entire packets. There is a slight difference in the capture payloads, two bytes each time.
The content between attacks changes at times, as do the source IPs, as they send different content. We've noticed at least 2 different akamai hosted sites packets being sent. 1460 is definitely the number. What gets me is that the 3-way should be complete to allow the 1460, and the modem bank is spamming host unreachable ICMP messages since that IP is offline.
I had another dial-up provider contact me off list, and he's seeing the same thing. I'm wondering if this is actually more widespread, but only dial-up providers are really seeing the effects since a 3-5Mbps burst is most noticeable for us on our smaller upstream links. //
This was my thought, though in my downstream's case, it's saturating his DS-3. The 45mb spikes were just enough for me to barely make it out on the akamai gig-e graphs. He's also not always receiving from my local node. Sometimes his other transit links saturate due to remote nodes doing the same thing. Jack
The issue has been reported to the proper people inside Akamai. They are investigating, we are not ignoring the issue. If any network with on-net Akamai servers has an issue, including this or any other, please e-mail NetSupport-tix@akamai.com and that will open a ticket with our Network Support group. -- TTFN, patrick On Jan 21, 2011, at 9:43 AM, Jack Bates wrote:
On 1/21/2011 8:38 AM, Tom Beecher wrote:
Jack-
This is exactly what we're seeing. The Akamai server starts a retransmission flood aimed at a specific address randomly. We're seeing thousands of retransmissions of the same packet over and over again, same sequence/ack numbers, all 1460 bytes. In the last capture I have, it was all JPEG data, although we weren't capturing entire packets. There is a slight difference in the capture payloads, two bytes each time.
The content between attacks changes at times, as do the source IPs, as they send different content. We've noticed at least 2 different akamai hosted sites packets being sent.
1460 is definitely the number. What gets me is that the 3-way should be complete to allow the 1460, and the modem bank is spamming host unreachable ICMP messages since that IP is offline.
I had another dial-up provider contact me off list, and he's seeing the same thing. I'm wondering if this is actually more widespread, but only dial-up providers are really seeing the effects since a 3-5Mbps burst is most noticeable for us on our smaller upstream links. //
This was my thought, though in my downstream's case, it's saturating his DS-3. The 45mb spikes were just enough for me to barely make it out on the akamai gig-e graphs.
He's also not always receiving from my local node. Sometimes his other transit links saturate due to remote nodes doing the same thing.
Jack
participants (4)
-
Chris Adams -
Jack Bates -
Patrick W. Gilmore -
Tom Beecher