Hi Folks, Possibly a little off-topic for nanog, but I couldn't think of anywhere else to ask this (suggestions please!): We just discovered a vulnerability the hard way - someone used one of our IPMI boards as a vector for a DDoS attack (well, I guess the real hard way would be to have been on the receiving end, but...). Anyway... aside from some obvious issues, I've been learning a lot about the vulnerabilities of Supermicro IPMI boards (and busily locking them down). The one that's tricky, though, is that this was a reflection/amplification attack. Conveniently, the attackee's data center operator managed to capture incoming packets with tcpdump, and they all looked like this: 8:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. (obviously, with the IP addresses removed). It could be that someone planted a bot in the IPMI board (just starting to do some forensics - currently hampered by being on travel, and having blocked all the ports from the outside world - need to get to the datacenter and make some hardwired connections) - but it looks a lot more like a reflection/amplification attack - particularly since the target seems to have been a game host, and port 27015 is used by the game halflife. But.... Now I understand reflected DNS and NTP attacks - but the outbound port, 2072 (registered for GlobeCom msync) is neither, nor is it anything that we're running - which kind of begs the question of how this might be working. Any thoughts? Any pointers? Any starting points? Immediate issue is dealt with (at least for us, target seems to be off the air) - but want to understand this, report it, all of that. Thanks very much, Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
On Aug 26, 2014, at 6:48 PM, Miles Fidelman <mfidelman@meetinghouse.net> wrote:
Immediate issue is dealt with (at least for us, target seems to be off the air) - but want to understand this, report it, all of that.
IPMI boards are reported as being used in reflection/amplification attacks of various kinds; the ntp one is straightforward, as you note. This may be some sort of chargen-like packet reflector that's either built into the firmware, or that an attacker has managed to insert, somehow. The 'mailto:' bit is interesting; it might work sort of like SNMP reflection/amplification attacks work, where the attacker is using some sort of management functionality to walk the device config or somesuch, packetize it, and blast it out as packet-padding. Does the target of the attack have flow telemetry records or complete packets? Because the one you posted looked incomplete (29 bytes?) . . . ---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Equo ne credite, Teucri. -- Laocoön
Roland Dobbins wrote:
On Aug 26, 2014, at 6:48 PM, Miles Fidelman <mfidelman@meetinghouse.net> wrote:
Immediate issue is dealt with (at least for us, target seems to be off the air) - but want to understand this, report it, all of that. IPMI boards are reported as being used in reflection/amplification attacks of various kinds; the ntp one is straightforward, as you note.
This may be some sort of chargen-like packet reflector that's either built into the firmware, or that an attacker has managed to insert, somehow. The 'mailto:' bit is interesting; it might work sort of like SNMP reflection/amplification attacks work, where the attacker is using some sort of management functionality to walk the device config or somesuch, packetize it, and blast it out as packet-padding.
Can you say a bit more about what I might look for in trying to track this down?
Does the target of the attack have flow telemetry records or complete packets? Because the one you posted looked incomplete (29 bytes?) . . .
Unfortunately, all I have is what they sent to our abuse address - understandably, they've been a bit busy and not as responsive to further inquiries as one might hope. But, having said that, this looks like all they have. They seem to be getting these from lots of different places around the net, they just sent a filtered excerpt - here's a larger sample: 18:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 18:33:58.484625 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 18:33:58.486137 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. On closer reading, what they captured does seem to be "proto UDP (17), length 29)" and "UDP, length 1" Thanks! Miles -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
On Aug 26, 2014, at 7:18 PM, Miles Fidelman <mfidelman@meetinghouse.net> wrote:
Can you say a bit more about what I might look for in trying to track this down?
Fuzz your IPMI boards? ;> My guess is that this is going to come to light sooner rather than later. We're getting reports of other DDoS attacks which seem to match this scenario involving IPMI boards. There's no real reason to try and track it down, from an operational standpoint, is there> Management-plane things like IMPI boards shouldn't be open to the general Internet; put them behind ACLs and use a VPN. Problem solved. ---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Equo ne credite, Teucri. -- Laocoön
qotd 17/udp quote You're not blocking small services outbound at the edge? On 08/26/2014 05:18 AM, Miles Fidelman wrote:
Roland Dobbins wrote:
On Aug 26, 2014, at 6:48 PM, Miles Fidelman <mfidelman@meetinghouse.net> wrote:
Immediate issue is dealt with (at least for us, target seems to be off the air) - but want to understand this, report it, all of that. IPMI boards are reported as being used in reflection/amplification attacks of various kinds; the ntp one is straightforward, as you note.
This may be some sort of chargen-like packet reflector that's either built into the firmware, or that an attacker has managed to insert, somehow. The 'mailto:' bit is interesting; it might work sort of like SNMP reflection/amplification attacks work, where the attacker is using some sort of management functionality to walk the device config or somesuch, packetize it, and blast it out as packet-padding.
Can you say a bit more about what I might look for in trying to track this down?
Does the target of the attack have flow telemetry records or complete packets? Because the one you posted looked incomplete (29 bytes?) . . .
Unfortunately, all I have is what they sent to our abuse address - understandably, they've been a bit busy and not as responsive to further inquiries as one might hope.
But, having said that, this looks like all they have. They seem to be getting these from lots of different places around the net, they just sent a filtered excerpt - here's a larger sample:
18:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 18:33:58.484625 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 18:33:58.486137 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
On closer reading, what they captured does seem to be "proto UDP (17), length 29)" and "UDP, length 1"
Thanks!
Miles
On Aug 26, 2014, at 8:26 PM, Stephen Satchell <list@satchell.net> wrote:
qotd 17/udp quote
No, that's the protocol number - 17 is UDP - not the port number. ---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Equo ne credite, Teucri. -- Laocoön
In this case, 17 is both the protocol and port number. Confusing coincidence :)
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Roland Dobbins Sent: Tuesday, August 26, 2014 8:32 AM To: nanog@nanog.org Subject: Re: where to go to understand DDoS attack vector
On Aug 26, 2014, at 8:26 PM, Stephen Satchell <list@satchell.net> wrote:
qotd 17/udp quote
No, that's the protocol number - 17 is UDP - not the port number.
---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Equo ne credite, Teucri.
-- Laocoön
On Aug 26, 2014, at 8:37 PM, John York <johny@griffintechnology.com> wrote:
In this case, 17 is both the protocol and port number. Confusing coincidence :)
Not in this output which the OP sent to the list:
8:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
18:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 18:33:58.484625 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 18:33:58.486137 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
Source port 2072, destination port 27015. ---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Equo ne credite, Teucri. -- Laocoön
On 08/26/2014 07:58 AM, Roland Dobbins wrote:
On Aug 26, 2014, at 8:37 PM, John York <johny@griffintechnology.com> wrote:
In this case, 17 is both the protocol and port number. Confusing coincidence :) Not in this output which the OP sent to the list:
8:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
18:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 18:33:58.484625 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 18:33:58.486137 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. Source port 2072, destination port 27015.
Been awhile since I got to dig into hex tcpdump but spent the time anyway. A UDP data segment that is 9 bytes long and only contains a "C" (0x43) ? And looks like to a Steam/Half-life (27015) gaming port. Not sure what the "C" is used for with those systems but guessing it's some sort of request? --john
---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Equo ne credite, Teucri.
-- Laocoön
On 8/26/2014 12:52 PM, me wrote:
On 08/26/2014 07:58 AM, Roland Dobbins wrote:
On Aug 26, 2014, at 8:37 PM, John York <johny@griffintechnology.com> wrote:
In this case, 17 is both the protocol and port number. Confusing coincidence :) Not in this output which the OP sent to the list:
8:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
18:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 18:33:58.484625 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 18:33:58.486137 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. Source port 2072, destination port 27015.
Been awhile since I got to dig into hex tcpdump but spent the time anyway. A UDP data segment that is 9 bytes long and only contains a "C" (0x43) ? And looks like to a Steam/Half-life (27015) gaming port. Not sure what the "C" is used for with those systems but guessing it's some sort of request?
It's pretty tough to say without knowing exactly what game is running there. While 27015 was originally used for Half Life, it's been used by a wide range of games at this point. Pretty much all the Valve games use this port, as well as a number of third party games that are based on the Steamworks SDK. Trying to figure out exactly what the game server thinks the packet is is not likely to help you figure out why it's being sent. You should instead be figuring out why your IPMI controller is compromised. It could also be reflection, 2072 is within the port range that is usually used for KVM or remote media by the IPMI controllers (though, they're usually TCP and not UDP).
me wrote:
On 08/26/2014 07:58 AM, Roland Dobbins wrote:
On Aug 26, 2014, at 8:37 PM, John York <johny@griffintechnology.com> wrote:
In this case, 17 is both the protocol and port number. Confusing coincidence :) Not in this output which the OP sent to the list:
8:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
18:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 18:33:58.484625 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 18:33:58.486137 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1 0x0000: 4500 001d 0000 4000 3811 088c cf9a 3b8c E.....@.8 <mailto:E.....@.8>.....;. 0x0010: 405e eebf 0818 6987 0009 10f8 4300 0000 @^....i.....C... 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. Source port 2072, destination port 27015.
Been awhile since I got to dig into hex tcpdump but spent the time anyway. A UDP data segment that is 9 bytes long and only contains a "C" (0x43) ? And looks like to a Steam/Half-life (27015) gaming port. Not sure what the "C" is used for with those systems but guessing it's some sort of request?
That's about as far as I've gotten. What has me scratching my head is what is setting the source port. This has all the earmarks of a reflection attack, except... I'm not running anything that presents as port 2072 (msync) - so either the attack is making very clever use of some other open server, or the board's BMC is infected by a bot. Unfortunately, with the port now blocked, and what was intermittent in any case - it's a little hard to monitor incoming traffic to see what might be trigger traffic. Sigh... Thanks, Miles
On 8/26/2014 10:40 AM, Miles Fidelman wrote:
That's about as far as I've gotten. What has me scratching my head is what is setting the source port. This has all the earmarks of a reflection attack, except... I'm not running anything that presents as port 2072 (msync) - so either the attack is making very clever use of some other open server, or the board's BMC is infected by a bot. Unfortunately, with the port now blocked, and what was intermittent in any case - it's a little hard to monitor incoming traffic to see what might be trigger traffic. Sigh...
From the traffic dump and description, this was highly likely to be a direct attack and not an amplification/reflection hit. I don't know of reflectors that run on port 2072; but, bots are routinely used to send UDP length 29 (payload length 1) packets. Older Supermicro IPMI devices have multiple published exploits including the much-publicized port-49152 vulnerability that provides the admin password in the clear (described at http://blog.cari.net/carisirt-yet-another-bmc-vulnerability-and-some-added-e... and other places). Many device owners also never change the default u/p, in which case an exploit doesn't even need to be used. The attacker will typically use a tool to scan the IPv4 space for vulnerable hosts; the tool logs in and installs a bot that connects to a C&C server and is later used for attacks. The same procedure is followed with other easily-compromised devices, including Hikvision DVRs/NVRs and various routers including the Chinese Telecom F420. Resulting botnets can be tens or even hundreds of thousands of hosts in size. IPMI devices have been used quite regularly for attacks for a couple of months now -- as soon as that vulnerability was made public, the toolmakers started using it. The best defense against current and yet-to-be-discovered IPMI vulnerabilities is to make sure that your IPMI devices are not open to the public internet, as Roland said. -John
On Tue, 26 Aug 2014 18:57:27 +0700, Roland Dobbins said:
. The 'mailto:' bit is interesting; it might work sort of like SNMP reflection/amplificati
Nope. It's a red herring, somebody's MUA trying to get *far* too clever with the fact that there's a literal "....@.8" in the ascii dump part of the packet. Took me a few seconds to figure it out too, am a tad low on caffeine today. :)
On Aug 26, 2014, at 11:02 PM, Valdis.Kletnieks@vt.edu wrote:
Took me a few seconds to figure it out too, am a tad low on caffeine today. :)
doh, lack of proper sanitization/escaping strikes again! ---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Equo ne credite, Teucri. -- Laocoön
participants (8)
-
Brian Rak -
John -
John York -
me -
Miles Fidelman -
Roland Dobbins -
Stephen Satchell -
Valdis.Kletnieks＠vt.edu