Server PS maximum input wattage is 900W. Present draw of 2.0A @ 208V is ~420W, so 420/900 = 46.67%
But in the real world an R640 would *never* draw 900W. Even if you were to load it up with the maximal CPU configuration (2 x 125W TDP CPU per socket), a full load of 2.5" 15K spinning drives, maximum RAM, and three high wattage low-profile PCI-E cards, while simultaneously running CPU, RAM and disk stress tests, you might get in the neighborhood of 550-600W under load. Much the same way that a desktop PC equipped with a nominally "850W" rated active PFC 80+ gold power supply might be powering a motherboard and CPU combo with a high CPU TDP, but total power consumption under stress tests/benchmarks would be nowhere near 850W. That rating exists to ensure that the power supply isn't running anywhere near its max capacity... On Fri, Mar 5, 2021 at 3:33 PM Brian Knight <ml@knight-networks.com> wrote:
On 2021-03-05 15:40, Eric Kuhnke wrote:
For comparison purposes, I'm curious about the difference in wattage results between:
a) Your R640 at 420W running DPDK
b) The same R640 hardware temporarily booted from a Ubuntu server live USB, in which some common CPU stress and memory disk/IO benchmarks are being run to intentionally load the system to 100% to characterize its absolute maximum AC load wattage.
We've got a few more hosts waiting to be deployed that are configured almost identically. I'll see what we can do.
I'm guessing those tests would pull slightly more power than the vEdge hosts, just because there's not much disk IO that happens on a networking VM. These hosts have four SSDs for local storage.
What's the delta between the 420W and absolute maximum load the server is capable of pulling on the 208VAC side?
https://manpages.ubuntu.com/manpages/artful/man1/stress-ng.1.html
Server PS maximum input wattage is 900W. Present draw of 2.0A @ 208V is ~420W, so 420/900 = 46.67%
One possible factor is whether ESXI is configured to pass the pci-e devices directly through to the guest VM, or if there is any abstraction in between. For non-ESXI stuff, in the world of Xen or KVM there's many different ways that a guest domU can access a dom0's network devices, some of which can have impact on overall steady-state wattage consumed by the system.
The 420W server has its interfaces routed through the ESXI kernel. We're moving quickly to SR-IOV on new servers.
If the greatest possible efficiency is desired for a number of 1U things, one thing to look at would be something similar to the open compute platform single centralized AC to DC power units, and servers that don't each have their own discrete 110-240VAC single or dual power supplies. In terms of cubic meters of air moved per hour vs wattage, the fans found in 1U servers are really quite inefficient. As a randomly chosen example of 12VDC 40mm (1U server height) fan:
https://www.shoppui.com/documents/9HV0412P3K001.pdf
If you have a single 12.0VDC fan that's a maximum load of 1.52A, that's a possible load of up to 18.24W for just *one* 40mm height fan. And your typical high speed dual socket 1U server may have up to eight or ten of those, in the typical front to back wind tunnel configuration. Normally fans won't be running at full speed, so each one won't be a 18W load, but more like 10-12W per fan is totally normal. Plus two at least two more fans in both hot swap power supplies. Under heavy load I would not be surprised at all to say that 80W to 90W of your R640's total 420W load is ventilation.
Which is of course dependent on the environmentals. Fan speeds on our two servers are 25% for the 260W vs. 29% for 420W, so not much difference. Inlet temp on both is ~17C.
I checked out another R640 heavily loaded with vEdge VMs, and it's pulling similar power, 415W, but the fan speed is at 45%, because inlet temp is 22C.
The TDP for the Xeon 6152 is 140W, which seems middle-of-the-road. From the quick survey I did of Dell's configurator, the R640 can take CPUs up to 205W. So we have headroom in terms of cooling.
In a situation where you're running out of power before you run out of rack space, look at some 1.5U and 2U high chassist that use 60mm height fans, which are much more efficient in ratio of air moved per time period vs watts.
Or ask the colo to turn the A/C lower ;) (that moves the power problem elsewhere, I know)
Thanks,
-Brian