For comparison purposes, I'm curious about the difference in wattage results between:
a) Your R640 at 420W running DPDK
b) The same R640 hardware temporarily booted from a Ubuntu server live USB, in which some common CPU stress and memory disk/IO benchmarks are being run to intentionally load the system to 100% to characterize its absolute maximum AC load wattage.
What's the delta between the 420W and absolute maximum load the server is capable of pulling on the 208VAC side?
One possible factor is whether ESXI is configured to pass the pci-e devices directly through to the guest VM, or if there is any abstraction in between. For non-ESXI stuff, in the world of Xen or KVM there's many different ways that a guest domU can access a dom0's network devices, some of which can have impact on overall steady-state wattage consumed by the system.
If the greatest possible efficiency is desired for a number of 1U things, one thing to look at would be something similar to the open compute platform single centralized AC to DC power units, and servers that don't each have their own discrete 110-240VAC single or dual power supplies. In terms of cubic meters of air moved per hour vs wattage, the fans found in 1U servers are really quite inefficient. As a randomly chosen example of 12VDC 40mm (1U server height) fan:
If you have a single 12.0VDC fan that's a maximum load of 1.52A, that's a possible load of up to 18.24W for just *one* 40mm height fan. And your typical high speed dual socket 1U server may have up to eight or ten of those, in the typical front to back wind tunnel configuration. Normally fans won't be running at full speed, so each one won't be a 18W load, but more like 10-12W per fan is totally normal. Plus two at least two more fans in both hot swap power supplies. Under heavy load I would not be surprised at all to say that 80W to 90W of your R640's total 420W load is ventilation.
In a situation where you're running out of power before you run out of rack space, look at some 1.5U and 2U high chassist that use 60mm height fans, which are much more efficient in ratio of air moved per time period vs watts.