The statement used on the survey "Are you aware that use of DPDK on a processor core keeps utilization at 100% regardless of packet activity?" can be easily distorted and badly used.

I sincerely do not agree with the approach of presuming and declaring "DPDK spent too much power".

Mainly because I lived some migrations where dedicated hardware was replaced by the consolidation of servers using NICs with DPDK, and what justified the CAPEX of these replacements was the energy savings (in addition to the capacity expansion)


I agree with Pawell when he says that is the Application that defines power consumption, and it can be improved or not by the skills of the developer...
But, for exemplification, you won't see a Stateful NAT Scenario that does not do ConnTrack Lookups... And those lookups spent CPU cycles, consequently spent energy.


My suggestion to clear things up is to do an honest comparison between DPDK implementations against Fully Hardware-Based Implementations. Considering CAPEX, and OPEX(from energy power and others).


To that, would be necessary to have information on both possibilities.

A) Simulate(with synthetic traffic generator) several scenarios using DPDK, and measure all the relevant data on that(traffic, latency, time of CPU dispended, power consumption, etc)
   Between those scenarios(not an exhaustive list):
   A.1) Soft-Based Router Scenario(stateless)
   A.2) Filtering packets(stateless)
   A.3) Filtering packets(stateful)
   A.4) CGNAT(obviously stateful)

In each of those scenarios, test(when applicable) the possibilities
  Statistics(considering that obtaining accurate statistics implicates more talking between northbridge and southbridge)
   i) Capturing packets statistics (input, drop, forwarding)
   j) Without Capturing packets statistics (input, drop, forwarding)

  Poll-Mode Drivers vs Low-Power-Idle against Traffic Demand
   y) Stateless instructions to DPDK, without collecting statistics demand near-zero interactions between CPU and NIC, so LPI is applicable.
   z) Even Stateful scenarios, with accurate measuring, demand fewer interactions between CPU and NIC when traffic is low (automatic redefinition of the period of pooling cycles)


B) Obtain that kind of information on other solutions available in market
   B.1) Hardware-Based Router (Cisco, Juniper, Huawei, Nokia)
   B.2) Stateless Filtering network devices (Ex.: L3 switchs and ACLs)
   B.3) Stateful Filtering network devices (Ex.: Firewalls)
   B.4) CGNAT Solutions (A10, F5, Hillstone, Juniper, Cisco, Etc...)





Em qua., 24 de fev. de 2021 às 04:48, Etienne-Victor Depasquale <edepa@ieee.org> escreveu:
Hello Robert,

Your statement that DPDK “keeps utilization at 100% regardless of packet activity” is just not correct.  You further pre-suppose "widespread DPDK's core operating inefficiency” without any data to backup the operating inefficacy assertion.  

This statement is incorrect. 
I have provided references (please see earlier e-mails) that investigate the operation of DPDK. 
These references are items of peer-reviewed research that investigate a perceived problem with deployment of DPDK. 
If the power consumption incurred while running DPDK were a corner case, 
then there would be little to no research value in investigating such behavior.

Please don’t mislead the community into believing that DPDK == power bad
I have to object to this statement. It does seem to imply malice, or, at best, amateurish behaviour, whether you intended it or not.

Everything following is informational.  Stop here if so inclined.
 Please stop delving into the detail of DPDK's facilities without regard for your logical omission: 
that whether the facilities are available or not, DPDK's deployment profile (meaning: how it's being used in general), as indicated by the references I've provided,
are leading to high power inefficiency on cores partitioned to the data plane.

The takeaway is that DPDK (and similar) doesn’t guarantee runaway power bills.
Of course it doesn't. 
Even the second question of that bare-bones survey tried to communicate this much.

If you have questions, I’d be happy to discuss off line
I would be happy to answer your objections in detail off line too.
Just let me know.

Cheers,

Etienne


On Wed, Feb 24, 2021 at 12:12 AM Robert Bays <robert@gdk.org> wrote:
Hi Etienne,

Your statement that DPDK “keeps utilization at 100% regardless of packet activity” is just not correct.  You further pre-suppose "widespread DPDK's core operating inefficiency” without any data to backup the operating inefficacy assertion.  Your statements, taken at face value, lead people to believe that if a project uses DPDK it’s going to increase their power costs.  And that’s just not the case.  Please don’t mislead the community into believing that DPDK == power bad.

Everything following is informational.  Stop here if so inclined.

DPDK does not dictate CPU utilization or power consumption, the application leveraging DPDK does.  It’s the application that decides how to poll packets.  If an application implements DPDK using only a tight polling loop, then it will keep CPU cores that are running DPDK threads at 100%.  But only the most simple and/or bespoke (think trading) applications are implemented this way.  You don’t need tight polling all the time to get the performance gains provided by DPDK or similar environments.  The vast majority of applications that this audience would actually install in their networks do not do tight polling all the time and therefore don’t consume 100% of the CPU all the time.   An interesting, helpful research effort you could lead would be to survey the ecosystem to catalog those applications that do fall into the power hungry category and help them to change their code. 

Intel DPDK application development guidelines don’t pre-suppose tight polling all the time and offer at least two methods for optimizing power against throughput.  The older method is to use adaptive polling; increasing the polling frequency as traffic load increases.  This keeps cpu utilization low when packet load is light and increases it as traffic levels warrant.  The second method is to use P-states and/or C-states to put the processor into lower power modes when traffic loads are lighter.  We have found that adaptive polling works better across a larger pool of hardware types, and therefore that is what DANOS uses, amongst other things. 

Further, performance and power consumption are dictated by a multivariate set of application decisions including: design patterns such as single thread run to completion models vs. passing mbufs between multiple threads, buffer sizes and cache management algorithms, combining and/or separating tx/rx threads, binding threads to specific lcores, reserved cores for DPDK threads, hyperthreading, kernel schedulers, hypervisor schedulers, interface drivers, etc.  All of these are application specific, not DPDK generic.  Well written applications that leverage DPDK provide knobs for the user to tune these settings for their specific environment and use case.  None of this unique to DPDK.  Solution designs were cribbed from previous technologies.

The takeaway is that DPDK (and similar) doesn’t guarantee runaway power bills.  Power consumption is dictated by the application.  Look for well behaved applications and everything will be alright.

If you have questions, I’d be happy to discuss off line.

Thanks,
Robert.


> On Feb 22, 2021, at 11:27 PM, Etienne-Victor Depasquale <edepa@ieee.org> wrote:
>
> Sorry, last line should have been:
> "intended to get an impression of how widespread ***knowledge of*** DPDK's core operating inefficiency is",
> not:
> "intended to get an impression of how widespread DPDK's core operating inefficiency is"
>
> On Tue, Feb 23, 2021 at 8:22 AM Etienne-Victor Depasquale <edepa@ieee.org> wrote:
> Beyond RX/TX CPU affinity, in DANOS you can further tune power consumption by changing the adaptive polling rate.  It doesn’t, per the survey, "keep utilization at 100% regardless of packet activity.”
> Robert, you seem to be conflating DPDK
> with DANOS' power control algorithms that modulate DPDK's default behaviour.
>
> Let me know what you think; otherwise, I'm pretty confident that DPDK does:
> "keep utilization at 100% regardless of packet activity.”   
>
> Keep in mind that this is a bare-bones survey intended for busy, knowledgeable people (the ones you'd find on NANOG) -
> not a detailed breakdown of modes of operation of DPDK or DANOS.
> DPDK has been designed for fast I/O that's unencumbered by the trappings of general-purpose OSes,
> and that's the impression that needs to be forefront.
> Power control, as well as any other dimensions of modulation,
> are detailed modes of operation that are well beyond the scope of a bare-bones 2-question survey
> intended to get an impression of how widespread DPDK's core operating inefficiency is.
>
> Cheers,
>
> Etienne
>
> On Mon, Feb 22, 2021 at 10:20 PM Robert Bays <robert@gdk.org> wrote:
> Beyond RX/TX CPU affinity, in DANOS you can further tune power consumption by changing the adaptive polling rate.  It doesn’t, per the survey, "keep utilization at 100% regardless of packet activity.”  Adaptive polling changes in DPDK optimize for tradeoffs between power consumption, latency/jitter and drops during throughput ramp up periods.  Ideally your DPDK implementation has an algorithm that tries to automatically optimize based on current traffic patterns.
>
> In DANOS refer to the “system default dataplane power-profile” config command tree for adaptive polling settings.  Interface RX/TX affinity is configured on a per interface basis under the “interfaces dataplane” config command tree.
>
> -robert
>
>
> > On Feb 22, 2021, at 11:46 AM, Jared Geiger <jared@compuwizz.net> wrote:
> >
> > DANOS lets you specify how many dataplane cores you use versus control plane cores. So if you put a 16 core host in to handle 2GB of traffic, you can adjust the dataplane worker cores as needed. Control plane cores don't stay at 100% utilization.
> >
> > I use that technique plus DANOS runs on VMware (not oversubscribed) which allows me to use the hardware for other VMs. NICS are attached to the VM via PCI Passthrough which helps eliminate the overhead to the VMware hypervisor itself.
> >
> > I have an 8 core VM with 4 cores set to dataplane and 4 to control plane. The 4 control plane cores are typically idle only processing BGP route updates, SNMP, logs, etc.
> >
> > ~Jared
> >
> > On Sun, Feb 21, 2021 at 11:30 PM Etienne-Victor Depasquale <edepa@ieee.org> wrote:
> > Hello folks,
> >
> > I've just followed a thread regarding use of CGNAT and noted a suggestion (regarding DANOS) that includes use of DPDK.
> >
> > As I'm interested in the breadth of adoption of DPDK, and as I'm a researcher into energy and power efficiency, I'd love to hear your feedback on your use of power consumption control by DPDK.
> >
> > I've drawn up a bare-bones, 2-question survey at this link:
> >
> > https://www.surveymonkey.com/r/J886DPY.
> >
> > Responses have been set to anonymous.
> >
> > Cheers,
> >
> > Etienne
> >
> > --
> > Ing. Etienne-Victor Depasquale
> > Assistant Lecturer
> > Department of Communications & Computer Engineering
> > Faculty of Information & Communication Technology
> > University of Malta
> > Web. https://www.um.edu.mt/profile/etiennedepasquale
>
>
>
> --
> Ing. Etienne-Victor Depasquale
> Assistant Lecturer
> Department of Communications & Computer Engineering
> Faculty of Information & Communication Technology
> University of Malta
> Web. https://www.um.edu.mt/profile/etiennedepasquale
>
>
> --
> Ing. Etienne-Victor Depasquale
> Assistant Lecturer
> Department of Communications & Computer Engineering
> Faculty of Information & Communication Technology
> University of Malta
> Web. https://www.um.edu.mt/profile/etiennedepasquale



--
Ing. Etienne-Victor Depasquale
Assistant Lecturer
Department of Communications & Computer Engineering
Faculty of Information & Communication Technology
University of Malta
Web. https://www.um.edu.mt/profile/etiennedepasquale


--
Douglas Fernando Fischer
Engº de Controle e Automação