Are they refreshing data they've already got, though?

This is the classic use case for client-side caching.

On Tue, Jul 16, 2019 at 5:56 PM Ken Gilmour <ken.gilmour@gmail.com> wrote:

We have a different use case to traditional analytics - We're aimed at consumers and small businesses, so instead of a SOC with one big screen refreshing 10000 rows of only alert data every 30 seconds, we have thousands of individuals refreshing all of their data every 30 seconds because there are comparatively less alerts for individuals than enterprises.

What you "should" do often doesn't translate to what you "do" do.

On Tue, 16 Jul 2019 at 11:23, Valdis Kl??tnieks <valdis.kletnieks@vt.edu> wrote:

On Tue, 16 Jul 2019 10:39:59 -0600, Ken Gilmour said:

> These are actual real problems we face. thousands of customers load and
> reload TBs of data every few seconds on their dashboards.

If they're reloading TBs of data every few seconds, you really should have been
doing summaries during data ingestion and only reloading the summaries.
(Overlooking the fact that for dashboards, refreshing every few seconds is
usually pointless because you end up looking at short-term statistical spikes
rather than anything that you can react to at human speeds.  If you *care* in
real time that the number of probes on a port spiked to 457% of average for 2
seconds you need to be doing automated responses....

Custom queries are more painful - but those don't happen "every few seconds".