After some troubleshooting I ended up having to increase my kafka partitions as well as my clickhouse collectors as it seemed like clickhouse would lack behind quite a bit 
I also has some issues with the server where CPU and RAM would max out all the time, my RAM usage is still quite high and seems to grow exponentially as the day goes by, but i don't think its causing the issues anymore. Storage wise I'm good tho

Thanks for the advice.

Kind regards,

 

Javier Gutierrez,

From: Vincent Bernat <bernat@luffy.cx>
Sent: Saturday, June 8, 2024 2:46 AM
To: Javier Gutierrez <GutierrezJ@westmancom.com>; Steven Bakker <steven.bakker@ams-ix.net>; Peter Phaal <peter.phaal@gmail.com>
Cc: nanog@nanog.org <nanog@nanog.org>
Subject: Re: Open source Netflow analysis for monitoring AS-to-AS traffic
 
CAUTION: This email is from an external source. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Without much information, I think this is more likely that you are
running out of disk space.

On 2024-06-05 23:15, Javier Gutierrez wrote:
> Hi everyone,
> I've been trying to get Akvorado to work on my environmnet but I keep
> getting the flows to stop collecting, it seems like the issue is related
> to the number of exporters I have sending data, can someone please share
> the max number they have gotten to work and the flows/s rate without the
> system crashing?
>
> Thanks in advance for your answers.
> ------------------------------------------------------------------------
> *From:* NANOG <nanog-bounces+gutierrezj=westmancom.com@nanog.org> on
> behalf of Steven Bakker <steven.bakker@ams-ix.net>
> *Sent:* Sunday, March 31, 2024 4:53 AM
> *To:* Peter Phaal <peter.phaal@gmail.com>
> *Cc:* nanog@nanog.org <nanog@nanog.org>
> *Subject:* Re: Open source Netflow analysis for monitoring AS-to-AS traffic
>
> *CAUTION: *This email is from an external source. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
> Hi Peter,
>
> Thanks for that link. I did read the spec, and while the definition
> itself is clear, the escape clause gives a lot of wiggle room:
>
> "/Hardware limitations may// prevent an exact reporting of the
> underlying frame length, but an agent should attempt to //be as accurate
> as possible./"
>
> I read that as, "the vendor will do whatever it pleases, and you should
> be grateful to receive a non-negative integer at all." I could be too
> cynical, though.
>
> Anyway, this particular vendor does other funny things (such as
> sometimes stripping the q-tag headers from the sampled frame; throttling
> the frame sampling on the box, but not adjusting the sampling interval
> in the sFlow exports) that make it a true joy to work with this gear. ;-)
>
> Cheers,
>
> -- Steven