I have used MapForce Pro edition, a generic commeercial tool for any-to-any data conversion, for similar tasks in the past. Specifically I was converting pcap logs of SMTP sessions to create a database of source and destination email for a custom security project. It worked pretty well, and the tool is reasonably priced. I did end up writing a Python pcap preprocessor but that was only to filter out stuff I didn't want. -mel ________________________________ From: Jonas Muecke via NANOG <nanog@lists.nanog.org> Sent: Thursday, January 15, 2026 9:57 AM To: nanog@lists.nanog.org <nanog@lists.nanog.org> Cc: Jonas Muecke <jonas.muecke@tu-dresden.de> Subject: Converting IPFIX flows to columnar format with all IEs - tool recommendations? Hi, I'm looking for recommendations on conversion and long-term storage of raw IPFIX flow data. Specifically, I need to convert IPFIX flows stored in pcap files into a columnar format like Parquet to enable easy and detailed historic analysis that isn't possible with aggregated data. Requirements: - Parse IPFIX from pcap files (including templates) - Preserve ALL information elements, including custom IEs with enterprise PENs - Output enterprise number + IE ID + data (detailed interpretation of the data not needed) - Handle IP fragment reassembly or large IP packets nfdump [1] gets close, but it skips custom IEs. Other tools require replaying the pcaps which risks overflowing buffers, so reading directly from pcap files would be preferred. Has anyone had similar requirements and found a solution? I'm open to multi-step conversions (e.g., via JSON). Long-term I'll capture directly to a better format, but need to process existing pcap archives first. Thanks, Jonas [1] https://github.com/phaag/nfdump -- Jonas Muecke Phd Student, TU Dresden _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/U7ZR5BJF...

Take a look at SiLK. Very robust, very powerful, command line: https://tools.netsa.cert.org/silk/silk.html FlowViewer sits on top and provides a full GUI interface: flowviewer.net Joe On 1/15/2026 12:57 PM, Jonas Muecke via NANOG wrote:

Hi,

I'm looking for recommendations on conversion and long-term storage of raw IPFIX flow data. Specifically, I need to convert IPFIX flows stored in pcap files into a columnar format like Parquet to enable easy and detailed historic analysis that isn't possible with aggregated data.

Requirements: - Parse IPFIX from pcap files (including templates) - Preserve ALL information elements, including custom IEs with enterprise PENs - Output enterprise number + IE ID + data (detailed interpretation of the data not needed) - Handle IP fragment reassembly or large IP packets

nfdump [1] gets close, but it skips custom IEs. Other tools require replaying the pcaps which risks overflowing buffers, so reading directly from pcap files would be preferred.

Has anyone had similar requirements and found a solution? I'm open to multi-step conversions (e.g., via JSON). Long-term I'll capture directly to a better format, but need to process existing pcap archives first.

Thanks, Jonas

[1] https://github.com/phaag/nfdump

-- Jonas Muecke Phd Student, TU Dresden

_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/U7ZR5BJF...