On 02.10.2020 at 12:03 Job Snijders wrote:
Marco Marzetti (PCCW) wrote an even faster compression tool! https://github.com/lamehost/aggregate-prefixes Both these python implementations are meant as replacements for ISC's vintage 'aggregate' Unix utility, with the notable difference that they also support IPv6.
Example:
job@bench ~$ pip3 install aggregate-prefix
job@bench ~$ wc -l dfz_ipv4 810607 job@bench ~$ cat dfz_ipv4 | time aggregate-prefixes - | wc -l 141645 1m40.17s real 1m37.39s user 0m01.60s system
Compressing the whole IPv4 DFZ prefix list takes only 100 seconds.
First time I uploaded/publish something to/on Github ... so please be kind: https://github.com/FvDxxx/pfxaggr In case you need it even faster (and can accept the little known issues and that it's old, ugly and never reviewed):
wc -l dfz-pfx-20201002-A-v4.txt 813542 dfz-pfx-20201002-A-v4.txt
time cat dfz-pfx-20201002-A-v4.txt | ./pfxagg -a1 > dfz-4-agg-pfx.log
real 0m1.034s user 0m0.909s sys 0m0.232s
time cat dfz-pfx-20201002-A-v4.txt | aggregate-prefixes > dfz-4-agg-pyth.log
real 1m11.691s user 1m10.879s sys 0m0.786s
diff dfz-4-agg-pyth.log dfz-4-agg-pfx.log wc -l dfz-4-agg-pyth.log dfz-4-agg-pfx.log 141754 dfz-4-agg-pyth.log 141754 dfz-4-agg-pfx.log 283508 total
Cheers, Markus