[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Sheflug] Pigz compression

Hi All,

So I was reading through the pigz manpages. pigz is a parallel gzip tool
and decided to spend some time testing it. This is all part of my
deduplication rabbit hole that I have now fallen down.

lscpu
    Thread(s) per core 2
    Core(s) per socket 6
    Socket(s) 1
So 2*6*1 gives us 12 as the max thread count on the machine.

I am using a 58GB directory as a test of which the files are mostly
compressed anyway.
    time tar cf - /spare/deduptest/Retro/DS/ | pigz -0 -p 12 > DS.tar.gz
takes 26 minutes with the -0 meaning no compression for a 58.3GB output

    time tar cf - /spare/deduptest/Retro/DS/ | pigz -p 12 > DS.tar.gz
takes 15 minutes 38 seconds with the default output for a 35.0GB output

    time tar cf - /spare/deduptest/Retro/DS/ | pigz -9 -p 12 > DS2.tar.gz
takes 16 minutes 55 seconds with the maximum compression for a 34.9GB output

    time tar cf - /spare/deduptest/Retro/DS/ | pigz -1 -p 12 > DS3.tar.gz
takes 16 minutes 10 seconds with the maximum speed for a 35.7GB output

The difference between the fastest and slowest is just 77 seconds with the
sizes varying from 34.9GB to 35.7GB. I would not expect much compression
difference as the files should be fairly compressed anyway.

Just for reference
    time tar cf DS4.tar /spare/deduptest/Retro/DS/
took 25 minutes 24 seconds for a 58.3GB output

and
    time tar -czf DS5.tar.gz /spare/deduptest/Retro/DS/
took

So the question is why does just tar or the zero compression take so long
and why have the -o option in the pigz command?


Regards
John
_______________________________________________
Sheffield Linux User's Group
http://sheflug.org.uk/mailman/listinfo/sheflug_sheflug.org.uk
FAQ at: http://www.sheflug.org.uk/mailfaq.html

GNU - The Choice of a Complete Generation