Disk image compression drops

Vax456 asked:

I’m using partclone to take disk images with this command:

partclone.ntfs -c -s /dev/$mypartition | lzop -c > /my/nfs/share/mypartition.lzo

The partition size is 34.1 GB with 33.8 GB used.

The problem is the performance drops after a while. At first the backup is being written at roughly 300 MB/minute, but the performance gradually drops to 150-200 MB/minute. The estimated time of completion increases from about 1.25 hours to about 2.5 hours.

I took an image with no compression:

partclone.ntfs -c -s /dev/$mypartition -o /my/nfs/share/mypartition

The backup speed hovers around 600 MB/minute, but the performance DOESN’T degrade after a while. It stays around 600 MB/minute more or less.

Is there anything I can do keep the performance from degrading when using compression?

EDIT: I’ve used gzip and lzop and both suffer from degrading performance. I also tried backing up the same image on a Pentium D machine (2 cores, 3 GB of RAM), and a Xen DomU (1 processor, 512 MB of RAM). They both have the same problem.

My answer:

With the old CPUs you’re using, it seems most likely that you’re CPU bound; compression eats a lot of CPU (and it can vary depending on the data being compressed) and is almost certainly the immediate cause of the issue.

If you can compress with a faster CPU somehow, perhaps by piping the data to a faster machine first, then that should help.

You can also try using a multi-core capable compression utility such as pbzip2, which should increase the rate of compression on your dual-core system, at least. It won’t do anything for you on the single-CPU VM, though. pbzip2 writes bzip2-compatible files and is already packaged for most Linux distributions.

View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.