Thursday, March 13, 2014

Createrepo and Parallel Delta RPMs on CentOS-6

Anyone who is using Delta RPMs (AKA - the yum-presto plugin) on CentOS-6 or Fedora knows how much of a PITA it is to generate the drpms.  It takes forever to generate.

Ian Mcleod has posted some new functionality on the Yum Developers mailing list that addresses this issue.

Here is the git repo that contains the new code.

I have created an RPM and SRPM for EL6 that use the tarball from the git repo and add the following functionality:

--delta-workers - The number of worker processes to use for delta
related tasks

--max-concurrent-delta-rpm-size - The maximum total size of uncompressed
rpm payloads that are actively being processed by makedeltarpm at any
given time.

You can get copies of the RPMS from my people.centos.org location for testing. 

The only things that I found with the rpm is:

1.  You will need to also install pyliblzma package from epel to use it.

2.  You should use the same (or lower) number of --delta-workers as compared to --workers 

3.  As discussed in the Yum Developers thread above, you want to make sure to assign a value to --max-concurrent-delta-rpm-size .  The number to assign is bytes ... so 16GB (if you have a 64GB ram machine, and the recommended amount is about 25%) then it would be 16 x 1024 x 1024 x 1024 = 17179869184 (as an example).  That machine also has 24 CPU cores ... so I used 24 for --workers and --delta-workers.