Createrepo and Parallel Delta RPMs on CentOS-6

Anyone who is using Delta RPMs (AKA - the yum-presto plugin) on CentOS-6 or Fedora knows how much of a PITA it is to generate the drpms.  It takes forever to generate.

Ian Mcleod has posted some new functionality on the Yum Developers mailing list that addresses this issue.

Here is the git repo that contains the new code.

I have created an RPM and SRPM for EL6 that use the tarball from the git repo and add the following functionality:

--delta-workers - The number of worker processes to use for delta
related tasks

--max-concurrent-delta-rpm-size - The maximum total size of uncompressed
rpm payloads that are actively being processed by makedeltarpm at any
given time.

You can get copies of the RPMS from my people.centos.org location for testing. 

The only things that I found with the rpm is:

1.  You will need to also install pyliblzma package from epel to use it.

2.  You should use the same (or lower) number of --delta-workers as compared to --workers 

3.  As discussed in the Yum Developers thread above, you want to make sure to assign a value to --max-concurrent-delta-rpm-size .  The number to assign is bytes ... so 16GB (if you have a 64GB ram machine, and the recommended amount is about 25%) then it would be 16 x 1024 x 1024 x 1024 = 17179869184 (as an example).  That machine also has 24 CPU cores ... so I used 24 for --workers and --delta-workers.

CentOS at Scale 12x

Several members of the CentOS team will be at Scale 12x (http://bit.ly/1hEH6t5) in Los Angeles, California on February 21st to 23nd, 2014.

CentOS will be part of the "Infrastructure.Next at Scale" event ( http://bit.ly/1irKWUm ) event that happens on Friday (21st), and we will have the following talk there:
http://bit.ly/1g2voUB

We will be at the Red Hat Community booth/table on both Saturday and Sunday (22nd and 23rd) with free swag (teeshirts, stickers, etc).

We will have a Birds of a Feather session, details of which will be provided at the table/booth when finalized.

Finally, we will also have a talk titled "CentOS Project Q&A Forum" (http://bit.ly/1bQ5rVS) on Sunday (23rd).

If you are in the Los Angeles area, please stop by and see us at Scale 12x !!!

Kernel-ml, NVIDIA Drivers, and the 3.13.x kernel

ELRepo has a Main Line kernel section to allow using the latest kernel with CentOS-6.  I use this kernel on my laptops and workstations as it usually means better hardware detection and battery life, etc.

If you have the need to use the proprietary NVIDIA drivers with this kernel, as I do on one of my laptops, then you would need to install the latest NVIDIA drivers for each kernel.  No one (that I know of at least) maintains an RPM based module for the kernel-ml series, so downloading the latest version of the drivers from NVIDIA's site is how I handle this situation.

Sometimes with a new kernel tree is begun, in this case the 3.13.x kernel tree, the NVIDIA drivers as posted with not work.  This is the current situation with the 3.13.2 kernel (as installed via ELRepo on CentOS 6.5 ) when using the latest NVIDIA drivers (currently version 338.31).

On the web, I found a good blog entry on how to generally fix the issue, so I thought I would go over how to make it work on CentOS-6.

• Enable the ELRepo repository (if required) using their instructions here.
• Install the ELRepo kernel-ml via their instructions here.
• Download the latest NVIDIA Drivers from the NVIDIA site.
• Change the permissions on the downloaded binary file so it can be executed:

chmod 755 NVIDIA-Linux-x86_64-331.38.run

• Extract the NVIDIA drivers with this command in a terminal:

./NVIDIA-Linux-x86_64-331.38.run -x

• Download the combined (2nd) patch from the above blog and save it (right click, save link as) into the NVIDIA-Linux-x86_64-331.38 directory created when extracting the driver above.
• Apply the patch with this command:

patch -p1 < 02_nvidia_3.13.patch

• Reboot into the new kernel (will get console mode, login as root) ... then go to the  NVIDIA-Linux-x86_64-331.38 that contains the patched code.
• Install the patched driver with this command:

./nvidia-installer

The install should now complete in the normal way, and you should be able to boot into the NVIDIA driver on the 3.13.x kernel-ml series from ELRepo on CentOS-6.5

CentOS-6.3 Release

July 9th, 2012 - Today the CentOS Project has released CentOS-6.3.  This release came 18 days after the upstream release of EL 6.3.  The major issues that we had with getting this release out were getting the i386 distribution to boot properly and adjusting the content of DVD-1 and DVD-2 of each architecture to allow for the most common install groups to come from DVD1 and not require DVD2.

Some of the major changes for this release include a move from OpenOffice 3.2.1 to LibreOffice 3.4.5 and the addition of Virtual to Virtual and Physical to Virtual (virt-v2v and virt-p2v) server migration tools to KVM Virtual Machines.  Please see the CentOS-6.3 release notes for more information on these and other features.  The Open H Website has a very good article on what's new in EL6.

We were pretty much ready to release the distribution by Friday, July 6th, but we had to then copy our trees to the several dozen internal mirror servers in the CentOS Project's infrastructure and then open those mirrors up to the more than 500 external mirrors that serve CentOS in more than 75 countries world wide.  While the CentOS Project is doing a point release, you can see what is happen by visiting our QA Development website and looking at the blog entries on the dashboard,

We hope you enjoy CentOS-6.3 !

CentOS Project Release Times

One of the issues we have had in the CentOS Project is that at point release time, we were lagging behind a bit on getting the releases out.  Recently, the CentOS Project has addressed this issue in 3 major ways:

1.  We created the CR repository where we can, if there are delays, push out multiple updates that work together while we take a care of problems with packages that don't build, if we have issues getting the ISOs to properly build, etc. during the point release process.  This gives us an avenue to release pieces of the point release without having to release the whole thing.  We have actually not had to use the CR repository on the last 2 release cycles (CentOS-5.8 and CentOS-6.2) as we got each of them out within 10 days... however, this is an option that we have if we need to get out pieces of the release (like critical updates) if there is any kind of delay in the main process.

2.  We now have corporate sponsors who sponsor 2 CentOS Developers to work on the CentOS Project full time.  That means that we now have 80 paid hours per week of CentOS Project time where we get do nothing but CentOS Project related work.  The sponsors do not ask for anything in return, just faster CentOS updates by the current CentOS developers who get to make the CentOS Project their daily work priority.  This should be huge in preventing future delays.

3.  We have given better tools to the CentOS QA team.  They now see the packages that we are building in the QA IRC channel, the status of the builds on the QA mailing list and we have a scratch build area setup where they can do test builds to help us develop patches for issues, etc.  We have also been using trello for collaboration during point release time to get updates out faster.

The end result of all of these efforts have shown that we can get packages out in a timely manner.  Since January 2012, there have been no updates (outside of point releases) that have taken more than 3 days to push.  In fact, only one package during the entire time has taken more than one day to get released.  As I mentioned before, the last two point releases were also completed very quickly.  Here is some press on that:

5.8 Release

6.2 Release

The bottom line is that the CentOS Project has expended great effort to ensure that point releases of current products will be released in a timely fashion.  With the changes we have put in place and with the ability to do CentOS as our main daily work focus the future looks quite good for timely CentOS updates.

DRBD 8.3.12 for CentOS-5 in testing

The ELRepo Project has DRBD packages for CentOS-5 and CentOS-6, named drbd83-utils or drbd84-utils.  The CentOS Project does not want to maintain extra packages that exist in other places unless we need to change them ... so we are not going to create DRBD packages for CentOS-6.

Since CentOS-4 is being EOL'ed in less than a month, we are also not going to publish updates for the DRBD in CentOS-4.

This leaves the DRBD for CentOS-5 that are part of CentOS Extras.  Since these have been released for CentOS-5, we will continue to maintain the DRBD version 8.3.x  tree (drbd83) in CentOS Extras.

A new version of DRBD 8.3 (drbd83-8.3.12) has been released to the testing repository for CentOS-5.  You can see the details here:

DRBD 8.3.12 for CentOS-5

If you want to use DRBD 8.4.x for CentOS-5, rather than releasing it separately, the CentOS Project recommends that you use drbd84-utils from ELRepo (linked above).

For users who want to use the drbd83-8.3.12 version ... please test the version that is currently in CentOS Testing and provide feedback.  With enough feedback I will move the packages from testing to CentOS Extras.

NOTE:  These packages have now been moved to CentOS Extras and can be installed normally with yum.

CentOS in 2012

The first thing I want to do is congratulate Karanbir and Tasha on the birth of their new baby girl Millie. She is the quite cute ... hello Millie :)

The CentOS Project has spent much time and effort into getting a new build system in place for CentOS 6 that can generate good and timely builds, as well as inform us of newly released upstream SRPMS and keep the CentOS QA team informed when we build any new packages.

The release of CentOS-6.2 on 12/20/2011, in less than 2 weeks and at the same time as Oracle's OEL as noted on Distrowatch, is where we would like to have all our future releases be. I think that we should see the standard 2-4 week time frame for point releases and within 24 hours for updates now that we have this new build system in place.

We have also put a Continuous Release (CR) repository in place for both CentOS 5 and CentOS 6. This repository can be installed via the simple command:

yum install centos-release-cr

The purpose of the CR repository is to allow the CentOS Project to push some of the security updates if we are having issues with a point release build (like we did with both CentOS-6.0 and CentOS-6.1). If we are not going to meet the 2-4 week goal for our point release, we will push out the packages we have gotten to build properly while continuing to work on the problem packages. This repository is totally optional and was not needed with CentOS-6.2, but we recommend it be installed if you want to get your security updates as fast as possible.

Karanbir gets the credit for the new build system, called reimzul. It uses beanstalkd work queues and allows adding new builders to process the work as required.

The build system has the flexibility to allow us to import SRPMS into a git repo for packages we want to change, generate a new SRPM after edits for those packages, and submit those modified SRPMS into the work queues. It also allows for the submission of non-modified SRPMS directly without the need to import them into git. It automates several things that we have done in the past by hand (automatically knowing which packages are not built by CentOS (for example the RHN packages that deal with upstream subscriptions) and automatically copies multilib 32bit packages into the 64 bit tree. The system also reliably produces the Yum-Presto DeltaRPMS and metadata for minimizing download times for updates.

We do need to announce that CentOS-4 will be reaching the End Of Life at the end of February 2012. That means that there will be no more CentOS-4 updates after March 1st, 2012. If you are still using CentOS-4, you need to upgrade to CentOS-5 or CentOS-6 or switch to Red Hat's paid Extended Update Support for EL4 to continue to get updates. Please see the CentOS-4 EOL announcement for more details.

So, news on the CentOS front for 2012 is very promising and we are looking forward to great things in the new year.