pterjan's diary

2013-01-03

Let's build it all over again... and again

Happy new year everyone. By posting something now I make sure I will post at least one this year :)

During Christmas I have been busy hacking an automated rebuild of Mageia.

I am using a single machine, and building only on x86_64. The machine has 12 cores and 32G ram so I am building 6 packages in parallel with -l12 -j12 on tmpfs, with some swap to prevent very large packages to break the build of others. It takes about 30 hours to build just under 11000 packages, creating a clean chroot each time.

I was happily surprised by the results of the first rebuild, 90.1% success (9845 packages built successfuly), and even more with the second (93.3%) after fixing a few packages breaking many others. I was less happy when the results went down in the recent runs (new version of rpm, new version of automake breaking 304 packages by removing AM_CONFIG_HEADER, %libexec changed to be different from %lib in our rpm config...), but at least we have the number, and better, the list of packages to fix!

Current main problems and limitations:

Packages only get rebuilt on x86_64

I add -lN flag in addition of -jN to _smp_mflags macro, this works well and allows me to build many packages in parallel to maximize cpu utilization but a few packages fail (at least the ones giving the flag to waf as waf handles -j but complains about -l)

There are some cases where the chroot is not removed (it happens for 175 chroots out of about 11000, but that's 73G used at the end of a rebuild that I manually clean, and because of that I am using a 100G swap file :) )

The build is started manually, and the results are copied manually

It runs as my user, in my home directory

The website is quite ugly, and lacking many things like the history of a package

A build creates 1.5G logs, and compressing them only gets it down to 500G at the cost of not being readable directly in the web browser, so I will have to remove some quite soon

Ah, and I will be attending FOSDEM as usual, so see you near some Belgian beers!

[Add a TSUKKOMI]

2004|06|07|08|09|11|

2005|01|02|05|06|07|08|09|10|11|12|

2006|01|02|03|06|08|10|11|12|

2007|01|02|03|04|05|06|07|08|09|10|11|12|

2008|01|03|04|05|06|07|08|09|10|11|12|

2009|01|02|03|04|05|06|07|08|09|10|11|12|

2010|01|02|03|04|05|06|07|08|09|10|12|

2011|02|04|06|

2012|01|05|11|

2013|01|02|04|06|

2014|02|

2015|06|

2017|05|07|12|