Top Latest Append

pterjan's diary


  Mageia Let's build it all over again... and again

Happy new year everyone. By posting something now I make sure I will post at least one this year :)

During Christmas I have been busy hacking an automated rebuild of Mageia.

I am using a single machine, and building only on x86_64. The machine has 12 cores and 32G ram so I am building 6 packages in parallel with -l12 -j12 on tmpfs, with some swap to prevent very large packages to break the build of others. It takes about 30 hours to build just under 11000 packages, creating a clean chroot each time.

I was happily surprised by the results of the first rebuild, 90.1% success (9845 packages built successfuly), and even more with the second (93.3%) after fixing a few packages breaking many others. I was less happy when the results went down in the recent runs (new version of rpm, new version of automake breaking 304 packages by removing AM_CONFIG_HEADER, %libexec changed to be different from %lib in our rpm config...), but at least we have the number, and better, the list of packages to fix!

Current main problems and limitations:

  • Packages only get rebuilt on x86_64
  • I add -lN flag in addition of -jN to _smp_mflags macro, this works well and allows me to build many packages in parallel to maximize cpu utilization but a few packages fail (at least the ones giving the flag to waf as waf handles -j but complains about -l)
  • There are some cases where the chroot is not removed (it happens for 175 chroots out of about 11000, but that's 73G used at the end of a rebuild that I manually clean, and because of that I am using a 100G swap file :) )
  • The build is started manually, and the results are copied manually
  • It runs as my user, in my home directory
  • The website is quite ugly, and lacking many things like the history of a package
  • A build creates 1.5G logs, and compressing them only gets it down to 500G at the cost of not being readable directly in the web browser, so I will have to remove some quite soon
  • Ah, and I will be attending FOSDEM as usual, so see you near some Belgian beers!