Building packages at scale

[Jonathan Perkin]

Various people have asked me for more details on our bulk build setup,
now that it is able to perform a full pkgsrc bulk build from scratch
(14,000+ packages) in 4.5 hours.

I've written a post which details some of these bits and the recent
performance work I've been doing, it's here:

http://www.perkin.org.uk/posts/building-packages-at-scale.html

I'm happy to answer any questions if you want me to go into more
detail about any aspects of our setup.

[Greg Troxel]

I'm curious if you are considering adding support for ccache to your
bulk setup. I realize that this departs from "from scratch" and has
some risk of trouble, but in general ccache speeds up building quite a
lot and I've been using it (not with pbulk).

[Jonathan Perkin]

Post by Greg Troxel
I'm curious if you are considering adding support for ccache to your
bulk setup. I realize that this departs from "from scratch" and
has some risk of trouble, but in general ccache speeds up building
quite a lot and I've been using it (not with pbulk).

It's something that I'd like to try at some point just to get some
hard data on it, but it's low priority as I'm almost certain it won't
help.

If anything it will probably reduce throughput, due to having to go to
NFS for each cc invocation, whereas I've been working to eliminate NFS
from the build side of things as much as possible - for example the
pkgsrc checkout is rsync'd to each build host and then loopback
mounted into chroots prior to starting the build.

But yeh, that's just a guess, so it'd be good to get some numbers on
it.

[Roy Marples]

Post by Jonathan Perkin
Various people have asked me for more details on our bulk build setup,
now that it is able to perform a full pkgsrc bulk build from scratch
(14,000+ packages) in 4.5 hours.
I've written a post which details some of these bits and the recent
http://www.perkin.org.uk/posts/building-packages-at-scale.html
I'm happy to answer any questions if you want me to go into more
detail about any aspects of our setup.

I'd just like to thank you and Joerg for the awesome work :)

Roy