Have pkgsrc building make use of SMP?

[Georg Schwarz]

Is there a way to make pkgsrc building make use of more than one CPU?

[Hubert Feyrer]

Post by Georg Schwarz
Is there a way to make pkgsrc building make use of more than one CPU?

Not yet.
The way to go is pass -j17 as MAKE_FLAGS to the Makefile being called for
the pkg to build. If you want to work on this, I may be able to review.


- Hubert

[Hubert Feyrer]

The way to go is pass -j17 as MAKE_FLAGS to the Makefile being called for the
pkg to build. If you want to work on this, I may be able to review.

For an implementation, try the following patch with MAKE_J=-j3 in
/etc/mk.conf:

Index: bsd.pkg.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/bsd.pkg.mk,v
retrieving revision 1.1607
diff -u -r1.1607 bsd.pkg.mk
--- bsd.pkg.mk 5 Apr 2005 03:18:00 -0000 1.1607
+++ bsd.pkg.mk 5 Apr 2005 12:28:03 -0000
@@ -2182,7 +2182,7 @@
# Build

BUILD_DIRS?= ${WRKSRC}
-BUILD_MAKE_FLAGS?= ${MAKE_FLAGS}
+BUILD_MAKE_FLAGS?= ${MAKE_J} ${MAKE_FLAGS}

.PHONY: do-build
.if !target(do-build)



What I'm afraid of is the non-pleasant, hard part - that pkg Makefiles
won't properly handle -j. :)


- Hubert

[Lars Nordlund]

Post by Hubert Feyrer

The way to go is pass -j17 as MAKE_FLAGS to the Makefile being called for the
pkg to build. If you want to work on this, I may be able to review.

For an implementation, try the following patch with MAKE_J=-j3 in
Index: bsd.pkg.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/bsd.pkg.mk,v
retrieving revision 1.1607
diff -u -r1.1607 bsd.pkg.mk
--- bsd.pkg.mk 5 Apr 2005 03:18:00 -0000 1.1607
+++ bsd.pkg.mk 5 Apr 2005 12:28:03 -0000
@@ -2182,7 +2182,7 @@
# Build
BUILD_DIRS?= ${WRKSRC}
-BUILD_MAKE_FLAGS?= ${MAKE_FLAGS}
+BUILD_MAKE_FLAGS?= ${MAKE_J} ${MAKE_FLAGS}
.PHONY: do-build
.if !target(do-build)
What I'm afraid of is the non-pleasant, hard part - that pkg Makefiles
won't properly handle -j. :)

It would be really cool to be able to just have pkgsrc build multiple
packages at the same time.

Example:

'foo' depends upon 'bar' and 'rab'. Neither of 'bar' or 'rab' depend on
the other.

# cd /usr/pkgsrc/meta-pkgs/foo; make -j 2
<pkgsrc spawns two build processes, one for bar and one for rab. when
both are done the build of foo continues..>


This way the various packages' broken build systems would not be an
obsticle!


Since the pkgsrc make knows about the dependancies it feels like this
should be possible to implement..


Best regards
Lars Nordlund

[Hubert Feyrer]

Post by Lars Nordlund
It would be really cool to be able to just have pkgsrc build multiple
packages at the same time.

Do you understand the difference of your suggestion to mine?

Why do you think running the whole pkgsrc infrastructure on two cpus in
parallel (with all the cache impact etc.), and adding the necessary
locking infrastructure to ensure dependencies are built before
applications that need them is better than just go and utilize all CPUs
with the compiler (which is already in cache) for all the CPUs in
parallel, and go for building one thing after another?

If you're keen on debugging such locking problems, why not have a
look at why make has issues with make -jX (X>1)?


- Hubert

[Lars Nordlund]

Post by Hubert Feyrer

Post by Lars Nordlund
It would be really cool to be able to just have pkgsrc build multiple
packages at the same time.

Do you understand the difference of your suggestion to mine?

Sure.

Post by Hubert Feyrer
Why do you think running the whole pkgsrc infrastructure on two cpus in
parallel (with all the cache impact etc.), and adding the necessary
locking infrastructure to ensure dependencies are built before
applications that need them is better than just go and utilize all CPUs
with the compiler (which is already in cache) for all the CPUs in
parallel, and go for building one thing after another?
If you're keen on debugging such locking problems, why not have a
look at why make has issues with make -jX (X>1)?
From what I have seen it is quite common for packages to not be make -j

safe. Securing those will take time.

On the other hand it is no problem building several packages from pkgsrc
at once, given their dependancy list does not clash (and PKGSRC_LOCKTYPE
will kick in, but this will of course just utilize one CPU since the
other build process is sleep waiting for the first one to finish).

Don't you ever build for example emacs and xmms at the same time, given
that you already have X and possibly other basic packages installed?


/Lars

[Hubert Feyrer]

Post by Lars Nordlund
Don't you ever build for example emacs and xmms at the same time, given
that you already have X and possibly other basic packages installed?

no. :)

So you basically want to work around the bug of make -jX not working
properly... hard to say anything against that.


- Hubert

[Lars Nordlund]

Post by Hubert Feyrer

Post by Lars Nordlund
Don't you ever build for example emacs and xmms at the same time, given
that you already have X and possibly other basic packages installed?

no. :)

Ok. Quite pointless on UP systems unless you have slow download times on
many of the distfiles, but on SMP systems.. mmmm :-)

Post by Hubert Feyrer
So you basically want to work around the bug of make -jX not working
properly... hard to say anything against that.

Is that a bug in make? I thought it was just a missing dependancy in
some makefile?


Best regards
Lars Nordlund

[Martin Husemann]

Post by Hubert Feyrer
So you basically want to work around the bug of make -jX not working
properly...

What bug in make(1) are you talking about here? What PR number?

Martin

[Hubert Feyrer]

Post by Martin Husemann

Post by Hubert Feyrer
So you basically want to work around the bug of make -jX not working
properly...

What bug in make(1) are you talking about here? What PR number?

The bug you'll likely see when you run this:
cd /usr/pkgsrc/foo/bar
env MAKE_FLAGS=-j17 make

AFAIK there's no PR on this.


- Hubert

[Martin Husemann]

Post by Hubert Feyrer
cd /usr/pkgsrc/foo/bar
env MAKE_FLAGS=-j17 make

I had no foo/bar pkg.
If you realy mean its a generic pkgsrc mk/* bug (or generic make(1) bug),
that should be fixable. Note that -j implies compatibility mode turned
off, so it may trigger makefile bugs that do not show up without it.

Post by Hubert Feyrer
AFAIK there's no PR on this.

Change that!

Martin

[Hubert Feyrer]

Post by Martin Husemann

Post by Hubert Feyrer
AFAIK there's no PR on this.

Change that!

Yes, but that needs time to dig up a testcase etc. - I'd be glad if
someone else (Adam? :) could do this.


- Hubert

[Havard Eidnes]

Post by Hubert Feyrer

Post by Lars Nordlund
Don't you ever build for example emacs and xmms at the same time, given
that you already have X and possibly other basic packages installed?

no. :)
So you basically want to work around the bug of make -jX not working
properly... hard to say anything against that.

Isn't what you are thinking of here "the problem that several packages
have not been prepared for parallel build using 'make -jX' with X>1"?

If that's the case, then it's not so much a case of a bug in make, but
rather a problem with insufficiently specified dependencies in the
packages' Makefiles. Since we're talking about third-party software
typically not exposed to parallel builds, I'd say it would be
difficult at best to fix all of pkgsrc's package Makefiles to play
nice with "make -jX", so following that avenue of attack to making
good use for more than one CPU is going to be an uphill battle at
best.

Going into whishlist mode: Making use of multiple CPUs on a single
physical machine could be the relatively simple first step in making
it possible to spread package builds across multiple machines in order
to speed up package builds for a given architecture... Clustered bulk
builds, anyone?

Regards,

- Håvard

[Adam Ciarcinski]

Post by Havard Eidnes

Post by Hubert Feyrer

Post by Lars Nordlund
Don't you ever build for example emacs and xmms at the same time, given
that you already have X and possibly other basic packages installed?

no. :)
So you basically want to work around the bug of make -jX not working
properly... hard to say anything against that.

Isn't what you are thinking of here "the problem that several packages
have not been prepared for parallel build using 'make -jX' with X>1"?
If that's the case, then it's not so much a case of a bug in make, but
rather a problem with insufficiently specified dependencies in the
packages' Makefiles. Since we're talking about third-party software
typically not exposed to parallel builds, I'd say it would be
difficult at best to fix all of pkgsrc's package Makefiles to play
nice with "make -jX", so following that avenue of attack to making
good use for more than one CPU is going to be an uphill battle at
best.
Going into whishlist mode: Making use of multiple CPUs on a single
physical machine could be the relatively simple first step in making
it possible to spread package builds across multiple machines in order
to speed up package builds for a given architecture... Clustered bulk
builds, anyone?

The problem is not with packages. It is either bmake, or pkgsrc
Makefiles, or both.

Speaking from experience :)
Adam

[Hubert Feyrer]

Post by Havard Eidnes
If that's the case, then it's not so much a case of a bug in make, but
rather a problem with insufficiently specified dependencies in the
packages' Makefiles. Since we're talking about third-party software
typically not exposed to parallel builds, I'd say it would be
difficult at best to fix all of pkgsrc's package Makefiles to play
nice with "make -jX", so following that avenue of attack to making
good use for more than one CPU is going to be an uphill battle at
best.

Yes. And given that GNUmake seems to deal ok, I blame BSDmake.

Given that the target-structure of a Makefile contains all the data on
what can/should be done an in parallel, and what to wait for before
proceeding to the next target, I wonder if that can't be used.
But I won't touch make. :)

Post by Havard Eidnes
Going into whishlist mode: Making use of multiple CPUs on a single
physical machine could be the relatively simple first step in making
it possible to spread package builds across multiple machines in order
to speed up package builds for a given architecture... Clustered bulk
builds, anyone?

Johnny wanted to talk about his "bobac" cluster bulkbuild thing at
pkgsrcCon. I have no idea what it is. Probably more than just an extended
"make".


- Hubert

[Lars Nordlund]

Post by Lars Nordlund
It would be really cool to be able to just have pkgsrc build multiple
packages at the same time.
'foo' depends upon 'bar' and 'rab'. Neither of 'bar' or 'rab' depend on
the other.
# cd /usr/pkgsrc/meta-pkgs/foo; make -j 2
<pkgsrc spawns two build processes, one for bar and one for rab. when
both are done the build of foo continues..>

After some thought and a quick trial and error I found that a temporary
generated Makefile like this would solve my problem:
------
foo: bar rab

bar:
(cd /usr/pkgsrc/editors/emacs; $(MAKE) install)

rab:
(cd /usr/pkgsrc/net/lftp; $(MAKE) install)
-------

'make -j 2 foo' will now build emacs and lftp at the same time. I had
some problems with ALLOW_VULNERABLE_PACKAGES but that should be possible
to solve.


Now all that is needed is a tool which will generate these kind of
temporary makefiles based on the wanted package's dependancy list. I am
checking out pkgtools/pkgdep right now (actually I am stuck waiting for
make index.. :-)

Ah, how nice. Another pet project. I have to start with this after I buy
a dual core cpu some day.. :-)


Best regards,
Lars Nordlund

[Adam Ciarcinski]

Post by Georg Schwarz
Is there a way to make pkgsrc building make use of more than one CPU?

On MacOSX I force pkgsrc to use (gnu)make -j2.
Unfortunatelly bmake doesn't really want to handle multiple jobs, or
pkgsrc does not handle "bmake -j2" correctly.
Anyone care to fix it?. :)

Kind regards,
Adam

[Hubert Feyrer]

Unfortunatelly bmake doesn't really want to handle multiple jobs, or pkgsrc
does not handle "bmake -j2" correctly.
Anyone care to fix it?. :)

Not me, but that's what I remember from last time I played with this, too.
That's some time ago though. I think this would be worth opening a PR, to
document error cases, how to reproduce, etc.

(With the final goal that as few Makefiles in pkgs need to be touched as
possible :)


- Hubert

[Georg Schwarz]

setting MAKE_FLAGS+=-j2 in /etc/mk.conf seems to do the trick at least
for packages using gmake. Of course one could try forcing the use of gmake,
but this would probably break a few packages.

[Hubert Feyrer]

Post by Georg Schwarz
setting MAKE_FLAGS+=-j2 in /etc/mk.conf seems to do the trick at least
for packages using gmake. Of course one could try forcing the use of gmake,
but this would probably break a few packages.

... and forcing GNUmake on people won't make you a lot of friends either. :)


- Hubert

[Georg Schwarz]

Post by Hubert Feyrer

Post by Georg Schwarz
setting MAKE_FLAGS+=-j2 in /etc/mk.conf seems to do the trick at least
for packages using gmake. Of course one could try forcing the use of gmake,
but this would probably break a few packages.

... and forcing GNUmake on people won't make you a lot of friends either. :)

this would of course also be an individual setting, e.g. by adding
USE_GNU_TOOLS+= make in /etc/mk.conf; however I am afraid it would break
a few packages that require BSD make.

[Hubert Feyrer]

Post by Georg Schwarz
this would of course also be an individual setting, e.g. by adding
USE_GNU_TOOLS+= make in /etc/mk.conf; however I am afraid it would break
a few packages that require BSD make.

It will also make those people unhappy that prefer BSDmake to GNUmake for
"normal" operation. I've heared there are some fanatics out there (hi
Matt! :)


- Hubert

[J Chapman Flack]

Post by Adam Ciarcinski
The problem is not with packages. It is either bmake, or pkgsrc
Makefiles, or both.
Speaking from experience :)

So, what *is* the experience? If there isn't a PR or a reasonably clear
example, how can the problem be fixed?

I'm not questioning the experience, just looking for details. I build
kernels with -j2 on a two-processor machine with no trouble. Seems to me
(if I try to imagine writing it) that writing a paralle make should not be
very difficult, but writing parallel makeFILES could be quite hard to get
right. Are you sure that's not the source of problems?

-Chap