Package split or package options?

[Edgar Fuß]

I'm asking myself wheter, in a particular case, it's preferable to use
options to selectively build components or use split into seperate packages?

I have imported nss-pam-ldapd. This thing consists of three parts: a daemon
(nslcd) and two clients, one for NSS and one for PAM (actually, there is a
fourth part, which appears to be an alterate version of the daemon written
in Python).
Unfortunately, almost every combination of wanting a subset of the components
makes sense: you may want NSS, but not PAM; you may want o stick with
nss_ldap and replace pam-ldap with the nss-pam-ldapd client; you may even
want a client, but not the daemon: there's a OpenLDAP overlay (slapo-nssov)
speaking the same protocol, but built into the LDAP server.

So, should I use package options controlling which of the three (or four)
components gets built (less overhead, less likely to accidently install
a client, but no server) or split it into three packages plus a
nss-pam-ldapd/Makefile.common (better for binary packages)?

Same question for OpenLDAP's slapo-nssov: I've currently implemented it as
an openldap-server option---should I factor out a package?

[Jonathan Perkin]

Post by Edgar Fuß
I'm asking myself wheter, in a particular case, it's preferable to use
options to selectively build components or use split into seperate packages?
I have imported nss-pam-ldapd. This thing consists of three parts: a daemon
(nslcd) and two clients, one for NSS and one for PAM (actually, there is a
fourth part, which appears to be an alterate version of the daemon written
in Python).
Unfortunately, almost every combination of wanting a subset of the components
makes sense: you may want NSS, but not PAM; you may want o stick with
nss_ldap and replace pam-ldap with the nss-pam-ldapd client; you may even
want a client, but not the daemon: there's a OpenLDAP overlay (slapo-nssov)
speaking the same protocol, but built into the LDAP server.
So, should I use package options controlling which of the three (or four)
components gets built (less overhead, less likely to accidently install
a client, but no server) or split it into three packages plus a
nss-pam-ldapd/Makefile.common (better for binary packages)?
Same question for OpenLDAP's slapo-nssov: I've currently implemented it as
an openldap-server option---should I factor out a package?

As a purveyor of a binary package distribution, I would always prefer
separate packages. Users are going to have different needs, and it
sucks to force a lot of them to have to build from source.

Especially in the cases above where appear to be distinct sets of
functionality.

[David Holland]

Post by Jonathan Perkin
As a purveyor of a binary package distribution, I would always prefer
separate packages. Users are going to have different needs, and it
sucks to force a lot of them to have to build from source.

...and I would like to figure out how to make options more useful for
binary packages.

In this case though it sounds like multiple packages is probably a
good idea.

[Fredrik Pettai]

Post by David Holland

Post by Jonathan Perkin
As a purveyor of a binary package distribution, I would always prefer
separate packages. Users are going to have different needs, and it
sucks to force a lot of them to have to build from source.

...and I would like to figure out how to make options more useful for
binary packages.

I've just thought of that too…

We always have to rebuild some of packages we use after a binary upgrade (because the defaults for some of the packages are not containing the knobs we need to have enabled.

I noticed the way MacPorts done it and it looks reasonable ok. They name the package "variants" as: pkgname @version+option(s). Perhaps pkgsrc could name those alternative bin pkgs as: pkgname+option(s)-version ?

OTOH, it would too time-consuming to build all pkgs with all the different make-option combinations, so instead a new variable in the Makefile (eg. ALSO_MAKE_BIN_PKG_WITH=tls (ok, not an optimal name, but you get the idea…)) could be created to make the "extra" bin pkgs + make-option(s) that's missing from PKG_DEFAULT_OPTIONS.
Then the maintainer (or any pkgsrc-dev) could add any new pkg bin options that is asked for, without influencing the old defaults.

/P

[Anthony Mallet]

On Wednesday, at 20:56, Fredrik Pettai wrote:
| I noticed the way MacPorts done it and it looks reasonable ok. They name the
| package "variants" as: pkgname @version+option(s). Perhaps pkgsrc could name
| those alternative bin pkgs as: pkgname+option(s)-version ?

I also did that in robotpkg (http://robotpkg.openrobots.org), which is a fork
of pkgsrc (targetting more specific goals and users and packages, hence the
fork).

Binary packages are named according to the following scheme:
${PKGBASE}-${PKGVERSION}~${PKG_OPTIONS}.tgz

$PKG_OPTIONS are joined with the "+" char.

This works well (tested on the ~300 pkgs in robotpkg), and required only
_minimal_ changes to pkgtools/pkg_install and can easily be "backported" to
pkgsrc. If people are interested I can give more details.

The hard part is to manage dependencies properly on options (see below).

| OTOH, it would too time-consuming to build all pkgs with all the different
| make-option combinations,

Sure. I also have developed an awk script that is able to expand patterns for
the above naming scheme, and I can define in robotpkg.conf (read: mk.conf) sets
of packages that can be used for bulk builds (or also for bulk installation
from source).

E.g.:
MY_SET = */* would expand to all possible packages with default options
(like current pbulk)
MY_SET = */*~* would expand to all possible packages (untracktable, though).
MY_SET = devel/*~foo (all packages in devel with option "foo" set)
MY_SET = */bar~!foo (all bar packages in any category without option "foo" set)
MY_SET = */bar~foo+* (all bar packages in any category with option "foo" and
all combination of other options)
etc.

This script is also used to expand dependencies with option requirements, e.g.:
DEPENDS+= foo>=1.0~bar or DEPENDS+= foo>=1.0~!bar+x11 etc.

This would probably require more work to integrate in pkgsrc/pbulk.

[Anthony Mallet]

On Thursday, at 10:48, Anthony Mallet wrote:
| On Wednesday, at 20:56, Fredrik Pettai wrote:
| | I noticed the way MacPorts done it and it looks reasonable ok. They name the
| | package "variants" as: pkgname @version+option(s). Perhaps pkgsrc could name
| | those alternative bin pkgs as: pkgname+option(s)-version ?
|
| I also did that in robotpkg (http://robotpkg.openrobots.org), which is a fork
| of pkgsrc (targetting more specific goals and users and packages, hence the
| fork).
|
| Binary packages are named according to the following scheme:
| ${PKGBASE}-${PKGVERSION}~${PKG_OPTIONS}.tgz
|
| $PKG_OPTIONS are joined with the "+" char.

Hi,

So.. back on this topic!

Here are the 3 patches for pkgtools/pkg_install implementing the aforementioned
naming scheme in robotpkg. As such, the patches are supposed to be harmless,
i.e. pkgsrc/pkg_install will work exactly as before (but of course implement
the necessary bits for the new naming scheme, mostly affecting pkg_admin pmatch).

Patches 1 & 2 are just code refactoring, patch 3 implements the real stuff.
There is my git commit message in the patch, hopefully explaining what is
implemented.

http://www.netbsd.org/~tho/0001-pkgtools-pkg_install-Factorize-code-adding-a-version.patch
http://www.netbsd.org/~tho/0002-pkgtools-pkg_install-Do-not-reinvent-pkg_match-for-m.patch
http://www.netbsd.org/~tho/0003-pkgtools-pkg_install-Introduce-the-tags-package-suff.patch

Examples:
pkg_admin pmatch 'x>=1.0~a' x-1.0 ->false
pkg_admin pmatch 'x>=1.0~a' x-1.0~a ->true
pkg_admin pmatch 'x>=1.0~a' x-0.0~a ->false

pkg_admin pmatch 'x~!a' x-1.0~a ->false
pkg_admin pmatch 'x~!a' x-1.0 ->true

pkg_admin pmatch 'x~a+b' x-1.0 ->false
pkg_admin pmatch 'x~a+b' x-1.0~a ->false

pkg_admin pmatch 'x~*' x-1.0 ->true
pkg_admin pmatch 'x~*' x-1.0~a ->true

pkg_admin pmatch 'x~!*' x-1.0 ->true
pkg_admin pmatch 'x~!*' x-1.0~a ->false

etc.

Some remarks:
- The matching algorithm with patterns on boolean variables is in general a
complex problem (known as SAT, I think), and the one implemented tries to
keep things simple and intuitve. But it's not been formalized, so there must
be some caveats ...

- As wiz@ already noted, the use of the not operator `!' might be replaced by a
`^' to avoid quoting issues, and instead of using pkgname-version~options,
one could also use pkgname~options-version¹.

I also provide an *untested* example patch for mk/bsd.pkg.mk that changes the
naming of the packages so that the ~options 'tag' is appended to PKGNAME. This
might possibly break pkgsrc in some places, but the patch is mostly here to
give you a concrete idea of the (almost only) change I did in robotpkg.

http://www.netbsd.org/~tho/PKGNAME.diff

Next step would be to review the places where this will break, but a
technical/philosophical discussion about all this must probably take place
beforehand. IIRC there was almost no issue when I did that in robotpkg, but the
code has diverged quite a lot from pkgsrc in some places, so I cannot tell
precisely right now.

So ... what do you think :)

--
Footnotes
¹ In robotpkg I have a shell script converting .tgz bsd binary package into
.deb ones (dpkg), and I had to name the .deb packages according to the second
alternative (i.e. pkgname~otpions-version), because dpkg is much more
restrictive about the name of the packages.

[Thomas Klausner]

Post by Anthony Mallet
Here are the 3 patches for pkgtools/pkg_install implementing the aforementioned
naming scheme in robotpkg. As such, the patches are supposed to be harmless,
i.e. pkgsrc/pkg_install will work exactly as before (but of course implement
the necessary bits for the new naming scheme, mostly affecting pkg_admin pmatch).
Patches 1 & 2 are just code refactoring, patch 3 implements the real stuff.
There is my git commit message in the patch, hopefully explaining what is
implemented.
http://www.netbsd.org/~tho/0001-pkgtools-pkg_install-Factorize-code-adding-a-version.patch

Ok, commit this after the branch.

Post by Anthony Mallet
http://www.netbsd.org/~tho/0002-pkgtools-pkg_install-Do-not-reinvent-pkg_match-for-m.patch

Good too.

Post by Anthony Mallet
http://www.netbsd.org/~tho/0003-pkgtools-pkg_install-Introduce-the-tags-package-suff.patch
From the patch age I assume you've been running this for years, so commit this too please.
pkg_admin pmatch 'x>=1.0~a' x-1.0 ->false
pkg_admin pmatch 'x>=1.0~a' x-1.0~a ->true
pkg_admin pmatch 'x>=1.0~a' x-0.0~a ->false
pkg_admin pmatch 'x~!a' x-1.0~a ->false
pkg_admin pmatch 'x~!a' x-1.0 ->true
pkg_admin pmatch 'x~a+b' x-1.0 ->false
pkg_admin pmatch 'x~a+b' x-1.0~a ->false
pkg_admin pmatch 'x~*' x-1.0 ->true
pkg_admin pmatch 'x~*' x-1.0~a ->true
pkg_admin pmatch 'x~!*' x-1.0 ->true
pkg_admin pmatch 'x~!*' x-1.0~a ->false
etc.
- The matching algorithm with patterns on boolean variables is in general a
complex problem (known as SAT, I think), and the one implemented tries to
keep things simple and intuitve. But it's not been formalized, so there must
be some caveats ...
`^' to avoid quoting issues, and instead of using pkgname-version~options,
one could also use pkgname~options-version¹.

I'd like some feedback from others here too what they prefer.
Has anyone looked what Debian or OpenBSD use here?

Cheers,
Thomas

[Alistair Crooks]

Post by Thomas Klausner

Post by Anthony Mallet
Here are the 3 patches for pkgtools/pkg_install implementing the aforementioned
naming scheme in robotpkg. As such, the patches are supposed to be harmless,
i.e. pkgsrc/pkg_install will work exactly as before (but of course implement
the necessary bits for the new naming scheme, mostly affecting pkg_admin pmatch).
Patches 1 & 2 are just code refactoring, patch 3 implements the real stuff.
There is my git commit message in the patch, hopefully explaining what is
implemented.
http://www.netbsd.org/~tho/0001-pkgtools-pkg_install-Factorize-code-adding-a-version.patch

Ok, commit this after the branch.

Post by Anthony Mallet
http://www.netbsd.org/~tho/0002-pkgtools-pkg_install-Do-not-reinvent-pkg_match-for-m.patch

Good too.

Post by Anthony Mallet
http://www.netbsd.org/~tho/0003-pkgtools-pkg_install-Introduce-the-tags-package-suff.patch
From the patch age I assume you've been running this for years, so commit this too please.

Personally, I have my doubts about this - I really have problems with
its scalability, and think, while it might work for one, or even two
options, it doesn't scale up beyond that.

Post by Thomas Klausner

Post by Anthony Mallet
pkg_admin pmatch 'x>=1.0~a' x-1.0 ->false
pkg_admin pmatch 'x>=1.0~a' x-1.0~a ->true
pkg_admin pmatch 'x>=1.0~a' x-0.0~a ->false
pkg_admin pmatch 'x~!a' x-1.0~a ->false
pkg_admin pmatch 'x~!a' x-1.0 ->true
pkg_admin pmatch 'x~a+b' x-1.0 ->false
pkg_admin pmatch 'x~a+b' x-1.0~a ->false
pkg_admin pmatch 'x~*' x-1.0 ->true
pkg_admin pmatch 'x~*' x-1.0~a ->true
pkg_admin pmatch 'x~!*' x-1.0 ->true
pkg_admin pmatch 'x~!*' x-1.0~a ->false
etc.
- The matching algorithm with patterns on boolean variables is in general a
complex problem (known as SAT, I think), and the one implemented tries to
keep things simple and intuitve. But it's not been formalized, so there must
be some caveats ...
`^' to avoid quoting issues, and instead of using pkgname-version~options,
one could also use pkgname~options-version?.

I have only ever seen ^ used as a negation or complement operator at
the start of character classes in regexps, which I think is an
esoteric case. I also don't think that quoting should have any
bearing on this.

I'd prefer to see more discussion before anything gets committed.

Best,
Alistair

[Tim Zingelman]

I am concerned about how this will affect our ability to correctly
produce patterns for the pkg-vulnerabilities file (used by
audit-packages.)
We too often have a hard time getting all the patterns right, and
unless I misunderstand how this change to package names and new
matching will work this will make things significantly harder.
Perhaps an example will help me understand... If a package has 6
possible non-mutually exclusive options, 2 of which are default and
there is a vulnerability in the base package (with or without options)
how do we specify a pattern? To be more concrete lets say the
vulnerability is found in versions of pkgname starting with version 4
and is fixed in pkgname-4.3.2nb1, and lets call the options aaa, bbb,
ccc, ddd, eee & fff, with bbb & fff being default options. Prior to
these proposed changes we would use the pattern pkgname>=4<4.3.2nb1
I appreciate any assistance you can provide in helping me understand
the fine details here.

Thanks,

- Tim

Post by Alistair Crooks

Post by Thomas Klausner

Post by Anthony Mallet
Here are the 3 patches for pkgtools/pkg_install implementing the aforementioned
naming scheme in robotpkg. As such, the patches are supposed to be harmless,
i.e. pkgsrc/pkg_install will work exactly as before (but of course implement
the necessary bits for the new naming scheme, mostly affecting pkg_admin pmatch).
Patches 1 & 2 are just code refactoring, patch 3 implements the real stuff.
There is my git commit message in the patch, hopefully explaining what is
implemented.
http://www.netbsd.org/~tho/0001-pkgtools-pkg_install-Factorize-code-adding-a-version.patch

Ok, commit this after the branch.

Post by Anthony Mallet
http://www.netbsd.org/~tho/0002-pkgtools-pkg_install-Do-not-reinvent-pkg_match-for-m.patch

Good too.

Post by Anthony Mallet
http://www.netbsd.org/~tho/0003-pkgtools-pkg_install-Introduce-the-tags-package-suff.patch
From the patch age I assume you've been running this for years, so commit this too please.

Personally, I have my doubts about this - I really have problems with
its scalability, and think, while it might work for one, or even two
options, it doesn't scale up beyond that.

Post by Thomas Klausner

Post by Anthony Mallet
pkg_admin pmatch 'x>=1.0~a' x-1.0 ->false
pkg_admin pmatch 'x>=1.0~a' x-1.0~a ->true
pkg_admin pmatch 'x>=1.0~a' x-0.0~a ->false
pkg_admin pmatch 'x~!a' x-1.0~a ->false
pkg_admin pmatch 'x~!a' x-1.0 ->true
pkg_admin pmatch 'x~a+b' x-1.0 ->false
pkg_admin pmatch 'x~a+b' x-1.0~a ->false
pkg_admin pmatch 'x~*' x-1.0 ->true
pkg_admin pmatch 'x~*' x-1.0~a ->true
pkg_admin pmatch 'x~!*' x-1.0 ->true
pkg_admin pmatch 'x~!*' x-1.0~a ->false
etc.
- The matching algorithm with patterns on boolean variables is in general a
complex problem (known as SAT, I think), and the one implemented tries to
keep things simple and intuitve. But it's not been formalized, so there must
be some caveats ...
`^' to avoid quoting issues, and instead of using pkgname-version~options,
one could also use pkgname~options-version?.

I have only ever seen ^ used as a negation or complement operator at
the start of character classes in regexps, which I think is an
esoteric case. I also don't think that quoting should have any
bearing on this.
I'd prefer to see more discussion before anything gets committed.
Best,
Alistair

[Anthony Mallet]

On Sunday, at 13:54, Tim Zingelman wrote:
| I am concerned about how this will affect our ability to correctly
| produce patterns for the pkg-vulnerabilities file (used by
| audit-packages.)
| We too often have a hard time getting all the patterns right, and
| unless I misunderstand how this change to package names and new
| matching will work this will make things significantly harder.
| Perhaps an example will help me understand... If a package has 6
| possible non-mutually exclusive options, 2 of which are default and
| there is a vulnerability in the base package (with or without options)
| how do we specify a pattern? To be more concrete lets say the
| vulnerability is found in versions of pkgname starting with version 4
| and is fixed in pkgname-4.3.2nb1, and lets call the options aaa, bbb,
| ccc, ddd, eee & fff, with bbb & fff being default options. Prior to
| these proposed changes we would use the pattern pkgname>=4<4.3.2nb1
| I appreciate any assistance you can provide in helping me understand
| the fine details here.

Well, if the options don't affect the vulnerability, pkgname>=4<4.3.2nb1 would
work just fine and match all packages in the version range whatever their
option.

But it the vulnerability is, say, only in option aaa (no matter if it's a
default option or not), then the vulnerable packages would be
pkgname>=4<4.3.2nb1~aaa

And if the vulnerability is instead present with all options but option bbb,
then then the vulnerable packages would be
pkgname>=4<4.3.2nb1~!bbb

('!' representing 'not', but it could be ^ as well or whatever char that is
deemed appropriate)

[Tim Zingelman]

Post by Anthony Mallet
| I am concerned about how this will affect our ability to correctly
| produce patterns for the pkg-vulnerabilities file (used by
| audit-packages.)
| We too often have a hard time getting all the patterns right, and
| unless I misunderstand how this change to package names and new
| matching will work this will make things significantly harder.
| Perhaps an example will help me understand... If a package has 6
| possible non-mutually exclusive options, 2 of which are default and
| there is a vulnerability in the base package (with or without options)
| how do we specify a pattern? To be more concrete lets say the
| vulnerability is found in versions of pkgname starting with version 4
| and is fixed in pkgname-4.3.2nb1, and lets call the options aaa, bbb,
| ccc, ddd, eee & fff, with bbb & fff being default options. Prior to
| these proposed changes we would use the pattern pkgname>=4<4.3.2nb1
| I appreciate any assistance you can provide in helping me understand
| the fine details here.
Well, if the options don't affect the vulnerability, pkgname>=4<4.3.2nb1 would
work just fine and match all packages in the version range whatever their
option.
But it the vulnerability is, say, only in option aaa (no matter if it's a
default option or not), then the vulnerable packages would be
pkgname>=4<4.3.2nb1~aaa
And if the vulnerability is instead present with all options but option bbb,
then then the vulnerable packages would be
pkgname>=4<4.3.2nb1~!bbb
('!' representing 'not', but it could be ^ as well or whatever char that is
deemed appropriate)

Thanks, that makes it more clear. I guess this is at worst break even
and at best an improvement for vulnerability matching.
I appreciate the clarification.

- Tim

[Anthony Mallet]

On Sunday, at 09:18, Alistair Crooks wrote:
| Personally, I have my doubts about this - I really have problems with
| its scalability, and think, while it might work for one, or even two
| options, it doesn't scale up beyond that.

In which context do you think this wouldn't scale properly?

From my experience, packages seldom need complex dependency requirements

regarding options, but I may be missing something.

| I'd prefer to see more discussion before anything gets committed.

The first two patches are unrelated to options and are just code
refactoring. Maybe I can just commit these for now?

http://www.netbsd.org/~tho/0001-pkgtools-pkg_install-Factorize-code-adding-a-version.patch
http://www.netbsd.org/~tho/0002-pkgtools-pkg_install-Do-not-reinvent-pkg_match-for-m.patch

[Anthony Mallet]

On Wednesday, at 10:26, Jonathan Perkin wrote:
| The main issue is dependencies, as you now have to pull in the correct
| binary package for the options involved, and how you express the
| matching in bl3 or DEPENDS.

Most of package options don't actually modify the dependencies. Of course, some
of them do. This is handled in robotpkg by the equivalent pkgsrc-like code:

Suppose a package has an option 'foo' that requires a dependency on bar built
with the foo option :

.if !empty(PKG_OPTIONS:Mfoo)
. include "../../category/bar/buildlink3.mk"
BUILDLINK_API_DEPENDS.bar+= bar~foo
.endif

The buildlink3 would define the dependency on bar (as usuall) and there is just
an additional line expressing the required 'foo' option on the bar package.

Then, the dependency resolution code merges the two requirements, for instance
if BUILDLINK_API_DEPENDS.bar ends up with the following two elements "bar>=42
bar~foo", they are merged into "bar>=42~foo", which is satisfied by any bar
package >=42 with the option foo enabled.

| And as soon as you want to introduce any PKG_OPTIONS which are handled
| by mk/pbulk/pbulk-index.mk then there is an explosion of combinations
| and difficulties in matching the correct version required.

Yes, there is an explosion if you want everything and this will definitely not
work. But, as I said, you don't want to build everything. I use a (very simple)
awk script that is able to expand lists like "foo~x11 foo~!x11" to for instance
build packages with and without x11, with all other options to their default
value.

| Given that I don't believe there is a way to solve these issues in the
| general case, and that you will need to be very careful in the options
| you allow to be supported, I don't see any advantage therefore in
| simply just providing additional packages for those instances, and
| this is the approach we will be taking.

The problem is mainly the conflicts on the filesystem of the binary package
names. You have to setup separate repos, duplicate many things. You cannot
verify that two given binary packages are compatible in terms of options, and
you cannot easily upgrade from one set of options to another set (e.g. with
pkgin).

[Anthony Mallet]

On Wednesday, at 04:24, Alistair Crooks wrote:
| On Mon, Mar 31, 2014 at 10:58:18AM +0200, Anthony Mallet wrote:
| > In which context do you think this wouldn't scale properly?
|
| [19:17:28] ***@netbsd-001 ...pkgsrc/www/curl [11626] > grep PKG_SUPPORTED_OPTIONS options.mk
| PKG_SUPPORTED_OPTIONS= inet6 libssh2 gssapi ldap rtmp libidn
| [19:17:40] ***@netbsd-001 ...pkgsrc/www/curl [11627] >

Sorry, I was not clear : I meant "what would not scale?".
I have many packages in robotpkg with 6 options or more, and I have not had any
"scalability" issue.

Of course, if you're talking about the number of different binary packages that
can be generated, this does not scale. For instance, the ~300 packages in
robotpkg can generate more than 9000 different binary packages in total. This
is not tractable in pratice, but you don't want to generate all possible binary
packages. Typically, you build only the default options and for some packages
a small set of "interesting" options combination.

| > The first two patches are unrelated to options and are just code
| > refactoring. Maybe I can just commit these for now?
| >
| > http://www.netbsd.org/~tho/0001-pkgtools-pkg_install-Factorize-code-adding-a-version.patch
|
| Yes, re-factoring, but the re-factored code is less readable,
| understandable, and obvious than the previous code. At a glance, I
| could see what
|
| pattern = xasprintf("%s-[0-9]*", pkgname);
|
| did. The following:
|
| pattern = addpkgwildcard(pkgname);
|
| is opaque. Moreover, the function itself where this is factored into
| is a one-liner:
|
| +char *
| +addpkgwildcard(const char *pkg)
| +{
| + return xasprintf("%s-[0-9]*", pkg);
| +}
|
| so I don't see the benefit of it. Please explain why it's better...


This is better for the patch implementing the options in package name (see line
274 in
http://www.netbsd.org/~tho/0003-pkgtools-pkg_install-Introduce-the-tags-package-suff.patch
) and this groups in the single file opattern.c all the code manipulating
the package name format and assumptions about this format.

|
| > http://www.netbsd.org/~tho/0002-pkgtools-pkg_install-Do-not-reinvent-pkg_match-for-m.patch
|
| Not sure why this was there previously - perhaps for efficiency, not
| allocating and freeing memory on every invocation. Perhaps joerg can
| shed some light?

Same reason: do not scatter assumptions about the package name format.

[Jonathan Perkin]

Post by Anthony Mallet
| > In which context do you think this wouldn't scale properly?
|
| PKG_SUPPORTED_OPTIONS= inet6 libssh2 gssapi ldap rtmp libidn
Sorry, I was not clear : I meant "what would not scale?".
I have many packages in robotpkg with 6 options or more, and I have not had any
"scalability" issue.
Of course, if you're talking about the number of different binary packages that
can be generated, this does not scale. For instance, the ~300 packages in
robotpkg can generate more than 9000 different binary packages in total. This
is not tractable in pratice, but you don't want to generate all possible binary
packages. Typically, you build only the default options and for some packages
a small set of "interesting" options combination.

I looked into this for a while at Joyent, and ultimately concluded it
wasn't worth the effort. It's fine for a very limited set of packages
and package options if you are very careful, but you quickly run into
intractable problems.

The main issue is dependencies, as you now have to pull in the correct
binary package for the options involved, and how you express the
matching in bl3 or DEPENDS.

And as soon as you want to introduce any PKG_OPTIONS which are handled
by mk/pbulk/pbulk-index.mk then there is an explosion of combinations
and difficulties in matching the correct version required.

Given that I don't believe there is a way to solve these issues in the
general case, and that you will need to be very careful in the options
you allow to be supported, I don't see any advantage therefore in
simply just providing additional packages for those instances, and
this is the approach we will be taking.

[Joerg Sonnenberger]

Post by Anthony Mallet

Post by Anthony Mallet
http://www.netbsd.org/~tho/0002-pkgtools-pkg_install-Do-not-reinvent-pkg_match-for-m.patch

Not sure why this was there previously - perhaps for efficiency, not
allocating and freeing memory on every invocation. Perhaps joerg can
shed some light?

More predictable behavior in case someone really wants to use <> etc. in
the argument.

Joerg

[Alistair Crooks]

Post by Anthony Mallet
| Personally, I have my doubts about this - I really have problems with
| its scalability, and think, while it might work for one, or even two
| options, it doesn't scale up beyond that.
In which context do you think this wouldn't scale properly?

[19:17:28] ***@netbsd-001 ...pkgsrc/www/curl [11626] > grep PKG_SUPPORTED_OPTIONS options.mk
PKG_SUPPORTED_OPTIONS= inet6 libssh2 gssapi ldap rtmp libidn

Post by Anthony Mallet

From my experience, packages seldom need complex dependency requirements

regarding options, but I may be missing something.
| I'd prefer to see more discussion before anything gets committed.
The first two patches are unrelated to options and are just code
refactoring. Maybe I can just commit these for now?
http://www.netbsd.org/~tho/0001-pkgtools-pkg_install-Factorize-code-adding-a-version.patch

Yes, re-factoring, but the re-factored code is less readable,
understandable, and obvious than the previous code. At a glance, I
could see what

pattern = xasprintf("%s-[0-9]*", pkgname);

did. The following:

pattern = addpkgwildcard(pkgname);

is opaque. Moreover, the function itself where this is factored into
is a one-liner:

+char *
+addpkgwildcard(const char *pkg)
+{
+ return xasprintf("%s-[0-9]*", pkg);
+}

so I don't see the benefit of it. Please explain why it's better...

Post by Anthony Mallet
http://www.netbsd.org/~tho/0002-pkgtools-pkg_install-Do-not-reinvent-pkg_match-for-m.patch

Not sure why this was there previously - perhaps for efficiency, not
allocating and freeing memory on every invocation. Perhaps joerg can
shed some light?

Thanks,
Alistair

[Thomas Klausner]

Post by Anthony Mallet
The first two patches are unrelated to options and are just code
refactoring. Maybe I can just commit these for now?
http://www.netbsd.org/~tho/0001-pkgtools-pkg_install-Factorize-code-adding-a-version.patch
http://www.netbsd.org/~tho/0002-pkgtools-pkg_install-Do-not-reinvent-pkg_match-for-m.patch

Fine with me.
Thomas

[Ryo ONODERA]

Hi,

Another freeze has started...
What is the stopper of this change.
If there is no stopper, please commit just after freeze end.

Thank you.

Post by Anthony Mallet
| | I noticed the way MacPorts done it and it looks reasonable ok. They name the
| | those alternative bin pkgs as: pkgname+option(s)-version ?
|
| I also did that in robotpkg (http://robotpkg.openrobots.org), which is a fork
| of pkgsrc (targetting more specific goals and users and packages, hence the
| fork).
|
| ${PKGBASE}-${PKGVERSION}~${PKG_OPTIONS}.tgz
|
| $PKG_OPTIONS are joined with the "+" char.
Hi,
So.. back on this topic!
Here are the 3 patches for pkgtools/pkg_install implementing the aforementioned
naming scheme in robotpkg. As such, the patches are supposed to be harmless,
i.e. pkgsrc/pkg_install will work exactly as before (but of course implement
the necessary bits for the new naming scheme, mostly affecting pkg_admin pmatch).
Patches 1 & 2 are just code refactoring, patch 3 implements the real stuff.
There is my git commit message in the patch, hopefully explaining what is
implemented.
http://www.netbsd.org/~tho/0001-pkgtools-pkg_install-Factorize-code-adding-a-version.patch
http://www.netbsd.org/~tho/0002-pkgtools-pkg_install-Do-not-reinvent-pkg_match-for-m.patch
http://www.netbsd.org/~tho/0003-pkgtools-pkg_install-Introduce-the-tags-package-suff.patch
pkg_admin pmatch 'x>=1.0~a' x-1.0 ->false
pkg_admin pmatch 'x>=1.0~a' x-1.0~a ->true
pkg_admin pmatch 'x>=1.0~a' x-0.0~a ->false
pkg_admin pmatch 'x~!a' x-1.0~a ->false
pkg_admin pmatch 'x~!a' x-1.0 ->true
pkg_admin pmatch 'x~a+b' x-1.0 ->false
pkg_admin pmatch 'x~a+b' x-1.0~a ->false
pkg_admin pmatch 'x~*' x-1.0 ->true
pkg_admin pmatch 'x~*' x-1.0~a ->true
pkg_admin pmatch 'x~!*' x-1.0 ->true
pkg_admin pmatch 'x~!*' x-1.0~a ->false
etc.
- The matching algorithm with patterns on boolean variables is in general a
complex problem (known as SAT, I think), and the one implemented tries to
keep things simple and intuitve. But it's not been formalized, so there must
be some caveats ...
`^' to avoid quoting issues, and instead of using pkgname-version~options,
one could also use pkgname~options-version¹.
I also provide an *untested* example patch for mk/bsd.pkg.mk that changes the
naming of the packages so that the ~options 'tag' is appended to PKGNAME. This
might possibly break pkgsrc in some places, but the patch is mostly here to
give you a concrete idea of the (almost only) change I did in robotpkg.
http://www.netbsd.org/~tho/PKGNAME.diff
Next step would be to review the places where this will break, but a
technical/philosophical discussion about all this must probably take place
beforehand. IIRC there was almost no issue when I did that in robotpkg, but the
code has diverged quite a lot from pkgsrc in some places, so I cannot tell
precisely right now.
So ... what do you think :)
--
Footnotes
¹ In robotpkg I have a shell script converting .tgz bsd binary package into
.deb ones (dpkg), and I had to name the .deb packages according to the second
alternative (i.e. pkgname~otpions-version), because dpkg is much more
restrictive about the name of the packages.

--
Ryo ONODERA // ***@yk.rim.or.jp
PGP fingerprint = 82A2 DC91 76E0 A10A 8ABB FD1B F404 27FA C7D1 15F3

[Anthony Mallet]

Fredrik Pettai wrote:
| I noticed the way MacPorts done it and it looks reasonable ok. They name the
| package "variants" as: pkgname @version+option(s). Perhaps pkgsrc could name
| those alternative bin pkgs as: pkgname+option(s)-version ?

Anthony Mallet wrote:
| I also did that in robotpkg (http://robotpkg.openrobots.org)
|
| Binary packages are named according to the following scheme:
| ${PKGBASE}-${PKGVERSION}~${PKG_OPTIONS}.tgz
|
| $PKG_OPTIONS are joined with the "+" char.
|
| Patches 1 & 2 are just code refactoring, patch 3 implements the real stuff.
| There is my git commit message in the patch, hopefully explaining what is
| implemented.
|
| http://www.netbsd.org/~tho/0001-pkgtools-pkg_install-Factorize-code-adding-a-version.patch
| http://www.netbsd.org/~tho/0002-pkgtools-pkg_install-Do-not-reinvent-pkg_match-for-m.patch
| http://www.netbsd.org/~tho/0003-pkgtools-pkg_install-Introduce-the-tags-package-suff.patch

On Tuesday, at 13:22, Ryo ONODERA wrote:
| Another freeze has started...
| What is the stopper of this change.
| If there is no stopper, please commit just after freeze end.

Hi,

Last time this was discussed, I more or less understood the conclusions as:
- This functionality is not really needed
- The efficiency of my proposed implementation is questionable (scalability,
in particular)

Which are both valid points. Here is the thread:
http://mail-index.netbsd.org/tech-pkg/2014/01/18/msg012591.html

But I'm ready to discuss this again :)

[Ryo ONODERA]

Post by Anthony Mallet
| I noticed the way MacPorts done it and it looks reasonable ok. They name the
| those alternative bin pkgs as: pkgname+option(s)-version ?
| I also did that in robotpkg (http://robotpkg.openrobots.org)
|
| ${PKGBASE}-${PKGVERSION}~${PKG_OPTIONS}.tgz
|
| $PKG_OPTIONS are joined with the "+" char.
|
| Patches 1 & 2 are just code refactoring, patch 3 implements the real stuff.
| There is my git commit message in the patch, hopefully explaining what is
| implemented.
|
| http://www.netbsd.org/~tho/0001-pkgtools-pkg_install-Factorize-code-adding-a-version.patch
| http://www.netbsd.org/~tho/0002-pkgtools-pkg_install-Do-not-reinvent-pkg_match-for-m.patch
| http://www.netbsd.org/~tho/0003-pkgtools-pkg_install-Introduce-the-tags-package-suff.patch
| Another freeze has started...
| What is the stopper of this change.
| If there is no stopper, please commit just after freeze end.
Hi,
- This functionality is not really needed
- The efficiency of my proposed implementation is questionable (scalability,
in particular)
http://mail-index.netbsd.org/tech-pkg/2014/01/18/msg012591.html
But I'm ready to discuss this again :)

Hmm...
I need this functionality.
As pkgsrc developer, I create the pkgsrc packages that has many options,
and I must preserve binary package tarball not to be overwritten by
another binary package that is built with another option.
As pkgsrc user, if binary packages are provided by options, it is
very convenient.

I am not a professional programmer and I cannot say about efficiency of
your program.
What is efficiency/scalability in this case?

--
Ryo ONODERA // ***@yk.rim.or.jp
PGP fingerprint = 82A2 DC91 76E0 A10A 8ABB FD1B F404 27FA C7D1 15F3

[Anthony Mallet]

On Tuesday, at 18:10, Ryo ONODERA wrote:
| From: Anthony Mallet <***@netbsd.org>, Date: Tue, 17 Jun 2014 10:38:02 +0200
| As pkgsrc developer, I create the pkgsrc packages that has many options,
| and I must preserve binary package tarball not to be overwritten by
| another binary package that is built with another option.
| As pkgsrc user, if binary packages are provided by options, it is
| very convenient.
|
| I am not a professional programmer and I cannot say about efficiency of
| your program.
| What is efficiency/scalability in this case?

The main issue is that even for a small number of options, the number of
possible packages quickly explodes. And it is completely untracktable if you
also consider the cartesian product of the dependencies. So it will never be
possible to e.g. bulk-build all possible packages and packages combinations.

The approach that I have chosen is to stick to bulk building packages with the
default set of options only, except for a very few specific cases which are
specified manually.

What is not clear yet is the algorithm matching a package requirement against a
set of packages names. For instance, if you require foo>=1.0~a (package foo
with option a), is foo-1.0~a+b (package foo with options a and b) "better" than
just foo-1.0~a ? Which one should be used? And it becomes even more complex
when you want wildcards (e.g. foo>=1.0~a+!* - with option a but no other
option). The generic matching problem is called SAT and this is considered to
be an NP-complete problem, so I just implemented some heuristics that do work
for the most usual cases, but which are not formally defined...

[Marc Espie]

Splitting stuff into separate packages is the most efficient way
to do things.


In OpenBSD, *all* of the complexity of flavors and multi-packages is
geared towards making this maximally useful.

Summary:

FLAVORS are options that are handled by a given port.

MULTI_PACKAGES are the different subpackages that will be built for a given
port.

If
DISTNAME = foo-1.0
MULTI_PACKAGES = -server -client -doc

then we will get the pkgnames from
PKGNAME-server
PKGNAME-client

for instance,
PKGNAME-server = foo-server-1.0
PKGNAME-client = foo-1.0

The FULLPKGNAME has flavors appended, by default, e.g.,
if FLAVORS=a b c
and FLAVOR=c
let's explicitly set
FULLPKGNAME-doc = foo-doc-1.0 (as it doesn't depend on the flavor in
that case).


then
FULLPKGNAME-server = foo-server-1.0-c
FULLPKGNAME-client = foo-1.0-c
FULLPKGNAME-doc = foo-doc-1.0

this works just fine.


One nifty additions is PSEUDO_FLAVORS, such as
PSEUDO_FLAVORS = no_server: these are flavors that only affect
the build, and that don't change the pkgnames.


e.g., if
FLAVOR=a b no_server, then I'll end up with
FULLPKGNAME-client = foo-1.0-a-b
FULLPKGNAME-doc = foo-doc-1.0
(and no server subpackage).

In the actual case of a full bulk build, we can uniquely identify each
binary packages easily...

By default, dependencies ignore flavors, unless we explicitly tell them
to require ONE specific flavor.

[Taylor R Campbell]

Date: Wed, 4 Dec 2013 18:02:25 +0100
From: Edgar =?iso-8859-1?B?RnXf?= <***@math.uni-bonn.de>

So, should I use package options controlling which of the three (or four)
components gets built (less overhead, less likely to accidently install
a client, but no server) or split it into three packages plus a
nss-pam-ldapd/Makefile.common (better for binary packages)?

Please split it into separate packages. That way, after I have done
my bulk build and partially deployed some packages, I don't need to
restart from the beginning when I realize I wanted to add a different
feature -- I just have to run pkg_add.