pkg_regress usage and errors

[Jeremy C. Reed]

1. Running "make" in pkgsrc/regress results in numerous ERRORs.

I see the pkgsrc guide mentions pkg_regress.

We should have a README in pkgsrc/regress/README that explains it also.
Is it okay to have errors when running make? Can someone please explain
this in a README (even if it just points to the pkgsrc guide)?

Should we have a make all target in pkgsrc/regress that has
understandable or informational output?

2. Running pkg_regress results in:

sunflower:regress$ /usr/pkg/sbin/pkg_regress
ERROR: Expected "USE_TOOLS\+=perl" in the output, but it is not there.
ERROR: Expected exit code 0, but got 1.
ERROR: Expected exit code 0, but got 1.
ERROR: Expected exit code 0, but got 1.
ERROR: Expected exit code 0, but got 1.
===> Cleaning for compiler-1.1
ERROR: Expected "^Cplain.*:.*Cplain" in the output, but it is not there.
===> Cleaning for compiler-1.1
ERROR: Expected "Required package bison" in the output, but it is not
there.
ERROR: Expected "Circular dependency detected" in the output, but it is
not there.
ERROR: Didn't expect "bison-must-be-deinstalled" in the output, but
found it.
ERROR: Expected "^The variable MAKE_ENV remains unchanged\.$" in the
output, but it is not there.
ERROR: Expected exit code 0, but got 1.
ERROR: Expected exit code 0, but got 1.
ERROR: Expected exit code 0, but got 1.
ERROR: Expected "^@comment Matched test file.$" in the output, but it is
not there.
ERROR: Expected "^print-plist/foo$" in the output, but it is not there.
ERROR: Expected "^@comment Matched test directory.$" in the output, but
it is not there.
ERROR: Expected "^@dirrm print-plist$" in the output, but it is not
there.
Password:
Sorry, try again.
Password:
ERROR: Expected exit code 0, but got 1.
Tests failed: barrier bootstrap-install-sh buildlink-libtool
buildlink-transform buildlink-unwrap compiler ignore-tools
make-env-phases make-quoting pkg-options print-plist tools

Statistics:
1 passed, 12 failed

Is the above what others are seeing?

pkg_regress needs a manual page.

Is any NetBSD system automated to run the pkg_regress tests? If not,
can we do this?

What other regression tests should we add?

I'd like to prove that a selection of packages work to install, run, and
deinstall. It would also be nice to simulate some upgrade type
scenario.

[Julio Merino]

Post by Jeremy C. Reed
1. Running "make" in pkgsrc/regress results in numerous ERRORs.
I see the pkgsrc guide mentions pkg_regress.
We should have a README in pkgsrc/regress/README that explains it also.
Is it okay to have errors when running make? Can someone please explain
this in a README (even if it just points to the pkgsrc guide)?
Should we have a make all target in pkgsrc/regress that has
understandable or informational output?

In my opinion, we should just replace pkg_regress with atf. Then
you'd just go into pkgsrc/regress/ and execute "atf-run | atf-report"
as usual.

[Richard PALO]

Post by Jeremy C. Reed
1. Running "make" in pkgsrc/regress results in numerous ERRORs.
I see the pkgsrc guide mentions pkg_regress.
We should have a README in pkgsrc/regress/README that explains it also.
Is it okay to have errors when running make? Can someone please explain
this in a README (even if it just points to the pkgsrc guide)?
Should we have a make all target in pkgsrc/regress that has
understandable or informational output?
sunflower:regress$ /usr/pkg/sbin/pkg_regress
ERROR: Expected "USE_TOOLS\+=perl" in the output, but it is not there.
ERROR: Expected exit code 0, but got 1.
ERROR: Expected exit code 0, but got 1.
ERROR: Expected exit code 0, but got 1.
ERROR: Expected exit code 0, but got 1.
===> Cleaning for compiler-1.1
ERROR: Expected "^Cplain.*:.*Cplain" in the output, but it is not there.
===> Cleaning for compiler-1.1
ERROR: Expected "Required package bison" in the output, but it is not
there.
ERROR: Expected "Circular dependency detected" in the output, but it is
not there.
ERROR: Didn't expect "bison-must-be-deinstalled" in the output, but
found it.
ERROR: Expected "^The variable MAKE_ENV remains unchanged\.$" in the
output, but it is not there.
ERROR: Expected exit code 0, but got 1.
ERROR: Expected exit code 0, but got 1.
ERROR: Expected exit code 0, but got 1.
not there.
ERROR: Expected "^print-plist/foo$" in the output, but it is not there.
it is not there.
there.
Sorry, try again.
ERROR: Expected exit code 0, but got 1.
Tests failed: barrier bootstrap-install-sh buildlink-libtool
buildlink-transform buildlink-unwrap compiler ignore-tools
make-env-phases make-quoting pkg-options print-plist tools
1 passed, 12 failed
Is the above what others are seeing?
pkg_regress needs a manual page.
Is any NetBSD system automated to run the pkg_regress tests? If not,
can we do this?
What other regression tests should we add?
I'd like to prove that a selection of packages work to install, run, and
deinstall. It would also be nice to simulate some upgrade type
scenario.

Apparently not much has changed since this posting.
For example, regress/tools seems to only need LANG=C or LANG=posix in
order to get it to pass directly...
Some errors, such is from regress/make-quoting are actually

Post by Jeremy C. Reed
===> Building for regress-make-1.0
Testing bmake(1) version 20110606
Running testcase bug2
Running testcase dblquote
Running testcase for
Running testcase ltarget
Running testcase nestfor
Running testcase quoting
Running testcase sglquote
1c1
< \a b z
---

\\a b z

*** Error code 1
Stop.

[Benny Siegert]

Post by Richard PALO
Apparently not much has changed since this posting.

Amitai and I were looking at pkg_regress during pkgsrccon. At the
time, all but one test was broken, and I fixed a couple ones. But for
most of them, I could not figure out what the actual problem was. And
the output from the pkg_regress tool is less than helpful IMHO.

Post by Richard PALO
For example, regress/tools seems to only need LANG=C or LANG=posix in order
to get it to pass directly...

Could you add that to the test runner please?

Post by Richard PALO

Post by Jeremy C. Reed
Running testcase sglquote
1c1
< \a b z
---

\\a b z

This is not a helpful message, at all. I believe way that the tests
are run could be done better.
In the Go project, the use of test assertion frameworks is forbidden,
instead you use a _custom_ error message for each type of test -- and
ideally give the test a descriptive name. "sglquote" is probably not
expressive enough.

Ideally, you would want to run the tests in a harness like Perl's TAP
(Test Anything Protocol), and you would want to make them small and
descriptive. Are you willing to help with working on this?

--Benny.