Return to regress.pl.1 CVS log

Up to [cvsweb.bsd.lv] / mandoc / regress

Finally port the OpenBSD regression suite.
Both kristaps@ and wiz@ repeated asked for this,
literally for years.

.\"	$Id: regress.pl.1,v 1.1 2017/02/08 03:02:13 schwarze Exp $
.\"
.\" Copyright (c) 2017 Ingo Schwarze <schwarze@openbsd.org>
.\"
.\" Permission to use, copy, modify, and distribute this software for any
.\" purpose with or without fee is hereby granted, provided that the above
.\" copyright notice and this permission notice appear in all copies.
.\"
.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
.\"
.Dd $Mdocdate: February 8 2017 $
.Dt REGRESS.PL 1
.Os
.Sh NAME
.Nm regress.pl
.Nd portable steering script for mandoc regression tests
.Sh SYNOPSIS
.Nm ./regress.pl
.Oo
.Ar directory Ns Op Pf : Ar test
.Op Ar modifier ...
.Oc
.Sh DESCRIPTION
The
.Nm
steering script allows running the
.Xr mandoc 1
regression suite on arbitrary operating systems,
even though the suite was designed for OpenBSD only.
.Pp
When run without an argument,
.Nm
runs the complete regression suite.
.Pp
When run with one argument, that argument can be:
.Bl -enum
.It
A single dot to run the complete suite.
.It
One of the top level directories, for example
.Pa mdoc ,
to run the test suite for a complete language or feature group.
.It
A subdirectory, for example
.Pa man/IP ,
to run the tests for a specific macro or an individual feature.
.It
A subdirectory with a test name appended with a colon, for example
.Pa char/unicode : Ns Pa named ,
to run the tests for one particular input file.
.El
.Pp
Any additional arguments modify the way the tests are run.
The default is
.Cm all .
The following modifiers are available:
.Bl -tag -width verbose
.It Cm all
Run all kinds of subtests.
This implies all other modifiers except
.Cm verbose
and
.Cm clean .
.It Cm ascii
Run subtests for
.Fl T Cm ascii
output mode.
.It Cm clean
Remove all output files created by running the tests.
.It Cm html
Run subtests for
.Fl T Cm html
output mode.
.It Cm lint
Run subtests for
.Fl T Cm lint
warning and error output.
.It Cm man
Run subtests for
.Fl T Cm man
output mode.
.It Cm utf8
Run subtests for
.Fl T Cm utf8
output mode.
.It Cm verbose
Display approximate indications of what is being done.
.El
.Pp
The amount of summary lines shown can be modified by giving an
argument consisting of a single digit:
.Bl -tag -width verbose
.It Cm 3
Show all summary lines for all directories entered.
Even without
.Cm verbose ,
this generates more than hundred lines of output when running the
complete regression suite.
.It Cm 2
This is the default.
It shows the summary lines for the
.Ar directory
given on the command line and its immediate children.
Except for
.Pa mdoc ,
the output usually fits on one screen.
.It Cm 1
Only show a single summary line for the whole run.
.It Cm 0
Do not show any summary lines.
No output means success.
Success or failure can also be seen from the exit status.
.El
.Pp
All failed tests are always reported, even when the
.Cm 0
modifier is given.
.Sh EXIT STATUS
.Ex -std
.Sh EXAMPLES
The recommended invocation for casual users:
.Pp
.Dl ./regress.pl
.Pp
Maximum output:
.Pp
.Dl ./regress.pl \&. verbose
.Pp
Complete check, but keep the tree clean:
.Pp
.Dl ./regress.pl \&. all clean
.Pp
Test all of
.Pa mdoc ,
but don't print the usual 65 lines of output:
.Pp
.Dl ./regress.pl mdoc 1
.Pp
Investigate a specific failure:
.Pp
.Dl ./regress.pl mdoc/Bd:broken man verbose
.Sh HISTORY
The
.Nm
script appeared in release 1.14.1 of the portable
.Sy mandoc
distribution.
.Sh AUTHORS
.An Ingo Schwarze Aq Mt schwarze@openbsd.org
.Sh CAVEATS
This script is not optimized for elegance.
Regression suites for other software should not copy the design.
.Pp
The problem it solves is that the
.Sy mandoc
regression suite is tightly integrated into the regression
testing system of the OpenBSD base system, which requires
both OpenBSD
.Xr make 1 ,
working neither with POSIX make nor with GNU make, and which
also requires the OpenBSD-specific Makefile fragments in
.Pa /usr/share/mk .
The workaround of parsing the Makefiles by hand and constructing
the required command lines by hand is unavoidably messy; it's
the classic no-no of parsing a language with an ad-hoc incomplete
parser.
But the problem of providing this regression suite for other
operating systems stood unsolved for many years, and no cleaner
solution was found that could be implemented with reasonable effort.
So maybe this is better than nothing.
.Pp
The top-level Makefiles for running this regression suite on
OpenBSD are not included in the portable distribution.
They are too OpenBSD-specific to be useful elsewhere,
and on OpenBSD itself, the suite ought be run natively from
.Pa /usr/src/regress/usr.bin/mandoc
and not from the portable distribution.
.Pp
The
.Pa db
subdirectory of the regression suite is not included.
It uses a Makefile structure that differs vastly from the
rest of the suite.