automake-1.16: Use TAP with the Automake test harness
15.4.2 Use TAP with the Automake test harness
---------------------------------------------
Currently, the TAP driver that comes with Automake requires some by-hand
steps on the developer’s part (this situation should hopefully be
improved in future Automake versions). You’ll have to grab the
‘tap-driver.sh’ script from the Automake distribution by hand, copy it
in your source tree, and use the Automake support for third-party test
drivers to instruct the harness to use the ‘tap-driver.sh’ script and
the awk program found by ‘AM_INIT_AUTOMAKE’ to run your TAP-producing
tests. See the example below for clarification.
Apart from the options common to all the Automake test drivers (⇒
Command-line arguments for test drivers), ‘tap-driver.sh’ supports the
following options, whose names are chosen for enhanced compatibility
with the ‘prove’ utility.
‘--ignore-exit’
Causes the test driver to ignore the exit status of the test
scripts; by default, the driver will report an error if the script
exits with a non-zero status. This option has effect also on
non-zero exit statuses due to termination by a signal.
‘--comments’
Instruct the test driver to display TAP diagnostics (i.e., lines
beginning with the ‘#’ character) in the testsuite progress output
too; by default, TAP diagnostics are only copied to the ‘.log’
file.
‘--no-comments’
Revert the effects of ‘--comments’.
‘--merge’
Instruct the test driver to merge the test scripts’ standard error
into their standard output. This is necessary if you want to
ensure that diagnostics from the test scripts are displayed in the
correct order relative to test results; this can be of great help
in debugging (especially if your test scripts are shell scripts run
with shell tracing active). As a downside, this option might cause
the test harness to get confused if anything that appears on
standard error looks like a test result.
‘--no-merge’
Revert the effects of ‘--merge’.
‘--diagnostic-string=STRING’
Change the string that introduces TAP diagnostics from the default
value of “‘#’” to ‘STRING’. This can be useful if your TAP-based
test scripts produce verbose output on which they have limited
control (because, say, the output comes from other tools invoked in
the scripts), and it might contain text that gets spuriously
interpreted as TAP diagnostics: such an issue can be solved by
redefining the string that activates TAP diagnostics to a value you
know won’t appear by chance in the tests’ output. Note however
that this feature is non-standard, as the “official” TAP protocol
does not allow for such a customization; so don’t use it if you can
avoid it.
Here is an example of how the TAP driver can be set up and used.
% cat configure.ac
AC_INIT([GNU Try Tap], [1.0], [bug-automake@gnu.org])
AC_CONFIG_AUX_DIR([build-aux])
AM_INIT_AUTOMAKE([foreign -Wall -Werror])
AC_CONFIG_FILES([Makefile])
AC_REQUIRE_AUX_FILE([tap-driver.sh])
AC_OUTPUT
% cat Makefile.am
TEST_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) \
$(top_srcdir)/build-aux/tap-driver.sh
TESTS = foo.test bar.test baz.test
EXTRA_DIST = $(TESTS)
% cat foo.test
#!/bin/sh
echo 1..4 # Number of tests to be executed.
echo 'ok 1 - Swallows fly'
echo 'not ok 2 - Caterpillars fly # TODO metamorphosis in progress'
echo 'ok 3 - Pigs fly # SKIP not enough acid'
echo '# I just love word plays ...'
echo 'ok 4 - Flies fly too :-)'
% cat bar.test
#!/bin/sh
echo 1..3
echo 'not ok 1 - Bummer, this test has failed.'
echo 'ok 2 - This passed though.'
echo 'Bail out! Ennui kicking in, sorry...'
echo 'ok 3 - This will not be seen.'
% cat baz.test
#!/bin/sh
echo 1..1
echo ok 1
# Exit with error, even if all the tests have been successful.
exit 7
% cp PREFIX/share/automake-APIVERSION/tap-driver.sh .
% autoreconf -vi && ./configure && make check
...
PASS: foo.test 1 - Swallows fly
XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
PASS: foo.test 4 - Flies fly too :-)
FAIL: bar.test 1 - Bummer, this test has failed.
PASS: bar.test 2 - This passed though.
ERROR: bar.test - Bail out! Ennui kicking in, sorry...
PASS: baz.test 1
ERROR: baz.test - exited with status 7
...
Please report to bug-automake@gnu.org
...
% echo exit status: $?
exit status: 1
% env TEST_LOG_DRIVER_FLAGS='--comments --ignore-exit' \
TESTS='foo.test baz.test' make -e check
...
PASS: foo.test 1 - Swallows fly
XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
# foo.test: I just love word plays...
PASS: foo.test 4 - Flies fly too :-)
PASS: baz.test 1
...
% echo exit status: $?
exit status: 0
Info Catalog
automake-1.16: Introduction to TAP
automake-1.16: Using the TAP test protocol
automake-1.16: Incompatibilities with other TAP parsers and drivers