fpc/tests

Directories
-----------
webtbs...........Tests for web-bug-database bugs (success in compilation)
Digits in filename refer to bug database entry
webtbf...........Tests for web-bug-database bugs (fail compile)
Digits in filename refer to bug database entry
test.............Testsuites for different aspects of the compiler/rtl etc
tbs..............Tests for other bugs, added by the fpc core team
(success in compilation) Digits in filename is a serial no
tbf..............Tests for other bugs, added by the fpc core team
(fail compile) Digits in filename is a serial no
units............Helper units for doing the tests
utils............Utilities for processing tests


Test directives
---------------
At the top of the test source code, some directives
can be used to determine how the tests will be
processed (if processed automatically via make),
e. g. {%CPU=i386} :

OPT................Compiler option required to compile
CPU................Only for these CPU's (i386,m68k,etc). Might be a list.
SKIPCPU............Not for these CPU's (i386,m68k,etc). Might be a list.
TARGET.............Only for these OS targets (win32,macos,etc).
Might be a list.
SKIPTARGET.........Not for these OS targets (win32,macos,etc).
Might be a list.
VERSION............Compiler with at lest this version number required.
MAXVERSION.........Compiler with at most this version number required.
RESULT.............Exit code of execution of test expected
GRAPH..............Requires graph unit
FAIL...............Compilation must fail
RECOMPILE..........After compiling a test, recompile the test for a second
time. This is needed to test ppu loading issues.
NORUN..............Do not execute test, only compile it
INTERACTIVE........Do not execute test, as it requires user intervention
NOTE...............Output note when compiling/executing test
NEEDLIBRARY........Adds -rpath to the linker for unix. This is needed to
test runtime library tests. The library needs the -FE.
option to place the .so in the correct directory.
KNOWNRUNERROR......Known bug, which manifest itself at runtime. To the
right of the equal sign is the expected exit code,
followed by an optional note. Will not be logged
as a bug.
KNOWNCOMPILEERROR..Known bug, which manifest itself at compile time. To
the right of the equal sign is the expected exit code
from compiler, followed by an optional note. Will not
be logged as a bug.

NOTE: A list consists of comma separated items, e. g. CPU=i386,m68k,linux
No space between the elements and the comma.

Usage
-----
To actually start the testsuite:
do a simple
make full This should create a log of all failed tests.

make rundigest scans the created log file and outputs some statistics
make rundigest USESQL=YES sends the results to an SQL database

When the tests are performed, first the units (e g rtl) needed by the tests
are compiled in a clean determined way and put in the units directory. Then
webtbs/webtbf/test/tbs/tbf are searched for t*.pp to be compiled
and executed as tests.

Controling testing in more detail
---------------------------------
Calling "make full" will preform tests in a standard manner. To have
more control of the test process we must differentiate between:

* Driver enviroment: compiler/rtl etc to be used by the tools which
runs and analyze the tests. All normal options to make, like FPC
OS_TARGET, OPT etc controls this.

* Test environment: compiler/rtl etc to be tested, to be used
*in* the tests. Ususal options, prepended with TEST_ , controls
this. If no such options are given, test and driver environment
will be the same.

This differentiation also enables cross testing.

The following test options can be given:

TEST_FPC defaults to FPC
TEST_OS_TARGET defaults to OS_TARGET
TEST_CPU_TARGET defaults to CPU_TARGET
TEST_OPT defaults to ""
TEST_FPC_VERSION defaults to version of TEST_FPC
TEST_CCOMPILER defaults to installed gcc compiler, but only
if driver and test full-targets are the same.

(Please add more test options if needed)

NOTE To clean after a test session, "make clean" must be given the same
options as when running the tests.

The utils directory is considerd to belong to the driver environment,
all other directories belongs to the test environment.

Remote execution
----------------
Also remote execution of the testsuite is possible
Requirements:
- current build tree contains a cross compiled rtl/fcl
- the cross compiler is installed works without passing extra parameters
- the tests tree is somewhere on the remote machine e.g. /mnt/cf/fpc/tests
- some dir, e.g. i386-utils contains a dotest executable for the host system
- ssh must work without keyboard interaction or extra parameters
then a example make command could be
make DOTEST=i386-utils/dotest FPC=ppcarm "DOTESTOPT=-Y-XParm-linux- [email protected] -P/mnt/cf/fpc/tests -T"