readme.txt 11 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272
  1. Usage
  2. -----
  3. To start the test suite execute:
  4. make full TEST_FPC=path_to_your_compiler
  5. This should run all tests, scan the created log file and output some
  6. statistics.
  7. make digest TEST_FPC=path_to_your_compiler
  8. scans the created log file and outputs some statistics
  9. make digest USESQL=YES TEST_FPC=path_to_your_compiler
  10. sends the results to an SQL database
  11. When the tests are performed, first the units (e.g. rtl) needed by the
  12. tests are compiled in a clean determined way and put in the units
  13. directory. Then webtbs/webtbf/test/tbs/tbf are searched for t*.pp to be
  14. compiled and executed as tests.
  15. Parallel test suite runs
  16. ------------------------
  17. It is possible to run the test suite in parallel, but only when using the
  18. "all", "full" or "fulldb" targets and only if SINGLEDOTESTRUNS is not used.
  19. Under those circumstances, it is safe to use -jx, with x the number of tests
  20. that can be compiled and run in parallel. E.g.
  21. make full TEST_FPC=path_to_your_compiler -j 2
  22. Make sure to clean the test suite between two runs for the same platform.
  23. Directories
  24. -----------
  25. webtbs...........Tests for web-bug-database bugs (should compile/run)
  26. Digits in filename refer to bug database entry
  27. webtbf...........Tests for web-bug-database bugs (should not compile/run)
  28. Digits in filename refer to bug database entry
  29. test.............Test suites for different aspects of the compiler/rtl etc
  30. test/packages....Tests depending on packages. The tests are sorted by package
  31. though this is not mandatory because tests can depend on
  32. multiple packages
  33. tbs..............Tests for other bugs, added by the fpc core team
  34. (success in compilation) Digits in filename is a serial no
  35. tbf..............Tests for other bugs, added by the fpc core team
  36. (fail compile) Digits in filename is a serial no
  37. units............Helper units for doing the tests
  38. utils............Utilities for processing tests
  39. Writing a test
  40. --------------
  41. A test should have a name on the form t*.pp, to be recognized as a test.
  42. It should return 0 on success, any other value indicates failure.
  43. Test directives
  44. ---------------
  45. At the top of the test source code, some directives can be used to
  46. determine how the tests will be processed (if processed automatically via
  47. make), e.g. {%CPU=i386}:
  48. OPT................Compiler option required to compile
  49. CPU................Only for these CPU's (i386,m68k,etc). Might be a list.
  50. SKIPCPU............Not for these CPU's (i386,m68k,etc). Might be a list.
  51. TARGET.............Only for these OS targets (win32,MacOS,etc).
  52. Might be a list.
  53. SKIPTARGET.........Not for these OS targets (win32,MacOS,etc).
  54. Might be a list.
  55. VERSION............Compiler with at lest this version number required.
  56. MAXVERSION.........Compiler with at most this version number required.
  57. RESULT.............Exit code of execution of test expected
  58. TIMEOUT............Timeout indication for test in seconds (only used if
  59. enabled by defining TEST_TIMEOUT)
  60. FILES..............List of files (separated by spaces) required by test,
  61. will be copied to remote before execution. If TEST_DELTEMP
  62. is set, all these files will be deleted after the test.
  63. DELFILES...........List of files (separated by spaces) to be deleted after
  64. the test. Files will be deleted only if TEST_DELTEMP is
  65. set. If no extension is specified for a file, the dotest
  66. program will try to delete executable and shared library
  67. of that name, using appropriate exe and dll extensions.
  68. GRAPH..............Requires graph unit
  69. FAIL...............Compilation must fail
  70. RECOMPILE..........After compiling a test, recompile the test for a second
  71. time. This is needed to test ppu loading issues. Can
  72. optionally specify a command line option to use only
  73. during the second compilation.
  74. NORUN..............Do not execute test, only compile it
  75. INTERACTIVE........Do not execute test, as it requires user intervention
  76. NOTE...............Output note when compiling/executing test
  77. NEEDLIBRARY........Adds -rpath to the linker for unix. This is needed to
  78. test runtime library tests. The library needs the -FE.
  79. option to place the .so in the correct directory.
  80. NEEDEDAFTER........Use it if the files are necessary for a later test.
  81. this option will avoid delteting generated file even when
  82. TEST_DELTEMP is used.
  83. KNOWNRUNERROR......Known bug, which manifest itself at runtime. To the
  84. right of the equal sign is the expected exit code,
  85. followed by an optional note. Will not be logged
  86. as a bug.
  87. KNOWNCOMPILEERROR..Known bug, which manifest itself at compile time. To
  88. the right of the equal sign is the expected exit code
  89. from compiler, followed by an optional note. Will not
  90. be logged as a bug.
  91. QUICKTEST..........If set, only tests without package dependencies are executed
  92. WPOPARAS...........Parameters to be added after -OW/-Ow to perform whole
  93. program optimization tests
  94. WPOPASSES..........Number of whole program optimization iterations to perform
  95. ("1" means compile once with "-FWsomefile -OW<wpoparas>"
  96. and then again with "-FWsomefile2 -OW<wpoparas>
  97. -Fwsomefile1 -Ow<wpoparas>", "2" means another pass but
  98. using somefile2 as input and somefile3 as output, etc.)
  99. NOTE: A list consists of comma separated items, e. g. CPU=i386,m68k,powerpc
  100. No space between the elements and the comma.
  101. Controling testing in more detail
  102. ---------------------------------
  103. Calling "make full" will perform tests in a standard manner. To have
  104. more control of the test process one must distinguish between:
  105. * Driver environment: compiler/rtl etc. to be used by the tools which
  106. runs and analyze the tests. All normal options to make, like FPC
  107. OS_TARGET, OPT etc. controls this.
  108. * Test environment: compiler/rtl etc. to be tested, to be used
  109. *in* the tests. Ususal options, prepended with TEST_ , controls
  110. this. If no such options are given, test and driver environment
  111. will be the same.
  112. This differentiation also enables cross testing.
  113. The following test options can be given:
  114. TEST_FPC compiler to test (no default)
  115. TEST_OS_TARGET defaults to default target of TEST_FPC
  116. TEST_CPU_TARGET defaults to default target of TEST_FPC
  117. TEST_OPT defaults to ""
  118. TEST_FPC_VERSION defaults to version of TEST_FPC
  119. TEST_CCOMPILER defaults to installed gcc compiler, but only
  120. if driver and test full-targets are the same.
  121. TEST_VERBOSE let dotest be more verbose, only useful for debugging
  122. TEST_BENCH display compilation/execution time of each test
  123. TEST_DELTEMP delete temporary executable/object/ppu file,
  124. default is off
  125. TEST_TIMEOUT use timeout wrapper for (remote) execution
  126. V print dotest commandline
  127. TEST_ABI test a certain abi, this influences where the
  128. c object files are taken from: TEST_ABI=eabi
  129. takes the c*.o files from
  130. test/cg/obj/linux/arm-eabi
  131. CHUNKSIZE Number of tests per chunk that can be potentially executed
  132. in parallel with other chunks
  133. (Please add more test options if needed)
  134. NOTE: To clean after a test session, "make clean TEST_FPC=path_to_your_compiler"
  135. must be given the same options as when running the tests.
  136. The utils directory is considerd to belong to the driver environment,
  137. all other directories belong to the test environment.
  138. Remote execution
  139. ----------------
  140. Also remote execution of the test suite is possible.
  141. Requirements:
  142. - rsh/ssh must work without keyboard interaction or extra parameters
  143. Test options:
  144. TEST_RSH set this to the hostname when you want to use rsh/rcp
  145. to execute/copy the test
  146. TEST_SSH set this to use ssh/scp to execute the test
  147. TEST_PUTTY test using putty when remote testing (pscp and plink)
  148. TEST_ADB run tests om remote Android device using ADB. Specify
  149. TEST_ADB=1 to run on default connected device. Specify
  150. TEST_ADB=<serial> to run on specific connected device.
  151. TEST_REMOTEOPT extra options to remote program
  152. TEST_REMOTEPATH set remote path to use, default is /tmp
  153. TEST_DELBEFORE delete remote executable before uploading
  154. TEST_DELTEMP delete executable after running, so the remote system
  155. doesn't need much free disk space
  156. TEST_REMOTEPW pass a password with -pw to remote tools,
  157. mainly useful for putty
  158. Examples:
  159. -------
  160. make TEST_FPC=$HOME/fpc/compiler/ppcsparc TEST_BINUTILSPREFIX=sparc-linux- \
  161. TEST_RSH=sunny TEST_REMOTEPATH=/tmp/tests
  162. make TEST_FPC=$HOME/fpc/compiler/ppcsparc TEST_BINUTILSPREFIX=sparc-linux- \
  163. TEST_SSH=fpc@sunny TEST_REMOTEPATH=/tmp/tests
  164. Example for win32/putty:
  165. make TEST_FPC=c:\fpc\compiler\ppcarm TEST_BINUTILSPREFIX=arm-linux- \
  166. [email protected] TEST_REMOTEPATH=/tmp TEST_DELTEMP=1 \
  167. "TEST_REMOTEPW=xxx" FPC=c:\fpc\compiler\ppc386
  168. Emulator execution
  169. ------------------
  170. Emulator execution is possible as well. It can't be combined with remote
  171. execution though.
  172. EMULATOR: name of the emulator to use
  173. Examples:
  174. make TEST_FPC=~/fpc/compiler/ppcrossarm TEST_OPT=-XParm-linux- \
  175. EMULATOR=qemu-arm
  176. make TEST_FPC=~/fpc/compiler/ppcrossarm TEST_OPT=-XParm-linux- \
  177. EMULATOR=qemu-arm digest DBDIGESTOPT="-C qemu-arm" USESQL=YES
  178. Example cross testing of target Mac OS with driver Darwin
  179. --------------------------------------------------------
  180. NOTE: Today, it is possible to run the test suite Mac OS native.
  181. A machine with both Mac OS X and classic Mac OS installed is required.
  182. Note that make will not run the tests. This has to be done in MPW with the
  183. scripts in utils/MacOS.
  184. make clean alltest TEST_OS_TARGET=MacOS TEST_OPT="-WT -st" \
  185. USEUNITDIR=/Projekt/Freepascal/fpc/rtl/MacOS
  186. To clean. Note that same options as above has to be given so that the
  187. correct files will be removed.
  188. make clean TEST_OS_TARGET=MacOS USEUNITDIR=/Projekt/Freepascal/fpc/rtl/MacOS
  189. Example cross testing of target arm-wince
  190. -----------------------------------------
  191. //arm-wince example:
  192. see FPCTRUNK\DEMO\WINCE\TESTEMU\ for additionally required tools
  193. Connect your device via ActiveSync and execute:
  194. make TEST_FPC=ppcrossarm TEST_CPU_TARGET=arm TEST_OS_TARGET=wince \
  195. TEST_OPT="-XParm-wince- -WC -Xs" \
  196. EMULATOR=MyDisc:\My\Path\to\wcetemu.exe
  197. Tests will be performed in the \fpctests folder on the device.
  198. Result uploading
  199. ----------------
  200. Results can be uploaded to the test suite result DB
  201. (http://www.freepascal.org/testsuite/cgi-bin/testsuite.cgi) by executing
  202. make uploadrun
  203. On unix:
  204. - The testing machine must be enabled to login on www.freepascal.org
  205. automatically as user fpc with ssh.
  206. On Windows:
  207. - The putty utilities plink.exe and pscp.exe must be in the path
  208. - There must be a putty session named [email protected] which is
  209. enabled to login automatically into www.freepascal.org
  210. Use TEST_USE_LONGLOG=1 as make parameter to pass information
  211. to server using longlog file instead of the whole output.