Document unit testing in the engine and modules (#4017)

development/cpp/custom_modules_in_cpp.rst

@@ -594,6 +594,72 @@ you might encounter an error similar to the following:
     ERROR: Can't write doc file: docs/doc/classes/@GDScript.xml
        At: editor/doc/doc_data.cpp:956
 
+.. _doc_custom_module_unit_tests:
+
+Writing custom unit tests
+-------------------------
+
+It's possible to write self-contained unit tests as part of a C++ module. If you
+are not familiar with the unit testing process in Godot yet, please refer to
+:ref:`doc_unit_testing`.
+
+The procedure is the following:
+
+1. Create a new directory named ``tests/`` under your module's root:
+
+.. code-block:: console
+
+    cd modules/summator
+    mkdir tests
+    cd tests
+
+2. Create a new test suite: ``test_summator.h``. The header must be prefixed
+   with ``test_`` so that the build system can collect it and include it as part
+   of the ``tests/test_main.cpp`` where the tests are run.
+
+3. Write some test cases. Here's an example:
+
+.. code-block:: cpp
+
+    // test_summator.h
+    #ifndef TEST_SUMMATOR_H
+    #define TEST_SUMMATOR_H
+
+    #include "tests/test_macros.h"
+
+    #include "modules/summator/summator.h"
+
+    namespace TestSummator {
+
+    TEST_CASE("[Modules][Summator] Adding numbers") {
+        Ref<Summator> s = memnew(Summator);
+        CHECK(s->get_total() == 0);
+
+        s->add(10);
+        CHECK(s->get_total() == 10);
+
+        s->add(20);
+        CHECK(s->get_total() == 30);
+
+        s->add(30);
+        CHECK(s->get_total() == 60);
+
+        s->reset();
+        CHECK(s->get_total() == 0);
+    }
+
+    } // namespace TestSummator
+
+    #endif // TEST_SUMMATOR_H
+
+4. Compile the engine with ``scons tests=yes``, and run the tests with the
+   following command:
+
+.. code-block:: console
+
+    ./bin/<godot_binary> --test --source-file="*test_summator*" --success
+
+You should see the passing assertions now.
 
 .. _doc_custom_module_icons:
 

development/cpp/index.rst

@@ -12,6 +12,7 @@ Engine development
    variant_class
    object_class
    inheritance_class_tree
+   unit_testing
    custom_modules_in_cpp
    binding_to_external_libraries
    custom_resource_format_loaders

development/cpp/unit_testing.rst

@@ -0,0 +1,290 @@
+.. _doc_unit_testing:
+
+Unit testing
+============
+
+Godot Engine allows to write unit tests directly in C++. The engine integrates
+the `doctest <https://github.com/onqtam/doctest>`_ unit testing framework which
+gives ability to write test suites and test cases next to production code, but
+since the tests in Godot go through a different ``main`` entry point, the tests
+reside in a dedicated ``tests/`` directory instead, which is located at the root
+of the engine source code.
+
+Platform and target support
+---------------------------
+
+C++ unit tests can be run on Linux, macOS, and Windows operating systems.
+
+Tests can only be run with editor ``tools`` enabled, which means that export
+templates cannot be tested currently.
+
+Running tests
+-------------
+
+Before tests can be actually run, the engine must be compiled with the ``tests``
+build option enabled (and any other build option you typically use), as the
+tests are not compiled as part of the engine by default:
+
+.. code-block:: shell
+
+    scons tests=yes
+
+Once the build is done, run the tests with a ``--test`` command-line option:
+
+.. code-block:: shell
+
+    ./bin/<godot_binary> --test
+
+The test run can be configured with the various doctest-specific command-line
+options. To retrieve the full list of supported options, run the ``--test``
+command with the ``--help`` option:
+
+.. code-block:: shell
+
+    ./bin/<godot_binary> --test --help
+
+Any other options and arguments after the ``--test`` command are treated as
+arguments for doctest.
+
+.. note::
+
+    Tests are compiled automatically if you use the ``dev=yes`` SCons option.
+    ``dev=yes`` is recommended if you plan on contributing to the engine
+    development as it will automatically treat compilation warnings as errors.
+    The continuous integration system will fail if any compilation warnings are
+    detected, so you should strive to fix all warnings before opening a pull
+    request.
+
+Filtering tests
+~~~~~~~~~~~~~~~
+
+By default, all tests are run if you don't supply any extra arguments after the
+``--test`` command. But if you're writing new tests or would like to see the
+successful assertions output coming from those tests for debugging purposes, you
+can run the tests of interest with the various filtering options provided by
+doctest.
+
+The wildcard syntax ``*`` is supported for matching any number of characters in
+test suites, test cases, and source file names:
+
++--------------------+---------------+------------------------+
+| **Filter options** | **Shorthand** | **Examples**           |
++--------------------+---------------+------------------------+
+| ``--test-suite``   | ``-ts``       | ``-ts="*[GDScript]*"`` |
++--------------------+---------------+------------------------+
+| ``--test-case``    | ``-tc``       | ``-tc="*[String]*"``   |
++--------------------+---------------+------------------------+
+| ``--source-file``  | ``-sf``       | ``-sf="*test_color*"`` |
++--------------------+---------------+------------------------+
+
+For instance, to run only the ``String`` unit tests, run:
+
+.. code-block:: shell
+
+    ./bin/<godot_binary> --test --test-case="*[String]*"
+
+Successful assertions output can be enabled with the ``--success`` (``-s``)
+option, and can be combined with any combination of filtering options above,
+for instance:
+
+.. code-block:: shell
+
+    ./bin/<godot_binary> --test --source-file="*test_color*" --success
+
+Specific tests can be skipped with corresponding ``-exclude`` options. As of
+now, some tests include random stress tests which take a while to execute. In
+order to skip those kind of tests, run the following command:
+
+.. code-block:: shell
+
+    ./bin/<godot_binary> --test --test-case-exclude="*[Stress]*"
+
+Writing tests
+-------------
+
+Test suites represent C++ header files which must be included as part of the
+main test entry point in ``tests/test_main.cpp``. Most test suites are located
+directly under ``tests/`` directory.
+
+All header files are prefixed with ``test_``, and this is a naming convention
+which the Godot build system relies on to detect tests throughout the engine.
+
+Here's a minimal working test suite with a single test case written:
+
+.. code-block:: cpp
+
+    #ifndef TEST_STRING_H
+    #define TEST_STRING_H
+
+    #include "tests/test_macros.h"
+
+    namespace TestString {
+
+    TEST_CASE("[String] Hello World!") {
+        String hello = "Hello World!";
+        CHECK(hello == "Hello World!");
+    }
+
+    } // namespace TestString
+
+    #endif // TEST_STRING_H
+
+The ``tests/test_macros.h`` header encapsulates everything which is needed for
+writing C++ unit tests in Godot. It includes doctest assertion and logging
+macros such as ``CHECK`` as seen above, and of course the definitions for
+writing test cases themselves.
+
+.. seealso::
+
+    `tests/test_macros.h <https://github.com/godotengine/godot/blob/master/tests/test_macros.h>`_
+    source code for currently implemented macros and aliases for them.
+
+Test cases are created using ``TEST_CASE`` function-like macro. Each test case
+must have a brief description written in parentheses, optionally including
+custom tags which allow to filter the tests at run-time, such as ``[String]``,
+``[Stress]`` etc.
+
+Test cases are written in a dedicated namespace. This is not required, but
+allows to prevent naming collisions for when other static helper functions are
+written to accommodate the repeating testing procedures such as populating
+common test data for each test, or writing parameterized tests.
+
+Godot supports writing tests per C++ module. For instructions on how to write
+module tests, refer to :ref:`doc_custom_module_unit_tests`.
+
+Assertions
+~~~~~~~~~~
+
+A list of all commonly used assertions used throughout the Godot tests, sorted
+by severity.
+
++-------------------+----------------------------------------------------------------------------------------------------------------------------------+
+| **Assertion**     | **Description**                                                                                                                  |
++-------------------+----------------------------------------------------------------------------------------------------------------------------------+
+| ``REQUIRE``       | Test if condition holds true. Fails the entire test immediately if the condition does not hold true.                             |
++-------------------+----------------------------------------------------------------------------------------------------------------------------------+
+| ``REQUIRE_FALSE`` | Test if condition does not hold true. Fails the entire test immediately if the condition holds true.                             |
++-------------------+----------------------------------------------------------------------------------------------------------------------------------+
+| ``CHECK``         | Test if condition holds true. Marks the test run as failing, but allow to run other assertions.                                  |
++-------------------+----------------------------------------------------------------------------------------------------------------------------------+
+| ``CHECK_FALSE``   | Test if condition does not hold true. Marks the test run as failing, but allow to run other assertions.                          |
++-------------------+----------------------------------------------------------------------------------------------------------------------------------+
+| ``WARN``          | Test if condition holds true. Does not fail the test under any circumstance, but logs a warning if something does not hold true. |
++-------------------+----------------------------------------------------------------------------------------------------------------------------------+
+| ``WARN_FALSE``    | Test if condition does not hold true. Does not fail the test under any circumstance, but logs a warning if something holds true. |
++-------------------+----------------------------------------------------------------------------------------------------------------------------------+
+
+All of the above assertions have corresponding ``*_MESSAGE`` macros, which allow
+to print optional message with rationale of what should happen.
+
+Prefer to use ``CHECK`` for self-explanatory assertions and ``CHECK_MESSAGE``
+for more complex ones if you think that it deserves a better explanation.
+
+.. seealso::
+
+    `doctest: Assertion macros <https://github.com/onqtam/doctest/blob/master/doc/markdown/assertions.md>`_.
+
+Logging
+~~~~~~~
+
+The test output is handled by doctest itself, and does not rely on Godot
+printing or logging functionality at all, so it's recommended to use dedicated
+macros which allow to log test output in a format written by doctest.
+
++----------------+-----------------------------------------------------------------------------------------------------------+
+| **Macro**      | **Description**                                                                                           |
++----------------+-----------------------------------------------------------------------------------------------------------+
+| ``MESSAGE``    | Prints a message.                                                                                         |
++----------------+-----------------------------------------------------------------------------------------------------------+
+| ``FAIL_CHECK`` | Marks the test as failing, but continue the execution. Can be wrapped in conditionals for complex checks. |
++----------------+-----------------------------------------------------------------------------------------------------------+
+| ``FAIL``       | Fails the test immediately. Can be wrapped in conditionals for complex checks.                            |
++----------------+-----------------------------------------------------------------------------------------------------------+
+
+Different reporters can be chosen at run-time. For instance, here's how the
+output can be redirected to a XML file:
+
+.. code-block:: shell
+
+    ./bin/<godot_binary> --test --source-file="*test_validate*" --success --reporters=xml --out=doctest.txt
+
+.. seealso::
+
+    `doctest: Logging macros <https://github.com/onqtam/doctest/blob/master/doc/markdown/logging.md>`_.
+
+Testing failure paths
+~~~~~~~~~~~~~~~~~~~~~
+
+Sometimes, it's not always feasible to test for an *expected* result. With the
+Godot development philosophy of that the engine should not crash and should
+gracefully recover whenever a non-fatal error occurs, it's important to check
+that those failure paths are indeed safe to execute without crashing the engine.
+
+*Unexpected* behavior can be tested in the same way as anything else. The only
+problem this creates is that the error printing shall unnecessarily pollute the
+test output with errors coming from the engine itself (even if the end result is
+successful).
+
+To alleviate this problem, use ``ERR_PRINT_OFF`` and ``ERR_PRINT_ON`` macros
+directly within test cases to temporarily disable the error output coming from
+the engine, for instance:
+
+.. code-block:: cpp
+
+    TEST_CASE("[Color] Constructor methods") {
+        ERR_PRINT_OFF;
+        Color html_invalid = Color::html("invalid");
+        ERR_PRINT_ON; // Don't forget to re-enable!
+
+        CHECK_MESSAGE(html_invalid.is_equal_approx(Color()),
+            "Invalid HTML notation should result in a Color with the default values.");
+    }
+
+Test tools
+----------
+
+Test tools are advanced methods which allow you to run arbitrary procedures to
+facilitate the process of manual testing and debugging the engine internals.
+
+These tools can be run by supplying the name of a tool after the ``--test``
+command-line option. For instance, the GDScript module implements and registers
+several tools to help the debugging of the tokenizer, parser, and compiler:
+
+.. code-block:: shell
+
+    ./bin/<godot_binary> --test gdscript-tokenizer test.gd
+    ./bin/<godot_binary> --test gdscript-parser test.gd
+    ./bin/<godot_binary> --test gdscript-compiler test.gd
+
+If any such tool is detected, then the rest of the unit tests are skipped.
+
+Test tools can be registered anywhere throughout the engine as the registering
+mechanism closely resembles of what doctest provides while registering test
+cases using dynamic initialization technique, but usually these can be
+registered at corresponding ``register_types.cpp`` sources (per module or core).
+
+Here's an example of how GDScript registers test tools in
+``modules/gdscript/register_types.cpp``:
+
+.. code-block:: cpp
+
+    #ifdef TESTS_ENABLED
+    void test_tokenizer() {
+        TestGDScript::test(TestGDScript::TestType::TEST_TOKENIZER);
+    }
+
+    void test_parser() {
+        TestGDScript::test(TestGDScript::TestType::TEST_PARSER);
+    }
+
+    void test_compiler() {
+        TestGDScript::test(TestGDScript::TestType::TEST_COMPILER);
+    }
+
+    REGISTER_TEST_COMMAND("gdscript-tokenizer", &test_tokenizer);
+    REGISTER_TEST_COMMAND("gdscript-parser", &test_parser);
+    REGISTER_TEST_COMMAND("gdscript-compiler", &test_compiler);
+    #endif
+
+The custom command-line parsing can be performed by a test tool itself with the
+help of OS :ref:`get_cmdline_args<class_OS_method_get_cmdline_args>` method.