Restructure to run the extract and annotate in a single tool.

The AST matcher is currently only able to extract the exposed class names.
[ci only: Annotate]

Docs/GettingStarted.dox

@@ -71,7 +71,6 @@ A number of build options can be defined when invoking the build scripts or when
 |RPI                  |0|Configure project using Raspberry Pi cross-compiler toolchain (cmake_generic.sh only), cmake-gui users need to specify RPI toolchain file for cross-compiling explicitly|
 |WIN32                |0|Configure project using MinGW (32-bit or 64-bit) cross-compiler toolchain (cmake_generic.sh only), cmake-gui users need to specify MinGW toolchain file for cross-compiling explicitly|
 |EMSCRIPTEN           |0|Configure project using Emscripten cross-compiler toolchain (cmake_generic.bat and cmake_generic.sh only), cmake-gui users need to specify Emscripten toolchain file for cross-compiling explicitly|
-|URHO3D_C++11         |0|Enable C++11 standard|
 |IOS                  |0|Configure project for targeting iOS platform (cmake_generic.sh and cmake-gui only)|
 |URHO3D_64BIT         |*|Enable 64-bit build, on MSVC default to 0, on other compilers the default is set based on the 64-bit capability of the chosen toolchain on host system|
 |URHO3D_ANGELSCRIPT   |1|Enable AngelScript scripting support|
@@ -285,7 +284,7 @@ The prerequisites are Doxygen and Graphviz. Tools to dump the \ref ScriptAPI "An
 
 \section Building_Clang_tools Clang-tools build
 
-If "URHO3D_CLANG_TOOLS" build option is set then CMake would generate a special build tree for the purpose of developing the Clang-tools. Before doing this, the development software package of LLVM/Clang must be already installed in the host system. Alternatively, you can download LLVM/Clang source, built it from source and install it into the host system. If it is not installed in a system-wide installation location then use the LLVM_CLANG_ROOT environment variable to point to the root path of this custom location (the root is the parent directory containing the bin, lib, include and share subdirectories). If you have built Emscripten-SDK in your host system then you can also just install the Fastcomp/Clang found inside the Emscripten-SDK by navigating to the Fastcomp/Clang build tree and issuing a 'make install' command. Using Fastcomp/Clang is fine because we are only interested in using Clang as 3rd-party library instead of as compiler.
+If "URHO3D_CLANG_TOOLS" build option is set then CMake would generate a special build tree for the purpose of developing the Clang-tools. Before doing this, the development software package of LLVM/Clang version 3.7.0 must be already installed in the host system. Alternatively, you can download LLVM/Clang source files from Clang's SVN repo, build and install it into the host system. If it is not installed in a system-wide installation location then use the LLVM_CLANG_ROOT environment variable to point to the root path of this custom location (the root is the parent directory containing the bin, lib, include and share subdirectories). If you have built Emscripten-SDK in your host system then you can also just install the Fastcomp/Clang found inside the Emscripten-SDK by navigating to the Fastcomp/Clang build tree and issuing a 'make install' command. Using Fastcomp/Clang is fine because we are only interested in using Clang as 3rd-party library instead of as compiler.
 
 Check the Source/Clang-Tools/CMakeLists.txt to get the list of targets currently available or under development. Normal build targets won't work properly in this special build tree. See also https://github.com/urho3d/Urho3D/issues/887 to find out more about the current development plan and leave a comment there if you would like to help to contribute the development.
 

Source/Clang-Tools/Annotator/Annotator.cpp

@@ -20,11 +20,12 @@
 // THE SOFTWARE.
 //
 
-#include <clang/Driver/Options.h>
+#include <clang/ASTMatchers/ASTMatchFinder.h>
 #include <clang/Tooling/CommonOptionsParser.h>
-#include <clang/Tooling/Tooling.h>
+#include <clang/Tooling/Refactoring.h>
 
 using namespace clang;
+using namespace clang::ast_matchers;
 using namespace clang::driver;
 using namespace clang::tooling;
 using namespace llvm;
@@ -39,25 +40,52 @@ static cl::extrahelp MoreHelp(
     "\tNote, that path/in/subtree and current directory should follow the\n"
     "\trules described above.\n"
     "\n"
+    "Most probably you want to invoke 'annotate' built-in target instead of invoking this tool\n"
+    "directly. The 'annotate' target invokes this tool in a right context prepared by build system.\n"
+    "\n"
 );
 
 static cl::OptionCategory AnnotatorCategory("Annotator options");
-static std::unique_ptr<opt::OptTable> Options(createDriverOptTable());
-static cl::opt<std::string> BindingsFile
-    ("b", cl::desc("Bindings file in JSON format (output of ScriptBindingExtractor tool)"), cl::cat(AnnotatorCategory));
 
-class AnnotateFrontendAction : public ASTFrontendAction
+class BindingCallback : public MatchFinder::MatchCallback
 {
-protected:
-    virtual std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance& CI, StringRef InFile)
+public :
+    virtual void run(const MatchFinder::MatchResult& result)
+    {
+        ASTContext* context = result.Context;
+        const StringLiteral* stringLiteral = result.Nodes.getNodeAs<StringLiteral>("StringLiteral");
+        if (stringLiteral)
+            // TODO: Store the exposed class name in a global hashset
+            outs() << stringLiteral->getString() << "\n";
+    }
+
+    virtual void onStartOfTranslationUnit()
     {
-        return make_unique<ASTConsumer>();
+        outs() << '.';
     }
 };
 
 int main(int argc, const char** argv)
 {
     CommonOptionsParser OptionsParser(argc, argv, AnnotatorCategory);
-    ClangTool Tool(OptionsParser.getCompilations(), OptionsParser.getSourcePathList());
-    return Tool.run(newFrontendActionFactory<AnnotateFrontendAction>().get());
+    ClangTool bindingExtractor(OptionsParser.getCompilations(), OptionsParser.getSourcePathList());
+    RefactoringTool annotator(OptionsParser.getCompilations(), OptionsParser.getSourcePathList());
+    BindingCallback bindingCallback;
+    MatchFinder bindingFinder;
+    bindingFinder.addMatcher(
+        memberCallExpr(
+            callee(
+                methodDecl(hasName("RegisterObjectType"))),
+            hasArgument(0, stringLiteral().bind("StringLiteral"))), &bindingCallback);
+    bindingFinder.addMatcher(
+        callExpr(
+            hasDeclaration(
+                functionDecl(hasParameter(1, hasName("className")))),
+            hasArgument(1, stringLiteral().bind("StringLiteral"))), &bindingCallback);
+    MatchFinder annotateFinder;
+    return (outs() << "Extracting", true) &&
+           bindingExtractor.run(newFrontendActionFactory(&bindingFinder).get()) == EXIT_SUCCESS &&
+           (outs() << "\nAnnotating", true) &&
+           annotator.runAndSave(newFrontendActionFactory(&annotateFinder).get()) == EXIT_SUCCESS &&
+           (outs() << "\n", true) ? EXIT_SUCCESS : EXIT_FAILURE;
 }

Source/Clang-Tools/CMakeLists.txt

@@ -44,27 +44,32 @@ set_tool_output_directories ()
 get_target_property (SOURCES Urho3D SOURCES)
 string (REGEX REPLACE "[^;]+\\.h" "" SOURCES "${SOURCES}")   # Stringify to preserve the semicolons
 string (REGEX REPLACE "[^;]+generated[^;]+\\.cpp" "" SOURCES "${SOURCES}")
-file (GLOB API_SOURCES RELATIVE ${CMAKE_SOURCE_DIR}/Source/Urho3D ${CMAKE_SOURCE_DIR}/Source/Urho3D/Script/*API.cpp)
+file (GLOB BINDING_SOURCES RELATIVE ${CMAKE_SOURCE_DIR}/Source/Urho3D ${CMAKE_SOURCE_DIR}/Source/Urho3D/Script/*API.cpp)
 
 # Define common dependency libs
-set (LIBS clangTooling clangFrontend clangDriver clangParse clangSerialization clangSema clangEdit clangAnalysis clangLex clangAST clangBasic
-        LLVMBitReader LLVMMC LLVMOption LLVMMCParser LLVMSupport)
+set (LIBS clangTooling clangFrontend clangDriver clangParse clangSerialization clangSema clangEdit clangAnalysis clangToolingCore
+        clangRewrite clangLex clangASTMatchers clangAST clangBasic
+        LLVMBitReader LLVMMC LLVMMCParser LLVMOption LLVMSupport)
 execute_process (COMMAND ${LLVM_CONFIG} --system-libs OUTPUT_VARIABLE LLVM_SYSLIBS OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_QUIET)
 string (REGEX REPLACE " *-l" ";" LLVM_SYSLIBS "${LLVM_SYSLIBS}")   # Stringify against empty output variable
 list (APPEND LIBS ${LLVM_SYSLIBS})
 
 # List of tools
-add_subdirectory (ScriptBindingExtractor)
 add_subdirectory (Annotator)
 
 # List of targets
-add_custom_target (ast      # Possible options are: dump|list|print; example usage: opt=dump make ast
-    COMMAND ${BINDIR}clang-check -p ${CMAKE_BINARY_DIR} -ast-$$opt ${API_SOURCES}
+add_custom_target (ast
+    COMMAND ${CMAKE_COMMAND} -E echo "Usage: option=-help make ast"
+    COMMAND ${BINDIR}clang-check -p ${CMAKE_BINARY_DIR} $$option ${SOURCES}
     WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/Source/Urho3D
-    COMMENT "Executing clang-check on AngelScript API bindings source files")
+    COMMENT "Executing clang-check on Urho3D library source files")
+add_custom_target (binding-ast
+    COMMAND ${CMAKE_COMMAND} -E echo "Usage: option=-help make binding-ast"
+    COMMAND ${BINDIR}clang-check -p ${CMAKE_BINARY_DIR} $$option ${BINDING_SOURCES}
+    WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/Source/Urho3D
+    COMMENT "Executing clang-check on (existing) AngelScript API bindings source files")
 add_custom_target (annotate
-    COMMAND ${CMAKE_BINARY_DIR}/bin/tool/ScriptBindingExtractor -p ${CMAKE_BINARY_DIR} ${API_SOURCES} >${CMAKE_BINARY_DIR}/bindings.json
-    COMMAND ${CMAKE_BINARY_DIR}/bin/tool/Annotator -p ${CMAKE_BINARY_DIR} -b ${CMAKE_BINARY_DIR}/bindings.json ${SOURCES}
-    DEPENDS ScriptBindingExtractor Annotator
+    COMMAND ${CMAKE_BINARY_DIR}/bin/tool/Annotator -p ${CMAKE_BINARY_DIR} ${SOURCES}
+    DEPENDS Annotator
     WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/Source/Urho3D
     COMMENT "Annotating Urho3D library source files")

Source/Clang-Tools/ScriptBindingExtractor/CMakeLists.txt

@@ -1,33 +0,0 @@
-#
-# Copyright (c) 2008-2015 the Urho3D project.
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-#
-
-# Define target name
-set (TARGET_NAME ScriptBindingExtractor)
-
-# Define source files
-define_source_files ()
-
-# Setup target
-if (APPLE)
-    setup_macosx_linker_flags (CMAKE_EXE_LINKER_FLAGS)
-endif ()
-setup_executable ()

Source/Clang-Tools/ScriptBindingExtractor/ScriptBindingExtractor.cpp

@@ -1,65 +0,0 @@
-//
-// Copyright (c) 2008-2015 the Urho3D project.
-//
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in
-// all copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-// THE SOFTWARE.
-//
-
-#include <clang/Driver/Options.h>
-#include <clang/Tooling/CommonOptionsParser.h>
-#include <clang/Tooling/Tooling.h>
-
-using namespace clang;
-using namespace clang::driver;
-using namespace clang::tooling;
-using namespace llvm;
-
-static cl::extrahelp CommonHelp(CommonOptionsParser::HelpMessage);
-static cl::extrahelp MoreHelp(
-    "\tFor example, to run ScriptBindingExtractor on all files in a subtree of the\n"
-    "\tsource tree, use:\n"
-    "\n"
-    "\t  find path/in/substree -name '*.cpp'|xargs ScriptBindingExtractor -p build/path\n"
-    "\n"
-    "\tNote, that path/in/subtree and current directory should follow the\n"
-    "\trules described above.\n"
-    "\n"
-);
-
-static cl::OptionCategory ScriptAPIExtractorCategory("ScriptBindingExtractor options");
-static std::unique_ptr<opt::OptTable> Options(createDriverOptTable());
-
-class ExtractASTConsumer : public ASTConsumer
-{
-};
-
-class ExtractFrontendAction : public ASTFrontendAction
-{
-protected:
-    virtual std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance& CI, StringRef InFile)
-    {
-        return make_unique<ExtractASTConsumer>();
-    }
-};
-
-int main(int argc, const char** argv)
-{
-    CommonOptionsParser OptionsParser(argc, argv, ScriptAPIExtractorCategory);
-    ClangTool Tool(OptionsParser.getCompilations(), OptionsParser.getSourcePathList());
-    return Tool.run(newFrontendActionFactory<ExtractFrontendAction>().get());
-}