Merge pull request #70 from SimeonEhrig/addBenchmarks

add benchmarks

CMakeLists.txt

@@ -23,6 +23,8 @@ option(VIKUNJA_ENABLE_EXTRA_WARNINGS "Enable extra warnings" OFF)
 option(BUILD_TESTING "Build the testing tree." OFF)
 cmake_dependent_option(VIKUNJA_SYSTEM_CATCH2 "Use your local installation of Catch2" ON BUILD_TESTING OFF)
 cmake_dependent_option(VIKUNJA_ENABLE_CXX_TEST "Builds test that checks if the C++ standard is set correctly" OFF BUILD_TESTING OFF)
+cmake_dependent_option(VIKUNJA_ENABLE_BENCHMARKS "Enable benchmarks" OFF BUILD_TESTING OFF)
+cmake_dependent_option(VIKUNJA_ENABLE_CUDA_THRUST_BENCHMARKS "Enable benchmarks using CUDA Thrust" OFF "VIKUNJA_ENABLE_BENCHMARKS;ALPAKA_ACC_GPU_CUDA_ENABLE" OFF)
 
 # activate support for host/device lambdas in cuda
 # needs to be set before alpaka is included

docs/source/advanced/cmake.rst

@@ -20,6 +20,7 @@ Common
 
 Testing
 +++++++
+.. _cmake-test:
 
 **BUILD_TESTING** (OFF)
     .. code-block::
@@ -29,16 +30,29 @@ Testing
 **VIKUNJA_SYSTEM_CATCH2** (OFF)
     .. code-block::
 
-        Only works if BUILD_TESTING is ON.
+        Requires BUILD_TESTING to be ON.
         Use your local installation of Catch2.
         Otherwise, it will be automatically downloaded and installed in the local build folder.
 
 **VIKUNJA_ENABLE_CXX_TEST** (OFF)
     .. code-block::
 
-        Only works if BUILD_TESTING is ON.
+        Requires BUILD_TESTING to be ON.
         Special test that checks if ALPAKA_CXX_STANDARD works correctly.
-        The implementation is very compiler specific, so it is possible that the test is not supported by your used C++ compiler.
+        The implementation is very compiler specific, so it is possible that the test is not
+        supported by your C++ compiler.
+
+**VIKUNJA_ENABLE_BENCHMARKS** (OFF)
+    .. code-block::
+
+        Requires BUILD_TESTING to be ON.
+        Enables the benchmarks. The benchmarks are built automatically and can be executed via CTest.
+
+**VIKUNJA_ENABLE_CUDA_THRUST_BENCHMARKS** (OFF)
+    .. code-block::
+
+        Requires VIKUNJA_ENABLE_BENCHMARKS and ALPAKA_ACC_GPU_CUDA_ENABLE to be ON.
+        Enables Thrust benchmarks for comparison.
 
 alpaka
 ++++++
@@ -64,7 +78,7 @@ The following CMake variables are provided by alpaka. This section contains only
         - ALPAKA_ACC_GPU_CUDA_ENABLE
         - ALPAKA_ACC_GPU_HIP_ENABLE
 
-        Important: Not all alpaka accelerator backends are tested together with vikunja, 
+        Important: Not all alpaka accelerator backends are tested together with vikunja,
         see CI tests.
 
 **ALPAKA_CUDA_NVCC_EXPT_EXTENDED_LAMBDA** (ON)
@@ -75,4 +89,4 @@ The following CMake variables are provided by alpaka. This section contains only
 **ALPAKA_CUDA_EXPT_EXTENDED_LAMBDA** (ON)
     .. code-block::
 
-        Enable lambda support in Alpaka 0.7.x and later for the CUDA accelerator.
+        Enable lambda support in Alpaka 0.7.x and later for the CUDA accelerator.

docs/source/basic/algorithm.rst

@@ -1,5 +1,5 @@
 Algorithms
-=========
+==========
 
 This page provides an overview of all algorithms implemented in vikunja.
 
@@ -8,7 +8,7 @@ All algorithms have the property that the order in which the input elements are
 Transform
 ---------
 
-Takes a range of elements as input, applies an unary operator to each element, and writes the result to an output range in the same order. 
+Takes a range of elements as input, applies an unary operator to each element, and writes the result to an output range in the same order.
 
 .. only:: html
 
@@ -35,4 +35,4 @@ Takes a range of elements as input and reduces it to a single element via an ope
 .. only:: latex
 
   .. image:: images/reduction.pdf
-    :alt: scheme: reduce algorithm
+    :alt: scheme: reduce algorithm

docs/source/basic/installation.rst

@@ -19,8 +19,8 @@ Vikunja builds and installs itself using `CMake <https://cmake.org/>`_. Before y
 .. code-block:: bash
 
   git clone https://github.com/alpaka-group/vikunja.git
-  mkdir vikunja/build 
-  cd vikunja/build  
+  mkdir vikunja/build
+  cd vikunja/build
   cmake ..
   cmake --build .
   cmake --install .
@@ -38,6 +38,8 @@ Enable and run the tests:
    cmake --build .
    ctest
 
+Read this :doc:`section </development/test>` for more information about the tests.
+
 Enable and run an example:
 
 .. code-block:: bash

docs/source/basic/introduction.rst

@@ -7,7 +7,7 @@ The basic concept of vikunja is to run an ``algorithm`` with an ``operator`` ove
 
   * **Transform**: Takes a range of elements as input, applies an operator to each element, and writes the result to an output range.
   * **Reduce**: Takes a range of elements as input and returns a single element. The reduce operator takes two elements of the input range, applies an operation to them, and returns a single element. The operator is applied up to the point where only one element remains.
-  * For more examples see: :ref:`Algorithm <Algorithm>`
+  * For more examples see: :doc:`Algorithm </basic/algorithm>`
 * An ``operator`` describes an algorithm which is applied to one (unary operator) or two (binary operator) elements and returns a result. The following examples assume that **i** is the first and **j** the second input element:
 
   * **sum**: `return i+j;`

docs/source/development/test.rst

@@ -0,0 +1,59 @@
+Testing and Benchmarking
+========================
+
+Vikunja offers different types of tests. The source code is tested via unit and integration tests with `Catch2 <https://github.com/catchorg/Catch2/tree/v2.x>`_. The CMake code is tested with integration tests and custom scripts.
+
+Source Code Tests
+-----------------
+
+Before you start writing source code tests, you should read the `Catch2 documentation <https://github.com/catchorg/Catch2/blob/v2.x/docs/tutorial.md#top>`_. Tests written with Catch2 are standalone executables. They have their own source code files and ``CMakeLists.txt`` files located in the ``test/unit`` and ``test/integ`` folders. If you set the CMake argument ``-DBUILD_TESTING=ON``, the tests will be built automatically. All test executables are registered via the CMake function ``add_test``. Therefore, you can automatically run all tests from the build folder with the ``ctest`` command:
+
+.. code-block:: bash
+
+    mkdir build && cd build
+    cmake .. -DBUILD_TESTING=ON
+    cmake --build .
+    ctest
+
+For more CMake arguments for the tests, see the :ref:`CMake section <cmake-test>`.
+
+If you only want to run a single test, you can run the test executable directly. All test executables are located in ``<build_folder>/tests``. It is also possible to run the executable with the ``--help`` flag to show additional options. For example, the ``-s`` flag displays additional information created with the Catch2 function ``INFO()``.
+
+.. code-block:: bash
+
+    mkdir build && cd build
+    cmake .. -DBUILD_TESTING=ON
+    cmake --build .
+    # display extra test options
+    test/integ/reduce/test_reduce --help
+    # run test with extra output
+    test/integ/reduce/test_reduce -s
+
+.. tip::
+
+    Each test is a CMake target that you can build separately. A test target always starts with ``test_``. To get all available test CMake targets, run ``cmake --build . -t help | grep 'test_'`` in the build folder. You can build a specific test with ``cmake --build . -t test_IndividualTestCase``.
+
+CMake Tests
+-----------
+
+The CMake integration tests check whether vikunja can be used correctly in another project via the CMake functions ``find_package()`` or ``add_subdirectory``. The CI contains test jobs which create dummy projects that use the vikunja library. The job names start with ``integration``. All associated files for the tests are in ``script/integration_test``.
+
+CXX Test
+++++++++
+
+There is a special Catch2 test that tests vikunja's build system to see if the C++ standard is set correctly. The name of the test is ``test_cxx``. It compares the C++ standard set by the compiler with an expected standard passed as an argument. By default, ``ctest`` automatically passes the expected C++ standard depending on the CMake variable ``ALPAKA_CXX_STANDARD``. If you run the test manually, you must pass it yourself:
+
+.. code-block:: bash
+
+    # expects, that the code was compiled with C++ 17
+    test/unit/cxx/test_cxx --cxx 17
+
+
+Benchmarks
+----------
+
+Vikunja uses `Catch2 benchmark <https://github.com/catchorg/Catch2/blob/v2.x/docs/benchmarks.md#top>`_ to automatically run benchmarks. By default, benchmarks are not enabled. To enable them, the CMake arguments ``-DBUILD_TESTING=ON -DVIKUNJA_ENABLE_BENCHMARKS=ON`` must be set. The benchmarks are created automatically and can be run with ``ctest``. As with the tests, you can run a particular benchmark directly from the executable file, e.g. ``test/benchmarks/transform/bench_vikunja_transform``. All benchmark executables are located in ``<build_folder>/test/benchmarks``.
+
+.. tip::
+
+    If you run ``<benchmark_exe> --help``, you get benchmark specific options.

docs/source/index.rst

@@ -41,6 +41,7 @@ Generally, **follow the manual pages in-order** to get started. Individual chapt
    :maxdepth: 1
    :caption: Development
 
+   development/test.rst
    development/docs.rst
    development/styleguide.rst
    development/ci.rst

test/CMakeLists.txt

@@ -33,8 +33,14 @@ target_link_libraries(vikunjaTestSetup
   PUBLIC
   Catch2::Catch2
   )
+if(VIKUNJA_ENABLE_BENCHMARKS)
+  target_compile_definitions(vikunjaTestSetup PRIVATE CATCH_CONFIG_ENABLE_BENCHMARKING)
+endif()
 add_library(vikunja::testSetup ALIAS vikunjaTestSetup)
 
 list(APPEND _VIKUNJA_TEST_OPTIONS "--use-colour yes")
 add_subdirectory("unit/")
 add_subdirectory("integ/")
+if(VIKUNJA_ENABLE_BENCHMARKS)
+    add_subdirectory("benchmarks/")
+endif()

test/benchmarks/CMakeLists.txt

@@ -0,0 +1,18 @@
+# Copyright 2022 Simeon Ehrig
+#
+# This file is part of vikunja.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+cmake_minimum_required(VERSION 3.18)
+
+add_library(vikunjaBenchSetup INTERFACE)
+target_compile_definitions(vikunjaBenchSetup INTERFACE CATCH_CONFIG_ENABLE_BENCHMARKING)
+target_include_directories(vikunjaBenchSetup INTERFACE include)
+add_library(vikunja::benchSetup ALIAS vikunjaBenchSetup)
+
+add_subdirectory("helper/")
+add_subdirectory("transform/")
+add_subdirectory("reduce/")

test/benchmarks/helper/CMakeLists.txt

@@ -0,0 +1,25 @@
+# Copyright 2022 Simeon Ehrig
+#
+# This file is part of vikunja.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+cmake_minimum_required(VERSION 3.18)
+
+set(_TARGET_NAME "test_bench_helper")
+
+alpaka_add_executable(
+  ${_TARGET_NAME}
+  test_bench_helper.cpp
+  )
+
+target_link_libraries(${_TARGET_NAME}
+  PRIVATE
+  vikunja::testSetup
+  vikunja::benchSetup
+  vikunja::internalvikunja
+)
+
+add_test(NAME ${_TARGET_NAME} COMMAND ${_TARGET_NAME} ${_VIKUNJA_TEST_OPTIONS})

test/benchmarks/helper/test_bench_helper.cpp

@@ -0,0 +1,123 @@
+/* Copyright 2022 Simeon Ehrig
+ *
+ * This file is part of vikunja.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+#include <vikunja/bench/memory.hpp>
+#include <vikunja/test/AlpakaSetup.hpp>
+#include <vikunja/test/utility.hpp>
+
+#include <alpaka/alpaka.hpp>
+#include <alpaka/example/ExampleDefaultAcc.hpp>
+
+#include <catch2/catch.hpp>
+
+
+TEMPLATE_TEST_CASE("allocate_mem_iota compare std::iota", "[iota]", int, float, double)
+{
+    using Data = TestType;
+    using Setup = vikunja::test::TestAlpakaSetup<
+        alpaka::DimInt<1u>, // dim
+        int, // Idx
+        alpaka::AccCpuSerial, // host type
+        alpaka::ExampleDefaultAcc, // device type
+        alpaka::Blocking // queue type
+        >;
+    using Vec = alpaka::Vec<Setup::Dim, Setup::Idx>;
+
+    Setup::Idx size = GENERATE(1, 10, 3045, 2'000'000);
+    Data begin = GENERATE(0, 1, 45, -42);
+
+    INFO((vikunja::test::print_acc_info<Setup::Dim>(size)));
+    INFO("begin: " + std::to_string(begin));
+
+    Setup setup;
+    Vec extent = Vec::all(static_cast<Setup::Idx>(size));
+
+    auto devMem = vikunja::bench::allocate_mem_iota<Data>(setup, extent, begin);
+    auto hostMem(alpaka::allocBuf<Data, typename Setup::Idx>(setup.devHost, extent));
+    Data* const hostMemPtr(alpaka::getPtrNative(hostMem));
+
+    alpaka::memcpy(setup.queueAcc, hostMem, devMem, extent);
+
+    std::vector<Data> expected_result(size);
+    std::iota(std::begin(expected_result), std::end(expected_result), begin);
+
+    for(Setup::Idx i = 0; i < size; ++i)
+    {
+        REQUIRE(static_cast<Data>(expected_result[i]) == hostMemPtr[i]);
+    }
+}
+
+TEMPLATE_TEST_CASE("allocate_mem_iota different increment", "[iota]", int, float, double)
+{
+    using Data = TestType;
+    using Setup = vikunja::test::TestAlpakaSetup<
+        alpaka::DimInt<1u>, // dim
+        int, // Idx
+        alpaka::AccCpuSerial, // host type
+        alpaka::ExampleDefaultAcc, // device type
+        alpaka::Blocking // queue type
+        >;
+    using Vec = alpaka::Vec<Setup::Dim, Setup::Idx>;
+
+    Setup::Idx size = GENERATE(1, 10, 3045);
+    Data begin = GENERATE(0, 1, 45, -42);
+    Data increment = GENERATE(1, -1, 45, -42);
+
+    INFO((vikunja::test::print_acc_info<Setup::Dim>(size)));
+    INFO("begin: " + std::to_string(begin));
+    INFO("increment: " + std::to_string(increment));
+
+    Setup setup;
+    Vec extent = Vec::all(static_cast<Setup::Idx>(size));
+
+    auto devMem = vikunja::bench::allocate_mem_iota<Data>(setup, extent, begin, increment);
+    auto hostMem(alpaka::allocBuf<Data, typename Setup::Idx>(setup.devHost, extent));
+    Data* const hostMemPtr(alpaka::getPtrNative(hostMem));
+
+    alpaka::memcpy(setup.queueAcc, hostMem, devMem, extent);
+
+    for(Setup::Idx i = 0; i < size; ++i)
+    {
+        Data expected_result = begin + static_cast<Data>(i) * increment;
+        REQUIRE_MESSAGE(expected_result == hostMemPtr[i], "failed with index: " + std::to_string(i));
+    }
+}
+
+TEMPLATE_TEST_CASE("allocate_mem_constant", "[iota]", int, float, double)
+{
+    using Data = TestType;
+    using Setup = vikunja::test::TestAlpakaSetup<
+        alpaka::DimInt<1u>, // dim
+        int, // Idx
+        alpaka::AccCpuSerial, // host type
+        alpaka::ExampleDefaultAcc, // device type
+        alpaka::Blocking // queue type
+        >;
+    using Vec = alpaka::Vec<Setup::Dim, Setup::Idx>;
+
+    Setup::Idx size = GENERATE(1, 10, 3045, 2'000'000);
+    Data constant = GENERATE(0, 1, 45, -42);
+
+    INFO((vikunja::test::print_acc_info<Setup::Dim>(size)));
+    INFO("constant: " + std::to_string(constant));
+
+    Setup setup;
+    Vec extent = Vec::all(static_cast<Setup::Idx>(size));
+
+    auto devMem = vikunja::bench::allocate_mem_constant<Data>(setup, extent, constant);
+    auto hostMem(alpaka::allocBuf<Data, typename Setup::Idx>(setup.devHost, extent));
+    Data* const hostMemPtr(alpaka::getPtrNative(hostMem));
+
+    alpaka::memcpy(setup.queueAcc, hostMem, devMem, extent);
+
+    for(Setup::Idx i = 0; i < size; ++i)
+    {
+        REQUIRE(static_cast<Data>(constant) == hostMemPtr[i]);
+    }
+}