Test Taxonomy: Scope

Tests have different scope.

• Unit Testing
• Integration Testing
• …

Test Taxonomy: Goal

Tests have different goals.

• Acceptance Testing
• Regression Testing
• …

Unit Testing

• Testing by developers for developers
• Code that tests code
• Automatic checks of what you know about code

Example Code

#include <solvers/solvers.h>

/**
 * @brief Solves k * x + d = 0 for x.
 */
double solve_linear_equation(const double k, const double d) {
  return -k / d;
}

Example Test

#include <gtest/gtest.h>
#include <solvers/solvers.h>

TEST(SolverTestSuite, solves_regular_linear_equation) {
  // Checks solution for k = 1/3 and d = 0.1.
  const auto result = solve_linear_equation(1.0 / 3.0, 0.1);
  EXPECT_DOUBLE_EQ(result, -0.3);
}

Run the Test

Not OK:

Running main() from gtest_main.cc
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SolverTestSuite
[ RUN      ] SolverTestSuite.solves_regular_linear_equation
../test/test_solvers.cpp:9: Failure
Expected equality of these values:
  solve_linear_equation(1.0 / 3.0, 0.1)
    Which is: -3.333333333333333
  -0.29999999999999999
[  FAILED  ] SolverTestSuite.solves_regular_linear_equation (0 ms)
[----------] 1 test from SolverTestSuite (0 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (0 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] SolverTestSuite.solves_regular_linear_equation

 1 FAILED TEST

Why testing?

Software development is complex.

• Codebase grows
• Requirements change
  • Code used differently
  • Used code changes

=> Rechecking manually not feasible

Test Driven Development

Write tests together with code.

1. Write tests until tests fail
2. Write code until tests pass
3. Refactor and repeat

Beyoncé Rule (Google)

🎵 If you liked it, then you shoulda put a test on it.

Project Structure

├── lib1
│   ├── CMakeLists.txt
│   ├── *.cpp
│   ├── include
│   │   └── lib1
│   │       └── *.h
│   └── test
│       ├── CMakeLists.txt
│       └── test_*.cpp
├── lib2
│   └── ...
├── ...
├── README.md
└── .gitignore

GTest Framework

• Nice output
• More checks
  • Float comparison
  • Expected exceptions
  • …

Remember Solver

#include <solvers/solvers.h>

/**
 * @brief Solves k * x + d = 0 for x.
 */
double solve_linear_equation(const double k, const double d) {
  return -k / d;
}

TestCase

#include <gtest/gtest.h>
#include <solvers/solvers.h>

namespace {
TEST(SolverTestSuite, solves_regular_linear_equation) {
  // Checks solution for k = 1/3 and d = 0.1.
  const auto result = solve_linear_equation(1.0 / 3.0, 0.1);
  EXPECT_DOUBLE_EQ(result, -0.3);
}

} // namespace

CMake config

cmake_minimum_required(VERSION 3.16.3)
project(solvers VERSION 0.0.1 LANGUAGES C CXX)

add_library(solvers solvers.cpp)
target_include_directories(solvers PUBLIC include)

enable_testing()
add_subdirectory(test)

find_package(GTest 1.8.1 CONFIG REQUIRED)

add_executable(test_solvers test_solvers.cpp)
target_link_libraries(test_solvers
                      PRIVATE solvers GTest::gtest_main)
add_test(NAME test_solvers
         COMMAND $<TARGET_FILE:test_solvers>
)

Run Tests

Start runner:

cmake --build build --target test
# or
cd build ; ctest # use "--output-on-failure" for verbose output

Test project solve_linear_equation/build
   Start 1: test_solvers
1/1 Test #1: test_solvers .....................***Failed    0.01 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.01 sec

The following tests FAILED:
      1 - test_solvers (Failed)
Errors while running CTest

Fix Tests

Correct wrong order of k and d:

#include <solvers/solvers.h>

/**
 * @brief Solves k * x + d = 0 for x.
 */
double solve_linear_equation(const double k, const double d) {
  return -d / k; // k and d replaced
}

Check Result

Restart runner:

cmake --build build && cmake --build build --target test

Test project solve_linear_equation/build
    Start 1: test_solvers
1/1 Test #1: test_solvers .....................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.00 sec

Special Cases

Add test to SolverTestSuite.

TEST(SolverTestSuite, throws_for_gradient_zero) {
  // Solve throws exception for gradient zero.
  EXPECT_THROW(solve_linear_equation(0.0, 0.1),
               std::invalid_argument);
}

Check Result

Restart runner:

cmake --build build && cmake --build build --target test

Test project solve_linear_equation/build
   Start 1: test_solvers
1/1 Test #1: test_solvers .....................***Failed    0.01 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.00 sec

The following tests FAILED:
      1 - test_solvers (Failed)
Errors while running CTest

Next Steps

• What happens when k=0 and d=0?
• …

Result

Verified Documentation.

TEST(SolverTestSuite, solves_regular_linear_equation) {
  // Checks solution for k = 1/3 and d = 0.1.
  const auto result = solve_linear_equation(1.0 / 3.0, 0.1);
  (...)
}

TEST(SolverTestSuite, throws_for_gradient_zero) {
  // Solve throws exception for gradient zero.
  (...)
}

Advantages

• High test coverage
• Simplified debugging
• Documentation
• Design

Exercise

gitlab.ethz.ch/sis/courses/cpp/testing/exercises/fizz_buzz

Know the Framework

Use suitable assertions.

• EXPECT_DOUBLE_EQ for floats
• Avoid EXPECT_TRUE(a == b)
• Many more… (see documentation)

Good Tests

Write your tests first. (Clean Code)

• F ast
• I solated
• R epeatable
• S elf-validating
• T imely

Fast

Prepare for 1000s of tests.

• Maximum: few milliseconds
• You should run all the tests often
• You will write many of them

Isolated

Minimal maintenance.

• Independent of each other
• Independent of behaviour tested elsewhere
• Independent of machine
• Independent of external data/network

Repeatable

Rerun to locate bugs.

• Same result every time you run it
• No random input
• Independent of environment (network, etc.)

Self-Validating

Do the work once.

• Test determines if result was expected
• No manual work involved
• Remember, you want to run them often

Timely

Write the test as soon as possible.

• You still know why you wrote the code that way
• You know the special cases
• You have (manual) test cases ready

Pattern

Given-When-Then Pattern. (Clean Code)

• Given: Setup of environment

• When: Execute what you want to test

• Then: Check result

• (if required): Cleanup

Recall Test

TEST(SolverTestSuite, solves_regular_linear_equation) {
  // Checks solution for k = 1/3 and d = 0.1.
  const auto result = solve_linear_equation(1./3., 0.1); // WHEN
  EXPECT_DOUBLE_EQ(result, -0.3); // THEN
}

Help the Reader

That’s why I chose 1/3 and 0.1.

TEST(SolverTestSuite, solves_regular_linear_equation) {
  // Checks solution for nonzero gradient, offset.
  const double nonzero_gradient = 1. / 3.;
  const double nonzero_offset = 0.1;

  const auto result = solve_linear_equation(nonzero_gradient,
                                            nonzero_offset);

  EXPECT_DOUBLE_EQ(result, -0.3);
}

Repeat

We could do the same thing here.

TEST(SolverTestSuite, throws_for_gradient_zero) {
  // Solve throws exception for gradient zero.
  EXPECT_THROW(solve_linear_equation(0.0, 0.1),
               std::invalid_argument);
}

Fixtures

We use a nontrivial test fixture.

struct SolverTest : public testing::Test {
protected:
  double nonzero_gradient = 1. / 3.;
  double nonzero_offset = 0.1;
  void SetUp() override {
    // Here you could do other stuff to initialize the fixture.
  }
};

Rollout

No need to copy and paste.

TEST_F(SolverTest, throws_for_gradient_zero) {
  // Solve throws exception for gradient zero.
  EXPECT_THROW(solve_linear_equation(0.0, nonzero_offset),
               std::invalid_argument);
}

Set the Stage

Use Fixtures.

• Provide data for the tests
• Initialize classes with nontrivial constructors
• Provide convenience methods
• Name a collection of tests

Challenges

You will be facing common challenges.

• I don’t know how to get started.
• I can’t test hidden/private code.
• I can’t test every combination of inputs.
• I don’t understand what a function does.
• The function has 20 parameters, so 2^20 tests?
• The function needs to read a file.

Getting Started

Break the vicious circle by adding integration tests.

• Testing requires refactoring.
• Refactoring requires tests.

Private Code

I can’t test hidden/private code.

No problem, everybody else can’t, either.

• Test what users can do with your code
• Extract code if this makes testing difficult

Lean Testing

I can’t test every combination of inputs.

Test until you are confident that your code works.

• Test the uses cases the unit was designed for
• Test ‘boundaries’ and special cases
• Add tests when you discover bugs
• ‘Code coverage’ tools can help

No Documentation

I don’t understand what a function does.

No problem, use unit testing.

• Write a test that checks any assumption
• Look at the actual output
• Fix the test
• Repeat

Complex Interface

The function has 20 parameters.

Unit testing tests atomic units.

• Add regression tests first
• Try to extract atomic units and test them
  • 20 one-boolean-parameter-functions: 40 tests
  • One 20-boolean-parameter-function: ~1M tests
• Add integration tests for the (important) use cases

Interoperation

The function needs to read a file.

#include <fstream>
#include <string>

std::string bad_function() {
  std::ifstream input;
  input.open("data.txt");
  std::string content;
  input >> content;
  return content + content;
}

Split IO from processing!

Split IO from processing:

#include <fstream>
#include <string>

std::string read(const std::string &path) {
  std::ifstream input;
  input.open(path);
  std::string content;
  input >> content;
  return content;
}

std::string process(const std::string &content) {
  return content + content;
}

Mocking

Might help in the “Interoperation” case.

More info: gMock.

Integration tests

Testing of higher level logic.

CMake setup

Example: run a Python script to orchestrate and check multiple binaries.

add_test(NAME integration_tests
         COMMAND python ./integration_tests/run.py)

Other languages

Unit Testing is not language-specific.

• Principles apply to software engineering in general
• Unit testing frameworks (e.g.)
  • C++: googletest
  • Python: unittest, pytest
  • R: testthat
  • Java: JUnit

Benefits

Correctness and more.

• Check new code for bugs
• Check old code for new bugs after modifications
• Documentation
• Tested/able code is often good code
  • Small units (simpler, reuseable)
  • Special cases covered
  • Easy to improve later

Questions?

Acknowledgements

• Slides:
  • Reveal.js, MIT
  • Pandoc, GPLv2+, BSD 3-clause
• Based on a talk by Manuel Weberndorfer and Uwe Schmitt