unit_test_framework

Tutorial: Unit Test Framework

In this tutorial, you will learn how to write test cases using a lightweight framework that functions similarly to unit test frameworks used in real-life software development.

Table of Contents

Setting Up

First, you will need the files unit_test_framework.h and unit_test_framework.cpp. These files are available with the starter files for Lab 2, but you can also download them with the following commands.

$ wget https://raw.githubusercontent.com/eecs280staff/unit_test_framework/master/unit_test_framework.cpp
$ wget https://raw.githubusercontent.com/eecs280staff/unit_test_framework/master/unit_test_framework.h

Second, create a file called lab02_tests.cpp. Add the following code to lab02_tests.cpp:

#include "lab02.h"
#include "unit_test_framework.h"

// We define a test case with the TEST(<test_name>) macro.
// <test_name> can be any valid C++ function name.
TEST(true_is_true) {
  ASSERT_TRUE(true);
}

TEST(numbers_are_equal) {
  ASSERT_EQUAL(42, 42);
}

TEST_MAIN() // No semicolon!

You’re probably wondering why some of the syntax in this code looks unusual. That’s because this testing framework uses preprocessor macros to achieve functionality that wouldn’t be possible with the plain C++ you’re used to seeing. Preprocessor macros are beyond the scope of this course and in general should be used sparingly, so here’s all you need to know:

Compile and run this test case with the following two commands:

$ g++ -Wall -Werror -pedantic -g -std=c++11 lab02.cpp lab02_tests.cpp unit_test_framework.cpp -o lab02_tests.exe
$ ./lab02_tests.exe
Running test: numbers_are_equal
PASS
Running test: true_is_true
PASS

*** Results ***
** Test case 'numbers_are_equal': PASS
** Test case 'true_is_true': PASS
*** Summary ***
Out of 2 tests run:
0 failure(s), 0 error(s)

Another nice feature of the framework is that we can tell it to run only a subset of our test cases. If we wanted to only run the test numbers_are_equal, we could do it with this command:

$ ./lab02_tests.exe numbers_are_equal
Running test: numbers_are_equal
PASS

*** Results ***
** Test case 'numbers_are_equal': PASS
*** Summary ***
Out of 1 tests run:
0 failure(s), 0 error(s)

You can pass any number of test names as command line arguments, and it will only run the ones you’ve specified.

Special Test Assertions

One of the main reasons for using the special assertions provided by the framework is that they allow the framework to run all of your tests and report which ones passed and which ones failed. As you may have noticed, when you use regular assert() in your test cases, they automatically stop at the first failure. This can make it difficult to debug errors in one test that are actually caused by a function whose test cases didn’t get a chance to run yet.

Here is a summary of all the special assertions that the framework provides:

Assertion Description
ASSERT_EQUAL(first, second) If first == second evaluates to false, the test will fail. Note: Do not use this if first and second are not comparable using the == operator. Other than this restriction, first and second may be any type.
ASSERT_NOT_EQUAL(first, second) If first != second evaluates to false, the test will fail. Note: Do not use this if first and second are not comparable using the != operator. Other than this restriction, first and second may be any type.
ASSERT_TRUE(bool value) If value is false, the test will fail.
ASSERT_FALSE(bool value) If value is true, the test will fail.
ASSERT_ALMOST_EQUAL(double first, double second, double precision) If first and second are not equal within precision, the test will fail.

Example: Tests for an add() function

Suppose we have an add() function that computes the sum of two double arguments:

double add(double first, double second) {
  return first + second;
}

We could write a test case for add() using assert() as follows:

void add_test_basic();
bool doubles_close(double first, double second, double range);

int main() {
  add_test_basic();
}

void add_test_basic() {
  double a = 0.1;
  double b = 0.2;
  double expected_sum = 0.3;
  double actual_sum = add(a, b);

  cout << setprecision(20);
  cout << "expected_sum: " << expected_sum << endl;
  cout << "actual_sum: " << actual_sum << endl;

  assert(doubles_close(expected_sum, actual_sum, .001));
}

bool doubles_close(double first, double second, double range) {
  double diff = abs(first - second);
  return diff < range;
}

The doubles_close() function above determines if two double values are equal to each other within a given range of precision. Since double values are not exact, it isn’t safe to compare non-integral double values with the == operator.

Let’s rewrite the test case above using the unit test framework. First, let’s turn add_test_basic into a TEST() and replace assert() with ASSERT_TRUE():

TEST(add_basic) {
  double a = 0.1;
  double b = 0.2;
  double expected_sum = 0.3;
  double actual_sum = add(a, b);

  cout << setprecision(20);
  cout << "expected_sum: " << expected_sum << endl;
  cout << "actual_sum: " << actual_sum << endl;

  ASSERT_TRUE(doubles_close(expected_sum, actual_sum, .001));
}

Next, instead of using our own doubles_close() function, let’s use ASSERT_ALMOST_EQUAL() from the framework:

TEST(add_basic) {
  double a = 0.1;
  double b = 0.2;
  double expected_sum = 0.3;
  double actual_sum = add(a, b);

  ASSERT_ALMOST_EQUAL(expected_sum, actual_sum, .001);
}

ASSERT_ALMOST_EQUAL() and ASSERT_EQUAL() will print out the expected and actual values for us if they’re different. Finally, let’s add TEST_MAIN() and any #includes that we need:

#include “eecs280math.h”
#include “unit_test_framework.h”

TEST(add_basic) {
  double a = 0.1;
  double b = 0.2;
  double expected_sum = 0.3;
  double actual_sum = add(a, b);

  ASSERT_ALMOST_EQUAL(expected_sum, actual_sum, .001);
}

TEST_MAIN()

Write Unit Tests for slideright() and flip() (Lab 2)

Add test cases for slideRight() and flip() to lab02_tests.cpp.

Compile and run the tests with the following commands:

$ g++ -Wall -Werror -pedantic -O1 -std=c++11 lab02.cpp lab02_tests.cpp unit_test_framework.cpp -o lab02_tests.exe
$ ./lab02_tests.exe

Of course, if you run your tests against your own (correct) implementations in lab02.cpp, the tests should all pass. (We call tests that pass when run against a correct implementation valid.) You can temporarily change one of your functions to contain a bug and observe that some tests fail.

Once you feel your tests are thorough, submit lab02_tests.cpp to the lab 2 autograder. It will check your tests against a set of buggy implementations of slideRight and flip. To earn points for the lab, your tests must detect (i.e. fail when run against) each of the bugs. Note that the autograder will discard any tests that are not valid when checked against a correct solution.

Write a compare_arrays() Function (Lab 2)

You may have noticed some duplicated code in your test cases, particularly when checking the expected and actual contents of arrays. Write a function called compare_arrays() that takes in two arrays and their lengths and checks that the contents of those arrays are equal using the framework’s special assertions. Then, refactor your test cases to use compare_arrays() instead of duplicating code.

Note: Since compare_arrays() is a test helper, it should be within lab02_tests.cpp!

Extra: Convert your Project 1 Tests to use the Framework

If you want extra practice with the framework before using it for your test cases in Project 2, update your unit tests for stats.cpp so that they use this framework.

Note: Do NOT submit these updated test cases to the Project 1 autograder, as the framework will not be available there.

Hint: You can take advantage of ASSERT_ALMOST_EQUAL() to compare doubles in your test cases.