p1-stats

Valgrind

This tutorial will run the Valgrind utility to check for undefined behavior. Valgrind can find problems like:

Uninitialized variables
Going off the end of an array or vector
Pointers wandering off target
Memory leaks

We recommend running Valgrind on CAEN Linux because it can give different results on different machines and is difficult to configure (more details).

Prerequisites

We’re assuming that you already have a folder with starter source code in it, e.g., p1-stats/.

$ pwd
/Users/awdeorio/src/eecs280/p1-stats
$ ls
Makefile      main_test.out.correct  p1_library.h  stats_public_test.cpp
main.cpp      main_test_data.tsv     stats.cpp     stats_tests.cpp
main_test.in  p1_library.cpp         stats.h

Your tests compile and run with no assertion failures. It’s OK if your tests don’t pass. If your test fails on an assertion, fix that before using Valgrind. Only use Valgrind on tests that produce no assertion failures.

$ make clean
rm -rvf *.exe *~ *.out *.dSYM *.stackdump
$ make test
g++ -Wall -Werror -pedantic -g --std=c++11 main.cpp stats.cpp p1_library.cpp -o main.exe
g++ -Wall -Werror -pedantic -g --std=c++11 stats_tests.cpp stats.cpp p1_library.cpp -o stats_tests.exe
g++ -Wall -Werror -pedantic -g --std=c++11 stats_public_test.cpp stats.cpp p1_library.cpp -o stats_public_test.exe
./stats_public_test.exe
count3
sum6
mean2
median2
mode1
min1
max3
stdev1
percentile2
./stats_tests.exe
test_sum_small_data_set
PASS!
./main.exe < main_test.in > main_test.out
diff main_test.out main_test.out.correct

You can connect to CAEN Linux. See the CAEN Linux tutorial for help.

$ ssh awdeorio@login.engin.umich.edu
The authenticity of host 'login.engin.umich.edu (141.213.74.65)' can't be established.
ECDSA key fingerprint is SHA256:LL0GPTtaVGa6gvv2kVpGq4ZULA1l5pw2wXC4dK3ymIk.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'login.engin.umich.edu,141.213.74.65' (ECDSA) to the list of known hosts.
Password: 
Duo two-factor login for awdeorio

Enter a passcode or select one of the following options:

 1. Duo Push to XXX-XXX-2735
 2. Phone call to XXX-XXX-2735
 3. SMS passcodes to XXX-XXX-2735

Passcode or option (1-3): 1
Success. Logging you in...
...
-bash-4.2$ exit
Connection to login.engin.umich.edu closed.

Copy code to CAEN Linux

We’re going to synchronize our code to CAEN Linux many times, so be sure to set up these short cuts.

Be sure that you can avoid repeated two-factor authentication (2FA). In a separate terminal, connect to CAEN Linux with SSH so you won’t have to enter your password again.

$ ssh awdeorio@login.engin.umich.edu
Password:
Duo two-factor login for awdeorio

Enter a passcode or select one of the following options:

 1. Duo Push to XXX-XXX-2735
 2. Phone call to XXX-XXX-2735
 3. SMS passcodes to XXX-XXX-2735

Passcode or option (1-3): 1
Success. Logging you in...

$

If you didn’t already, add a make sync shortcut to your Makefile.

Clean up (from your laptop).

$ hostname
manzana.local
$ make clean
rm -rvf *.exe *~ *.out *.dSYM *.stackdump
removed 'main.exe'
removed 'stats_tests.exe'
removed 'stats_public_test.exe'

Copy (from your laptop).

$ hostname
manzana.local
$ make sync
rsync \
  -rtv \
  --delete \
  --exclude '.git*' \
  --filter=':- .gitignore' \
  ../p1-stats/ \
  awdeorio@login.engin.umich.edu:p1-stats-copy/
building file list ... done
created directory p1-stats-copy
./
Makefile
main.cpp
main_test.in
main_test.out.correct
main_test_data.tsv
p1_library.cpp
p1_library.h
stats.cpp
stats.h
stats_public_test.cpp
stats_tests.cpp

sent 9557 bytes  received 268 bytes  19650.00 bytes/sec
total size is 8818  speedup is 0.90

Connect to CAEN Linux

Connect to CAEN Linux via ssh. If you’re already connected, you don’t need to connect again.

$ ssh awdeorio@login.engin.umich.edu
Success. Logging you in...
...
$ hostname
caen-vnc-vm16.engin.umich.edu

Change directory to the folder containing your source code. Depending on whether you copied with rsync or cloned with git, your folder name might be different.

$ ls
p1-stats
$ cd p1-stats

Run Valgrind on a unit test

Reminder: you should be SSH’ed into CAEN Linux, and in your project directory.

$ hostname
caen-vnc-vm16.engin.umich.edu
$ pwd
/home/awdeorio/p1-stats

Clean and build.

$ make clean
rm -rvf *.exe *~ *.out *.dSYM *.stackdump
$ make stats_tests.exe
g++ -Wall -Werror -pedantic -g --std=c++11 stats_tests.cpp stats.cpp p1_library.cpp -o stats_tests.exe

Run without Valgrind. Make sure that no assertions fail. If your program fails on an assertion, stop and fix that first.

$ ./stats_tests.exe 
test_sum_small_data_set
PASS!

Run with Valgrind. Notice the output said 0 errors from 0 contexts, and didn’t generate any errors.

$ valgrind ./stats_tests.exe 
==24672== Memcheck, a memory error detector
==24672== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==24672== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info
==24672== Command: ./stats_tests.exe
==24672== 
test_sum_small_data_set
PASS!
==24672== 
==24672== HEAP SUMMARY:
==24672==     in use at exit: 0 bytes in 0 blocks
==24672==   total heap usage: 4 allocs, 4 frees, 80 bytes allocated
==24672== 
==24672== All heap blocks were freed -- no leaks are possible
==24672== 
==24672== For counts of detected and suppressed errors, rerun with: -v
==24672== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Run Valgrind on a system test

Reminder: you should be SSH’ed into CAEN Linux, and in your project directory.

$ hostname
caen-vnc-vm16.engin.umich.edu
$ pwd
/home/awdeorio/p1-stats

Clean and build.

$ make clean
rm -rvf *.exe *~ *.out *.dSYM *.stackdump
$ make main.exe
g++ -Wall -Werror -pedantic -g --std=c++11 main.cpp stats.cpp p1_library.cpp -o main.exe

Run without Valgrind. Make sure that no assertions fail. If your program fails on an assertion, stop and fix that first.

$ ./main.exe
enter a filename
main_test_data.tsv
enter a column name
B
reading column B from main_test_data.tsv
Summary (value: frequency)
6: 1
7: 1
8: 1
9: 1
10: 1

count = 5
sum = 40
mean = 8
stdev = 1.58114
median = 8
mode = 6
min = 6
max = 10
  0th percentile = 6
 25th percentile = 7
 50th percentile = 8
 75th percentile = 9
100th percentile = 10

Run with Valgrind. Enter user input if the program prompts you. Verify that the output said 0 errors from 0 contexts, and didn’t generated any errors.

$ valgrind ./main.exe
==25049== Memcheck, a memory error detector
==25049== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==25049== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info
==25049== Command: ./main.exe
==25049== 
enter a filename
main_test_data.tsv
enter a column name
B
reading column B from main_test_data.tsv
Summary (value: frequency)
6: 1
7: 1
8: 1
9: 1
10: 1

count = 5
sum = 40
mean = 8
stdev = 1.58114
median = 8
mode = 6
min = 6
max = 10
  0th percentile = 6
 25th percentile = 7
 50th percentile = 8
 75th percentile = 9
100th percentile = 10
==25049== 
==25049== HEAP SUMMARY:
==25049==     in use at exit: 0 bytes in 0 blocks
==25049==   total heap usage: 75 allocs, 75 frees, 11,226 bytes allocated
==25049== 
==25049== All heap blocks were freed -- no leaks are possible
==25049== 
==25049== For counts of detected and suppressed errors, rerun with: -v
==25049== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Different program inputs can produce different Valgrind output. Let’s run it again on the Real Data Example from project 1.

$ valgrind ./main.exe 
==11464== Memcheck, a memory error detector
==11464== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==11464== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info
==11464== Command: ./main.exe
==11464== 
enter a filename
HCMST_ver_3.04.tsv
enter a column name
qflag
reading column qflag from HCMST_ver_3.04.tsv
Summary (value: frequency)
1: 3009
2: 993

count = 4002
sum = 4995
mean = 1.24813
stdev = 0.431979
median = 1
mode = 1
min = 1
max = 2
  0th percentile = 1
 25th percentile = 1
 50th percentile = 1
 75th percentile = 1
100th percentile = 2
==11464== 
==11464== HEAP SUMMARY:
==11464==     in use at exit: 0 bytes in 0 blocks
==11464==   total heap usage: 18,474 allocs, 18,474 frees, 5,396,190 bytes allocated
==11464== 
==11464== All heap blocks were freed -- no leaks are possible
==11464== 
==11464== For counts of detected and suppressed errors, rerun with: -v
==11464== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Understanding Valgrind errors

Let’s say we made a mistake in the sum() function in stats.cpp. We accidentally went off the end of a vector (notice <= instead of <).

double sum(vector<double> v) {
  assert(!v.empty());
  double sum = 0;
  for (size_t i = 0; i <= v.size(); ++i) {
    sum += v[i];
  }
  return sum;
}

Build and run without Valgrind. It passes on CAEN Linux, but because going off the end of a vector (or array) is undefined, it could get the wrong answer on the autograder!

$ make stats_tests.exe
g++ -Wall -Werror -pedantic -g --std=c++11 stats_tests.cpp stats.cpp p1_library.cpp -o stats_tests.exe
main awdeorio@caen-vnc-vm04 p1-stats-copy
$ ./stats_tests.exe 
test_sum_small_data_set
PASS!

Run with Valgrind.

$ valgrind ./stats_tests.exe 
==12081== Memcheck, a memory error detector
==12081== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==12081== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info
==12081== Command: ./stats_tests.exe
==12081== 
test_sum_small_data_set
==12081== Invalid read of size 8
==12081==    at 0x402A61: sum(std::vector<double, std::allocator<double> >) (stats.cpp:55)
==12081==    by 0x4016FF: test_sum_small_data_set() (stats_tests.cpp:44)
==12081==    by 0x401645: main (stats_tests.cpp:29)
==12081==  Address 0x5a19158 is 0 bytes after a block of size 24 alloc'd
==12081==    at 0x4C2A203: operator new(unsigned long) (vg_replace_malloc.c:334)
==12081==    by 0x402369: __gnu_cxx::new_allocator<double>::allocate(unsigned long, void const*) (new_allocator.h:104)
==12081==    by 0x4021AC: std::_Vector_base<double, std::allocator<double> >::_M_allocate(unsigned long) (stl_vector.h:168)
==12081==    by 0x401FB4: std::_Vector_base<double, std::allocator<double> >::_M_create_storage(unsigned long) (stl_vector.h:181)
==12081==    by 0x401B80: std::_Vector_base<double, std::allocator<double> >::_Vector_base(unsigned long, std::allocator<double> const&) (stl_vector.h:136)
==12081==    by 0x4018EA: std::vector<double, std::allocator<double> >::vector(std::vector<double, std::allocator<double> > const&) (stl_vector.h:312)
==12081==    by 0x4016F3: test_sum_small_data_set() (stats_tests.cpp:44)
==12081==    by 0x401645: main (stats_tests.cpp:29)
==12081== 
PASS!
==12081== 
==12081== HEAP SUMMARY:
==12081==     in use at exit: 0 bytes in 0 blocks
==12081==   total heap usage: 4 allocs, 4 frees, 80 bytes allocated
==12081== 
==12081== All heap blocks were freed -- no leaks are possible
==12081== 
==12081== For counts of detected and suppressed errors, rerun with: -v
==12081== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

To debug this issue, look in the Valgrind output for line numbers from your own source code. In this example, stats.cpp:55 is the line which goes off the end of the vector.

for (size_t i = 0; i <= v.size(); ++i) {  // line 54
  sum += v[i];                            // line 55
}

Let’s look at another example, where we forgot to initialize the variable i:

for (size_t i; i < v.size(); ++i) {       // line 54
  sum += v[i];                            // line 55
}

When running stats_tests.exe with Valgrind, the test case fails, but before it does so, we get a Valgrind error:

==208== Conditional jump or move depends on uninitialised value(s)
==208==    at 0x402F94: sum(std::vector<double, std::allocator<double> >) (stats.cpp:54)
==208==    by 0x401924: test_sum_small_data_set() (stats_tests.cpp:44)
==208==    by 0x40185E: main (stats_tests.cpp:29)

This tells us that line 54 in stats.cpp depends on the value of an unitialized variable.

Checking for memory leaks

Skip this section for project 1. We’ll use dynamic memory (new and delete) on projects 2 through 5.

Use Valgrind to check for memory leaks. If you use the new operator in your code, you should run Valgrind with the --leak-check=full argument.

$ valgrind --leak-check=full ./main.exe

Let’s comment out the deletion of the Matrix object in test_fill_basic in Matrix_tests.cpp from project 2.

TEST(test_fill_basic) {
  Matrix *mat = new Matrix; // create a Matrix in dynamic memory

  const int width = 3;
  const int height = 5;
  const int value = 42;
  Matrix_init(mat, 3, 5);
  Matrix_fill(mat, value);

  for(int r = 0; r < height; ++r){
    for(int c = 0; c < width; ++c){
      ASSERT_EQUAL(*Matrix_at(mat, r, c), value);
    }
  }

  //delete mat; // delete the Matrix
}

Recompile the code.

$ make Matrix_tests.exe
g++ --std=c++11 -Wall -Werror -pedantic -g Matrix_tests.cpp Matrix.cpp Matrix_test_helpers.cpp unit_test_framework.cpp -o Matrix_tests.exe

Run Valgrind with the --leak-check=full argument.

$ valgrind --leak-check=full ./Matrix_tests.exe 
==100== Memcheck, a memory error detector
==100== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==100== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==100== Command: ./Matrix_tests.exe
==100== 
Running test: test_fill_basic
PASS

*** Results ***
** Test case 'test_fill_basic': PASS
*** Summary ***
Out of 1 tests run:
0 failure(s), 0 error(s)
==100== 
==100== HEAP SUMMARY:
==100==     in use at exit: 1,072,712 bytes in 2 blocks
==100==   total heap usage: 6 allocs, 4 frees, 1,073,992 bytes allocated
==100== 
==100== 1,000,008 bytes in 1 blocks are definitely lost in loss record 2 of 2
==100==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==100==    by 0x401F47: test_fill_basic() (Matrix_tests.cpp:15)
==100==    by 0x403B46: TestCase::run(bool) (unit_test_framework.cpp:28)
==100==    by 0x4041FB: TestSuite::run_tests(int, char**) (unit_test_framework.cpp:102)
==100==    by 0x401FFF: main (Matrix_tests.cpp:42)
==100== 
==100== LEAK SUMMARY:
==100==    definitely lost: 1,000,008 bytes in 1 blocks
==100==    indirectly lost: 0 bytes in 0 blocks
==100==      possibly lost: 0 bytes in 0 blocks
==100==    still reachable: 72,704 bytes in 1 blocks
==100==         suppressed: 0 bytes in 0 blocks
==100== Reachable blocks (those to which a pointer was found) are not shown.
==100== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==100== 
==100== For counts of detected and suppressed errors, rerun with: -v
==100== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Valgrid reports a leak, and the output includes the stack trace of where the leaked object is created. The first line of the trace indicates that the object was created by the new operator:

==100==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)

The second line tells us that it occurred in line 15 of Matrix_tests.cpp, in test_fill_basic:

==100==    by 0x401F47: test_fill_basic() (Matrix_tests.cpp:15)

Line 15 of Matrix_test.cpp is the following:

  Matrix *mat = new Matrix; // create a Matrix in dynamic memory

The remaining lines in the stack trace are internal to the unit test framework.

Let’s run Valgrind again, this time with the deletion line included.

$ valgrind --leak-check=full ./Matrix_tests.exe 
==123== Memcheck, a memory error detector
==123== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==123== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==123== Command: ./Matrix_tests.exe
==123== 
Running test: test_fill_basic
PASS

*** Results ***
** Test case 'test_fill_basic': PASS
*** Summary ***
Out of 1 tests run:
0 failure(s), 0 error(s)
==123== 
==123== HEAP SUMMARY:
==123==     in use at exit: 72,704 bytes in 1 blocks
==123==   total heap usage: 6 allocs, 5 frees, 1,073,992 bytes allocated
==123== 
==123== LEAK SUMMARY:
==123==    definitely lost: 0 bytes in 0 blocks
==123==    indirectly lost: 0 bytes in 0 blocks
==123==      possibly lost: 0 bytes in 0 blocks
==123==    still reachable: 72,704 bytes in 1 blocks
==123==         suppressed: 0 bytes in 0 blocks
==123== Reachable blocks (those to which a pointer was found) are not shown.
==123== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==123== 
==123== For counts of detected and suppressed errors, rerun with: -v
==123== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Valgrind no longer reports an error.

Next steps

Return to the main set up tutorial.