c++ TDD quest

[bbguimaraes]

Hi. After a short programming-absence on the last semester (academic purposes), I started working on a company which applies TDD on its development process (ironically, the same university). They use the python language, so I decided to look for an alternative for c++, to use on my own projects. So I thought: "Hey, I could post my research on the forum, so people can use and learn from it too.". Then I thought there probably was already some posts about it. But I was wrong. If you're curious, search for "unit test": you'll find two unanswered posts. But I decided to do it anyway.

I'm not going to explain what TDD is and how it works. First, there's google. And I think the basic idea can be learned from the examples. I'm going to do my research and post it here, from time to time, so you can check it out, maybe learn something new and maybe help me with something or add new material. I'll leave this introduction on a post of its own.

Index:

• Part I: Finding alternatives
• Part II: Hello world

[bbguimaraes]

Here are the options I found after a short google search:

• CppUnit: c++ port of JUnit, one of the first testing frameworks. This will probably be the last one tested, because I have preconcieved (bad) ideas about anything slightly related to Java.
• boost::test: part of the famous boost library. If it's as good as the other boost libraries, I already have a favorite.
• QtTest: part of the famouse Qt framework. If it's as good as the other Qt libraries, I already have a second favorite.
• googletest: this one surprised me. Google has its own c++ test framework!

This is not a definitive list, I'll add items if I find others. I'm also looking for suggestions.

[dandymcgee]

I'm not going to explain what TDD is and how it works.

That's fine, but you could have at least said what the acronym stands for..
http://en.wikipedia.org/wiki/Test-driven_development

[bbguimaraes]

Following the short introduction, here is the basic setup. I've covered installation on my Ubuntu test machine. I'm using apt to install the packages, so much of the complexity of setting up paths is hidden. For all the libraries though, installation basically meant putting headers and libraries on /usr/include and /usr/lib.

If you're wondering where is the CppUnit section: I'm sorry. I just couldn't find a simple program to run the tests. They all require a huge amount of code to setup the simplest of tests. I have a working example, that I'll post on the next sections.

Update Aug 09, 2012: I decided to include cxxtest too. I really liked the idea of parsing the headers to create the test runners. Let's see how it scales to more complex tests.

I've also decided to give CppUnit another chance. No, I'm kidding. I just looked at the code for the "minimal problem" and thought there were only two possibilities: (a) I clearly missed something, so someone can correct me (b) the code is so ridiculously long they deserve to be mocked.

This sections show:

• installation of the library and compilation of a simple program
• a simple program and its output

Index:

• boost::test
• QtTest
• googletest
• cxxtest
• CppUnit

[bbguimaraes]

boost::test

Install libraries

Code: Select all

sudo apt-get install libboost-test-dev

Compilation

Code: Select all

g++ main.cpp -o boost-test

Minimal program

Code: Select all

#define BOOST_TEST_MAIN
#include <boost/test/included/unit_test.hpp>
BOOST_AUTO_TEST_CASE(test) {}

Output

Code: Select all

Running 1 test case...
*** No errors detected

[bbguimaraes]

QtTest

Install libraries

Code: Select all

sudo apt-get install libqt4-test

Compilation

Code: Select all

qmake -project "CONFIG += qtestlib"
qmake
make

Minimal program

Code: Select all

#include <QtTest/QtTest>
class Test : public QObject {};
QTEST_MAIN(Test)
#include <main.moc>

Output

Code: Select all

********* Start testing of QObject *********
Config: Using QTest library 4.8.1, Qt 4.8.1
PASS   : QObject::initTestCase()
PASS   : QObject::cleanupTestCase()
Totals: 2 passed, 0 failed, 0 skipped
********* Finished testing of QObject *********

[bbguimaraes]

googletest

Installation

Code: Select all

sudo apt-get install libgtest-dev
cd /usr/src/gtest
sudo cmake CMakeLists.txt
sudo make
cd /usr/lib
sudo ln -s /usr/src/gtest/libgtest.a libgtest.a
sudo ln -s /usr/src/gtest/libgtest_main.a libgtest_main.a

Compilation

Code: Select all

g++ main.cpp -lgtest_main -lgtest -pthread -o googletest

Minimal program

Code: Select all

#include <gtest/gtest.h>
TEST(TestCase, Test) {}

Output

Code: Select all

Running main() from gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from TestCase
[ RUN      ] TestCase.Test
[       OK ] TestCase.Test (0 ms)
[----------] 1 test from TestCase (0 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (0 ms total)
[  PASSED  ] 1 test.

[bbguimaraes]

cxxtest

Installation

Code: Select all

add-apt-repository ppa:dhart/ppa
sudo apt-get install cxxtest

Compilation

Code: Select all

cxxtestgen --error-printer -o main.cpp Test.h
g++ main.cpp -o cxxtest

Minimal program

Code: Select all

include <cxxtest/TestSuite.h>
using CxxTest::TestSuite;

class Test : public CxxTest::TestSuite {
   public:
       void testCxxtest() {}
};

Output

Code: Select all

Running 1 test.OK!

[bbguimaraes]

CppUnit

Installation

Code: Select all

sudo apt-get install libcppunit-dev

Compilation

Code: Select all

g++ main.cpp -lcppunit -o cppunit

Minimal program

Code: Select all

#include <cppunit/extensions/HelperMacros.h>
#include <cppunit/TestFixture.h>
using CPPUNIT_NS::TestFixture;

#include <cppunit/ui/text/TestRunner.h>
using CPPUNIT_NS::TextTestRunner;

#include <cppunit/extensions/TestFactoryRegistry.h>
using CPPUNIT_NS::TestFactoryRegistry;

class Test : public TestFixture {
    virtual void runTest() {}
};

int main() {
    TextTestRunner runner;
    TestFactoryRegistry & registry = TestFactoryRegistry::getRegistry();
    runner.addTest(registry.makeTest());
    runner.run();
    return 0;
}

Output (no whitespaces added)

Code: Select all

OK (0 tests)

[Falco Girgis]

So what are you liking about boost::test better than QT's testing framework? I have been a long-time QT loyalist, and you have gotten me curious. I have never done much looking into a testing framework.

[bbguimaraes]

Actually, I'm still in a very early stage of comparison. What I meant is that, if boost::test and QtTest were in an equal level, I'd choose boost, for the single fact that it's more modularized than Qt (smaller footprint if you want to get fancy). That's inherent to the purpose of both of them. If the project already has Qt dependencies, I think I'd choose Qt. It probably has easier ways do test GUIs (at least its own). So far (I haven't tested cxxtest properly, which seems pretty promising), they seem like the two best alternatives.

Yes, I'm biased. Sue me.

[bbguimaraes]

Now we're going to de some simple tests, just to learn how each library structures each test and combines all of them on execution, including the output of running for many tests. The function used on the tests is the following:

Code: Select all

int sum(int n1, int n2) {
    return 3;
}

That is not much useful for everyday use, but it will serve in our tests to show success and failure messages.

Index:

• boost::test
• QtTest
• googletest
• cxxtest

[bbguimaraes]

boost::test

Code

Code: Select all

#define BOOST_TEST_MODULE Test
#include <boost/test/unit_test.hpp>

int sum(int n1, int n2) {
    return 3;
}

BOOST_AUTO_TEST_CASE(test1plus2) {
    BOOST_CHECK_EQUAL(sum(1, 2), 3);
}

BOOST_AUTO_TEST_CASE(test2plus2) {
    BOOST_CHECK_EQUAL(sum(2, 2), 4);
}

Output

Code: Select all

Running 2 test cases...
main.cpp(13): error in "test2plus2": check sum(2, 2) == 4 failed [3 != 4]

*** 1 failure detected in test suite "Test"

Note
Since we're moving out of compiling the test runner with our executable to only linking to the pre-compiled library, we need to alter our compilation command to:

Code: Select all

g++ main.cpp -lboost_test_exec_monitor -o boost-test

[bbguimaraes]

QtTest

Code

Code: Select all

int sum(int n1, int n2) {
    return 3;
}

class Test : public QObject {
    Q_OBJECT

    private slots:
        void test1plus2() {
            QCOMPARE(sum(1, 2), 3);
        }

        void test2plus2() {
            QCOMPARE(sum(2, 2), 4);
        }
};

QTEST_MAIN(Test)
#include <main.moc>

Output

Code: Select all

********* Start testing of Test *********
Config: Using QTest library 4.8.1, Qt 4.8.1
PASS   : Test::initTestCase()
PASS   : Test::test1plus2()
FAIL!  : Test::test2plus2() Compared values are not the same
   Actual (sum(2, 2)): 3
   Expected (4): 4
   Loc: [main.cpp(16)]
PASS   : Test::cleanupTestCase()
Totals: 3 passed, 1 failed, 0 skipped
********* Finished testing of Test *********

Note
I'm just going to get out of the way here to say that it took me two minutes to learn how to write this program. Once again, Qt proved to have an excelent documentation. You can also see how complete the output is, showing the expressions and values of both parameters of QCOMPARE, and the line on the file where it's being called.

[bbguimaraes]

googletest

Code

Code: Select all

#include <gtest/gtest.h>

int sum(int n1, int n2) {
    return 3;
}

TEST(test1plus2, Test) {
    ASSERT_EQ(3, sum(1, 2));
}

TEST(test2plus2, Test) {
    ASSERT_EQ(4, sum(1, 2));
}

Output

Code: Select all

Running main() from gtest_main.cc
[==========] Running 2 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from test1plus2
[ RUN      ] test1plus2.Test
[       OK ] test1plus2.Test (0 ms)
[----------] 1 test from test1plus2 (0 ms total)

[----------] 1 test from test2plus2
[ RUN      ] test2plus2.Test
main.cpp:12: Failure
Value of: sum(1, 2)
  Actual: 3
Expected: 4
[  FAILED  ] test2plus2.Test (0 ms)
[----------] 1 test from test2plus2 (0 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 2 test cases ran. (1 ms total)
[  PASSED  ] 1 test.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] test2plus2.Test

 1 FAILED TEST

(Yet another) Note
I'm attaching a screenshot, because you can't see here that the output is colored.