Effectively testing Qt 5 using Squish Coco

Sébastien Fricker and Amanda Burma

09/18/2012

PDF version:

1 Introduction

With more than tree million lines of code, Qt 5 is one of the largest open source projects. Creating a challenge for every contributing developer. In fact how can we be sure:

code modifications are sufficiently tested?
code modifications do not impact other features?
a reviewer can approve a patch by simply reading a basic diff?

The approach to expect unit testing by a developer and letting the maintainer or approver take responsibility when integrating an extension, can be vastly improved by Squish Coco as it:

helps the Qt contributor find areas containing untested portions of code.
helps the approver verify the modification is thoroughly tested and no regressions exist.

2 System Prerequisites

2.1 Installing Squish Coco

Squish Coco is available in two editions:

A Non-Commercial Edition which includes a compiler replacement, performing the code coverage analysis and a set of command line tools creating the ability to generate reports and manipulate code coverage data.
A Professional Edition which includes also a graphical user interface, professional support and a license expanding its usage beyond non-commercial applications.

To install first sign up for an evaluation or permanent license:

for Squish Coco: http://www.froglogic.com/squish/coco/
for Squish Coco Non-Commercial: http://www.froglogic.com/squish/coco/non-commercial.php

This article only uses Squish Coco’s command line tools, allowing either edition to be used.

After installing Squish Coco modify the path variable to access csg++, the Squish Coco compiler replacement, from the command line.

2.2 System Prerequisites for Building Qt 5

Qt 5 depends of an important set of libraries. A full description of the requirements can be found on Qt homepage: http://qt-project.org/wiki/Building_Qt_5_from_Git

2.3 Downloading Qt 5 Source

The system variable $WORK refers to the working directory. $QT5 is the Qt 5 source check out location. And $PROCESSORS indicates the number of available CPU cores.

export WORK=$HOME/coco # or use an other location
export QT5=$WORK/qt5
export QT5BASE=$QT5/qtbase
export PROCESSORS=$(cat /proc/cpuinfo | grep processor | wc -l)

mkdir -p $WORK
cd $WORK
git clone git://gitorious.org/qt/qt5.git
cd $QT5
./init-repository -f
git pull
git submodule update

The section (above) downloads the complete Qt 5 source.

3 Building Qt 5 and Executing Unit Test

Qt 5 provides a command line switch called -testcocoon, which permits the instrumentation of Squish CocoṀore precisely:

the Makefile uses csg++ instead of g++ for code generation.
for each unit test execution, an execution report is generated, displaying the code coverage for one test or an entire test suite.

Execute the following to compile:

./configure -no-qpa-platform-guard -platform  linux-g++-64 \
            -developer-build -no-gtkstyle -no-pch \
            -opensource -testcocoon -confirm-license -prefix $QT5BASE
cd $QT5BASE
make -j$PROCESSORS

At this stage, QtBase is generated along with $QT5BASE/lib for each generated shared library a file with the extension .csmes is produced. This file contains the code coverage instrumentation, and upon completion also contains the code coverage report for each unit test.

Perform the following to execute the test suite: ¹

make check -k

Now, as each test executes, the instrumentation files in $QT5BASE/lib are not updated, however import the unit test results.

4 Code Coverage Report of Qt 5 Libraries

The following suppresses all coverage reports from the uic, moc and rcc tools:

find $QT5 -name "rcc.csexe" -exec rm -v {} \;
find $QT5 -name "moc.csexe" -exec rm -v {} \;
find $QT5 -name "uic.csexe" -exec rm -v {} \;

The following imports the execution report (.csexe file) for each unit test into the instrumentation database (.csmes file):

cd $QT5BASE
find . -name "tst_*.csexe" | while read LINE
do
    CSEXE="$LINE"
    CSMES=$(dirname "$CSEXE")/$(basename "$CSEXE" .csexe).csmes
    if [ -e "$CSMES" ]
    then
        echo -n "Generating Unit Test Database $CSMES ... "
        cmcsexeimport -m "$CSMES" -e "$CSEXE" -t "unittest" -p merge
        echo "done"
    fi
    rm "$CSEXE"
done

cmcsexeimport performs the import operation. After which the imported execution report is no longer needed, and can be safely removed.

At this stage an instrumentation database exists for each unit test, which contains the code coverage information for each execution. The following retrieves and places this information into the Qt 5 libraries’ instrumentation databases:

cd $QT5BASE
for QTLIB in lib/lib*.csmes
do
    QTLIBTMP="$1".tmp
    UNITTESTS=$(find . -name "tst_*.csmes")
    echo -n "Importing Unit Tests Result in $QTLIB ... "
    cmmerge -o "$QTLIBTMP"  -i "$QTLIB" $UNITTESTS
    mv -f "$QTLIBTMP" "$QTLIB"
    echo "done"
done

cmmerge imports the unit test results into Qt 5 library’s database. It also ignores the code coverage in the test code. After this operation, the instrumentation database for each unit test is no longer needed.

The following generates an HTML report from the code coverage database:

cd $QT5BASE
for QTLIB in lib/lib*.csmes
do
    HTML=$(dirname "$QTLIB")/$(basename "$QTLIB" .csmes).html
    TTTLE=$(basename "$QTLIB" .csmes)
    cmreport -m "$QTLIB" -s '.*' --html="$HTML" --toc --title="$TITLE" \
             --source=all --method=all --global=all --dead-code \
             -bargraph --source-sort=coverage --method-sort=coverage
done

In $QT5BASE/lib an HTML report is generated for each Qt library, making it possible to analyze which source code line was not covered by the test suite.

5 Using Squish Coco Qt 5 Development

Having a code coverage report of Qt 5 in its entirety does not have any real value for a Qt developer: The developer is only focused on developing a feature or fixing a bug and does not need to be aware of the quality of the entire Qt project. Instead, the developer needs to know how the modification can negatively impact the project and how well the changes are tested.

5.1 Hacking in Qt 5

The following example demonstrates how Squish Coco can be used to better understand the impact of a change and the required testing: To support the € symbol when converting a Unicode string to a Latin-1 codec, the € (unicode 0x20AC) will be converted to the character ’E’.

Modify QLatin1Codec::convertFromUnicode as follows (modifications are underlined):

   1 QByteArray QLatin1Codec::convertFromUnicode(const QChar *ch,
   2                                             int len, ConverterState *state) const
   3 {
   4     const char replacement = (state && state->flags & ConvertInvalidToNull) ? 0 : '?';
   5     QByteArray r(len, Qt::Uninitialized);
   6     char *d = r.data();
   7     int invalid = 0;
   8     for (int i = 0; i < len; ++i) {
   9         if (ch[i] == 0x20AC) {      // MODIFICATION: Euro symbol handling
  10             d[i] = ’E’;             // MODIFICATION: Euro symbol handling
  11         } else if (ch[i] > 0xff) {
  12             d[i] = replacement;
  13             ++invalid;
  14         } else {
  15             d[i] = (char)ch[i].cell();
  16         }
  17     }
  18     if (state) {
  19         state->invalidChars += invalid;
  20     }
  21     return r;
  22 }

Before generating the Qt library, make a copy of the current instrumentation database as follows:

cd $QT5BASE
export QT5REF=$QT5/../qt5_lib_ref/
mkdir -p $QT5REF
cp $QT5BASE/lib/*.csmes $QT5REF/

Then run make to compile. $QT5BASE/lib/libQtCore.so.5.0.0.csmes is updated at the same time as $QT5BASE/lib/libQtCore.so.5.0.0.so.

No tests are executed at this stage; however Squish Coco can analyze which test is impacted by the modification:

cd $QT5BASE
cmreport -m "$QT5REF/libQtCore.so.5.0.0.csmes" -s '.*' --html="euro_sym.html" \
         --toc --title="Euro symbol support" \
         --source=all --method=all --global=all \
         -bargraph --source-sort=coverage --method-sort=coverage \
         -mr "$QT5BASE/lib/libQtCore.so.5.0.0.csmes" --execution=all

euro_sym.html report is the coverage report for the modified functions, and in our case only the QLatin1Codec::convertFromUnicode function. The list of executions is limited to two entries:

tst_QString with 78% coverage
tst_QTextStream with 64% coverage

Tests Impacted By Code Modification

Tests Impacted By Code Modification

These are the only tests impacted by the modification, meaning all other tests are irrelevant. The report (above) also illustrates this function is not fully covered by the test suite, leaving the following as our potential regressions:

A character not able to be concerted to Latin-1 remains untested (Unicode value greater than 255)
A null replacement character remains untested.

Source Code View

Source Code View

5.2 The Unit Test

The two missing unit tests are illustrated below: one for testing the € symbol conversion, and one for testing other Unicode values.

Modify $QT5BASE/tests/auto/corelib/tools/qstring/tst_qstring.cpp and define two new slots:

1 void euroSym();
2 void invalidUnicodeConversionToLatin1();

The slot code is:

   1 void tst_QString::invalidUnicodeConversionToLatin1()
   2 {
   3     QTextCodec *codec = QTextCodec::codecForName("Latin1");
   4     QVERIFY(codec != NULL);
   5
   6     QString str;
   7     str += QChar(0x21AC);
   8     QByteArray latin1_converted = codec->fromUnicode(str);
   9     QCOMPARE(latin1_converted , QByteArray("?"));
  10 }
  11
  12 void tst_QString::euroSym()
  13 {
  14     QTextCodec *codec = QTextCodec::codecForName("Latin1");
  15     QVERIFY(codec != NULL);
  16
  17     QString str;
  18     str += QChar(0x20AC);
  19     QByteArray latin1_converted = codec->fromUnicode(str);
  20
  21     QCOMPARE(latin1_converted , QByteArray("E"));
  22 }

Run the test suite again:

cd $QT5BASE
make -j$PROCESSORS
make -k check
find $QT5 -name "rcc.csexe" -exec rm -v {} \;
find $QT5 -name "moc.csexe" -exec rm -v {} \;
find $QT5 -name "uic.csexe" -exec rm -v {} \;
find . -name "tst_*.csexe" | while read LINE
do
    CSEXE="$LINE"
    CSMES=$(dirname "$CSEXE")/$(basename "$CSEXE" .csexe).csmes
    if [ -e "$CSMES" ]
    then
        echo -n "Generating Unit Test Database $CSMES ... "
        cmcsexeimport -m "$CSMES" -e "$CSEXE" -t "unittest" -p merge
        echo "done"
    fi
    rm "$CSEXE"
done
for QTLIB in lib/lib*.csmes
do
    QTLIBTMP="$1".tmp
    UNITTESTS=$(find . -name "tst_*.csmes")
    echo -n "Importing Unit Tests Result in $QTLIB ... "
    cmmerge -o "$QTLIBTMP"  -i "$QTLIB" $UNITTESTS
    mv -f "$QTLIBTMP" "$QTLIB"
    echo "done"
done

Following the test suite execution, the code coverage report contains the modification:

cd $QT5BASE
cmreport -m "$QT5BASE/lib/libQtCore.so.5.0.0.csmes" -s '.*' --html="$WORK/euro_sym.html" \
         --toc --title="Euro symbol support" \
         --source=all --method=all --global=all \
         -bargraph --source-sort=coverage --method-sort=coverage \
         -mr "$QT5REF/libQtCore.so.5.0.0.csmes" --execution=all

$WORK/euro_sym.html is generated, displaying the code coverage result for the modification.

The modification’s coverage is 94%:

Code Coverage Statistics Of The Code Modification

Code Coverage Statistics Of The Code Modification

The only line not covered corresponds to the missing test for selecting the replacement character:

Source Code View

Source Code View

6 Conclusion

Using Squish Coco it is possible to reduce the code coverage analysis to only what the contributor and the approver needs:

Is the patch correctly tested?
What are the possible regressions?
What is the impact, in terms of the number of tests, for the modification?

Of course, such analysis should be made by the integration server and performed on all Qt modules, integrated with the code review tool; however, this is topic for another day…

In order to help the Qt community to improve the quality, froglogic GmbH publish now code coverage statistics of QtBase computed on the daily snapshot. Once the official Qt5 will be available, we will also provide the code coverage analysis on the modifications between the actual snapshot and the last stable release.
Here the link: http://download.froglogic.com/public/qt5-squishcoco-report

7 Script Files

The following script was tested on Debian and performs the steps described in this article automatically.

To execute:

Copy qt5-coco.sh and qt5-euro.patch into a folder.
Execute ./qt5-coco.sh
The generated files appear within $HOME/coco

Be patient: compiling and executing a Qt test suite as well as instrumenting it can be time consuming.

7.1 Build Script

To download: qt5-coco.sh

7.2 Patch

To download: qt5-euro.patch