Developments are ongoing in the quest for better testing methods for proxy models. I wrote before about testing a proxy model by visual inspection in the proxymodeltestapp. For identifying the sources of issues this has generally worked really well, but the involvement of human interaction means that it doesn’t scale and is error prone. Unit testing fares better in that regard because they can be run automatically without a human operator or observer, are faster than unit testing and are a persistent test for regressions so they allow for broad refactoring if needed for performance reasons.
This is not a very screenshot-able subject, so I have added some cat pictures at the suggestion of and for the amusement of tokoe.
Getting the blues
The biggest problem with Qt Model View is the amount of API that must be implemented to write a proxy model. Aside from implementing the QAbstractItemModel interface and the small additional interface in QAbstracyProxyModel, an observer interface for the source model must also be implemented. Source models emit signals such as rowsAboutToBeInserted and rowsInserted, and similar signals for removing, moving, restructuring the entire layout and a complete reset of the source model. Depending on the proxy model implementation, it may also be relevant that items with children are inserted, or that an entire subtree is removed or moved. Proxy models might also have boundary errors if items are removed or inserted at the top or bottom of an existing list, or if a row is moved from a visible row to an invisible row or vice-versa.
There are a large but finite number of updates that a source model can signal to a proxy model which must be reacted to in some way (even if the reaction of the proxy model is “do nothing”, if for example the inserted row is filtered out). It should be possible to write unit tests for proxy models such that the signals emitted by the source model are always the same, predictable and comprehensive.
My first attempt at doing so resulted in a unit test suite which commanded a source model through a sequence of steps covering insertion, moving, removal and layout changes. This did work in that it was possible to find bugs with it, but it did not scale well to different configurations of the proxy model under test and was inflexible to adding new tests. I started writing a different test file per configuration (kselectionproxymodeltest-onlyselectedchildren and kselectionproxymodeltest-selectedbranches for example, and they didn’t cover different selections), and adding tests anywhere but the end of the sequence meant updating all testcases in all test classes. Additionally, the sequence of tests did not make bug finding any easier. Executing a single test for one data point in the test was possible in most, but not all cases, and even then, all previous tests in the sequence would be executed before the particular data test of interest, causing a lot of noise.
My latest venture is to create tests which always initialize the source model to a known state and execute a single data driven test. Each data test is then standalone and can be executed independently or as part of a specified list. The minimal nature of the tests means that the developer can write a failing test and then easily execute only the failing code without first having to skip through a sequence of steps of prior tests.
The suite for executing, running and verifying the tests has also had many improvements. There are now predefined test data for indicating things like “This signal should be forwarded as-is through the proxy model”, which would be the case for a QSortFilterProxyModel without sorting or filtering enabled. This allows creation of very concise tests.
The configuration of a proxy model adds an extra dimension to how it is expected to behave in response to a stimulus from the source model. A QSortFilterProxyModel can have various filtering and sorting rules, although many could be described as generalizations of other tests. The behaviour of a KSelectionProxyModel is determined by the selection in a QItemSelectionModel and by the behaviour configured on the proxy model.
The way these large variety of configuration possibilities is handled is to define test data in template specializations of the test methods. For example in testing the KSelectionProxyModel there could be template specialisations like:
void TestData<SelectionStrategy<9>, KSelectionProxyModel::ExactSelection>::testRemoveFromTopLevelData()
// test data for when 9 is selected and the configuration of the proxy is ExactSelection
void TestData<SelectionStrategy<4, 7>, KSelectionProxyModel::SubTrees>::testRemoveFromTopLevelData()
// test data for when 4 and 7 are selected and the configuration of the proxy is SubTrees
In perhaps more familiar terms, QSortFilterProxyModels for different filter regexps could be specified like this:
// Need external linkage
extern const QRegExp re1_2_4(“1|2|4”);
extern const QRegExp re7_12_24(“7|12|24”);
// test data for when the filter regexp is “1|2|4”
// test data for when the filter regexp is “7|12|24”
In each case, the test data is a description of the signals expected to be emitted by the proxy and the expected update patterns to QPersistentModelIndexes in the proxy.
A consequence of using template specialization is that it allows partial implementation of test code for new proxy model configurations. For example, if a new bug is discovered when inserting rows for a particular implementation, but removing and moving rows are not affected, a test specialization for the relevant filter could be implemented for the insert test method only, leaving the others as the default implementation.
Proxy models without crashes
One of the more difficult crashes to track down in models or proxies looks something like this:
ASSERT failure in QPersistentModelIndex::~QPersistentModelIndex: "persistent model indexes corrupted", file kernel/qabstractitemmodel.cpp, line 544
which can happen on shutdown, or with an innocent mouse movement some time after changing the model in some way by inserting or moving rows for example.
QModelIndexes updated over event loop returns
This happens when internal QPersistentModelIndexes are not updated consistently with what is reported to the outside via signals, and an observer creates a new QPersistentModelIndex as a result, or the proxy updates an existing persistent index to a conflicting position. In the test suite, this is covered by having different instances of the tests creating persistent indexes at different times. Each test is run twice- once for each persistent index taking strategy, that is creating QPersistentModelIndexes before running the test, and creating them during the test when the model emits rowsAboutToBeInserted for example. Additionally, the test suite monitors all QPersistentModelIndexes before and after running each test to ensure that updates are consistent with emitted signals.
Look up the index and double check
Another aspect of correct generic proxy models is that they should work well with other proxy models. To ensure this the test suite is run in a further two configurations for when the source model of the proxy under test is a “base” QAbstractItemModel or an intermediate QSortFilterProxyModel. That means that each data test in the suite is executed 4 times! The tests of course verify that the model() of all indexes always indicates the correct model.
Vigourous testing is concise
Currently the KSelectionProxyModel test executes tests for 1122 different data points, but that will get much larger soon when more configurations are tested.
The QtTest module provides a test suite for running a test with a particular test data set already. It is possible to run the test executable on the command line with arguments for the exact test data point required. The proxy model test suite extends that system by making it possible to run all available tests by running the ./kselectionproxymodeltest executable. When finding bugs it is useful to write new tests and execute only the failing code to discover the fix. The proxy model test suite allows executing a particular test for all configurations of the proxy, or all tests for just one configuration of the proxy, or some combination of tests for a selection of configurations.
In short, the new test suite is very versatile.