Archive for September, 2010

Grantlee pairs up with the “other” template system

September 29, 2010

For the first time in a while a post about Grantlee which is not a release announcement. 🙂

QObject introspection

Since I started working on Grantlee it has been possible to put QObject derived types into the template system as context, as well as QLists or QHash containers of QObject derived types in a QVariant. Properties on the instances are then available through the Q_PROPERTY system and can be used when rendering a template.

  QVariantList peopleList;
  // Inconvenient!
  peopleList.append(QVariant::fromValue(static_cast<QObject*>(new Person("Alice"))));
  peopleList.append(QVariant::fromValue(static_cast<QObject*>(new Person("Bob"))));

  context.insert("people", peopleList);
  template->render(&context);

  ...

  <ul>
  {% for person in peopleList %}
    <li>{{ person.name }}, {{ person.age }}</li>
  {% endfor %}
  </ul>

The restriction that the type be QObject derived was because the only way to introspect them was through the Qt Property system. Only QVariantList and QVariantHash were supported because that provides sequential and associative containers, and both already have built in support in QVariant.

It is not an ideal situation however, because existing applications already have a data API which might not use QObject derived types, so wrappers would have to be written. That is a common situation in KDEPIM where we have KMime::Message, KABC::Addressee and KCalCore::Incidence and related classes which do not inherit QObject. Additionally, QList is not often the best container to use as a sequential container. Commonly people use QVector or std::vector instead.

Generic introspection

Near the end of July Michael Jansen did some work on generic type support in Grantlee. It took me until the middle of September to finally get around to reviewing the patch, and it showed me something I didn’t know was possible. There are several aspects to the work on the mjansen-experimental branch, but the core concept is being able to use C++ templates (very different to Grantlee templates) and function pointers to extract custom types from QVariants from within the Grantlee processing code. This post is about how it works, not how to use it. For the documentation about how it is used, see here.

Because it uses C++ templates, Grantlee doesn’t need specific code to support the custom types, but can expose a static interface which applications can implement for their own custom types. The real magic happens by using function pointers to create a registry of available types. It works something similar to this (some details and magic omitted for clarity):

  // static interface
  template<typename T>
  struct TypeAccessor
  {
    static QVariant lookUp(const T &object, const QString property);
  };

  // Converts the object parameter from QVariant to T for use with the TypeAccessor
  template<typename T>
  struct LookupTrait
  {
    static QVariant doLookUp( const QVariant &object, const QString &property )
    {
      typedef typename Grantlee::TypeAccessor<T> Accessor;
      return Accessor::lookUp( object.value<T>() ), property );
    }
  };

  struct MetaType
  {
    typedef QVariant (*LookupFunction)(const QVariant &, const QString &);

    QVariant doLookup(const QVariant &object, const QString &property);

    static void registerLookUpOperator( int id, LookupFunction f );
  };

  template<typename T>
  void registerMetaType()
  {
    QVariant ( *lf )(const QVariant&, const QString&) = LookupTrait<T>::doLookUp;

    const int id = qMetaTypeId<T>();

    MetaType::registerLookUpOperator( id, lf );
  }

  QVariant MetaType::doLookup(const QVariant &object, const QString &property)
  {
    LookupFunction func = registry.value(object.userType());

    // Calls LookUpTrait<T>::doLookUp(QVariant, QString);
    return func( object, property);
  }

There are some things to note there:

  • The signature for looking up a property on an object is available as a typedef on the MetaType class.
  • The same signature is available as a static method on the TypeAccessor class
  • Custom types can be registered using a template function. That retrieves a function pointer to the TypeAccessor template implementation and stores in in the MetaType using the qMetaTypeId of the type as an identifier.
  • The MetaType can later use the type in the QVariant to call the correct function to return a property
  • The custom type must have already been declared as a metatype with Q_DECLARE_METATYPE, but that’s necessary anyway to put it in a QVariant.
  • In practice the details of the TypeAccessor are hidden behind the GRANTLEE_BEGIN_LOOKUP macro.

With the above infrastructure, the rest is just the bread and butter of full and partial template specialization in applications to supply the correct hooks to Grantlee.

  // Application code:

  class Person // Not a QObject
  {
    QString name() const;
    int age() const;
  };

  Q_DECLARE_METATYPE(Person)

  template<>
  struct TypeAccessor<Person>
  {
    static QVariant lookUp(const Person &object, const QString &property)
    {
      // can only call const methods here
      if (property == "name")
        return object.name(); // Returns a QString
      if (property == "int")
        return object.age(); // Returns an int
      else
        return QVariant();
    }
  };

  void init() {
    Grantlee::registerMetaType();
  }

Any type that can be put in a QVariant can be returned from such a function, including other custom types, and including containers. All of this is not necessary for QObject derived types of course, as it is provided automatically by moc. This method is safer however as constness provides compile-time safety which is not provided by Q_PROPERTY. You can not define a clear() method on Person and then call it in the TypeAccessor specialization, whereas with a Q_PROPERTY, the READ method is not enforced to be const.

Template containment

Aside from support for generic types in Grantlee, I wanted generic container support too. Grantlee provides tags to perform {% for %} loops as above, as well as index based lookup for sequential containers and key lookup in associative containers. Rather than just allowing that with QVariantList and QVariantHash, I wanted to be able to loop over or access any Qt container, or any other user specified container. That means that if you already have an API which needs to stay as it is (for binary compatiblity reasons for example), you can use it even without changing the types of any containers in the API to QVariantList or providing conversion wrappers.

The following containers have automatic, built in support in Granltee:

  • QVector<T>, QList<T>, QStack<T>, QQueue<T>, QLinkedList<T>, QSet<T>
  • std::vector<T>, std::deque<T>, std::list<T>
  • QHash<Key, T>, QMap<Key, T>, std::map<Key, T> where Key is a QString or integral

and where T is one of:

  • bool
  • a number
  • QString
  • QVariant
  • QDateTime
  • QObject*
  • Any custom type registered with Grantlee
  • A container of any type on this list.

The container must also be registered as a QMetaType with Q_DECLARE_METATYPE.

Did you see the recursion by the way? It is also possible to use nested containers like QList<std::vector<QSet<Person> > > or StackMapListVectorInt, though possible doesn’t mean should.

Additional containers can easily be supported with a macro:

  // Add support to Grantlee for boost::circular_buffer<T>
  GRANTLEE_REGISTER_SEQUENTIAL_CONTAINER(boost::circular_buffer)

  // Make it possible to put a boost::circular_buffer<Person> into a QVariant
  Q_DECLARE_METATYPE(boost::circular_buffer<Person>)

The unit tests use std::tr1::array and std::tr1::unordered_map containers to demonstrate how to support ‘third party’ containers.

Recursive Autoconditional Container registration

The best part about the container support is that it is completely automatically conditional and determined at build time. That is, if you declare QSet<Person>, but not QList<Person> with Q_DECLARE_METATYPE, Grantlee will support only QSet<Person>, saving the build-time (building template code takes a long time) and the start-up time of registering the other container. The unit test which tests all possible builtin containers (non-nested) takes 5 minutes alone to build. The same is true of the numeric and Qt types. Support for them is determined at build time using sufficient magic

  enum
  {
    Magic,
    MoreMagic
  };

  template<typename T, int n>
  struct RegisterTypeContainer
  {
    static void reg()
    {
    }
  };

  namespace Grantlee {
  template<typename T>
  struct RegisterTypeContainer<QList<T>, MoreMagic>
  {
    static int reg()
    {
      // registerSequentialContainer is like registerMetaType
      return registerSequentialContainer<QList<T> >();
    }
  };
  }

  template<typename T>
  void registerMetaType()
  {
    // This registers the container if QList<T> has been declared as a metatype, and does nothing otherwise.
    Grantlee::RegisterTypeContainer<QList<T>, QMetaTypeId2<QList<T>>::Defined>::reg();
  }

Do you see how the magic works?

Smart pointers

There’s a memory leak on this page. In the first example I created Person QObjects on the heap and then never called delete for them. They also don’t have a parent in the example, so they leak. There are several reasons not to give a QObject a parent. Some short-lived objects have an unclear end-point, but shouldn’t be around as long as their parent. In some threading situations there are also reasons not to give a QObject a parent.

In those cases we can use QSharedPointer to manage the memory of the QObject. Grantlee now also has special (automatic condtional) support for QSharedPointers which manage a QObject derived type. QSharedPointer can now be put into a QVariant and into a Grantlee Context just like any other object, and its Q_PROPERTIES are automatically available in the Grantlee::Template.

  Q_DECLARE_METATYPE(QSharedPointer<Person>)
  Q_DECLARE_METATYPE(QVector<QSharedPointer<Person> >)

...

  QVector<QSharedPointer<Person> > peopleList;
  QSharedPointer<Person> alice(new Person("Alice"));
  peopleList.append(alice);
  QSharedPointer<Person> bob(new Person("Bob"));
  peopleList.append(bob);

  context.insert("people", QVariant::fromValue(peopleList));
  template->render(&context);

Of course it is also possible to use other smart pointers, and smart pointers to non QObject derived types. To support an additional smart pointer, a simple macro is used:

  GRANTLEE_SMART_PTR_ACCESSOR(boost::shared_ptr)
  Q_DECLARE_METATYPE(boost::shared_ptr<Person>)

In KMail, we use objects like boost::shared_ptr<KMime::Message>, so these kinds of features may prove quite useful for advanced theming in the message viewer.

TTP without TTP

There were several times while implementing the support for templated types that TTP could have been useful. Apparently support for TTP is not widespread in some compilers yet, so I avoided using that language feature. What it allows is constructs like this:

  template<typename <typename> class Container, typename T>
  struct ContainerCleaner
  {
    QVariant clean(Container<T> &container)
    {
      // Do some cleaning on Container<T> 
    }
  };

That struct would work with QList<T>, QVector<T> etc automatically as long as the symbolic API used is the same. I wanted something like that in unit tests for clean-up code. After using a Container<QObject*> I wanted to call qDeleteAll() on the container. That would be possible with the above template by partially specializing it for T=T*. Doing the same thing without TTP is almost as easy

  template<typename Container, typename T = typename Container::value_type>
  struct ContainerCleaner
  {
    void clean(Container &container)
    {
      // Do nothing in the common case.
    }
  };

  // Partial specialization for Container<QObject*>
  template<typename Container>
  struct ContainerCleaner<Container, QObject*>
  {
    void clean( Container &container)
    {
      qDeleteAll(container);
    }
  };

  template<typename Container>
  void cleanupContainer c)
  {
    CleanupContainer<Container>::clean(c);
  }

  ...
  // After test:
  cleanup(container);

The only disadvantage is that I need two clean up methods, one for sequential containers, and one for associative containers, because they are not statically polymorphic (value_type vs mapped_type). I also need another specialization for std::map types because qDeleteAll does not work for those.

Conclusion

If you have read this far I can only assume a strong interest in template programming. I’d encourage you to read the commits in the branch. They are sequential and easy to read, and if you spot anything that could be done better, do please tell me.

Advertisements

Grantlee version 0.1.6 now available

September 26, 2010

The Grantlee community is pleased to announce the release of Grantlee version 0.1.6.

This is the first release where the tarballs are mirrored outside of grantlee.org provided by Yury G. Kudryashov. My service provider blocks all traffic from Russia. They also removed the ssh access feature which was a particular reason for me to get a contract with them. If anyone has tips for a better service provider I’m open to moving.

This release is not an exciting one in terms of features (that’s version 0.1.7 🙂 ), but it has several essential and useful changes. Janne Hakonen identified and diagnosed several memory leaks apparent when using plugins written in QtScript, which have now been fixed. Other changes include the use of QT_NO_CAST_FROM_ASCII, QT_NO_CAST_TO_ASCII and QT_STRICT_ITERATORS. There have been lots of patches to KDE lately from Laurent Montel applying similar changes. The ASCII cast defines help QString initialize static string data more efficiently and go some way to prevent encoding related issues cropping up.

Other changes take heed of the recommendations from Qt with regard to static data, and a note Ingo posted regarding order of include directives. Nothing very exciting all-in-all, but an essential maintenance release.

Practicing what I preach

September 17, 2010

Two days ago I came across a Qt bug about adding extra columns in proxy models.

Gabriel de Dietrich is right to think QSortFilterProxyModel is not right for the job of adding extra columns. There are a few dangerous corners in using it for that purpose, such as proper handling of ranges in dataChanged signals, added complications from possible reordering of columns, and the one I mentioned in the bug report – that is – when the source model changes its layout, the QSortFilterProxyModel will only change the layout of the columns that the source model knows about, but not your extra columns, which could lead to a crash in various situations. I also mentioned that although it’s a bad idea, we in Akonadi also have a proxy which does such a thing for adding statistics columns in kmail (unread count, size etc). We also didn’t handle the crash-case I mentioned in the bug report.

Then yesterday I started working on a crash in akonadiconsole, a development tool which uses the same statistics proxy model. The crash happened when moving items between different folders. That in turn caused an intermediate proxy to perform a layout change, which was signaled to our statistics proxy, which then lead to a crash with this backtrace:

Thread 1 (Thread 0xb2b33700 (LWP 16733)):
[KCrash Handler]
#7  0xb60a5a37 in QSortFilterProxyModelPrivate::index_to_iterator (this=0x9eb3be0, proxy_index=...) at /home/kde-devel/kde/src/qt47/src/gui/itemviews/qsortfilterproxymodel.cpp:193
#8  0xb60a1dce in QSortFilterProxyModel::parent (this=0x9eb4028, child=...) at /home/kde-devel/kde/src/qt47/src/gui/itemviews/qsortfilterproxymodel.cpp:1654
#9  0xb7569834 in QModelIndex::parent (this=0x49455467) at /home/kde-devel/kde/qt47/include/QtCore/qabstractitemmodel.h:389
#10 0xb7565e96 in Future::KIdentityProxyModel::parent (this=0x9eb31c0, child=...) at /home/kde-devel/kde/src/KDE/kdepim/akonadi_next/kidentityproxymodel.cpp:353
#11 0xb5409448 in QModelIndex::parent (this=0xa9df368) at /home/kde-devel/kde/src/qt47/src/corelib/kernel/qabstractitemmodel.h:389
#12 0xb54025a2 in QPersistentModelIndex::parent (this=0xa933cf0) at /home/kde-devel/kde/src/qt47/src/corelib/kernel/qabstractitemmodel.cpp:347
#13 0xb600f93b in QItemSelectionRange::parent (this=0xa933cf0) at ../../include/QtGui/../../../../src/qt47/src/gui/itemviews/qitemselectionmodel.h:78
#14 0xb606871e in QItemSelectionRange::contains (this=0xa933cf0, index=...) at /home/kde-devel/kde/src/qt47/src/gui/itemviews/qitemselectionmodel.h:85
#15 0xb6060d93 in QItemSelection::contains (this=0x9eb7890, index=...) at /home/kde-devel/kde/src/qt47/src/gui/itemviews/qitemselectionmodel.cpp:423
#16 0xb6065fed in QItemSelectionModel::isSelected (this=0x9eb7828, index=...) at /home/kde-devel/kde/src/qt47/src/gui/itemviews/qitemselectionmodel.cpp:1200
#17 0xb604b48c in QTreeView::drawRow (this=0x9e26310, painter=0xbfcbd7c8, option=..., index=...) at /home/kde-devel/kde/src/qt47/src/gui/itemviews/qtreeview.cpp:1602
#18 0xb604a861 in QTreeView::drawTree (this=0x9e26310, painter=0xbfcbd7c8, region=...) at /home/kde-devel/kde/src/qt47/src/gui/itemviews/qtreeview.cpp:1441
#19 0xb6049ee3 in QTreeView::paintEvent (this=0x9e26310, event=0xbfcbe0e8) at /home/kde-devel/kde/src/qt47/src/gui/itemviews/qtreeview.cpp:1274
#20 0xb5a150e9 in QWidget::event (this=0x9e26310, event=0xbfcbe0e8) at /home/kde-devel/kde/src/qt47/src/gui/kernel/qwidget.cpp:8346

It’s not obvious where the bug is because it comes from some random paint event rather than something directly related to the move action. After some investigation I found that I was being hit by the exact issue I was describing in the bug report. A QPersistentModelIndex was corrupted and then some time later the QTreeView tried to draw the row for it. The fix was as described and only took a little time to write.

Another lesson that this nicely illustrates is that you need to Q_ASSERT as much as possible when using Qt model-view. Constructs like if (index.model() != this) return hide bugs and will result in you reading backtraces unrelated to the code causing the bug because a segfault or assert will occur later. Although KIdentityProxyModel appeared in the above backtrace it was unrelated to the problem. Assert as early as possible if you expect something from a QModelIndex.