Implementing QVariant/QMetaType features with template tricks

Grantlee::Templates depends largely on QVariant and the Qt MetaType system to provide core features. It is possible to store objects of many different types in a QVariant, making it possible to create non-typesafe systems such as string template systems and javascript wrappers like QtScript. I’ve had several frustrations with QVariant and related classes while implementing Grantlee, and when last Friday a developer unfamiliar with the requirements of the Qt MetaType system hit problems using Grantlee I decided to try to finally fix the issues.

The short of it is that with the patches you can use QVariant::value<T*>() and QVariant::fromValue<T*>() where T is a QObject subclass, and you can use Q_PROPERTY(T* myProp READ myProp ) all without using Q_DECLARE_METATYPE or qRegisterMetaType(). The QObject introspection is used instead of QMetaType introspection in that case. It’s not in Qt yet, and there’s the slight problem that the Q_PROPERTY shortcut doesn’t work in all cases and reports a false negative, so we’ll see what happens there.

That’s not half as interesting as how it is implemented though :).

Problem statement

One issue was that when using custom types, the only type you can extract from a QVariant is the type you put in. Of course for basic types like numbers and strings it is possible to mix and match to some extent, but the issue I hit was that if you put a QObjectDerived* into a QVariant, you can’t take a QObject* out of it … :

  class QObjectDerived : public QObject
  {
    Q_OBJECT
    // ...
  };
  // We declare a pointer to the custom type as a metatype so we can put it in a QVariant.
  Q_DECLARE_METATYPE(QObjectDerived*)
  ...
  QObjectDerived *object = new QObjectDerived(this);
  QVariant variant = QVariant::fromValue(object);
  qDebug() << variant;                          // prints QVariant(QObjectDerived*, )
  qDebug() << variant.value<QObject*>();        // prints QObject(0x0)
  qDebug() << variant.value<QObjectDerived*>(); // QObjectDerived(0x8c491c8)

  variant = QVariant::fromValue(static_cast<QObject*>(object));
  ( or more simply, QVariant::fromValue<QObject*>(object); )
  qDebug() << variant;                          // prints QVariant(QObject*, QObjectDerived(0x8c491c8) )
  qDebug() << variant.value<QObject*>();        // prints QObjectDerived(0x8c491c8)
  qDebug() << variant.value<QObjectDerived*>(); // QObject(0x0)

… in Qt 4.7 :).

If we leave out or forget the Q_DECLARE_METATYPE line, the example doesn’t even build. More on that later.

It seems that we can either extract QObject* or the derived type depending on how we insert it. But can’t we get the best of both worlds? If we hit that bullseye the rest of the dominoes will fall like a house of cards.

? qvariant_cast : qobject_cast

The reason I need to take a QObject* out of a QVariant like that is because Grantlee doesn’t know about the QObjectDerived type. It knows about QObject though, and that’s where all the useful introspection features are. That means that Grantlee users are required to put QObject* into QVariants when populating a Grantlee::Context, and not pointers to QObject derived types. We as developers know that QObjectDerived is-a QObject and can be static_cast to use the introspection features, but the Qt MetaType system doesn’t know that.

So the first task is for QVariant to know whether value() is a valid conversion for the type it contains. The way QVariant knows what type it contains is by storing the type as a simple number. The number is allocated by QMetaType at runtime. Therefore, we can only decide whether to perform the cast or not at runtime. Internal to QVariant is a union of data which includes a void* and a QObject*. So the implementation of value<QObject*>() only needs to return data.o if it a QObject. However, if the internal data in the QVariant is not a QObject*, we’ll get a segfault when we do that.

We need to know ahead of time whether the internal data is-a QObject* and constrain our access to the data based on that.

The way to do that is to store the information about whether a metatype is-a QObject along with the metatype identifier.

SFINASNAFU

So far all we’ve done is to bump the is-a QObject fact finding responsibility from QVariant to QMetaType. We still can’t determine that at runtime, so we have to determine it at compile time. Because QMetaType works with template specializations, we can determine lots of things at compile time using type traits. Qt uses traits already for the QTypeInfo system, so all we need to do is add a trait for whether the metatype inherits QObject and is a pointer. The ‘is a pointer’ part of the trait is easily specified by specializing on T*, and to determine the QObject inheritance, we use SFINAE.

You can only use features you know about

template <typename T>
struct QTypeInfo { enum { isQObjectPointer = false }; };

// Partial Specialization
template <typename T*>
struct QTypeInfo
{
    typedef int yes_type;
    typedef char no_type;

    // SFINAE
    static yes_type check(QObject*);
    static no_type check(...);
    enum { isQObjectPointer = sizeof(check(static_cast<T>(0))) == sizeof(yes_type) };
};

SFINAE makes for some pretty strange C++ code. The class has a static method called ‘check’ which returns a char (typedef’ed) and takes anything at all as arguments, specified by ‘…’. The ‘…’ is a special and rare construct in C++ which means ‘any type and number of arguments’. The check method is also overloaded with a version that specifically takes a QObject* and returns an int (typedef’ed).

When code calls a method like check(), the compiler determines the most specific method to call given the type of the arguments. If check is called with an argument of type pointer to QObject derived type, that is more specific than ‘any arguments at all’, so the compiler determines the QObject* overload to be the correct one. As a side effect, the compiler can now determine the return type of the check() call based on whether the argument is a QObject derived pointer or not. If it is, the return type will be an int (yes_type).

In the enum section we have a call to check with an argument of type T*, which is cast from a null pointer. If T inherits QObject, the return type will be yes_type, otherwise no_type. Because the size of the yes_type and no_type are different (also known to the compiler), the enum, which is also evaluated at compile time, will be given the value true or false based on which overload of check() was called, and whether the return type is yes_type. This it can only determine because int and char have different sizes. As the enum evaluation is replaced with its result at compile time, the call to check() never actually happens, so it doesn’t even need an implementation.

The compiler makes the result available in the form of a boolean which is stored and accessible along with the metatype id at runtime.

Abstractions are good. Let’s have more of those

After that deep-dive, the result is that we can now implement a specialization of the QVariant::value() method something like this:

template<T>
QObject* QVariant::value<QObject*>() {
  if (QMetaType::isQObjectStar(this->type))
    return this->data.o;
  return 0;
}

That means that this now works:

  QObjectDerived *object = new QObjectDerived(this);
  QVariant variant = QVariant::fromValue(object);
  qDebug() << variant;                          // prints QVariant(QObject*, QObjectDerived(0x8c491c8))
  qDebug() << variant.value<QObject*>();        // prints QObjectDerived(0x8c491c8)
  qDebug() << variant.value<QObjectDerived*>(); // QObjectDerived(0x8c491c8)

Notable improvements are
* When we qDebug() the variant we see the address of the pointer, not just the type.
* We can extract the data from the variant in the form of a pointer to the base QObject class

Special in parts, but which parts?

Once we have a QObject* we can qobject_cast that to whatever QObject derived type we like, but ideally that would be built in to our QVariant::value call so that we can do something like this:

class SubQObjectDerived : public QObjectDerived
{
  Q_OBJECT
  // ...
};
Q_DECLARE_METATYPE(SubQObjectDerived*)
...
  SubQObjectDerived *object = new SubQObjectDerived;
  QVariant variant = QVariant::fromValue(object);
  qDebug() << variant;                             // prints QVariant(QObject*, SubQObjectDerived(0x8c491c8))
  qDebug() << variant.value<QObject*>();           // prints SubQObjectDerived(0x8c491c8)
  qDebug() << variant.value<QObjectDerived*>();    // QObject(0x0)
  qDebug() << variant.value<SubQObjectDerived*>(); // prints SubQObjectDerived(0x8c491c8)

So why do we get a QObject(0x0) when we try to extract the data as a QObjectDerived*? The reason is that there is no template specialization for that yet, but we can add one. All we need to do is provide a class to to qvariant conversion, and another class to be used if the template argument is a pointer to a class which inherits QObject.

template<typename T, bool>
struct QVariantConverter
{
  static T convert(const QVariant &variant)
  {
    // Do regular conversion
  }
};
// Partial specialization used only when the second template argument is true.
typename<typename T>
struct QVariantConverter<T, true>
{
  static T convert(const QVariant &variant)
  {
    return qobject_cast<T>(regular_variant_cast<QObject*>(variant));
  }
};

So we have a class that we can use to do a regular conversion in the normal case, and a QObject* conversion followed by a qobject_cast if the second template argument is true.

Now we can implement our QVariant method like this, internalizing all the complexity:

template<typename T>
QVariant::fromValue(const QVariant &variant)
{
  return QVariantConverter<T, QTypeInfo<T>::isQObjectPointer>::convert(variant);
}

So now QVariant::value works for all data that qobject_cast works with. Neat.

As a piece of API sugar we can also implement QVariant::canConvert<T*>() to return true for pointers to QObject derived types.

In(ter)ference

We haven’t quite achieved nerdvana yet.

Because of all the work on value<T>(), we can now extract QObject derived types from QVariants even if the derived type does not have a corresponding Q_DECLARE_METATYPE declaration.

  QObject *object = new CustomQObjectNoMetaType(this);
  QVariant variant = QVariant::fromValue(object);
  qDebug() << variant.value<QObject*>();                 // works !!
  qDebug() << variant.value<CustomQObjectNoMetaType*>(); // works !!

The disadvantage is that we still have to insert the object into the QVariant as a QObject* rather than a CustomQObjectNoMetaType*.

However, QVariant::fromValue() is a template method too, and that means it can be specialized. The implementation is very similar to that of the QVariantConverter above, but in reverse.

template<typename T, bool>
struct QVariantFromValueConverter
{
  static QVariant convert(const T &t)
  {
    // Do regular conversion
  }
};
// Partial specialization used only when the second template argument is true.
typename<typename T>
struct QVariantFromValueConverter<T, true>
{
  static QVariant convert(const T &t)
  {
    return regular_convert_from_value(qobject_cast<QObject*>(t));
  }
};

// ...

template<typename T>
QVariant::fromValue(const T &t)
{
  return QVariantFromValueConverter<T, QTypeInfo<T>::isQObjectPointer>::convert(t);
}

We once again use our compile time check for QObject inheritance, QTypeInfo<T>::isQObjectPointer, to determine how to perform the conversion. By replacing the regular conversion from Derived* with a qobject_cast and a conversion from QObject*, we avoid the call that the regular conversion makes to QMetaType<Derived*>::qt_metatype_id(), which is only defined if we use the Q_DECLARE_METATYPE macro. So now if you forget or omit the macro with QObject derived types, you can still use it in a QVariant.

  CustomQObjectNoMetaType *object = new CustomQObjectNoMetaType(this);
  QVariant variant = QVariant::fromValue(object);
  qDebug() << variant.value<QObject*>();                 // works !!
  qDebug() << variant.value<CustomQObjectNoMetaType*>(); // works !!

But we’re still not done :).

pre-compile-time checks?

The final piece of the puzzle is to make it possible to use pointers to QObject derived types in Q_PROPERTY definitions without any extra work. Currently if using a Derived* in a Q_PROPERTY you do still need to use Q_DECLARE_METATYPE(Derived*) and you need to call qRegisterMetaType<Derived*>() somewhere in your program. Easily forgotten, though the warning message you get is pretty clear. Even all the compile-time checks introduced above won’t help. We need to go earlier.

The problem is that QMetaProperty::read uses the QMetaType system to determine the type of the property at runtime. The type name (as text) is extracted at moc time and used with QMetaType::type at runtime to determine the integer metatype of the property. The the mapping between the text typeName and the integer type is only made at runtime. The qRegisterMetaType<Derived*>() call creates that mapping, and it depends on the existence of Q_DECLARE_METATYPE(Derived*) to work.

However, we know that if the type of the property is a pointer to a QObject derived type, then we can work with the property as a QObject* using QVariant. So if QMetaProperty::read can know whether a property type is a pointer to QObject derived it’s all gravy. The only information QMetaProperty::read has (apart from the information in QMetaType) is the text name of the property and the flags generated by moc.

The answer, reversed

moc doesn’t know anything about the inheritance structure of the code it processes, so it is not as simple as checking at moc time whether the property type inherits QObject and setting a flag in the generated code for that. moc knows the question, but not the answer.

The complete solution is a moc-time check to generate the answer to the question of whether the type matches QObject*, a build-time evaluation of the answer, and then a run-time query of the answer.

The answer is generated by moc by using a simple bitwise operation by generating code something like:

  // property flags
  (QTypeInfo<T>::isQObjectPointer << 24) | 0xf1465

At compile time the bit shift causes a generates a 1 or 0, determined by our now familiar SFINAE class at the appropriate position and the flags will be changed to 0x10f1465 if the property is a QObject. The problem is that when the moc file is built, it’s possible for T to be only a forward declaration. In that case it is not possible to determine if T inherits QObject or not and the negative is assumed. That means that in some cases it would still be necessary to either include the header for T in the Derived Object header file, or revert to the Q_DECLARE_METATYPE/qRegisterMetaType system.

At runtime it’s a regular simple check in QMetaProperty::read for if (flags & IsQObjectPointer).

It’s all very simple really.

I’ve submitted the whole thing to Qt, so maybe it will reach developers in Qt 4.8.