Archive for the ‘Qt’ Category

Another one bytes the dust

April 5, 2011

I’ve just pushed a change to kdepim 4.4 which removes this annoying dialog in a few annoying cases.

For users, this dialog was coming up, and not seeming to give any useful information, and when dismissed, the application was usable.

Showing the dialog was actually a bug that was fixed some time in February 2010 with improvements to the kdepim libraries, but because there was no KDEPIM applications 4.5 release, didn’t make it to users.

The fix was to make the applications not call dangerous API with sub-eventloops.

Making KDEPIM less annoying

April 4, 2011

I’ve started looking into KDEPIM 4.6 on Kubuntu Natty to see if it can be made less annoying to use. There are two unpopular dialogs which appear when using KDEPIM. Both are telling the user that essential PIM services are not fully operational.

The essential PIM services are Akonadi and Nepomuk. Akonadi provides access to all PIM data (emails, contacts, events etc) the user has. It is started automatically if using a PIM application like KMail2, KAddressBook, KOrganizer, KJots and more. There is no configuration option to turn Akonadi off. Akonadi is a cache server which uses a database like MySQL or SQLite to cache data.

Nepomuk provides indexing and searching capabilities to the PIM infrastructure. If you want to search your email, or use autocompletion when typing in email addresses, you need Nepomuk. These are currently considered essential features for a useful PIM stack, so Akonadi depends on Nepomuk being operational. Unfortunately it can be turned off, and when it is off, that’s when the user gets the two unpopular dialogs.

There may be a case for coming up with a unified framework for how services can depend on each other and give the user the opportunities to start essential dependent services. It might be something to discuss at the Platform 11 sprint.

However, there are things we can change in the short-term that can benefit the user. For one, I’ve turned one of the annoying dialogs into a passive notification using KNotification.

A notification is less annoying than a dialog

Next I’ll have to consider how to show the other annoying dialog only when attempting to search or autocomplete email addresses…

QIODevice Cat is(-not)-a QIODevice

March 30, 2011

In some ways, cat is an opposite of tee. Whereas tee reads from a single source and writes to multiple, cat reads from multiple sources including optionally stdin, and writes to a single target, stdout.

Reading from stdin is actually a key difference between cat and echo, in that echo ignores stdin.

# Doesn't work (where's the echo?):
cat somefile | echo

However, in this example cat will uselessly write the input back to the output.

# Does work:
cat somefile | cat

Initially considered making Grantlee::Cat implement QIODevice so that it too could be written to. That would make it analogous to the unix command, but would make the implementation of Cat more complex. It would need to use a Grantlee::Channel and maybe a Grantlee::Reserviour internally (which I haven’t written yet) and wouldn’t really have any advantages that can’t be solved some other way.

Then I decided that I should make it implement QIODevice anyway, because it would make the title of this blog post funnier.

In the end though I decided that blog titles are not a good yardstick to measure the aptness of a design, so Grantlee::Cat is-a QObject instead.

Once I have Grantlee::Channel written it might actually make sense to be able to write to Cat and be easy to implement, so I might reconsider it then anyway.

Cat reads sequentially from a list of QIODevices and writes to a QIODevice target. The internal implementation is very simple. It uses a Grantlee::Pump pointing at the target, and sets the source of the pump to each source device in the list until the source device is closed.

So Cat is easy to use with readable QIODevices, but if you want to write to Cat (from multiple devices), that’s where Grantlee::Channel will come in.

Pumpin’ ain’t easy

March 23, 2011

An example in my last post used a Grantlee::Pump to transfer data from one QIODevice to another. I’ve just added the Pump class to the Grantlee::Tubes library.

Pumping tends to limits of capacity and drainage

QIODevice provides an asynchronous API for clients to use. A call to readAll() will return available data from the device, but with turning of time and the event loop, more data may eventually become available.

  QTcpSocket *socket = getSocket();
  QFile *logFile = getLogFile();

  // Read all data from the socket and write it to the log file as it becomes available.
  Grantlee::Pump *pump = new Grantlee::Pump(this);
  pump->setSource(socket);
  pump->setTarget(logFile);

The Pump encapsulates the handling of the readyRead() signal so that clients don’t need to do that themselves.

Actually, pumpin’ IS easy

I used this class just a short time ago as a debugging tool. Working on an embedded platform with only a single serial cable can be a significant constraint. While evaluating performance on the system I was attempting to run top -d 1 -b & and then starting the target application along with a command line interface to the application. The problem is that any attempt to make top run and record in the background failed. The command line interaction system seemed to conflict with proper execution of top, which simply terminated.

Enter QProcess with Grantlee::Pump.

The trick was to make QProcess run top instead of starting it over the serial connection. The output of top (batched) would be written out by QProcess. Of course, QProcess implements QIODevice, so all I needed to do was pump from the QProcess into a QFile:

int main(int argc, char **argv) 
{
  QProcess topProc;
  topProc.start(QLatin1String("top"), QStringList() << QString::fromLatin1("-d") << QString::fromLatin1("1") << QString::fromLatin1("-b"));

  QApplication app(argc, argv);

  QFile logFile(app.applicationDirPath() + QLatin1String("/topoutput"), this);
  logFile.open(QFile::WriteOnly);

  Grantlee::Pump pump;
  pump.setSource(topProc);
  pump.setTarget(logFile);

  int exitCode = app.exec();

  logFile.close();
  return exitCode;
}

Of course, this is equally possible without Grantlee::Pump. The class does not solve a hard problem, but it solves it in an object orientated way, making it easy to use and reuse as part of larger systems.

Pump takes care of the limiting rate of drainage from the source device. To handle the limiting capacity of the target will require a different Tube.

Tee is for Tubes

March 22, 2011

It is a curiosity that both existing Grantlee libraries begin with the letter ‘T’. This can not have been coincidence. I thought it best that I continue the trend and keep adding libraries whose name begins with ‘T’.

So was renamed the Grantlee Pipes library to Grantlee Tubes. Grantlee Tubes is a library of classes supporting the QIODevice API in an object-orientated way. The developer can connect a series of Tubes to achieve exacting goals in much the same way that the Unix programmer connects commands with pipes (‘|’). The first class to hit the public repo is, appropriately, Grantlee::Tee.

Trial and error

I encountered the tee command when I first started using Ubuntu and came across instructions like this to add repositories to the system:

echo "deb http://security.ubuntu.com/ubuntu jaunty-security main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list

I already knew about shell redirection by then so I had wondered why I couldn’t simply do this:

echo "deb http://security.ubuntu.com/ubuntu jaunty-security main restricted universe multiverse" >> /etc/apt/sources.list

Firstly, permission denied, so try Plan B

sudo echo "deb http://security.ubuntu.com/ubuntu jaunty-security main restricted universe multiverse" >> /etc/apt/sources.list

This doesn’t work because echo is run as superuser, but that permission does not cross the ‘>>’ boundary. It still attempts to write to the root-owned file as a normal user.

tee resolves that issue by reading from stdin and writing to standard out (like cat) AND to any specified targets.

Tee for Two

The name Tee comes from a plumbing term for a device which allows splitting a stream of fluid to two targets. The tee command allows duplicating a stream to N targets, allows quite a bit of versatility.

Tee: Duplicating targets

The GNU Coreutils manual is informative and illustrative in the use and use cases for tee. Primarily the advantages of its use are that it allows processing of streams which is efficient both in terms of memory use (the entire input does not need to be held in memory), and in terms of speed by enabling parallel processing.

 wget -O - http://example.com/dvd.iso \
       | tee dvd.iso | sha1sum > dvd.sha1 

Piping and Tubing

QIODevice is one of the most dominant interfaces in Qt. It is the base for most data reading and writing APIs such as QFile, QTcpSocket, QNetworkReply etc, and it is the interface used in most collaborators to data reading and writing, such as QDataStream, QTextStream, QTextDocumentWriter and more. This means for example that a QTextDocument may be written to a Tcp socket or a QProcess just as easily as it can be written to a file, from a Qt API point of view. The asynchronous nature of QIODevice adds to its versatility and suitability for many tasks around streaming data in the real world.

Just as we combine commands in a Unix shell with pipes, so too we want to be able to combine source and target QIODevices in efficient and familiar ways. It should be possible to implement something like this in Qt:

tardir=grantlee_src
     tar chof - "$tardir" \
       | tee >(gzip -9 -c > grantlee.tar.gz) \
       | bzip2 -9 -c > grantlee.tar.bz2

This is where Grantlee::Tubes comes in.

Tubular

Want to write large data to multiple files without multiple read/write cycles, holding all data in memory or file copying? Easy:

QIODevice *sourceDevice = getSourceData();
QFile *file1 = getFile("file1");
QFile *file2 = getFile("file2");
QFile *file3 = getFile("file3");

Grantlee::Tee *tee = new Grantlee::Tee(this);

tee->appendTarget(file1);
tee->appendTarget(file2);
tee->appendTarget(file3);

// Now write to Tee. readAll for demostration but is not
// memory efficient.
tee->write(sourceDevice->readAll());

Want to write to a log file while also writing to a QTcpSocket? Easy:

QIODevice *sourceDevice = getSourceData();
QTcpSocket *output = getOutput();
QFile *logFile = getLogfile();

Grantlee::Tee *tee = new Grantlee::Tee(this);

tee->appendTarget(output);
tee->appendTarget(logFile);

tee->write(sourceDevice->readAll());

An important point to grasp is that Grantlee::Tee implements QIODevice. That means that you can write to a Tee (and therefore multiple targets) just as easily as you can write to a single target.

This can be made easier (and asyncronous) using Grantlee::Pump:

tee->appendTarget(output);
tee->appendTarget(logFile);

// Replaced with Pump.
// tee->write(sourceDevice->readAll());

Grantlee::Pump *pump = new Grantlee::Pump(this);
pump->setSource(sourceDevice);
pump->setTarget(tee);

Grantlee::Pump listens to the readyRead signal and writes all available data in the source to the target until there’s no more left. That means that data can be fed into Tee in smaller chunks and it is not required to readAll data into memory up front.

Compression is equally possible. Here’s an example where a Grantlee::Template is rendered to a QIODevice which is piped through a compressor, and then a TcpSocket:

QTcpSocket *socket = getSocket();
Grantlee::Compressor *compressor = new Grantlee::Compressor(this);
compressor->setTarget(socket);

Template t = engine->newTemplate("main.html");
Context c = getContext();
// The string template is rendered to the compressor, which writes to the socket. 
t->render(compressor, c);

Down to a Tee

Several of these classes are already in various stages of being written, getting tests and documentation. For now I’ve added Tee to a volatile branch of Grantlee destined for some future release. Along with that all of the infrastructure for a third library in Grantlee has been added: documentation, CMake infrastructure, test framework etc. Grantlee::Tubes will continue growing in the coming days.

Gory technical details

March 20, 2011

In a previous post I wrote some details about how SFINAE works and provides type introspection, and how it can be used to make QVariant more effective at providing access to QObject derived classes. I submitted the patches to Qt where it got thoroughly reviewed.

One of the issues raised in review was that if the SFINAE template is used with a T which is only forward declared it does not work as expected.

  class MyObject;

  void fwdDeclared() {
    qDebug() << "fwdDeclared" << QTypeInfo<MyObject*>::isQObjectPointer;
  }

  class MyObject : public QObject 
  {
    Q_OBJECT
    // ...
  }

  void fullyDefined() {
    qDebug() << "fullyDefined" << QTypeInfo<MyObject*>::isQObjectPointer;
  }

  int main() {
    fwdDeclared();
    fullyDefined();
  }

...

$ ./test
fwdDeclared 0
fullyDefined 0

Why do we get 0 instead of one in both cases? MyClass clearly inherits QObject. Let’s try commenting out the body of fwdDeclared.

  void fwdDeclared() {
//    qDebug() << "fwdDeclared" << QTypeInfo<MyObject*>::isQObjectPointer;
  }
  
$ ./test
fullyDefined 1

Now it works. We get the expected 1 resulting from the isQObjectPointer being assigned a true. What’s going on?

It is often wise to avoid false negatives

At the point where the fwdDeclared function is defined, MyClass has only been forward declared – it is an incomplete type.

Recall that our SFINAE template depends on function overloading depending on the type of the argument to the template:

template<typename T>
struct QTypeInfo<T*>
{  
  yes_type check(QObject*);
  no_type check(...);
  enum { isQObjectPointer = sizeof(check(static_cast<T*>(0)) == sizeof(yes_type) };
};

The first time QTypeInfo<FirstType*>::isQObjectPointer is encountered is in fwdDeclared. At that point in the file, any FirstType* (such as the one in the SFINAE template) will be treated as a void pointer. That means that the catch-all check method which is not the QObject* overload will be used which returns a no_type, and the enum will be resolved to false.

This result as it is first defined will be used everywhere in the translation unit. QTypeInfo<FirstType*>::isQObjectPointer will be treated as an alias to false even after FirstType is fully defined. When we comment out the body of the fwdDeclared function, then the first time QTypeInfo<FirstType*>::isQObjectPointer is encountered is in the fullyDefined function so the base type can be seen by the compiler to be a QObject and the enum evaluates to true.

So what is a translation unit? Can’t we just consider not using them if they’re a problem?

Lost in translation

Translation is what a compiler does to turn source code into object code. A translation unit or compilation unit, is effectively the output of the preprocessor after resolving all the #if, #ifdef and #include etc directives. A forward declaration can appear multiple times in a translation unit, but a definition can only appear zero or one times. This is the one definition rule.

In the example above, the first time QTypeInfo<FirstType*>::isQObjectPointer is encountered it is evaluated to false, therefore if ODR is to be enforced within the translation unit, it must must be false throughout the entire translation unit.

QTypeInfo<FirstType*>::isQObjectPointer might even have a different value in different translation units depending on whether FirstType was defined or forward declared in each one, or the compiler might just spit it out if you try it, though GCC isn’t there yet.

The ODR is not necessarily enforced by the compiler. Causing the template trait to be evaluated differently would also conceivably be an ODR violation and it is undefined whether the compiler needs to do anything about that because compilers don’t necessarily have all the information to make that detection.

Relying on undefined behaviour would be dangerous.

You just have to deliberately the whole thing!

To bring this back to the QVariant/QMetaType context, the SFINAE template as it was before did function if the type T was forward declared, although it gave the wrong answer. Any of the QVariant functions that make use of the template require T to be a complete type anyway, but that still leaves the issues of maintainability (Qt developer in the future might use the trait in a way that doesn’t require the full type) and C++ header voyeurism (third party sees something in internal API that looks useful and uses it, with undefined, undebuggable results).

Both are valid issues of course, but we already know that using the trait with a forward declared T always gives the wrong answer simply because it doesn’t always give the right answer and might cause ODR violations. So we want to enforce that the type is fully defined.

The way to do that is to simply do use the type in a way that requires the full type to be known. One option would be to try to call a static method like connect() on the QObject that T is supposed to inherit. If a call to connect() can be made the type is fully defined. This fails to compile of course if T happens to not have a static connect() method in it’s interface. That option goes out the window as we do still need to compile if it is not a QObject.

Size *does* matter

Another language feature that requires the full type to be defined is the sizeof operator. sizeof(T*) == sizeof(void*) in most cases and that works even if T is forward declared. sizeof(T) however requires that T be fully defined. So all we need to do is invoke sizeof(T) somewhere in our SFINAE template and we’ll get a compiler error if it isn’t.

template<typename T>
struct QTypeInfo<T*>
{  
  yes_type check(QObject*);
  no_type check(...);
  enum { isQObjectPointer = sizeof(check(static_cast<T*>(0)) == sizeof(yes_type) + (0 * sizeof(T)) };
};

Consequence

Because we now require that the full type is known when evaluating the SFINAE template, we compromise the feature of automatic Q_PROPERTY handling. The translation unit that the .moc file is in does not necessarily contain a full definition for T so QTypeInfo<T*>::isQObjectPointer won’t necessarily compile and we can’t put it in .moc files.

The final twist in the tale is that we can’t even use QMetaType to store the information about whether a type is a QObject subclass anymore. In the previous patch that information was stored in the QMetaType data structures as a result of the qRegisterMetaType() call. However, it turns out that qRegisterMetaType also does not require the T to be a complete type. Using our SFINAE class inside that method imposes that as a new restriction which is source incompatible.

So instead of storing that information once per metatype in QMetaType, we have to store it in the QVariant as part of the data type stored in it. This turned out to be a better solution in the end anyway because it eliminates calls to QMetaType which require locking and unlocking a mutex.

It’s still not in Qt yet though, we’ll have to see if it makes it.

[Aside: A picture is worth a thousand words, and this post is now just over a thousand. I guess the picture is only worth the words if you know the words...]

ODR compromise

March 17, 2011

Implementing QVariant/QMetaType features with template tricks

March 16, 2011

Grantlee::Templates depends largely on QVariant and the Qt MetaType system to provide core features. It is possible to store objects of many different types in a QVariant, making it possible to create non-typesafe systems such as string template systems and javascript wrappers like QtScript. I’ve had several frustrations with QVariant and related classes while implementing Grantlee, and when last Friday a developer unfamiliar with the requirements of the Qt MetaType system hit problems using Grantlee I decided to try to finally fix the issues.

The short of it is that with the patches you can use QVariant::value<T*>() and QVariant::fromValue<T*>() where T is a QObject subclass, and you can use Q_PROPERTY(T* myProp READ myProp ) all without using Q_DECLARE_METATYPE or qRegisterMetaType(). The QObject introspection is used instead of QMetaType introspection in that case. It’s not in Qt yet, and there’s the slight problem that the Q_PROPERTY shortcut doesn’t work in all cases and reports a false negative, so we’ll see what happens there.

That’s not half as interesting as how it is implemented though :).

Problem statement

One issue was that when using custom types, the only type you can extract from a QVariant is the type you put in. Of course for basic types like numbers and strings it is possible to mix and match to some extent, but the issue I hit was that if you put a QObjectDerived* into a QVariant, you can’t take a QObject* out of it … :

  class QObjectDerived : public QObject
  {
    Q_OBJECT
    // ...
  };
  // We declare a pointer to the custom type as a metatype so we can put it in a QVariant.
  Q_DECLARE_METATYPE(QObjectDerived*)
  ...
  QObjectDerived *object = new QObjectDerived(this);
  QVariant variant = QVariant::fromValue(object);
  qDebug() << variant;                          // prints QVariant(QObjectDerived*, )
  qDebug() << variant.value<QObject*>();        // prints QObject(0x0)
  qDebug() << variant.value<QObjectDerived*>(); // QObjectDerived(0x8c491c8)

  variant = QVariant::fromValue(static_cast<QObject*>(object));
  ( or more simply, QVariant::fromValue<QObject*>(object); )
  qDebug() << variant;                          // prints QVariant(QObject*, QObjectDerived(0x8c491c8) )
  qDebug() << variant.value<QObject*>();        // prints QObjectDerived(0x8c491c8)
  qDebug() << variant.value<QObjectDerived*>(); // QObject(0x0)

… in Qt 4.7 :).

If we leave out or forget the Q_DECLARE_METATYPE line, the example doesn’t even build. More on that later.

It seems that we can either extract QObject* or the derived type depending on how we insert it. But can’t we get the best of both worlds? If we hit that bullseye the rest of the dominoes will fall like a house of cards.

? qvariant_cast : qobject_cast

The reason I need to take a QObject* out of a QVariant like that is because Grantlee doesn’t know about the QObjectDerived type. It knows about QObject though, and that’s where all the useful introspection features are. That means that Grantlee users are required to put QObject* into QVariants when populating a Grantlee::Context, and not pointers to QObject derived types. We as developers know that QObjectDerived is-a QObject and can be static_cast to use the introspection features, but the Qt MetaType system doesn’t know that.

So the first task is for QVariant to know whether value() is a valid conversion for the type it contains. The way QVariant knows what type it contains is by storing the type as a simple number. The number is allocated by QMetaType at runtime. Therefore, we can only decide whether to perform the cast or not at runtime. Internal to QVariant is a union of data which includes a void* and a QObject*. So the implementation of value<QObject*>() only needs to return data.o if it a QObject. However, if the internal data in the QVariant is not a QObject*, we’ll get a segfault when we do that.

We need to know ahead of time whether the internal data is-a QObject* and constrain our access to the data based on that.

The way to do that is to store the information about whether a metatype is-a QObject along with the metatype identifier.

SFINASNAFU

So far all we’ve done is to bump the is-a QObject fact finding responsibility from QVariant to QMetaType. We still can’t determine that at runtime, so we have to determine it at compile time. Because QMetaType works with template specializations, we can determine lots of things at compile time using type traits. Qt uses traits already for the QTypeInfo system, so all we need to do is add a trait for whether the metatype inherits QObject and is a pointer. The ‘is a pointer’ part of the trait is easily specified by specializing on T*, and to determine the QObject inheritance, we use SFINAE.

You can only use features you know about

template <typename T>
struct QTypeInfo { enum { isQObjectPointer = false }; };

// Partial Specialization
template <typename T*>
struct QTypeInfo
{
    typedef int yes_type;
    typedef char no_type;

    // SFINAE
    static yes_type check(QObject*);
    static no_type check(...);
    enum { isQObjectPointer = sizeof(check(static_cast<T>(0))) == sizeof(yes_type) };
};

SFINAE makes for some pretty strange C++ code. The class has a static method called ‘check’ which returns a char (typedef’ed) and takes anything at all as arguments, specified by ‘…’. The ‘…’ is a special and rare construct in C++ which means ‘any type and number of arguments’. The check method is also overloaded with a version that specifically takes a QObject* and returns an int (typedef’ed).

When code calls a method like check(), the compiler determines the most specific method to call given the type of the arguments. If check is called with an argument of type pointer to QObject derived type, that is more specific than ‘any arguments at all’, so the compiler determines the QObject* overload to be the correct one. As a side effect, the compiler can now determine the return type of the check() call based on whether the argument is a QObject derived pointer or not. If it is, the return type will be an int (yes_type).

In the enum section we have a call to check with an argument of type T*, which is cast from a null pointer. If T inherits QObject, the return type will be yes_type, otherwise no_type. Because the size of the yes_type and no_type are different (also known to the compiler), the enum, which is also evaluated at compile time, will be given the value true or false based on which overload of check() was called, and whether the return type is yes_type. This it can only determine because int and char have different sizes. As the enum evaluation is replaced with its result at compile time, the call to check() never actually happens, so it doesn’t even need an implementation.

The compiler makes the result available in the form of a boolean which is stored and accessible along with the metatype id at runtime.

Abstractions are good. Let’s have more of those

After that deep-dive, the result is that we can now implement a specialization of the QVariant::value() method something like this:

template<T>
QObject* QVariant::value<QObject*>() {
  if (QMetaType::isQObjectStar(this->type))
    return this->data.o;
  return 0;
}

That means that this now works:

  QObjectDerived *object = new QObjectDerived(this);
  QVariant variant = QVariant::fromValue(object);
  qDebug() << variant;                          // prints QVariant(QObject*, QObjectDerived(0x8c491c8))
  qDebug() << variant.value<QObject*>();        // prints QObjectDerived(0x8c491c8)
  qDebug() << variant.value<QObjectDerived*>(); // QObjectDerived(0x8c491c8)

Notable improvements are
* When we qDebug() the variant we see the address of the pointer, not just the type.
* We can extract the data from the variant in the form of a pointer to the base QObject class

Special in parts, but which parts?

Once we have a QObject* we can qobject_cast that to whatever QObject derived type we like, but ideally that would be built in to our QVariant::value call so that we can do something like this:

class SubQObjectDerived : public QObjectDerived
{
  Q_OBJECT
  // ...
};
Q_DECLARE_METATYPE(SubQObjectDerived*)
...
  SubQObjectDerived *object = new SubQObjectDerived;
  QVariant variant = QVariant::fromValue(object);
  qDebug() << variant;                             // prints QVariant(QObject*, SubQObjectDerived(0x8c491c8))
  qDebug() << variant.value<QObject*>();           // prints SubQObjectDerived(0x8c491c8)
  qDebug() << variant.value<QObjectDerived*>();    // QObject(0x0)
  qDebug() << variant.value<SubQObjectDerived*>(); // prints SubQObjectDerived(0x8c491c8)

So why do we get a QObject(0x0) when we try to extract the data as a QObjectDerived*? The reason is that there is no template specialization for that yet, but we can add one. All we need to do is provide a class to to qvariant conversion, and another class to be used if the template argument is a pointer to a class which inherits QObject.

template<typename T, bool>
struct QVariantConverter
{
  static T convert(const QVariant &variant)
  {
    // Do regular conversion
  }
};
// Partial specialization used only when the second template argument is true.
typename<typename T>
struct QVariantConverter<T, true>
{
  static T convert(const QVariant &variant)
  {
    return qobject_cast<T>(regular_variant_cast<QObject*>(variant));
  }
};

So we have a class that we can use to do a regular conversion in the normal case, and a QObject* conversion followed by a qobject_cast if the second template argument is true.

Now we can implement our QVariant method like this, internalizing all the complexity:

template<typename T>
QVariant::fromValue(const QVariant &variant)
{
  return QVariantConverter<T, QTypeInfo<T>::isQObjectPointer>::convert(variant);
}

So now QVariant::value works for all data that qobject_cast works with. Neat.

As a piece of API sugar we can also implement QVariant::canConvert<T*>() to return true for pointers to QObject derived types.

In(ter)ference

We haven’t quite achieved nerdvana yet.

Because of all the work on value<T>(), we can now extract QObject derived types from QVariants even if the derived type does not have a corresponding Q_DECLARE_METATYPE declaration.

  QObject *object = new CustomQObjectNoMetaType(this);
  QVariant variant = QVariant::fromValue(object);
  qDebug() << variant.value<QObject*>();                 // works !!
  qDebug() << variant.value<CustomQObjectNoMetaType*>(); // works !!

The disadvantage is that we still have to insert the object into the QVariant as a QObject* rather than a CustomQObjectNoMetaType*.

However, QVariant::fromValue() is a template method too, and that means it can be specialized. The implementation is very similar to that of the QVariantConverter above, but in reverse.

template<typename T, bool>
struct QVariantFromValueConverter
{
  static QVariant convert(const T &t)
  {
    // Do regular conversion
  }
};
// Partial specialization used only when the second template argument is true.
typename<typename T>
struct QVariantFromValueConverter<T, true>
{
  static QVariant convert(const T &t)
  {
    return regular_convert_from_value(qobject_cast<QObject*>(t));
  }
};

// ...

template<typename T>
QVariant::fromValue(const T &t)
{
  return QVariantFromValueConverter<T, QTypeInfo<T>::isQObjectPointer>::convert(t);
}

We once again use our compile time check for QObject inheritance, QTypeInfo<T>::isQObjectPointer, to determine how to perform the conversion. By replacing the regular conversion from Derived* with a qobject_cast and a conversion from QObject*, we avoid the call that the regular conversion makes to QMetaType<Derived*>::qt_metatype_id(), which is only defined if we use the Q_DECLARE_METATYPE macro. So now if you forget or omit the macro with QObject derived types, you can still use it in a QVariant.

  CustomQObjectNoMetaType *object = new CustomQObjectNoMetaType(this);
  QVariant variant = QVariant::fromValue(object);
  qDebug() << variant.value<QObject*>();                 // works !!
  qDebug() << variant.value<CustomQObjectNoMetaType*>(); // works !!

But we’re still not done :).

pre-compile-time checks?

The final piece of the puzzle is to make it possible to use pointers to QObject derived types in Q_PROPERTY definitions without any extra work. Currently if using a Derived* in a Q_PROPERTY you do still need to use Q_DECLARE_METATYPE(Derived*) and you need to call qRegisterMetaType<Derived*>() somewhere in your program. Easily forgotten, though the warning message you get is pretty clear. Even all the compile-time checks introduced above won’t help. We need to go earlier.

The problem is that QMetaProperty::read uses the QMetaType system to determine the type of the property at runtime. The type name (as text) is extracted at moc time and used with QMetaType::type at runtime to determine the integer metatype of the property. The the mapping between the text typeName and the integer type is only made at runtime. The qRegisterMetaType<Derived*>() call creates that mapping, and it depends on the existence of Q_DECLARE_METATYPE(Derived*) to work.

However, we know that if the type of the property is a pointer to a QObject derived type, then we can work with the property as a QObject* using QVariant. So if QMetaProperty::read can know whether a property type is a pointer to QObject derived it’s all gravy. The only information QMetaProperty::read has (apart from the information in QMetaType) is the text name of the property and the flags generated by moc.

The answer, reversed

moc doesn’t know anything about the inheritance structure of the code it processes, so it is not as simple as checking at moc time whether the property type inherits QObject and setting a flag in the generated code for that. moc knows the question, but not the answer.

The complete solution is a moc-time check to generate the answer to the question of whether the type matches QObject*, a build-time evaluation of the answer, and then a run-time query of the answer.

The answer is generated by moc by using a simple bitwise operation by generating code something like:

  // property flags
  (QTypeInfo<T>::isQObjectPointer << 24) | 0xf1465

At compile time the bit shift causes a generates a 1 or 0, determined by our now familiar SFINAE class at the appropriate position and the flags will be changed to 0x10f1465 if the property is a QObject. The problem is that when the moc file is built, it’s possible for T to be only a forward declaration. In that case it is not possible to determine if T inherits QObject or not and the negative is assumed. That means that in some cases it would still be necessary to either include the header for T in the Derived Object header file, or revert to the Q_DECLARE_METATYPE/qRegisterMetaType system.

At runtime it’s a regular simple check in QMetaProperty::read for if (flags & IsQObjectPointer).

It’s all very simple really.

I’ve submitted the whole thing to Qt, so maybe it will reach developers in Qt 4.8.

Grantlee version 0.1.8 (Codename: Der Hammer) now available

January 26, 2011

The Grantlee community is pleased to announce the release of Grantlee version 0.1.8 (Mirror).

This release includes the internationalization and localization features, and the version number is entirely coincidental :). There has been more development in this release than between any other two Grantlee releases. Most of the work went into i18n along with its tests, examples and documentation, but there was also the usual round of whitespace and const cleanups, support for QDate and QTime objects (with a very simple patch) and a couple of Windows issues resolved. All the unit tests pass on Windows again.

Another change though looks inconsequential, but is quite relevant to the future development of Grantlee. Internally, grantlee_core was renamed grantlee_templates, and grantlee_gui was renamed grantlee_textdocument. Grantlee will soon be getting some new libraries which are not directly related to string templating. Actually it already has one, as the rich text builders do not relate to string templating, though the two systems can be used together.

The new names reflect that the libraries are independent – grantlee_gui doesn’t depend on grantlee_core and never did – and also reflects better what the libraries actually relate to. Now though they’re also in completely separate directory trees, which will make it easy to add additional libraries.

Localization: Tricky issue

November 11, 2010

Vernacular localization: Tricky issue

For a foreigner living in Germany there are several cultural and vernacular differences that you notice straight away, and several more subtle ones that take a while for you to realize, even though they have been staring you in the face the whole time.

Something that is important to get correct from the start is time-keeping. 7:30 is called “halb acht” in German, which translates as “half eight”. While it first appears that the Germans are calling it half to eight instead of half past eight, the reality is they’re saying half of eight. This allows constructs like “drei viertel acht” or “three quarters of eight” for 7:45, and “fünf vor halb acht” or “five to half of eight” for 7:25. This seems perfectly normal in Germany. Eventually. There are several other time-based peculiarities in German, such as the word for “tomorrow” being the same as the word for “morning” (“morgen”), so that if you want to say “tomorrow morning”, you have to say “early tomorrow” (“morgen früh”), whereas if you want to say “this morning”, you have to say “today morning” (“heute morgen”).

Date handling in Germany is standardized in dd.mm.yyyy format instead of dd-mm-yyyy or dd/mm/yyyy. At least it’s not mm/dd/yyyy, which is the bane of my life. Anytime I see 4/6/2010 I have to look for more dates on the same page or document until I find one where the first or second element is greater than 12, or I will have no idea whether it’s an April or June date.

When the number 100.001 is written in English speaking locales it (usually?) means “one hundred point zero zero one”, whereas in Germany, it means “one hundred thousand and one”. 100,001 in Germany means “one hundred point zero zero one”. For a reason I can’t grasp, the German and English localization systems evolved to use exact opposite characters for thousands separation and decimal separation.

Something I didn’t notice until this week though is that currency formatting differs between Ireland and Germany. In Germany the Euro symbol is put after the amount (Except in a badly localized flash ad that’s there now), but in Ireland the symbol is in front of the amount.

Literal or near-literal translation brings more entertainment. Most German people I know have a Chef that they see most days of the week, but who does not cook for them. How’s that for a false friend?

Although most simple sentences similar to “I am hungry” can be translated to “Ich bin hungrig”, it is not correct to translate “I am cold” as “Ich bin kalt”, because that means “I am a cold-hearted person”. Instead it must be “Mir ist kalt”. In the same vein, “How are you” – “I’m good” doesn’t work in German. The “Ich bin gut” means “I’m a morally good person”, which is a non-sequitur in this context. The answer has to be “Mir ist gut” or “Mir geht’s gut”. I also found out that when a waiter asks if I’d like another beer, and I don’t, “No, I’m Ok” doesn’t translate to German very well. Sometimes these things are discovered when the reply is laughter and a “Yeah, we think you’re ok too Steve…”.

My old favorite German term is “krass”, which is pretty much an emphasizer, usually I hear it as a negative, but can be used in a positive too. The weather can be krass. A movie can be “krass lustig”. The best (“krasste”?) moment for krass was when a friend of mine described a particular situation as being “krass krass”. My new favorite German phrase is “Der Hammer!”. Der Hammer is the best thing there is. In en_US that would be ‘Da Bomb!’ I guess.

Gut > Besser > Am Besten > Der Hammer

On top of all the peculiarities and differences in expression, native speakers of English living in or visiting Germany are called “native speakers”. To me “native” means “indigenous to the current geography”, so I still find that a bit weird.

It’s not that English doesn’t have it’s fair share of peculiarities. Why does inflammable mean flammable, for a common example? Even within English there are wild deviations from the norm. That’s a great article by the way. If you want to know what “long monophthongs are often dipthongized, and while some diphthongs are tripthongized” means, take a break from the MeeGo conference and go to Moore Street for “Bananids five for a pow-und”. A notable omission from the Lexicon on the page though is ‘fierce’, which translates to German as approximately ‘krass’. With all that, it’s not uncommon for natives of the Queens English to have problems understanding Hiberno-English, and not just because of the accent. Check this Video with NSFW Audio to more from the Wexford Lexicon without the accent and see if you can make any sense of it.

The point is that localization issues are things we encounter everyday, within and without computer systems but they have to be handled in computer systems too.

Mann, eh?

Abstract localization: Tricky issue

For Qt developers the first thing localization means is a choice. Do the Qt built-in localization facilities meet the applications needs? Would gettext be different? Can KLocale be used instead? The answer to all three is, of course, Yes. Qt localization is quite extensive for most Qt developers which don’t target very uncommon locales. KLocale is a more powerful localization system, and is part of the KDE platform which can be used without many extra dependencies than QtCore and gettext, which it extends.

Of course, KLocale and Qt localization systems are incompatible – They use different conventions in strings for plurals and different forms of catalogs, so a Qt developer has to pick one up front. For developers of libraries like Grantlee, that brings a need for abstraction. Grantlee now features an AbstractLocalizer which can be implemented to support any localization system. I’ve already implemented it for QLocale and for KLocale. That means that Grantlee templates can now be localized easily in both Qt applications and KDE applications.

We have a joke in the Grantlee community there are so many examples distributed in Grantlee for text editing, code generation etc that KDE is almost obsoleted (har-har). To test out the l10n feature I’ve added an address book application:

Contacts with plurals en_US

Contacts with plurals

Contacts with plurals de_DE

There are several important things to note here:

  • I suck at creating good looking html
  • Strings are translated
  • Strings with plural forms are translated
  • Numbers are localized properly. The German version uses comma for decimal separation.
  • Money is localized properly. The German version puts the Euro symbol on the right.
  • Dates are localized properly. The en_US version is wrong, and the German version uses dots for separators.

    The localization supports all of the common localization needs for application development.

  • Using _() to localize strings: {{ _(“Table of contents”) }}
  • Using _() to localize anything: {{ _(some_date_variable) }} where some_date_variable is a QDate outputs 14/3/2010 in en_GB, 3/14/2010 in en_US, and 14.3.2010 in de_DE. Numbers like {{ _(100001) }} are also localized properly.
  • Using i18n for argument substitution: {% i18n “Hello %1, Welcome to %2″ username website_name %}
  • Using i18nc for disambiguated localization: {% i18nc “Name of a person” “Name” %}, {% i18nc “Name of a category” “Name” %}
  • Using i18np for plurals: {% i18np “%1 person” “%1 people” numPeople %}
  • Using i18ncp for disambiguated plurals {% i18ncp “The amount of people logged in” “%1 person” “%1 people” numPeople %}
  • Using l10n_money to localize monetary values: {% l10n_money amount “EUR” %} outputs for example, “€ 99.99″ in the Irish locale, and “99,99 €” in the German locale.

    Isolated localization: Tricky issue

    Usually if you are creating content, the content is created in the locale the application is running in. For example, I can export address book entries from KAddressBook and birthdays will be formatted as 20/11/2009. If I want to export the address book entries and send them to a German person it should be localized for Germany. To do that I would have to restart the application with the correct locale or switch my system locale. With Grantlee and AbstractLocalizer it is now possible to construct a localizer with the appropriate QLocale or KLocale and the exported document will be correctly localized.

    So if you use Grantlee to export content you can let the user decide what locale to target when exporting.

    Mixed localization: Tricky issue

    When learning a language it can be helpful to see the same terms and phrases in the same place in different languages. By using Grantlee for theming you can now allow users to use multiple locales/languages in the same rendering.

    Tables in multiple locales

    Table with multi-lingual header

    Admittedly, I only added this feature because I wanted to see if it would be possible. I don’t know if anyone would use it.

    Extended localization: Tricky issue

    The whole point of Grantlee is to enable accessors to become mutators – enable users to create and share themes for applications like KAddressBook, KMail etc. Those themes will have to be localized, even if the theme introduces strings which were not already in the original application. Now themers will be able to custom create themes and even add custom fields which can be translated. The translations of course would need to be shipped with the theme.

    In this example, a themer has added a calculation to the theme (in javascript, because we can) for the age/salary ratio.

    Theme with age/salary ratio en_US

    This can then be translated and localized easily:

    Theme with age/salary ratio de_DE

    Localization: Easy

    i18n/l10n has been on the roadmap for Grantlee since day one. I started working on this at Akademy in Tampere, and am only nearly finished now. It took a few attempts to get the design right and learn everything needed about localization in general and computerized localization in particular. Now it’s easy for everyone else. :)

    All of this works of course with “pure” Qt and with KLocale and the extensions it provides.

    BTW for more on the elite Germans of Berlin check out Wash Echte.


  • Follow

    Get every new post delivered to your Inbox.