The Germans have a great sense of humour.
I’ve been telling people that for years, but I’m battling against established stereotypes. It always gives me a good laugh when I hear a new expression or turn of phrase that they use. Back in April sebas introduced me to a new one which still gets a chuckle every time I hear it: “Mir ist alles Wurst!”, or sometimes simply “Wurst!”. It is used to express indifference and translates as “It’s all just sausage to me”. The equivalent English phrase would be “It’s all the same to me.”
I always think of “Mir ist alles Wurst” when I think of how Akonadi works. Akonadi is the next generation PIM system being developed to fill the needs of the KDE4 platform. There is work ongoing to port existing KDE PIM applications like KMail, KAddressBook etc to Akonadi, as well as bring it to new platforms like Plasma, because the data it stores is completely independent from the applications used to access and modify it. That is a design feature which makes it possible to access your emails in Mailody, KMail and plasma, and have it all in sync. Always. By using Akonadi, KJots notes can be synced between the KJots application and a plasmoid in plasma. Updates to the notes in either user context are automatically and immediately transferred to the other.
Of course, this is not limited to emails, contacts and notes. In fact it’s not limited at all. Throughout most of the Akonadi stack, all items are treated the same way from an API point of view through the Akonadi::Item class and are serialized and deserialized as needed when retrieved from the Akonadi server. The Akonadi server itself doesn’t care what type a particular Item is. Whether it’s an email, a contact, a note, or anything else, it is simply stored in serialized form, and can be fetched, replaced, removed, moved etc without knowing its type. It’s all sausage.
The way that a particular item is serialized depends on its type. Emails are all serialized and stored the same way. Contacts are serialized and stored the same way (But of course, differently to emails). Rather than using types of sausage to name the different types of data stored in Akonadi (opportunity missed IMO), the Akonadi designers decided to use mimetypes. Emails have the mimetype “message/rfc822”, contacts have the mimetype “text/directory”, etc. These types are not chosen by the Akonadi team, but are specified in several international standards and are core to how email and address systems function. The mimetype describe the structure of the data, and standard compliant libraries can be used to convert it into something that’s easy to use for applications. By serializing an email in “message/rfc822” format, not only KDE applications, but any application capable of deserializing that type can access and manipulate the data. In KDE the data gets transformed to a KMime::Message. In a python world, the same message could potentially be deserialized into an email.Message object and handled in a python only (read: Non-KDE) application. Again, all changes would be immediately and automatically synchronized.
Caches All The Way Down
It makes sense to keep this type independence in as much of the Akonadi stack as possible to allow reusing the same code for different types, including types we haven’t even thought about storing in Akonadi yet. The highest level API we have in Akonadi for dealing with Items is the EntityTreeModel. This class is an implementation of a QAbstractItemModel, which means that the data represented in an application is completely separate to the way it is interacted with. By using this class, any type of Item stored in Akonadi can be stored in a tree which mirrors the structure in Akonadi. In its vanilla form, the EntityTreeModel represents all items in a generic way- just the same as the Akonadi server, they have a mimetype, which is used to determine how to deserialize the Item, and several identifiers. It can also be configured to contain only a certain type, eg only emails, and processing only the subset, kind of like a sharding system.
The numbers in the tree represent Akonadi::Items. Each number is a unique identifier for an item. In its vanilla form, EntityTreeModel has 3 columns.
Akonadi is an asynchronous system. All interaction is done via Job objects which get created to fetch a bunch of items for example. Rather than waiting on the Job to finish and freezing the application UI until the items are returned, the job is “forgotten” until it reports that it has retrieved the items, or has falied to retrieve the items. In the meantime, the user can continue to interact with the application. This has many advantages, but it doesn’t suit the Qt Model-View API, which requires synchronous access, although the model can be filled asynchronously. For that reason, the EntityTreeModel maintains a cache of data from Akonadi, which is itself a cache of data on a remote server, or somewhere in the filesystem. Data can be retrieved and manipulated in the model in a way that seems synchronous to the application by using the QAbstractItemModel API.
Swings and Roundabouts
Having type independence in EntityTreeModel is useful from a library point of view, but for applications, it is necessary to know exactly what type the model is storing. That allows for example to show Subject, Sender and Date if the model contains emails, and Given Name, Last Name and Email Address if it contains contacts. The way this is achieved is by subclassing. EntityTreeModel forms part of a Template Pattern (Template Pattern, not to be confused with C++ Templates as in template class Foo;.). It generically handles fetching, inserting, removing, moving, drag and drop, lazy population and more. It also defines some extra API beyond the Qt Model API to make applications function as required for different types, such as representing different Items differently if they are emails or contacts for example. For those two cases we have subclasses of EntityTreeModel like ContactsModel and MailModel, but a subclass could be implemented for any type known to Akonadi.
Notice that the Items in the tree, the emails and contacts, are not shown as numbers anymore, but show the contact details in the case of the ContactsModel and the Email subject etc in the case of the MailModel. Part of the template interface to implement when subclassing EntityTreeModel is to specify how many columns should be in the model, and what data each column should show. Notice also that there is a resource of tags. Each folder below that, eg “Something” is a Tag in Nepmouk, and the emails in that folder are emails tagged with the Nepmouk tag “Something”. That’s just a teaser for a future post.
Those are typically processed a little bit using proxy models to separate mail folders from emails so they can be represented in different views, for example. That’s a typical pattern in KDEPIM applications, but one not followed by the KJotsModel, which represents Collections of note pages in a tree together.
The mail model split into two views using a series of proxy models. Notice that each view now has different header data. The headers read “Folder” in one view and “Subject”, “From”, “Date” in the other. Defining the header data to be shown is another part of the EntityTreeModel interface that subclasses should implement to customize its appearance. It’s not obvious how to do that using the Qt Model View API, but eventually I did find a way to do it relatively cleanly. To see the result of the split in the case of contacts, see Tobias Koenigs excellent blog post.
One of the more interesting implementations of EntityTreeModel is the AkonadiBrowserModel. We have developed several tools to make Akonadi development possible, including akonadiconsole, which allows browsing the contents of Akonadi, debugging the low level jobs and the notification system and more. The AkonadiBrowserModel is a browsable tree of Collections and Items in Akonadi without any filtering. It can represent the data differently whether the contents of the Item is an email, a contact, an event etc.
Akonadi browser model showing items in their generic form, represented as emails, and represented as contacts. This is implemented internally using a State Pattern. There is an individual State object for each way the data is to be represented.
If Collections and items are usually separated, this begs the question of why keep them together in the EntityTreeModel in the first place? The answer is to make better synchronous caching possible, and to minimize the number of classes to maintain. The previous design used separate Models for Collections and Items. Whenever a Collection was selected, like a mail folder for example, the items contained in it would have to be retrieved from Akonadi. If a different Collection was then selected, the old items would be discarded, and might have to be fetched again later. Additionally, Having separate classes for these things meant implementing and maintaining in two places all the logic to store and keep up-to-date the synchronous cache of Akonadi data, handle drops etc. It’s almost twelve months since I started writing EntityTreeModel, and I can confirm that that’s not trivial. There’s a lot to consider such as handling drops, insertions, removals and moves of items and collections etc. I did toy with having a separate class for actual storage, and making models use that internally, but that turned out to be the wrong approach.
Finally, KJots required that it was possible to represent them all in one tree, so that’s what’s important. You’ve gotta look out for number 1🙂.
Putting Collections and Items together in one model also has some disadvantages. Because items are not removed when a mail folder is un-selected, the amount of data stored in the model can get large and must be invalidated. Also, because the entire data structure of Akonadi is represented by default in EntityTreeModel, application writers have to be a little bit more specific about what they want their instance of it to include. Finally, it provides a single point of failure, and I’ve been breaking it a lot recently to add new features. It’s swings and roundabouts, but this is the working solution today.
Just because you have a model doesn’t mean it must be put in a view. EntityTreeModel can perfectly well be used without a view. That is how the Model for retrieving Grantlee Templates works. Nowhere in the UI is the tree of template data shown, but it does exist in the AkonadiTemplateLoader. It is essentially a headless model. The match() method is used to locate wanted data, and all notifications from Akonadi about templates being changed, inserted or removed are handled internally by the model. By using the lower level Akonadi API like Jobs and Monitor directly, application writers would need to ensure that they connect to Monitor signals related to the items they retrieve using Jobs. I have doubts that application developers would remember to do that each time, so hopefully this “Headless model” approach to accessing data from Akonadi can be used to full effect in the future. EntityTreeModel is missing some pieces to make it a general solution for that currently, but it should be in place in a few months.
Akonadi deals with any type of data you would care to use it for. Serializers exist to serialize data to emails, contact, events, templates, notes, images etc, and writing new serializers can be trivial. Throughout the Akonadi API we have tried to keep functionality of dealing with items separate to knowledge about the type of data in those items, and allow a lot of versatility in how that data can represented to the user. This means that many Akonadi using applications will have a similar code design, and share a lot of code. The application design is of course a separate issue, and limited only by creativity in writing QAbstractItemView implementations. It is too late to rename Akonadi::Item to Akonadi::Sausage because that would be binary incompatible and source incompatible. EntityTreeModel is @since 4.4 though, so I cold potentially rename it as SausageModel. I’m not sure it would pass API review though…