Generating Python Bindings with Clang

Python Bindings

Python provides a C API to define libraries which can be imported into a python environment. That python C interface is often used to provide bindings to existing libraries written in C++ and there are multiple existing technologies for that such as PyQt, PySide, Boost.Python and pybind.

pybind seems to be a more modern implementation of what Boost.Python provides. To create a binding, C++ code is written describing how the target API is to be exposed. That gives the flexibility of defining precisely which methods get exposed to the python environment and which do not.

PyQt and PySide are also similar in that they require the maintenance of a binding specification. In the case of PySide, that is an XML file, and in the case of PyQt that is a DSL which is similar to a C++ header. The advantage both of these systems have for Qt-based code is that they ship with bindings for Qt, relieving the binding author of the requirement to create those bindings.

PyKF5 application using KItemModels and KWidgetsAddons

PyKF5 application using KItemModels and KWidgetsAddons

Generated Python Bindings

For libraries which are large library collections, such as Qt5 and KDE Frameworks 5, manual binding specification soon becomes a task which is not suitable for manual creation by humans, so it is common to have the bindings generated from the C++ headers themselves.

The way that KDE libraries and bindings were organized in the time of KDE4 led to PyQt-based bindings being generated in a semi-automated process and then checked into the SCM repository and maintained there. The C++ header parsing tools used to do that were written before standardization of C++11 and have not kept pace with compilers adding new language features, and C++ headers using them.

Automatically Generated Python Bindings

It came as a surprise to me that no bindings had been completed for the KDE Frameworks 5 libraries. An email from Shaheed Hague about a fresh approach to generating bindings using clang looked promising, but he was hitting problems with linking binding code to the correct shared libraries and generally figuring out what the end-goal looks like. Having used clang APIs before, and having some experience with CMake, I decided to see what I could do to help.

Since then I’ve been helping get the bindings generator into something of a final form for the purposes of KDE Frameworks, and any other Qt-based or even non-Qt-based libraries. The binding generator uses the clang python cindex API to parse the headers of each library and generate a set of sip files, which are then processed to create the bindings. As the core concept of the generator is simply ‘use clang to parse the headers’ it can be adapted to other binding technologies in the future (such as PySide). PyQt-based bindings are the current focus because that fills a gap between what was provided with KDE4 and what is provided by KDE Frameworks 5.

All of that is internal though and doesn’t appear in the buildsystem of any framework. As far as each buildsystem is concerned, a single CMake macro is used to enable the build of python (2 and 3) bindings for a KDE Frameworks library:


  ecm_generate_python_binding(
    TARGET KF5::ItemModels
    PYTHONNAMESPACE PyKF5
    MODULENAME KItemModels
    SIP_DEPENDS
      QtCore/QtCoremod.sip
    HEADERS
      ${KItemModels_HEADERS}
  )

Each of the headers in the library are parsed to create the bindings, meaning we can then write this code:


  #!/usr/bin/env python
  #-*- coding: utf-8 -*-

  import sys

  sys.path.append(sys.argv[1])

  from PyQt5 import QtCore
  from PyQt5 import QtWidgets

  from PyKF5 import KItemModels

  app = QtWidgets.QApplication(sys.argv)

  stringListModel = QtCore.QStringListModel(
    ["Monday", "Tuesday", "Wednesday",
    "Thursday", "Friday", "Saturday", "Sunday"]);

  selectionProxy = KItemModels.KSelectionProxyModel()
  selectionProxy.setSourceModel(stringListModel)

  w = QtWidgets.QWidget()
  l = QtWidgets.QHBoxLayout(w)

  stringsView = QtWidgets.QTreeView()
  stringsView.setModel(stringListModel)
  stringsView.setSelectionMode(
    QtWidgets.QTreeView.ExtendedSelection)
  l.addWidget(stringsView)

  selectionProxy.setSelectionModel(
    stringsView.selectionModel())

  selectionView = QtWidgets.QTreeView()
  selectionView.setModel(selectionProxy)
  l.addWidget(selectionView)

  w.show()

  app.exec_()

and it just works with python 2 and 3.

Other libraries’ headers are more complex than KItemModels, so they have an extra rules file to maintain. The rules file is central to the design of this system in that it defines what to do when visiting each declaration in a C++ header file. It contains several small databases for handling declarations of containers, typedefs, methods and parameters, each of which may require special handling. The rules file for KCoreAddons is here.

The rules file contains entries to discard some method which can’t be called from python (in the case of heavily templated code for example) or it might replace the default implementation of the binding code with something else, in order to implement memory management correctly or for better integration with python built-in types.

Testing Automatically Generated Python Bindings

Each of the KDE Frameworks I’ve so far added bindings for gets a simple test file to verify that the binding can be loaded in the python interpreter (2 and 3). The TODO application in the screenshot is in the umbrella repo.

The binding generator itself also has tests to ensure that changes to it do not break generation for any framework. Actually extracting the important information using the cindex API is quite difficult and encounters many edge cases, like QStringLiteral (actually includes a lambda) in a parameter initializer.

Help Testing Automatically Generated Python Bindings

There is a call to action for anyone who wishes to help on the kde-bindings mailing list!

2 Responses to “Generating Python Bindings with Clang”

  1. Links 22/5/2016: Systemd 230, Debian Installer Alpha 6 | Techrights Says:

    […] Generating Python Bindings with Clang […]

  2. Michael Lutynski Says:

    This is so great to hear!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: