The Future of AST-Matching refactoring tools (EuroLLVM and ACCU)

April 30, 2019

I recently made a trip to LLVM in Brussels and ACCU in Bristol. It was a busy week. I gave a talk at both conferences on the topic of the future of AST Matchers-based refactoring.

As usual, the ‘hallway track’ also proved useful at both conferences, leading to round-table discussions at the LLVM conference with other interested contributors and getting to talk to other developers interested in refactoring tooling at ACCU.


The learning curve for AST-Matcher-based refactoring is currently too steep. Most C++ developers who are not already familiar with the internal Clang APIs need to invest a lot in order to learn how to make such bespoke tooling to improve and maintain their codebase.

The presentations both include demos of steps I’ve been taking to try to address these problems.

The first demo is of clang-query discovery features which aim to reduce the need to infer AST Matcher code by examining the Clang AST itself. I also showed the debugging features I am preparing to upstream to clang-query. Finally – in terms of demo content – I showed a Qt tool which can eliminate some of the additional difficulties and problems of long-developer-iteration-time.

The debugging features and the Qt tool were world exclusives at the LLVM conference (and at the ACCU conference because most people didn’t attend both 🙂 ). I hadn’t shown them to anyone else before, so I was quite happy the demos went well.


My 25 minute presentation to the LLVM developers tried to show that these changes can make mechanical refactoring more easily available to C++ developers.

The aim was to show the features to the LLVM community to

  1. illustrate the issues as I see them
  2. get some feedback about whether this is a good direction
  3. introduce myself for the sake of further code reviews (and collaborators). As this was my first LLVM conference, I am not already familiar with most of the attendees.

My 1.5 hour ACCU presentation is a far-less-rushed presentation of the same tools and a repetition of some of the content at code::dive 2018. In the ACCU presentation, the new demo content starts about 30 minutes in. This talk is the one to watch if you are interested in using mechanical refactoring on your own code.

Feedback was very positive from both talks, so I’m happy with that.

Qt Tooling

Earlier this year I refactored the clang AST dump functionality. It was previously implemented in one class, ASTDumper, which had the dual responsibilities of traversing the clang AST and creating a textual representation of it. I separated the textual output from the generic output independent traversal, which introduces the possibility of alternative output formats such as JSON.

Of course, given my KDE and Qt contribution history, I would only create a generic tree traversal class in order to implement QAbstractItemModel for it.

The demos show all of the features you would expect from a point-and-click refactoring tool including exploring, feature discovery, debugging with in-source feedback, live source updates, experimental refactoring etc.

Of course, all of this requires changes to Clang upstream (for example to add the debugging interface) which was the point of my visit to EuroLLVM. Hopefully, once enough of the changes are upstream, I’ll be able to open source the tool.

The idea as always is to hopefully have enough functionality in Clang itself that IDEs such as Qt-Creator, KDevelop and Visual Studio would be able to integrate these features using their own GUI APIs, making the simple tool I made obsolete anyway. I only made it for demo purposes.

This will take the mechanical refactoring workflow which is currently

and turn it into

You will still do the same things, but with much faster development iteration to achieve the same result.

There is even more that can be done to make the process of mechanical refactoring with clang easier and faster. We discussed some of that at EuroLLVM, and hopefully all the pieces can come together soon. Meanwhile I’ll be upstreaming some of this work, talking at NDC Oslo, and at my local meetup group on this topic.

Debugging Clang AST Matchers

April 16, 2019

Last week I flew to Brussels for EuroLLVM followed by Bristol for ACCU.

At both conferences I presented the work I’ve been doing to make it easier for regular C++ programmers to perform ‘mechanical’ bespoke refactoring using the clang ASTMatchers tooling. Each talk was prepared specifically for the particular audience at that conference, but both were very well received. The features I am working on require changes to the upstream Clang APIs in order to enable modern tooling, so I was traveling to EuroLLVM to try to build some buy-in and desire for those features.

I previously delivered a talk on the same topic about AST Matchers at code::dive 2018. This week I presented updates to the tools and features that I have worked on during the 6 months since.

One of the new features I presented is a method of debugging AST Matchers.

Part of the workflow of using AST Matchers is an iterative development process. For example, the developer wishes to find functions of a particular pattern, and creates and ever-more-complex matcher to find all desired cases without false-positives. As the matcher becomes more complex, it becomes difficult to determine why a particular function is not found as desired.

The debugger features I wrote for AST Matchers intend to solve that problem. It is now possible to create, remove and list breakpoints, and then enable debugger output to visualize the result of attempting to match at each location. A simple example of that is shown here.

When using a larger matcher it becomes obvious that the process of matching is short-circuited, meaning that the vertically-last negative match result is the cause of the overall failure to match the desired location. The typical workflow with the debugger is to insert break points on particular lines, and then remove surplus breakpoints which do not contribute useful output.

This feature is enabled by a new interface in the Clang AST Matchers, but the interface is also rich enough to implement some profiling of AST Matchers in the form of a hit counter.

Some matchers (and matcher sub-trees) are slower/more expensive to run than others. For example, running a matcher like `matchesName` on every AST node in a translation unit requires creation of a regular expression object, and comparing the name of each AST node with the regular expression. That may result in slower runtime than trimming the search tree by checking a parameter count first, for example.

Of course, the hit counter does not include timing output, but can give an indication of what might be relevant to change. Comparison of different trees of matchers can then be completed with a full clang-tidy check.

There is much more to say about both conferences and the tools that I demoed there, but that will be for a future log post. I hope this tool is useful and helps discover and debug AST Matchers!

Refactor with Clang Tooling at code::dive 2018

January 2, 2019

I delivered a talk about writing a refactoring tool with Clang Tooling at code::dive in November. It was uploaded to YouTube today:

The slides are available here and the code samples are here.

This was a fun talk to deliver as I got to demo some features which had never been seen by anyone before. For people who are already familiar with clang-tidy and clang-query, the interesting content starts about 15 minutes in. There I start to show new features in the clang-query interpreter command line.

The existing clang-query interpreter lacks many features which the replxx library provides, such as syntax highlighting and portable code completion:

It also allows scrolling through results to make a selection:

A really nice feature is value-based code-completion instead of type-based code completion. Existing code completion only completes candidates based on type-compatibility. It recognizes that a parameterCountIs() matcher can be used with a functionDecl() matcher for example. If the code completion already on the command line is sufficiently constrained so that there is only one result already, the code completion is able to complete candidates based on that one result node:

Another major problem with clang-query currently is that it is hard to know which parenthesis represents the closing of which matcher. The syntax highlighting of replxx help with this, along with a brace matching feature I wrote for it:

I’m working on upstreaming those features to replxx and Clang to make them available for everyone, but for now it is possible to experiment with some of the features on my Compiler Explorer instance on

I wrote about the AST-Matcher and Source Location/Source Range discovery features on my blog here since delivering the talk. I also wrote about Composing AST Matchers, which was part of the tips and tricks section of the talk. Over on the Visual C++ blog, I wrote about distributing the refactoring task among computers on the network using Icecream. My blogs on that platform can be seen in the Clang category.

All of that blog content is repeated in the code::dive presentation, but some people prefer to learn from conference videos instead of blogs, so this might help the content reach a larger audience. Let me know if there is more you would like to see about clang-query!

Composing AST Matchers in clang-tidy

November 20, 2018

When creating clang-tidy checks, it is common to extract parts of AST Matcher expressions to local variables. I expanded on this in a previous blog.

auto nonAwesomeFunction = functionDecl(

  , this);

  , this);

Use of such variables establishes an emergent extension API for re-use in the checks, or in multiple checks you create which share matcher requirements.

When attempting to match items inside a ForStmt for example, we might encounter the difference in the AST depending on whether braces are used or not.

#include <vector>

void foo()
    std::vector<int> vec;
    int c = 0;
    for (int i = 0; i < 100; ++i)

    for (int i = 0; i < 100; ++i) {

In this case, we wish to match the push_back method inside a ForStmt body. The body item might be a CompoundStmt or the CallExpr we wish to match. We can match both cases with the anyOf matcher.

auto pushbackcall = callExpr(callee(functionDecl(hasName("push_back"))));

    , this);

Having to list the pushbackcall twice in the matcher is suboptimal. We can do better by defining a new API function which we can use in AST Matcher expressions:

auto hasIgnoringBraces = [](auto const& Matcher)
    return anyOf(

With this in hand, we can simplify the original expression:

auto pushbackcall = callExpr(callee(functionDecl(hasName("push_back"))));

    , this);

This pattern of defining AST Matcher API using a lambda function finds use in other contexts. For example, sometimes we want to find and bind to an AST node if it is present, ignoring its absense if is not present.

For example, consider wishing to match struct declarations and match a copy constructor if present:

struct A

struct B
    B(B const&);

We can match the AST with the anyOf() and anything() matchers.

    , this);

This can be generalized into an optional() matcher:

auto optional = [](auto const& Matcher)
    return anyOf(

The anything() matcher matches, well, anything. It can also match nothing because of the fact that a matcher written inside another matcher matches itself.

That is, matchers such as


match ‘trivially’.

If a functionDecl() in fact binds to a method, then the derived type can be used in the matcher:


The optional matcher can be used as expected:

    , this);

Yet another problem writers of clang-tidy checks will find is that AST nodes CallExpr and CXXConstructExpr do not share a common base representing the ability to take expressions as arguments. This means that separate matchers are required for calls and constructions.

Again, we can solve this problem generically by creating a composition function:

auto callOrConstruct = [](auto const& Matcher)
    return expr(anyOf(

which reads as ‘an Expression which is any of a call expression or a construct expression’.

It can be used in place of either in matcher expressions:

        hasArgument(0, integerLiteral().bind("port_literal"))
    , this);

Creating composition functions like this is a very convenient way to simplify and create maintainable matchers in your clang-tidy checks. A recently published RFC on the topic of making clang-tidy checks easier to write proposes some other conveniences which can be implemented in this manner.

Future Developments in clang-query

November 11, 2018

Getting started – clang-tidy AST Matchers

Over the last few weeks I published some blogs on the Visual C++ blog about Clang AST Matchers. The series can be found here:

I am not aware of any similar series existing which covers creation of clang-tidy checks, and use of clang-query to inspect the Clang AST and assist in the construction of AST Matcher expressions. I hope the series is useful to anyone attempting to write clang-tidy checks. Several people have reported to me that they have previously tried and failed to create clang-tidy extensions, due to various issues, including lack of information tying it all together.

Other issues with clang-tidy include the fact that it relies on the “mental model” a compiler has of C++ source code, which might differ from the “mental model” of regular C++ developers. The compiler needs to have a very exact representation of the code, and needs to have a consistent design for the class hierarchy representing each standard-required feature. This leads to many classes and class hierarchies, and a difficulty in discovering what is relevant to a particular problem to be solved.

I noted several problems in those blog posts, namely:

  • clang-query does not show AST dumps and diagnostics at the same time<
  • Code completion does not work with clang-query on Windows
  • AST Matchers which are appropriate to use in contexts are difficult to discover
  • There is no tooling available to assist in discovery of source locations of AST nodes

Last week at code::dive in Wroclaw, I demonstrated tooling solutions to all of these problems. I look forward to video of that talk (and videos from the rest of the conference!) becoming available.

Meanwhile, I’ll publish some blog posts here showing the same new features in clang-query and clang-tidy.

clang-query in Compiler Explorer

Recent work by the Compiler Explorer maintainers adds the possibility to use source code tooling with the website. The compiler explorer contains new entries in a menu to enable a clang-tidy pane.

clang-tidy in Compiler Explorer

clang-tidy in Compiler Explorer

I demonstrated use of compiler explorer to use the clang-query tool at the code::dive conference, building upon the recent work by the compiler explorer developers. This feature will get upstream in time, but can be used with my own AWS instance for now. This is suitable for exploration of the effect that changing source code has on match results, and orthogonally, the effect that changing the AST Matcher has on the match results. It is also accessible via

It is important to remember that Compiler Explorer is running clang-query in script mode, so it can process multiple let and match calls for example. The new command set print-matcher true helps distinguish the output from the matcher which causes the output. The help command is also available with listing of the new features.

The issue of clang-query not printing both diagnostic information and AST information at the same time means that users of the tool need to alternate between writing

set output diag


set output dump

to access the different content. Recently, I committed a change to make it possible to enable both output and diag output from clang-query at the same time. New commands follow the same structure as the set output command:

enable output detailed-ast
disable output detailed-ast

The set output command remains as an “exclusive” setting to enable only one output feature and disable all others.

Dumping possible AST Matchers

This command design also enables the possibility of extending the features which clang-query can output. Up to now, developers of clang-tidy extensions had to inspect the AST corresponding to their source code using clang-query and then use that understanding of the AST to create an AST Matcher expression.

That mapping to and from the AST “mental model” is not necessary. New features I am in the process of upstreaming to clang-query enable the output of AST Matchers which may be used with existing bound AST nodes. The command

enable output matcher

causes clang-query to print out all matcher expressions which can be combined with the bound node. This cuts out the requirement to dump the AST in such cases.

Inspecting the AST is still useful as a technique to discover possible AST Matchers and how they correspond to source code. For example if the functionDecl() matcher is already known and understood, it can be dumped to see that function calls are represented by the CallExpr in the Clang AST. Using the callExpr() AST Matcher and dumping possible matchers to use with it leads to the discovery that callee(functionDecl()) can be used to determine particulars of the function being called. Such discoveries are not possible by only reading AST output of clang-query.

Dumping possible Source Locations

The other important discovery space in creation of clang-tidy extensions is that of Source Locations and Source Ranges. Developers creating extensions must currently rely on the documentation of the Clang AST to discover available source locations which might be relevant. Usually though, developers have the opposite problem. They have source code, and they want to know how to access a source location from the AST node which corresponds semantically to that line and column in the source.

It is important to make use a semantically relevant source location in order to make reliable tools which refactor at scale and without human intervention. For example, a cursory inspection of the locations available from a FunctionDecl AST node might lead to the belief that the return type is available at the getBeginLoc() of the node.

However, this is immediately challenged by the C++11 trailing return type feature, where the actual return type is located at the end. For a semanticallly correct location, you must currently use


It should be possible to use getReturnTypeSourceRange(), but a bug in clang prevents that as it does not appreciate the trailing return types feature.

Once again, my new output feature of clang-query presents a solution to this discovery problem. The command

enable output srcloc

causes clang-query to output the source locations by accessor and caret corresponding to the source code for each of the bound nodes. By inspecting that output, developers of clang-tidy extensions can discover the correct expression (usually via the clang::TypeLoc heirarchy) corresponding to the source code location they are interested in refactoring.

Next Steps

I have made many more modifications to clang-query which I am in the process of upstreaming. My Compiler explorer instance is listed as the ‘clang-query-future’ tool, while the clang-query-trunk tool runs the current trunk version of clang-query. Both can be enabled for side-by-side comparison of the future clang-query with the exising one.

API Changes in Clang

September 13, 2018

I’ve started contributing to Clang, in the hope that I can improve the API for tooling. This will eventually mean changes to the C++ API of Clang, the CMake buildsystem, and new features in the tooling. Hopefully I’ll remember to blog about changes I make.

The Department of Redundancy Department

I’ve been implementing custom clang-tidy checks and have become quite familiar with the AST Node API. Because of my background in Qt, I was immediately disoriented by some API inconsistency. Certain API classes had both getStartLoc and getLocStart methods, as well as both getEndLoc and getLocEnd etc. The pairs of methods return the same content, so at least one set of them is redundant.

I’m used to working on stable library APIs, but Clang is different in that it offers no API stability guarantees at all. As an experiment, we staggered the introduction of new API and removal of old API. I ended up replacing the getStartLoc and getLocStart methods with getBeginLoc for consistency with other classes, and replaced getLocEnd with getEndLoc. Both old and new APIs are in the Clang 7.0.0 release, but the old APIs are already removed from Clang master. Users of the old APIs should port to the new ones at the next opportunity as described here.

Wait a minute, Where’s me dump()er?

Clang AST classes have a dump() method which is very useful for debugging. Several tools shipped with Clang are based on dumping AST nodes.

The SourceLocation type also provides a dump() method which outputs the file, line and column corresponding to a location. The problem with it though has always been that it does not include a newline at the end of the output, so the output gets lost in noise. This 2013 video tutorial shows the typical developer experience using that dump method. I’ve finally fixed that in Clang, but it did not make it into Clang 7.0.0.

In the same vein, I also added a dump() method to the SourceRange class. This prints out locations in the an angle-bracket format which shows only what changed between the beginning and end of the range.

Let it bind

When writing clang-tidy checks using AST Matchers, it is common to factor out intermediate variables for re-use or for clarity in the code.

auto valueMethod = cxxMethodDecl(hasName("value"));

clang-query has an analogous way to create intermediate matcher variables, but binding to them did not work. As of my recent commit, it is possible to create matcher variables and bind them later in a matcher:

let valueMethod cxxMethodDecl(hasName("value"))
match valueMethod.bind("methodDecl")
match callExpr(callee(valueMethod.bind("methodDecl"))).bind("methodCall")

Preload your Queries

Staying on the same topic, I extended clang-query with a --preload option. This allows starting clang-query with some commands already invoked, and then continue using it as a REPL:

bash$ cat cmds.txt
let valueMethod cxxMethodDecl(hasName("value"))

bash$ clang-query --preload cmds.txt somefile.cpp
clang-query> match valueMethod.bind("methodDecl")

Match #1:

somefile.cpp:4:2: note: "methodDecl" binds here
        void value();

1 match.

Previously, it was only possible to run commands from a file without also creating a REPL using the -c option. The --preload option with the REPL is useful when experimenting with matchers and having to restart clang-query regularly. This happens a lot when modifying code to examine changes to AST nodes.


Embracing Modern CMake

November 5, 2017

I spoke at the ACCU conference in April 2017 on the topic of Embracing Modern CMake. The talk was very well attended and received, but was unfortunately not recorded at the event. In September I gave the talk again at the Dublin C++ User Group, so that it could be recorded for the internet.

The slides are available here. The intention of the talk was to present a ‘gathered opinion’ about what Modern CMake is and how it should be written. I got a lot of input from CMake users on reddit which informed some of the content of the talk.

Much of the information about how to write Modern CMake is available in the CMake documentation, and there are many presentations these days advocating the use of modern patterns and commands, discouraging use of older commands. Two other talks from this year that I’m aware of and which are popular are:

It’s very pleasing to see so many well-received and informative talks about something that I worked so hard on designing (together with Brad King) and implementing so many years ago.

One of the points which I tried to labor a bit in my talk was just how old ‘Modern’ CMake is. I recently was asked in private email about the origin and definition of the term, so I’ll try to reproduce that information here.

I coined the term “Modern CMake” while preparing for Meeting C++ 2013, where I presented on the topic and the developments in CMake in the preceding years. Unfortunately (this happens to me a lot with CMake), the talk was not recorded, but I wrote a blog post with the slides and content. The slides are no longer on the KDAB website, but can be found here. Then already in 2013, the simple example with Qt shows the essence of Modern CMake:

find_package(Qt5Widgets 5.2 REQUIRED)

add_executable(myapp main.cpp)
target_link_libraries(myapp Qt5::Widgets)

Indeed, the first terse attempt at a definition of “Modern CMake” and first public appearance of the term with its current meaning was when I referred to it as approximately “CMake with usage requirements”. That’s when the term gained a capitalized ‘M’ and its current meaning and then started to gain traction.

The first usage I found of “Modern CMake” in private correspondence was March 13 2012 in an email exchange with Alex Neundorf about presenting together on the topic at a KDE conference:

Hi Alex

Are you planning on going to Talinn for Akademy this year? I was thinking about sumitting a talk along the lines of Qt5, KF5, CMake (possibly along the lines of the discussion of ‘modern CMake’ we had before with Clinton, and what KDE CMake files could look like as a result).

I thought maybe we should coordinate so either we don’t submit overlapping proposals, or we can submit a joint talk.



The “discussion with Clinton” was probably this thread and the corresponding thread on the cmake mailing list where I started to become involved in what would become Modern CMake over the following years.

The talk was unfortunately not accepted to the conference, but here’s the submission:

Speakers: Stephen Kelly, Alexander Neundorf
Title: CMake in 2012 – Modernizing CMake usage in Qt5 and KDE Frameworks 5
Duration: 45 minutes

KDE Frameworks 5 (KF5) will mark the start of a new chapter in the history of KDE and of the KDE platform. Starting from a desire to make our developments more easy to use by 3rd parties and ‘Qt-only’ developers, the effort to create KF5 is partly one of embracing and extending upstreams to satisfy the needs of the KDE Platform, to enable a broadening of the user base of our technology.

As it is one of our most important upstreams, and as the tool we use to build our software, KDE relies on CMake to provide a high standard of quality and features. Throughout KDE 4 times, KDE has added extensions to CMake which we consider useful to all developers using Qt and C++. To the extent possible, we are adding those features upstream to CMake. Together with those features, we are providing feedback from 6 years of experience with CMake to ensure it continues to deliver an even more awesome build experience for at least the next 6 years. Qt5 and KF5 will work together with CMake in ways that were not possible in KDE 4 times.

The presentation will discuss the various aspects of the KDE buildsystem planned for KF5, both hidden and visible to the developer. These aspects will include the CMake automoc feature, the role of CMake configuration files, and how a target orientated and consistency driven approach could change how CMake will be used in the future.

There is a lot to recognize there in what has since come to pass and become common in Modern CMake usage, in particular the “target orientated and consistency driven approach” which is the core characteristic of Modern CMake.

Boost dependencies and bcp

August 21, 2016

Recently I generated diagrams showing the header dependencies between Boost libraries, or rather, between various Boost git repositories. Diagrams showing dependencies for each individual Boost git repo are here along with dot files for generating the images.

The monster diagram is here:

Edges and Incidental Modules and Packages

The directed edges in the graphs represent that a header file in one repository #includes a header file in the other repository. The idea is that, if a packager wants to package up a Boost repo, they can’t assume anything about how the user will use it. A user of Boost.ICL can choose whether ICL will use Boost.Container or not by manipulating the ICL_USE_BOOST_MOVE_IMPLEMENTATION preprocessor macro. So, the packager has to list Boost.Container as some kind of dependency of Boost.ICL, so that when the package manager downloads the boost-icl package, the boost-container package is automatically downloaded too. The dependency relationship might be a ‘suggests’ or ‘recommends’, but the edge will nonetheless exist somehow.

In practice, packagers do not split Boost into packages like that. At least for debian packages they split compiled static libraries into packages such as libboost-serialization1.58, and put all the headers (all header-only libraries) into a single package libboost1.58-dev. Perhaps the reason for packagers putting it all together is that there is little value in splitting the header-only repository content in the monolithic Boost from each other if it will all be packaged anyway. Or perhaps the sheer number of repositories makes splitting impractical. This is in contrast to KDE Frameworks, which does consider such edges and dependency graph size when determining where functionality belongs. Typically KDE aims to define the core functionality of a library on its own in a loosely coupled way with few dependencies, and then add integration and extension for other types in higher level libraries (if at all).

Another feature of my diagrams is that repositories which depend circularly on each other are grouped together in what I called ‘incidental modules‘. The name is inspired by ‘incidental data structures’ which Sean Parent describes in detail in one of his ‘Better Code’ talks. From a packager point of view, the Boost.MPL repo and the Boost.Utility repo are indivisible because at least one header of each repo includes at least one header of the other. That is, even if packagers wanted to split Boost headers in some way, the ‘incidental modules’ would still have to be grouped together into larger packages.

As far as I am aware such circular dependencies don’t fit with Standard C++ Modules designs or the design of Clang Modules, but that part of C++ would have to become more widespread before Boost would consider their impact. There may be no reason to attempt to break these ‘incidental modules’ apart if all that would do is make some graphs nicer, and it wouldn’t affect how Boost is packaged.

My script for generating the dependency information is simply grepping through the include/ directory of each repository and recording the #included files in other repositories. This means that while we know Boost.Hana can be used stand-alone, if a packager simply packages up the include/boost/hana directory, the result will have dependencies on parts of Boost because Hana includes code for integration with existing Boost code.

Dependency Analysis and Reduction

One way of defining a Boost library is to consider the group of headers which are gathered together and documented together to be a library (there are other ways which some in Boost prefer – it is surprisingly fuzzy). That is useful for documentation at least, but as evidenced it appears to not be useful from a packaging point of view. So, are these diagrams useful for anything?

While Boost header-only libraries are not generally split in standard packaging systems, the bcp tool is provided to allow users to extract a subset of the entire Boost distribution into a user-specified location. As far as I know, the tool scans header files for #include directives (ignoring ifdefs, like a packager would) and gathers together all of the transitively required files. That means that these diagrams are a good measure of how much stuff the bcp tool will extract.

Note also that these edges do not contribute time to your slow build – reducing edges in the graphs by moving files won’t make anything faster. Rewriting the implementation of certain things might, but that is not what we are talking about here.

I can run the tool to generate a usable Boost.ICL which I can easily distribute. I delete the docs, examples and tests from the ICL directory because they make up a large chunk of the size. Such a ‘subset distribution’ doesn’t need any of those. I also remove 3.5M of preprocessed files from MPL. I then need to define BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS when compiling, which is easy and explained at the end:

$ bcp --boost=$HOME/dev/src/boost icl myicl
$ rm -rf boostdir/libs/icl/{doc,test,example}
$ rm -rf boostdir/boost/mpl/aux_/preprocessed
$ du -hs myicl/
15M     myicl/

Ok, so it’s pretty big. Looking at the dependency diagram for Boost.ICL you can see an arrow to the ‘incidental spirit’ module. Looking at the Boost.Spirit dependency diagram you can see that it is quite large.

Why does ICL depend on ‘incidental spirit’? Can that dependency be removed?

For those ‘incidental modules’, I selected one of the repositories within the group and named the group after that one repository. Too see why ICL depends on ‘incidental spirit’, we have to examine all 5 of the repositories in the group to check if it is the one responsible for the dependency edge.

boost/libs/icl$ git grep -Pl -e include --and \
  -e "thread|spirit|pool|serial|date_time" include/

Formatting wide terminal output is tricky in a blog post, so I had to make some compromises in the output here. Those ICL headers are including Boost.DateTime headers.

I can further see that gregorian.hpp and ptime.hpp are ‘leaf’ files in this analysis. Other files in ICL do not include them.

boost/libs/icl$ git grep -l gregorian include/
boost/libs/icl$ git grep -l ptime include/

As it happens, my ICL-using code also does not need those files. I’m only using icl::interval_set<double> and icl::interval_map<double>. So, I can simply delete those files.

boost/libs/icl$ git grep -l -e include \
  --and -e date_time include/boost/icl/ | xargs rm

and run the bcp tool again.

$ bcp --boost=$HOME/dev/src/boost icl myicl
$ rm -rf myicl/libs/icl/{doc,test,example}
$ rm -rf myicl/boost/mpl/aux_/preprocessed
$ du -hs myicl/
12M     myicl/

I’ve saved 3M just by understanding the dependencies a bit. Not bad!

Mostly the size difference is accounted for by no longer extracting boost::mpl::vector, and secondly the Boost.DateTime headers themselves.

The dependencies in the graph are now so few that we can consider them and wonder why they are there and can they be removed. For example, there is a dependency on the Boost.Container repository. Why is that?

include/boost/icl$ git grep -C2 -e include \
   --and -e boost/container
#   include <boost/container/set.hpp>
#   include <set>

#   include <boost/container/map.hpp>
#   include <boost/container/set.hpp>
#   include <map>

#   include <boost/container/set.hpp>
#   include <set>

So, Boost.Container is only included if the user defines ICL_USE_BOOST_MOVE_IMPLEMENTATION, and otherwise not. If we were talking about C++ code here we might consider this a violation of the Interface Segregation Principle, but we are not, and unfortunately the realities of the preprocessor mean this kind of thing is quite common.

I know that I’m not defining that and I don’t need Boost.Container, so I can hack the code to remove those includes, eg:

index 6f3c851..cf22b91 100644
--- a/include/boost/icl/map.hpp
+++ b/include/boost/icl/map.hpp
@@ -12,12 +12,4 @@ Copyright (c) 2007-2011:
-#   include <boost/container/map.hpp>
-#   include <boost/container/set.hpp>
 #   include <map>
 #   include <set>
-#else // Default for implementing containers
-#   include <map>
-#   include <set>

This and following steps don’t affect the filesystem size of the result. However, we can continue to analyze the dependency graph.

I can break apart the ‘incidental fusion’ module by deleting the iterator/zip_iterator.hpp file, removing further dependencies in my custom Boost.ICL distribution. I can also delete the iterator/function_input_iterator.hpp file to remove the dependency on Boost.FunctionTypes. The result is a graph which you can at least reason about being used in an interval tree library like Boost.ICL, quite apart from our starting point with that library.

You might shudder at the thought of deleting zip_iterator if it is an essential tool to you. Partly I want to explore in this blog post what will be needed from Boost in the future when we have zip views from the Ranges TS or use the existing ranges-v3 directly, for example. In that context, zip_iterator can go.

Another feature of the bcp tool is that it can scan a set of source files and copy only the Boost headers that are included transitively. If I had used that, I wouldn’t need to delete the ptime.hpp or gregorian.hpp etc because bcp wouldn’t find them in the first place. It would still find the Boost.Container etc includes which appear in the ICL repository however.

In this blog post, I showed an alternative approach to the bcp --scan attempt at minimalism. My attempt is to use bcp to export useful and as-complete-as-possible libraries. I don’t have a lot of experience with bcp, but it seems that in scanning mode I would have to re-run the tool any time I used an ICL header which I had not used before. With the modular approach, it would be less-frequently necessary to run the tool (only when directly using a Boost repository I hadn’t used before), so it seemed an approach worth exploring the limitations of.

Examining Proposed Standard Libraries

We can also examine other Boost repositories, particularly those which are being standardized by newer C++ standards because we know that any, variant and filesystem can be implemented with only standard C++ features and without Boost.

Looking at Boost.Variant, it seems that use of the Boost.Math library makes that graph much larger. If we want Boost.Variant without all of that Math stuff, one thing we can choose to do is copy the one math function that Variant uses, static_lcm, into the Variant library (or somewhere like Boost.Core or Boost.Integer for example). That does cause a significant reduction in the dependency graph.

Further, I can remove the hash_variant.hpp file to remove the Boost.Functional dependency:

I don’t know if C++ standardized variant has similar hashing functionality or how it is implemented, but it is interesting to me how it affects the graph.

Using a bcp-extracted library with Modern CMake

After extracting a library or set of libraries with bcp, you might want to use the code in a CMake project. Here is the modern way to do that:

add_library(boost_mpl INTERFACE)
target_compile_definitions(boost_mpl INTERFACE
target_include_directories(boost_mpl INTERFACE 

add_library(boost_icl INTERFACE)
target_link_libraries(boost_icl INTERFACE boost_mpl)
target_include_directories(boost_icl INTERFACE 
add_library(boost::icl ALIAS boost_icl)

Boost ships a large chunk of preprocessed headers for various compilers, which I mentioned above. The reasons for that are probably historical and obsolete, but they will remain and they are used by default when using GCC and that will not change. To diverge from that default it is necessary to set the BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS preprocessor macro.

By defining an INTERFACE boost_mpl library and setting its INTERFACE target_compile_definitions, any user of that library gets that magic BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS define when compiling its sources.

MPL is just an internal implementation detail of ICL though, so I won’t have any of my CMake targets using MPL directly. Instead I additionally define a boost_icl INTERFACE library which specifies an INTERFACE dependency on boost_mpl with target_link_libraries.

The last ‘modern’ step is to define an ALIAS library. The alias name is boost::icl and it aliases the boost_icl library. To CMake, the following two commands generate an equivalent buildsystem:

target_link_libraries(myexe boost_icl)
target_link_libraries(myexe boost::icl)

Using the ALIAS version has a different effect however: If the boost::icl target does not exist an error will be issued at CMake time. That is not the case with the boost_icl version. It makes sense to use target_link_libraries with targets with :: in the name and ALIAS makes that possible for any library.

Opt-in header-only libraries with CMake

August 9, 2016

Using a C++ library, particularly a 3rd party one, can be complicated affair. Library binaries compiled on Windows/OSX/Linux can not simply be copied over to another platform and used there. Linking works differently, compilers bundle different code into binaries on each platform etc.

This is not an insurmountable problem. Libraries like Qt distribute dynamically compiled binaries for major platforms and other libraries have comparable solutions.

There is a category of libraries which considers the portable binaries issue to be a terminal one. Boost is a widespread source of many ‘header only’ libraries, which don’t require a user to link to a particular platform-compatible library binary. There are also many other examples of such ‘header only’ libraries.

Recently there was a blog post describing an example library which can be built as a shared library, or as a static library, or used directly as a ‘header only’ library which doesn’t require the user to link against anything to use the library. The claim is that it is useful for libraries to provide users the option of using a library as a ‘header only’ library and adding preprocessor magic to make that possible.

However, there is yet a fourth option, and that is for the consumer to compile the source files of the library themselves. This has the
advantage that the .cpp file is not #included into every compilation unit, but still avoids the platform-specific library binary.

I decided to write a CMake buildsystem which would achieve all of that for a library. I don’t have an opinion on whether good idea in general for libraries to do things like this, but if people want to do it, it should be easy as possible.

Additionally, of course, the CMake GenerateExportHeader module should be used, but I didn’t want to change the source from Vittorio so much.

The CMake code below compiles the library in several ways and installs it to a prefix which is suitable for packaging:

cmake_minimum_required(VERSION 3.3)


# define the library


add_library(library_static STATIC ${library_srcs})
add_library(library_shared SHARED ${library_srcs})

add_library(library_iface INTERFACE)

add_library(library_srcs INTERFACE)
target_sources(library_srcs INTERFACE

# install and export the library

  EXPORT library_targets
install(EXPORT library_targets
  NAMESPACE example_lib::
  DESTINATION lib/cmake/example_lib

install(FILES example_lib-config.cmake
  DESTINATION lib/cmake/example_lib

This blog post is not a CMake introduction, so to see what all of those commands are about start with the cmake-buildsystem and cmake-packages documentation.

There are 4 add_library calls. The first two serve the purpose of building the library as a shared library and then as a static library.

The next two are INTERFACE libraries, a concept I introduced in CMake 3.0 when it looked like Boost might use CMake. The INTERFACE target can be used to specify header-only libraries because they specify usage requirements for consumers to use, such as include directories and compile definitions.

The library_iface library functions as described in the blog post from Vittorio, in that users of that library will be built with LIBRARY_HEADER_ONLY and will therefore #include the .cpp files.

The library_srcs library causes the consumer to compile the .cpp files separately.

A consumer of a library like this would then look like:

cmake_minimum_required(VERSION 3.3)


find_package(example_lib REQUIRED)


## uncomment only one of these!
# target_link_libraries(myexe 
#     example_lib::library_static)
# target_link_libraries(myexe
#     example_lib::library_shared)
# target_link_libraries(myexe
#     example_lib::library_iface)

So, it is up to the consumer how they consume the library, and they determine that by using target_link_libraries to specify which one they depend on.

Generating Python Bindings with Clang

May 18, 2016

Python Bindings

Python provides a C API to define libraries which can be imported into a python environment. That python C interface is often used to provide bindings to existing libraries written in C++ and there are multiple existing technologies for that such as PyQt, PySide, Boost.Python and pybind.

pybind seems to be a more modern implementation of what Boost.Python provides. To create a binding, C++ code is written describing how the target API is to be exposed. That gives the flexibility of defining precisely which methods get exposed to the python environment and which do not.

PyQt and PySide are also similar in that they require the maintenance of a binding specification. In the case of PySide, that is an XML file, and in the case of PyQt that is a DSL which is similar to a C++ header. The advantage both of these systems have for Qt-based code is that they ship with bindings for Qt, relieving the binding author of the requirement to create those bindings.

PyKF5 application using KItemModels and KWidgetsAddons

PyKF5 application using KItemModels and KWidgetsAddons

Generated Python Bindings

For libraries which are large library collections, such as Qt5 and KDE Frameworks 5, manual binding specification soon becomes a task which is not suitable for manual creation by humans, so it is common to have the bindings generated from the C++ headers themselves.

The way that KDE libraries and bindings were organized in the time of KDE4 led to PyQt-based bindings being generated in a semi-automated process and then checked into the SCM repository and maintained there. The C++ header parsing tools used to do that were written before standardization of C++11 and have not kept pace with compilers adding new language features, and C++ headers using them.

Automatically Generated Python Bindings

It came as a surprise to me that no bindings had been completed for the KDE Frameworks 5 libraries. An email from Shaheed Hague about a fresh approach to generating bindings using clang looked promising, but he was hitting problems with linking binding code to the correct shared libraries and generally figuring out what the end-goal looks like. Having used clang APIs before, and having some experience with CMake, I decided to see what I could do to help.

Since then I’ve been helping get the bindings generator into something of a final form for the purposes of KDE Frameworks, and any other Qt-based or even non-Qt-based libraries. The binding generator uses the clang python cindex API to parse the headers of each library and generate a set of sip files, which are then processed to create the bindings. As the core concept of the generator is simply ‘use clang to parse the headers’ it can be adapted to other binding technologies in the future (such as PySide). PyQt-based bindings are the current focus because that fills a gap between what was provided with KDE4 and what is provided by KDE Frameworks 5.

All of that is internal though and doesn’t appear in the buildsystem of any framework. As far as each buildsystem is concerned, a single CMake macro is used to enable the build of python (2 and 3) bindings for a KDE Frameworks library:

    TARGET KF5::ItemModels
    MODULENAME KItemModels

Each of the headers in the library are parsed to create the bindings, meaning we can then write this code:

  #!/usr/bin/env python
  #-*- coding: utf-8 -*-

  import sys


  from PyQt5 import QtCore
  from PyQt5 import QtWidgets

  from PyKF5 import KItemModels

  app = QtWidgets.QApplication(sys.argv)

  stringListModel = QtCore.QStringListModel(
    ["Monday", "Tuesday", "Wednesday",
    "Thursday", "Friday", "Saturday", "Sunday"]);

  selectionProxy = KItemModels.KSelectionProxyModel()

  w = QtWidgets.QWidget()
  l = QtWidgets.QHBoxLayout(w)

  stringsView = QtWidgets.QTreeView()


  selectionView = QtWidgets.QTreeView()


and it just works with python 2 and 3.

Other libraries’ headers are more complex than KItemModels, so they have an extra rules file to maintain. The rules file is central to the design of this system in that it defines what to do when visiting each declaration in a C++ header file. It contains several small databases for handling declarations of containers, typedefs, methods and parameters, each of which may require special handling. The rules file for KCoreAddons is here.

The rules file contains entries to discard some method which can’t be called from python (in the case of heavily templated code for example) or it might replace the default implementation of the binding code with something else, in order to implement memory management correctly or for better integration with python built-in types.

Testing Automatically Generated Python Bindings

Each of the KDE Frameworks I’ve so far added bindings for gets a simple test file to verify that the binding can be loaded in the python interpreter (2 and 3). The TODO application in the screenshot is in the umbrella repo.

The binding generator itself also has tests to ensure that changes to it do not break generation for any framework. Actually extracting the important information using the cindex API is quite difficult and encounters many edge cases, like QStringLiteral (actually includes a lambda) in a parameter initializer.

Help Testing Automatically Generated Python Bindings

There is a call to action for anyone who wishes to help on the kde-bindings mailing list!