Archive for the ‘Events’ Category

The Future of AST-Matching refactoring tools (EuroLLVM and ACCU)

April 30, 2019

I recently made a trip to LLVM in Brussels and ACCU in Bristol. It was a busy week. I gave a talk at both conferences on the topic of the future of AST Matchers-based refactoring.

As usual, the ‘hallway track’ also proved useful at both conferences, leading to round-table discussions at the LLVM conference with other interested contributors and getting to talk to other developers interested in refactoring tooling at ACCU.

Presentations

The learning curve for AST-Matcher-based refactoring is currently too steep. Most C++ developers who are not already familiar with the internal Clang APIs need to invest a lot in order to learn how to make such bespoke tooling to improve and maintain their codebase.

The presentations both include demos of steps I’ve been taking to try to address these problems.

The first demo is of clang-query discovery features which aim to reduce the need to infer AST Matcher code by examining the Clang AST itself. I also showed the debugging features I am preparing to upstream to clang-query. Finally – in terms of demo content – I showed a Qt tool which can eliminate some of the additional difficulties and problems of long-developer-iteration-time.

The debugging features and the Qt tool were world exclusives at the LLVM conference (and at the ACCU conference because most people didn’t attend both 🙂 ). I hadn’t shown them to anyone else before, so I was quite happy the demos went well.

Videos

My 25 minute presentation to the LLVM developers tried to show that these changes can make mechanical refactoring more easily available to C++ developers.

The aim was to show the features to the LLVM community to

  1. illustrate the issues as I see them
  2. get some feedback about whether this is a good direction
  3. introduce myself for the sake of further code reviews (and collaborators). As this was my first LLVM conference, I am not already familiar with most of the attendees.

My 1.5 hour ACCU presentation is a far-less-rushed presentation of the same tools and a repetition of some of the content at code::dive 2018. In the ACCU presentation, the new demo content starts about 30 minutes in. This talk is the one to watch if you are interested in using mechanical refactoring on your own code.

Feedback was very positive from both talks, so I’m happy with that.

Qt Tooling

Earlier this year I refactored the clang AST dump functionality. It was previously implemented in one class, ASTDumper, which had the dual responsibilities of traversing the clang AST and creating a textual representation of it. I separated the textual output from the generic output independent traversal, which introduces the possibility of alternative output formats such as JSON.

Of course, given my KDE and Qt contribution history, I would only create a generic tree traversal class in order to implement QAbstractItemModel for it.

The demos show all of the features you would expect from a point-and-click refactoring tool including exploring, feature discovery, debugging with in-source feedback, live source updates, experimental refactoring etc.

Of course, all of this requires changes to Clang upstream (for example to add the debugging interface) which was the point of my visit to EuroLLVM. Hopefully, once enough of the changes are upstream, I’ll be able to open source the tool.

The idea as always is to hopefully have enough functionality in Clang itself that IDEs such as Qt-Creator, KDevelop and Visual Studio would be able to integrate these features using their own GUI APIs, making the simple tool I made obsolete anyway. I only made it for demo purposes.

This will take the mechanical refactoring workflow which is currently

and turn it into

You will still do the same things, but with much faster development iteration to achieve the same result.

There is even more that can be done to make the process of mechanical refactoring with clang easier and faster. We discussed some of that at EuroLLVM, and hopefully all the pieces can come together soon. Meanwhile I’ll be upstreaming some of this work, talking at NDC Oslo, and at my local meetup group on this topic.

Debugging Clang AST Matchers

April 16, 2019

Last week I flew to Brussels for EuroLLVM followed by Bristol for ACCU.

At both conferences I presented the work I’ve been doing to make it easier for regular C++ programmers to perform ‘mechanical’ bespoke refactoring using the clang ASTMatchers tooling. Each talk was prepared specifically for the particular audience at that conference, but both were very well received. The features I am working on require changes to the upstream Clang APIs in order to enable modern tooling, so I was traveling to EuroLLVM to try to build some buy-in and desire for those features.

I previously delivered a talk on the same topic about AST Matchers at code::dive 2018. This week I presented updates to the tools and features that I have worked on during the 6 months since.

One of the new features I presented is a method of debugging AST Matchers.

Part of the workflow of using AST Matchers is an iterative development process. For example, the developer wishes to find functions of a particular pattern, and creates and ever-more-complex matcher to find all desired cases without false-positives. As the matcher becomes more complex, it becomes difficult to determine why a particular function is not found as desired.

The debugger features I wrote for AST Matchers intend to solve that problem. It is now possible to create, remove and list breakpoints, and then enable debugger output to visualize the result of attempting to match at each location. A simple example of that is shown here.

When using a larger matcher it becomes obvious that the process of matching is short-circuited, meaning that the vertically-last negative match result is the cause of the overall failure to match the desired location. The typical workflow with the debugger is to insert break points on particular lines, and then remove surplus breakpoints which do not contribute useful output.

This feature is enabled by a new interface in the Clang AST Matchers, but the interface is also rich enough to implement some profiling of AST Matchers in the form of a hit counter.

Some matchers (and matcher sub-trees) are slower/more expensive to run than others. For example, running a matcher like `matchesName` on every AST node in a translation unit requires creation of a regular expression object, and comparing the name of each AST node with the regular expression. That may result in slower runtime than trimming the search tree by checking a parameter count first, for example.

Of course, the hit counter does not include timing output, but can give an indication of what might be relevant to change. Comparison of different trees of matchers can then be completed with a full clang-tidy check.

There is much more to say about both conferences and the tools that I demoed there, but that will be for a future log post. I hope this tool is useful and helps discover and debug AST Matchers!

Refactor with Clang Tooling at code::dive 2018

January 2, 2019

I delivered a talk about writing a refactoring tool with Clang Tooling at code::dive in November. It was uploaded to YouTube today:

The slides are available here and the code samples are here.

This was a fun talk to deliver as I got to demo some features which had never been seen by anyone before. For people who are already familiar with clang-tidy and clang-query, the interesting content starts about 15 minutes in. There I start to show new features in the clang-query interpreter command line.

The existing clang-query interpreter lacks many features which the replxx library provides, such as syntax highlighting and portable code completion:

It also allows scrolling through results to make a selection:

A really nice feature is value-based code-completion instead of type-based code completion. Existing code completion only completes candidates based on type-compatibility. It recognizes that a parameterCountIs() matcher can be used with a functionDecl() matcher for example. If the code completion already on the command line is sufficiently constrained so that there is only one result already, the code completion is able to complete candidates based on that one result node:

Another major problem with clang-query currently is that it is hard to know which parenthesis represents the closing of which matcher. The syntax highlighting of replxx help with this, along with a brace matching feature I wrote for it:

I’m working on upstreaming those features to replxx and Clang to make them available for everyone, but for now it is possible to experiment with some of the features on my Compiler Explorer instance on ce.steveire.com.

I wrote about the AST-Matcher and Source Location/Source Range discovery features on my blog here since delivering the talk. I also wrote about Composing AST Matchers, which was part of the tips and tricks section of the talk. Over on the Visual C++ blog, I wrote about distributing the refactoring task among computers on the network using Icecream. My blogs on that platform can be seen in the Clang category.

All of that blog content is repeated in the code::dive presentation, but some people prefer to learn from conference videos instead of blogs, so this might help the content reach a larger audience. Let me know if there is more you would like to see about clang-query!

Future Developments in clang-query

November 11, 2018

Getting started – clang-tidy AST Matchers

Over the last few weeks I published some blogs on the Visual C++ blog about Clang AST Matchers. The series can be found here:

I am not aware of any similar series existing which covers creation of clang-tidy checks, and use of clang-query to inspect the Clang AST and assist in the construction of AST Matcher expressions. I hope the series is useful to anyone attempting to write clang-tidy checks. Several people have reported to me that they have previously tried and failed to create clang-tidy extensions, due to various issues, including lack of information tying it all together.

Other issues with clang-tidy include the fact that it relies on the “mental model” a compiler has of C++ source code, which might differ from the “mental model” of regular C++ developers. The compiler needs to have a very exact representation of the code, and needs to have a consistent design for the class hierarchy representing each standard-required feature. This leads to many classes and class hierarchies, and a difficulty in discovering what is relevant to a particular problem to be solved.

I noted several problems in those blog posts, namely:

  • clang-query does not show AST dumps and diagnostics at the same time<
  • Code completion does not work with clang-query on Windows
  • AST Matchers which are appropriate to use in contexts are difficult to discover
  • There is no tooling available to assist in discovery of source locations of AST nodes

Last week at code::dive in Wroclaw, I demonstrated tooling solutions to all of these problems. I look forward to video of that talk (and videos from the rest of the conference!) becoming available.

Meanwhile, I’ll publish some blog posts here showing the same new features in clang-query and clang-tidy.

clang-query in Compiler Explorer

Recent work by the Compiler Explorer maintainers adds the possibility to use source code tooling with the website. The compiler explorer contains new entries in a menu to enable a clang-tidy pane.

clang-tidy in Compiler Explorer

clang-tidy in Compiler Explorer

I demonstrated use of compiler explorer to use the clang-query tool at the code::dive conference, building upon the recent work by the compiler explorer developers. This feature will get upstream in time, but can be used with my own AWS instance for now. This is suitable for exploration of the effect that changing source code has on match results, and orthogonally, the effect that changing the AST Matcher has on the match results. It is also accessible via ce.steveire.com.

It is important to remember that Compiler Explorer is running clang-query in script mode, so it can process multiple let and match calls for example. The new command set print-matcher true helps distinguish the output from the matcher which causes the output. The help command is also available with listing of the new features.

The issue of clang-query not printing both diagnostic information and AST information at the same time means that users of the tool need to alternate between writing

set output diag

and

set output dump

to access the different content. Recently, I committed a change to make it possible to enable both output and diag output from clang-query at the same time. New commands follow the same structure as the set output command:

enable output detailed-ast
disable output detailed-ast

The set output command remains as an “exclusive” setting to enable only one output feature and disable all others.

Dumping possible AST Matchers

This command design also enables the possibility of extending the features which clang-query can output. Up to now, developers of clang-tidy extensions had to inspect the AST corresponding to their source code using clang-query and then use that understanding of the AST to create an AST Matcher expression.

That mapping to and from the AST “mental model” is not necessary. New features I am in the process of upstreaming to clang-query enable the output of AST Matchers which may be used with existing bound AST nodes. The command

enable output matcher

causes clang-query to print out all matcher expressions which can be combined with the bound node. This cuts out the requirement to dump the AST in such cases.

Inspecting the AST is still useful as a technique to discover possible AST Matchers and how they correspond to source code. For example if the functionDecl() matcher is already known and understood, it can be dumped to see that function calls are represented by the CallExpr in the Clang AST. Using the callExpr() AST Matcher and dumping possible matchers to use with it leads to the discovery that callee(functionDecl()) can be used to determine particulars of the function being called. Such discoveries are not possible by only reading AST output of clang-query.

Dumping possible Source Locations

The other important discovery space in creation of clang-tidy extensions is that of Source Locations and Source Ranges. Developers creating extensions must currently rely on the documentation of the Clang AST to discover available source locations which might be relevant. Usually though, developers have the opposite problem. They have source code, and they want to know how to access a source location from the AST node which corresponds semantically to that line and column in the source.

It is important to make use a semantically relevant source location in order to make reliable tools which refactor at scale and without human intervention. For example, a cursory inspection of the locations available from a FunctionDecl AST node might lead to the belief that the return type is available at the getBeginLoc() of the node.

However, this is immediately challenged by the C++11 trailing return type feature, where the actual return type is located at the end. For a semanticallly correct location, you must currently use

getTypeSourceInfo()->getTypeLoc().getAs().getReturnLoc().getBeginLoc()

It should be possible to use getReturnTypeSourceRange(), but a bug in clang prevents that as it does not appreciate the trailing return types feature.

Once again, my new output feature of clang-query presents a solution to this discovery problem. The command

enable output srcloc

causes clang-query to output the source locations by accessor and caret corresponding to the source code for each of the bound nodes. By inspecting that output, developers of clang-tidy extensions can discover the correct expression (usually via the clang::TypeLoc heirarchy) corresponding to the source code location they are interested in refactoring.

Next Steps

I have made many more modifications to clang-query which I am in the process of upstreaming. My Compiler explorer instance is listed as the ‘clang-query-future’ tool, while the clang-query-trunk tool runs the current trunk version of clang-query. Both can be enabled for side-by-side comparison of the future clang-query with the exising one.

Embracing Modern CMake

November 5, 2017

I spoke at the ACCU conference in April 2017 on the topic of Embracing Modern CMake. The talk was very well attended and received, but was unfortunately not recorded at the event. In September I gave the talk again at the Dublin C++ User Group, so that it could be recorded for the internet.

The slides are available here. The intention of the talk was to present a ‘gathered opinion’ about what Modern CMake is and how it should be written. I got a lot of input from CMake users on reddit which informed some of the content of the talk.

Much of the information about how to write Modern CMake is available in the CMake documentation, and there are many presentations these days advocating the use of modern patterns and commands, discouraging use of older commands. Two other talks from this year that I’m aware of and which are popular are:

It’s very pleasing to see so many well-received and informative talks about something that I worked so hard on designing (together with Brad King) and implementing so many years ago.

One of the points which I tried to labor a bit in my talk was just how old ‘Modern’ CMake is. I recently was asked in private email about the origin and definition of the term, so I’ll try to reproduce that information here.

I coined the term “Modern CMake” while preparing for Meeting C++ 2013, where I presented on the topic and the developments in CMake in the preceding years. Unfortunately (this happens to me a lot with CMake), the talk was not recorded, but I wrote a blog post with the slides and content. The slides are no longer on the KDAB website, but can be found here. Then already in 2013, the simple example with Qt shows the essence of Modern CMake:


find_package(Qt5Widgets 5.2 REQUIRED)

add_executable(myapp main.cpp)
target_link_libraries(myapp Qt5::Widgets)

Indeed, the first terse attempt at a definition of “Modern CMake” and first public appearance of the term with its current meaning was when I referred to it as approximately “CMake with usage requirements”. That’s when the term gained a capitalized ‘M’ and its current meaning and then started to gain traction.

The first usage I found of “Modern CMake” in private correspondence was March 13 2012 in an email exchange with Alex Neundorf about presenting together on the topic at a KDE conference:

Hi Alex

Are you planning on going to Talinn for Akademy this year? I was thinking about sumitting a talk along the lines of Qt5, KF5, CMake (possibly along the lines of the discussion of ‘modern CMake’ we had before with Clinton, and what KDE CMake files could look like as a result).

I thought maybe we should coordinate so either we don’t submit overlapping proposals, or we can submit a joint talk.

Thanks,

Steve.

The “discussion with Clinton” was probably this thread and the corresponding thread on the cmake mailing list where I started to become involved in what would become Modern CMake over the following years.

The talk was unfortunately not accepted to the conference, but here’s the submission:

Speakers: Stephen Kelly, Alexander Neundorf
Title: CMake in 2012 – Modernizing CMake usage in Qt5 and KDE Frameworks 5
Duration: 45 minutes

KDE Frameworks 5 (KF5) will mark the start of a new chapter in the history of KDE and of the KDE platform. Starting from a desire to make our developments more easy to use by 3rd parties and ‘Qt-only’ developers, the effort to create KF5 is partly one of embracing and extending upstreams to satisfy the needs of the KDE Platform, to enable a broadening of the user base of our technology.

As it is one of our most important upstreams, and as the tool we use to build our software, KDE relies on CMake to provide a high standard of quality and features. Throughout KDE 4 times, KDE has added extensions to CMake which we consider useful to all developers using Qt and C++. To the extent possible, we are adding those features upstream to CMake. Together with those features, we are providing feedback from 6 years of experience with CMake to ensure it continues to deliver an even more awesome build experience for at least the next 6 years. Qt5 and KF5 will work together with CMake in ways that were not possible in KDE 4 times.

The presentation will discuss the various aspects of the KDE buildsystem planned for KF5, both hidden and visible to the developer. These aspects will include the CMake automoc feature, the role of CMake configuration files, and how a target orientated and consistency driven approach could change how CMake will be used in the future.

There is a lot to recognize there in what has since come to pass and become common in Modern CMake usage, in particular the “target orientated and consistency driven approach” which is the core characteristic of Modern CMake.

Grantlee v5.1.0 (Codename Außen hart und innen ganz weich) now available

April 19, 2016

The Grantlee community is pleased to announce the release of Grantlee version 5.1 (Mirror). Grantlee contains an implementation of the Django template system in Qt.

This release is binary and source compatible with the previous Qt5-based Grantlee release.

Following the pattern of naming Grantlee releases with German words and phrases I encounter, this release codename reflects the API being stable while the internals change a lot. Grantlee is “Allzeit bereit”, “einfach unerzetzlich” – it does everything “ganz ganz genau” :).

For the benefit of the uninitiated, Grantlee is a set of Qt based libraries including an advanced string template system in the style of the Django template system.

{# This is a simple template #}
{% for item in list %}
    {% if item.quantity == 0 %}
    We're out of {{ item.name }}!
    {% endif %}
{% endfor %}

The showcase feature in this release is the introduction of ==, !=, <, >, <=, >= and in operators for the if tag. Django has had this feature for many years, but it was missing from Grantlee until now.

Also notable in this release is several changes from Daniel of Cutelyst. Daniel ported Grantlee from QRegExp to the more modern QRegularExpression, better error checking, and lots of prototyping of the if operators feature.

In order to accommodate CMake IDE generators (Visual Studio/Xcode) plugins built in debug mode now gain a ‘d‘ postfix in their name. Grantlee built in debug mode will first search for a plugin with a ‘d‘ postfix and fall back to a name without the postfix. This should make it possible to use a different generator to build Grantlee than is used to build 3rd-party plugins (eg, NMake and Visual Studio), and still be able to use the resulting binaries.

All of the work on making everything work well with all CMake generators was concurrent with making use of Travis and AppVeyor. All changes to Grantlee now have a large quality gate configuration matrix to pass. Of course, Qt handles all the platform abstractions needed here in the C++, but people build with many different configurations, CMake generators and platforms which need to continue to work.

All of the other changes since the last release are mostly clean-ups and minor modernizations of various kinds

  • Make it possible to build without QtScript and QtLinguistTools
  • clazy-discovered cleanups
  • Modernization
  • Bump the CMake requirement to version 3.1 and the Qt requirement to version 5.3

I’m giving an introductory talk later today about Grantlee.

How do you use Grantlee?

April 11, 2016

Grantlee has been out in the wild for quite some years now. Development stagnated for a while as I was concentrating on other things, but I’m updating it now to prepare for a new release.

I’m giving a talk about it in just over a week (the first since 2009), and I’m wondering how people are using it these days. The last time I really investigated this was 6 years ago.

I’m really interested in knowing about other users of Grantlee and other use-cases where it fits. Here are some of the places I’m already aware of Grantlee in use:

Many areas of KDE PIM use Grantlee for rendering content such as addressbook entries and rss feeds along with some gui editors for creating a new look. The qgitx tool also uses it for rendering commits in the view with a simple template.

qgit-grantlee

It is also used in the Cutelyst web framework for generating html, templated emails and any other use-cases users of that framework have.

There is also rather advanced Grantlee integration available in KDevelop for new class generation, using the same principles I blogged about some years ago.

It is also used by the subsurface application for creating dive summaries and reports. Skrooge also uses it for report generation.

It is used in Oyranos buildsystem, seemingly to generate some of the code compiled into the application.

Also on the subject of generating things, it seems to be used in TexturePacker, a tool for game developers to create efficient assets for their games. Grantlee enables one of the core selling points of that software, enabling it to work with any game engine.

Others have contacted me about using Grantlee to generate documentation, or to generate unit tests to feed to another DSL. That’s not too far from how kitemmodels uses it to generate test cases for proxy model crashes.

Do you know of any other users or use-cases for Grantlee? Let us know in the comments!

CMake Daemon for user tools

January 24, 2016

I’ve been working for quite some time on a daemon mode for CMake in order to make it easier to build advanced tooling for CMake. I made a video about this today:

The general idea is that CMake is started as a long-running process, and can then be communicated with via a JSON protocol.

So, for example, a client sends a request like

{
  "type": "code_completion_at",
  "line": 50,
  "path": "/home/stephen/dev/src/cmake-browser/CMakeLists.txt",
  "column": 7
}

and the daemon responds with

{  
   "completion":{  
      "commands":[  
         "target_compile_definitions",
         "target_compile_features",
         "target_compile_options",
         "target_include_directories",
         "target_link_libraries",
         "target_sources"
      ],
      "matcher":"target_"
   }
}
Many more features are implemented such as semantic annotation, variable introspection, contextual help etc, all without the client having to implement it themselves.
Aside from the daemon, I implemented a Qt client making use of all of the features, and a Kate plugin to use the debugging features in that editor. This is the subject of my talk at FOSDEM, which I previewed in Berlin last week.
Come to my talk there to learn more!

Holy Cheese Batman!

June 1, 2011

* What?

* Uh, the cheese. It has lots of holes in it…

That’s right. I’m in Switzerland

I was a bit cryptic in my last post, though posts from others may be more clear that I’m at the Platform 11 sprint. There is a growing list of discussion topics, but the final list of topics will be determined at runtime.

It seems though that there is a lot of stuff to untangle here.

Untangling this is going to be a lot of work

We’re going to spend the next 6 or so days talking about the future of the KDE platform, the future of the KDE libraries, what we should decouple and what not, what Qt 5 means for KDE, the relevance of QML etc. All the stuff that everyone else is wondering about, we are too, and there are about 20 people here in Randa Switzerland, to discuss it in high volume, high bandwidth and low distraction.

There is another few sprints co-located in time and space to discuss Nepomuk, Amarok and KDevelop, so in total we are about 60 people concentrated in a tiny village. This might be the highest concentration of KDE developers compared to ‘normies’ in a single place ever.

It’s been an evening of arrivals and re-acquainting with familiar faces. I finally got to meet Gregory Schlomoff in person too and catch up with what is happening in BetterInbox. This is sure to be a good sprint.

MeeGo@Dublin

November 9, 2010

I’ll be in Dublin on Monday only for the first day of the MeeGo conference at Lansdowne Road.

Try to catch up if you want to talk about things I know about such as Qt ModelView, string templating, QML, KDE, PIM or KDAB.

I’ve also been getting more interested in state machines and l10n/i18n lately, so they’re also good topics.