Archive for the ‘Events’ Category

Future Developments in clang-query

November 11, 2018

Getting started – clang-tidy AST Matchers

Over the last few weeks I published some blogs on the Visual C++ blog about Clang AST Matchers. The series can be found here:

I am not aware of any similar series existing which covers creation of clang-tidy checks, and use of clang-query to inspect the Clang AST and assist in the construction of AST Matcher expressions. I hope the series is useful to anyone attempting to write clang-tidy checks. Several people have reported to me that they have previously tried and failed to create clang-tidy extensions, due to various issues, including lack of information tying it all together.

Other issues with clang-tidy include the fact that it relies on the “mental model” a compiler has of C++ source code, which might differ from the “mental model” of regular C++ developers. The compiler needs to have a very exact representation of the code, and needs to have a consistent design for the class hierarchy representing each standard-required feature. This leads to many classes and class hierarchies, and a difficulty in discovering what is relevant to a particular problem to be solved.

I noted several problems in those blog posts, namely:

  • clang-query does not show AST dumps and diagnostics at the same time<
  • Code completion does not work with clang-query on Windows
  • AST Matchers which are appropriate to use in contexts are difficult to discover
  • There is no tooling available to assist in discovery of source locations of AST nodes

Last week at code::dive in Wroclaw, I demonstrated tooling solutions to all of these problems. I look forward to video of that talk (and videos from the rest of the conference!) becoming available.

Meanwhile, I’ll publish some blog posts here showing the same new features in clang-query and clang-tidy.

clang-query in Compiler Explorer

Recent work by the Compiler Explorer maintainers adds the possibility to use source code tooling with the website. The compiler explorer contains new entries in a menu to enable a clang-tidy pane.

clang-tidy in Compiler Explorer

clang-tidy in Compiler Explorer

I demonstrated use of compiler explorer to use the clang-query tool at the code::dive conference, building upon the recent work by the compiler explorer developers. This feature will get upstream in time, but can be used with my own AWS instance for now. This is suitable for exploration of the effect that changing source code has on match results, and orthogonally, the effect that changing the AST Matcher has on the match results. It is also accessible via cqe.steveire.com.

It is important to remember that Compiler Explorer is running clang-query in script mode, so it can process multiple let and match calls for example. The new command set print-matcher true helps distinguish the output from the matcher which causes the output. The help command is also available with listing of the new features.

The issue of clang-query not printing both diagnostic information and AST information at the same time means that users of the tool need to alternate between writing

set output diag

and

set output dump

to access the different content. Recently, I committed a change to make it possible to enable both output and diag output from clang-query at the same time. New commands follow the same structure as the set output command:

enable output dump
disable output dump

The set output <feature> command remains as an “exclusive” setting to enable only one output feature and disable all others.

Dumping possible AST Matchers

This command design also enables the possibility of extending the features which clang-query can output. Up to now, developers of clang-tidy extensions had to inspect the AST corresponding to their source code using clang-query and then use that understanding of the AST to create an AST Matcher expression.

That mapping to and from the AST “mental model” is not necessary. New features I am in the process of upstreaming to clang-query enable the output of AST Matchers which may be used with existing bound AST nodes. The command

enable output matcher

causes clang-query to print out all matcher expressions which can be combined with the bound node. This cuts out the requirement to dump the AST in such cases.

Inspecting the AST is still useful as a technique to discover possible AST Matchers and how they correspond to source code. For example if the functionDecl() matcher is already known and understood, it can be dumped to see that function calls are represented by the CallExpr in the Clang AST. Using the callExpr() AST Matcher and dumping possible matchers to use with it leads to the discovery that callee(functionDecl()) can be used to determine particulars of the function being called. Such discoveries are not possible by only reading AST output of clang-query.

Dumping possible Source Locations

The other important discovery space in creation of clang-tidy extensions is that of Source Locations and Source Ranges. Developers creating extensions must currently rely on the documentation of the Clang AST to discover available source locations which might be relevant. Usually though, developers have the opposite problem. They have source code, and they want to know how to access a source location from the AST node which corresponds semantically to that line and column in the source.

It is important to make use a semantically relevant source location in order to make reliable tools which refactor at scale and without human intervention. For example, a cursory inspection of the locations available from a FunctionDecl AST node might lead to the belief that the return type is available at the getBeginLoc() of the node.

However, this is immediately challenged by the C++11 trailing return type feature, where the actual return type is located at the end. For a semanticallly correct location, you must currently use

getTypeSourceInfo()->getTypeLoc().getAs().getReturnLoc().getBeginLoc()

It should be possible to use getReturnTypeSourceRange(), but a bug in clang prevents that as it does not appreciate the trailing return types feature.

Once again, my new output feature of clang-query presents a solution to this discovery problem. The command

enable output srcloc

causes clang-query to output the source locations by accessor and caret corresponding to the source code for each of the bound nodes. By inspecting that output, developers of clang-tidy extensions can discover the correct expression (usually via the clang::TypeLoc heirarchy) corresponding to the source code location they are interested in refactoring.

Next Steps

I have made many more modifications to clang-query which I am in the process of upstreaming. My Compiler explorer instance is listed as the ‘clang-query-future’ tool, while the clang-query-trunk tool runs the current trunk version of clang-query. Both can be enabled for side-by-side comparison of the future clang-query with the exising one.

Advertisements

Embracing Modern CMake

November 5, 2017

I spoke at the ACCU conference in April 2017 on the topic of Embracing Modern CMake. The talk was very well attended and received, but was unfortunately not recorded at the event. In September I gave the talk again at the Dublin C++ User Group, so that it could be recorded for the internet.

The slides are available here. The intention of the talk was to present a ‘gathered opinion’ about what Modern CMake is and how it should be written. I got a lot of input from CMake users on reddit which informed some of the content of the talk.

Much of the information about how to write Modern CMake is available in the CMake documentation, and there are many presentations these days advocating the use of modern patterns and commands, discouraging use of older commands. Two other talks from this year that I’m aware of and which are popular are:

It’s very pleasing to see so many well-received and informative talks about something that I worked so hard on designing (together with Brad King) and implementing so many years ago.

One of the points which I tried to labor a bit in my talk was just how old ‘Modern’ CMake is. I recently was asked in private email about the origin and definition of the term, so I’ll try to reproduce that information here.

I coined the term “Modern CMake” while preparing for Meeting C++ 2013, where I presented on the topic and the developments in CMake in the preceding years. Unfortunately (this happens to me a lot with CMake), the talk was not recorded, but I wrote a blog post with the slides and content. The slides are no longer on the KDAB website, but can be found here. Then already in 2013, the simple example with Qt shows the essence of Modern CMake:


find_package(Qt5Widgets 5.2 REQUIRED)

add_executable(myapp main.cpp)
target_link_libraries(myapp Qt5::Widgets)

Indeed, the first terse attempt at a definition of “Modern CMake” and first public appearance of the term with its current meaning was when I referred to it as approximately “CMake with usage requirements”. That’s when the term gained a capitalized ‘M’ and its current meaning and then started to gain traction.

The first usage I found of “Modern CMake” in private correspondence was March 13 2012 in an email exchange with Alex Neundorf about presenting together on the topic at a KDE conference:

Hi Alex

Are you planning on going to Talinn for Akademy this year? I was thinking about sumitting a talk along the lines of Qt5, KF5, CMake (possibly along the lines of the discussion of ‘modern CMake’ we had before with Clinton, and what KDE CMake files could look like as a result).

I thought maybe we should coordinate so either we don’t submit overlapping proposals, or we can submit a joint talk.

Thanks,

Steve.

The “discussion with Clinton” was probably this thread and the corresponding thread on the cmake mailing list where I started to become involved in what would become Modern CMake over the following years.

The talk was unfortunately not accepted to the conference, but here’s the submission:

Speakers: Stephen Kelly, Alexander Neundorf
Title: CMake in 2012 – Modernizing CMake usage in Qt5 and KDE Frameworks 5
Duration: 45 minutes

KDE Frameworks 5 (KF5) will mark the start of a new chapter in the history of KDE and of the KDE platform. Starting from a desire to make our developments more easy to use by 3rd parties and ‘Qt-only’ developers, the effort to create KF5 is partly one of embracing and extending upstreams to satisfy the needs of the KDE Platform, to enable a broadening of the user base of our technology.

As it is one of our most important upstreams, and as the tool we use to build our software, KDE relies on CMake to provide a high standard of quality and features. Throughout KDE 4 times, KDE has added extensions to CMake which we consider useful to all developers using Qt and C++. To the extent possible, we are adding those features upstream to CMake. Together with those features, we are providing feedback from 6 years of experience with CMake to ensure it continues to deliver an even more awesome build experience for at least the next 6 years. Qt5 and KF5 will work together with CMake in ways that were not possible in KDE 4 times.

The presentation will discuss the various aspects of the KDE buildsystem planned for KF5, both hidden and visible to the developer. These aspects will include the CMake automoc feature, the role of CMake configuration files, and how a target orientated and consistency driven approach could change how CMake will be used in the future.

There is a lot to recognize there in what has since come to pass and become common in Modern CMake usage, in particular the “target orientated and consistency driven approach” which is the core characteristic of Modern CMake.

Grantlee v5.1.0 (Codename Außen hart und innen ganz weich) now available

April 19, 2016

The Grantlee community is pleased to announce the release of Grantlee version 5.1 (Mirror). Grantlee contains an implementation of the Django template system in Qt.

This release is binary and source compatible with the previous Qt5-based Grantlee release.

Following the pattern of naming Grantlee releases with German words and phrases I encounter, this release codename reflects the API being stable while the internals change a lot. Grantlee is “Allzeit bereit”, “einfach unerzetzlich” – it does everything “ganz ganz genau” :).

For the benefit of the uninitiated, Grantlee is a set of Qt based libraries including an advanced string template system in the style of the Django template system.

{# This is a simple template #}
{% for item in list %}
    {% if item.quantity == 0 %}
    We're out of {{ item.name }}!
    {% endif %}
{% endfor %}

The showcase feature in this release is the introduction of ==, !=, <, >, <=, >= and in operators for the if tag. Django has had this feature for many years, but it was missing from Grantlee until now.

Also notable in this release is several changes from Daniel of Cutelyst. Daniel ported Grantlee from QRegExp to the more modern QRegularExpression, better error checking, and lots of prototyping of the if operators feature.

In order to accommodate CMake IDE generators (Visual Studio/Xcode) plugins built in debug mode now gain a ‘d‘ postfix in their name. Grantlee built in debug mode will first search for a plugin with a ‘d‘ postfix and fall back to a name without the postfix. This should make it possible to use a different generator to build Grantlee than is used to build 3rd-party plugins (eg, NMake and Visual Studio), and still be able to use the resulting binaries.

All of the work on making everything work well with all CMake generators was concurrent with making use of Travis and AppVeyor. All changes to Grantlee now have a large quality gate configuration matrix to pass. Of course, Qt handles all the platform abstractions needed here in the C++, but people build with many different configurations, CMake generators and platforms which need to continue to work.

All of the other changes since the last release are mostly clean-ups and minor modernizations of various kinds

  • Make it possible to build without QtScript and QtLinguistTools
  • clazy-discovered cleanups
  • Modernization
  • Bump the CMake requirement to version 3.1 and the Qt requirement to version 5.3

I’m giving an introductory talk later today about Grantlee.

How do you use Grantlee?

April 11, 2016

Grantlee has been out in the wild for quite some years now. Development stagnated for a while as I was concentrating on other things, but I’m updating it now to prepare for a new release.

I’m giving a talk about it in just over a week (the first since 2009), and I’m wondering how people are using it these days. The last time I really investigated this was 6 years ago.

I’m really interested in knowing about other users of Grantlee and other use-cases where it fits. Here are some of the places I’m already aware of Grantlee in use:

Many areas of KDE PIM use Grantlee for rendering content such as addressbook entries and rss feeds along with some gui editors for creating a new look. The qgitx tool also uses it for rendering commits in the view with a simple template.

qgit-grantlee

It is also used in the Cutelyst web framework for generating html, templated emails and any other use-cases users of that framework have.

There is also rather advanced Grantlee integration available in KDevelop for new class generation, using the same principles I blogged about some years ago.

It is also used by the subsurface application for creating dive summaries and reports. Skrooge also uses it for report generation.

It is used in Oyranos buildsystem, seemingly to generate some of the code compiled into the application.

Also on the subject of generating things, it seems to be used in TexturePacker, a tool for game developers to create efficient assets for their games. Grantlee enables one of the core selling points of that software, enabling it to work with any game engine.

Others have contacted me about using Grantlee to generate documentation, or to generate unit tests to feed to another DSL. That’s not too far from how kitemmodels uses it to generate test cases for proxy model crashes.

Do you know of any other users or use-cases for Grantlee? Let us know in the comments!

CMake Daemon for user tools

January 24, 2016

I’ve been working for quite some time on a daemon mode for CMake in order to make it easier to build advanced tooling for CMake. I made a video about this today:

The general idea is that CMake is started as a long-running process, and can then be communicated with via a JSON protocol.

So, for example, a client sends a request like

{
  "type": "code_completion_at",
  "line": 50,
  "path": "/home/stephen/dev/src/cmake-browser/CMakeLists.txt",
  "column": 7
}

and the daemon responds with

{  
   "completion":{  
      "commands":[  
         "target_compile_definitions",
         "target_compile_features",
         "target_compile_options",
         "target_include_directories",
         "target_link_libraries",
         "target_sources"
      ],
      "matcher":"target_"
   }
}
Many more features are implemented such as semantic annotation, variable introspection, contextual help etc, all without the client having to implement it themselves.
Aside from the daemon, I implemented a Qt client making use of all of the features, and a Kate plugin to use the debugging features in that editor. This is the subject of my talk at FOSDEM, which I previewed in Berlin last week.
Come to my talk there to learn more!

Holy Cheese Batman!

June 1, 2011

* What?

* Uh, the cheese. It has lots of holes in it…

That’s right. I’m in Switzerland

I was a bit cryptic in my last post, though posts from others may be more clear that I’m at the Platform 11 sprint. There is a growing list of discussion topics, but the final list of topics will be determined at runtime.

It seems though that there is a lot of stuff to untangle here.

Untangling this is going to be a lot of work

We’re going to spend the next 6 or so days talking about the future of the KDE platform, the future of the KDE libraries, what we should decouple and what not, what Qt 5 means for KDE, the relevance of QML etc. All the stuff that everyone else is wondering about, we are too, and there are about 20 people here in Randa Switzerland, to discuss it in high volume, high bandwidth and low distraction.

There is another few sprints co-located in time and space to discuss Nepomuk, Amarok and KDevelop, so in total we are about 60 people concentrated in a tiny village. This might be the highest concentration of KDE developers compared to ‘normies’ in a single place ever.

It’s been an evening of arrivals and re-acquainting with familiar faces. I finally got to meet Gregory Schlomoff in person too and catch up with what is happening in BetterInbox. This is sure to be a good sprint.

MeeGo@Dublin

November 9, 2010

I’ll be in Dublin on Monday only for the first day of the MeeGo conference at Lansdowne Road.

Try to catch up if you want to talk about things I know about such as Qt ModelView, string templating, QML, KDE, PIM or KDAB.

I’ve also been getting more interested in state machines and l10n/i18n lately, so they’re also good topics.

Flight of the Pechvogel

January 29, 2010

On Wednesday afternoon I boarded a Greyhound bus in Las Vegas, and by Thursday morning I was at the airport in San Diego to fly home to Berlin by Friday evening. It’s 8 am by my frame of reference, and I didn’t sleep on the plane because of the old guy beside me who looked so uncomfortable in his chair I thought he might collapse.

Of course, just like last time, my luggage didn’t get as far as I did. There must be some way to measure the hysteresis of my luggage–self movements. Or something, I’m very tired and I have to stay up until nighttime. But at least I learned a new German word.

There was a lot of interesting content at CampKDE which will be shared with the world in due course. Aside from the talks, it was very cool to meet KDE people I’d never met before and some that I’ve met a few times now. This side of the pond had a good turnout, but it was mostly North Americans attending. The trainings in CMake, Qt and QtEmbedded were very useful to attend too.

I should be back to the usual interesting blog posts about kde pim, more ModelView stuff and Grantlee in the next week now that I’m out of the land of distraction.

Getting (mostly) to CampKDE

January 15, 2010

A guy walks to the checkin desk of his airline with a lot of luggage:

Flyer: Hi there. I’d like you to send my suitcase to Beijing, my rucksack to Washington, my surfboard to Berlin, and I’m flying to San Diego.
Airline: Uh, I’m sorry sir I don’t think we can do that.
Flyer: Well you did it before without me even asking you to…

In fairness I had only one checked in bag on my flights, but it got lost somewhere through my cancelled and delayed connections. Hopefully it will turn up in the next day or two, or it could shaft my plans for after the event.

Right now though, they’re playing Moondance by Van Morrison in the lobby. The day is looking up. 🙂 Time to debug my presentation methinks.

Award for the most gratuitous use of the word “QAbstractItemModel” in a serious presentation

December 23, 2009

It seems I’m one of the recipients of the first annual NQDF award for outstanding achievement in the field of excellence. It seems the phone will be delivered in a few weeks, but they haven’t yet revealed when the big award ceremony is, or when I will get money, power, fame and influence as a result. I’ll be patient, but until then, thanks for the phone!

The schedule for CampKDE was published last week. I’ll be giving a talk on the challenges the Akonadi team faced in using the Qt Model View framework to implement the posted KDE PIM applications and how it results in reusable components and allows a lot of scope for innovation. I have been blogging on the subject for some time now, but there will be a lot of new content in the presentation that I haven’t covered here yet which is relevant to all users of Qt Model/View – not just Akonadi and not just KDE. Registration is still open if you want to come to attend.

By the way, I didn’t know this until just now. Very funny.