Thinking back all the way to the beginning, I’m quickly overwhelmed by how many great people I have had the opportunity to work with thanks to the Eclipse CDT project. At the very beginning was Sky Matthews and John Prokopenko who let me weasel my way on as Rational’s technical lead on the project just as it was starting out in 2002 also at a time when I needed a change. Of course, I had a great team of developers at Rational with me that made it fun and easy. Not to mention the original team at QNX who were welcoming and made it easy to get involved. I have a special mention for Sebastien Marineau, CDT’s first project lead, who let me take a leadership role on the project and eventually hired me on at QNX to take over.

Then there was the early years on the CDT where we made our mark. Those early CDT Summits were so fun and really helped built up a team atmosphere. We had about a dozen companies sending contributors, a few of them competitors in the spirit of co-opetition, and we made it work. Then over the years we started getting independent contributors who just did it for the passion of building great C++ tooling they wanted to use themselves. It’s been a great mix and I am so lucky and proud to have been a part of it.

And of course, it was all topped off with our yearly EclipseCons. I am proud to have attended every one of the EclipseCon North America ones and was able to attend quite a few of the EclipseCon Europes in Germany. I have to thank Anne and Mike and Ralph and Wayne and Sharon and Perri and Donald and Ian and Lynn and all the Eclipse Foundation staff past and present for making me feel a part of the family. I always looked forward to the EclipseCon Express flights out of and return to Ottawa with many of them.

My fellow attendees at these conferences were amazing, from the first one at Disneyland where we had an overflow crowd at the CDT BOF and where I gave my first of many CDT talks, to all the friends I met at the bar or ran into at sessions, many of whom had nothing to do with CDT but made me feel so much a part of the bigger community. I will never forget the late nights in the bars chatting with friends like Michael Scharf and Ian Bull and Eric Cloninger and Gilles and Adrian and Jonah and Tom and so many others. As it turns out, last year in Ludwigsburg was a perfect finale where we had such a great time at the Nestor on Wednesday night. I will never forget you all.

I’m incredibly proud of what we built for the CDT. It still has the best indexer in the business, thanks to the parser we built back at Rational and the database I built at QNX and then with so many hands continually making it better and adjusting to the now ever changing C++ language spec. The Launch Bar achieved what I wanted by simplifying the Eclipse launch experience. CDT’s new Core Build fits naturally with the Launch Bar and makes it much simpler to integrate external build systems like CMake. And we have just started a GDB debug adapter using the debug adapter protocol which will pave the way to simplify integrating debuggers with the CDT.

The current set of active committers on the CDT have lately been pulling almost all the weight evolving it and getting releases out the door. Their great work has made my transition easier and will keep the CDT rolling along for years to come. And hopefully vendors will come back too and help provide funding for all this activity. We have an action plan to transition the project lead role. Follow the cdt-dev mailing list to find out more.

It’s sad to leave and the memories and friendships will be forever. I will keep my cdtdoug personal gmail account as a reminder of where I came from. But my new role will give me some much needed energy to keep things going for the next decade. I once questioned why you hardly see any retired engineers helping with open source projects or sharing their passion with the next generation. I promise you this, you will see me again.

Take care, and thank you.

When I joined the Eclipse CDT project back in 2002 (yeah, it’s been a long time), I was working on modeling tools for “real time”, or more accurately, embedded reactive systems. Communicating state machines. I wrote code generators that generated C and C++ from ROOM models and then eventually UML-RT. ROOM was way better by the way and easier to generate for because it was more semantically complete and well defined. That objective is key later in this story.

We had the vision to integrate our modeling tools more closely with Integrated Development Environments. We started looking at Visual Studio but Eclipse was the young up and comer. That and IBM bought us, Rational by that point, and had already bought OTI who built Eclipse so it was a natural fit. And we were all in Ottawa. And by chance, Ottawa-based QNX had already written a C/C++ IDE based on Eclipse and were open sourcing it and it was perfect for our customers as well. It’s amazing how that all happened and led to my life as CDT Doug.

Our first order of business was to help the CDT become an industry class C/C++ IDE and become a foundation for integrating our modeling tools. Since we wanted to be able to generate model elements from code, it required we have accurate C and C++ parsers and indexers. No one figured we could do it, but we were able to put together a somewhat decent system written in Java in the org.eclipse.cdt.core plug-in.

Scaling is Hard

However, as the community started to try it out on real projects, especially ones of a significant size, we started to run into pretty massive performance problems with the indexer. We were essentially doing full builds of the user’s projects and storing the results in a string table. On large projects, builds take a long time. But users expect that and put up with it because they really need those binaries it produces. They don’t have the same patience for their IDEs building indexes the don’t really see and we paid a pretty high price for that.

As a solution, I wondered if we could store the symbol information that we were gathering in a way that we could load it up from disk as we were parsing other files and plug the symbol info into the AST the same way we do symbols normally. This would allow us to parse header files once and reuse the results, similar to how precompiled headers work. The price you pay is in accuracy since some systems parse header files multiple times with different macro settings. But my guess was that it wouldn’t be that bad.

It was hard to convince my team at IBM Rational to take this road. Accuracy was king for our modeling tools. But when I moved to join QNX, I decide to forgo that requirement and give this “fast indexer” strategy a go. And the rest is history. Performance on large projects was an order of magnitude faster. Incremental indexing of files as they were saved isn’t even noticeable. It was a huge success and my proudest contribution to the CDT. And I was even better when other community members handed us their expertise to make the accuracy better and better so you barely notice that at all either.

C++ Rises from the “Dead”

Move the clock a decade later and we started running into a problem. The C++ standards community has new life and are adding a tonne of new features at a three year cadence. The CDT community has long lost most of the experts that build the original parsers. Lucky for us a new crop of contributors has come along and are doing heroes work to keep up. But it’s getting harder and harder. One thing we benefit from is how slow embedded developers, the majority of users of CDT, are to adopt the new standards. It gives us time, but not forever. We need to find a better way.

Then along came the Language Server Protocol and a small handful of language servers that do C/C++. I’ve investigated four of them. Three of them are based on llvm and clang. One of them is in tree with llvm and clang in clang-tools-extra, i.e., clangd. The other two are projects that use libclang with parts of the tree, i.e., cquery and ccls. Those two projects are what I call “one person projects” and with cquery at least, that person found something else to do last November. Beware of the one person project.

clangd

I’ve spent a lot of time with clangd when experimenting with Visual Studio Code. For what it does, clangd is very accurate and really fast. It uses compile_commands.json files to find out what source files are built and what compiler and command lines they use. I’ve had to fork the tree to add in support for discovering compilers it doesn’t know about, but that was pretty easy to put together. It gives great content assist and you get the benefit of clang’s awesome compilation error diagnostics as you type. It shows a lot of promise.

However clangd for the longest time lacked an indexer. When you search for references it only finds them in files you have opened previously. The thought as I understand it is that you use another process to build the index and that is usually done at build time. However, not all users have such an environment set up so having an index created by the IDE is a mandatory feature. Now, clangd did eventually get an indexer but it does what the old CDT indexer did and completely parses the source three. That predictably takes forever on large projects and I don’t think users have the appetite to take a huge step backwards like that.

IntelliSense

While waiting for the right solution to arrive for clangd, I thought I’d give the Microsoft C/C++ Tools for VS Code a try. My initial experience was quite surprising. It actually worked well with a gnu tools cross compiler project I used for testing. You have to teach it how to parse your code using a magic JSON file, which fits right in with the rest of VS Code. It’s able to pick out the default include path when you point it at your compiler. It has a MI support for debugging, though no built-in support for remote debugging but that was hackable. It seemed like a reasonable alternative, at least for VS Code.

However when I tried it with one of our production projects it quickly fell apart. It does a great job trying to figure out include paths, similar to the heuristics we use in CDT. That includes things like treating all the folders in your workspace as a potential include path entry. But it tended to make mistakes. It even has support for compile_commands.json files so I could tell it the command lines that were use. It did better but still tried to do too much and gave incorrect results.

That and it doesn’t have an index yet either. One is coming soon, but if it can’t figure out how to parse my files correctly, it’s not going to be a great experience. Still a lot of work to do there.

Where do we go from here?

As it stands today, at least from a CDT perspective, there really isn’t a language server solution that comes near what we have in CDT. Yes, some things are better. Both these language servers are using real parsers to parse the code. (or at least clangd is. Microsoft’s, of course, is closed source so I can only assume). They give really good content assist and error diagnostics and open declaration works. But without a usable indexer, you don’t get accurate symbol references. And I haven’t even mentioned refactoring which CDT has and which is not even suggested in the language server protocol.

So if all your doing is typing in code, the new language servers are great. But if you need to do some code mining to understand the code before you change it, you’re out of luck. The good news is that we are continuing to see investment in them so who knows. But then, maybe the CDT parsers catch up with the language standards before these other language servers grow great indexers making the whole thing moot. I wouldn’t bet against that right now.

Recently with version 8 of clang, they have made the wasm target a first class citizen available with the standard distribution. I thought I’d take a look and found at least one example on github that showed how. Here’s a quick summary on how to get started. Be warned, though, one of the arguments is nostdlib which means this is a very barebones example. But that’s another area where I think Emscripten has gone a little to far with. More on that later.

To start this example is a pretty basic Fibonacci calculator, pretty standard for WebAssembly. Here’s the C++ file.

#include “wasm.h”

WASM_IMPORT void log(int i);

WASM_EXPORT int fib(int i) {
    int res = i <= 1 ? i : fib(i — 1) + fib(i — 2);
    log(res);
    return res;
}

I wanted to show C++ calling back into JavaScript so there’s a very contrived log method we import. The fib function itself is pretty basic. I’ve created a couple of macros in the wasm.h file to manage marking functions as import or export.

#define WASM_EXPORT __attribute__((visibility(“default”))) \
    extern “C”
#define WASM_IMPORT extern “C”

Since I’m writing C++ I want to make sure the compiler doesn’t mangle the names so I declare them as extern “C”. As you can see, the export also turns on the visibility of the symbol which is hidden by default in the Makefile.

I’m running this with node.js which has had WebAssembly support since at least version 8 that I have on my Linux box. The idea is to do some of the more computationally expensive tasks in my node server using wasm. Here’s my js file.

const fs = require(‘fs’)

async function run() {
  const buf = fs.readFileSync(‘./fib.wasm’)
  return await WebAssembly.instantiate(buf, {
    ‘env’: {
      ‘log’: function(i) { console.log(`log: ${i}`) }
    }
  })
}

run().then(res => {
  const { fib } = res.instance.exports
  console.log(fib(10))
})

It simply loads up the wasm file and instantiates it passing in the log function. When that’s complete, I extract my fib function from the exports and run it. You should see the output of the log function (more times that I was expecting at least), then the result, 55.

As with most things C++, the magic is actually in the Makefile.

CXX = $(HOME)/wasm/clang-8/bin/clang
CXXFLAGS = \
    -Wall \
    --target=wasm32 \
    -Os \
    -flto \
    -nostdlib \
    -fvisibility=hidden \
    -std=c++14 \
    -ffunction-sections \
    -fdata-sections

LD = $(HOME)/wasm/clang-8/bin/wasm-ld
LDFLAGS = \
    --no-entry \
    --strip-all \
    --export-dynamic \
    --initial-memory=131072 \
    -error-limit=0 \
    --lto-O3 \
    -O3 \
    --gc-sections

fib.wasm: fib.o
    $(LD) $(LDFLAGS) -o $@ $<

There’s lots of magic flags here and I have to thank the author of the example I linked above for getting me started. I’ll have to play with them to see what’s actually necessary. The key here is that it isn’t Emscripten but straight clang 8 that I downloaded from llvm.org. There’s no standard library, so don’t go and try and do a printf. You’re a bit on your own for now.

But that’s somewhat a conclusion I reached. Emscripten allows C++ developer to easily port their apps to run on the web. It doesn’t make the C++ developer think like a Web developer. What would be interesting to me is what it would look like if you weren’t handed those fancy libraries you get with Emscripten and really just wanted to build a web app, like a game, using the standard JavaScript APIs you get with node or the browser. I think you’d end up writing programs like an Arduino developer where you don’t have printf either…

In my last article, I walked through a bit of the history of the Eclipse CDT project. It’s a great story of the trials and tribulations of community building, working with a group of like minded platform vendors, even competitors, who just wanted a great IDE and who knew the only way to get that was to work together.

What sticks with me most looking back on my ump-teen years on the project is the people. Each of these vendors were led by passionate tools developers who put aside any hint of corporate agenda and just wanted to work with other developers from around the world, to solve hard problems, and build something they would be proud of. Through the semi regular face-to-face CDT Summits and CDT days at EclipseCons past we were able to build relationships and become great friends. Even after some have moved on, I still remain in contact with many of them and we still share a laugh or two, and even the odd beer!

So I offer no apologies if, as participation has dwindled on the project in recent years, I yearn for those days. But my personal enjoyment working in this environment is one thing, I don’t think the job is done. The C++ language continues to evolve and become more usable and powerful than ever before. Even C continues to receive updates. And more importantly, the IDE landscape is changing and we need to keep up to keep our users and customers happy.

Over the years, I have always had to deal with users and even colleagues, who are hard core engineers, who refuse to give up their text editors and custom bash scripts. In the end, it’s been hard to argue with them. For these folk, IDEs don’t work the way they want to work. And, initially at least, it does slow them down as they fight the paradigm and they just end up frustrated and toss it aside losing all those gains we are trying to give them.

But in the end, I guess it was only a matter of time that an IDE platform would come along and offer the best of both worlds. Be a great unopinionated text editor, facilitate hooking up users’s bash scripts, be super extensible to add in little accelerators. It’s been brewing for a while. Sublime was the first such thing I saw developers really enjoy. And now, today, we have Visual Studio Code, which is becoming the dominant editor platform.

And funny enough, an interesting thing happens when you take a bunch of IDE developers and tell them to build a great editor, they end up building an editor so extensible that it slowly becomes an IDE. The VS Code team, many of them former Eclipsers, has managed to create a mechanism to add powerful language services to the editor. And, of all things, add support for debuggers. OK, you add debug support, you’re no longer an editor, you are an IDE. Stop pretending :).

But even greater, the VS Code team has done this by creating standardized APIs and protocols that other IDEs can use. You can build common services and then plug them into your favorite IDE, even Eclipse!

With this new architecture, the path forward for the Eclipse CDT project is clear. We need to take all that C/C++ IDE knowledge we’ve built over the years and use that to build common components that can be used by other IDE platforms. It means growing beyond our Eclipse IDE roots to give users that great experience in the IDE that best works they way they want to work.

We as the CDT community have a lot to offer, and there really isn’t a community like it, that focuses on the needs of the C/C++ developer. The challenge will be to harness that energy again, to work together for the greater good that we can only accomplish together. It really is an exciting time to be an IDE developer and Eclipse is a great home for us to come together.

In my next article I will start getting into the technical meat of what this new vision entails and what fun we can have building it! Time to roll up our sleeves and get to work.

The Eclipse CDT project has been around for a long time. The first code commits were in June 2002 when QNX contributed the core C/C++ tooling components of it’s fresh new Momentics IDE. For the time it was quite visionary. Platform vendors are more often focused on their platforms and provide tools as an enabler for developers to leverage the power of those platforms. Having every company replicate that effort only leads to poor tools and unhappy developers. It doesn’t scale. Why not join together with like-minded vendors and build something we can all benefit from?

As an added bonus, we end up with a standard platform that allows developers to work with multiple environments on the same platform leading to some interesting partnership opportunities. At the very least developers acquire skills working with that IDE they can take to other jobs or even other projects in the same company. It truly is a win/win for everyone and makes for a very rich ecosystem.

But as with all long lasting projects, it has had it’s ebbs and flows. As we started, QNX and my team at Rational Software (later part of IBM) were the main contributors. There were a few others lurking around with a few contributions. The first few monthly CDT conference calls were cool as new voices appeared from around the world. It was a great start that led to the first EclipseCon where we had people standing in the hall trying to listen into our overflow birds of a feather session. We were overwhelmed by the interest.

But it didn’t last forever and the powers that be decided my team at IBM was needed elsewhere in the company. Obviously I wasn’t pleased and certainly wasn’t ready to leave the community. So when Sebastien Marineau, the CDT project lead at the time suggested I come work for him at QNX, I jumped at the chance. It wasn’t the easiest of moves but it was my chance to do what I could to help keep the CDT machine rolling and ended up getting a good start on my pride and joy, CDT’s super-fast indexer.

At the first CDT summit held in the fall of 2005 shortly after I had joined QNX, we had a lot of interest and it was well attended with over a dozen vendors present. We had some good presentations on what we were working on and some interesting presentations from other vendors on what they’d like to see in the project. Well someone said the wrong thing and I had to stop the proceedings. Of the 20 or so people in the room I asked the active contributors to stand up. There were four of us. How are they expecting this little team to do their bidding when the whole idea was to share in the effort and do something great together.

That’s probably the proudest and hardest moment of my time on the CDT. It worked. And from the chart above you can see we had healthy growth and lots of vendors coming to help after that. Those were the best years for the project. Though for me personally, I never really did recover from my lost team at Rational/IBM and ended up moving around a bit, never leaving the CDT project but not really giving it my all.

Luckily and happily, for the last four years, I’ve been back at QNX working on that original Momentics IDE and contributing to Eclipse trying to make it easier to use for C/C++ developers including my favorite add, the Launch Bar. Our customers are happy, the community seems happy, and there are more and more vendors delivering product based on our work in open source.

But also from the chart you can see participation in the CDT has been on a worrying trajectory to a point where we once again have about 4 people actively working on it. Despite being as popular as ever, only a small percentage of people and companies out there are helping with the common cause. It’s made me sad, and frankly angry. And I must apologize to my friends at last year’s EclipseCon for losing control of that a bit, but when you have so many companies leveraging something you are passionately doing for free for them, thanks to my sympathetic employer, you feel taken advantage of.

But as the leader of the project, it’s up to me with the help of my open source colleagues to find a way to turn that around. And we have some ideas that I’m very excited about. My next few posts will talk about that and go into details of some very cool new efforts underway in the CDT project. The IDE world has changed but it’s needed more than ever and we are ready to adapt. We really hope you can join us and make it great, again…