Builder 3.20 development underway

I’ve been busy working on the plumbing for what will become Builder 3.20. We have a really ambitious cycle ahead of us, so getting these core changes in place as soon as possible will help give us time to stabilize.

From the early design phase of Builder, we knew we wanted multi-process plugins. Easier said than done. This cycle we are starting to follow through on that design goal by making it simple for plugins to achieve this.

I just landed the multi-process plumbing and ported our Jedi auto-completion engine to use it. Previously we were using Jedi from Python threads. It would occasionally cause the main loop to stall (likely due to GIL) as we have plugins written in Python inside the UI. Now that the work is being done in a worker process, we simply receive a large GVariant of the result set from the worker process over a private GDBus connection. This allows us to allocate/free large memory chunks instead of so many small strings which is nice. There are still a lot of strings created when sizing the completion window, but we can address that later.

Since we require GDBus, and PyGObject support for GDBus is not very convenient, we ported some of Martin Pitt’s work into the Builder G-I overrides. This means you can quickly create a GDBus service in Python with a few lines of code. It’s still not ideal, but again, we can iterate now. One thing I’d like to see is the ability to use yield to wait until an asynchronous operation completes. Python Twisted has been doing this for nearly a decade, and it’s quite pleasant to use. Also, we had to go out of our way to use GVariant from Python without unwrapping the variants. We want to avoid doing that until absolutely necessary.

The next big step in multi-process is to port the clang and vala plugins to use the same plumbing. After that, we need to get the plugins to recycle when they hit various thresholds such as undue memory consumption.

I think our story with plugins is going to become pretty compelling this cycle. Being able to write a single plugin that can take advantage of multiple-processes without destroying your code is an area where we can really shine above the competition.

I’ve also started work on distributing Builder as an Xdg-App. We have some unique constraints that will help us push Xdg-App to support some very difficult corner cases. Psuedo terminal support landed last week, so the terminal now works. Although, your terminal is inside of the mount namespace, so it won’t match your host system. That might feel a bit odd at first.

Additionally, we need to come up with a good design that allows Builder to run in one xdg-app, while building/running/testing your application from another xdg-app/runtime/sdk.

As always, lots to do! If you are interested in contributing to Builder or Xdg-App, this is a great time to get started! Come join us on irc.gimp.net in #gnome-builder and we’ll find a way for you to contribute.

post-GUADEC catchup!

Builder 3.18

It’s been an extremely busy couple of months since we last saw each other at GUADEC. During that time, I’ve been feverishly getting 3.18 out the door, and preparing some special bits for 3.18.1.

Builder Auto-Completion with Clang

Coverity Scans

Garrett Regier got us signed up for coverity, so we will start doing regular scans of Builder and as many of our depenent libraries as we can. I did the first batch of those yesterday. GLib and Gtk+, for example, already do this occasionally. It’s a free (as in beer) service to open source projects, and I’d consider it if you are writing C or C++ regularly.

Joining Red Hat

The real news, however, is that I’ve accepted a wonderful role at Red Hat. I’ll be focusing on the Xdg-App developer story, and Builder is an important part of that. We want to make it as easy as possible for you to create and deploy software that users can trust.

This was a difficult choice for me, as I had a few offers from people I respect greatly. Ultimately, I think Red Hat is the right place to push two hugely important projects forward. They were respectful of my desire to continue living like a Gnome in the forest, so I’ll be remote.

The GNOME Foundation’s Board of Directors already has two people who happen to be employed at Red Hat. Two is the maximum number from any single employer as set out in our bylaws. With much sadness, I have therefore resigned from the Board of Directors. I want to thank everyone who voted for me. I’m sure I’ll have an opportunity to serve you again soon enough. The Board was notified a couple of weeks ago and has started the process to replace me.

This week I have orientation and then I’m headed up to Boston for the summit over the long weekend.

Boston GNOME Summit

Saturday is the start of the Boston GNOME Summit! I hope those of you in the area can come say hi and visit for a bit. I’ll be demoing some Builder features and taking feature requests.

One thing I’d like to see this year is more people join us on IRC to be part of the event even if they can’t make it in person. Usually we are in #summit on irc.gimp.net.

I’d also like to say thank you to everyone who has been putting together this years summit. It’s a lot more work than you would think.

Builder Campaign

I’ve been so focused on writing software that other aspects of the campaign were not tended to in the manner I envisioned. If I were to ever take on a project of this magnitude again, I would definitely not go it alone. I would highly recommend having at least a second person in place that is dedicated to communication and administrative tasks. They are a huge time sink and really would have prevented me from pushing the software forward as fast as I did.

If you do not see your name in the Builder credits, and you purchased a perk that included that, please contact me (or submit a patch). Chances are that you selected the “anonymous donation” box on indiegogo. While I can see your name, I’m legally restricted from mentioning it (such as in the credits).

So… All of you who are waiting on perks from me, they are coming. I owe you more than just software and I’ll be pushing hard on that front over the next month to finish the preparations and shipments. My goal is to have the campaign perks completed and shipped by end of November (Thanksgiving time here).

New UI styling

It’s no secret that we are working towards an ambitious UI design. Where we are today is not a product of being where we want to be, but a step on the iterative process to where we are going.

One thing you’ll notice is that the floating bar is much less intrusive now.

Sébastien Lafargue landed support in Gtk+ for margins in GtkTextView. We are using this to allow you to scroll past the bottom of the document. (It was annoying editing lines along the bottom of the window). However, there is still more to do. Rather than using margin to achieve this, we’d like to add support in GtkTextView to allow us to scroll the last row to the top of the view. Vim users are familiar with this using `ctrl+e` to scroll.

Workbench Panels

The workbench has gained some panels. Of course, as I said above, this is intermediate design. We aren’t totally sure on final designs yet, but we needed something to move forward.

workbench

One thing that was particularly interesting to optimize was animating panels. Most IDEs cheat here and make them floating. Why? Transience is one, but another that it is difficult to do smoothly. It’s hard to manage 60 FPS animations if you have relayout and redraw cycles to perform on every frame. I was particularly careful about avoiding gtk_widget_queue_relayout() and ensuring we keep as much content in the GtkPixelCache as possible.

The source editor is one area that is very tricky to get right for animations. Editors are a tricky business. When you change the horizontal sizing of a GtkTextView, we have to discard the pixel cache because we might have right aligned text that needs to be moved. One way we can get around this in the future is to track if we have center or right aligned text in the BTree (where we store text tags and such). Upon an allocation change, we would have a fast lookup to determine if we need to destroy the cache. However, in the meantime, I employed some dirty tricks that are so dirty, you’ll have to read the source code to find out.

The symbol tree has been pulled out of the header (which was hidden behind a popover) and now has prominent viewing in the right-hand sidebar. Documentation is also now browsable and searchable from the sidebar.

We have a terminal in the bottom panel or `ctrl+shift+t` to have one inline with your editor. I find I’m switching to gnome-terminal a lot less now.

I decided to leave sysmon, a “debug” feature, in the release because I find it both beautifully smooth and useful for keeping me honest about CPU usage. If I really screw something up in Builder, I see it pretty immediately. Perhaps it’s the motivational factor of why Builder feels like such a lightweight editor despite being powered by IDE-like compiler features.

These scrolling CPU graphs were based on ideas I discussed in a blog series a few years back. They don’t do the ring buffer drawing, but I’ll add that back at some point. The code is in `contrib/rg` if you are interested. At least one other project is using the code today too.

Shortcuts

Many of you saw my screenshots for the new shortcut view which Allan so graciously hurried a design for (in the last couple weeks of the GNOME 3.18 cycle). It was one of our biggest complaints and I felt it had to be done even if it meant major code changes landing at the last minute.

OS X Support

While we in no way “officially” support OS X, you can now build Builder on OS X if you are so inclined. (It also works on FreeBSD and OpenBSD thanks to some gracious patches on Bugzilla).

It turns out cairo-quartz needs some love. You can find my cairo tree with various forward-ported patches from Mozilla, including CGLayer support on my github. It also has fixes for HiDPI when using GtkPixelCache (which I got from cairo bugzilla).

Something interesting I found was that colorspace conversion was happening on every frame draw. This is some seriously slow code. Both cairo and gdk-quartz were using the RGB colorspace, which you would think is fine. Apparently not. Instead, I changed both cairo and gtk to use the colorspace of the primary monitor to avoid that conversion altogether.

I’m curious what others do. I guess external monitors just always take the extra color conversion cost? Almost everything has a color profile on OS X it seems. Perhaps native applications adjust their colorspace when the window crosses a monitor boundary? They seem to employ similar tricks with DPI at least.

I was curious to learn more about how GdkFrameClock was implemented and wondering if it would help us with our frame rate on quartz. I wrote one of those too. You can find it on my gtk+ branch at wip/chergert/quartz-frame-clock.
It uses CVDisplayLink to thaw the GdkFrameClock when a frame needs to be drawn.

On my test machine, I was able to get two finger scroll on a retina 13″ macbook pro up to about 45 FPS. Still short of the “buttery smooth” feeling of TextEdit, but further progress can be made. This morning I did a few tests before heading out east, and we seem to be drawing fast enough. So the issue is probably somewhere else. More investigation needed.

Adding Support for Vala (in Vala)

One of the most requested features from the Builder campaign was support for Vala. Once 3.18 was out the door, I finally had a chance to sit down and see what that would look like.

Thankfully, the Anjuta codebase already had working code for auto-completion. So most of the work was a matter of adapting that code to fit into the compiler toolchain and editor design we use in Builder.

Vala Auto-Completion

I actually prototyped this in C, since the vala compiler of course compiles to header files. But I soon realized it was not code I would want to be the sole maintainer of. So I rewrote those bits in Vala and now the vala-pack plugin is our first plugin written in Vala. That means that you too can start writing plugins in Vala.

I do have concerns about memory leaks though. I find it particularly difficult to track those down compared to our C code. We have robust counters in Builder for tracking this type of stuff, but they aren’t usable from Vala yet.

I assume it’s in our usage of libvala for auto-completion, but I’ve not had the chance to investigate. I would love for the Vala community to help us make Builder the best tool for writing software in Vala. In particular, we still need build system integration to discover compiler flags like –pkg and such.

Python Auto Completion

We have auto-completion for Python as well. It’s powered by python-jedi but also knows about GObject Introspection. One caveat right now, by the very nature that jedi is running in process, is that this is for python3 only. We need to add support for moving the auto-completion out of process to support both python2 and python3. Thanks much to Elad Alfassa for implementing GObject Introspection and fixing my broken python.

python auto completion

Auto Indenter Designs

After having written four auto indenters, I started to see an abstraction that would make things easier. So this week I started on a new Python indenter which I hope to land soon. It’s design seems novel, but I’m not sure if it has been done this way before. Probably, nothing is new under the sun.

The first phase is to scan backwards for various points of interest (not new). While doing that, we collect information about things we discover. Such as things like:

  • This looks like a class start
  • This looks like a function start
  • This looks like the start of a comment
  • This is a tuple, dict, list, etc
  • Here is a block (if, elif, else, with, while, etc)

Then, we can match rules to selectors based on what we found.
For example, to determine if we are in a function parameter list, we might do something like:

match = selector(Rank.CLASS, Rank.FUNCTION, Rank.TUPLE)

We can do the same to discover if we are an `elif` or `else` nested inside an `if` block. That might indicate we should unindent the row.

I assume after some battle testing, I’ll port the C, XML, and Vala indenters to do the same. I’m sure there is a further abstraction to help us simplify these more.

Making Auto-Completion Fast

Those that follow my snark on twitter know that I spent some time this week on improving our auto-completion story. That was some fun work, so I’ll share a bit about it.

There are three major things that need to be done during the autocompletion process. First, you need to look at the context of the insertion cursor and execute a query to find all possible matches. In the background, that might mean walking a recent AST, looking up a ctags index, or walking an idex of words within the file. Different strategies are employed for different languages and desired effect.

This can take some time, depending on the method employed. Our ctags index, however, is one of the fastest out there. Even with fuzzy search.

Second, you need to sort the results. This can be expensive, and sometimes you’ll try to merge it into the first job of lookup. If you keep your index sorted, you can have sorting naturally during the search. Databases also do this, and when you want reverse sort, they just walk the index backwards.

Third, the results need to be displayed to the user. This can be very expensive for a couple reasons. One of my typical complaints was that GtkSourceCompletionContext requires a linked list of entries. (I’ve since changed my stance and quite like that this is the interface).

It’s not so bad when there are a 100 options, much worse when there are a couple thousand. Ctags can easily give us many thousands. Until you’ve typed enough text.

One might argue that we shouldn’t display 1000 items to the user, because what are they going to do with that. However, relevancy scoring is yet another layer of work. (We do score the results while searching them).

Building, traversing, and mutating linked lists is particularly expensive for sorting. Usually this is done with a merge sort (which GList does in fact do). But malloc()’ing a few bytes (24 on 64-bit, or rather 32 since they require 16 byte alignment) for each completion result only to further pointer chase is rather unhelpful. So instead we abstracted GtkSourceCompletion a bit. We now have two new data structures that allow us to almost completely avoid the pain.

IdeCompletionResults is a container for IdeCompletionItem (a GtkSourceCompletionProposal). We embed a GList node directly in the IdeCompletionItem structure (similar to what Alex Larsson did with GdkWindow a couple weeks ago). This allows us to have a linked list without an additional allocation (just our individual allocation for IdeCompletionItem proposals).

Further more, if we keep that linked list around, every character you type can continually re-filter that list of items. That means that our fuzzy search gets faster with each keystroke.

You’ll find that Clang, Ctags, Python, and Vala auto completion are much faster now and fuzzy search friendly (for 3.18.1).

Thanks for reading!