Builder 3.18
It’s been an extremely busy couple of months since we last saw each other at GUADEC. During that time, I’ve been feverishly getting 3.18 out the door, and preparing some special bits for 3.18.1.
Coverity Scans
Garrett Regier got us signed up for coverity, so we will start doing regular scans of Builder and as many of our depenent libraries as we can. I did the first batch of those yesterday. GLib and Gtk+, for example, already do this occasionally. It’s a free (as in beer) service to open source projects, and I’d consider it if you are writing C or C++ regularly.
Joining Red Hat
The real news, however, is that I’ve accepted a wonderful role at Red Hat. I’ll be focusing on the Xdg-App developer story, and Builder is an important part of that. We want to make it as easy as possible for you to create and deploy software that users can trust.
This was a difficult choice for me, as I had a few offers from people I respect greatly. Ultimately, I think Red Hat is the right place to push two hugely important projects forward. They were respectful of my desire to continue living like a Gnome in the forest, so I’ll be remote.
The GNOME Foundation’s Board of Directors already has two people who happen to be employed at Red Hat. Two is the maximum number from any single employer as set out in our bylaws. With much sadness, I have therefore resigned from the Board of Directors. I want to thank everyone who voted for me. I’m sure I’ll have an opportunity to serve you again soon enough. The Board was notified a couple of weeks ago and has started the process to replace me.
This week I have orientation and then I’m headed up to Boston for the summit over the long weekend.
Boston GNOME Summit
Saturday is the start of the Boston GNOME Summit! I hope those of you in the area can come say hi and visit for a bit. I’ll be demoing some Builder features and taking feature requests.
One thing I’d like to see this year is more people join us on IRC to be part of the event even if they can’t make it in person. Usually we are in #summit on irc.gimp.net.
I’d also like to say thank you to everyone who has been putting together this years summit. It’s a lot more work than you would think.
Builder Campaign
I’ve been so focused on writing software that other aspects of the campaign were not tended to in the manner I envisioned. If I were to ever take on a project of this magnitude again, I would definitely not go it alone. I would highly recommend having at least a second person in place that is dedicated to communication and administrative tasks. They are a huge time sink and really would have prevented me from pushing the software forward as fast as I did.
If you do not see your name in the Builder credits, and you purchased a perk that included that, please contact me (or submit a patch). Chances are that you selected the “anonymous donation” box on indiegogo. While I can see your name, I’m legally restricted from mentioning it (such as in the credits).
So… All of you who are waiting on perks from me, they are coming. I owe you more than just software and I’ll be pushing hard on that front over the next month to finish the preparations and shipments. My goal is to have the campaign perks completed and shipped by end of November (Thanksgiving time here).
New UI styling
It’s no secret that we are working towards an ambitious UI design. Where we are today is not a product of being where we want to be, but a step on the iterative process to where we are going.
One thing you’ll notice is that the floating bar is much less intrusive now.
Sébastien Lafargue landed support in Gtk+ for margins in GtkTextView. We are using this to allow you to scroll past the bottom of the document. (It was annoying editing lines along the bottom of the window). However, there is still more to do. Rather than using margin to achieve this, we’d like to add support in GtkTextView to allow us to scroll the last row to the top of the view. Vim users are familiar with this using `ctrl+e` to scroll.
Workbench Panels
The workbench has gained some panels. Of course, as I said above, this is intermediate design. We aren’t totally sure on final designs yet, but we needed something to move forward.
One thing that was particularly interesting to optimize was animating panels. Most IDEs cheat here and make them floating. Why? Transience is one, but another that it is difficult to do smoothly. It’s hard to manage 60 FPS animations if you have relayout and redraw cycles to perform on every frame. I was particularly careful about avoiding gtk_widget_queue_relayout() and ensuring we keep as much content in the GtkPixelCache as possible.
The source editor is one area that is very tricky to get right for animations. Editors are a tricky business. When you change the horizontal sizing of a GtkTextView, we have to discard the pixel cache because we might have right aligned text that needs to be moved. One way we can get around this in the future is to track if we have center or right aligned text in the BTree (where we store text tags and such). Upon an allocation change, we would have a fast lookup to determine if we need to destroy the cache. However, in the meantime, I employed some dirty tricks that are so dirty, you’ll have to read the source code to find out.
The symbol tree has been pulled out of the header (which was hidden behind a popover) and now has prominent viewing in the right-hand sidebar. Documentation is also now browsable and searchable from the sidebar.
We have a terminal in the bottom panel or `ctrl+shift+t` to have one inline with your editor. I find I’m switching to gnome-terminal a lot less now.
I decided to leave sysmon, a “debug” feature, in the release because I find it both beautifully smooth and useful for keeping me honest about CPU usage. If I really screw something up in Builder, I see it pretty immediately. Perhaps it’s the motivational factor of why Builder feels like such a lightweight editor despite being powered by IDE-like compiler features.
These scrolling CPU graphs were based on ideas I discussed in a blog series a few years back. They don’t do the ring buffer drawing, but I’ll add that back at some point. The code is in `contrib/rg` if you are interested. At least one other project is using the code today too.
Shortcuts
Many of you saw my screenshots for the new shortcut view which Allan so graciously hurried a design for (in the last couple weeks of the GNOME 3.18 cycle). It was one of our biggest complaints and I felt it had to be done even if it meant major code changes landing at the last minute.
OS X Support
While we in no way “officially” support OS X, you can now build Builder on OS X if you are so inclined. (It also works on FreeBSD and OpenBSD thanks to some gracious patches on Bugzilla).
It turns out cairo-quartz needs some love. You can find my cairo tree with various forward-ported patches from Mozilla, including CGLayer support on my github. It also has fixes for HiDPI when using GtkPixelCache (which I got from cairo bugzilla).
Something interesting I found was that colorspace conversion was happening on every frame draw. This is some seriously slow code. Both cairo and gdk-quartz were using the RGB colorspace, which you would think is fine. Apparently not. Instead, I changed both cairo and gtk to use the colorspace of the primary monitor to avoid that conversion altogether.
I’m curious what others do. I guess external monitors just always take the extra color conversion cost? Almost everything has a color profile on OS X it seems. Perhaps native applications adjust their colorspace when the window crosses a monitor boundary? They seem to employ similar tricks with DPI at least.
I was curious to learn more about how GdkFrameClock was implemented and wondering if it would help us with our frame rate on quartz. I wrote one of those too. You can find it on my gtk+ branch at wip/chergert/quartz-frame-clock.
It uses CVDisplayLink to thaw the GdkFrameClock when a frame needs to be drawn.
On my test machine, I was able to get two finger scroll on a retina 13″ macbook pro up to about 45 FPS. Still short of the “buttery smooth” feeling of TextEdit, but further progress can be made. This morning I did a few tests before heading out east, and we seem to be drawing fast enough. So the issue is probably somewhere else. More investigation needed.
Adding Support for Vala (in Vala)
One of the most requested features from the Builder campaign was support for Vala. Once 3.18 was out the door, I finally had a chance to sit down and see what that would look like.
Thankfully, the Anjuta codebase already had working code for auto-completion. So most of the work was a matter of adapting that code to fit into the compiler toolchain and editor design we use in Builder.
I actually prototyped this in C, since the vala compiler of course compiles to header files. But I soon realized it was not code I would want to be the sole maintainer of. So I rewrote those bits in Vala and now the vala-pack plugin is our first plugin written in Vala. That means that you too can start writing plugins in Vala.
I do have concerns about memory leaks though. I find it particularly difficult to track those down compared to our C code. We have robust counters in Builder for tracking this type of stuff, but they aren’t usable from Vala yet.
I assume it’s in our usage of libvala for auto-completion, but I’ve not had the chance to investigate. I would love for the Vala community to help us make Builder the best tool for writing software in Vala. In particular, we still need build system integration to discover compiler flags like –pkg and such.
Python Auto Completion
We have auto-completion for Python as well. It’s powered by python-jedi but also knows about GObject Introspection. One caveat right now, by the very nature that jedi is running in process, is that this is for python3 only. We need to add support for moving the auto-completion out of process to support both python2 and python3. Thanks much to Elad Alfassa for implementing GObject Introspection and fixing my broken python.
Auto Indenter Designs
After having written four auto indenters, I started to see an abstraction that would make things easier. So this week I started on a new Python indenter which I hope to land soon. It’s design seems novel, but I’m not sure if it has been done this way before. Probably, nothing is new under the sun.
The first phase is to scan backwards for various points of interest (not new). While doing that, we collect information about things we discover. Such as things like:
- This looks like a class start
- This looks like a function start
- This looks like the start of a comment
- This is a tuple, dict, list, etc
- Here is a block (if, elif, else, with, while, etc)
Then, we can match rules to selectors based on what we found.
For example, to determine if we are in a function parameter list, we might do something like:
match = selector(Rank.CLASS, Rank.FUNCTION, Rank.TUPLE)
We can do the same to discover if we are an `elif` or `else` nested inside an `if` block. That might indicate we should unindent the row.
I assume after some battle testing, I’ll port the C, XML, and Vala indenters to do the same. I’m sure there is a further abstraction to help us simplify these more.
Making Auto-Completion Fast
Those that follow my snark on twitter know that I spent some time this week on improving our auto-completion story. That was some fun work, so I’ll share a bit about it.
There are three major things that need to be done during the autocompletion process. First, you need to look at the context of the insertion cursor and execute a query to find all possible matches. In the background, that might mean walking a recent AST, looking up a ctags index, or walking an idex of words within the file. Different strategies are employed for different languages and desired effect.
This can take some time, depending on the method employed. Our ctags index, however, is one of the fastest out there. Even with fuzzy search.
Second, you need to sort the results. This can be expensive, and sometimes you’ll try to merge it into the first job of lookup. If you keep your index sorted, you can have sorting naturally during the search. Databases also do this, and when you want reverse sort, they just walk the index backwards.
Third, the results need to be displayed to the user. This can be very expensive for a couple reasons. One of my typical complaints was that GtkSourceCompletionContext requires a linked list of entries. (I’ve since changed my stance and quite like that this is the interface).
It’s not so bad when there are a 100 options, much worse when there are a couple thousand. Ctags can easily give us many thousands. Until you’ve typed enough text.
One might argue that we shouldn’t display 1000 items to the user, because what are they going to do with that. However, relevancy scoring is yet another layer of work. (We do score the results while searching them).
Building, traversing, and mutating linked lists is particularly expensive for sorting. Usually this is done with a merge sort (which GList does in fact do). But malloc()’ing a few bytes (24 on 64-bit, or rather 32 since they require 16 byte alignment) for each completion result only to further pointer chase is rather unhelpful. So instead we abstracted GtkSourceCompletion a bit. We now have two new data structures that allow us to almost completely avoid the pain.
IdeCompletionResults is a container for IdeCompletionItem (a GtkSourceCompletionProposal). We embed a GList node directly in the IdeCompletionItem structure (similar to what Alex Larsson did with GdkWindow a couple weeks ago). This allows us to have a linked list without an additional allocation (just our individual allocation for IdeCompletionItem proposals).
Further more, if we keep that linked list around, every character you type can continually re-filter that list of items. That means that our fuzzy search gets faster with each keystroke.
You’ll find that Clang, Ctags, Python, and Vala auto completion are much faster now and fuzzy search friendly (for 3.18.1).
Thanks for reading!