Happenings in GNOME

Did you know that Builder has a build panel finally? That’s right, you can actually do what the name says now. Of course, we only have one IdeBuildSystem implementation at the moment (autotools), but if you implement that interface, you too can have your project building.

Note that you can write build system implementations in Python or Vala now. That should lower the barrier for contributions quite a bit.

Never mind the CSS errors in the theme, those will be fixed soon enough 😉

One of the things necessary in an IDE is the ability to create projects from templates. This is something we’ve been lacking since the beginning, and I’m finally in a position to follow through on it.

So, templates. What is a template? We have a couple layers of templates needed. We have, at a macro level, project templates. These are a collections of files, that get tweaked based on the project options.

Then we have per-file templates. These need to go through various tweaking and can get quite detailed, based on how much you want to do for the user.

I did a quick survey of what was available to reuse, and was not entire satisfied with the options. Thankfully I like programming languages and compilers, and decided to create exactly what I was hoping to find.

The result, was written over the past few days, and is called template-glib. It will look familiar to people who’ve used other template engines. Bits and pieces had inspiration from all over the place. I rather like python’s mako template API, but not much of a fan of the implementation. I also don’t want a bidirectional run-time^[1] involved.

Some people really seem to like Go’s text templating. I’ve never used it, but like any language community full of crusaders, they’ll come to your door to let you know why they love it so much. Some things will be familiar to those people.

The API is designed in such a way that parsing and expanding the template are separate operations. If you were to use it in such a fashion, the parsing operation will be much less expensive.

Expanding of templates uses a TmplScope which can have inheritance. That allows for some nifty scoping control as you work your way down. I haven’t finished this to the level I’d like, but “it works”.

You can control resolving of included files using TmplTemplateLocator. This is similar to python mako’s TemplateLookup. Works as advertised.

You can include files, like:
{{include "foo.tmpl"}}

You can cycle through certain types of objects
{{for item in my_list}}
{{item.title}}
{{end}}

You can perform various expressions, including assignment.
{{x=sqrt(10)*sqrt(20)}}

You can call into methods via GObject Introspection.
{{item.short_description("with", "params")}}

You can use expressions in conditionals.
{{if item.year > 2016}}
Current: {{item.title}}
{{else}}
Archived: {{item.title}}
{{end}}


This code is very early, in fact less than a week old, but I’ll keep pushing it forward to add the features we need in Builder. Perhaps you’ll find it useful too. I’d certainly love to see someone make a nice GNOME 3 static blog generator.

You can find the lexer here and the parser here for the expression language.

[1] Bidirectional run-time meaning that objects can live en either the run-time or outside of it, and coordination of GC must occur.

There is one thing that has been important to me for a long time in Builder, but I’ve been unable to solve the problem correctly until recently. And that is command line tools.

I’m not naive enough to believe that I can convince every multi-decade Emacs and Vim user to switch to a Gtk-based application. Even if our emulation of such systems are perfect, the muscle-memory of launching them from the command line is entrenched in work-flows. Because of that, I wanted to export many of our tools via command line tools.

In git, Builder now has support for an ide command line tool. The goal is for plugins to export their features via a command line tool. The added benefit is that it can provide an easier way to test core functionality of your plugin.

These aren’t all finished, but this gives you an idea of where we are going.


# list available commands
ide --list-commands

# -d allows selecting target device
ide build -j 12 -d local

# clone/bootstrap/build, prepares for contribution
ide contribute gtk+

# create a shared library project
ide create-project foo-glib --template=library --with-tests


You get the idea. None of those work as advertised, but you know it goes, that’s the plan.

I’m nearly a month down on a branch for Builder 3.20. It’s goal is to radically simplify the process of creating plugins, and prepare for external plugins. We really wanted to create a solid plugin story before doing that and things are progressing nicely.

However, I wanted to point out I put together a short(ish) video on creating your first GObject in C. It does expect some familiarity with C. You can find the video on Youtube, or alternatively, I applaud your use of youtube-dl.

youtube-dl https://www.youtube.com/watch?v=1-_EBEr0fxI

I hope to do more of these videos (with better planning, storyboarding, and production quality) as time permits. I’d also like companion material (latex chapters or similar) if possible.

And I’ll leave you with a few screenshots to whet your appetite. All UI elements are non-final. We’ve been iterating and playing with a few ideas to see what sticks.

Builder 3.18

It’s been an extremely busy couple of months since we last saw each other at GUADEC. During that time, I’ve been feverishly getting 3.18 out the door, and preparing some special bits for 3.18.1.

Coverity Scans

Garrett Regier got us signed up for coverity, so we will start doing regular scans of Builder and as many of our depenent libraries as we can. I did the first batch of those yesterday. GLib and Gtk+, for example, already do this occasionally. It’s a free (as in beer) service to open source projects, and I’d consider it if you are writing C or C++ regularly.

Joining Red Hat

The real news, however, is that I’ve accepted a wonderful role at Red Hat. I’ll be focusing on the Xdg-App developer story, and Builder is an important part of that. We want to make it as easy as possible for you to create and deploy software that users can trust.

This was a difficult choice for me, as I had a few offers from people I respect greatly. Ultimately, I think Red Hat is the right place to push two hugely important projects forward. They were respectful of my desire to continue living like a Gnome in the forest, so I’ll be remote.

The GNOME Foundation’s Board of Directors already has two people who happen to be employed at Red Hat. Two is the maximum number from any single employer as set out in our bylaws. With much sadness, I have therefore resigned from the Board of Directors. I want to thank everyone who voted for me. I’m sure I’ll have an opportunity to serve you again soon enough. The Board was notified a couple of weeks ago and has started the process to replace me.

This week I have orientation and then I’m headed up to Boston for the summit over the long weekend.

Boston GNOME Summit

Saturday is the start of the Boston GNOME Summit! I hope those of you in the area can come say hi and visit for a bit. I’ll be demoing some Builder features and taking feature requests.

One thing I’d like to see this year is more people join us on IRC to be part of the event even if they can’t make it in person. Usually we are in #summit on irc.gimp.net.

I’d also like to say thank you to everyone who has been putting together this years summit. It’s a lot more work than you would think.

Builder Campaign

I’ve been so focused on writing software that other aspects of the campaign were not tended to in the manner I envisioned. If I were to ever take on a project of this magnitude again, I would definitely not go it alone. I would highly recommend having at least a second person in place that is dedicated to communication and administrative tasks. They are a huge time sink and really would have prevented me from pushing the software forward as fast as I did.

If you do not see your name in the Builder credits, and you purchased a perk that included that, please contact me (or submit a patch). Chances are that you selected the “anonymous donation” box on indiegogo. While I can see your name, I’m legally restricted from mentioning it (such as in the credits).

So… All of you who are waiting on perks from me, they are coming. I owe you more than just software and I’ll be pushing hard on that front over the next month to finish the preparations and shipments. My goal is to have the campaign perks completed and shipped by end of November (Thanksgiving time here).

New UI styling

It’s no secret that we are working towards an ambitious UI design. Where we are today is not a product of being where we want to be, but a step on the iterative process to where we are going.

One thing you’ll notice is that the floating bar is much less intrusive now.

Sébastien Lafargue landed support in Gtk+ for margins in GtkTextView. We are using this to allow you to scroll past the bottom of the document. (It was annoying editing lines along the bottom of the window). However, there is still more to do. Rather than using margin to achieve this, we’d like to add support in GtkTextView to allow us to scroll the last row to the top of the view. Vim users are familiar with this using `ctrl+e` to scroll.

Workbench Panels

The workbench has gained some panels. Of course, as I said above, this is intermediate design. We aren’t totally sure on final designs yet, but we needed something to move forward.

One thing that was particularly interesting to optimize was animating panels. Most IDEs cheat here and make them floating. Why? Transience is one, but another that it is difficult to do smoothly. It’s hard to manage 60 FPS animations if you have relayout and redraw cycles to perform on every frame. I was particularly careful about avoiding gtk_widget_queue_relayout() and ensuring we keep as much content in the GtkPixelCache as possible.

The source editor is one area that is very tricky to get right for animations. Editors are a tricky business. When you change the horizontal sizing of a GtkTextView, we have to discard the pixel cache because we might have right aligned text that needs to be moved. One way we can get around this in the future is to track if we have center or right aligned text in the BTree (where we store text tags and such). Upon an allocation change, we would have a fast lookup to determine if we need to destroy the cache. However, in the meantime, I employed some dirty tricks that are so dirty, you’ll have to read the source code to find out.

The symbol tree has been pulled out of the header (which was hidden behind a popover) and now has prominent viewing in the right-hand sidebar. Documentation is also now browsable and searchable from the sidebar.

We have a terminal in the bottom panel or `ctrl+shift+t` to have one inline with your editor. I find I’m switching to gnome-terminal a lot less now.

I decided to leave sysmon, a “debug” feature, in the release because I find it both beautifully smooth and useful for keeping me honest about CPU usage. If I really screw something up in Builder, I see it pretty immediately. Perhaps it’s the motivational factor of why Builder feels like such a lightweight editor despite being powered by IDE-like compiler features.

These scrolling CPU graphs were based on ideas I discussed in a blog series a few years back. They don’t do the ring buffer drawing, but I’ll add that back at some point. The code is in `contrib/rg` if you are interested. At least one other project is using the code today too.

Shortcuts

Many of you saw my screenshots for the new shortcut view which Allan so graciously hurried a design for (in the last couple weeks of the GNOME 3.18 cycle). It was one of our biggest complaints and I felt it had to be done even if it meant major code changes landing at the last minute.

OS X Support

While we in no way “officially” support OS X, you can now build Builder on OS X if you are so inclined. (It also works on FreeBSD and OpenBSD thanks to some gracious patches on Bugzilla).

It turns out cairo-quartz needs some love. You can find my cairo tree with various forward-ported patches from Mozilla, including CGLayer support on my github. It also has fixes for HiDPI when using GtkPixelCache (which I got from cairo bugzilla).

Something interesting I found was that colorspace conversion was happening on every frame draw. This is some seriously slow code. Both cairo and gdk-quartz were using the RGB colorspace, which you would think is fine. Apparently not. Instead, I changed both cairo and gtk to use the colorspace of the primary monitor to avoid that conversion altogether.

I’m curious what others do. I guess external monitors just always take the extra color conversion cost? Almost everything has a color profile on OS X it seems. Perhaps native applications adjust their colorspace when the window crosses a monitor boundary? They seem to employ similar tricks with DPI at least.

I was curious to learn more about how GdkFrameClock was implemented and wondering if it would help us with our frame rate on quartz. I wrote one of those too. You can find it on my gtk+ branch at wip/chergert/quartz-frame-clock.
It uses CVDisplayLink to thaw the GdkFrameClock when a frame needs to be drawn.

On my test machine, I was able to get two finger scroll on a retina 13″ macbook pro up to about 45 FPS. Still short of the “buttery smooth” feeling of TextEdit, but further progress can be made. This morning I did a few tests before heading out east, and we seem to be drawing fast enough. So the issue is probably somewhere else. More investigation needed.

Adding Support for Vala (in Vala)

One of the most requested features from the Builder campaign was support for Vala. Once 3.18 was out the door, I finally had a chance to sit down and see what that would look like.

Thankfully, the Anjuta codebase already had working code for auto-completion. So most of the work was a matter of adapting that code to fit into the compiler toolchain and editor design we use in Builder.

I actually prototyped this in C, since the vala compiler of course compiles to header files. But I soon realized it was not code I would want to be the sole maintainer of. So I rewrote those bits in Vala and now the vala-pack plugin is our first plugin written in Vala. That means that you too can start writing plugins in Vala.

I do have concerns about memory leaks though. I find it particularly difficult to track those down compared to our C code. We have robust counters in Builder for tracking this type of stuff, but they aren’t usable from Vala yet.

I assume it’s in our usage of libvala for auto-completion, but I’ve not had the chance to investigate. I would love for the Vala community to help us make Builder the best tool for writing software in Vala. In particular, we still need build system integration to discover compiler flags like –pkg and such.

Python Auto Completion

We have auto-completion for Python as well. It’s powered by python-jedi but also knows about GObject Introspection. One caveat right now, by the very nature that jedi is running in process, is that this is for python3 only. We need to add support for moving the auto-completion out of process to support both python2 and python3. Thanks much to Elad Alfassa for implementing GObject Introspection and fixing my broken python.

Auto Indenter Designs

After having written four auto indenters, I started to see an abstraction that would make things easier. So this week I started on a new Python indenter which I hope to land soon. It’s design seems novel, but I’m not sure if it has been done this way before. Probably, nothing is new under the sun.

The first phase is to scan backwards for various points of interest (not new). While doing that, we collect information about things we discover. Such as things like:

• This looks like a class start
• This looks like a function start
• This looks like the start of a comment
• This is a tuple, dict, list, etc
• Here is a block (if, elif, else, with, while, etc)

Then, we can match rules to selectors based on what we found.
For example, to determine if we are in a function parameter list, we might do something like:

match = selector(Rank.CLASS, Rank.FUNCTION, Rank.TUPLE)

We can do the same to discover if we are an `elif` or `else` nested inside an `if` block. That might indicate we should unindent the row.

I assume after some battle testing, I’ll port the C, XML, and Vala indenters to do the same. I’m sure there is a further abstraction to help us simplify these more.

Making Auto-Completion Fast

Those that follow my snark on twitter know that I spent some time this week on improving our auto-completion story. That was some fun work, so I’ll share a bit about it.

There are three major things that need to be done during the autocompletion process. First, you need to look at the context of the insertion cursor and execute a query to find all possible matches. In the background, that might mean walking a recent AST, looking up a ctags index, or walking an idex of words within the file. Different strategies are employed for different languages and desired effect.

This can take some time, depending on the method employed. Our ctags index, however, is one of the fastest out there. Even with fuzzy search.

Second, you need to sort the results. This can be expensive, and sometimes you’ll try to merge it into the first job of lookup. If you keep your index sorted, you can have sorting naturally during the search. Databases also do this, and when you want reverse sort, they just walk the index backwards.

Third, the results need to be displayed to the user. This can be very expensive for a couple reasons. One of my typical complaints was that GtkSourceCompletionContext requires a linked list of entries. (I’ve since changed my stance and quite like that this is the interface).

It’s not so bad when there are a 100 options, much worse when there are a couple thousand. Ctags can easily give us many thousands. Until you’ve typed enough text.

One might argue that we shouldn’t display 1000 items to the user, because what are they going to do with that. However, relevancy scoring is yet another layer of work. (We do score the results while searching them).

Building, traversing, and mutating linked lists is particularly expensive for sorting. Usually this is done with a merge sort (which GList does in fact do). But malloc()’ing a few bytes (24 on 64-bit, or rather 32 since they require 16 byte alignment) for each completion result only to further pointer chase is rather unhelpful. So instead we abstracted GtkSourceCompletion a bit. We now have two new data structures that allow us to almost completely avoid the pain.

IdeCompletionResults is a container for IdeCompletionItem (a GtkSourceCompletionProposal). We embed a GList node directly in the IdeCompletionItem structure (similar to what Alex Larsson did with GdkWindow a couple weeks ago). This allows us to have a linked list without an additional allocation (just our individual allocation for IdeCompletionItem proposals).

Further more, if we keep that linked list around, every character you type can continually re-filter that list of items. That means that our fuzzy search gets faster with each keystroke.

You’ll find that Clang, Ctags, Python, and Vala auto completion are much faster now and fuzzy search friendly (for 3.18.1).

Thanks for reading!

Many of you have asked for help with keyboard shortcuts in Builder. It was always something we wanted to do, but I was humbly waiting for upstream to get that into the toolkit so we would get it for “free”.

Since that didn’t happen this cycle, I rushed Allan to put together an updated design and I set out to implement it for 3.18. You’ll notice type-ahead search and a nice visual style.

The original “hack” of an implementation used XMACROs. For those of you that write C and don’t know what that is, look it up. I’ll wait. Now that you can see the shock and horror, you’ll be happy to know I didn’t go that route in the long term. The code isn’t ready to be generally usable, but it’s in better shape now that I added a custom GtkBuilder parser. Example here.

There is still much to do, but this will help you get started!

Early on in the Builder development road-map, I discussed how I didn’t want to use a GtkTreeView for a sidebar (or much of a sidebar at all). But everything changes when you need to ship software.

I’ve been working on some ideas for a replacement file browser based on some immature designs. But at the moment, they are very much just that. Ideas.

For those who have jhbuild installs, I’d love for you to try Builder’s “wip/path-bar” branch. It’s not an accurately named branch, but oh well. In particular, try the “tests/test-stack-list” program and see how you like it from a usability standpoint. Everything has trade-offs, and I’d like to inform myself before pushing further on the design.

cd tests
make test-stack-list
./test-stack-list

You can watch a video of what it looks like if you don’t have a Builder checkout. Sorry for the youtube link, I don’t want to host the bandwidth. Alternatively, the following works.

youtube-dl https://www.youtube.com/watch?v=GYox__Wwatg
totem “hacks on a plane-GYox__Wwatg.mp4”