Smooth transition to new major versions of a set of libraries

With GTK+ 4 in development, it is a good time to reflect about some best-practices to handle API breaks in a library, and providing a smooth transition for the developers who will need to port their code.

But this is not just about one library breaking its API. It’s about a set of related libraries all breaking their APIs at the same time. Like what will happen with (at least a subset of) the GNOME libraries in addition to GTK+.

Smooth transition, you say?

What am I implying by “smooth transition”, exactly? If you know the principles behind code refactoring, the goal should be obvious: doing small changes in the code, one step at a time, and – more importantly – being able to compile and test the code after each step. Not in one huge commit or a branch with a lot of un-testable commits.

So, how to achieve that?

Reducing API breaks to the minimum

When developing a non-trivial feature in a library, designing a good API is a hard problem. So often, once an API is released and marked as stable, we see some possible improvements several years later. So what is usually done is to add a new API (e.g. a new function), and deprecating an old one. For a new major version of the library, all the deprecated APIs are removed, to simplify the code. So far so good.

Note that the deprecated APIs still need to work as advertised. In a lot of cases, we can just leave the code as-is. But in some other cases, the deprecated API needs to be re-implemented in terms of the new API, usually for a stateful API where the state is stored only wrt the new API.

And this is one case where library developers may be tempted to introduce the new API only in a new major version of the library, removing at the same time the old API to avoid the need to adapt the old API implementation. But please, if possible, don’t do that! Because an application would be forced to migrate to the new API at the same time as dealing with other API breaks, which we want to avoid.

So, ideally, a new major version of a library should only remove the deprecated APIs, not doing other API breaks. Or, at least, reducing to the minimum the list of the other, “real” API breaks.

Let’s look at another example: what if you want to change the signature of a function? For example adding or removing a parameter. This is an API break, right? So you might be tempted to defer that API break for the next major version. But there is another solution! Just add a new function, with a different name, and deprecate the first one. Coming up with a good name for the new function can be hard, but it should just be seen as the function “version 2”. So why not just add a “2” at the end of the function name? Like some Linux system calls: umount() -> umount2() or renameat() -> renameat2(), etc. I admit such names are a little ugly, but a developer can port a piece of code to the new function with one (or several) small, testable commit(s). The new major version of the library can rename the v2 function to the original name, since the function with the original name was deprecated and thus removed. It’s a small API break, but trivial to handle, it’s just renaming a function (a git grep or the compiler is your friend).

GTK+ timing and relation to other GNOME libraries

GTK+ 3.22 as the latest GTK+ 3 version came up a little as a surprise. It was announced quite late during the GTK+/GNOME 3.20 -> 3.22 development cycle. Other GTK+-based libraries will need to be ported to GTK+ 4, which will require a fair amount of code changes, and might force API breaks in those higher-level libraries. So what will happen is that for those libraries, a new major version will also be released, removing their own deprecated APIs, and doing other API breaks.

If you are a maintainer of one of those higher-level libraries, you might have a list of things you want to improve in the API, some corners that you find a little ugly but you never took the time to add a better API. So you think, “now is a good time” since you’ll release a new major version.

Let’s say you released libfoo 3.22 in September. If you follow the new GTK+ numbering scheme, you’ll release libfoo 3.90 in March (if everything goes well). But remember, porting an application to libfoo 3.90/4.0 should be as smooth as possible. So instead of introducing the new APIs directly in libfoo 3.90 (and removing the old, ugly APIs at the same time), you should release one more version based on GTK+ 3: libfoo 3.24. To reduce the API delta between libfoo-3 and libfoo-4.

So the unusual thing about this development cycle is that, for some libraries, there will be two new versions in March (excluding the micro/patch versions). Or, alternatively, one new version released in the middle of the development cycle. That’s what will be done for GtkSourceView, at least (the first option), and I encourage other library developers to do the same if they are in the same situation (wanting to get rid of APIs which were not yet marked as deprecated in GNOME 3.22).

Porting, one library at a time

If each library maintainer has reduced to the minimum the real API breaks, this eases greatly the work to port an application (or higher-level library).

But in the case where (1) multiple libraries all break their APIs at the same time, and (2) they are all based on the same main library (in our case GTK+), and (3) the new major version of those other libraries all depend on the new major version of the main library (in our case, libfoo 3.90/4.0 can be used only with GTK+ 3.90/4.0, not with GTK+ 3.22). Then… it’s again the mess to port an application – except with the following good practice that I will just describe!

The problem is easy but must be done in a well-defined order. So imagine that libfoo 3.24 is ready to be released (you can either release it directly, or create a branch and wait March to do the release, to follow the GNOME release schedule). What are the next steps?

  • Do not port libfoo to GTK+ 3.89/3.90 directly, stay at GTK+ 3.22.
  • Bump the major version of libfoo, making it parallel-installable with previous major versions.
  • Remove the deprecated APIs and then release libfoo 3.89.1 (development version). With a git tag and a tarball.
  • Do the (hopefully few) other API breaks and then release libfoo 3.89.2. If there are many API breaks, more than one release can be done for this step.
  • Port to GTK+ 3.89/3.90 for the subsequent releases (which may force other API breaks in libfoo).

The same for libbar.

Then, to port an application:

  • Make sure that the application doesn’t use any deprecated APIs (look at compilation warnings).
  • Test against libfoo 3.89.1.
  • Port to libfoo 3.89.2.
  • Test against libbar 3.89.1.
  • Port to libbar 3.89.2.
  • […]
  • Port to GTK+ 3.89/3.90/…/4.0.

This results in smaller and testable commits. You can compile the code, run the unit tests, run other small interactive/GUI tests, and run the final executable. All of that, in finer-grained steps. It is not hard to do, provided that each library maintainer has followed the above steps in the good order, with the git tags and tarballs so that application developers can compile the intermediate versions. Alongside a comprehensive (and comprehensible) porting guide, of course.

And you, what is your list of library development best-practices when it comes to API breaks?

Posted in GtkSourceView, Library development, Programming, Thoughts | Leave a comment

gspell and LaTeXila – progress report

In September I’ve launched two small fundraisings on gspell and LaTeXila. The two goals are now reached, thanks!

I’ve started working on those two projects, here is a progress report.

gspell – adding GtkEntry support

The basic infrastructure is there, and red wavy underlines are inserted to highlight misspelled words. The code is written in a way that it is unit-testable, and unit tests have been written.

Statistics so far:

21 files changed, 1921 insertions(+), 14 deletions(-)

Next steps:

  • The context menu (right-click menu).
  • Do not check the word currently typed.
  • Better word boundaries, to take into account apostrophes and dashes.

For all those next steps, the idea is to have common code between the GtkTextView and GtkEntry support.

LaTeXila – port to GAction/GMenu?

This progress report is a bit more technical.

To create the menu and toolbars, LaTeXila still uses the deprecated GtkUIManager and GtkAction. What I’ve done this week is investigation work, to figure out if it is possible to use GMenu (the new way menus are now usually created). GAction is great and will be used, but for GMenu it is more complicated.

The conclusion is that it will not be possible to use GMenu in LaTeXila. Because in LaTeXila, when hovering a menu item, a longer description is displayed in the status-bar – which makes the user interface self-discoverable – and this is not possible with GMenu (see this comment in bugzilla for more details). Thankfully GTK+ has a more basic way to create menus, with GtkMenuBar, GtkMenuItem etc. So that’s what LaTeXila will use.

Another reason to create a GtkMenuBar manually is to be able to add a sub-menu with the list of recently used files, with GtkRecentChooserMenu. This doesn’t exist as a GMenu, it would need to be implemented.

Also, the nice thing about GtkUIManager is that the information was encoded just once for both the menus and the toolbars. For a certain menu or toolbar item, the needed information is: the short description, the long description, the icon, the keyboard shortcut and the function to call when the action is activated. With GMenu, some of the information needs to be duplicated for creating a toolbar. So that’s another reason why not using GMenu (at least not directly).

So what I will do is to create a simplified GtkUIManager based on GAction. In other words, store the information just once, and have a convenient way to create menus and toolbars. To create a menu or toolbar item, all of what will be needed is to provide the action name; it will fetch the required missing information from a central store.

Posted in gspell, LaTeXila | 6 Comments

gspell and LaTeXila fundraisings – thanks!

The gspell fundraising has reached its initial goal! So thanks a lot for your support!

Expect GtkEntry support in the next version of gspell, which is planned for March 2017.

I’ve added a second milestone for the gspell fundraising, because there are a lot of other possible things to do.

The LaTeXila fundraising is going well too, it is currently at 80%. So thanks to the people who donated so far!

If you write documents in LaTeX, and care about having a good LaTeX editor, well maintained and well integrated to GNOME, then consider donating to LaTeXila ;) There will hopefully be other milestones in the future, for example to improve the auto-completion (e.g. complete the label in \ref commands), or implementing a full-screen mode.

Posted in gspell, LaTeXila | Leave a comment

Two Small Fundraisings – on gspell and LaTeXila

We live in a world where it’s increasingly possible to have a part-time job and being paid for other small tasks on the side; think about Uber or airbnb.

I have a half-time job, and I care about Free Software. So the natural thing to do for me is to find ways to be funded for the contributions I do.

That’s why I was really pleased to hear Alexander Larsson, at the end of his talk at GUADEC about Flatpak:


[…]
And also an interesting thing – I think actually super important – is payment. I want people to be able to pay for Free Software.

Which was met with applause. But unfortunately such support in Flatpak and GNOME Software are not going to happen anytime soon, maybe in a few years with some hope.

In the meantime, I’m launching two small fundraisings!

If this is successful, the idea is to add further milestones, after the work is done on the first one.

gspell

gspell is a library that I created last year to share the spell-checking code between gedit and LaTeXila. The code comes from the gedit spell plugin, as such there is only the support for GtkTextView. The goal of the fundraiser is to add support for GtkEntry, a single line text entry field.

Go to the gspell fundraising page if you’re interested!

LaTeXila

GTK+ 3.22, which will be released this Wednesday, will be the last GTK+ 3 stable version (see this announcement). After that, the deprecated functionality of GTK+ 3 will be removed. LaTeXila is not a new application, it is developed since 2009. A fundraising on a new piece of software can concentrate on features. For LaTeXila, what is more important is that it is well maintained, to be still relevant in the 2020’s. So the fundraiser is to make the code ready for GTK+ 4, at least as a next step.

Code maintenance is not really what we can call interesting, but it is useful. We can see this interesting/useful dichotomy quite often in the computer science world: for a researcher, is proof of correctness interesting? Yes. But in practice it is rarely applied. Is writing unit tests interesting? No, but it is very useful.

Go to the LaTeXila fundraising page if you’re convinced ;)

Posted in gspell, LaTeXila | 8 Comments

Announcing Gtef, an incubator for GtkSourceView

Gtef – the acronym for “GNOME Text Editor Framework” – is a new library that eases the development of GtkSourceView-based text editors and IDEs. It can serve as an incubator for some GtkSourceView features.

See the Gtef repo on GitHub for more details. In particular, Gtef has an interesting way to handle API (un)stability. The Gtef Roadmap is alongside the GtkSourceView one.

Posted in Gtef, GtkSourceView | 2 Comments

Using the GtkSourceView API to write scripts that manipulate text

In the gnome-c-utils repository, I wrote some scripts that use the GtkSourceView library.

When a script needs to read some text, search something in it, and possibly edit the content, then having a GtkTextBuffer is really convenient.

GtkTextBuffer is good at navigating through text, with all the GtkTextIter functions. The content can be edited easily. And the GtkSourceView library adds interesting features like regex search & replace and file loading and saving with character encoding handling (GtkTextBuffer accepts only valid UTF-8).

So, that’s it. I just wanted to share that I find it convenient to write scripts with GtkSourceView (but maybe because I speak the GtkTextView/GtkSourceView API fluently).

Posted in GtkSourceView | Leave a comment

Semi-Object-Oriented Programming in C

Although I said last year that I’m not the right person to write a book about developing GLib/GTK+ applications and libraries, I continue slowly but surely.

There is now a new section called “Semi-Object-Oriented Programming in C”. This serves as a nice warming up for GObject.

Download the PDF:

The GLib/GTK+ Development Platform (version 0.5)

See also the NEWS file to know what changed since my last blog post on the subject.

Any comments or patches are welcome (the Git repo is here).

Posted in book | 2 Comments

Thoughts on the Linux Mint X-Apps forks

You may be aware that Linux Mint has forked several GNOME applications, either directly from GNOME (Totem -> Xplayer, Evince -> Xreader, Eye of GNOME -> Xviewer), or indirectly via MATE (gedit -> pluma -> XEd).

GNOME is like the Debian of the Linux desktops. But is it a good thing? In the current state of the code, I don’t think so and I’ll explain why, with a solution: creating more shared libraries.

At the end of the day, it’s just a matter of design and usability concerns. We can safely say that the main reason behind the forks is that the Linux Mint developers don’t like the new design of GNOME applications with a GtkHeaderBar.

And there are perfectly valid reasons to not like headerbars. For gedit for example, see the list of usability regressions at this wiki page.

Currently the trend is GtkHeaderBar, but what will it be in 5 years, 10 years? Let’s face it, GNOME is here just following the new trend that came with smartphones and tablets.

So, a GNOME application developer needs to know that:

  • A GUI is an ever-changing thing, exactly like the clothes that you bought last year are already obsolete, right?
  • When the GUI changes too much, other developers don’t like it and fork the project. For valid reasons or not, this doesn’t matter.

The four X-Apps forks account for roughly 200k lines of code. In the short-term it works, Linux Mint has apps with a traditional UI. But hey, porting the code to GTK+ 4 will be another beast, because the four X-Apps still use the deprecated GtkUIManager and GtkAction APIs, among other things.

But when we look at the codebase, there are a lot of code that could be shared between a GNOME app and its fork(s). So there is a solution: creating more shared libraries. The shared libraries would contain the backend code, of course, but also some basic blocks for the UI. The application would just need to glue things up together, assembling objects, binding GObject properties to GSettings, create the main GtkWindow and a few other things.

The difference would be that instead of forking 200k lines of code, it would be forking maybe 20k lines, which is more manageable to maintain in the long term.

In the case of gedit, making its code more re-usable is exactly what I do since several years, but for another reason: being able to create easily specialized text editors or small IDEs.

Beside avoiding code duplication, creating a shared library has the nice side effect that it is much better documented (usually), and with an API browser like Devhelp, it’s a breeze to discover and understand a new codebase, it permits to have a nice overview of the classes. It’s of course possible to have such documentation for application code, but in practice few developers do that, although it would be a big step towards lowering the barrier to entry for newcomers.

When untangling some code from an application and putting it in a shared library, it is also easier to make the code unit testable (and unit tested!), or at least write a mini interactive test in case of frontend code. Making the code more stable, getting closer to bug-free code and thus more successful software.

Developing a shared library doesn’t necessarily mean to provide backward compatibility during 10 years. Nothing prevents you from bumping the major version of the library every 6 months if needed, making the new version parallel-installable with the previous major versions. So that applications are not forced to update the code when there is an API break.

But creating a library is more difficult, API design is hard. But in my opinion it is worth it. GNOME is not only a desktop environment with an application suite, it is also a development platform.

Posted in gedit, gspell, GtkSourceView, Library development, Programming, Thoughts | 12 Comments

Doing things that scale

In the software world, and with internet, we can do a lot of things that scale.

Answering a user question on IRC doesn’t scale, only one person and a few lurkers will benefit from it. Answering a user question on a mailing list scales a little better, since the answer is archived and can be searched for. What really scales is instead to improve the reference manual. Yes, you know, documentation. That way, normally the question will not surface again. You have written the documentation once and N readers benefit from it, regardless of N. It scales!

Another example. Writing an application versus writing a library. Writing an application or end-user software already scales in the sense that the programmer writes the code once, but it can be copied and installed N times for almost the same cost as one time. Writing a library scales more, since it can benefit N programs, and thus N^2 users if we use the asymptotic notation. It’s no longer linear, it’s quadratic! The lower-level the library, the more it scales.

Providing your work under a free/libre license scales more, since it can be re-distributed freely on the internet. There are some start-ups in the world that reinvent the wheel by writing proprietary software and provide technical support in their region only. By writing a free software instead, other start-ups can pop up in other regions of the world, and will start contributing to your codebase (if it’s a copyleft license, that is). For the human being advancement, free software is better since it more easily permits to not reinvent the wheel. Provided that it’s a high-quality software.

This can go on and on. You get the picture. But doing something that scales is generally more difficult. It takes more time in the short-term, but in the long-term everybody wins.

Posted in Library development, Programming, Thoughts | Comments Off on Doing things that scale

API vs ABI

I repeatedly see other people doing the mistake, so a little reminder doesn’t hurt.

  • API: Application Programming Interface
  • ABI: Application Binary Interface

The difference can be easily explained by knowing what to do for some code when the API or ABI breaks in a library that the code depends on:

  • If only the ABI breaks, then you just need to re-compile the code.
  • If the API breaks, you need to adapt the code.
  • When the code doesn’t compile anymore, you need to adapt it, so it’s both an API and ABI break.
  • When the code still compiles fine but you need to adapt it, it’s only an API break.

Example of an ABI break only: when the size of a public struct changes.

Example of an ABI and API break: when a function is renamed.

Example of an API break only: CSS in GTK+ 3.20, or when a function doesn’t do the same as what was documented before (in an incompatible way).

That’s it.

Posted in Library development, Programming | 6 Comments