I’ve finally convinced myself that I need to make a Pango 2.0 release to clean up the API, and introduce some new APIs without breaking users that expect Pango to be very stable.

So, here it is… well not quite. What I am presenting today is not Pango 2.0  yet,  but 1.90 – an unstable preview of the coming changes, to gather feedback and give some heads-up about whats coming.

Whats changed?

Pango is now shipped as a single shared object, libpango-2.so, which contains the high-level cross-platform code as well as platform-specific fontmap implementations and the cairo support (if it is enabled). All of the APIs have been cleaned up and modernized.

PangoFontMap  has seen some significant changes. It is now possible to instantiate a PangoFontMap, and populate it manually with PangoFontFamily and PangoFontFace objects.

There are still platform-specific subclasses

•  PangoFcFontMap
• PangoCoreTextFontMap
• PangoDirectWriteFontMap

which will use platform APIs to enumerate fonts and populate the fontmap.

Whats new?

PangoLineBreaker is the core of pango’s line-breaking algorithm,
broken out from PangoLayout. Having this available independent
from PangoLayout will facilitate uses such as multi-column
layout, text flow between frames and shaping paragraphs around
images.

Here is an example that shows changing the column width mid-paragraph:

PangoLines is the ‘formatted output’ part of a PangoLayout, and can be used to collect the output of a PangoLineBreaker.

PangoHbFont is a font implementation that is a thin wrapper around HarfBuzz font and face objects. This the way in which Pango handles fonts on all platforms now.

PangoUserFont is a  callback-based font implementation to allow for entirely application-defined font handling, including glyph drawing. This is similar to cairo user fonts, where this example was borrowed:

Many smaller changes, such as better control over line height with line-height attributes and control over the trimming of leading, or guaranteed font ↔ description roundtrips with face-ids.

How can I try this?

The Pango code lives on the pango2 branch, and there is a corresponding pango2 branch of GTK, which contains a port of GTK to the new APIs.

The tarballs are here.

Summary

If you have an interest in text rendering, please try this out and tell us what you think. Your feedback will make Pango 2 better.

For more details about the changes in this release, see the NEWS, and have a look at the migration guide.

If you want to learn more about the history of Pango and the background for some of these changes, come to my Guadec talk in Guadalajara!

I was hoping to wrap up my Pango work after the previous update, but unexpected trouble came in from the side – Benjamin made GtkLabel more serious about height-for-width, and that uncovered some inaccuracies in Pango’s line wrapping implementation. Sometimes, we would make our lines shorter than necessary, and sometimes, we would let a hyphen leak out of the allotted width, creating an overlong line.

Fixing all this up took some serious effort,  but I think it was time well spent. One of the outcomes is that Pango now has APIs to serialize PangoLayout objects, and these are used  in the testsuite.

A Layout Editor

To get some (visual) insight into what was going wrong with line breaking, I wrote a quick utility called layout-editor. This is how it looks:

It lets you tweak all the parameters of a PangoLayout object and shows you the results of your changes. It can also show details about pango’s analysis of the text. And it can overlay extra information, such as extents of lines, runs, glyphs, caret positions, and more.

Since the layout editor also uses the serialization APIs to load and save your layouts, you can directly use it to inspect the test cases in Pango’s testsuite and create new ones. This should help improve test coverage, going forward.

If you want to gain more insight into what is happening inside Pango,  this tool might be for you.

Better Tabs

With this new tool in hands, I felt the urge to see if it can help for feature development too. One long-standing feature gap in Pango is the lack of support for tab stops with alignments other than left.

Amazingly, an almost 15 year old patch found in this bug still mostly applied, and worked, after some small adaptations. The new tool was indeed very helpful in working out some of the finer points.

If you always felt like you should be able to line up numbers properly at their decimal point, instead of picking a monospace font and hoping for the best, voila! now you can.

Wrapping up

All of this is available in Pango 1.50. Enjoy

Here is another update on what will appear in Pango 1.50 soon. Quite a few things, as it turns out. (This is a continuation of my last Pango update)

Bidi Improvements

Pango has long claimed to have good support for bidirectional text rendering and editing. But until recently, there were some annoying bugs: you could end up in loops when moving the cursor through mixed-direction text.

The relevant code has been rewritten, closing a very old bug (#157).

Useful Information

Modern fonts can contain a lot of different information – there can be colormaps, and glyphs can be represented not just as splines, but also as pngs or svgs.

For GTK, an important piece of information for each glyph is whether it is colored or monochrome. With the new PangoGlyphVisAttr.is_color field,  GTK no longer needs to poke directly at the font data to find out.

Another case where GTK needed data that wasn’t available through Pango APIs has been closed with pango_font_get_languages(). This data is used in the font chooser filter popup:

Superscripts and subscripts

Superscripts and subscripts now use font metrics information for size changes and baseline shift.  They can also be properly nested, so markup like

2<sup>2<sup>2</sup></sup>

yields the expected rendering:

Customizing segmentation

Pango determines word and sentence boundaries according to the Unicode Text Segmentation Spec (TR29). The specification can’t deal with all the complexities of formatted text, so sometimes its results need tailoring, which can now be done with the new word and sentence attributes:

$ pango-segmentation --kind=word --text=\
    "<span segment='word'>1-based</span> index"

|1-based| |index|

Better Markup

Besides the new attributes that have been mentioned, Pango markup now lets you specify many attributes in more natural units.

For example,  you can now say

<span font_size='12.5pt'>

instead of

<span font_size='12800'>

Small Caps

Pango has had the PangoVariant enumeration with its PANGO_VARIANT_SMALL_CAPS value since forever, but it has never done anything. Since we’ve added support for OpenType features (in Pango 1.37), it has been possible to produce Small Caps by using the smcp=1 feature, if the font supports it. Sadly, most fonts don’t.

With the text transformation and font scaling infrastructure now in place, it was easy to emulate Small Caps for fonts that don’t  support this natively:

While at it, we’ve expanded the PangoVariant enumeration to cover all the CSS casing variants, and made the GTK CSS engine use them instead of OpenType features.

Better debugging

Pango ships with a versatile test utility called pango-view, which has options to test many of Pangos layout features. It has recently learned to show more auxiliary information in visual form, like glyph extents, caret positions and slopes:

$ pango-view --text Boxes \ --annotate=glyph,caret,slope

Enjoy!

I’ve spent some time on Pango, recently. Here is a little update on the feature work that I’ve done there. All of these changes will appear in Pango 1.50 and GTK 4.6.

The general directions of this work are:

• Take advantage of the fact that we are now using harfbuzz on all platforms. Among other things, this gives us much easier access to font information.
• Match CSS where it makes sense. If nothing else, this makes it much easier to connect new Pango features to the CSS machinery in GTK.

CSS features

Lets start with the second point: matching CSS.

Line spacing has historically been a bit painful in GtkTextView. You can set distances before and after paragraphs, and between wrapped lines inside a paragraph. But this does not take font sizes into account—it is a fixed number of pixels.

A while ago, I added a line-spacing factor to Pango, which was meant to help with the font size dependency. You basically tell Pango: I want the baselines of this paragraph spaced apart 1.33 times as wide as they would normally be. The remaining problem is that Pango handles text one paragraph at a time. So as far as it is concerned, there is not previous baseline above the first line in a paragraph, and it does not increase the spacing between paragraphs.

The CSS solution to this problem is to just make the lines themselves taller, and place them flush next to each other.  With this approach, you need to be a little careful to make sure that you still get consistent baseline-to-baseline distances. But at least it solves the paragraph spacing issue.

Pango recently gained a line-height attribute that does just that, and GTK now supports the corresponding CSS property.

Another feature that has come to Pango from the CSS side is support for text transformation (also in the screenshot). This lets you change the capitalization of text. Note that this is just for presentation purposes—if you select STRASSE in the example and copy it to the clipboard,  you get the original straße.

And again, GTK supports the corresponding CSS property.

Font features

As I said, harfbuzz makes it much easier for us to access features of the fonts we use. One of these that I have recently looked into is caret metrics. High-quality italic fonts can contain information about the slope at which the text caret is best drawn to match the text:

I’ve added a new api to get this information, and made GTK use it, with this result:

Another useful bit of font information concerns placement or carets inside ligatures.

 Historically, Pango has just divided the width of the glyph evenly among the characters that are forming the ligature (w and i, in this example), but high-quality fonts can provide this information. This is most relevant for scripts using many ligatures, such as Arabic.I’ve made Pango use the ligature caret data if it is available from the font, and got this result:

The wi ligature in this test is what I could come up with after a struggling for a few hours with fontforge (clearly, font design is not in my future). The only other fonts I’ve found with ligature caret information are Arabic, and I sadly can’t read or write that script.

The last feature closed a 15 year old bug – not something you get to do every day!

The last time I talked about flatpak updates, I explained how flatpak apps can detect that a newer version has been installed, and restart themselves. That is great, and may almost be good enough when you have automatic updates. But that is not always the case.

Thankfully, we can do better. Since 1.5, Flatpak has a portal API that lets applications monitor for updates, and request updating themselves.

Here is how this looks when it is all put together:

In the terminal, I’m building a new version of the the portal test app, and update my (local) repository. The flatpak portal is noticing that the update appeared (I’m running it with a short poll timeout here, instead of the usual 30 minutes), and sends out a D-Bus signal to the application, which requests to be updated, and then restarts itself.

Using the portal API directly is not very convenient, since you have to listen to D-Bus signals and whatnot. Therefore, we now have a library called libportal, which is providing simple async wrappers for most portals. That is what the portal test app in the demo is using, and you should be using it too in your applications.

The first stable release of libportal will appear very soon, with Flatpak 1.6, and then it will find its way into runtimes.

Update: Since this is a portal, users are in control of what apps are allowed to do. If you don’t want an application to update itself, you can put an end to it with

flatpak permission-set flatpak updates $APPID no

Use ‘ask’ instead of ‘no’ to get a confirmation dialog. The permission-set command is new in flatpak 1.6.

Flatpak development is not standing still. Here is a quick summary of recent and coming changes.

Better extensions

In 1.4.2, Flatpak gained the ability to use extra-data for extensions. This mechanism has been around for applications for a long time, but it is a new feature for extensions.

The 19.08 version of the freedesktop runtime uses it for its new org.freedesktop.Platform.openh264 extension, which uses the Cisco openh264 builds.

Since we are taking the ‘run everywhere’ aspect of Flatpak seriously, we’ve backported this feature from the 1.4 branch to older stable branches and released 1.2.4 and 1.0.9, so even users on very stable distributions can enjoy this new feature.

Future plans

We’ve quietly started to work on Flatpak 1.6, which should be out before the end of the year.

On the roadmap for the this release, we have

• Support for masking updates and pinning apps.  This gives users more control about what updates Flatpak installs, without having to answer questions every time.
• Parental controls. This optional feature uses libmalcontent to implement policies about what applications users can install and run, based on OARS content ratings.
• Disk space checks. This is an ongoing effort to improve the accuracy of our disk- and download-size handling and to handle low disk space situations more gracefully.
• Infrastructure for purchases/donations. This is still a bit of a research topic.

You can follow the discussion around these features, the flatpak roadmap and general flatpak topics on the flatpak mailing list.

Coming soon to portals

Things are happening on the portal side too. Some of these have already landed, and will appear in a release soon.

Secrets

We have a secrets portal now.  It works by providing a master secret to the sandboxed app, which is then used to store the applications secrets in an encrypted file inside the sandbox . The master secret is stored in the session keyring.

This is nice in that applications don’t leave their secrets behind in the keyring when they are uninstalled, and the application secrets are safe from others.

The backend for this portal will be provided by gnome-keyring and libsecret will automatically use it inside a sandbox. Backend implementations for other environments are more than welcome.

The secret portal is the work of Daiki Ueno, who gave a talk about it at Guadec.

Self-updates

The Flatpak commandline and tools like Discover or the Elementary app store do a fine job of handling updates for Flatpak apps and runtimes.

But the reality is that self-updating is a popular feature for applications, so we added an update portal that lets them do this in a clean way, with proper integration in the Flatpak machinery.

Backgrounds 1

The background portal monitors applications that are running in the background (without open windows). It gives apps a way to request permission to run in the background, and it notifies users when apps are trying to do so sneakily without permission. The portal also lets applications request to be started automatically when the user logs in.

To implement this, the portal needs information from the compositor about open windows, and which applications they belong to. Currently, this is implemented for gnome-shell, other backends are more than welcome.

Window sharing

The screencast portal now lets you select individual windows, in addition to screens, if the application asks for this.

For now, the portal identifies windows by the application icon and window title. We are looking to improve this by using thumbnails.

Backgrounds 2

We will add a small bit of desktop integration with a portal for setting desktop wallpapers.

A portal library

In the ideal case, portal functionality is used transparently by existing desktop libraries without the need for apps to do anything special. Examples for this are GtkFileChooserNative using the file chooser portal, or libsecret using the new secret portal.

But for some portals, there is no natural library api, and in these cases, doing the portal interaction with D-Bus calls can be a bit cumbersome.

Therefore, we are working on a libportal library that will provide GIO-style async apis for portal requests.

Open for contribution

If you want to get involved with Flatpak development, or are just curious, check out the flatpak project on github, chime in on the Flatpak mailing list, or find us on IRC in #flatpak on freenode.

In my last post discussing changes in Pango 1.44, I’ve asked for feedback. We’ve received some, thanks to everybody who reported issues!

We tried to address some of the fallout in several follow-up releases. I’ll do a 1.44.4 release with the last round of fixes before too long.

Here is a summary.

Bitmap fonts

As expected, not supporting Type 1 and BDF fonts anymore is an unwelcome change for people whose favorite fonts are in these formats.

Clearly, a robust conversion script would be a very good thing to have; people have had mixed success with fontforge-based scripts (see this issue). I hope that we can get some help from the font packager community with this.

One follow-up fix that we did here is to make sure that Pango’s font enumeration code does not return fonts in formats that we don’t support. This makes font fallback work to replace bitmap fonts, and helps to avoid ‘black box’ output.

Subpixel positioning

Font rendering is a sensitive topic; every change here is likely to upset some people (in particular those with carefully tuned font setups).

We did not help things by enabling subpixel positioning unconditionally in Pango, when it is only supported in cairo master.  When used with the released cairo, this leads to unpleasantly uneven glyph placement.  Even with cairo master, some compositors have not been updated to support subpixel positioning (e.g. win32, xcb).

To address this problem, subpixel positioning is now optional, and off by default. Use

pango_context_set_round_glyph_positions (context, FALSE)

to turn it on.

Even without subpixel positioning, there is are still small differences in glyph positioning between Pango 1.43 and 1.44. These are caused by differences in glyph extent calculations between cairo and harfbuzz; see this issue for the ongoing discussion.

API changes

I was a bit overzealous in my attempt to reduce our dependency on freetype when I changed the return type of pango_fc_font_lock_face() to gpointer. This is a harmless change for the C API, but it broke some users of Pango in C++. The next release will have the old return type back.

Line spacing

Another new feature that turned out to be better of being off by default is the new line spacing. In the initial 1.44 release, it was on by default, causing line spacing UIs (e.g. in the GIMP) to stop working, which is not acceptable. It is now off by default. Call

pango_layout_set_line_spacing (layout, factor)

to enable it.

Hyphenation

We’ve received one bug report pointing out that hyphens could be confusing in some contexts, for example when breaking filenames. As a consequence, there is  now a text attribute to suppress the insertion of hyphens.

Miscellaneous bugs

Naturally, some bugs crept in; there were some crash fixes, and some hyphens got inserted in the wrong place (such as: hyphens after hyphens, or hyphens after spaces). These were easy.

One bug that took me a while to track down was making lines grow higher when they are ellipsized, causing misrendering. It turned out to be a mixup with text attributes, that let us to pick the wrong font  for the ellipsis character. This will be fixed in the next release.

There is a Pango 1.44 release now. It contains all the changes I outlined recently. We also managed to sneak in a few features and fixes for longstanding bugs. That is the topic of this post.

Line breaking

One area for improvements in this release is line breaking.

Hyphenation

We don’t have TeX-style automatic hyphenation yet (although it may happen eventually). But at least, Pango inserts hyphens now when it breaks a line in the middle of a word (for example, at a soft hyphen character).

This is something i have wanted to do for a very long time, so I am quite happy that switching to harfbuzz for shaping on all platforms has finally enabled us to do this without too much effort.

Better line breaks

Pango follows Unicode UAX14 and UAX29 for finding word boundaries and line break opportunities.  The algorithm described in there is language-independent, but allows for language-specific tweaks. The Unicode standard calls this tailoring.

While Pango has had implementations for both the language-independent and -dependent parts before, we didn’t have them clearly separated in the API, until now.

In 1.44, we introduce a new pango_tailor_break() function which applies language-specific tweaks to a segment of text that has a uniform language. It is meant to be called after pango_default_break().

Line break control

Since my focus was on line-breaking already, I’ve added support for a text attribute to control line breaking. You can now say:

Don't break <span allow_breaks="false">here!</span>

in Pango markup, and Pango will obey.

In the hyphenation example above, the words showing possible hyphenation points (like im‧peachment) are marked up in this way.

Placement

Another area with significant changes is placement, both of lines and of individual glyphs.

Line height

Up to now, Pango has been placing the lines of a paragraph directly below each other, possibly with a fixed amount of spacing between them. While this works ok most of the time, a more typographically correct way to go about this is to control the baseline-to-baseline distance between lines.

Fonts contain a recommended value for this distance, so the first step was to make this value available with a new pango_font_metrics_get_height() API.

To make use of it, we added a new parameter to PangoLayout that tells it to place lines according to baseline-to-baseline distance. Once we had this, it was very easy to turn the parameter into a floating point number and allow things like double-spaced lines, by saying

pango_layout_set_line_spacing (layout, 2.0)

You can still use the old way of spacing if you set line-spacing to 0.

Subpixel positions

Pango no longer rounds glyph positions and font metrics to integral pixel numbers. This lets consumers of the formatted glyphs (basically, implementations of PangoRenderer) decide for themselves if they want to place glyphs at subpixel positions or pixel-aligned.

The cairo renderer in libpangocairo will do subpixel positioning, but you need cairo master for best results. GTK master will soon have the necessary changes to take advantage of it for its GL and Vulkan renderers too.

This is likely one of the more controversial changes in this release—any change to font rendering causes strong reactions. One of the reasons for doing the release now is that it gives us enough time to make sure it works ok for all users of Pango before going out in the next round of upstream and distro releases in the fall.

Visualization

Finally, I spent some time implementing  some long-requested features around missing glyphs, and their rendering as hex boxes. These are also known as tofu (which is the origin of the name for the Noto fonts – ‘no tofu’).

Invisible space

Some fonts don’t have a glyph for the space character – after all, there is nothing to draw. In the past, Pango would sometimes draw a hex box in this case. This is entirely unnecessary – we can just leave a gap of the right size and pretend that nothing happened.  Pango 1.44 will do just that: no more hex boxes for space.

Visible space

On the other hand, sometimes you do want to see where spaces and other whitespace characters such as tabs, are. We’ve added an attribute that lets you request visible rendering of whitespace:

<span show="spaces">Some space here</span>

This is implemented in the cairo backend, so you will need to use pangocairo to see it.

Special characters

In the same vein, sometimes it is helpful to see special characters such as left-to-right controls in the output.  Unicode calls these characters default-ignorable.

The show attribute also lets you make default-ignorables visible:

<span show=”ignorables”>Hidden treasures</span>

As you can see, we use nicknames for ignorables.

Font information

Pango has been shipping a simple tool called pango-list for a while. It produces a list of all the fonts Pango can find.  This can be very helpful in tracking down changes between systems that are caused by differences in the available fonts.

In 1.44, pango-list can optionally show font metrics and variation axes as well. This may be a little obsure, but it has helped me fix the CI tests for Pango.

Summary

This release contains a significant amount of change; I’ve closed a good number of ‘teenage’ bugs while working on it. Please let us know if you see problems or unexpected changes with it!

After Behdad left, Christian and I turned our attention to GtkTextView, and made some progress.

Scrolling

GtkTextView is a very old widget. It started out as a port of the tk text widget, and it has not seen a lot of architectural updates over the years. A few years ago, we added a pixel cache to it, to improve its scrolling, but on a high resolution display, its still a lot of pixels to shovel around.

As we’ve moved widgets to GTK4’s rendering models, everybody avoided GtkTextView, so it was using the fallback cairo rendering path, even as we ported other text rendering in GTK to a new pango renderer which produces render nodes.

Until yesterday. We decided to just have a look at how hard it would be to switch the text view over to the new pango renderer. This went much more smoothly than we expected, and the new code is in master today.

So far, this is just a straight port with no optimizations (we want to look at smarter caching of render nodes for the visible range). But it is already noticeably smoother to scroll text.

The video does not really do it justice. If you want to try for yourself, the commit is here.

Blinking

After this unexpected success, we looked for another small thing we could to make text editing in GTK feel more modern: better blinking cursors.

For the last 20 years, our cursor blinking was very simple: We turn it off, and then we turn it on again. With GTK4, it is very straightforward to do a little better, and fade the cursor in and out smoothly.

A subtle change, but it improves the experience.

I have recently spent some time on Pango again, in preparation for the Westcoast hackfest. Behdad is here, and we’ve made great progress on the first day.