Comments on: Should UI strings in source code have non-ASCII characters?

By: Alexander Jones

Alexander Jones — Mon, 19 May 2008 23:43:09 +0000

@simos:

UTF-8 is designed so that subsequences are unambiguous. You won’t get a byte less than 0×80 in any part of a multi-byte sequence. bytes 0×00-0×7F map directly to 7-bit ASCII.

Some people are worried about string functions breaking. I really don’t see how this is the case, seeing as we’re doing g_some_function (_(”Some ASCII string”)) which is replaced with a UTF-8 string at runtime anyway.

Does anyone have any actual proof of UTF-8 in our translatable strings breaking C?

By: Yevgen Muntyan

Yevgen Muntyan — Sat, 17 May 2008 00:20:38 +0000

UTF-8 strings like “\xCE\x80″ *are* portable. They are not “portable to different encodings”, but nobody needs that (whatever that means). We need UTF-8 in C strings, and that’s the way to have them. If you want nice in po files, make xgettext convert C escape sequences to nice UTF-8 symbols.

As to universal character names, it is implementation-defined what actually will be contained in the character array. I.e. if you have char *s = “\u…” then you have no idea how to display text pointed to by that variable in a gtk label. Also, once you have that line of code, it won’t by magic change “to different encodings”, it will be whatever byte sequences the compiler will put in there and that’s it. It’s pretty much the same as “abc” – it won’t by magic be valid UTF-16, no matter how you compile the file.

And by the way, MS does not implement C99 (surprised?).

By: simos

simos — Sat, 17 May 2008 00:02:33 +0000

@nona: For the narrow scope of GNOME, it appears that all POT/PO files follow the UTF-8 encoding. Indeed, if some translation teams were to use another encoding such as SHIFT-JIS, it would make a bit of a mess.

SHIFT-JIS is almost backward-compatible with ASCII (two characters differ).

@behdad: C99 defines a super-portable way to encode non-ASCII strings (using UCNs, as described in the added section in the post above). This is what gcc says about UCNs:

$ cat t.c
int main(void)
{
char* str = “\u0399″;

return 0;
}
$ gcc t.c -o t
t.c:3:14: warning: universal character names are only valid in C++ and C99
$ _

This means that UCNs work in gcc, but they produce a warning by default.

Using hand-encoded UTF-8 strings (such as “\xCE\x80″) makes the code less portable to different encodinds.

By: nona

nona — Fri, 16 May 2008 22:10:04 +0000

What about non-unicode multibyte character sets that might still be popular in some countries? What happens when there’s UTF-8 and, let’s say, SHIFT-JIS in the same PO file?

By: behdad

behdad — Fri, 16 May 2008 03:07:47 +0000

I think the reason some think C source code should be 7-bit is that your *compiler* can screw up if run under a non-UTF-8 locale. And that may actually be required by the C standard. Not motivated enough to test it.

By: simos

simos — Thu, 15 May 2008 11:16:54 +0000

@Alexander: Some compilers complain when the source code has non-ASCII characters.

That is, bytes with the 8th bit set. Both iso-8859-x and utf-8 can have bytes that the value is >127.

Or, bytes with value <32 (control characters). That could be the case with UTF-8 when a character has codepoint value >127.

By: Alexander Jones

Alexander Jones — Thu, 15 May 2008 10:42:14 +0000

UTF-8 translates to ISO-8859 fine, insofar that it remains valid, even if it is garbage. I don’t see why a compiler would screw up on parsing UTF-8 characters, as they just appear like a series of ISO-8859-x characters.

Maybe I’m missing something?

By: Yevgen Muntyan

Yevgen Muntyan — Wed, 14 May 2008 19:58:54 +0000

First fix all text editors, so they don’t screw up your unicode (on the way to there remove iso8559-15 markers from all source files). Next, fix the C and C++ standards so all compilers understand UTF-8 source by default. Then use UTF-8 in C code
Note that gcc is not the only C compiler for desktops, MS makes some too. UTF-8 in source code is GNU-C-ism which makes code less portable.

By: Sebastian Benitez

Sebastian Benitez — Wed, 14 May 2008 15:55:52 +0000

Like Phil says, languages like spanish, german and french use different quotes than english. For spanish it would be «these quotes».

By: Phil

Phil — Wed, 14 May 2008 12:05:02 +0000

It isn’t always a valid assumption that apps are written with american english strings, so why not just go the whole hog and use semi-symbolic strings by default?

printf(_(”file not found: %s\n”));

might not be very friendly, but it’s direct and equally easy to translate, regardless of what quotes your region uses. There exist english translations of a lot of english software already (en_UK etc) so adding en_US isn’t creating a major new translation job.

Obviously I wouldn’t bother changing old strings, as long as compilers aren’t erroring anyway, but a gradual shift doesn’t seem to be a lot of work.