Mallard and EPUB

2011-11-10

Did you know you can easily convert a Mallard document into an EPUB file? It’s built right into Yelp Tools, the collection of documentation tools built on top of the core of the Yelp help viewer. Install yelp-tools from your package manager and run the following command in a directory containing Mallard page files:

yelp-build epub .

If you have Yelp customization stylesheets you use for the web, you can use them for your EPUB output too. Or you can create customizations to better fit the form factor of an e-book reader. Of course, as with everything from Yelp, the formatting is non-ugly by default.

Don’t forget that a Mallard document is just a collection of pages that link among themselves dynamically. If you want to publish an EPUB of only certain pages (for example, to provide your accessibility topics for people to read from another device), just limit it to only the pages you want. There’s no need to hand-edit a driver file.

For some reason, blogs.gnome.org doesn’t let me upload webm or ogv files. Go watch a video of searchable menus on Google+. Some notes on what’s going on:

  • The initial menu items are not specified by Nautilus. The menu is populated with items based on tags in the Mallard help files.
  • Start typing to search. This is searching on title and desc elements from the GNOME desktop help. The results are filtered to only include pages relevant to Nautilus. The scroll arrows showing momentarily as you type is an unfortunate glitch I hope to iron out.
  • Click a topic and Yelp opens immediately to that topic.
  • What’s more, Yelp knows that you arrived there from a menu item you got after searching in a menu, and offers a link at the top to perform a full text search on your search terms.

Yelp Tools

2011-05-26

Over the last year or so, I’ve been restructuring the various bits of code that have traditionally been Yelp and gnome-doc-utils. The reusable XSLT stylesheets used by Yelp now live in yelp-xsl. I’ve worked on itstool as a successor to xml2po. And now yelp-tools holds some really handy command-line tools for Mallard and DocBook, plus some simple build magic for autotools projects.

I think hardly anybody knows about the tools in yelp-tools. Let’s fix that.

yelp-new lets you create a new Mallard page file from a template. If you want to use the “task” template (shipped with yelp-tools), to create a page with the ID ducksinarow, do this:

yelp-new task ducksinarow

Or, give it a title straight away

yelp-new task ducksinarow "Put your ducks in a row"

You can create your own templates by giving them the extension .page.tmpl. yelp-new picks up both installed templates and templates in the current directory. It substitutes variables surrounded by @. This should look familiar to people who’ve used autoconf. The easiest way to create a new template is by basing it on an existing template with the –tmpl option, like so:

yelp-new --tmpl task my-new-task

yelp-build allows you to build output formats from Mallard or DocBook files. Currently, it can create HTML or XHTML, Mallard cache files, and EPUB. EPUB only works for Mallard right now. Create HTML for some Mallard page files:

yelp-build html *.page

You can even create your own XSLT customizations. You’ll need to learn the templates and modes used by yelp-xsl. (It’s pretty well commented with a home-brewed XSLT documentation system I made.) yelp-build lets you pass a customization with the -x option. The nice part is that your XSLT doesn’t have to xsl:import anything, so you don’t have to worry about the correct file path. When you pass a file with -x, yelp-build automatically creates a new stylesheet that imports yelp-xsl and includes your customization.

yelp-build html -x my-customization.xsl *.page

Finally, yelp-check is full of handy routines that help you keep track of your work while you write. You can check to make sure all xref attributes point to valid IDs:

yelp-check links *.page

You can check which pages can’t be reached by topic links from the index page:

yelp-check orphans *.page

You can get a report of the revision status of all your page files:

yelp-check status *.page

The status subcommand takes options to let you specify version, docversion, and pkgversion attributes to match against, provide an upper or lower cutoff date, restrict the output to only a few status markers, or just print totals. See the –help output for details.

Finally, you can validate your document against a schema:

yelp-check validate *.page

The validate subcommand implements dynamic schemas with the Mallard version attribute, allowing you to do stricter validation when using extensions like conditional processing, faceted navigation, or glossary indexes.

What else would you find useful?

Open Help Conference

We’ve talked about doing run-time conditional processing in our help for a very long time, nearly as long as I’ve been around. When you have applications that can run in multiple environments, you sometimes need to write different help text for difference cases.

For example, take an instant messaging application like Empathy. In GNOME 3, the Telepathy integration means that users can chat directly from the tray. Hackers know it’s not actually Empathy, but users don’t care. It’s chat. In GNOME 2, you usually interact with Empathy through the notification tray. In Unity, it integrates with the messaging menu. These are all things we should write about in the help.

But it’s hard to write about things when you don’t know where your help is read. The only way to deal with this is to patch the help for different environments. My experience is that that rarely happens. Plus, you don’t always create different packages for different environments. The exact same Empathy package in Fedora is used in GNOME Shell, GNOME fallback, and XFCE.

Other XML formats have conditional processing, but their design isn’t suitable for our needs. They usually work by having a set of attributes that take some tags. The behavior isn’t usually specified, because it’s assumed that they’ll be handled by a vendor-specific build tool that strips things out before creating a deliverable. Our source format is our deliverable format, and we want to adapt it at run-time.

So I’ve just landed some changes in Yelp to support a Mallard extension format for run-time conditional processing. Any block element can have the if:test conditional attribute added. This is evaluated as an XPath expression with extension functions that can tell you things about the environment. This is much more flexible than a set of tagging attributes, because you can combine multiple tests with “and” and “or“.

<p if:test="if:env('gnome-shell')">This is only shown in GNOME Shell</p>
<p if:test="if:env('gnome-shell') or if:env('gnome-panel')">
  This is shown under GNOME Shell or GNOME Panel
</p>

The if:test attribute is a convenient short-hand syntax. There are also conditional processing elements that allow you to do basic branching and fallback.

<if:choose>
  <if:if test="if:env('unity')">
    <p>You're using Unity</p>
  </if>
  <if:if test="if:env('xfce')">
    <p>You're using XFCE</p>
  </if:if>
  <if:else>
    <p>You're using something else</p>
  </if:else>
</if:choose>

The full syntax has another advantage. What happens if you hand a document with conditional processing to a Mallard processor that doesn’t support it? It turns out that Mallard has very well-defined rules for how unknown extensions are handled. In the first case, you just have an external attribute on a known element. The attribute is ignored and the p element is processed as normal.

In the second case, you have an external element in a block context. A Mallard implementation that doesn’t do conditional processing will visit the children in restricted block context. In restricted block context, unknown external elements are ignored.

Yelp 3.0 doesn’t have the conditional processing support. The following two snippets will do what they say in Yelp 3.0, but they will be exactly equivalent to Yelp 3.2 with conditional processing support.

<p if:test="if:env('foo')">
  This will be displayed in Yelp 3.0
</p>
<if:choose>
  <if:if test="if:env('foo')">
    <p>This will not be displayed in Yelp 3.0</p>
  </if:test>
</if:choose>

Details are all subject to change, of course. This does basically follow a proposal I made to the Mallard mailing list last August. But for now, it’s in an experimental namespace. I just need to iron out the kinks and write up a specification on projectmallard.org.

Open Help Conference

GNOME 3

2011-04-07

I’ve never been more excited to be a GNOME developer. After years of hard work and planning, GNOME 3 was released yesterday. Check out the Introduction to GNOME from our brand-new help to learn all about it.

GNOME 3 shows the innovation that open source communities can bring. Hundreds of developers, designers, writers, testers, and translators worked hard to deliver an amazing new user experience. Among them are the fantastic people who helped create an all-new help system that rivals anything I’ve seen elsewhere. My release notes list Phil Bull, Jim Campbell, Tiffany Antapolski, Natalia Ruz Leiva, Shaun McCance, Paul Frields, Mike Hill, Aline Bessa, Marina Zhurakhinskaya, and Kelly Sinnott. If I missed anyone, I’m really sorry. It’s hard to keep track of so much awesome.

We tossed out the old manual (who reads those?) and started fresh with topic-oriented help, building on the dynamic Mallard language. The results are amazing. The initial release has 214 pages, carefully organized and cross-linked to help you find the information you need quickly and get back to your life. What is a workspace? Select files by pattern. Enter special characters.

Frederic Peters did amazing work on library.gnome.org so all the new help is available online. But remember that all of this is available from the Help application on your GNOME 3 desktop. The help viewer was completely rewritten for GNOME 3, and I think you’ll really like what we’ve done.

By the way, if you want to meet up and learn about GNOME’s fresh approach to help, you should come to the Open Help Conference this June. At least Phil, Jim, and I will be there, and we’ll be joined by documentation teams from other great open source projects.

We’re not done yet.

Help needs to constantly improve and evolve as we learn more about what our users need. The GNOME documentation team is already hard at work on more pages and revisions to existing pages. We’ll be pushing updates to the help weekly. If you want to get involved, subscribe to our mailing list and send an email to gnome-doc-list@gnome.org.

We also welcome drive-by contributions. One of the really nice things about topic-oriented documents with Mallard is that it’s easy to just write up a page about something without worrying about making revisions to an entire book. If you love using GNOME, a great way to contribute is by writing a short page about an awesome time-saving trick. We’ll put these under Tips & Tricks, and users worldwide will learn something cool because of you.

I am GNOME

Phil mentioned the future of Desktop Help, but he was sparse on details. Let me fill in the blanks. With Mallard, we worry mostly about structure and content, and we leave the rendering details to Yelp. Sometimes we want to influence the style, and for that we use Mallard’s style hints. But sometimes, just sometimes, for very special pages, we want to inject some pizzazz.

We can do that too. Watch:

This is using some custom style hints and a very small Mallard extension, which falls back gracefully thanks to Mallard’s well-defined rules for extensions and fallback.

This is still a work in progress. And with work continuing into the night at the help hackfest in Toronto, it’s very much in progress.

I blogged earlier this year about doing faceted navigation with Mallard. The idea sat dormant for a while, but with the new GNOME developer demos, I had a really fun use case.

Here’s the basic idea: Topic pages declare values for facets. These are basically just categorized tags. So my Message Board tutorial has some simple markup that says it’s written in C and it’s teaching you WebKit. You then have a page that collects all the topics and provides selectors to narrow what you’re looking at.

So as I browse the demos, I can say that I’m only interested in C. And perhaps I’m looking at GTK+ and Clutter to see which one is a better fit for a project I’m starting. I can uncheck everything else. Then I’m looking at only demos in C that teach either GTK+ or Clutter. Screencast:

All of the scripting is handled by Yelp. Document authors only need to write some simple declarative markup in a Mallard extension format.

(Aside: When I try to upload an ogv file to WordPress, it tells me the file type doesn’t meet security guidelines. If I rename it to ogg, I can upload it, but WordPress inserts it as audio. I have to write the HTML for the video by hand. Could somebody please make it easier to use Ogg Theora on blogs.gnome.org?)

Open Help Conference

I attended the GNOME developer documentation hackfest in frigid Berlin last week. This was, I think, a very successful hackfest. We came out of it with a concrete plan, a new documentation effort firmly started, and a host of improvements to tools in the pipelines.

The plan: We will have an updated Platform Overview. We will have a large and continually growing set of demo tutorials. We will have conceptual overviews for all libraries and major subsets of libraries. We will have complete API references. And we will have a beautiful new developer.gnome.org that acts as a portal to all this information.

Everybody kicked in to create an initial set of demo tutorials. The initial demos focus on getting you up to speed with one of our platform libraries. Further demos will build on this knowledge and help you dive deeper into what the GNOME platform can offer. The demos are in git.gnome.org/gnome-devel-docs/demos/C. We will do our best to make all demos available in multiple programming languages.

Get involved by contacting the GNOME documentation team.

The demos are written in Mallard. This is interesting to me, because it’s pushing Mallard in places where it hasn’t been pushed before. This helps me develop useful features. (As a rule, I do not add language features without real-world use cases.)

Jonh asked for line numbering, so I’ve added that in Yelp using a simple style hint. Chris wondered about starting line numbers, continuations, and line number references. These sorts of features require more than an on/off style hint, and I want to think about them more. I think that, with dynamic pages, we can do better than callout lists. Regardless, any improvements there should be in a Mallard extension.

Everybody wanted syntax highlighting. Fortunately, Mallard allows you to provide the MIME type of the content of code blocks, so we already have a way of declaring what should be parsed and highlighted. I looked through half a dozen JS libraries to do syntax highlighting. In the end, I chose jQuery.Syntax. This is now integrated into Yelp, and syntax highlighting works with appropriate values of the mime attribute. Obligatory screenshot:

Line numbering and syntax highlighting in Yelp

A big thanks to Openismus for hosting the hackfest, and to Kat and David for putting up Phil and me. And of course, thanks to the GNOME Foundation for their sponsorship.

Sponsored by the GNOME Foundation

Earlier this month, I attended the GNOME accessibility hackfest at the AEGIS conference. My goal was to start planning the accessibility user help for GNOME 3.0. The documentation team often doesn’t understand the needs of users of accessible technologies, and understanding user needs is the first step to writing good help.

I got Joanie Diggs started working on Orca help, and Brian Nitz on Accerciser help. I led them through the basics of planning topic-oriented help and helped them revise their plans. It’s always interesting to teach people topic-oriented help with Mallard, and to see what they grasp quickly and what needs more explanation. It helps me develop my ability to teach people to write.

I also got a much clearer sense of how we should work accessibility information into our desktop and application help. The entire team helped me understand what types of information users might be looking for. For example, Joanie mentioned keyboard shortcut reference pages as something that will often be useful to users of accessible technologies.

This led us to a feature idea for Mallard and Yelp. Joanie wondered if it would be possible to aggregate the keyboard shortcut references from across the desktop. This is basically remixing Mallard pages into a new document, which is entirely doable. But in this case, we want to do it dynamically. And for that, we need a tagging system. I once blogged about tags for faceted navigation, and I think those would work for this kind of use as well. This probably not 3.0 material (unless Phil Bull asks for it, because Phil always gets the backburner features he asks for). But it’s a potentially powerful feature that’s made possible by Mallard’s design.

I also watched demos that gave me a better appreciation of the kinds of needs some people have. Tools like screen readers and magnifiers have high visibility, but there are so many other accessible technologies aimed at people with lots of different needs. All GNOME developers should have the chance to see this stuff first-hand.

On a personal note, I left my wallet in a taxi. Brian Nitz and Mario Sanchez Prada spent two hours with me at the hotel where I picked up the taxi, hoping that the same cabbie would come back there to wait for customers. He didn’t, but Mario heroically talked to every single cabbie that stopped there. End result: my wallet was recovered the next day, minus 30€. Thanks, Mario. You’re my hero.

Sponsored by the GNOME Foundation

Yelp and the DOM

2010-06-08

Xan just landed a patch to get the DOM node from a hit test result in WebKit. This allows Yelp to have meaningful link text for the “Read Later” feature I blogged about earlier. But it also opens the door to all sorts of nice features.

One of the nice things about working with source formats like Mallard and DocBook is that the HTML that gets fed to WebKit is very predictable. In fact, it’s completely under my control. So now, if you use a Mallard code block, Yelp can offer some extra goodness when you right-click:

And if the author was good enough to put the code block inside a listing element (as I have in this example), Yelp will even suggest a filename:

Little features like this make Yelp really pleasant for reading system documentation and programming guides.