Ducktype 1.0

2019-04-13

Earlier this week, I released the final specification for Ducktype 1.0. If you’d like a quick primer on what Ducktype looks like, check out our Learn Ducktype tutorial.

Ducktype is a lightweight syntax designed to be able to do everything Mallard can do. It doesn’t sacrifice semantics. It doesn’t sacrifice metadata. It doesn’t sacrifice nesting and content model.

It took a long time to reach 1.0, because I take compatibility and correctness very seriously. Syntax is API. I work with multiple documentation formats on an almost daily basis. It is frustrating to constantly run into language quirks, especially quirks that vary between implementations. I realize quirks tend to accumulate over time, but I think we’ve prevented  a lot of them by writing an actual specification, developing a regression test suite, and focusing on the cohesiveness of the language.

I’m really happy with how it’s turned out so far. I hope you enjoy it too.

We’ve just released Mallard 1.1. Let’s take a look at what’s new. All of these features are already supported in tools like Yelp and Pintail.

This is part 3 in a 3-part series. Read part 1 and part 2.

Keywords

MEP-0014: Informational keywords element

One of the most visible additions in Mallard 1.1 is the introduction of the keywords element. If you look through a lot of real-world documents, you’ll find some that stuff synonyms into the desc element just to match things that users search for. This is not ideal.

In Mallard 1.1, we’ve introduced the keywords element to aid search systems built into tools like Yelp and Pintail. We decided to keep the element simple, using just a comma-separated list of terms instead of any nested keyword elements.

External info links

MEP-0007: External Info Links

Mallard features a number of types of automatic links, like seealso links. These links tend to automatically link in both directions, hence why we call them automatic links. The exact way they automatically link back depends on the link type. They all use informational link elements with an xref attribute, and the link text for these links is taken from informational elements like title and desc of the target node.

Since automatic links use the xref attribute, they were only able to link to pages within the same document, not to anything on the web. This makes some sense. After all, we can’t very well make random web pages automatically link back to our page. But sometimes you really do just want to have a seealso link to an external web page, even if it can’t automatically link back.

In Mallard 1.1, you can use the href attribute on informational links. Exactly how this works depends on the link type, but in general for things like seealso links, the link will appear in the list with your internal links. Because we can’t reliably fetch external resources for the link text every time we build a document, the informational link element can now have a title and desc element to specify the title and desc of the target.

Inline highlights

MEP-0009: Inline Highlight Element

Another highly visible change is the new hi element. This element can be used to highlight text that you’ve added or changed when progressively building up code examples. You can see it in action in the Mallard Ten Minute Tour.

In fact, we’ve been using this element (in the experimental namespace) in the Ten Minute Tour and other places since before Mallard 1.0 was even released. It was (I think) the first thing we added to the experimental namespace, and certainly the longest element in continual use without being in an actual spec. I wasn’t sure back then if we really wanted to add it, because it was unlike any inline element in any other semantic format. Years and years of usage have proven it’s worth having.

You can also use the ins and del style hints when including a diff in a Mallard page. These will do what you expect: green background, and red background with a strikethrough, respectively. Also, the yelp-xsl stylesheets let you use any of red, orange, yellow, green, blue, or purple as style hints to set the background color. Have fun.

Linkable sequences

Mallard features ubiquitous linking, meaning that any of the three linking attribute can be used on any inline element, so you don’t have to use two separate element to make a function name be both code and a link. Well, almost any inline element. In Mallard 1.0, you couldn’t put linking attributes on the guiseq and keyseq elements. I have completely forgotten whatever reason I might have had for doing that, and multiple people have since convinced me that was a mistake. Mallard 1.1 fixes this. You can now do ubiquitous linking on guiseq and keyseq.

Speaking of guiseq and keyseq, we’re considering creating a shorthand syntax for them so you can easily write simple GUI paths and key sequences without nested gui and key elements. Comment on the issue to let us know your thoughts.

And more

That’s it for part 3 of 3. If you haven’t already, read part 1 and part 2. For more information, you can read the Mallard 1.1 changes, the Mallard 1.1 enhancement proposals, and the 1.1 milestone issues.

Want to get involved? Take a look at our 1.2 milestone issues. Or check out the Documentation issue label and help us write tutorials. Keep in touch on the Mallard mailing list.

We’ve just released Mallard 1.1. Let’s take a look at what’s new. All of these features are already supported in tools like Yelp and Pintail.

This is part 2 in a 3-part series. Read part 1.

Table header cells

MEP-0012: Table Header Cells

Mallard mostly follows the HTML table model, with some simplifications of how things are styled. One element that was notably absent from Mallard tables, however, was the th element. The th element allows you to mark a cell as being a header for a row or column, even if it’s not in a thead element (as row headers wouldn’t be).

This was a pretty obvious and easy addition. I was so confident Mallard 1.1 would get a th element that I added a Ducktype shorthand for it well before Mallard 1.1 was released.

Custom roles for links

MEP-0003: The role Attribute on the links Element

This one’s pretty exciting, though a bit advanced. In Mallard, links can have roles which affect things like link text. These work with the multiple informational titles you can provide for a page or section. So, for example, if your language needs to do declensions for different parts of speech, you could write your links like this:

<link role="subject" xref="useful"/> is really useful.
For useful info, check out <link role="object" xref="useful"/>.

Then in useful.page, you would have something like this:

<info>
  <title type="link" role="subject">Subject Title</title>
  <title type="link" role="object">Object Title</title>
</info>
<title>Normal Title</title>

Mallard will pick up the right title. More often, however, you don’t write your links inline, but instead you do automatic linking with informational links and possibly the links element. What happens then?

In Mallard 1.0, different types of automatic links have implicit roles. So the topic links on a guide page will automatically use the "topic" role to select link text from titles, for example. There are implicit roles for topic, guide, seealso, and series links.

So far, so good. But what if you want to use different titles for link text when coming from different guide pages with different topic link styles? This is where the new Mallard 1.1 feature comes in. In Mallard 1.1, you can add a role attribute to a links element to set the primary role to use for all the links it generates. The implicit role for that links type will still be used as a secondary role.

<links type="topic" role="fancyrole"/>

Boring schema changes

In Mallard 1.0, sections were required to have id. There were a couple of reasons I made that decision, but in the end it turned out to annoy more people than it helped. So in Mallard 1.1, section IDs are optional.

We also made a perfectly boring schema change to make it easier for the Cache Files schema to extend the Mallard schema. (There’s also a new Cache Files 1.1 release.) Although RELAX NG is a mostly great schema language for extensible schema design, it does take some effort to design schemas that can be extended. Mallard got a lot of things right, but sometimes we find something to improve.

And more

That’s it for part 2. If you haven’t already, go read part 1, and keep your eye out for part 3. For more information, you can read the Mallard 1.1 changes, the Mallard 1.1 enhancement proposals, and the 1.1 milestone issues.

Want to get involved? Take a look at our 1.2 milestone issues. Or check out the Documentation issue label and help us write tutorials. Keep in touch on the Mallard mailing list.

We’ve just released Mallard 1.1. Let’s take a look at what’s new. All of these features are already supported in tools like Yelp and Pintail.

This is part 1 in a (probably) 3-part series.

New type attribute for code blocks

MEP-0010: The type Attribute on the code Element

In Mallard 1.0, we used the mime attribute to specify the content type of elements like code and screen. The mime attribute took a MIME type, of course, which seemed like a good idea at the time. Unfortunately, there are very few registered MIME types for the types of content we put in code blocks. So you had to memorize arbitrary long strings.

<code mime="text/x-python">

In Mallard 1.1, we’ve deprecated the mime attribute in favor of the new type attribute. The type attribute takes simple strings, like "xml", "py", and "c". It can also take a space-separated list, so you can provide specific types as well as more generic types to catch different syntax highlighters, like "xml mallard".

<code type="py">

As an added bonus, the type attribute has a shorthand syntax in Ducktype. So instead of typing this:

[code type="py"]

You can just type this:

[code py]

New div element

MEP-0005: Generic div Element

Mallard gives you a handful of semantic block elements, but sometimes you just need to group some block elements together without semantics. This is useful for extensions like Conditionals that operate on block elements. It can make it easier to control things like translatability with ITS. And it can help with transclusions using XInclude.

Mallard 1.1 introduces the new div element as a generic block element. The div element is a formal element, accepting optional title and desc elements. One extra bonus of this addition is that it provides better fallback behavior for extensions that have formal block elements.

Making example formal

Speaking of formal block elements, we made the example element formal, accepting optional title and desc elements. The example element was the only block container element that didn’t allow any sort of title, and that just seemed silly.

Giving blocks info

MEP-0002: Block info Element

All formal block elements now take an optional info element. This can be used, for example, to provide credits and license information for people who drew figures or wrote code listings. Or you can use it to provide alternate titles for UI expanders.

More importantly, this is a crucial first step in making block elements participate in both inline and automatic links, tentatively slated for Mallard 1.2

And more

That’s it for part 1. Keep your eye out for parts 2 and 3. For more information, you can read the Mallard 1.1 changes, the Mallard 1.1 enhancement proposals, and the 1.1 milestone issues.

Want to get involved? Take a look at our 1.2 milestone issues. Or check out the Documentation issue label and help us write tutorials. Keep in touch on the Mallard mailing list.

Mallard allows you to declare a content type for code blocks, which can then be used for syntax highlighting. We ship a copy of the excellent highlight.js with yelp-xsl, which in turn is used by tools like Yelp, yelp-build, and Pintail. And we colorize what highlight.js gives us using the same color templates used in the rest of the stack.

With Mallard 1.0, the way to declare the content type of a code block is using a mime attribute, which takes a MIME type.

<code mime="text/x-csrc">

This seemed like a good idea at the time. MIME types are the most universal standard for identifying stuff like this, for better or for worse. In practice, it’s been pretty cumbersome. Almost none of the kinds of things you’d put in a code block have registered MIME types. You just have to look up and remember arbitrary long strings. Sad face.

In Mallard 1.1, we’re introducing the type attribute and deprecating the mime attribute. The type attribute can take a simple, short string identifier, just like other documentation formats use.

<code type="c">

The strings are shorter than ad hoc MIME types, and generally easier to remember, or even to guess. For most programming languages, both the file extension and the lowercase name of the language probably work. If they don’t, tell us. The type attribute can also take a space-separated list. So you can list a more specific type alongside a generic type you know will match. This is useful for XML vocabularies.

<code type="xml mallard">

This is also particularly useful when you’re also doing things besides syntax highlighting with code blocks of a certain type. The type attribute (and mime before it) is useful for syntax highlighting, but that’s not its only purpose. You could use it to extract code samples for linting, for example. On projectmallard.org, we automatically extract the RELAX NG schemas from the specifications using (currently) the mime attribute. So it can be useful to list multiple types.

One other really nice thing about using the type attribute is that it takes advantage of a syntax shorthand in Ducktype. Ducktype is a compact syntax designed to do everything Mallard can do. Attributes like type, style, and xref are so common that Ducktype gives you a shorthand way of writing them. For the type attribute, you just put the value there on its own. So in Ducktype, it’s as easy as this:

[code c]

Or this:

[code xml mallard]

We could always use help with testing and with updating the documentation. See my post to mallard-list for more information.

When designing Ducktype, I wanted people to be able to extend the syntax, but I wanted extensions to be declared and defined, so we don’t end up with something like the mess of Markdown flavors. So a Ducktype file can start with a @ducktype/ declaration that declares the version of the Ducktype syntax and any extensions in use. For example:

@ducktype/1.0 if/1.0

This declares that we’re using version 1.0 of the Ducktype syntax,that we want an extension called if, and that we want version 1.0 of that extension.

Up until last week, extensions were just theoretical. I’ve now added two extension points to the Ducktype parser, and I plan to add three or four more. Both of these are exercised in the _test extension, which is fairly well commented so you can learn from it.

Let’s look at the extensions we have, plus the ones I plan to add.

Block line parser

This extension is implemented. It allows extensions to handle really any sort of line in block context, adding any sort of new syntax. Extensions only get to access to lines after headings, comments, fences, and a few other things are handled. This is a limitation, but it’s one that makes writing extensions much easier.

Let’s look at an actual example that uses this extension: Mallard Conditionals. You can use Mallard Conditionals in Ducktype just fine without any syntax extension. Just declare the namespace and use the elements like any other block element:

@ducktype/1.0
@namespace if http://projectmallard.org/if/1.0/

= Conditional Example

[if:if test=target:html]
  This is a paragraph only shown in HTML.

But with the if/1.0 Ducktype syntax extension, we can skip the namespace declaration and use a shorthand for tests:

@ducktype/1.0 if/1.0

= Conditional Example

? target:html
  This is a paragraph only shown in HTML.

We even have special syntax for branching with if:choose elements:

@ducktype/1.0 if/1.0

= Conditional Branching Example

??
  ? platform:fedora
    This paragraph is only shown on Fedora.
  ? platform:ubuntu
    This paragraph is only shown on Ubuntu.
  ??
    This paragraph is shown on any other operating system.

(As of right now, you actually have to use if/experimental instead of if/1.0. But that extension is pretty solid, so I’ll change it to if/1.0 along with the 1.0 release of the parser.)

Directive handler

Ducktype files can have parser directives at the top. We’ve just seen the @namespace parser directive to declare a namespace. There is an implemented extension point for extensions to handle parser directives, but not yet a real-world extension that uses it.

Extensions only get to handle directives with a prefix matching the extension name. For example, the _test extension only gets to see directives that look like @_test:foo.

Block element handler

This extension is not yet implemented. I want extensions to be able to handle standard-looking block declarations with a prefix. For example, I want the _test extension to be able to do something with a block declaration that looks like this:

[_test:foo]

In principle, you could handle this with the current block line parser extension point, but you’d have to handle parsing the block declaration by yourself, and it might span multiple lines. That’s not ideal.

Importantly, I want both block line parsers and block element handlers to be able to register themselves to handle future lines, so they can have special syntax in following lines. Here is how an extension for CSV-formatted tables might look:

@ducktype/1.0 csv/1.0

= CSV Table Example

[csv:table frame=all rules=rows]
one, two, three
eins, zwei, drei
uno, dos, tres

Inline element handler

This extension is not yet implemented. Similar to block element handlers, I want extensions to be able to handle standard-looking inline markup. For example, I want the _test extension to be able to do something with inline markup that looks like this:

$_test:foo(here is the content)

For example, a gnome extension could make links to GitLab issue reports easier:

$gnome:issue(yelp#138)

Inline text parser

This extension is not yet implemented. I also want extensions to be able to handle arbitrary inline markup extensions, things that don’t even look like regular Ducktype markup. This is what you would need to create Markdown-like inline markup like *emphasis* and `monospace`.

This extension might have to come in two flavors: before standard parsing and after. And it may be tricky because you want each extension to get a crack at whatever text content was output by other extensions, except you probably also want extensions to be able to block further parsing in some cases.

All in all, I’m really happy with the Ducktype syntax and parser, and how easy it’s been to write extension points so far.

It’s now easier to add sidebars to HTML output in yelp-xsl 3.30. When I finally landed the HTML modernization changes in 3.28, I added the ability to have sidebars without completely mucking up the layout. There’s a main element that’s a horizontal flexbox, and you could implement html.sidebar.custom to put stuff in main. But you’d have to do sizing and styling yourself, and you’d have to implement all the stuff you want in there.

I wanted to make sidebars easy in Pintail, so I started implementing some stock sidebars there. Then I realized I could move most of that work into yelp-xsl.

With 3.30, you can now set what you want to see in the sidebars using the html.sidebar.left and html.sidebar.right parameters. These are both space-separated lists of words, where each word is a sidebar component. In yelp-xsl, we have two components out of the box: contents to give you a table of contents for the whole document, and sections to give you a list of sections on the current page. You can also use the special blank token to force a sidebar to appear without actually adding anything to it.

More importantly, you can add your own components. So you can still make fully custom sidebar content while letting yelp-xsl do the rest of the work. For example, let’s say you wanted a left sidebar with a table of contents followed by a Google ad. You could set html.sidebar.left to "contents googlead". Then add a template to your extension stylesheet like this:

<xsl:template
     mode="html.sidebar.mode"
     match="token[. = 'googlead']">
  <!-- Put Google's stuff here -->
</xsl:template>

Pintail will add further sidebar components, such as one to switch between versions of a document, a language selector, and a search bar.

If you use yelp-build, you can pass the extension stylesheet with -x. The extension styleshet is also where you’d set the parameters when using yelp-build.

With Pintail, you’ll be able to set the parameters in your pintail.cfg file.

[pintail]
sidebar_left = contents
sidebar_right = search languages

What’s more, you’ll be able to set these on a per-directory basis. So, for example, on nightly builds you could add a warning message. In your extension stylesheet:

<xsl:template
     mode="html.sidebar.mode"
     match="token[. = 'nightly']">
  <p style="color: red">NIGHTLY BUILD!</p>
</xsl:template>

Then your pintail.cfg would look like this:

[pintail]
sidebar_left = contents
sidebar_right = search languages

[/path/to/nightly/]
sidebar_left = contents nightly

I’m going to share some math tricks for converting between kilograms and pounds, something I often deal with when weightlifting. This post is long, but if you stick around to the end, I’ll share the super secret divisibility rule for 11. (Originally posted to Facebook. Reposted on my blog at Jim Campbell’s suggestion.)

If you find a mathematician or engineer who’s good at doing math in their head and ask them how they do it, you’ll find they have a handful of techniques they apply in different situations. Many of these techniques involve turning problems into things we’re already good at. What are we good at? Well, multiplying and dividing by 10 is trivial. It’s just moving a decimal point. And most of us are pretty good at doubling and halving things. So, multiplying and dividing by 10 and 2 are sweet spots.

1kg is approximately 2.2lb. (If you need better precision, use a computer.) So to convert kilograms to pounds, we have to multiply by 2.2. Let’s pull out that distributive law. 2.2x = (2 + 0.2)x = 2x + 0.2x = 2x + (2x)/10. We’ve reduced the problem to multiplication by 2 and division by 10.

“Woah Shaun, stop with the algebra!” OK. Take the kilograms. Double it. Take that double, shift the decimal place. Add the double and the decimal-shifted double. For example, take 150kg, a respectable squat weight. Double it = 300. Divide by 10 = 30. Add these two together = 330lb. (Google will tell you the answer is 330.693lb.)

But what about converting pounds back to kilograms? Do we have to divide by 2.2? That sounds awful. But division by 2.2 is multiplication by 5/11. Does that make it any better? Yes! Really? YES! Division by 11 is awesome, and for some reason, nobody learns it in school.

To divide by 11, first divide by 10. That’s your current total. Divide that by 10 and subtract the result from the current total. That’s your new current total. Divide that by 10 and add it to the current total. That’s your new current total. Continue dividing by 10 and alternating addition and subtraction. Do this until you die of exhaustion, you see the two-digit repeating pattern, or you’re happy with the precision. Recall that 2.2 was only an approximate conversion to begin with, so I stop when all the action is after the decimal point. Round whole numbers are good enough for me.

But we needed to multiply by 5/11, not just 1/11. No worries. To multiply by x/11, instead of starting with 1/10, start with x/10. Luckily, 5/10 is just 1/2. We like dividing by 2.

My body weight is about 190lb. What is that in kilograms? Half of 190 is 95. Divide by 10 for 9.5. Subtract 9.5 from 95 for 85.5. Divide 9.5 by 10 for almost one. Addition is next, so let’s call it 86kg. (Google will tell you the answer is 86.1826kg.)

So there you go. Quick tricks to help you get approximate conversions between kilograms and pounds in your head.

But what about the super secret divisibility rule for 11 I promised? It follows the same pattern as the technique I gave for dividing by 11. Just do alternating addition and subtraction on the digits of a number. If the result is divisible by 11, so is the original number. Is 1936 divisible by 11? 1-9+3-6 = -11. It sure is.

Eleven is awesome.

*tap* *tap* *tap* testing testing *tap* *tap* *tap*

Still here? Cool. Here’s a bonus tip that wasn’t in my original post. All that stuff about 11? It works the same way for whatever number “11” happens to represent in any radix. Need to divide by 11_8 (decimal 9) in octal? Same division algorithm. Need to check divisibility by 11_16 (decimal 17) in hexadecimal? Same division rule. Looking for some fun weekend math? Take a look at the divisibility rules you know, figure out why they work, and use that to figure out what divisibility rules you have in other bases. Hint: every divisibility rule I can name stems from three basic kinds of tests.

I’ve served as treasurer for three separate organizations over the last six years. Two of them are US 501(c)(3) non-profit organizations. The other is a consumer-owned cooperative. I’m not an accountant, but I’ve learned a lot about accounting, and each organization has forced me to learn something new.

Today’s adventure is learning how to deal with restricted funds, or funds that have to be used for a particular purpose. I’m going to show four different techniques for dealing with restricted funds, along with some pros and cons.

Restricted funds are conceptually similar to earmarked funds and fiscal sponsorships. In all cases, you have a specific amount of money that is supposed to be used for a specific purpose. I’ll make the following distinction: Both restricted funds and earmarked funds are short-term situations where you expect to spend the money, zero the balance, and stop looking at it. When I talk about restricted funds, I mean funds where you are legally obligated to spend and account for every penny, and pay back any money you did not spend. This could be a grant where the granting organization places requirements on the funds. When I talk about earmarked funds, I mean funds that you have designated for a particular purpose, but where you’re not legally obligated to spend it all, and you could roll a small leftover balance into general funds. This could be money from a targeted fundraising campaign.

Fiscal sponsorships, however, are long-term arrangements where you handle the money for a group of people doing something in line with your charitable cause. They officially operate under your organization so they don’t have to deal with all the non-profit paperwork, but they have a certain amount of autonomy, and importantly, their funds are theirs.

I’ll use the term targeted funds to refer to all three. The following techniques all conceivably work for each type of targeted funds, with different pros and cons on each. None of these techniques are, in my opinion, perfect. Also, by the way, I have to assume you have a passing familiarity with double-entry accounting and split transactions, among other things. Explaining those is another blog post. Onward…

Asset Subaccount

With this technique, you create a subaccount of your primary asset account (probably a checking account), and post all transactions for the targeted funds to that account. This has the advantage of letting you use expense and income accounts however you want, and it might be OK if you have very small and very simple transactions.

I don’t like this technique, though, for a number of reasons. It can lead to really crazy split transactions. Split transactions where money comes from separate income accounts are fine. Split transactions where money goes to multiple asset accounts are weird. Split transactions where both happen hurt my brain.

This really falls apart when you have multiple very liquid asset accounts. By that I mean accounts you regularly make payments from. This could be a checking account, a PayPal account with money in it, a prepaid postage account for bulk mailings, or a prepaid purchase card at a store. (I’ve dealt with all of these.) Or you might have liability accounts you regularly expense from, like a credit card or an expense account at a store.

If your targeted funds are a subaccount of your primary checking, what happens when you use another account to pay with those funds? If you used a credit card, you probably need to use a split transaction when you make your monthly payment. I hope you remember how much. You’d better look through your transaction report. I don’t even know what to do if you carry a balance on the card. And if you paid with PayPal or another asset account, you should really transfer money from checking to cover it, and record that as coming out of the subaccount. What a pain. The easier approach is probably just to use a journal entry to move money from the targeted subaccount to its parent. But that basically means your books will be littered with fake adjustment entries.

Also, there’s a big question mark around how well your accounting tools deals with transactions in a subaccount of a checking account when doing things like reconciliation or automatic transaction importing. My advice is that asset accounts on your books should match real accounts. This technique is just a hack.

Actual Separate Account

If accounts in your books should always match real accounts, why not create an actual separate checking account for the targeted funds? This technique involves quite a bit of overhead, and it’s not something you’re going to do for short-term restricted or earmarked funds. But it could work for fiscal sponsors, with the added benefit that you could even allow your sponsored organization access to that account.

This at least ensures that accounts are accounts and transactions are transactions, but it can still be tricky if you want to spend those targeted funds with other payment methods like a prepaid account or a credit card. If you do that, you’ll have to write an actual check (or make an actual bank transfer) to pay one part of your organization from another part. That’s a hassle, although it is sensible.

There’s also an issue of minimum balances. Your bank may insist on a minimum balance on all checking accounts. You probably don’t care about a minimum balance for your sponsored organizations. You might even be OK with them holding a zero balance. (In the case of restricted or earmarked funds, a zero balance is your goal.) You just might be OK with allowing sponsored funds to be temporarily negative. Banks don’t care for that, so you would have to spend money from your general account, and then record the money owed in some sort of weird entry that is both accounts payable and accounts receivable.

Liability Account

This is probably the technique your accountant will recommend, especially for legally restricted funds like restricted donations. A liability account is a type of account you use to record money you owe. You can make purchases with a liability account. You can pay into it to pay it off. A credit card, for example, is a liability account. A credit card purchase is a transaction involving a liability account and (usually) an expense account. A payment toward your credit card balance is a transaction involving a liability account and an asset account. Liability accounts can be other types of debts, or other types of money you have to spend, like taxes payable.

Money you have to spend. Like that big grant that has to be spent for a specific purpose. It’s strange to think about a grant as a liability, but from an accounting perspective, it fits.

This has the advantage of quickly showing you how much you need to spend and how much money you have in general funds, right on the balance sheet. No other technique makes those numbers quite as clear. The downside is that it’s a black box on an activity statement. You don’t get to use expense and income accounts for anything, and if you want to get any sense of how much you’ve actually spent, or how you’ve spent it, you have to dig into transaction reports. That’s doable, but I don’t enjoy it.

One other thing to consider with this technique is what to do with a not-quite-zero balance. If you overspend, you’ll need to do a split transaction with part of it coming from a general expense account. Otherwise you’ll spend the rest of your life with a weird negative liability on your balance sheet. And if it’s earmarked funds where you can roll a small remaining balance into general funds, make sure to post a transaction that moves that money from the liability account to an income account

If you deal with a very large number of restricted grants, and you need to be absolutely certain they all get spent entirely and in a timely fashion, you should strongly consider this approach. You should also talk to an actual accountant, preferably one with non-profit experience, instead of reading a blog by some guy on the internet.

Income and Expense Accounts

Finally, you can just create specific income and expense accounts to track money into and out of targeted funds, respectively. You can do this alongside the first two techniques involving separate asset accounts, but one of the advantages of those techniques is that you don’t absolutely have to use targeted income and expense accounts. With those techniques, you could record payments to a contractor in your general “Contractor Payments” expense account, if you want.

With this technique, you have to make absolutely certain that every transaction involving the targeted funds is posted to a targeted income or expense account. If you don’t, you will never be able to find the targeted fund balance, you’ll start treating that money as general funds, and some people will be very very upset with you. But frankly, with any technique, you need to be careful to file things correctly.

The biggest downside of this technique is that it tells you nothing on your balance sheet. Your balance sheet won’t tell you how much targeted funds you’ve spent, how much you still need to spend, or how much general funds you actually have after accounting for targeted funds. For some people, that will be a really big deal. If restricted grants or fiscal sponsorships constitute a very large portion of your finances, you probably want to see them on the balance sheet. If you handle targeted funds only rarely, and your general funds are comparatively large, it’s less of an issue.

The really nice thing about this technique is that you still get to see income and expenses in actual income and expense accounts. It shows up in your activity statement. It can show up in your budget report, if you want to budget it. You aren’t even limited to using a single account for expenses. You could have a parent expense account for the specific targeted fund, then subaccounts for things like contractor pay, material costs, etc.

Since this technique doesn’t show you anything on a balance sheet, you’ll need to use an activity statement to see the balance of any targeted funds. Grab an activity statement covering at least the entire period of time the targeted funds have been active (or just for all time), then subtract the targeted expense account from the targeted income account to see how much you have left to spend. Or much more simply, most accounting software lets you run an activity statement on a specific list of accounts. Run one on just your targeted accounts, and it will tell you the balance right at the bottom.

This might seem awkward, but it’s actually no different from what you probably do for non-targeted funds with matched income and expense accounts. For example, if you run a large event, you probably have one or more income categories specifically for that event, and one or more expense categories specifically for that event. If you want to see if the event made or lost money, you have to compare numbers in an activity statement, just like you would do to see the balance of targeted funds in this technique.

This technique has no problem with overspending. Just use the same expense categories and don’t bother with splits. You’ll just have higher expenses than income, just like in an event that lost money. It doesn’t show up on your balance sheet, and eventually it will roll off the activity statements you regularly look at. And if you underspend earmarked funds and intend to retain them in general funds, you can just literally do nothing. Just stop looking at the targeted fund activity statement and the money is in general funds. Or, if you want to be really pedantic, post a transaction that moves money from the targeted expense account to an income account like “Retained Earmarked Income”. That way you can go back and see how much earmarked funds you retain over time, if you’re into that.

This technique is actually the one I’ve decided to use for restricted grants in the organization I’m currently treasurer of. Each year, we deal with just a handful of restricted grants, six at the most. We also run exactly one fundraising event where we earmark the funds, and we retain a portion of those earmarked funds into general funds upfront. I have an income account called “Restricted Grants” and an expense account called “Restricted Grant Spending”. For each restricted grant, I create subaccounts of both the income and expense accounts. I can set up custom activity reports for each grant and put them in the pile with other reports that I review monthly. When a restricted grant is spent and accounted for, I just remove it from my regular reports.

Ideally, I’d like a system where I can record transactions as if in income and expense accounts, but somehow group those accounts in a way that makes their difference show up as a liability on the balance sheet. I don’t know if that’s a thing, but it should be.

I build a number of sites with Pintail, including projectmallard.org and yelp.io. Some of these sites I build and upload manually. Others are hooked up to continuous deployment. I’ve been using python-github-webhooks on my server, which is a very simple tool to receive GitHub notifications and do stuff in response. What I was doing in response was building sites with Pintail.

The problem with this approach is that GitHub wants endpoints to respond within 30 seconds. And although building Mallard with Pintail is fast, there are things you don’t want to block on. In particular, you don’t want network operations to hold things up. At the very least, building requires updating one git repository, and possibly more. The seemingly simple yelp.io configuration pulls in two more git repositories. (Yes, it’s that easy.)

So I needed a job queue. I don’t want to just background the build tasks, because then I could end up starting a new build before a previous build finished, and down that path lies madness. I looked into using AMQP queues or using a full-blown CI tool like Buildbot. But I wanted something simple that didn’t involve a lot of new software on my servers. (Side note: I’m building a handful of relatively small sites with fairly low traffic. If you’re doing more, go use a tool like Buildbot and ignore the rest of this post.)

What I finally decided to do was to manage a simple build queue with mkfifo. I have a program that creates a FIFO and reads from it indefinitely, triggering builds when it receives data. Slightly stripped down version:

wdir=/var/pintail

rm -f "$wdir/queue"
mkfifo "$wdir/queue"
chmod a+w "$wdir/queue"

while read repo <"$wdir/queue"; do
    if [ "x$repo" = "xyelp.io" ]; then
        git='https://github.com/projectmallard/yelp.io.git'
    # Other sites get elif statements here.
    else
        continue
    fi
    if [ ! -d "$wdir/$repo" ]; then
        (cd "$wdir" && git clone "$git")
    else
        (cd "$wdir/$repo" && git pull -r)
    fi

    outdir="$repo"-$(date +%Y-%m-%d)-$(uuidgen)
    mkdir -p "/var/www/$outdir"
    (cd "$wdir/$repo" &&
        LANG=en_US.utf-8 scl enable python33 -- pintail build -v -o "/var/www/$outdir" &&
        cd "/var/www" &&
        ln -sf "$outdir" "$repo".new &&
        mv -T "$repo".new "$repo"
    ) 2>&1 >> "$wdir/$repo"-log
done   

Now the only thing my hook endpoints actually do is write a line to the FIFO. Importantly, the build process only looks for known strings in the FIFO, and ignores any other input. It doesn’t, for example, execute arbitrary commands placed in the FIFO. So the worst an attacker could do is trigger builds (potentially resulting in a DoS).

This script has one other trick: It uses symlinks to atomically update sites. The actual built site is in a unique directory named with the actual site name, the date, and a uuid. The actual directory pointed to by my httpd config files is a symlink. Overwriting a symlink with mv -T is an atomic operation, so your site is never half-updated or half-broken. This is a trick I learned at a previous employer, where it was very very important that our very very large documentation site was updated exactly as our release announcement went out.