Category: General

After thinking about it a bit more I made some changes to my versioning model. I thought since it was fairly simple I’d add the ability for lightweight data copies, separate meta-data from data, and as a side-effect I simplified some of the queries and probably made them more efficient to boot.

With a hangover for most of Saturday it was pretty slow going but I managed to work something out.

I added an extra level of indirection between the entity, revision and it’s content. I also removed the revision from the data, so it is simply a key-value, which maps nicely and more efficiently to Berkeley DB, so I can remove any data marshalling for what will always be the largest data block. It let me remove some of the indexes as well.

The revision and entry tables stay the same:

create table rev (
  id serial not null primary key,
  branchid int not null references rev(id),
  name text,
  author text,
  ctime text,
  constraint rev_name_uc unique (name)
);

create index rev_branchid_idx on rev(branchid);

create table entry (
 id serial not null primary key,
 revid int not null references rev(id),
 classid int
);

But the data table is simpler, and I added a meta-data table, which just contains a name for now:

create table data (
  id serial primary key,
  content text
);

create table meta (
  id serial primary key,
  name text not null
  -- other stuff goes here
);

Then there is a new table which maps revisions of a given entry to its content.

create table entryrev {
  id serial primary key,
  entryid int references entry(id),
  revid int references rev(id),
  metaid int references meta(id),
  dataid references data(id)
};

create index entryrev_entryid on entryrev(entryid);

Since I did this using Berkely DB, I also created an indirect index of ‘name’ to ‘entryrev’, so name based lookups are very simple. This is easy to maintain since data is never deleted. Something similar could be done in SQL by adding a redundant name column to the entryrev table and indexing on that, or you could just use a join and index on metaid.

All the branch matching calculations are the same, but it only has to be done once now and will return keys to all data. So the basic query returns more information, and doesn’t have to be repeated for each set of data (if you want to version more than one). Now to perform a ‘branch merge’ (for a new item or one that doesn’t require merging) just create a new entryrev with a new revision on the target branch and the metaid, entryid, and dataid from the source branch. i.e. create a lightweight copy.

I’ve been consumed by thinking about database versioning. Why I wonder? What a silly thing to be consumed about. Is there just nothing to watch on TV? Hmm, no there isn’t. Still – I’m starting to get a bit sick of it … but before I drop it never to return I thought id’ better put brain to keyboard and dump out some of what I worked out.

I just wrote an article covering the basics of how to version records in a database. The basic algorithm and data structures, how to perform some of the typical things you want to perform with versioned data-sets, enhancement ideas and so on. Maybe someone will find it useful.

Well it wasn’t that wet, but it was close enough and it was cold and dreary. I spent an awful lot of the weekend playing with an idea I had last week. Another piece of software to run a blog/wiki/etc – i.e. a ‘cms’. I don’t really need one, and the world doesn’t need another one, I just want to update some of my skills and play with some web stuff again (as much as a vowed never to do any web coding again after a php experience, wpf is worse and has driven me to these depths) and exercise a few other ideas i’ve had lately.

And I want to do it with raw c and no frameworks, just to get back to basics again for a while. I’m only using libdb, and gperf for a static hash table. It is also a cgi programme so I can (hopefully) run it on my isp’s web site (that is another reason I can’t use frameworks). Being a fork/exec cgi means I can be lazy with resources too – implicit garbage collection by means of calling exit. At least for small strings, and that simplifies the code a bit more. I’m also avoiding xml. I don’t really like xml, and I don’t think it aids productivity in most cases – to me it is a technology that lets proprietary products ‘open up’ a little controlled access into their internal logic (thinking xslt mainly here). But it is not a natural way to code for most developers and is extremely resource intensive (at least the c implementations are – java/.net is so resource intensive anyway it’s hard to tell). Anyway – I don’t need it, so I wont use it (unless I start looking at dynamic asynchronous javascript down the track).

So after doing the basic stuff of a simple cgi argument parser, and refreshing some ‘form’ knowledge, I started looking into presentation, and programme flow. I have not looked at css before – so there’s plenty to look at there, and it looks like it’ll be flexible enough to not need anything like xslt for presentation.

But how to add all the basic support HTML around the content? First I thought of templates parsed at run-time. Something like server side includes but with a little more flexibility. But I want a minimum of processing at run-time, and then there’s all the hassle of template-to-function linkage, showing pages recursively and all that guff. I thought i’d just pre-compile the templates, a bit like the way xaml works, but without all the crap. Compile into tables which are executed at run time? No hang on, why not just into a function-per-template and a string of function calls? Hmm, that suddenly made the problem tiny. A very simple ‘compiler’ that converts the templates into a sequence of function calls – output string, call function, output string, etc. Let the compiler handle the linkage and storage issues, and I just have to handle a little run-time context. And the added advantage is that run-time is very fast and the memory use is minimal – and shareable – since the page is read directly in it’s executable form. Problem solvered. I didn’t really want to go the php route and have embedded logic which can end up becoming a huge mess, so you can only call functions and each page outputs in its entirety and the control logic is in the c source. After all I’m not writing a framework here, just a solution to a specific problem, so there’s no need to add more smarts than is required. I wrote a little makefile magic to compile these templates, and some bourne-shell to generate the ‘/* AUTOMATICALLY GENERATED */’ header files, and now I have a nice ‘void page_front_page(void)’ I can call from any other page or c code to output whatever is in ‘front-page.html’.

I had to get up to speed on libdb again – I looked at it for evolution just before I left, and I wanted everything to be transaction protected, use sequences, multiple secondary indices, and possibly queues in the future. Didn’t take long – libdb has excellent documentation. One issue is how to store the data since libdb only gives you a key-value pair, you need to pack the rows yourself. I decided for now on a simple tagged format – extremely simple (and fast of course) to parse on loading and no need to have it human readable. And being tagged means it can change in the future without breaking everything (that was a mistake I made with camel – trying to compress the data too much, the order and number of items was important). Initially I was going to store the records as files and just store the meta-data in the table, but I decided to put everything in the database, at least for the moment. Conceptually, each record is a little like an ldap record, a field followed by a value with the capability of having multi-valued fields. And everything is binary-safe.

That let me add a few more pieces – now I could throw together a front page – 2 small template files, the main page which calls the function to output the content, and a one to display the summary of the post. An editor. A bit of messy logic in the c file to track the lifetime of the editing session, plus one editor page template to communicate with the user. Simple basic stuff – and that’s half of a blog done. Oh I threw together an rss feed with a little bit more work as well.

And then I had enough time to look at the user interface to the content – the ‘wiki’ language. Rather than re-inventing another weird wiki language, I thought i’d try using texinfo (and only texinfo – no html/special stuff). This has been an idea swirling around in my head for a while now. What if you could edit a complete texinfo document, where each @node is a separate page, and the node links (next/prev/up) are used to automagically bind all works into a coherent whole. One problem I find with wiki’s is that the nodes mostly sit by themselves, and the linking process is ad-hoc (usually it is not done since it isn’t completely necessary too – pages stand alone as articles) and it is difficult if not impossible to convert part or all of a wiki into a a printed or all-in-one-page document. Add in the cross referencing and indexing features in texinfo and you could really have a powerful system. Well that’s the idea anyway. For now i’ve got only a small part of texinfo implemented and all it really does is formatting and external links. I’m still not sure how to handle node/navigation – the wiki/texinfo way is to use a node name – but if you change that I want it to go and update any links too – well I have a transactional database with multiple indices, so it should be doable. And I probably need some sort of namespace mechanism to manage multiple separate documents. I can’t run any texinfo tools because they might not be available and it’s too much of a security issue anyway, so I need to do the processing internally (I need to do that anyway for any automatic manipulation).

But I dunno – the texinfo thing is a big task, so i’ll see how it goes – for now it provides a reasonable markup syntax for many types of technical documents. There are plenty of other issues like user management and concurrent access too, should I bother looking into those. Hmm, versioning – that might be something quite interesting to look into too – and something else i’ve been thinking about recently.

I also ordered a new thinkpad. There’s nothing particularly wrong with my old one – T40 – 5 years old and still in good shape (motherboard/keyboard replaced after 3 years on warrantee, and i dropped it a week later, but just a tiny bit of the case broke off). I just felt like spending money and I thought i’d try an X series this time since they were on special last week, and came with a dock/dvd burner thrown in at a reasonable price. I’ll miss the touchpad and the keyboard light, but the x300 is too expensive.

Didn’t notice the comment approval thing in this CMS … so I just approved all the comments.

BTW i’ve been assured tomboy wasn’t political. I guess it was made using some other decision process I don’t understand.

Update
I unapproved the SPAM. I guess I might have to tone things down a bit – from the comments some people actually read this drivel, which is both good and bad to know. I’ll try to go back to coding and leave the pythonites and debianites alone.

Sigh. It’s just about the last straw for me and debian-based systems.

I wanted to read about the texinfo format, so naturally I ran ‘info texinfo’, and all I got was a man page which told me to run ‘info texinfo’ to get the full documentation. Oh funny – GNU is full of recursive jokes but this is just silly. Ahh well off to try and install it. Hmm, no texinfo-doc package, huh? Just some non-free ‘info files’ which didn’t seem related to texinfo at all. Oh what? They ARE the documentation. WTF Is going on?

Ho hum. Ok, so i’m a bit late to the party – 2 and a bit years late – but really, these debian guys have lost the plot a bit – they’ve always had a ‘*BSD crowd’ feel about them and things like this just reinforce that impression. Maybe that’s why i’ve had so much trouble finding the documentation for just about everything I looked for.

Complete documentation is a core strength of Unix and by extension the GNU system. It is one of it’s main benefits over other so-called ‘operating systems’ like Windows. Without readily accessible man pages how can you learn to use the system? To write software? I learnt perl from the rather excellent man pages that at least used to exist – why buy a book which will be out of date quickly and is hard to search?

Documentation IS part of an application. To not install it by default is bad enough, to not install it on purpose for quite petty reasons is utterly atrocious.

My house mate wanted to edit her CV, but of course it is in microsoft’s shitty binary format. Loading it into open office has about as many formatting issues as it did the last time I tried something similar years ago … maybe open office devs aren’t focusing on those kind of things (i thought it was a focus at least at some point). I’m not blaming them – its ms, or more directly, TAFE for wasting limited educational resources on such rubbish.

Google docs was actually a little better – it lost more formatting, but it did it in a more consistent and visually pleasing manner. Still, not much use for this case either. The file hasn’t really been formatted properly (not using styles/margins properly) – but what can you – that is the mode of operation wysisyg editors enforce.

Microsoft re-announcing that they’re going to support ODF has no positive meaning either. Even if they ever do what they say, they will still have enough incompatibilities that they will just set the ‘standard’ on any ambiguous language in the specification, or simply break it on purpose. You can quite legitimately claim to support a standard, yet still have ‘bugs’ which make your product the one the others have to follow or work with. Just look at internet explorer, or outlook

At work we had a shitty office 2003 xml loader for excel files, but I got sick of its api and wrote a csv loader instead. Oh joy, so nice to be able to edit excel ‘files’ in emacs, and now they load almost instantly, using fewer lines of code, and much less memory. Rather than having to run-time compile an xml de-serialiser and load the whole thing into memory into objects that get translated into arrays of strings, which mostly get thrown away. So it has fewer features, but they’re not features I actually need.

While i’m here – I’m a bit sick of the ‘xml solves the worlds woes’ rubbish – if anything the whole ooxml debacle should prove otherwise. XML is not really a panacea for anything, it’s just a convenient if a bit complicated file format for data interchange. There are far simpler and far more efficient binary formats for the same thing too, which would probably make an awful lot more sense when you don’t need a text editor to edit them (xmlrpc, soap, anyone?). I find it rather disturbing that people talk about using ‘the dom’ as an internal api for an applications which work with XML as a ‘good thing’ too. It isn’t. It is a terrible api for internal data manipulation. XML should stick to what it is good at (hmm, is it even?), data interchange. An external format. Letting external file formats directly become your internal data representation is just as bad as doing it the other way around. It might be convenient for one-off simple applications, but all you’re doing is replacing native language features with amorphous language neutral ones which will never be as easy to write or maintain or scale.

Had a strange episode on my laptop last night. Not sure if it’s the update I ran a few hours before, or flakey hardware, but all of a sudden I couldn’t start any new shells, and the cpu was locked so hard the machine turned into a hair dryer (which is why I wanted to run a shell – to run top). I used the xfce process manager to kill a busted bash process which was locking the cpu (although unlike windows, the machine was still responsive enough, I didn’t want to burn the thing out).

Turns out that /bin/bash somehow got corrupted. Hmm, tricky. Bit hard to fix something like that if you can’t log in – so much for gui tools. Hmm, what to do. I thought of emacs and shell mode, and guessed the way to specify the shell to run. After trying all the obvious things like ash, tcsh, and csh and discovering none of those shells were installed, I took a punt on the oddly named but completely unfamiliar dash. Horrah, finally I could run a shell! So I changed my login shell to dash and started poking around. Hmm, apt-get --reinstall install bash perhaps? Oh no, it needs bash to run the pre/post install scripts – it just always failed. Poked around man pages in dpkg and apt-get. Not much luck, no obvious way to disable the scripts. But at least I found where the package was cached. Hmm, dpkg-reconfigure? Oh helpful. Package bash is not working or only partially installed. No shit sherlock. Bear with me here, I really don’t grok debian’s package system – all I remember from the Evolution days is that they made up their own packages and versions which just gave the developers extra headaches. Ahaah, dpkg-deb -x bash* /tmp/foo. We have a binary. cp bin/bash /bin/bash, re-fix my shell, and finally it works.

Bit of a panic there – it’s been so long since i’ve had to administer a box at that level (apart from disabling all the crap that runs at login), and quite frankly there’s a reason for that. Hmm, so maybe there’s some hardware issue with my machine – firefox has started crashing a lot too (firefox 2 that is, 3 was way too much to use), or maybe it’s just related to the 8.04 install or an update.