Writing Tests is Humbling

I have spent some time recently writing import/export tests cases for Gnumeric. It is what you do when you see a mistake that it should not have been possible to make, in this case a hang when writing certain strings to the obsolete xls/biff7 format.

Writing these tests has been a very humbling experience. Highly recommended.

A lot of the code being subjected to tests is quite old: 10-15 years old. You would think that by now it would have had any obvious bugs beating out of it. No such luck. Not only were there ancient bugs — such as the direction of diagonal borders being flipped on load+save — but there were also bugs accidentally introduced when fixing other things.

Bugs happen, even where you think you are being careful. And that is not a problem. The problem arises when the bugs are not caught and make it into releases. This is where a brutal test suite is needed. Our test suite clearly was not evil enough for import and export of various formats, so I have been adding something I call round-trip tests for ods, xls/biff7, xls/biff8, and xlsx.

A round-trip test is a test that when we convert to a given format and back to our own Gnumeric format, nothing changes. Or, if something does change, then only parts we understand change. If the format is deficient, there is not much we can do. For example: ods cannot store patterned background or the sheet size; xlsx cannot store solver parameters; xls/biff7 cannot store arbitrary unicode; and xls/biff8 has a fixed sheet size of 64k-by-256. Note, that by itself a round-trip test does not guarantee that what we produce is correct. We could, hypothetically, have swapped division and multiplication are still gotten a perfect round-trip. To test that the generated files are correct one has to load the resulting sheet in Excel or LibreOffice (which, despite claims to the contrary, are what really defines xls/xlsx and ods formats). Unfortunately, I do not know how to script that so it is not automatic.

As a result of all the new tests, the recently released Gnumeric 1.12.12 should interact better with other spreadsheets.

Note: some of these new tests were probably a decade overdue. My excuse is that The Gnumeric Team is fairly small. I do hope that LO/OO already have an evil test suite, but I am not optimistic. I ran a few of my test sheets through LO and saw things like truncated strings.