Technical Blog of Richard Hughes

An ambient light sensor is a little light-to-frequency chip that you’ve certainly got in your tablet, most probably in your phone and you might even have one in your laptop if you’re lucky. Ambient light sensors let us change the panel brightness in small ways so that you can still see your screen when it’s sunny outside, but we can dim it down when the ambient room light is lower to save power. Lots of power.

There is a chicken and egg problem here. Not many laptops have ambient light sensors; some do, but driver support is spotty and they might not work, or work but return values with some unknown absolute scale. As hardware support is so bad, we’ve not got any software that actually uses the ALS hardware effectively, and so most ALS hardware goes unused. Most people don’t actually have any kind of ALS at all, even on high-end models like Thinkpads

So, what do we do? I spent a bit of time over the last few weeks designing a small OpenHardware USB device that acts as a ALS sensor. It’s basically a ColorHug1 with a much less powerful processor, but speaking the same protocol so all the firmware update and test tools just work out of the box. It sleeps between readings too, so only consumes a tiiiiny amount of power. I figure that with hardware that we know works out of the box, we can get developers working on (and testing!) the software integration without spending hours and hours compiling kernels and looking at DSDTs. I was planning to send out devices for free to GNOME developers wanting to hack on ALS stuff with me, and sell the devices for perhaps $20 to everyone else just to cover costs.

The device would be a small PCB, 12x22mm in size which would be left in a spare USB slot. It only sticks out about 9mm from the edge of the laptop as most of the PCB actually gets pushed into the USB slot. It’s obviously non-ideal, and non-sexy, but I really think this is the way to break the chicken/egg problem we have with ALS sensors. It obviously costs money to make a device like this, and the smallest batch possible is about 60 – so before I spend any more of my spare money/time on this, is anyone actually interested in saving tons of power using an ALS sensor and dimming the display? Comment here or email me if you’re interested. Thanks.

I’ve been using F3 to check my flash drives, and this is how I discovered my drives were counterfeit. It seems to me this kind of feature needs to be built inside gnome-multi-writer itself to avoid sending fake flash out to customers. Last night I wrote a simple tool called gnome-multi-writer-probe which does the following few things:

* Reads the existing data from the drive in 32kb chunks every 32Mbish into RAM
* Writes random blocks of 32kb every 32MBish, and also stores in RAM
* Resets the drive
* Reads all the 32k blocks from slightly different addresses and sizes and compares them to the random data in RAM
* Writes all the saved data back to the drive.

I only takes a few seconds on most drives. It also tries to be paranoid, and saves the data back to the drive the best it can when it encounters an error. That said, please don’t use this tool on any drives that have important data on them; assume you’ll have to reformat them after using this tool. Also, it’s probably a really good idea to unmount any drives before you try this.

If you’ve got access to gnome-multi-writer from git (either from jhbuild, or from my repo) then please could you try this:

sudo gnome-multi-writer-probe /dev/sdX

Where sdX is the USB drive you want to test. I’d be interested of the output, and especially interested if you have any fake flash media you can test this with. Either leave a comment here, grab me on IRC or send me an email. Thanks.

tl;dr Don’t use promo-newa.com, they are scammers that sell fake flash.

Longer version: For the ColorHug project we buy a lot of the custom parts direct from China at a fraction of the price available to us in the UK, even with import tax considered. It would be impossible to produce such a low cost device and still make enough money to make it worth giving up our evenings and weekends. This often means sending thousands of dollars to sketchy-looking companies willing to take on small (to them!) custom orders of a few thousand parts.

So far we’ve been very lucky, until last week. I ordered 1000 customized 1GB flash drives to use as a LiveUSB image rather than using a LiveCD. I checked out the company as usual, and ordered a sample. The sample came back good quality, with 1GB of fast flash. Payment in full was sent, which isn’t unusual for my other suppliers in China.

Fast forward a few weeks. 1000 USB drives arrived, which look great. Great, until you start using them with GNOME MultiWriter, which kept throwing validation warnings. Using the awesome F3 and a few remove-insert cylces later, the f3probe tool told me the flash chip was fake, reporting the capacity to be 1GB, when it was actually 96Mb looped around 10 times.

Taking the drives apart you could also see the chip itself was different from the sample, and the plastic molding and metal retaining tray was a lower quality. I contacted the seller, who said he would speak to the factory later that day. The seller got back to me today, and told me that the factory has produced “B quality drives” and basically, that I got what I paid for. For another 1600USD they would send me the 1GB ICs, which I would have to switch in the USB units. Fool me once, shame on you; fool me twice, shame on me.

I suppose people can use the tiny flash drives to get the .icc profile off the LiveCD image, which was always a stumbling block for some people, but basically the drives are worthless to me as LiveUSB devices. I’m still undecided whether to include them in the ColorHug box; i.e. is a free 96Mb drive better than them all going into landfill?

As this is China, I understand all my money is gone. The company listing is gone from Alibaba, so there’s not a lot I can do there. So other people can hopefully avoid this same mistake, I’ve listed all the details here, which hopefully will become googleable:

Promo-Newa Electronic Limited(Shenzhen)
Wei and Ping Group Limited(Hongkong)  

Office: Building A, HuaQiang Garden, North HuaQiang Road, Futian district, Shenzhen China, 0755-3631 4600
Factory: Building 4, DengXinKeng Industrial Zone, JiHua Road,LongGang District, Shenzhen, China
Registered Address: 15/B—15/F Cheuk Nang Plaza 250 Hennessy Road, HongKong
Email: sales@promo-newa.com
Skype: promonewa

Joshua from Plugable sent me 4 different USB hubs this week so they could be added as quirks to gnome-multi-writer. If you’re going to be writing lots of USB drives, the Plugable USB3 hubs now work really well. I’ve got a feeling that inserting and removing the drive is going to be slower than the actual writing and verifying now…

I’ve been talking to various people about the update descriptions we show to the user. Without exception, the messages we show to end users are really bad. For example, the overly-complex-but-not-actually-useful:

Or, the even more-to-the-point:

Update to 3.15.4

I’m guilty of both myself. Why is this? Typically this text is written an over-worked and under-paid packager doing updates to many applications and packages. Sometimes the packager might be the upstream maintainer, or at least involved in the project, but many times it’s just some random person that got fingered to maintain a particular package. This doesn’t make an awesome person to write beautiful prose and text that thousands of end users are going to read. It also doesn’t make sense to write the same beautiful prose again and again for every distribution out there.

So, what do we need? We need a person who probably wrote the code, or at least signed it off, who cares about the project and cares about the user experience. i.e. the upstream maintainer.

What I’m proposing is that we ask upstream maintainers to write the release information in a way that can be shown in the software center. NEWS files are not stanardized, and you don’t typically have a NEWS file for each application in your upstream tarball, so we need something else.

Suprise suprise, it’s AppStream to the rescue. AppStream has a <release> object that fits the bill almost completely; you can put upstream version information and long (optionally translated) formatted descriptions.

Of course, you don’t want to write both NEWS and the various appdata files at release time, as that just increased the workload of the overly-busy upstream maintainer. In this case we can use appstream-util appdata-to-news in the buildsystem and generate the former from the latter automatically. We’re outputting markdown for the NEWS file, which seems to be a fairly good approximation of what NEWS files actually look like at least for GNOME.

For a real-world example, see the GNOME MultiWriter example commit that uses this.

There are several problems with this approach. One is that the translators might have to translate lots more text; and the obvious solution to that seems to be to only mark strings to be translated for stable versions. Alas, projects like GNOME don’t allow any new strings in stable versions, so we’ll either have to come up with an except for release notes ammendment to that, or just say that all the release notes are only ever available in C locale.

The huge thing to take away from this blog, if you are intending to use this new feature is that update descriptions have to be understandable by end users. Various bug fixes is not helpful, but Fixes a crash when adding a joystick is. End users neither care or understand Use libgusb rather than libusbx and technical details that do not affect the UI or UX of the application should be omitted.

This doesn’t work for distribution releases, e.g. 3.14.1-1 to 3.14.1-2, but typically these are not huge changes that we need to show so prominently to the user.

I’m also writing a news-to-appdata.py script, so if anyone wants to take the plunge on an existing project it might be good to wait for that unless you like lots of copy and pasting.

Comments, as always, welcome.

When you do a search in GNOME Software it returns any result of any application with AppStream metadata and with a package name it can resolve in any remote repository. This works really well for software you’re installing from the main distribution repos, but less well for some other common cases.

Lets say I want to install Google Chrome so that my 2 year old daughter can ring me on hangouts, and tell me that dinner is ready. Lets search for Chrome on my Fedora Rawhide system.

Whoa! Wait, how did you do that? First, this exists in /etc/yum.repos.d/google-chrome.repo — the important line being enabled_metadata=1. This means “download just the metadata even when enabled=0” and means we can get information about what packages are available in repos we are not enabling by default for legal or policy reasons.

[google-chrome]
name=google-chrome
baseurl=http://dl.google.com/linux/chrome/rpm/stable/x86_64
enabled=0
gpgcheck=1
repo_gpgcheck=1
enabled_metadata=1
gpgkey=https://dl-ssl.google.com/linux/linux_signing_key.pub

We’ve also got a little XML document with the AppStream metadata (just the long description and keywords) called /usr/share/app-info/xmls/google-chrome.xml which could be included in the usual vendor-supplied fedora-22.xml if that’s what we want to do.

The other awesome feature this unlocks is when we have addon repos that are not enabled by default. For instance, my utopia repo full of super new free software applications could be included in Fedora, and if the user types in the search word we ask if the repo should be enabled. This would solve a lot of use cases if we could ship .repo files for a few popular COPRs of stuff we don’t (yet) ship in Fedora, but are otherwise free and open source software.

All the components to do this are upstream in Fedora 22 (you need a new librepo, libhif, PackageKit, libappstream-glib and gnome-software, phew!) although I’m sure we’ll be tweaking the UI and UX before Fedora 22 is released. Comments welcome.

 

I’ve just released GNOME MultiWriter 3.15.2, which is the first release that makes it pretty much feature complete for me.

Reads and writes are now spread over root hubs to increase throughput. If you’ve got a hub with more than 7 ports and the port numbers don’t match the decals on the device please contact me for more instructions.

In this release I’ve also added the ability to completely wipe a drive (write the image, then NULs to pad it out to the size of the media) and made that and the verification step optional. We also now show a warning dialog to the user the very first time the application is used, and some global progress in the title bar so you can see the total read and write throughput of the application.

With this release I’ve now moved the source to git.gnome.org and will do future releases to ftp.gnome.org like all the other GNOME modules. If you see something obviously broken and you have GNOME commit access, please just jump in and fix it. The translators have done a wonderful job using transifex, but now I’m leaving the just-as-awesome GNOME translator teams handle localisation.

If you’ve got a few minutes, and want to try it out, you can clone the git repo or install a package for Fedora.

Richard

Today I released the first version of GNOME MultiWriter, and built it for Rawhide and F21. It’s good enough for a first release, although there are still a few things left to do. The most important now is probably the self-profiling mode so that we know the best number of parallel threads to use for the read and the write. I want this to Just Work without any user interaction, but I’ll have to wait for my shipment of USB drives to arrive before I can add that functionality.

Also important to the UX is how we display failed devices. Most new USB devices accept the ISO image without a fuss, but the odd device will disconnect before completion or throw a write error. In this case it’s important to know which device is the one that belongs in the rubbish bin. This is harder than you think, as the electrical port number is not always what matches the decal on the plastic box.

For my test system I purchased a 10-port USB hub. I was interested to know how the vendor implemented this, as I don’t know of a SOIC chip that can drive more than 7 ports. It turns out, my 10-port hub is actually a 4-port hub, with a 7-port hub attached to the last port of the first hub. The second hub was also wired 1,2,3,4,7,6,5 rather than 1,2,3,4,5,6,7. This could cause my dad some issues when we tell him that device #5 needs removing.

I’ve merged some code into GNOME MultiWriter to work around this, but if you’ve got a hub with >7 ports I’d be interested to know if this works for you, or if we need to add some more VID/PID matches. If you do try this out you need libgusb from git master today. Helpfully gnome-multi-writer outputs quirk info to the command line if you use --verbose, so that makes debugging this stuff easier.

I spent last night writing a GNOME application to duplicate a ton of USB devices. I looked at mdcp, Clonezilla and also just writing something loopy in bash, but I need something simple my dad could use for a couple of hours a week without help.

It’s going to be super useful for me when I start sending our LiveUSB disks in the ColorHug box (rather than LiveCDs) and possibly useful to other people just wanting to copy a USB drive for QA testing on a small group of people, a XFCE live CD of Fedora rawhide for a code sprint, and that kind of thing.

GNOME MultiWriter allows you to write and verify an ISO file to up to 20 USB devices at once.

Bugs (and especially pull requests) accepted on GitHub; if there’s sufficient interest I’ll move the project to git.gnome.org after a few releases.

My OpenHardware post about an entropy source got loads of high quality comments, huge thanks to all who chimed in. There look to be a few existing projects producing OpenHardware, and the various comments have links to the better ones. I’ll put this idea back on the shelf for another holiday-hacking session. I’ve still not given up on the SD card interface, although it looks like emulating a storage device might be the easiest and quickest route for any future project.

So, on to the next idea. An OpenHardware USB ambient light sensor. A lot of hardware doesn’t have a way of testing the ambient light level. Webcams don’t count, they use up waaaay too much power and the auto-white balence is normally hardcoded in hardware. So I was thinking of producing a very low cost mini-dongle to measure the ambient light level so that lower-spec laptops could be saving tons of power. With smartphones people are now acutely aware than up to 60% of their battery power is just making the screen light up and I’m sure we could be smarter about what we do in GNOME. The problem traditionally, has been the lack of hardware with this support.

Anyone interested?