Posts Tagged ‘GNOME’

LibreOffice Con in Bern, Switzerland

Sunday, September 28th, 2014

I was invited to give a talk in Bern, Switzerland, for the LibreOffice Conference. The LibreOffice people are a nice crowd with diverse backgrounds. I talked to design people, coders doing rather low-level GL things, marketing folks, some being new to Free Software, and to some being old farts. It sounds like a lot of people and one is inclined to think of boat loads of people attending the conference when having the community statistics in mind. But it has been a very cosy event, with less than a hundred people. I found that surprising, but not necessarily in a bad way.

I couldn’t make it to many talks, because the conference took place on week days. But judging from the schedule there were many interesting talks. The only thing I didn’t like about the schedule was the weird formatting. Seriously, who makes the track’s name more visible than the talk’s title..? Also grouping by room and not by time is a bit weird.

Anyway, my talk went well although it was in the first slot after the free beer party ;-) You can find my slides in the collection. I was talking about GNOME in general, but with a twist for those who migrate from proprietary software to Free Software. I hope I could convey that the GNOME desktop might be a viable alternative to proprietary products.

As this was a great, comfortable conference, I’m looking forward to visiting next year’s event.

GUADEC 2014 in Strasbourg

Sunday, August 3rd, 2014

This year, GUADEC took place in the lovely Strasbourg in France. It was really nice to attend the conference and to hang around with people who care about Free Software. In fact, the venue itself ran Debian which was nice to see :-)

Unfortunately, I wasn’t able to attend many of the great talks as I wasn’t available for all days. And when I was, I was busy meeting people. Although it felt smaller than the last GUADEC, I think I’ve never met so many people who I wanted to talk to.

The conference offered a two-track program. Interestingly many of them looking out for a future of GNOME. John Stowers gave one of the more important talks, I think. He was describing the situation in academia. Python is very popular in the scientific computing space, he said. He was not satisfied with JavaScript being the new “default” language for GNOME applications, because the contestants are numerous and powerful. So we would compete at least against the Web and Qt. The former apparently being nice on other platforms such as Windows. GNOME’s bindings, however, were very good, he said. The technological foundation is excellent and we should leverage that potential and make people use it. However, GNOME’s story on Windows is not all too good, he said. GTK+ is becoming more and more irrelevant and even Wx appears to be as popular as Gtk. I also heard others claiming that the Windows situation is a problem. What I don’t understand is whether there are technical problems blocking easy to use ports. Apparently introspected GNOME libraries for Pyhon on Windows exist, but I don’t understand why that doesn’t do the job.

Another talk related to the future of GNOME was given byAllan Day. In order for GNOME to be successful, amongst other things, a focus on quality must be established, he said. Various ways to improve the current release process were mentioned and the audience engaged in a vivid discussion. I don’t remember the detail so I hope this will be followed up and discussed more broadly in the GNOME community.

“Why do we do desktop”, asked Matthew Garrett in his presentation. When I read that title for the first time I thought the question of the desktop becoming irrelevant was being picked up. But that was not the case. Instead, he wanted GNOME to differentiate from the existing desktops which, as he claimed, are continuing to be simple multiplexors for running several programs (such as clocks) at the same time. In contrast to existing desktop, GNOME should become the secure desktop. Other desktops, he said, would only exist in order to sell more things to the user, i.e. to tie the user to an existing ecosystem. An advantage of GNOME is it being free from corporate control. Decisions are made very transparently which enables it to focus on brining privacy and security to the user. Even if the user is not aligned with our core values and principles. As such, every user deserves as much privacy and security as we can possible provide.

Many thanks to the local team for having organised the conference. I hope next year in Gothenburg will be at least as good.

Sponsored by GNOME!

GPN 2014 in Karlsruhe

Monday, June 30th, 2014

The Gulash Programmier Nacht (GPN) took place in Karlsruhe, Germany. The local subsidiary of the Chaos Computer Club organised that event, which apparently took place for the 14th time. So far, I wasn’t able to attend, but this time I made it.

It’s a 200 to 300 people event, focussed at hacking, making, and talks around that. It’s very cosy and somewhat similar to the mrmcds. Most of the talks were held in German, a few in English, but I think that could easily change if there is a demand.

The conference was keynoted by tante, who talked about the political aspects of code and the responsibility every developer has. It was good to hear someone saying that you do create reality for people with the software you write and that you are indeed responsible for the view on the world the users of your software have. There were a few other interesting thoughts and I think I agree with the results of the analysis conducted to a great extent. But I think a few areas are not well covered. For example, he said that you limit the people with your software. I don’t think that’s necessarily true. If you provide your users with enough freedoms, i.e. by choosing a Free Software license, than I don’t think his argument is valid anymore.

On the more funny side, a chemist taught us about chemistry based on the stories of Walter White. It was a funny talk with many interludes of the TV series. She explained what the people in the episodes were doing and how close that is to reality. Turns out, it is quite close and at least stupid mistakes were not done.

We also learned about Perl 6. If you think Perl is ugly, he said, it’s not modern Perl. The new and shiny Perl 6 allows you to write short code while looking nice, he said. He showed some features that make it easy to write command line tools. You can simply declare an argument to your main function and Perl would expose that to the user, e.g. by presenting a help screen. It would also detect the types provided and do some magic fancy stuff like checking whether the provided argument is an existing (or empty) file.

A very interesting talk was given on the Enigma, the German crypto machine. He showed the machine that broke the crypto and now stands in Bletchley Park. He told stories about the development and operation of that machine. Very interesting indeed. Also well done on a technical level, the slides were really well done.

I was invited to give talk on GNOME. As you can see in the video, my battery didn’t even last the full 90 minutes slot I was assigned. Something is certainly wrong, either this Linux thing or my battery. Anyway, the talk itself went very well, and it was particularly well attended for that early slot. I was also positively surprised by the audience asking many questions and while I specifically asked for flames, I didn’t get that many.

GNOME.Asia Summit 2014

Friday, June 20th, 2014

I was fortunate to be able to attend this year’s GNOME Asia Summit in Beijing, China.

It was co-hosted with FUDCon, the Fedora Users and Developers Conference. We had many attendees and the venue provided good facilities to talk about Free Software and the Free Desktop.

Fudcon Beijing Logo

The venue was the Beihai University somewhat north of Beijing. Being Chinese, the building was massive in size. So we had loads of space, anyway ;-) The first day was reserved for trainings and attendees could get their feet wet with thinks like developing a GNOME application. I took part myself and was happy to learn new GNOME APIs. I think the audience was interested and I hope we could inspire a few attendees to create their next application using GNOME technologies.

I was invited to keynote the conference. It was my first time to do such a thing and I chose to give a talk that I would expect from a keynote, namely something that leads the conference and gives a vision and ideas about what to discuss during the conference. I talked on GNOME, GNOME 3, and GNOME 3.12. I tried to promote the ideas of GNOME and of Free Software. Unfortunately, I prepared for 60 minutes rather than 45, so I needed to cut off a good chunk of my talk :-/ Anyway, I am happy with how it went and especially happy with the fact that I wasn’t preaching to the choir only, as we had e.g. Fedora people in the audience, too.

We had RMS explaining Free Software to the audience and I think the people enjoyed his talking. I certainly did, although I think it doesn’t address problems we face nowadays. People have needs, as the discussion with the audience revealed. Apparently, people do want to have the functionality Facebook or Skype offers. I think that addressing these needs with the warning “you must not fall for the convenience trap” is too short sighted. We, the Free Software community, need to find better answers.

The event was full of talks and workshops from a diverse range of topics, which is a good thing for this conference. Of course, co-hosting with FUDCon helped that. The event is probably less technical than GUADEC and attendees can learn a lot from listening and talking to other people. I hope we can attract more Asian people to Free Software this way. I am not entirely sure we need to have the same setup as with GUADEC though. With GUADEC, we change the country every year. But Asia is about ten times larger than Europe. In fact, China alone is larger than all of Europe. It makes it somewhat hard for me to justify the moving around. We do need more presence in Asia, so trying to cover as much as possible might be an approach to attract more people. But I think we should investigate other approaches, such as focussing on an annual event in one location to actually create a strong Free Software location in Asia, before moving on. I wouldn’t know how to define “strong” right now, but we have absolutely no measure of success right now, anyway. That makes it a bit frustrating for me to pour money over Asia without actually seeing anything in return.

Anyway, Beijing is fun. We went to see the Great Wall and enjoyed the subway ;)

I would like to thank the organisers for having provided a great place us, the Free Software community, to spread the word about the benefits of free computing. I would also like to thank the GNOME Foundation for enabling people like me to attend the event.

Sponsored by GNOME!

Installing OpenSuSE 13.1 on an Lenovo Ideapad S10-3t

Monday, June 9th, 2014

I tried to install the most recent OpenSuSE image I received when I attended the OpenSuSE Conference. We were given pendrives with a live image so I was interested how smooth the OpenSuSE installation was, compared to installing Fedora. The test machine is a three to four year old Intel Ideapad s10-3t, which I received from Intel a while ago. It’s certainly not the most powerful machine, but it’s got some dual core CPU, a gigabyte of RAM, and a widescreen touch display.

The initial boot took a while. Apparently it changed something on the pendrive itself to expand to its full size, or so. The installation was a bit painful and, at the end of the day, not successful. The first error I received was about my username being wrong. It told me that I must only contain letters, digits, and other things. It did not tell me what was actually wrong; and I doubt it could, because my username was very legit. I clicked away the dialogue and tried again. Then it worked…

When I was asked about my partitioning scheme I was moderately confused. The window didn’t present any “next” button. I clicked the three only available buttons to no avail until it occurred to me that the machine has a wide screen so the vertical space was not sufficient to display everything. And yeah, after moving the window up, I could proceed.

While I was positively surprised to see that it offered full disk encryption, I wasn’t too impressed with the buttons. They were very tiny on the bottom of the screen, barely clickable.

Anyway, I found my way to proceed, but when attempting to install, YaST received “system error code -1014″ and failed to partition the disk. The disk could be at fault, but I have reasons to believe it was not the disks fault:

Apparently something ate all the memory so that I couldn’t even start a terminal. I guess GNOME’s system requirements are higher than I expected.

OpenSuSE Conference 14 in Dubrovnik, Croatia

Saturday, May 3rd, 2014

I had the pleasure to be invited to the 2014 edition of the OpenSuSE Conference in Dubrovnik, Croatia. That event was flying under my radar for a long time and I am glad that I finally found out about it.

The first thing that impressed me was Dubrovik. A lovely city with a walled old town. Even a (rather high) watch tower is still there. The city manages to create an inspiring atmosphere despite all the crowds moving through the narrow streets. It’s clean and controlled, yet busy and wild. There are so many small cafés, pubs, and restaurants, so many walls and corners, and so many friendly people. It’s an amazing place for an amazing conference.

The conference itself featured three tracks, which is quite busy already. But in addition, an unconference was held as a fourth track. The talks were varying in topic, from community management, to MySQL deployment, and of course, GNOME. I presented the latest and greatest GNOME 3.12. Despite the many tracks, the hallway track was the most interesting one. I didn’t know too many faces and as it’s a GNU/Linux distribution conference which I have never attended before, many of the people I met had an interesting background which I was not familiar with. It was fun meeting new people who do exciting things. I hope to be able to stay in touch with many of them.

The conference was opened by the OpenSuSE Board. I actually don’t really know how OpenSuSE is governed and if there is any legal entity behind it. But the Board seems to be somehow elected by the community and was to announce a few changes to OpenSuSE. The title of the conference was “The Strength to Change” which is indeed inviting to announce radical changes. For better or worse, both the number and severity of the changes announced were limited. First and foremost, handling marketing materials is about to change. A new budget was put in place to allow for new materials to be generated to have a much bigger presence in the world. Also, the materials were created by SuSE’s designers on staff. So they are considered to be rather high quality. To get more contributors, they introduce formalised sponsorship program for people to attend conferences to present OpenSuSE. I don’t know what the difference to their Travel Support Program is, though. They will also reimburse for locally produced marketing materials which cannot be shipped around the world to encourage more people to spread the word about OpenSuSE. A new process will be put in place which will enable local contributors to produce materials up to 200 USD from a budget of 2000 USD per quarter. Something that will change, but not just yet, is the development and release model. Andrew Wafaa said that OpenSuSE was a victim of its own success. He mentioned the number of 7500 packages which should probably indicate that it is a lot for them to handle. The current release cycle of 8 months is to be discussed. There is a strong question of whether something new shall be tried. Maybe annual releases, or even longer to have more time for polish. Or maybe not do regular releases at all, like rolling releases or just take as long as it takes. A decision is expected after the next release which will happen as normal at the end of this year. There was an agreement that OpenSuSE wants to be easy to contribute to. The purpose of this conference is to grow the participants’ knowledge and connections in and about the FLOSS environment.

The next talk was Protect your MySQL Server by Georgi Kodinov. Being with MySQL since 2006 he talked about the security of MySQL in OpenSuSE. The first point he made was how the post-installation situation is on OpenSuSE 13.1. It ships version 5.6.12 which is not too bad because it is only 5 updates behind of what upstream released. Other distros are much further away from that, he said. Version 5.6 introduced cool security related features like expiring passwords, password strength policies, or SHA256 support. He urged the audience to stop using passwords on the command line and look into the 5.6 documentation instead. He didn’t make it any more concrete, though, but mentioned “login paths” later. He also liked that the server was not turned on by default which encourages you to use your self-made configuration instead of a default one. He also liked the fact that there is no pre-packaged database as that does not configure users that are not very well protected. Finally, he pointed out that he is pleased to see that no remote access is configured in the default configuration. However, he did not like that OpenSuSE does not ship the latest version. The newest upstream version 5.6.15 not only fixes around 25 security problems but also adds advanced AES functionalities such as keys being bigger than 128 bits. He also disliked that a mysql_secure_installation script is not run after installation. That script would put random passwords to the root account, would disallow anonymous access, and would do away with empty default passwords. Another regret he had was that mysql_config_editor is not packaged. That tool would help to get rid of passwords in scripts using MySQL by storing credentials in encrypted files. That way you would have to protect only one file, not a lot of scripts. For some reason OpenSuSE activates the “federated plugin” which is disabled upstream.
Another weird plugin is the archive plugin which, he said, is not needed. In fact, it is not even available so that the starting server throws errors… Also, authentication plugins which should only be used for testing are enabled by default which can be a problem as it could allow someone to log in as any user. After he explained how this was a threat, the actual attack seems to be a bit esoteric. Anyway, he concluded that you get a development installation when you install MySQL in OpenSuSE, rather than an installation suited for production use.

He went on to refer about how to harden it after installation. He proposed to run mysql_secure_installation as it wouldn’t cause any harm even if run multiple times. He also recommended to make it listen on specific interfaces only, instead of all interfaces which is does by default. He also wants you to generate SSL keys and certificates to allow for encrypted communication over the network.

Even more security can be achieved when turning off TCP access altogether, so you should do it if the environment allows it. If you do use TCP, he recommended to use SSL even if there is no PKI. An interesting advice was to use external authentication such as PAM or LDAP. He didn’t go into details how to actually do it, though. The most urgent tip he gave was to set secure_file_priv to a certain directory as it will restrict the paths MySQL can write to.

As for new changes that come with MySQL 5.7, which is the current development version accumulating changes over 18 months of development, he mentioned the option to log to syslog. Interestingly,
a --ssl option on the client is basically a no-op (sic!) but will actually enforce SSL in the upcoming version. The new version also adds more crypto functions such as RANDOM_BYTES() which interface with the SSL libraries. He concluded his talk with a quote: “Security is like plastic surgery. the more you invest, the prettier it gets.”.

Michael Meeks talked next on the history of the Document Foundation. He explained how it used to be in the StarOffice days. Apparently, they were very process driven and believed that the more processes with even more steps help the quality of the software they produced. He didn’t really share that view. The mind set was, he said, that people would go into a shop and buy a box with the software. He sees that behaviour declining steeply. So then hackers came and branched StarOffice into OpenOffice which had a much shorter release cycle than the original product and incorporated fixes and features of the future version. Everyone shipped that instead of the original thing. The 18 months of the original product were a bit of a long thing in the free software world, he said. He quoted someone saying “StarDivision a problem for every solution.”

He went on to rant about Contributor License Agreements and showed a graph of Fedora contributions which spiked off when they dropped the requirement of a CLA. The graph was impressive but really showed the number of active accounts in an unspecified system. He claimed that by now they have around the same magnitude of contributions as the kernel does and with set a new record with 3000 commits in February 2014. The dominating body of contributors is volunteers which is quite different when compared to the kernel. He talked about various aspects of the Document Foundation like the governance or the fact that they want to make it as easy to contribute to the project as possible.

The next talk was given on bcache by Oliver Neukum. Bcache is a disk cache which is probably primarily used to cache rotational disks with SSDs. He first talked about the principles of caching, like write-back, write-through, and write-around. That is, the cache is responsible for writing to the backing store, the cache places the data to be written in its buffer, or write to the backing storage, but not the cache, respectively. Subsequently, he explained how to actually use bcache. A demo given later revealed that it’s not fool proof and that you do need to get your commands straight in order to make it work properly. As to when to actually use Bcache, he explained that SSDs are cool as they are fast, but they are small and expensive. Fast, as he continued, can either mean throughput or latency. SSDs are good with regards to latency, but not necessarily with throughput. Other, probably similar options to Bcache are dm-cache, but it does not support safe writes. I guess that you cannot use it if you have the requirement of a write-through or write-around scenario. A different alternative is EnhanceIO, written originally by Facebook, which keeps hash structure of the data to be cached in RAM. Bcache, on the other hand, stores a b-tree on the SSD instead of in the RAM. It works on block devices, so anything goes. Tape drives, RAIDs, … It places a special superblock to indicate the partition is a bcache partition. A second block is created to indicate what the backing store is. Currently, the kernel does not auto detect these caches, hence making it work with the root filesystem is a bit tricky. He did a proper evaluation of the effects of the cache. So his statements were well founded which I liked a lot.

It was announced that the next year’s conference, oSC15, will be in The Hague, Netherlands. The city we had our GUADEC in, once. If you have some time in spring, probably in April, consider to go.

Installing Fedora 20 on an Exo PC Slate

Saturday, March 1st, 2014

For our booth at FOSDEM we had some hardware to show off the latest and greatest GNOME. I brought the tablet I got from the Desktop Summit. In order to prepare it I installed Fedora 20 which comes with a nice and shiny installer. I found a few issues and glitches and will present then them in the course of this post.

It worked well enough, but it has a few glitches. One of them is that GNOME apparently does not detect that it is running on hardware which does not have a keyboard. So it was a bit to enter a password for a wifi when there is no (soft) keyboard…

Some incoherences exist. One of them is that it shows the “Next” button on the bottom right. Which is what I’d expect. But sometimes it also asks the user to press a button on the upper left. I didn’t even remotely expect having to press a button on the “back” side of the screen in order to continue installation.

It was very nice though that it seems to offer installation along an existing operating system *and* full disk encryption. The Ubuntu installer can install you a fully encrypted system nicely, but only if you install Ubuntu on the whole disk. The Fedora installer seems to manage that nicely.

As it seems to be normal nowadays, installation starts even though you haven’t provided all the necessary information yet. That is very convenient and have a much fast installation experience.

Another coherence issue is on the user dialogue. I can actually guess what the thinking was when designing these menus. You have this “overview” screen as seen above and then you “dive” into the sub menus. I expected a more linearly following set of menus. Why would I need return to the overview menu at all? I claim that it is much easier to just continue, not to go forth and back… Anyway, a real bug is visible: Mnemonics are not formatted properly.

The user dialogue, while being at it, seemed to have forced me to enter a strong password. I just wanted to install a system for a demo machine. Probably not the usual usecase, but annoying enough if it doesn’t work smoothly. I think I found out later that I needed to double press the “Next” button (labelled “done” and being placed in the area of the screen where I’d expect “back” buttons to be).

Turns out, that the same thing happened with the root password, which really annoyed me. Especially as the soft keyboard doesn’t really allow for convenient input of complicated characters.

But then I discovered something. On the very bottom there something weirdly coloured. It was a notification for the current menu. Why on earth complain about the password I’ve entered on the very bottom when the widget is on the very top? That was surprising and confusing. Plus, the warning itself was not very visible due to the onscreen keyboard obstructing the view. So I guess it’d be smarter to not have the warnings there.

Anyway, I was pleasantly surprised how smooth the installation experience was. It could, of course, have been better, but all in all it was quite good. I finished in less than half and hour. Too bad that I didn’t know that neither Eye of GNOME nor Epiphany were installed by default.

Converting Mailman archives (mboxes) to maildir

Tuesday, November 13th, 2012

I wanted to search discussions on mailing lists and view conversations. I didn’t want to use some webinterface because that wouldn’t allow me to search quickly and offline. So making my mail client aware of these emails seemed to be the way to go. Fortunately, the GNOME mailinglists are mbox archived. So you download the entire traffic in a standardised mbox.

But how to properly get this into your email clients then? I think Thunderbird can import mbox natively. But I wanted to access it from other clients, too, so I needed to make my server aware of these emails. Of course, I configured my mailserver to use maildir, so some conversion was needed.

I will present my experiences dealing with this problem. If you want to do similar things, or even only want to import the mbox directly, this post might be for you.

The archives

First, we need to get all the archives. As I had to deal with a couple of mailinglists and more than a couple of month, I couldn’t be arsed to click every single mbox file manually.

The following script scrapes the mailman page. It makes use of the interesting Splinter library, basically a wrapper around selenium and other browsers for Python.

#!/usr/bin/env python

import getpass
from subprocess import Popen, list2cmdline
import sys

import splinter

def fill_password(b, username=None, password=None):
    if not username:
        username = getpass.getpass('username: ')
    if not password:
        password = getpass.getpass('password: ')
        
    b.fill('username', username)
    b.fill('password', password)
    b.find_by_name('submit').click()


def main(url, username=None):
    b = splinter.Browser()
    
    try:
        #url = 'https://mail.gnome.org/mailman/private/board-list/'
        b.visit(url)
        
        if 'Password' in b.html:
            fill_password(b, username=username)


        links = [l['href'] for l in b.find_link_by_partial_text('Text')]

        cookie = b.driver.get_cookies()[0]
        cookie_name = cookie['name']
        cookie_value = cookie['value']
        cookie_str = "Cookie: {name}={value}".format(name=cookie_name, value=cookie_value)
        wget_cookie_arg = '--header={0}'.format(cookie_str)
        #print  wget_cookie_arg
        
        b.quit()

        
        for link in links:
            #print link
            cmd = ['wget', wget_cookie_arg, link]
            print list2cmdline(cmd)
            # pipe that to "parallel -j 8"

    except:
        b.quit()


if __name__ == '__main__':
    site = sys.argv[1]
    user = sys.argv[2]
    
    if site.startswith('http'):
        url=site
    else:
        url = 'https://mail.gnome.org/mailman/private/{0}'.format(site)
    
    main(username=user, url=url)

        

You can download the thing, too.

I use splinter because handling cookies is not fun as well as parsing the web page. So I just use whatever is most convenient for me, I wanted to get things done, after all. The script will print a line for each link it found, nicely prefixed with wget and its necessary arguments for the authorization cookie. You can pipe that to sh but if you want to download many month, you want to do it in parallel. And fortunately, there is an app for that!

Conversion to maildir

After having received the mboxes, it turned out to be a good idea nonetheless to convert to maildir; if only to extract properly formatted mails only and remove duplicates.

I came around mb2md-3.20.pl from 2004 quite soon, but it is broken. It cannot parse the mboxes I have properly. It will create broken mails with header lingering around as it seems to be unable to detect the beginning of new mails reliably. It took me a good while to find the problem though. So again, be advised, do not use mb2md 3.20.

As I use mutt myself I found this blog article promising. It uses mutt to create a mbox out of a maildir. I wanted it the other way round, so after a few trial and errors, I figured that the following would do what I wanted:

mutt -f mymbox -e 'set mbox_type=maildir; set confirmcreate=no; set delete=no; push "T.*;s/tmp/mymuttmaildir"'

where “mymbox” is your source file and “/tmp/mymuttmaildir” the target directory.

This is a bit lame right? We want to have parameters, because we want to do some batch processing on many archive mboxes.

The problem is, though, that the parameters are very deep inside the quotes. So just doing something like

mutt -f $source -e 'set mbox_type=maildir; set confirmcreate=no; set delete=no; push "T.*;s$target"'

wouldn’t work, because the $target would be interpreted as a raw string due to the single quotes. And I couldn’t find a way to make it work so I decided to make it work with the language that I like the most: Python. So an hour or so later I came up with the following which works (kinda):

import os
import subprocess
source = os.environ['source']
destination = os.environ['destination']

conf = 'set mbox_type=maildir; set confirmcreate=no; set delete=no; push "T.*;s{0}"'.format(destination)

cmd = ['mutt', '-f', source, '-e', conf]
subprocess.call(cmd)

But well, I shouldn’t become productive just yet by doing real work. Mutt apparently expects a terminal. It would just prompt me with “No recipients were specified.”.

So alright, this unfortunately wasn’t what I wanted. I you don’t need batch processing though, you might very well go with mutt doing your mbox to maildir conversion (or vice versa).

Damnit, another two hours or more wasted on that. I was at the point of just doing the conversion myself. Shouldn’t be too hard after all, right? While researching I found that Python’s stdlib has some email related functions *yay*. Some dude on the web wrote something close to what I needed. I beefed it up a very little bit and landed with the following:

#!/usr/bin/env python

# http://www.hackvalue.nl/en/article/109/migrating%20from%20mbox%20to%20maildir

import datetime
import email
import email.Errors
import mailbox
import os
import sys
import time


def msgfactory(fp):
    try:
        return email.message_from_file(fp)
    except email.Errors.MessageParseError:
        # Don't return None since that will
        # stop the mailbox iterator
        return ''
dirname = sys.argv[1]
inbox = sys.argv[2]
fp = open(inbox, 'rb')
mbox = mailbox.UnixMailbox(fp, msgfactory)


try:
        storedir = os.mkdir(dirname, 0750)
        os.mkdir(dirname + "/new", 0750)
        os.mkdir(dirname + "/cur", 0750)
except:
        pass

count = 0
for mail in mbox:
        count+=1
        #hammertime = time.time() # mail.get('Date', time.time())
        hammertime = datetime.datetime(*email.utils.parsedate(mail.get('Date',''))[:7]).strftime('%s')
        hostname = 'mb2mdpy'
        filename = dirname + "/cur/%s%d.%s:2,S" % (hammertime, count, hostname)
        mail_file = open(filename, 'w+')
        mail_file.write(mail.as_string())


print "Processed {0} mails".format(count)

And it seemed to work well! It recovered many more emails than the Perl script (hehe) but the generated maildir wouldn’t work with my IMAP server. I was confused. The mutt maildirs worked like charm and I couldn’t see any difference to mine.

I scped the file onto my .maildir/ on my server, which takes quite a while because scp isn’t all too quick when it comes to many small files. Anyway, it wouldn’t necessarily work for some reason which is way beyond me. Eventually I straced the IMAP server and figured that it was desperately looking for a tmp/ folder. Funnily enough, it didn’t need that for other maildirs to work. Anyway: Lesson learnt: If your dovecot doesn’t play well with your maildir and you have no clue how to make it log more verbosely, check whether you need a tmp/ folder.

But I didn’t know that so I investigated a bit more and I found another PERL script which converted the emails fine, too. For some reason it put my mails in “.new/” and not in “.cur/“, which the other tools did so far. Also, it would leave the messages as unread which I don’t like.

Fortunately, one (more or less) only needs to rename the files in a maildir to end in S for “seen”. While this sounds like a simple

for f in maildir/cur/*; do mv ${f} ${f}:2,S

it’s not so easy anymore when you have to move the directory as well. But that’s easily being worked around by shuffling the directories around.

Another, more annoying problem with that is “Argument list too long” when you are dealing with a lot of files. So a solution must involve “find” and might look something like this: find ${CUR} -type f -print0 | xargs -i -0 mv '{}' '{}':2,S

Duplicates

There was, however, a very annoying issue left: Duplicates. I haven’t investigated where the duplicates came from but it didn’t matter to me as I didn’t want duplicates even if the downloaded mbox archive contained them. And in my case, I’m quite confident that the mboxes are messed up. So I wanted to get rid of duplicates anyway and decided to use a hash function on the file content to determine whether two file are the same or not. I used sha1sum like this:

$ find maildir/.board-list/ -type f -print0 | xargs -0 sha1sum   | head
c6967e7572319f3d37fb035d5a4a16d56f680c59  maildir/.board-list/cur/1342797208.000031.mbox:2,
2ea005ec0e7676093e2f488c9f8e5388582ee7fb  maildir/.board-list/cur/1342797281.000242.mbox:2,
a4dc289a8e3ebdc6717d8b1aeb88959cb2959ece  maildir/.board-list/cur/1342797215.000265.mbox:2,
39bf0ebd3fd8f5658af2857f3c11b727e54e790a  maildir/.board-list/cur/1342797210.000296.mbox:2,
eea1965032cf95e47eba37561f66de97b9f99592  maildir/.board-list/cur/1342797281.000114.mbox:2,

and if there were two files with the same hash, I would delete one of them. Probably like so:

    #!/usr/bin/env python
    import os
    import sys


    hashes = []
    for line in sys.stdin.readlines():
        hash, fname = line.split()
        if hash in hashes:
            os.unlink(fname)
        else:
            hashes.append(hash)

But it turns out that the following snippet works, too:

find /tmp/maildir/ -type f -print0 | xargs -0 sha1sum | sort | uniq -d -w 40 | awk '{print $2}' | xargs rm

So it’ll check the files for the same contents via a sha1sum. In order to make uniq detect equal lines, we need to give it sorted input. Hence the sort. We cannot, however, check the whole lines for equality as the filename will show up in the line and it will of course be different. So we only compare the size of the hex representation of the hash, in this case 40 bytes. If we found such a duplicate hash, we cut off the hash, take the filename, which is the remainder of the line, and delete the file.

Phew. What a trip so far. Let’s put it all together:

The final thing


LIST=board-list

umask 077

DESTBASE=/tmp/perfectmdir

LISTBASE=${DESTBASE}/.${LIST}

CUR=${LISTBASE}/cur
NEW=${LISTBASE}/new
TMP=${LISTBASE}/tmp

mkdir -p ${CUR}
mkdir -p ${NEW}
mkdir -p ${TMP}

for f in  /tmp/${LIST}/*; do /tmp/perfect_maildir.pl ${LISTBASE} < ${f} ; done
mv ${CUR} ${CUR}.tmp
mv ${NEW} ${CUR}
mv ${CUR}.tmp ${NEW}
find ${CUR} -type f -print0 | xargs -i -0 mv '{}'  '{}':2,S
find ${CUR} -type f -print0 | xargs -0 sha1sum | sort | uniq -d -w 40 | awk '{print $2}' | xargs rm

And that’s handling email in 2012…

Interview for gnome.org

Friday, August 3rd, 2012

I was interviewed recently for GNOME.org and while you can read the interview over there, I felt like copying it over here. So enjoy the questions and the answers

Why is open source/free software important to you?

I believe that Free Software makes the world a better place. Also, as I am a bit of a computer security person, it is absolutely crucial to be able to see how the software in question works and be able to eventually fix issues (or have someone to fix them).

How/when/why did you become involved in GNOME?

I was using GNOME ever since and started to follow it more and more until I came greatly involved as a Summer of Code student.

Why did you run for the GNOME Foundation Board?

I am sticking around GNOME for about 5 years now and while I enjoy being in the community I do wanted to progress within GNOME and take new responsibilities.

What do you hope to accomplish during your term on the board?

I hope to push the revamp of the bylaws and enable people to work together more effectively.

Do you think GNOME is heading in the right direction? Why or why not?

I think GNOME is doing well so far, but it must not rest (decadence anyone?). We have very smart people in our community and we should enable them to get awesome stuff for GNOME and the Free Software world done.

Have you attended GUADEC in the past? If so, when/where?

My first GUADEC was the one in Birmingham in 2007.

What are you looking forward to most at GUADEC?

To see friends again and having nice discussions.

Any other thoughts on GUADEC and/or GNOME?

GNOME is a great community and GUADEC is a good place to get in touch. Sometimes though, it might not be comfortable for newcomers to chime in. I still remember myself being too shy to talk to some great GNOME people. So while I recommend to young GNOMErs to not be shy, maybe a “This is GNOME” introductory session for new GNOME people might be a good idea.

GNOME @ FOSDEM 2011

Sunday, February 27th, 2011

I am very excited about having attended this years FOSDEM. Unfortunately, times were a bit busy so I am a bit late reporting about it, but I still want to state a couple of things.

I'm going to FOSDEM, the Free and Open Source Software Developers' European Meeting (I wonder how that image will look in 2012 ;-) )

First of all, I am very happy that our GNOME booth went very well. Thanks to Frederic Peters and Frederic Crozat for manning to booth almost all the time. I tried to organise everything remotely and I’d say I partly succeeded. We got stickers, t-shirts and staff for the booth. We lacked presentation material and instructions for the booth though. But it still worked out quite well. For the next time, I’d try to be communicate more clearly who is doing what to prevent duplicate work and ensure that people know who is responsible for what.

Secondly, I’d like to thank Canonical for their generosity to sponsor a GNOME Event Box. After the orginal one went missing, Canocical put stuff like a PC, a projector, a monitor and lots of other stuff together for us to be able to show off GNOME-3. The old Box, however, turns out to be back again *yay*!

Sadly, we will not represent GNOME at upcoming CeBIT. But we will at LinuxTag. Latest.

Anyway, during FOSDEM, we got a lot of questions about GNOME 3 and Ubuntu, i.e. will it be easily possible to run GNOME 3 on Ubuntu. I hope we can make it possible to have a smooth transition from Unity to GNOME Shell. Interestingly enough, there isn’t a gnome-shell package in the official natty repositories yet :(

It was especially nice to see and talk to old GNOME farts. And I enjoyed socialising with all the other GNOME and non-GNOME people as well. Sadly, I didn’t like the GNOME Beer Event very much because it was very hot in the bar so I left very quickly.

So FOSDEM was a success for GNOME I’d say. Let’s hope that future events will work at least as well and that we’ll have a strong GNOME representation even after the GNOME 3 release.