Archive for the ‘hacking’ Category

Installing OpenSuSE 13.1 on an Lenovo Ideapad S10-3t

Monday, June 9th, 2014

I tried to install the most recent OpenSuSE image I received when I attended the OpenSuSE Conference. We were given pendrives with a live image so I was interested how smooth the OpenSuSE installation was, compared to installing Fedora. The test machine is a three to four year old Intel Ideapad s10-3t, which I received from Intel a while ago. It’s certainly not the most powerful machine, but it’s got some dual core CPU, a gigabyte of RAM, and a widescreen touch display.

The initial boot took a while. Apparently it changed something on the pendrive itself to expand to its full size, or so. The installation was a bit painful and, at the end of the day, not successful. The first error I received was about my username being wrong. It told me that I must only contain letters, digits, and other things. It did not tell me what was actually wrong; and I doubt it could, because my username was very legit. I clicked away the dialogue and tried again. Then it worked…

When I was asked about my partitioning scheme I was moderately confused. The window didn’t present any “next” button. I clicked the three only available buttons to no avail until it occurred to me that the machine has a wide screen so the vertical space was not sufficient to display everything. And yeah, after moving the window up, I could proceed.

While I was positively surprised to see that it offered full disk encryption, I wasn’t too impressed with the buttons. They were very tiny on the bottom of the screen, barely clickable.

Anyway, I found my way to proceed, but when attempting to install, YaST received “system error code -1014″ and failed to partition the disk. The disk could be at fault, but I have reasons to believe it was not the disks fault:

Apparently something ate all the memory so that I couldn’t even start a terminal. I guess GNOME’s system requirements are higher than I expected.

Split DNS Resolution

Thursday, April 17th, 2014

For the beginning of the year, I couldn’t make resolutions. The DNS server that the DHCP server gave me only resolves names from the local domain, i.e. acme.corp. Every connection to the outside world needs to go through a corporate HTTP proxy which then does the name resolution itself.

But that only works as long as the HTTP proxy is happy, i.e. with the destination port. It wouldn’t allow me to CONNECT to any other port than 80 (HTTP) or 443 (HTTPS). The proxy is thus almost useless for me. No IRC, no XMPP, no IMAP(s), no SSH, etc.

Fortunately, I have an SSH server running on port 443 and using the HTTP proxy to CONNECT to that machine works easily, i.e. using corkscrew with the following in ~/.ssh/config:

Host myserver443
  User remote-user-name
  HostName ssh443.example.com
  ProxyCommand corkscrew proxy.acme.corp 8080 %h %p
  Port 443

And with that SSH connection, I could easily tunnel TCP packets using the DynamicForward switch. That would give a SOCKS proxy and I only needed to configure my programs or use tsocks. But as I need a destination IP address in order to assemble TCP packets, I need to have DNS working, first. While a SOCKS proxy could do it, the one provided by OpenSSH cannot (correct me, if I am wrong). Obviously, I need to somehow get onto the Internet in order to resolve names, as I don’t have any local nameserver that would do that for me. So I need to tunnel. Somehow.

Most of the problem is solved by using sshuttle, which is half a VPN, half a tunnelling solution. It recognises your local machine sending packets (using iptables), does its magic to transport these to a remote host under your control (using a small python program to get the packets from iptables), and sends the packets from that remote host (using a small daemon on the server side). It also collects and forwards the answers. Your local machine doesn’t really realise that it is not really connecting itself.

As the name implies it uses SSH as a transport for the packets and it works very well, not only for TCP, but also for UDP packets you send to the nameserver of your choice. So external name resolution is done, as well as sending TCP packets to any host. You may now think that the quest is solved. But as sshuttle intercepts *all* queries to the (local) nameserver, you don’t use that (local nameserver) anymore and internal name resolution thus breaks (because the external nameserver cannot resolve printing.acme.corp). That’s almost what I wanted. Except that I also want to resolve the local domain names…

To clarify my setup, marvel at this awesome diagram of the scenario. You can see my machine being inside the corporate network with the proxy being the only way out. sshuttle intercepts every packet sent to the outside world, including DNS traffic. The local nameserver is not used as it cannot resolve external names. Local names, such as printing.acme.corp, can thus not be resolved.



  +-----------------------------------------+
  | ACME.corp                               |
  |-----------------------------------------|
  |                                         |
  |                                         |
  | +----------------+        +-----------+ |
  | |My machine      |        | DNS Server| |
  | |----------------|        +-----------+ |
  | |                |                      |
  | |sshuttle        |        +-----------+ |
  | |       corkscrew+------->| HTTP Proxy| |
  | +----------------+        +-----+-----+ |
  |                                 |       |
  +---------------------------------|-------+
                                    |
  +-----------------------------------------+
  | Internet                        |       |
  |-----------------------------------------|
  |                                 v       |
  |       +----------+        +----------+  |
  |       |DNS Server|<-------+SSH Server|  |
  |       +----------+        +----------+  |
  |                            +  +  +  +   |
  |                            |  |  |  |   |
  |                            v  v  v  v   |
  +-----------------------------------------+

To solve that problem I need to selectively ask either the internal or the external nameserver and force sshuttle to not block traffic to the internal one. Fortunately, there is a patch for sshuttle to specify the IP address of the (external) nameserver. It lets traffic designated for your local nameserver pass and only intercept packets for your external nameserver. Awesome.

But how to make the system select the nameserver to be used? Just entering two nameservers in /etc/resolv.conf doesn’t work, of course. One solution to that problem is dnsmasq, which, fortunately, NetworkManager is running anyway. A single line added to the configuration in /etc/NetworkManager/dnsmasq.d/corp-tld makes it aware of a nameserver dedicated for a domain:

server=/acme.corp/10.1.1.2

With that setup, using a public DNS server as main nameserver and make dnsmasq resolve local domain names, but make sshuttle intercept the requests to the public nameserver only, solves my problem and enables me to work again.

~/sshuttle/sshuttle --dns-hosts 8.8.8.8 -vvr myserver443 0/0 \
	--exclude 10.0.2.15/8 \
	--exclude 127.0.1.1/8 \
	--exclude 224.0.0.1/8 \
	--exclude 232.0.0.1/8 \
	--exclude 233.252.0.0/14 \
	--exclude 234.0.0.0/8 \

Applying international Bahn travel tricks to save money for tickets

Thursday, November 21st, 2013

Suppose you are sick of Tanzverbot and you want to go from Karlsruhe to Hamburg. As a proper German you’d think of the Bahn first, although Germany started to allow long distance travel by bus, which is cheap and surprisingly comfortable. My favourite bus search engine is busliniensuche.de.

Anyway, you opted for the Bahn and you search a connection, the result is a one way travel for 40 Euro. Not too bad:
bahn-ka-hh-40

But maybe we can do better. If we travel from Switzerland, we can save a whopping 0.05 Euro!
bahn-basel-hh-40
Amazing, right? Basel SBB is the first station after the German border and it allows for international fares to be applied. Interestingly, special offers exist which apparently make the same travel, and a considerable chunk on top, cheaper.

But we can do better. Instead of travelling from Switzerland to Germany, we can travel from Germany to Denmark. To determine the first station after the German border, use the Netzplan for the IC routes and then check the local map, i.e. Schleswig Holstein. You will find Padborg as the first non German station. If you travel from Karlsruhe to Padborg, you save 17.5%:
bahn-ka-padborg-33

Sometime you can save by taking a Global ticket, crossing two borders. This is, however, not the case for us:
bahn-basel-padborg-49

In case you were wondering whether it’s the very same train and route all the time: Yes it is. Feel free to look up the CNL 472.
db-cnl-472

I hope you can use these tips to book a cheaper travel.
Do you know any ways to “optimise” your Bahn ticket?

Finding Maloney

Wednesday, July 3rd, 2013

Every so often I feel the need to replace the music coming out of my speakers with an audio drama. I used to listen to Maloney which is a detective story with, well, weird plots. The station used to provide MP3 files for download but since they revamped their website that is gone as the new one only provides flash streaming.

As far as I know, there is only one proper library to access media via Adobe HDS. There are two attempts and a PHP script.

There is, however, a little trick making things easier. The website exposes a HTML5 player if it thinks you’re a moron. Fortunately, it’s easy to make other people think that. The easiest thing to do is to have an IPaid User-Agent header. The website will play the media not via Adobe HDS (and flash) but rather via a similar, probably Apple HTTP Live Streaming, method. And that uses a regular m3u playlist with loads of tiny AAC fragments :-)

The address of that playlist is easily guessable and I coded up a small utility here. It will print the ways to play the latest Maloney episode. You can then choose to either use HDS or the probably more efficient AAC version.

$ python ~/vcs/findmaloney/maloney.py 
mplayer -playlist http://srfaodorigin-vh.akamaihd.net/i/world/maloney/04df3324-4096-4dd5-b7c3-6f9b904e3f91.,q10,q20,.mp4.csmil/master.m3u8

livestreamer "hds://http://srfaodorigin-vh.akamaihd.net/z/world/maloney/04df3324-4096-4dd5-b7c3-6f9b904e3f91.,q10,q20,.mp4.csmil/manifest.f4m" best

enjoy!

Scale Text to the maximum of a page with LaTeX

Friday, April 19th, 2013

Being confronted with having to produce a simple poster that holds just a few letter but prints them as big as possible, I found myself needing to scale text (or a letter) on a page.

At first, I found \scalebox, which unfortunately takes a scaling factor, and not two dimensions. Instead of trying to do math, I found \resizebox which does take dimensions (width and height).

You could think that simply scaling up to the \textwidth is enough, but it’s not as you can see from the following “l” which was typeset using this code:

\documentclass[
landscape,
a6paper,
]{scrartcl}
\usepackage[pdftex]{graphicx}
\usepackage{palatino}
\begin{document}
\resizebox{\textwidth}{!}{l}%
\end{document}

And here’s the result:

"l" doesn't scale on A6 landscape paper

So the character doesn’t scale well in the sense that if it is too narrow, it would grow too tall. Unfortunately, it doesn’t automatically keep the aspect ratio and it doesn’t take such an argument as \includegraphic does. Fortunately, you can still make it keep the aspect ratio by globally setting the appropriate flag! So the following will work as expected:

\documentclass[landscape]{minimal}
\usepackage[showframe,a4paper]{geometry}
\usepackage{graphicx}
\setkeys{Gin}{keepaspectratio}

\newcommand{\vstretch}[1]{\vspace*{\stretch{#1}}}
\usepackage{palatino}
\begin{document}
\resizebox{\textwidth}{\textheight}{l}%
\end{document}

Another last thing is then multiline and centered output. The awesome people over at texexchange have a solution:

\documentclass[landscape]{minimal}
\usepackage[showframe,a6paper]{geometry}
\usepackage{varwidth}
\usepackage{graphicx}
\setkeys{Gin}{keepaspectratio}

\newcommand{\vstretch}[1]{\vspace*{\stretch{#1}}}
\usepackage{palatino}
\begin{document}
\topskip0pt
% This seems to fully work
\vstretch{1}
\centering\noindent\resizebox*\textwidth\textheight{\begin{varwidth}{\textwidth}%
\centering%
foooooooooooooooo

\centering
bar%
\end{varwidth}}

\vstretch{1}

\pagebreak
% Trying to other method with the table
\vstretch{1}
\centering\noindent\resizebox*\textwidth\textheight{\begin{varwidth}{\textwidth}%
\begin{tabular}{@{}c@{}}
foooooooooooooooo\\

bar
\end{tabular}%
\end{varwidth}}
\vstretch{1}

\end{document}

And the rendered result:

Converting Mailman archives (mboxes) to maildir

Tuesday, November 13th, 2012

I wanted to search discussions on mailing lists and view conversations. I didn’t want to use some webinterface because that wouldn’t allow me to search quickly and offline. So making my mail client aware of these emails seemed to be the way to go. Fortunately, the GNOME mailinglists are mbox archived. So you download the entire traffic in a standardised mbox.

But how to properly get this into your email clients then? I think Thunderbird can import mbox natively. But I wanted to access it from other clients, too, so I needed to make my server aware of these emails. Of course, I configured my mailserver to use maildir, so some conversion was needed.

I will present my experiences dealing with this problem. If you want to do similar things, or even only want to import the mbox directly, this post might be for you.

The archives

First, we need to get all the archives. As I had to deal with a couple of mailinglists and more than a couple of month, I couldn’t be arsed to click every single mbox file manually.

The following script scrapes the mailman page. It makes use of the interesting Splinter library, basically a wrapper around selenium and other browsers for Python.

#!/usr/bin/env python

import getpass
from subprocess import Popen, list2cmdline
import sys

import splinter

def fill_password(b, username=None, password=None):
    if not username:
        username = getpass.getpass('username: ')
    if not password:
        password = getpass.getpass('password: ')
        
    b.fill('username', username)
    b.fill('password', password)
    b.find_by_name('submit').click()


def main(url, username=None):
    b = splinter.Browser()
    
    try:
        #url = 'https://mail.gnome.org/mailman/private/board-list/'
        b.visit(url)
        
        if 'Password' in b.html:
            fill_password(b, username=username)


        links = [l['href'] for l in b.find_link_by_partial_text('Text')]

        cookie = b.driver.get_cookies()[0]
        cookie_name = cookie['name']
        cookie_value = cookie['value']
        cookie_str = "Cookie: {name}={value}".format(name=cookie_name, value=cookie_value)
        wget_cookie_arg = '--header={0}'.format(cookie_str)
        #print  wget_cookie_arg
        
        b.quit()

        
        for link in links:
            #print link
            cmd = ['wget', wget_cookie_arg, link]
            print list2cmdline(cmd)
            # pipe that to "parallel -j 8"

    except:
        b.quit()


if __name__ == '__main__':
    site = sys.argv[1]
    user = sys.argv[2]
    
    if site.startswith('http'):
        url=site
    else:
        url = 'https://mail.gnome.org/mailman/private/{0}'.format(site)
    
    main(username=user, url=url)

        

You can download the thing, too.

I use splinter because handling cookies is not fun as well as parsing the web page. So I just use whatever is most convenient for me, I wanted to get things done, after all. The script will print a line for each link it found, nicely prefixed with wget and its necessary arguments for the authorization cookie. You can pipe that to sh but if you want to download many month, you want to do it in parallel. And fortunately, there is an app for that!

Conversion to maildir

After having received the mboxes, it turned out to be a good idea nonetheless to convert to maildir; if only to extract properly formatted mails only and remove duplicates.

I came around mb2md-3.20.pl from 2004 quite soon, but it is broken. It cannot parse the mboxes I have properly. It will create broken mails with header lingering around as it seems to be unable to detect the beginning of new mails reliably. It took me a good while to find the problem though. So again, be advised, do not use mb2md 3.20.

As I use mutt myself I found this blog article promising. It uses mutt to create a mbox out of a maildir. I wanted it the other way round, so after a few trial and errors, I figured that the following would do what I wanted:

mutt -f mymbox -e 'set mbox_type=maildir; set confirmcreate=no; set delete=no; push "T.*;s/tmp/mymuttmaildir"'

where “mymbox” is your source file and “/tmp/mymuttmaildir” the target directory.

This is a bit lame right? We want to have parameters, because we want to do some batch processing on many archive mboxes.

The problem is, though, that the parameters are very deep inside the quotes. So just doing something like

mutt -f $source -e 'set mbox_type=maildir; set confirmcreate=no; set delete=no; push "T.*;s$target"'

wouldn’t work, because the $target would be interpreted as a raw string due to the single quotes. And I couldn’t find a way to make it work so I decided to make it work with the language that I like the most: Python. So an hour or so later I came up with the following which works (kinda):

import os
import subprocess
source = os.environ['source']
destination = os.environ['destination']

conf = 'set mbox_type=maildir; set confirmcreate=no; set delete=no; push "T.*;s{0}"'.format(destination)

cmd = ['mutt', '-f', source, '-e', conf]
subprocess.call(cmd)

But well, I shouldn’t become productive just yet by doing real work. Mutt apparently expects a terminal. It would just prompt me with “No recipients were specified.”.

So alright, this unfortunately wasn’t what I wanted. I you don’t need batch processing though, you might very well go with mutt doing your mbox to maildir conversion (or vice versa).

Damnit, another two hours or more wasted on that. I was at the point of just doing the conversion myself. Shouldn’t be too hard after all, right? While researching I found that Python’s stdlib has some email related functions *yay*. Some dude on the web wrote something close to what I needed. I beefed it up a very little bit and landed with the following:

#!/usr/bin/env python

# http://www.hackvalue.nl/en/article/109/migrating%20from%20mbox%20to%20maildir

import datetime
import email
import email.Errors
import mailbox
import os
import sys
import time


def msgfactory(fp):
    try:
        return email.message_from_file(fp)
    except email.Errors.MessageParseError:
        # Don't return None since that will
        # stop the mailbox iterator
        return ''
dirname = sys.argv[1]
inbox = sys.argv[2]
fp = open(inbox, 'rb')
mbox = mailbox.UnixMailbox(fp, msgfactory)


try:
        storedir = os.mkdir(dirname, 0750)
        os.mkdir(dirname + "/new", 0750)
        os.mkdir(dirname + "/cur", 0750)
except:
        pass

count = 0
for mail in mbox:
        count+=1
        #hammertime = time.time() # mail.get('Date', time.time())
        hammertime = datetime.datetime(*email.utils.parsedate(mail.get('Date',''))[:7]).strftime('%s')
        hostname = 'mb2mdpy'
        filename = dirname + "/cur/%s%d.%s:2,S" % (hammertime, count, hostname)
        mail_file = open(filename, 'w+')
        mail_file.write(mail.as_string())


print "Processed {0} mails".format(count)

And it seemed to work well! It recovered many more emails than the Perl script (hehe) but the generated maildir wouldn’t work with my IMAP server. I was confused. The mutt maildirs worked like charm and I couldn’t see any difference to mine.

I scped the file onto my .maildir/ on my server, which takes quite a while because scp isn’t all too quick when it comes to many small files. Anyway, it wouldn’t necessarily work for some reason which is way beyond me. Eventually I straced the IMAP server and figured that it was desperately looking for a tmp/ folder. Funnily enough, it didn’t need that for other maildirs to work. Anyway: Lesson learnt: If your dovecot doesn’t play well with your maildir and you have no clue how to make it log more verbosely, check whether you need a tmp/ folder.

But I didn’t know that so I investigated a bit more and I found another PERL script which converted the emails fine, too. For some reason it put my mails in “.new/” and not in “.cur/“, which the other tools did so far. Also, it would leave the messages as unread which I don’t like.

Fortunately, one (more or less) only needs to rename the files in a maildir to end in S for “seen”. While this sounds like a simple

for f in maildir/cur/*; do mv ${f} ${f}:2,S

it’s not so easy anymore when you have to move the directory as well. But that’s easily being worked around by shuffling the directories around.

Another, more annoying problem with that is “Argument list too long” when you are dealing with a lot of files. So a solution must involve “find” and might look something like this: find ${CUR} -type f -print0 | xargs -i -0 mv '{}' '{}':2,S

Duplicates

There was, however, a very annoying issue left: Duplicates. I haven’t investigated where the duplicates came from but it didn’t matter to me as I didn’t want duplicates even if the downloaded mbox archive contained them. And in my case, I’m quite confident that the mboxes are messed up. So I wanted to get rid of duplicates anyway and decided to use a hash function on the file content to determine whether two file are the same or not. I used sha1sum like this:

$ find maildir/.board-list/ -type f -print0 | xargs -0 sha1sum   | head
c6967e7572319f3d37fb035d5a4a16d56f680c59  maildir/.board-list/cur/1342797208.000031.mbox:2,
2ea005ec0e7676093e2f488c9f8e5388582ee7fb  maildir/.board-list/cur/1342797281.000242.mbox:2,
a4dc289a8e3ebdc6717d8b1aeb88959cb2959ece  maildir/.board-list/cur/1342797215.000265.mbox:2,
39bf0ebd3fd8f5658af2857f3c11b727e54e790a  maildir/.board-list/cur/1342797210.000296.mbox:2,
eea1965032cf95e47eba37561f66de97b9f99592  maildir/.board-list/cur/1342797281.000114.mbox:2,

and if there were two files with the same hash, I would delete one of them. Probably like so:

    #!/usr/bin/env python
    import os
    import sys


    hashes = []
    for line in sys.stdin.readlines():
        hash, fname = line.split()
        if hash in hashes:
            os.unlink(fname)
        else:
            hashes.append(hash)

But it turns out that the following snippet works, too:

find /tmp/maildir/ -type f -print0 | xargs -0 sha1sum | sort | uniq -d -w 40 | awk '{print $2}' | xargs rm

So it’ll check the files for the same contents via a sha1sum. In order to make uniq detect equal lines, we need to give it sorted input. Hence the sort. We cannot, however, check the whole lines for equality as the filename will show up in the line and it will of course be different. So we only compare the size of the hex representation of the hash, in this case 40 bytes. If we found such a duplicate hash, we cut off the hash, take the filename, which is the remainder of the line, and delete the file.

Phew. What a trip so far. Let’s put it all together:

The final thing


LIST=board-list

umask 077

DESTBASE=/tmp/perfectmdir

LISTBASE=${DESTBASE}/.${LIST}

CUR=${LISTBASE}/cur
NEW=${LISTBASE}/new
TMP=${LISTBASE}/tmp

mkdir -p ${CUR}
mkdir -p ${NEW}
mkdir -p ${TMP}

for f in  /tmp/${LIST}/*; do /tmp/perfect_maildir.pl ${LISTBASE} < ${f} ; done
mv ${CUR} ${CUR}.tmp
mv ${NEW} ${CUR}
mv ${CUR}.tmp ${NEW}
find ${CUR} -type f -print0 | xargs -i -0 mv '{}'  '{}':2,S
find ${CUR} -type f -print0 | xargs -0 sha1sum | sort | uniq -d -w 40 | awk '{print $2}' | xargs rm

And that’s handling email in 2012…

Loopback monting huge gzipped file

Wednesday, October 10th, 2012

This is basically a note to myself for future reference which I hope is interesting to others.

I just had to loopback mount a gzipped image file. I didn’t want, however, to unpack the file, because I am very short on disk space right now. Also, I didn’t care too much about processing power. I searched quite a bit until I found “avfs“.

AVFS is a system, which enables all programs to look inside archived or compressed files, or access remote files without recompiling the programs or changing the kernel.

At the moment it supports floppies, tar and gzip files, zip, bzip2, ar and rar files, ftp sessions, http, webdav, rsh/rcp, ssh/scp

muelli@xbox:/tmp$ avfsd -o allow_root ~/.avfs
muelli@xbox:/tmp$ cd  ~/.avfs/home/muelli/qemu
muelli@xbox:~/.avfs/home/muelli/qemu$ sudo losetup /dev/loop1 XP-4G.ntfs.dd.gz#
muelli@xbox:~/.avfs/home/muelli/qemu$ sudo mount /dev/loop1 -oro,noatime /home/muelli/empty/

Note that the filename I’m accessing is suffixed with a hash.

x61s and the backlight, replacing a CCFL and shorten a fuse

Friday, April 6th, 2012

Over the last months, I tried to repair my broken x61s which suffered from a missing backlight.

First, I changed the inverter, which is easy and relatively cheap to do. In case you read the Hardware Maintenance Manual, don’t follow it too closely. The inverter is easily changeable by removing the three screws on the bottom of the opened panel and carefully detaching the clipped front cover of the panel. The inverter sits on the bottom right and is the part that also lights the LEDs. You can’t miss it. But be careful: The inverter puts out high voltage, in the range of 600V to 800V, peaking at 1500V. Hence it’s hard to measure with normal home equipment :-/

Anyway, changing the inverter didn’t bring my backlight back up. So I got myself a LCD cable which is more expensive and a bit harder to attach than the inverter. You need to remove the LCD panel from its case which involves a lot of screws. Don’t miss to have separate bowls for the screws and even better: take pictures or notes to remember where the screws have been. Or be very disciplinary to follow the official instructions.

However, changing the LCD cable didn’t bring any remedy. So the only culprit, that I could think of, was the CCFL that’s actually responsible for lighting up the whole thing. So I got myself a new CCFL for a couple of euros.

Changing the CCFL is a bit messy, especially because it involves soldering. There are good instructions on the web as to how to change the CCFL. It also requires you to be very kind with the the tube so that it doesn’t break. And losing any part will probably result in a substantial loss of quality for you, so be careful. I mean it. I lost a tiny rubber ring which is to be placed around the tube to hold it tight in its channel and now the tube vibrates nicely in the panel making interesting noises.

Anyway, changing the CCFL didn’t bring back the backlight. I was very confused. There was no part I knew of that I didn’t change. With the exception of the motherboard… So I asked a friend of mine to provide his perfectly working x61s so that we’d have a reference platform that we knew was working. Thanks again to that friendly fellar that allowed the disassembly and reassembly of his machines several times :-) We switched several parts and it turned out that my panel with the new CCFL kinda works with the other inverter card. It didn’t with my inverter though. Again, very weird. As it turned out later, the CCFL contacts were not correctly isolated and short circuited :-/ But we didn’t know and thought the CCFL was broken from the very beginning. So my recommendation, which is also more (unpleasant) work, is: Checkpoint your work, i.e. run a test every now and then. It
would have saved us a lot of trouble.

So after having cross checked that my inverter was working correctly and the backlight was acting weird, I came across the fact that there might be a blown fuse. And well, the F2 fuse, which was not findable without the helping picture, was not letting anything through. Since it’s a SMD fuse there was no chance of soldering a new fuse onto the mainboard. So we just shortened the fuse with conducting silver paint.

Fortunately, the laptop’s backlight is working again, now. However, the keyboard is not. I presume that the whole dis- and reassembly shortened the lifespan of the keyboard cable. Also, as I mentioned, the CCFL is humming in the panel…

Dump Firefox passwords using Python (and libnss)

Friday, February 3rd, 2012

I was travelling and I didn’t have my Firefox instance on my laptop. I wanted, however, access some websites and the passwords were stored safely in my Firefox profile at home. Needless to say that I don’t upload my passwords to someone’s server. Back in the day, when I first encountered that problem, there were only ugly options to run the server yourself. Either some PHP garbage or worse: some Java Webapp. I only have so many gigabytes of RAM so I didn’t go that route. FWIW: Now you have a nice Python webapp and it might be interesting to set that up at some stage.

I could have copied the profile to my laptop and then ran Firefox with that profile. But as I use Zotero my profile is really big. And it sounds quite insane, because I only want to get hold of my 20 byte password, and not copy 200MB for that.

Another option might have been to run the Firefox remotely and do X-forwarding to my laptop. But that’d be crucially slow and I thought that I was suffering enough already.

So. I wanted to extract the passwords from the Firefox profile at home. It’s my data after all, right? Turns out, that Firefox (and Thunderbird for that matter) store their passwords encryptedly in a SQLite database. So the database itself is not necessarily encrypted, but the contained data. Turns out later, that you can as well encrypt the database (to then store encrypted passwords).

So a sample line in that database looks like this:

$ sqlite3 signons.sqlite 
SQLite version 3.7.11 2012-03-20 11:35:50
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .schema
CREATE TABLE moz_deleted_logins (id                  INTEGER PRIMARY KEY,guid                TEXT,timeDeleted         INTEGER);
CREATE TABLE moz_disabledHosts (id                 INTEGER PRIMARY KEY,hostname           TEXT UNIQUE ON CONFLICT REPLACE);
CREATE TABLE moz_logins (id                 INTEGER PRIMARY KEY,hostname           TEXT NOT NULL,httpRealm          TEXT,formSubmitURL      TEXT,usernameField      TEXT NOT NULL,passwordField      TEXT NOT NULL,encryptedUsername  TEXT NOT NULL,encryptedPassword  TEXT NOT NULL,guid               TEXT,encType            INTEGER, timeCreated INTEGER, timeLastUsed INTEGER, timePasswordChanged INTEGER, timesUsed INTEGER);
CREATE INDEX moz_logins_encType_index ON moz_logins(encType);
...
...
sqlite> SELECT * FROM moz_logins LIMIT 2;
1|https://nonpublic.foo.bar|Non-Pulic Wiki||||MDoEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvcNAwcECFKoZIhvcNAwcECFKoZIhvcNAwcECF|MDIEEPgAAAAAAAAAAAAAACJ75YchXUCAAAAAEwFAYIKoZIhvcNAwcE==|{4711r2d2-2342-4711-6f00b4r6g}|1|1319297071173|1348944692451|1319297071173|6
2|https://orga.bar.foo|ToplevelAuth||||MDoEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvcNAwcECIy5HFAYIKoZIhtnRFAYIKoZIh|MDoEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvFAYIKoZIhBD6PFAYIKoZIh|{45abc67852-4222-45cc-dcc1-729ccc91ceee}|1|1319297071173|1319297071173|1319297071173|1
sqlite> 

You see the columns you’d more or less expect but you cannot make sense out of the actual data.

If I read correctly, some form of 3DES is used to protect the data. But I couldn’t find out enough to decrypt it myself. So my idea then was to reuse the actual libraries that Firefox uses to read data from the database.

I first tried to find examples in the Firefox code and found pwdecrypt. And I even got it built after a couple of hours wrestling with the build system. It’s not fun. You might want to try to get hold of a binary from your distribution.

So my initial attempt was to call out to that binary and parse its output. That worked well enough, but was slow. Also not really elegant and you might not have or not be able to build the pwdecrypt program. Also, it’s a bit stupid to use something different than the actual Firefox. I mean, the code doing the necessary operations is already on your harddisk, so it seems much smarter to reuse that.

Turns out, there is ffpwdcracker to decrypt passwords using libnss. It’s not too ugly using Python’s ctypes. So that’s a way to go. And in fact, it works well enough, after cleaning up loads of things.

Example output of the session is here:


$ python firefox_passwd.py | head
FirefoxSite(id=1, hostname=u'https://nonpublic.foo.bar', httpRealm=u'Non-Pulic Wiki', formSubmitURL=None, usernameField=u'', passwordField=u'', encryptedUsername=u'MDoEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvcNAwcECFKoZIhvcNAwcECFKoZIhvcNAwcECF', encryptedPassword=u'MDIEEPgAAAAAAAAAAAAAACJ75YchXUCAAAAAEwFAYIKoZIhvcNAwcE==', guid=u'{4711r2d2-2342-4711-6f00b4r6g}', encType=1, plain_username='wikiuser', plain_password='mypass')
FirefoxSite(id=2, hostname=u'https://orga.bar.foo', httpRealm=u'ToplevelAuth', formSubmitURL=None, usernameField=u'', passwordField=u'', encryptedUsername=u'MDoEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvcNAwcECIy5HFAYIKoZIhtnRFAYIKoZIh', encryptedPassword=u'MDoEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvFAYIKoZIhBD6PFAYIKoZIh', guid=u'{45abc67852-4222-45cc-dcc1-729ccc91ceee}', encType=1, plain_username='suspect', plain_password='susicious')

The file is here: https://hg.cryptobitch.de/firefox-passwords/

It has also been extended to work with Thunderbird and, the bigger problem, with encrypted databases. I couldn’t really find out, how that works. I read some code, especially the above mentioned pwdecrypt program, but couldn’t reimplement it, because I couldn’t find the functions used in the libraries I had. At some stage, I just explained the problem to a friend of mine and while explaining and documenting, which things didn’t work, I accidentally found a solution \o/ So now you can also recover your Firefox passwords from an encrypted storage.

Pwnitter

Saturday, September 10th, 2011

Uh, I totally forgot to blog about a funny thing that happened almost a year ago which I just mentioned slightly *blush*. So you probably know this Internet thing and if you’re one of the chosen and carefully gifted ones, you confused it with the Web. And if you’re very special you do this Twitter thing and expose yourself and your communications pattern to some dodgy American company. By now, all of the following stuff isn’t of much interest anymore, so you might as well quit reading.

It all happenend while being at FOSS.in. There was a contest run by Nokia which asked us to write some cool application for the N900. So I did. I packaged loads of programs and libraries to be able to put the wireless card into monitor mode. Then I wiretapped (haha) the wireless and sniffed for Twitter traffic. Once there was a Twitter session going on, I sniffed the necessary authentication information was extracted and a message was posted on the poor user’s behalf. I coined that Pwnitter, because it would pwn you via Twitter.

That said, we had great fun at FOSS.in, where nearly everybodies Twitter sessions got hijacked ;-) Eventually, people stopped using plain HTTP and moved to end to end encrypted sessions via TLS.

Anyway, my program didn’t win anything because as it turned out, Nokia wanted to promote QML and hence we were supposed to write something that makes use of that. My program barely has a UI… It is made up of one giant button…

Despite not getting lucky with Nokia, the community apparently received the thing very well.

So there is an obvious big elephant standing in the room asking why would you want to “hack” Twitter. I’d say it’s rather easy to answer. The main point being that you should use end to end encryption when doing communication. And the punchline comes now: Don’t use a service that doesn’t offer you that by default. Technically, it wouldn’t be much of a problem to give you an encrypted link to send your messages. However, companies tend to be cheap and let you suffer with a plain text connection which can be easily tapped or worse: manipulated. Think about it. If the company is too frugal to protect your communication from pimpled 13yr olds with a wifi card, why would you want to use their services?

By now Twitter (actually since March 2011, making it more than 6 month ago AFAIK) have SSL enabled by default as far as I can tell. So let’s not slash Twitter for not offering an encrypted link for more than 5 years (since they were founded back in 2006). But there are loads of other services that suffer from the very same basic problem. Including Facebook. And it would be easy to adapt the existing solution stuff like Facebook, flickr, whatnot.

A noteable exception is Google though. As far as I can see, they offer encryption by default except for the search. If there is an unencrypted link, I invite you to grab the sources of Pwnitter and build your hack.

If you do so, let me give you an advise as I was going nuts over a weird problem with my Pwnitter application for Maemo. It’s written in Python and when building the package with setuptools the hashbang would automatically be changed to “#!/scratchbox/tools/bin/python“, instead of, say, “/usr/bin/python“.

I tried tons of things for many hours until I realised, that scratchbox redirects some binary paths.

However, that did not help me to fix the issue. As it turned out, my problem was that I didn’t depend on a python-runtime during build time. Hence the build server picked scratchbox’s python which was located in /scratchbox/bin.