Uncategorised

urlparse considered harmful

Over the weekend, I spent a number of hours tracking down a bug caused by the cache in the Python urlparse module. The problem has already been reported as Python bug 1313119, but has not been fixed yet. First a bit of background. The urlparse module does what you'd expect and parses a URL into its components: >>> from urlparse import urlparse >>> urlparse('http://www.gnome.org/') ('http', 'www.gnome.org', '/', '', '', '') As well as accepting byte strings (which you'd be using at the HTTP protocol level), it also accepts Unicode strings (which you'd be using at the HTML or XML content level): >>> urlparse(u'http://www.ubuntu.com/') (u'http', u'www.ubuntu.com', u'/', '', '', '') As the result is immutable, urlparse implements a cache of up to 20 previous results. Unfortunately, the cache does not distinguish between byte strings and Unicode strings, so parsing a byte string may return unicode components if the result is in the cache: >>> urlparse('http://www.ubuntu.com/') (u'http', u'www.ubuntu.com', u'/', '', '', '') When you combine this with Python's automatic promotion of byte strings to unicode when concatenating with a unicode string, can really screw things up when you do want to work with byte strings. If you hit such a problem, the code may all look correct but the problem was introduced 20 urlparse calls ago. Even if your own code never passes in Unicode strings, one of the libraries you use might be doing so. The problem affects more than just the urlparse function. The urljoin function from the same module is also affected since it uses urlparse internally: >>> from urlparse import urljoin >>> urljoin('http://www.ubuntu.com/', '/news') u'http://www.ubuntu.com/news' It seems safest to avoid the module all together if possible, or at least until the underlying bug is fixed.

OpenID 2.0 Specification Approved

It looks like the OpenID Authentication 2.0 specification has finally been released, along with OpenID Attribute Exchange 1.0. While there are some questionable features in the new specification (namely XRIs), it seems like a worthwhile improvement over the previous specification. It will be interesting to see how quickly the new specification gains adoption. While this is certainly an important milestone, there are still areas for improvement. Best Practices For Managing Trust Relationships With OPs The proposed Provider Authentication Policy Extension allows a Relying Party to specify what level of checking it wants the OpenID Provider to perform on the user (e.g. phishing resistant, multi factor, etc). The OP can then tell the RP what level of checking was actually performed. What the specification doesn't cover is why the RP should believe the OP. I can easily set up an OP that performs no checking on the user but claims that it performed "Physical Multi-Factor Authentication" in its responses. Any RP that acted on that assertion would be buggy. This isn't to say that the extension is useless. If the entity running the RP also runs the OP, then they might have good reason to believe the responses and act on them. Similarly, they might decide that JanRain are quite trustworthy so believe responses from myOpenID. What is common in between these situations is that there is a trust relationship between the OP and RP that is outside of the protocol. As the specification gives no guidance on how to set up these relationships, they are likely to be ad-hoc and result in some OpenIDs being more useful than others. At a minimum, it'd be good to see some best practices document on how to handle this. Trusted Attribute Exchange As mentioned in my previous article on OpenID Attribute Exchange, I mentioned that attribute values provided by the OP should be treated as being self asserted. So if the RP receives an email address or Jabber ID via attribute exchange, there is no guarantee that the user actually owns them. This is a problem if the RP wants to start emailing or instant messaging the user (e.g. OpenID enabled mailing list management software). Assuming the RP doesn't want to get users to revalidate their email address, what can it do? One of the simplest solutions is to use a trust relationship with the OP. If the RP knows that the OP will only transfer email addresses if the user has previously verified them, then they need not perform a second verification. This leaves us in the same situation as described in the previous situation. Another solution that has been proposed by Sxip is to make the attribute values self-asserting. This entails making the attribute value contain both the desired information plus a digital signature. Using the email example, if the email address has a valid digital signature and the RP trusts the signer to perform email address verification, then it can accept the email address without further verification. This means that…

States in Version Control Systems

Elijah has been writing an interesting series of articles comparing different version control systems. While the previous articles have been very informative, I think the latest one was a bit muddled. What follows is an expanded version of my comment on that article. Elijah starts by making an analogy between text editors and version control systems, which I think is quite a useful analogy. When working with a text editor, there is a base version of the file on disk, and the version you are currently working on which will become the next saved version. This does map quite well to the concepts of most VCS's. You have a working copy that starts out identical to a base tree from the branch you are editing. You make local changes and eventually commit, creating a new base tree for future edits. In addition to these two "states", Elijah goes on to list three more states that are actually orthogonal to the original two. These additional states refer to certain categorisations of files within the working copy, rather than particular versions of files or trees. Rather than simplifying things, I believe that mingling the two concepts together is more likely to cause confusion. I think this is evident from the fact that the additional states do not fit the analogy we started with. Versioned and Unversioned Files If you are going to use a version control system seriously, it is worth understanding how files within a working copy are managed. Rather than thinking of a flat list of possible states, I think it is helpful to think of a hierarchy of categories. The most basic categorisation is whether a file is versioned or not. Versioned files are those whose state will be saved when committing a new version of the tree. Conversely, unversioned files exist in the working copy but are not recorded when committing new versions of the tree. This concept does not map very well to the original text editor analogy. If text editors did support such a feature, it would be the ability to add paragraphs to the document that do not get stored to disk when you save, but would persist inside the editor. Types of Versioned Files There are various ways to categorise versioned files, but here are some fairly generic ones that fit most VCS's. unchanged modified added removed Each of these categorisations is relative to the base tree for the working copy. The modified category contains both files whose contents have changed and whose metadata has changed (e.g. files that have been renamed). The removed category is interesting because files in this category don't actually exist in the working copy. That said the VCS knows that such files did exist, so it knows to delete the files when committing the next version of the tree. Types of Unversioned Files There are two primary categories for unversioned files: ignored unknown The ignored category consists of unversioned files that the VCS knows the user does not want…

Inkscape Migrated to Launchpad

Yesterday I performed the migration of Inkscape's bugs from SourceForge.net to Launchpad. This was a full import of all their historic bug data – about 6900 bugs. As the import only had access to the SF user names for bug reporters, commenters and assignees, it was not possible to link them up to existing Launchpad users in most cases. This means that duplicate person objects have been created with email addresses like $USERNAME@users.sourceforge.net. If you are a Launchpad user and have previously filed or commented on Inkscape bugs, you can clean up the duplicate person object by going to the following URL and entering your $USERNAME@users.sourceforge.net address: https://launchpad.net/people/+requestmerge After following the instructions in the email confirmation, all references to the duplicate person will be fixed up to point at your primary account (so bug mail will go to your preferred email address rather than being redirected through SourceForge).

OpenID Attribute Exchange

In my previous article on OpenID 2.0, I mentioned the new Attribute Exchange extension. To me this is one of the more interesting benefits of moving to OpenID 2.0, so it deserves a more in depth look. As mentioned previously, the extension is a way of transferring information about the user between the OpenID provider and relying party. Why use Attribute Exchange instead of FOAF or Microformats? Before deciding to use OpenID for information exchange, it is worth looking at whether it is necessary at all. There are existing solutions for transferring user data such as FOAF and the hCard microformat. As the relying party already has the user's identity URL, it'd be trivial to discover a FOAF file or hCard content there. That said, there are some disadvantages to this method: Any information published in this way is available to everyone. This might be fine for some classes of information (your name, a picture, your favourite colour), but not for others (your email address, phone number or similar). The same information is provided to all parties. Perhaps you want to provide different email addresses to work related sites. The RP needs to make an additional request for the data. If we can provide the information as part of the OpenID authentication request, it will reduce the number of round trips that need to be made. In turn, this should reduce the amount of time it takes to log the user in. Why use Attribute Exchange instead of the Simple Registration extension? There already exists an OpenID extension for transferring user details to the RP, in the form of the Simple Registration extension. It has already been used in the field, and works with OpenID 1.1 too. One big downside of SREG is that it only supports a limited number of attributes. If you need to transfer more attributes, you basically have two choices: use some other extension to transfer the remaining attributes make up some new attribute names to send with SREG and hope for the best. The main problem with (2) is that there is no way to tell between your own extensions to SREG and someone else's which will likely create interoperability problems if when an attribute name conflict occurs. So this solution is not a good idea outside of closed systems. This leaves (1), for which Attribute Exchange is a decent choice. What can I do with Attribute Exchange? There are two primary operations that can be performed with the extension: fetch some attribute values store some attribute values Both operations are performed as part of an OpenID authentication request. Among other things, this allows: The OP to ask the user which requested attributes to send If the OP has not stored values for the requested attributes, it could get the user to enter them in and store them for next time. The OP could use a predefined policy to decide what to send the RP. One possibility would be to generate one-time email addresses specific…