Dan Williams’ blog

Thomas Johnson (High School Janitor)

“Oh yeah, I’ve seen that code.  It’s worse than what I clean up in the bathrooms after Prom or Homecoming.  The kids get high and drunk and party too hard and puke all over the place.  I deal with enough vomit from 7:30 to 6; I wouldn’t touch the staging drivers with a mop twice as long as the one I have at work.”

Just Say No

Thomas just found out that none of the “staging” wifi drivers will work with hidden access points because they don’t set the IW_SCAN_CAPA_ESSID capability bit.  Furthermore, the most popular “staging” drivers (for the Ralink hardware used in many netbooks) don’t even have specific SSID scanning capability at all.

Why do you care?  Hidden APs don’t broadcast their network ID, which misinformed people think is more secure (hint: it’s not).  Before a driver can associate to the network, it needs to discover available APs and capabilities, which requires a probe-request, which exposes the network ID to everyone anyway.  But that requires driver support which none of the staging drivers have.

I fixed this issue upstream two years ago by adding IW_SCAN_CAPA_ESSID to Wireless Extensions.  Of course the staging WiFi drivers that many distros enable never got fixed because the vendor it came from didn’t bother to work with the community in the first place.  And people wonder why they don’t work.

Broadly speaking, staging WiFi drivers come in two flavors: (a) old dried gum from under the cafeteria table (drivers with a future), and (b) fresh vomit from the hung-over kid in your math class (those without a future).

The drivers with a future (winbond, rtl81xx) are or will based on the kernel-standard mac80211 wireless stack, which implements the 802.11 WiFi specification in the kernel.  Since they use the standard mac80211 stack, they get all these nice features like probe-scanning and the correct capability bits for free.  All you have to do is work on supporting the hardware itself.

The drivers without a future (rt2860, rt2870, rt3070, rt3090, wlan-ng, vt665x) are based on forks of the ancient ieee80211 stack that Intel’s ipw2×00 drivers forked from the hostap driver.  Each of these drivers includes their own copy of the core ieee80211 stack forked at different times and with different hacks.  When a bug shows up, that means 4x the work, and 4x the chance for the fix to slip through the cracks.  Which is why these drivers have no future.  They are a maintenance nightmare.  Besides, they have crap like this:

pAdapter->StaCfg.bScanReqIsFromWebUI = TRUE;

It just blows my mind why people think staging wifi drivers are a great idea.  There’s a reason staging drivers set the TAINT_CRAP flag in your kernel; because that’s what they literally are.

So what’s the right thing to do?

There’s one huge reason why dead-end staging drivers are a bad idea: there aren’t enough developers.  So do you spend that effort  on maintaining unmaintainable shit code?  Or do you spend it on fixing the code that has a future?  Most of the time you can’t do both.

If you choose to maintain the staging drivers, then things become worse over time since the staging code is simply less tested and less maintainable.   So you continue to drop hacks and fixes onto an ever-growing steaming pile of manure.  Nobody cares much about the driver (because it doesn’t use the standard kernel interfaces and thus doesn’t have a future), so your staging driver never benefits from all the great feature work and bug fixing that the mac80211 and wireless developers are doing.

But if you choose to help fix the upstream drivers that do use mac80211 (like rt2×00), and thus have a future, maybe for a few months some users won’t have great wireless.  But they didn’t before either.  But then 6 months later, all the users get great wireless with features like power saving, background scanning, WiFi Direct, Bluetooth 3, access-point mode, etc.  Those things will never be done to the staging drivers, because those drivers are a dead-end maintenance nightmare, because their code is awful, and because they don’t use the standard kernel wireless stack.

I know I’d invest the effort where it helps users the most, even if it means a few more months of subpar driver support while the official upstream drivers get fixed and the staging drivers go untouched.  That’s how things actually get better when you can’t fix everything at once.

Lately ofono and ConnMan have been in the news, and that’s sparked some discussion about how these two projects relate to NetworkManager.  I’ve mostly just been ignoring that discussion and focusing on making NetworkManager better.  But at some point the discussion needs to become informed and the facts need to be straightened out.

So what makes NetworkManager great?

• Flexibility: NM’s D-Bus interface provides a ton of control and information about the network connections of your machine.  Developers and applications simply don’t take enough advantage of this.  Imagine mail automatically pulled whenever the corporate VPN is up.  Or more restrictive firewalls when connected to public networks.  Yeah, you can do that today with NetworkManager.
• Works everywhere: from the mainframe to the power desktop to the netbook and lower.  There’s nothing stopping you from running NetworkManager on an s390 or a Palm Pre.
• Integration: most users like NetworkManager’s distro integration, so it’s on by default (but can be turned off for running bare-metal).  NM will read your distro’s network config files: ifcfg on Fedora, /etc/network/interfaces on Debian, etc.  It doesn’t pretend the rest of the world doesn’t exist, but it can if you tell it to.
• Connection Sharing: you can share your 3G connection to the wired or the wifi interface, or the other way around.  How you share is completely up to you.
• VPN: it’s got plugins for Cisco (vpnc), openvpn, openconnect, and pptp.  An ipsec/openswan plugin is being written.  It’s just easy to use the VPN of your choice.
• Makes Linux better: by not working around stupid vendor drivers or other broken components, NetworkManager drives many improvements in drivers, kernel APIs, the supplicant, and desktop applications.    Five years ago I posted a list of wifi problems, many of which got fixed because NetworkManager users complained about them.  Stuff like WPA capability fixes, hidden SSID fixes, suspend/resume improvements, Ad-Hoc mode fixes, and lots of improvements to wpa_supplicant to name just a few.  By encouraging drivers to be open, by fixing bugs in the open drivers and the stack instead of hacking around them, and by encouraging vendors to work upstream, NetworkManager makes Linux better for you.

What great stuff is coming next?

• Bluetooth
  
• IPv6
• Bridging
• Getting rid of HAL
• ModemManager and better mobile broadband

All in all, a lot of great stuff is on the plate.  NetworkManager already works well for a ton of people, but we’d like to make it work better for a lot more people.  And it will.

So what about ConnMan?

I recently came across a slide deck about ConnMan which makes both disappointing and inaccurate claims about NetworkManager.  It’s also worth emphasizing the philosophical differences between the two projects.

First, ConnMan primarily targets embedded devices, netbooks, and MIDs (slide #1).  When ConnMan was first released in early 2008, NetworkManager 0.7 was under heavy development, and NetworkManager 0.6 clearly did not meet the requirements.  But 0.7, released in November 2008, works well for a wide range of use-cases and hardware platforms.

NetworkManager scales from netbooks, MIDs, and embedded devices with custom-written UIs to desktops to large systems like IBM’s s390.  You get the best of both worlds: from phenomenal cosmic power down to itty-bitty living space.

ConnMan explicitly doesn’t try to integrate with existing distributions (slide #5), partly due to it’s requirements to be as light-weight as possible.  But NetworkManager will use your distro’s normal network config and startup scripts if you tell it to do (but you don’t have to).  Early in NetworkManager days we tried to ignore the rest of the world too.  Turns out that doesn’t work so well; users demand integration with their distribution.  But ConnMan doesn’t pretend to be general purpose, and due to its embedded focus, it can wave this issue away.

Both ConnMan and ofono reject well-established technologies like GObject (but still uses glib) in favor of re-implementing much of GObject internally anyway. This is a curious decision as GObject is not a memory hog and not a performance drag for these cases.  The NIH syndrome continues with libgppp, libgdhcp, and libgdbus, where instead of improving existing, widely-used tools like dhclient/dhcpcd, pppd, and dbus-glib, ConnMan opts to re-implement them in the name of being more “lightweight”.  With embedded projects that ConnMan targets (like Maemo and Moblin) already using GObject and dhcpcd, I don’t understand why this tradeoff was made.  Perhaps this visceral dislike of GObject and dbus-glib was one reason the project’s creators decided to write their own connection manager instead of helping to improve existing ones.

NetworkManager in contrast re-uses and helps improve components all over the Linux stack.  Because of that, more people benefit from the fixes and improvements that NetworkManager drives in projects like avahi, wpa_supplicant, the kernel, pppd, glib, dbus-glib, ModemManager, libnl, PolicyKit, udev, etc.

Taking a look at the deck

I have things to say about most of the slides, but I’ll concentrate on the most interesting and misinformed ones instead.

Supposed Words About NetworkManager

• Very Complex Design: a complete strawman, because it doesn’t say anything.  NetworkManager 0.7 is a mature project with many useful features.  NM is based around a core of objects, each one performing actions based on signals and events from other objects.  It’s modular and flexible.  It’s just not a ConnMan-style box of lego blocks with a rigid plugin API and all the problems that causes.
• Large Dependency List: NM requires things like wpa_supplicant, udev, dbus, glib, libuuid, libnl, and a crypto library.  pppd and avahi are optional. This list is certainly not large.  When you take ConnMan and its optional dependencies (most of which are needed in a useful system) the list is just about the same.
• Too Much Decision-making in the UI: Completely bogus and frankly incomprehensible.  The core NM daemon provides a default policy which is in no way connected to the UI, and the rest is up to the user.  nm-applet contains no policy whatsoever.  If the objection is to nm-applet’s desktop-centric interaction model, then it’s important to know there is no lack of applets for different use-cases.
• Tries to work around distro problems: this is completely a matter of perspective.  Since Intel was creating its own Linux distribution (moblin), they didn’t have to work around any existing issues; these were simply waved away.  Unfortunately NetworkManager lives in the world of reality and not some universe full of ponies.  For users that expect it, NetworkManager integrates with your distros existing network config, init scripts, and DNS resolution.  For users that don’t care, NetworkManager can run bare-metal.
• Too much GNOME-like source code: seriously, what the hell?  I’m not sure where to begin with this one.  The NetworkManager core does not depend on GNOME.  At all. Yeah, the source-code is in the Gnome style, but is that seriously an issue?

Uninformed diagram of NetworkManager architecture

(Misinformation shaded blue for your protection)

The User Settings service is contained in the applet, and it’s completely optional.  The System Settings service has been merged back into the NetworkManager core daemon and is no longer a separate process.  That same commit ported NM from HAL to udev; thus HAL is no longer required.  NetworkManager always used HAL/udev for device detection instead of RTNL (ie, netlink).  NetworkManager also hasn’t used WEXT for a long time; wpa_supplicant handles kernel wireless configuration.  NetworkManager uses distro networking scripts only for service control, as does ConnMan.  The rest of the slide is quite petty and just splits hairs.

Where to?

It’s unlikely that either NetworkManager or ConnMan will disappear in the near future.  That means we’ll all have to live with two mutually exclusive connection managers and two completely different network configuration systems.  I think that’s pretty pointless, but I don’t get the last word anyway, since that’s not how Open Source works.  The users will decide which solution works best for them.  And that means NetworkManager will keep getting better, keep getting more useful, and will continue to be the easiest network management solution around.

A few of the 3G cards NetworkManager gets tested with

… to shoot myself in the head.  Some mobile broadband cards are like a nice, quiet child that does everything you tell them to do; they’d even clean out your family’s slurry tank if you asked.  Unfortunately, most of the cards you just want to throw right into the slurry tank strapped to the side of a large brick.  Hopefully the tank is full, and the card doesn’t have a snorkel, not that a snorkel would help it much.

Yes, there are standards.  But as we all know, given 10 people and a standard, you’ll end the day with 12 or 13 differently behaving “standards-compliant” implementations.  People suck. You’d think it would be easy to agree on an AT command for “prefer 3G / prefer 2G / 3G only / 2G only”.  NO SIMPLE FOR YOU.  But NetworkManager has to work around huge amounts of stupid.  Here’s a run-down of some of the mobile broadband hardware that’s available today and what about it sucks.

HUAWEI (PHEAR THE DRAGON)

Europe apparently got carpet-bombed with these things.  They provide two usable serial ports; one for data and another for stuff like signal strength, mode switches, etc.  But asking it anything on the second port makes the modem cry, grab its toys, and run home to tell mommy what you’ve done.  This caused problems with the new modem capability probing code in NetworkManager 0.7.1.  Thanks guys (not). Dropping unhandled input on the floor would apparently have been too easy.  And, of course, they use AT^SYSCFG with some magic numbers to indicate 2G/3G preference.  That said, Huawei does participate upstream and proactively adds IDs and support for their hardware.

Qualcomm Gobi (NEW HOTNESS ALERT)

Apparently now all the rage State-side (though they’ve been out for a while); even in the ultra-small Atom-based Poulsbong-smoking Sony Vaio P series.  These parts can do GSM/HSPA and CDMA/EVDO, depending on the firmware they load.  Now that’s pretty cool.  There’s even a driver for them (qcserial) queued up in gregkh’s tree.  Unfortunately, because Qualcomm still can’t get their head out of their ass you won’t get signal strength, cell tower reports, or mode change signals, since the driver only exposes one serial port which is used for PPP (it might support GSM multiplexing, in which case this rant is wrong).  Everything else seems to get done from userspace with libusb and a proprietary protocol.  WTF is so awesome about proprietary protocols?  You get to sell people an SDK for $20,000 or something?  Nice try.

Modern Sierra (MAGICALLY DELICIOUS)

Driven by the ’sierra’ driver (surprise!), these cards expose multiple serial ports; two or more of which accept AT commands.  Only one of these ports has a full AT interpreter and gets used for PPP, the other ports get used for signal strength and GPS during the PPP session.  I hear new Sierra gear is switching to the tty+netdevice model though, so these will be old-but-not-busted soon.  But of course, somebody took a huge pull off the bong, and came up with AT!SELRAT for 2G/3G preference.  Yay!  Variation #2!

Old-School Sierra (OLD BUT NOT BUSTED)

Sweet bliss.  Works like a champ.  16-bit PCMCIA.  HSDPA even.  GSM multiplexing support makes up for the fact that it only exposes one serial port to the OS (even though we don’t support userspace muxing yet).  It’s been supported in NetworkManager since, like, day #1.  Like the newer Sierra cards, it also uses AT!SELRAT, so at least Sierra is consistent.  Which is more than I can say for some other hardware I’ve seen.

Option “HSO” (THE NEED FOR SPEED)

PPP sucks; it’s only between you and the card, not over the air.  So why bother?  Which is why Option killed PPP dead.  These devices expose multiple AT-capable ttys, and an ethernet network interface.  Do the setup on the AT ports, do the data in high speed on the network interface.  This is the current trend.  Sierra is going to do it soon.  So is Huawei.  But unfortunately, everyone does the authentication and the IP configuration differently.  And Option’s 2G/3G preference command is AT_OPSYS.  Variant #3!  Go to hell.  In any case, big thanks to Option for providing me with hardware and also working with the upstream kernel community; you guys rock.

Ericsson F3507g (SWEDISH INVASION)

Dude, you got a Dell Mini?  If you’ve coughed up for the 5530 Mobile Broadband option, it’s probably got one of these inside.  The Sony Ericsson MD300 is the same hardware.  For once, somebody uses standard interfaces too; these parts expose multiple cdc-acm serial ports (like most mobile phones), and one cdc-ether network device used for data.  The interesting thing is that to get an IP address, you use DHCP on the ethernet interface.  We don’t yet know how to set 2G/3G preference, but you can get it with AT*ERINFO.  All hail variant #4.  This is getting rediculous.  At least Ericsson pays people to make their stuff work with Linux, though the AT reference document is NDA-encumbered.  Need to hit somebody with the cluebat for that.

BUSlink SCWi275u (DEAR GOD DON’T BUY THIS)

Really.  If you find one, put it out of its misery by burning it alive.  Yeah, it’s really old, and it’s only GPRS, and it’s from the land before time when they put WiFi into cellular modems because nobody had it onboard.  And hey, its firmware is as clueless about standards as Qualcomm is about Open Source.  But it works fine with NetworkManager

As you can see, nobody in this industry talks to each other, and none of the carriers care about making it easier to write software for the devices they sell.  Everywhere you look there are silos, walled gardens, and revenue stream protection.  But that’s where NetworkManager comes in.

The Bright, Shiny New Mobile Future

NM 0.7 delivered the promise of Mobile Broadband.  We took a limited set of devices (ie, no phones) and made those work out of the box.  Now it’s time to get bigger, faster, and stronger.  We can’t support everything in the current architecture inside NetworkManager, so Tambet started a new project called ModemManager.  All mobile broadband handling gets punted out to ModemManager (similar to how WiFi is handled with wpa_supplicant), making the NetworkManager core simpler, easier to maintain, and more robust.  ModemManager provides a nice D-Bus API for everything modem-related; data connections, SMS, phonebook, signal strength, GPS, etc.  It rocks.  It’s more flexible.  It spews out cute, cuddly kittens by the thousands.  It’s definitely the right architecture and the way forward.

The Slightly Less-Bright Now

But until ModemManager drops some awesome on y’all, we need to better support modems in NetworkManager 0.7.x too.  A few problems we’ve been tackling over the past few months:

• multiple serial ports – most modems provide more than one port; but nothing tells you what that port gets used for.  Sometimes asking the port what’s purpose in life is doesn’t work either.  So we have to special-case some modems in the udev prober, and some in NetworkManager.  This gets as ugly as your first girlfriend/boyfriend.
• modem capabilities -  this is why your mobile phone didn’t work with NetworkManager 0.7.0.   We need to know whether the modem is CDMA/EVDO or GSM/HSPA since the operation and UI needs to change based on which kind of modem it is.  Previously we used hal-info’s 10-modem.fdi, which simply doesn’t scale.  Asking the modem freaks some of them out (ie, Huawei) and others just lie for various reasons.  So with NM 0.7.1, we probe serial ports with a udev helper and are careful not to touch things that shouldn’t be touched.
• modem init strings – because, of course, consistent handling of initialization strings between devices would just be too easy.  Some devices puke up half-eaten puppies when given the same init string that every other device on the planet supports.  No standardization here.  So NetworkManager 0.7.1 tries different init strings until one works.
• registration commands -  some Huawei modems want to use AT+CGREG instead of AT+CREG.  Yeah, I know why it seems to think it can be special, but it’s not.  It’s just plain stupid.  And this seems to change based on firmware version of all things.  Dear God, why do you toy with me?  So in lieu of finding a Huawei engineer and asking them what the fuck they were thinking, we hacked around it for now.

We’ve gotten most of worked out in the NetworkManager 0.7.1 release candidate series.  And all this crap is exactly why NM 0.7.1 isn’t out yet.  Like when NM 0.7.1-rc1 broke people’s Huawei cards due to modem probing freaking out the firmware, I spent $100 for a Huawei E160G off eBay.  It took a week to get here, and two days to fix it.

But that’s why NetworkManager rocks; we pony up the cash to make sure our shit works.  Users appreciate that.