Dan Williams’ blog

(Note: the final 0.9.10 will be out later this week…  read on for the awesome that it will contain)

Hey wouldn’t it be great if NetworkManager did X and made my life so awesome I could retire to a private island surrounded by things I love?  Like kittens and teddy bears and bright copper kettles and Domaine Leflaive Montrachet Grand Cru?

If only your dream was reality…  Oh wait!  NetworkManager 0.9.10 will be your genie from Aladdin, granting every wish you dream of, except this time you can wish for more wishes.  But you still can’t bring awful networking back from the dead because it’s just not pretty.  Don’t do it.

 

What is pretty is NetworkManager 0.9.10; it’s like the lightning-quick racing yacht that Larry Ellison doesn’t have and really, really wants, but which somehow also adds a Triple-E-Class-worth of new features just for you.  Somebody (maybe you!) wished for every single thing you’re about to see.  And then a magic genie showed up, snapped its fingers, and gave it to them.

nmtui

We found a usability gap between full-fledged CLI tools like nmcli and GUI-based ones, and thus nmtui was born.  Sometimes you don’t want to remember esoteric commands and options, but you also don’t want to run X. Boom, first wish granted: a curses-based tool for configuring and managing your network, no X involved:

nmcli

The command-line still rules with divine mandate, and we’re here to please so nmcli was a huge focus for this release.  We’ve added interactive editing support, single-command editing, detailed help, tab completion, and enhanced bash completion.  You really need to check this out; almost anything you can do with GUI tools can now be done with nmcli, and there’s even some stuff nmcli can do that the GUI tools can’t.  If you’re comfortable with terminals, NetworkManager 0.9.10 is right up your alley.

Size Does Matter

Continuing on the quest to be more nimble and streamlined, we’ve split Wi-Fi, WWAN, Bluetooth, ADSL, and WiMAX device support into plugins which you don’t need to install if you like a minimal system.  Distributions should package these separately so they can be added/removed independently of NetworkManager itself, which reduces disk usage, runtime memory usage, and packaging dependency chains.  We’ve also spent time slimming down and optimizing the code.  The core NetworkManager daemon is now just over 1MB in size!

dbus-daemon is also no longer required for root-only or early-boot operation, with communication using a private root-only Unix socket. Similarly, PolicyKit is no longer used for root operation, though it could always be disabled at build-time anyway.

To facilitate remote and SSH-based management, the “at_console” D-Bus permission has been removed, which also helpfully harmonizes authorization settings between Fedora and Debian-based distributions.  All permissions authorization now happens through PolicyKit instead.

The Enterprise

When you Absolutely Positively MUST have your ethernet frames delivered on-time and without loss you turn to Data Center Bridging.  DCB provides the reliability and robustness that iSCSI and FibreChannel over Ethernet (FCoE) need so you don’t have to keep shovelling money into a proprietary SAN.  Since users requested it, we snapped our fingers and added support to NetworkManager 0.9.10 for configuring DCB on your ethernet interfaces.

We’ve also upped our game with IP-level configuration support for many more software interfaces like GRE, macvlan, macvtap, tun, tap, veth, and vxlan.  And when you have services that aren’t yet network-aware, the NetworkManager-wait-online systemd service is more reliable to ensure your legacy services start up with the resources they require.

Customization Galore

You dreamed, we listened.  Creepy, no?  Yeah, we know what you want.  And top of the list was more flexible configuration:

• Connection configuration files are no longer watched for changes by default, which used to cause problems with backups, filesystem copies, half-configured connections, etc.  If you want that behavior you can turn it back on (monitor-connection-files=true), but instead, edit them as much as you want and when you’re done, “nmcli con reload“.
• Connections can now be locked to interface names instead of just MAC addresses
• A new “ignore-carrier” option is available to ensure your critical app doesn’t fail just because you got drunk on Captain Morgan + Coke, and tripped over a cable
• Want to manage /etc/resolv.conf yourself?  You can!  “dns=none” is your new best friend.
• Configuration file snippets can be dropped into /etc/NetworkManager/conf.d to change smaller sets of configuration options

The NetworkManager dispatcher got some enhancements too.  It now has a “pre-up” event that allow scripts to execute before NetworkManager announces connectivity to applications.  We also added a “pre-down” event that lets network filesystems flush data before the interface is actually disconnected from the network.

Seamless Cooperation

Do you love /sbin/ip?  ifconfig?  brctl?  vconfig?  Keep using them!  Changes you make outside of NetworkManager get picked up, respected, and reflected in the D-Bus API.  NetworkManager 0.9.10 also goes to great lengths to read the existing configuration of interfaces and not touch them.  Most network interfaces known to the kernel are now exposed in the D-Bus API, and you can even change their IP configuration right from NetworkManager.  There’s more work to do here but we hope you’ll appreciate the new situational awareness as much as we do.

Get Your VPN On

We’ve improved support for routing-only VPNs like Openswan/Libreswan/Strongswan.  We’ve added full details of the VPN’s IP configuration to the D-Bus API.  And best yet, VPN plugins can now request additional passwords during the connection process if the ones you previously gave them are wrong or changed.

All the Rest

For clients, more properties are exposed in the D-Bus API.  We’ve added support for custom IP ranges to the Internet Connection Sharing functionality.  We’ve added WWAN autoconnect support and more reliable airplane mode behavior.  Fatal connection errors now more reliably block reconnect, which means better handling of wrong Wi-Fi passwords and access point failures.  Captive portal/hotspot support is moving forward, as are DNSSEC enhancements.

Geez, are we done yet?

Not even close!  Seriously, there’s more but I’m kinda tired of typing.  Try it out (the final release will be out later this week) and tell us what you think.  Then tell us what you want.  Don’t be afraid to dream a little bigger, darling!

Recently we posted a Fedora 21 update delivering the huge new awesome that is NetworkManager 0.9.10-beta1.  Among a literal Triple E Class boatload of enhancements and fixes, this update continues our fine tradition of making the core of NetworkManager smaller and more flexible by splitting out Wi-Fi support into the NetworkManager-wifi package.  If you don’t have or don’t use Wi-Fi on your system, you don’t need to install stuff for it and you can save some disk space and RAM.

The Problem

To ensure upgrades work correctly and people didn’t unexpectedly lose Wi-Fi support we set RPM Obsoletes to ensure that when upgrading the new package was installed even though it didn’t exist before.  Unfortunately this didn’t work for those using DNF instead of yum for package management.

It turns out that DNF treats RPM obsoletes differently than yum, and it’s unclear right now how package splitting is actually supposed to work with DNF.  You can track the issue here.

The Workaround

If you suddenly find yourself without Wi-Fi, find a wired network connection and:

dnf install NetworkManager-wifi
systemctl restart NetworkManager

and harmony will return to the Universe.  We understand the pain, will continue to monitor the situation, and will update the Fedora NetworkManager packages when DNF has a solution.

We’ve recently binged on making NetworkManager work better in more places, mostly enterprise and virtualization related.  But one thing we’ve wanted to do for a long time was make things more modular.  And that got landed this week via the dev-plugins branch, which makes ATM, Bluetooth, and WWAN into shared libraries loaded optionally at startup.

Distro packagers can now create separate NetworkManager-atm, NetworkManager-bluetooth, and NetworkManager-wwan packages, each with their own dependencies, while the NetworkManager core goes forward a slimmer, smaller, more efficient version of its previous self.  If you’re installing NetworkManager into minimal environments, you can just ignore or remove these plugins and revel in your newfound minimalism.

The core NM binary is now about 15% smaller, and there’s a corresponding 7.5% RSS reduction at runtime when no plugins are loaded.  What’s next?  Possibly WiFi, which would save about 6 – 8% of the core binary size.

Courtesy of Zeeshan I came into an N950 last year, so that I could make the N9/N950 work great with ModemManager.  Mission accomplished.  And one of the most annoying things about it is that when the battery is charged, it stops charging even though it’s still plugged in.  So, of course, I wake up in the morning and since it’s stopped charging while plugged in, the battery is now down to 75% or even lower.

So I humbly ask the Internet, is there a way to tell the N9/950 not to drain the battery when it’s done charging and still plugged in?

I’m trying to reverse engineer the EVDO Pilot Sets V2 QCDM log item so we can get EVDO signal strength while connected with ModemManager.  I’ve got most of it figured out, but half the battle of reverse engineering is getting enough variation in the data to see the patterns.  That’s where you come in.  I’m specifically interested in getting results from EVDO on 850MHz (Americas), 450MHz (Europe and Russia), and 1700MHz/AWS (North America), so if you’re in the US and you have Leap, Cricket, MetroPCS, US Cellular, C-Spire, nTelos, then you get bonus points.  But even if you have Verizon or Sprint, the data is still useful.

You obviously have to have an EVDO-capable WWAN data card or phone that uses Qualcomm chipsets and exposes a DIAG port, but luckily almost all devices that speak CDMA/EVDO are Qualcomm-based.  You don’t even need to have an active subscription, as the tool is read-only.

So if you’re game, grab this tarball and run ‘make’ in the extracted directory.  That’ll give you an ‘evdolog’ binary which when run dumps out the data I’m interested in.  When you’ve got the dump, mail it to me.  There’s a README in the tarball that has more detail on everything.

The tool doesn’t dump any personal data, just information about what radio channels your modem is listening on and information about what radio channels the modem might jump to if you move around.  The output looks like this:

SS: State: 1
SS: Band Class: 1
S: Channel: 75

L: PN:      3
L: AS ct:   1
L: AS win:  60
L: AS chan: 2123 (0x084B)
L: UNK 1:   63 (0x3F)
L: CA ct:   0
L: CA win:  0
L: RE ct:   14
L: RE win:  100
L: UNK 2:   0 (0x00)
L: Act 0:   PN: 30   6f 01 11 00 03 00 ba 3b
L: Rem 0:   PN: 33   00 00 4b 08 64 00 00 00
L: Rem 1:   PN: 36   00 00 4b 08 64 00 00 00
L: Rem 2:   PN: 402  34 00 4b 08 64 00 00 00
L: Rem 3:   PN: 279  06 00 4b 08 64 00 00 00
L: Rem 4:   PN: 27   00 00 4b 08 64 00 00 00
L: Rem 5:   PN: 21   00 00 4b 08 64 00 00 00
L: Rem 6:   PN: 126  00 00 4b 08 64 00 00 00
L: Rem 7:   PN: 399  00 00 4b 08 64 00 00 00
L: Rem 8:   PN: 318  00 00 4b 08 64 00 00 00
L: Rem 9:   PN: 24   00 00 4b 08 64 00 00 00
L: Rem 10:  PN: 300  00 00 4b 08 64 00 00 00
L: Rem 11:  PN: 504  00 00 4b 08 64 00 00 00
L: Rem 12:  PN: 507  00 00 4b 08 64 00 00 00
L: Rem 13:  PN: 510  00 00 4b 08 64 00 00 00

Which, if you know CDMA, you know there is no channel 2123 in band class 0 or 1, which is one reason I’m asking for data dumps here 🙂

Thanks!

And why are we sipping sophisticated cocktails with an air of restrained joy, you might ask?  The answer is that we don’t need a reason.  But if we had one (and I’m still not saying we do), it would involve a couple of great releases we’ve made over the past few weeks.  Thanks to everyone how contributed!

NetworkManager 0.9.6

Lots of great stuff in this update to the NetworkManager stable series.  Tons of fixes to the glib bindings, enhanced IPv6 functionality, more capable nmcli tool, support for ADSL modems, on-demand WiFi scan support, new Vala bindings, documentation updates, and a lot more.  This release is recommended for anyone using older 0.9.x versions.

Where do we go from here?  Bridging support is scheduled to land in the next release, as are some cleanups of the D-Bus and glib APIs that make it easier for app developers.  We’ll probably also have real hotspot support (AP-mode, yay!), bring back WPA Ad-Hoc mode for kernel drivers that actually support it, toss in an enhanced connection editor with support for bridging, bonding, VLAN, WiMAX, ADSL, etc, and yet more fixes all around.  We’re hoping to release this next version a bit later this fall.

ModemManager 0.5.4 and 0.6

Finally a 0.6 release, which adds support for Cinterion devices and Iridium satphone modems.  Plus an enhanced API for things like PIN/PUK retry counts, facility locks, network time and date indications, and better support for devices that want PPP on specific ports.  It also includes all the enhancements that are also present in 0.5.4 such as more compatible SMS handling, support for more ZTE, Ericsson, and Sierra devices, and random bug fixes.  They’re also the most tested ModemManager releases ever!

Next up for ModemManager is beating git master into shape, which will debut as 0.8 sometime later this year.  It properly supports multi-mode devices that can talk CDMA/EVDO and GSM/UMTS and LTE all at the same time, including all the new Qualcomm devices that use the QMI protocol.  It also drastically reworks the D-Bus API to ensure things stay working with future hardware capabilities.  Yes, a huge D-Bus API break, but it’s properly namespaced so that supporting both the old and new APIs can be done in the same binary without compile-time switches.

So cheers!  And play the *Manager drinking game: whenever it works, take a drink!  You’ll be wasted in no time…

There’s a major ModemManager release coming up soon, and that means testing.  What better to do on a Friday than test a @#%!-load of modems and phones?  But hey, I know what’s better…  drowning your incredible frustration with firmware and hardware in a kiddy-pool full of cheap booze.  Isn’t hardware great?

Further analysis clarifies some of the actual bugs in dhclient/dhcp code.  The ‘infinite lease’ problem only happens on 64-bit hosts due to the size of time_t.  Plus the problem isn’t just an ‘infinite’ lease issue, it’s a ‘infinite lease minus seconds since the epoch’ problem due to how the dhclient code internally handles timeout calculation.  Remember, an ‘infinite’ lease is 0xFFFFFFFF, or UINT_MAX.

On IA-32, where time_t is 32-bit:

• after the lease is ACK-ed, some code adds the lease seconds to gettimeofday() seconds to determine the lease expiry time, which of course is now wrapped
• this same code then checks if the lease expiry is less than gettimeofday() seconds (which of course it is, since it just wrapped around to gettimeofday() – 1) and helpfully resets the lease expiry to INT_MAX
• checks in isc_time_nowplusinterval() which guard against 32-bit overflow (by checking expiry seconds plus gettimeofday() against UINT_MAX) succeed and everything works

On x64, where time_t is 64-bit:

• adding gettimeofday() seconds to the lease seconds does not wrap, thus the lease expiry is larger than UINT_MAX
• since the value did not wrap, the checks for negative lease expiry do not trigger, and the expiry is still UINT_MAX + gettimeofday()
• the overflow checks in isc_time_nowplusinterval() trigger because our values are already larger than UINT_MAX, and the lease fails

The core problem is that most of the dhcp/dhclient code simply doesn’t deal with large time values that may occur on 64-bit platforms.  A few suggestions for dhclient:

1. Change the TIME define to ‘u_int64_t’ instead of ‘time_t’; that makes the relevant client_lease structure fields large enough to survive a wrap
2. Change the fields of struct isc_time_t to ‘u_int64_t’ instead of ‘unsigned int’; maybe even change ‘struct isc_interval_t’ to use 64-bit values just to ensure wrapping doesn’t happen there too
3. Stop using ‘struct timeval’; instead use the fixed ‘struct isc_time_t’
4. Wrap gettimeofday() in a utility function that returns a ‘struct isc_time_t’ instead of ‘struct timeval’
5. Then you don’t need the 32-bit checks in isc_time_nowplusinterval() or at least you could change them to be 64-bit safe instead.

For Fedora, fixed test packages are available here.  Not sure about other distros.

The DHCP server at GUADEC apparently gives out infinite leases, which have a value of 0xFFFFFFFF (in seconds).  Leaving aside the question of whether infinite leases are actually a good idea at a conference where many people come and go (hint: they usually aren’t),  why won’t it just @#%@#%!&&* connect?

By default NetworkManager uses dhclient for DHCP, and that usually works fairly well.  It’s a well-understood program developed by the ISC that’s used by millions every day.  But dhclient apparently fails with infinite leases.  Here’s the dhclient code:

    #define DHCP_SEC_MAX  0xFFFFFFFF

/*
* The ISC timer library doesn’t seem to like negative values
* and can’t accept any values above 4G-1 seconds so we limit
* the values to 0 <= value < 4G-1.  We do it before
* checking the trace option so that both the trace code and
* the working code use the same values.
*/

sec  = when->tv_sec – cur_tv.tv_sec;
usec = when->tv_usec – cur_tv.tv_usec;

<…>
} else if (sec > DHCP_SEC_MAX) {
log_error(“Timeout requested too large ”
“reducing to 2^^32-1”);
sec = DHCP_SEC_MAX;
<…>
}

isc_interval_set(&interval, sec & DHCP_SEC_MAX, usec * 1000);
status = isc_time_nowplusinterval(&expires, &interval);
if (status != ISC_R_SUCCESS) {
log_fatal(“Unable to set up timer: %s”,
isc_result_totext(status));
}

The code attempts to add a timeout that triggers when the lease expires; “when” is the lease interval coming from the DHCP server (which is 0xFFFFFFFF, remember).  “cur_tv” is just gettimeofday().  Let’s enumerate the fail purely for pedagogic purposes:

• Despite the comment and logged error, the code makes no attempt to limit the value to UINT_MAX – 1.  Even if it did, this wouldn’t help; see below.  So now we’re passing UINT_MAX into isc_time_nowplusinterval() as interval->seconds.  Condensed code for that function:

isc_result_t isc_time_nowplusinterval(isc_time_t *t, const isc_interval_t *i) {
struct timeval tv;

if (gettimeofday(&tv, NULL) == -1)
return (ISC_R_UNEXPECTED);

/* Ensure the resulting seconds value fits in the size of an
* unsigned int.  (It is written this way as a slight optimization;
* note that even if both values == INT_MAX, then when added
* and getting another 1 added below the result is UINT_MAX.)
*/
if ((tv.tv_sec > INT_MAX || i->seconds > INT_MAX) &&
((long long)tv.tv_sec + i->seconds > UINT_MAX))
return (ISC_R_RANGE);

<…>

• tv.tv_sec could be greater than INT_MAX since time_t is often 8 bytes wide.  So this code is much more likely to fail in early 2038.
• Oops!  i->seconds is UINT_MAX, and clearly that’s larger than INT_MAX.  So onward to the next bit.
• Oops I did it again!  I played with your timer!  The second check passes (unless you have a time machine and you like 1970) because tv.tv_sec is the current time (clearly a large value) and i->seconds is already UINT_MAX.
• An error is returned, and your lease fails.

Looks like this code just doesn’t expect to deal with large values of Unix time; hopefully that’ll get corrected soon.  The Red Hat bug report for this is bug 662254.

What can I do?

NetworkManager supports two different DHCP clients, selectable at runtime.  Install dhcpcd, set dhcp=dhcpcd in the [main] section of /etc/NetworkManager/NetworkManager.conf, restart NM, and voila, DHCP.