NetworkManager is a daemon for configuring the network on Linux. It’s all about connection profiles, which are created by the user and “activated” by NetworkManager. A profile is just a bunch of settings and configuration values with defined keys and values, that can be sent over D-Bus or persisted to disk. Currently two file formats exists for that purpose: keyfile and ifcfg-rh. Let’s take a look at the history and future of the ifcfg-rh format.
Eliška no longer uses ifcfg files with NetworkManager.
About initscripts
The very short version of the history is that various Linux distributions use the System V “initscripts” for system configuration and management. For example, the Fedora variant is (un)maintained at github/fedora-sysv. One part of initscripts is concerned about network and provides two scripts ifup and ifdown and a SysV-style init service network. On Fedora and RHEL, users would edit files like /etc/sysconfig/network-scripts/ifcfg-eth0 according to the documentation. These files are the “profiles” in the sense that they are configurations for a network interface.
The network-scripts part of initscripts are really a bunch of shell scripts that parse ifcfg files (by executing them as shell scripts) and configure the network accordingly. They do so by calling iproute2 (the ip tool) to setup interfaces and IP addresses in kernel. But the scripts can also start ISC’s dhclient to do DHCP, in which case the DHCP daemon keeps running in the background.
As such, initscripts are a simple thing that work well for what they are. But they are one-shot commands that do their task and quit. This is a fundamental limitation, despite that they leave some state for the next time when they run — like the PID file of dhclient or an IP address configured on an interface. Another problem is that their API essentially is to have users write an ifcfg file and reboot. Yes, you can also manually call ifup and ifdown to apply changes, but that might not reliably work in subtle cases. For example if you change the DEVICE= name in the ifcfg file that is currently up, ifdown wouldn’t know that there was something to tear down. Or calling ifup twice in a row can lead to errors. The scripts simply don’t know the current state. Such an API is not well suited to build applications that configure the network, like a GUI or cockpit. Providing such an API is the main reason why NetworkManager exists.
NetworkManager handles ifcfg Files
When NetworkManager was first introduced at around 2004, initscripts and ifcfg was the way to configure the network on Fedora Core. Consequently, NetworkManager persisted its connection profiles also in ifcfg format because users were already familiar with the format and because it already existed. Also initscripts on Fedora/RHEL started to integrate with NetworkManager. When you called ifup, the script would check whether NetworkManager is running and manages the profile, in which case the script will call nmcli connection up to let NetworkManager handle it. This can be prevented by configuring a key NM_CONTROLLED=no in the ifcfg file.
While a goal was that the same ifcfg file can be used both by initscripts and NetworkManager in a similar way, in practice this only works with simple configurations. NetworkManager defines a large number of additional shell variables that are not supported by initscripts. And likewise, in several ways initscripts will behave differently than NetworkManager. For example, NetworkManager does not support IP aliases and the way Openvswitch and Wi-Fi configuration is expressed in ifcfg files is fundamentally incompatible. Also, not every NetworkManager profile can be persisted in ifcfg format, in which case NetworkManager always uses the keyfile format.
In Fedora 29 the network-scripts package was split out of the initscripts package. On recent Fedora and RHEL, the network-scripts package is not installed by default. They are considered deprecated but still available for installation. Note that the initscripts package itself does things beyond network configuration and that part is still in use.
Moving away from ifcfg files
As far as NetworkManager is concerned, it keeps handling ifcfg files. Regardless whether network-scripts are in use, the file format is used by NetworkManager, along side the keyfile format which has ini style files in /{etc,run,usr/lib}/NetworkManager/system-connections. Note that the keyfile backend is the preferred format of NetworkManager and the one that can handle all connection types. It is thus always enabled in NetworkManager. You can only select whether to use the ifcfg-rh backend in addition. This is done via the main.plugins setting in NetworkManager.conf. Before Fedora 34, this setting had a default of main.plugins=ifcfg-rh,keyfile. Since Fedora 34 it defaults to main.plugins=keyfile,ifcfg-rh. The difference is that now NetworkManager preferably writes new profiles in keyfile format. This change is a step away from using the ifcfg-rh format.
It will probably take a while until NetworkManager completely gets rid of supporting ifcfg-rh format. The difficulty is that this concerns user configuration on disk, and it is important to NetworkManager to not break during package upgrade or during upgrade to a newer Fedora/RHEL release. To do this, we will also need to write migration tools that can convert ifcfg files to keyfile format. And in the end, if a user or tool relied on finding an ifcfg file, the change will be noted and might break existing setups. There is no hurry for you to migrate, but be aware that eventually we want to get rid of the ifcfg file format. The reason is that maintaining multiple formats has an overhead, not only for the development but for users who need to learn two formats and for tools that not work if an unexpected format is used. It’s also desirable that NetworkManager on Fedora/RHEL uses the same format as it does on most other Linux distributions.
Example of the ifcfg Format
If we create a new connection profile with
nmcli connection add type ethernet con-name eth0 \
ifname eth0 autoconnect no
Subjectively, I find the ini style format nicer. It’s also based on glib’s GKeyfile API that can parse these files into some form of dictionary. Also, libnm provides an API to convert NMConnection instaces to/from keyfile.
NetworkManager is a software to provide an API for configuring the network on Linux. It aims to make configuration simple and reliable.
A new NetworkManager version 1.28.0 was released today, on 6th December 2020.
Alice wants to try the latest NetworkManager release.
1.28 was already in release candidate phase for several weeks. That is longer than usual or planned, but there were some issues related to DNS which delayed the release. In the future, we should do a better job at sticking to more regular release schedules.
See the NEWS file for what is new and find the release tarball at the GNOME download page.
This release introduces no big features, but instead there are a lot of small fixes and improvements. I only would like to highlight two points that are more important to developers and maintainers:
Fixes for building NetworkManager with LTO. Building with Link Time Optimization tends to reveal issues, especially because we want NetworkManager to build without emitting compiler warnings. Now that Fedora 33 enables LTO by default, we fixed some remaining issues and have tests that continuously check that the build works.
We now reformatted our entire C source code with clang-format. The advantage is that our coding style is now governed by the code formatter and there is no more need to argue about trivialities. In particular we now use spaces for indentation, contrary to before, when tabs were used for indentation and spaces for horizontal alignment. Such so called “smart tabs” were cumbersome for contributors to get right, also because editors don’t support that style well. The downside is that this introduces a huge source code diff between 1.26 and 1.28 and makes backporting code to 1.26 harder.
NetworkManager is a software to provide an API for configuring the network on Linux. It aims to make configuration simple and reliable.
A new NetworkManager version 1.26.0 was released today.
Speedy has already upgraded to the latest version.
See the NEWS file for what is new and find the release tarball at the GNOME download page.
New manual pages nm-settings-nmcli and nm-settings-dbus. NetworkManager is all about the API that it provides. And this API is mostly about connection profiles. Depending on whether you use nmcli, D-Bus, keyfile or ifcfg-rh files, the properties of a profile are handled slightly different. For example, in nmcli all properties are strings, but on D-Bus they are structured data. So we also need different documentation. For that, we have now the manual pages nm-settings-dbus, nm-settings-nmcli, nm-settings-keyfile and nm-settings-ifcfg-rh.
Match profiles by device driver, PCI address and kernel command line. Connection profiles can be restricted to a certain device (interface). Commonly the interface name or the permanent MAC address can be specified. Since 1.26.0 there are two new matches: by device driver and by the device path. The device path essentially corresponds to the PCI address of the interface. Also, you can match a profile based on the kernel command line. That means, a profile will only be usable, if the kernel command line is as specified. For that, there are 3 new properties. In nmcli, they are called match.driver, match.path, and match.kernel-command-line. See the nm-settings-nmcli manual for details.
nm-cloud-setup now supports Google Compute Engine (GCE). nm-cloud-setup is a tool that automatically configures the network when running in a cloud environment. It already supported Amazon EC2 cloud and got support for Google Compute Engine (GCE).
Support for ethtool settings for coalesce and ring options. NetworkManager now supports ethtool coalesce and ring options in addition to offload features. For that, configure the ethtool properties like ethtool.coalesce-tx-frames.
Also, NetworkManager now restores the previous ethtool auto-negotiations settings when deactivating a profile. That is important, because with the wrong ethernet speed settings the link will have no carrier.
nmcli now colors profiles for externally configured devices differently. When a device is configured outside of NetworkManager, then NetworkManager creates an in-memory profile to represent that the device is connected. But it does not touch the device in any way, so these devices and profiles are special. This is often a cause for confusion. Now nmcli color codes these settings differently.
NetworkManager is a tool to configure the network on a Linux host. I am a NetworkManager developer and work at Red Hat. In my opinion the biggest value it provides is the API. That is what makes NetworkManager unique among other network configuration projects. Here is why.
Fleur and Eliška use NetworkManager on all their in-house production servers.
Configuring the Network
To configure the local network on the host, you need a stateful process.
Configuring networking means to setup the Linux kernel and user space so that applications can use the network. This involves setting up network interfaces, addresses and routes, DNS and the system’s hostname, and more. For most scenarios a stateful service is necessary to monitor and re-configure the system. This is for example the case for DHCP (ISC’s dhclient), Wi-Fi (wpa_supplicant), IPv6 SLAAC (kernel itself or a user space daemon), bluetooth (bluez), ppp (pppd), and more. For example, if you write a shell script to configure a networking interface with DHCP, it needs to spawn a manager deamon like ISC’s dhclient or dhcpcd.
Today we have various software that aims to manage networking exhaustively. We have NetworkManager, which is a freedesktop.org project. There is also ConnMan, netctl (on ArchLinux), systemd-networkd, wicd (unmaintained), wicked (on SUSE). Any of these will work well to setup networking on your machine.
Providing an API for other Tools
NetworkManager is more than just a tool to configure the network. Its main purpose is to provide an API for other applications.
Did you wonder why GNOME or KDE only has UI integration with NetworkManager? NetworkManager was created from the start as a configuration daemon that serves the needs of applications, independent from the UI. It is ahead of any other solution providing such an API, and that is why desktop environments integrate with it. It fits the requirements like no other. Most alternatives mentioned above don’t even have such an API as their explicit goal. Their focus is to configure the network. Of course, every program that is usable will have some form of API and a manner how to use it. But such APIs are often targeted directly towards a human, towards a user who edits a file, who invokes an ad-hoc command and who is around to react to with failures.
NetworkManager’s API aims to be used by other programs, the clients of NetworkManager. Some of these clients (GUIs) may directly target a human end user too. Or they provide integration with OpenStack or OpenShift, where the human administrator is much more detached from the host. When no human is around, it becomes critical that the current state of the system can be determined programmatically. The API is not only for configuration actions, but also to determine the current state of the system.
Generalist and Featureful
NetworkManager is a generalist and featureful and complex.
When your application relies on NetworkManager, then NetworkManager must be at least as portable as your application. That means, NetworkManager must be suitable for a wide range of environments and not only for the desktop. NetworkManager works well on the phone, the server, in a container, on a notebook and on a workstation. Likewise, NetworkManager must support a whole lot of use cases, technologies and features. The API must be powerful and ubiquitous. If there are shortcomings, then these problems are supposed to be fixable and should be fixed. Running NetworkManager on your phone or in your container is not out of scope, it is something that is supposed to work well and what is an explicit goal.
NetworkManager may not be the optimal solution in every scenario, yet. For example, on a BGP router with thousands of routes, there are still performance issues. Or running inside a container doesn’t work well with macvlan devices. Or NetworkManager running a DHCP server is simple but not very flexible. But most scenarios that you imagine are scenarios where NetworkManager wants to excel. The Linux kernel itself is the prime example of the benefits of being a generalist, and NetworkManager is also one.
This means NetworkManager is not a simple piece of software. But its complexity is there to provide features for other applications, so they themselves can be simpler. Solving hard problems once.
Integration of Tools
Using NetworkManager API allows the integration of tools.
Whether you use nmcli or nmtui on the command line, Cockpit, the GUIs, or Ansible, they all are just front ends for the same underlying configuration. And of course, you can write your own applications that uses NetworkManager’s API, and your application will naturally integrate with these other tools.
Wherever you have NetworkManager, it works and behaves the same. On RHEL and Fedora, you configure a server and the workstation using the same tools. Most Linux distributions provide NetworkManager. If you know how to configure NetworkManager on Ubuntu, you know how to configure networking on Fedora, too.
What’s Next?
NetworkManger also has issues and lacking features. Some people dislike it for being too large or complex, think it is not stable, or not suitable for anything except the desktop. NetworkManager is software, and like every software could be replaced by another piece of software. Should we instead replace it with some hypothetical, future alternative? I believe that it will be significantly simpler to incrementally improve NetworkManager, than to throw it away and start over.
In my opinion, the majority of shortcomings we face in free software comes down to lack of contributors and people. Under such circumstances, the solution for a problem is not to provide more alternatives, but to focus on making one solution work well.
One current area of focus is to integrate with Openstack, Openshift, CoreOS and dracut. I am also excited about the prospect of running NetworkManager on phones, like with UBPorts or Librem5.
NetworkManager is an active free software project. Let’s work together and run NetworkManager everywhere.
NetworkManager 1.16 got native support for WireGuard VPN tunnels (NEWS). WireGuard is a novel VPN tunnel protocol and implementation that spawned a lot of interest. Here I will not explain how WireGuard itself works. You can find very good documentation and introduction at wireguard.com.
Having support in NetworkManager is great for two main reasons:
NetworkManager provides a de facto standard API for configuring networking on the host. This allows different tools to integrate and interoperate — from cli, tui, GUI, to cockpit. All these different components may now make use of the API also for configuring WireGuard. One advantage for the end user is that a GUI for WireGuard is now within reach.
By configuring WireGuard with NetworkManager you get other features beyond the plain WireGuard tunnel setup. Most notably you get DNS and firewalld setup in a consistent manner.
For Alice it is now easy to configure WireGuard with NetworkManager.
NetworkManager’s support for WireGuard requires the kernel module for Linux. As of March 2019, it is not yet upstream in mainline kernel but easy to install on most distributions.
Import an existing WireGuard profile
The WireGuard project provides a wg-quick tool to setup WireGuard tunnels. If you are using WireGuard already, chances are that you use this tool. In that case you would have a configuration file and issue wg-quick up. Here is the example configuration file from wg-quick’s manual page:
Note that wg-quick up wg0.conf does something fundamentally different from what nmcli connection import does. When you run wg-quick up, it reads the file, configures the WireGuard tunnel, sets up addresses and routes, and exits.
This is not what “connection import” does. NetworkManager is profile based. That means you create profiles instead of issuing ad-hoc commands that configure ephemeral settings (like ip address add, wg set, or wg-quick up). NetworkManager calls these profiles “connections”. Configuring something in NetworkManager usually boils down to create a suitable profile and “activate” it for the settings to take effect.
nmcli connection import is just one way to create a profile. Note that the imported profile is configured to autoconnect, so quite possibly the profile gets activated right away. But regardless of that, think of “import” creating just a profile. You would only do this step once, but afterwards activate the profile many times.
There is no difference to NetworkManager how the profile was created. You could also create a WireGuard profile from scratch.
and finally let’s activate it. Note you will be asked to enter the private key that you may generate with wg genkey:
$ nmcli --show-secrets --ask connection up my-wg0
Secrets are required to connect WireGuard VPN 'my-wg0'
WireGuard private-key (wireguard.private-key): eD8wqjLABmg6ClC+6egB/dnMLbbUYSMMrDsrHUwmQlI=
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/30)
Confirm that the VPN tunnel is now up:
$ nmcli
[...]
wg0: connected to my-wg0
"wg0"
wireguard, sw, mtu 1420
inet6 fe80::720b:6576:1650:d26/64
route6 ff00::/8
route6 fe80::/64
$ ip link show wg0
34: wg0: mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/none
$ sudo WG_HIDE_KEYS=never wg
interface: wg0
public key: SymChsQwTX5yZrtwtsWpYfHLMgnJpOJ25YOfs7/ImT0=
private key: eD8wqjLABmg6ClC+6egB/dnMLbbUYSMMrDsrHUwmQlI=
listening port: 56389
Note that above wireguard.private-key-flags are set to 0. The secret flags determine whether the secret is not-required, to be stored to disk or a keyring, or always asked. In this case, the private key got stored to disk in /etc/NetworkManager/system-connections/.
Note that above output also shows the current device information with upper-cased properties. This is because the profile is currently activated. As you modify the profile, you’ll note that the changes don’t take effect immediately. For that you have to (re-) activate the profile with
$ nmcli connection up my-wg0
Note that this time we don’t need to provide the private key. The key was stored to disk according to the secret flags. This will allow the profile to automatically connect in the future upon boot.
Configuring Peers
As of now, nmcli does not yet support configuring peers. This is a missing feature. Until this is implemented you have the following possibilities, which are all a bit inconvenient.
1.) Import Peers from a wg-quick configuration file
See above. This does not allow you to modify an existing profile, as nmcli connection import always creates a new profile.
2.) Use the Python Example Script nm-wg-set
There is a python example script. It uses pygobject with libnm and accepts similar parameters as wg set. I mention this example script to give you an idea how you could use NetworkManager from python (in this case based on libnm and pygobject).
libnm is the client library for NetworkManager. It gained API for fully configuring WireGuard profiles. This is what the nm-wg-set example script above uses.
4.) Use D-Bus directly
NetworkManager’s D-Bus API is what all clients use — from libnm, nmcli to GUIs. NetworkManager is really all about the (D-Bus) API that it provides. Everything that a tool does with NetworkManager will always be possible by using D-Bus directly. When NetworkManager 1.16 introduces WireGuard support, then the tools are still lacking, but the API is ready for implementing them.
5.) Edit the Profile on Disk
NetworkManager persists WireGuard profiles in the keyfile format. These are files under /etc/NetworkManager/system-connections and it is always fully supported that you just edit these files by hand. This is the other, file-base API of NetworkManager beside D-Bus. This leaves you with the problem to know what to edit there exactly. Let’s look at what we got so far:
The WireGuard peer settings should be pretty straight forward. See also NetworkManager’s keyfile documentation. Edit the file and issue sudo nmcli connection reload or sudo nmcli connection load /etc/NetworkManager/system-connection/my-wg0.nmconnection. This causes NetworkManager to update the profile with the changes from disk.
Finally, reactivate the profile and check the result:
$ nmcli connection up my-wg0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/31)
$ sudo WG_HIDE_KEYS=never wg
interface: wg0
public key: SymChsQwTX5yZrtwtsWpYfHLMgnJpOJ25YOfs7/ImT0=
private key: eD8wqjLABmg6ClC+6egB/dnMLbbUYSMMrDsrHUwmQlI=
listening port: 50000
fwmark: 0x500
peer: llG3xkDWcEP4KODf45zjntuvUX0oXieRyxXdl5POYX4=
allowed ips: 192.168.7.0/24
persistent keepalive: every 2 minutes
peer: 2Gl0SATbfrrzxfrSkhNoRR9Jg56y533y07KtIVngAk0=
preshared key: qoNbN/6ABe4wWyz4jh+uwX7vqRpNeGEtgAnUbwNjEug=
allowed ips: (none)
Reapply and Runtime Configuration
We said that after modifying a profile we have to fully reactivate the profile for the changes to take effect. That’s not the only way. NetworkManager supports nmcli device reapply wg0 which makes changes to the profile effective without doing a full re-activation cycle. That is less disruptive as the interface does not go down. Likewise, nmcli device modify wg0 allows you to change only the runtime configuration, without modifying the profile. It is fully supported to modify WireGuard settings of an active tunnel via reapply.
Dynamically Resolving Endpoints
In WireGuard, peers may have an endpoint configured but also roaming is built-in. NetworkManager supports peer endpoints specified as DNS names: it will resolve the names before configuring the IP address in kernel. NetworkManager resolves endpoint names every 30 minutes or whenever the DNS configuration of the host changes, in order to pick up changes to the endpoint’s IP address.
MTU
In the NetworkManager profile you can configure wireguard.mtu for the MTU. In absence of an explicit configuration, the default is used. That is different from wg-quick up, which tries to autodetect the MTU by looking at how to reach all peers. NetworkManager does not do such automatism.
Peer Routes, AllowedIPs and Cryptokey Routing
In WireGuard you need to configure the “AllowedIPs” ranges for the peers. This is what WireGuard calls Cryptokey Routing. It also implies, that you usually configure direct routes for these “AllowedIPs” ranges via the WireGuard tunnel. NetworkManager will add those routes automatically if wireguard.peer-routes option of the profile is enabled (which it is by default).
Routing All Your Traffic
When routing all traffic via the WireGuard tunnel, then peer endpoints must be still reached outside the tunnel.
For other VPN plugins NetworkManager adds a direct route to the external VPN gateway on the device that has the default route. That works well in most cases, but is an ugly hack because NetworkManager doesn’t reliably know the correct direct route in unusual scenarios.
NetworkManager currently does not provide any additional automatism to help you with that. As workaround you could manually add an explicit route to the profile of the device via which the endpoint is reachable:
An alternative solution is to configure policy routing. The wg-quick tool does this with the Table=auto setting (which is the default).
NetworkManager supports configuring routes in other routing tables than the “main” table. Hence, using policy-routing works in parts by configuring "ipv4.route-table" and "ipv6.route-table". The problem is that currently NetworkManager does not support configuring the routing policy rules themselves. For now, the rules must be configured outside of NetworkManager. You could do so via a dispatcher script in /etc/NetworkManager/dispatcher.d, but yes, this is lacking. See the NetworkManager manual about dispatcher scripts.
update 2019/08/02 NetworkManager supports since 1.18.0 configuring policy routing rules in the profile. However, there are still two caveats to manually configure what wg-quick does with TABLE=auto (and what WireGuard calls “Improved Rule-based Routing“). First, it requires the suppress_prefixlength rule attribute. That attribute is only supported since NetworkManager 1.20.0. The second problem is that it requires to put the default-route in a dedicated table. While you can configure the routing table for manual routes in NetworkManager, you currently cannot configure a default route (with prefix lenth 0) like a manual route. That needs fixing. On the upside, 1.20.0 brings also new options wireguard.ip4-auto-default-route and wireguard.ip6-auto-default-route. These options are enabled by default and NetworkManager will now automatically configure policy routing like wg-quick with TABLE=auto. This automatism offers a nice solution for the problem.
Key, Peer, and IP Address Management
The beauty of WireGuard is its simplicity. But it also leaves all questions about key distribution, peer management and IP address assignment to the upper layers. For the moment NetworkManager does not provide additional logic on top of WireGuard and exposes just the plain settings. This leaves the user (or external tools) to manually distribute private keys and configure peers, IP addresses and routing. I expect that as WireGuard matures there will be schemes for simplifying this and NetworkManager may implement such protocols or functionality. But NetworkManager won’t come up with a homegrown, non-standard way of doing this.
WireGuard is Layer3 only. That means you cannot run DHCP on a WireGuard link and ipv4.method=auto is not a valid configuration. Instead, you have to configure static addresses or IPv6 link local addresses.
Namespaces
WireGuard, like most tunnel based solutions, have neat applications regarding networking namespaces. This is not implemented in NetworkManager yet, but we would be interested to do so. Note that this isn’t specific to WireGuard tunnels and namespace isolation would be a useful feature in general.
What’s next?
Add support for policy-routing rules (rhbz#1652653).
Automatically help avoiding routing loops when routing all traffic.
Add nmcli support for configuring WireGuard peers.
Add WireGuard support to other NetworkManager clients, like nm-connection-editor.
See where management tools for WireGuard go and what NetworkManager can do to simplify management of keys, peers and addressing.
Provide an API in NetworkManager to isolate networks via networking namespaces. This is not specific to WireGuard but will be useful in that context.
