The big names mostly can't yet, but some lesser-known Linux distributions offer the ability to undo updates and recover from damage, even automatically.

With a new version of Linux Mint available, a lot of people - especially less-technical types - are about to upgrade their PCs. As with almost all major upgrades, inevitably some some stuff won't work with the new version. Going back is difficult, although Mint does have a solution, and so do some smaller distros.

Linux Mint includes the Timeshift utility, originally developed by Teejeetech, but that doesn't make it invulnerable. No operating system is, as amply demonstrated recently by CrowdStrike. If installed on a file system that supports copy-on-write (COW) snapshots, Timeshift can use that, but if you chose some other disk format, it can also back up your system files using rsync to another drive.

For now, most Linux file systems can't handle snapshotting, including the default ext4 or its predecessors, or XFS, or the now-vanishing ReiserFS. Several existing file systems that work on Linux do include snapshot support, including bcachefs, ZFS, and Btrfs. However, there are solid reasons why most distros don't include the functionality.

For instance, while OpenZFS does support snapshots, and several distributions include ZFS, including TrueNAS Scale, Ubuntu, Proxmox, Void Linux, and NixOS, the ZFS code can't be compiled into the Linux kernel due to an incompatible software license. Thus, from appropriate levels of caution, OpenZFS isn't part of either SUSE's or Red Hat's enterprise offerings. Their compliance with multiple legal requirements and certifications is a big part of their corporate appeal. (For the curious, both companies maintain lists of their certifications. Red Hat has a 16-section list and SUSE a respectable dozen sections. Some companies will pay good money for this stuff.)

Red Hat removed Btrfs from RHEL six years ago, and is working on its own, all-GPL combined volume management system and file system called Stratis. Since Btrfs was invented there, Oracle had a simpler route available for its RHEL-compatible distro: Oracle Linux offers a kernel with Btrfs.

Bcachefs, meanwhile, is relatively new and immature. It only appeared at the beginning of this year in kernel 6.7, meaning that it's only been in three releases of the kernel so far. It's a little bit too new for anyone else to support it just yet.

So, for now, Btrfs is the most mature file system in the Linux kernel that can do snapshots. This is why SUSE's Snapper uses Btrfs by default, although it also supports LVM, and can be configured to use that instead.

Whatever distro family you favor, there's a flavor that includes Snapper and the ability to roll back from a failed update to an older, working state of the OS. In the RPM Package Manager family, obviously, that means both the SUSE Enterprise and openSUSE distros. After all, Snapper is a SUSE tool, it's installed by default. In the Arch family, Snapper is in Garuda Linux; for Debian Stable, there's Spiral Linux, and for the bleeding-edge unstable Debian Sid, in siduction.

However, Snapper does have a weakness. It's usually deployed on Btrfs, which can be fragile. The way that Snapper works is by making a COW snapshot of the OS before installing updates, and integrating those snapshots into the bootloader, so that if something goes wrong, you can choose a pre-installation snapshot and boot into that instead. The ALP enterprise distro that the company is actively working on can even automate this recovery process. The weakness that this vulture previously encountered with this is that it's very easy to fill up the root partition with snapshots, because Btrfs doesn't give accurate estimates of free space - and attempting to write to a full Btrfs volume almost always corrupts it. Worse still, there are no safe, working Btrfs repair tools.

For this reason if nothing else, The Reg FOSS desk really hopes to see an implementation of snapshots and rollback that also works on OpenZFS and bcachefs some day soon. Ubuntu could use ZFS for this, but Canonical hasn't done much work on its ZSys code in some years. We suspect it's focusing on Ubuntu Core and more recently Core Desktop instead. Ubuntu Core is entirely built using the Snap packaging format, and as that makes it easy to revert to older versions of packages, it doesn't need a fancy file system for this.

This may be why ChromeOS - and the free ChromeOS Flex for PCs and Intel Macs - chose a simpler system that's more amenable to automation. ChromeOS has two bootable root partitions. Both are normally read-only when the OS is running, and when it installs updates, each one updates the other, inactive copy. Next boot, the newer one is used - and if there's a problem, it can "fall back" to the other one, which hasn't been modified since it was last used.

To achieve this, ChromeOS has a complex partitioning system with a dozen or so volumes, most small and used by the system for its own inscrutable purposes. However, this double-root-volume setup is gradually also making its way to more conventional, flexible distros. CoreOS was based on ChromeOS, but after acquiring the company, Red Hat killed off the distro. It lives on in the form of a fork called Flatcar Linux.

On the desktop, we know of two distributions that are trying the dual-root-volume model. One is the Chinese Linux Deepin, whose partitioning we looked at in 2022. Its vendor claims over three million sales of its commercial version. In the West, the experimental Vanilla OS recently put out its second release, which also has dual root partitions managed by LVM.

Between Mint with the optional Timeshift, and openSUSE, Garuda, siduction, Spiral Linux, Linux Deepin, and Vanilla OS, multiple efforts are exploring distros where undoing an inadvisable software update is as easy as hitting the undo button in a desktop application. Similar efforts are afoot in the server space too. There are big gains to be made by any outfit that can make Linux as robust as Android. ®

https://www.theregister.com//2024/08/01/linux_rollback_options/