Part 1: Hosing Down your Computers

In an earlier post about setting up and maintaining a Linux audio lab I wrote:

Above I mentioned that cron-apt is used to check package status (daily) and use the email packages to email package status to the system admin (me). This, however, is not configured to auto update anything. To date we have followed the model of preferring stability to security. That means that the systems aren’t updated more than once or twice per month and then are done manually first on a live-swap machine to make sure nothing major breaks, and then on all the lab machines via ssh. KDE Neon has proven very reliable with updates and, to date, has not resulted in workflow interruption.

Well, “hilariously” I had not updated our core OS during the pandemic because the labs were closed to students. In preparing for this semester I started updating both our server and lab machines so we would be up-to-date and ready to go. Of course this resulted in a not-inconsiderable amount of chaos. I will spare you the gory details, but a few notes on our situation and the remedies we arrived at.

1. Our machines were really out-of-date — Plasma desktop was at version 5.17 (Fall 2019) and our LTS was 2018. This meant we were not getting security updates any longer. Not good.
2. Despite the above, everything *was* working.

I updated via command-line using apt dist-upgrade on one machine (that machine broke rather badly) and a combo of the Discover GUI and pkcon on the other four machines, one of which also broke. It turns out there were several issues that were out of my control, including a weird bug that bit me because of the extremely out of dateness of the previous system.

On machine 1, the *really* hosed one, I had no graphical login and no amount of cleaning, purging, and reinstalling programs solved the problem. Ultimately, I had to (because of a bug) roll back a few packages to earlier versions to trick another program (with a bug) to update them correctly (and then itself.) ITS helped me get up-and-running by using a different graphical login and DM and then it was just trial and error (and finding the right obscure bug report.)

For machine 2, the symptoms were similar, but I never lost Plasma as a usable desktop, and a series of dpkg reconfigures, purges, and reinstalls to some core plasma libraries got the machine back up in a day.

Part 2: Lessons

Lesson one, do not let your machines get seriously old. This, ultimately, was the problem I faced. In normal times, these machines are updated every few months at the end or beginning of terms. By sitting around without updates for > 2 years incompatibilities existed between old and new things and ultimately, *some* machines broke.

Lesson two, do not update your computers in a production environment unless you have time to fix problems. I scheduled our updates and was able to scramble to fix broken machines because it was the beginning of the semester and no one was using the lab yet.

Lesson three, all machines are different, even when they are “identical.” If the first machine I updated hadn’t have broken, I would have thought everything was cool and pushed updates to all my machines. Because that first machine broke, I was able to plan and plot for the remaining machines, making sure they worked as I went so there would be *some* working machines available if needed. All of the systems are identical in terms of hardware and all were really close in terms of their software configuration and out-of-dateness. Still, two of five experienced problems while three did not. This just illustrates a computing truth: some things “break for no reason.” Well, of course they don’t, but the level of complexity required for figuring out why one machine breaks on update and another doesn’t when they started out almost identical is absurdly high.

Lesson four, not all upgrade methods are good/valid **or** Choose the right tool for the job. KDE Neon uses the pkcon package to manage differences between their packages and those required/used by the Ubuntu base to ensure compatibility and preclude breakage. Ultimately, this works, but with caveats.

• To get some things like firmware updates, you must use the Discover GUI or manually use fwupd — it doesn’t happen with pkcon or apt.
• For normal updates you *can* use apt update && apt dist-upgrade which is handy because auto-complete (for searching packages) doesn’t work with pkcon. However, your mileage may vary. You also, AFAIK, cannot de-cruft your machine in the same way you can with apt using apt autoclean && apt autoremove which means just using pkcon you will eventually have a *lot* of old kernels and other programs sitting around on your system taking up space.

Outro

In the end, I “got away with it” which is to say, everything is fine in lab land. Careful timing and ability to scramble and fix saved me (along with expert linux support from ITS…) I wish you better luck in your tech audio endeavors. Hit up the comments with your favorite tails of audio-land woe, or find me at The Playwrite and we’ll drink to hosed and un-hosed computers. Cheers!