This is your annual reminder to do a snapshot (timeshift or whatever you prefer) before doing relatively minor changes to your system.
I was supposed to be in bed now, but instead I am stuck troubleshooting xorg refusing to start after an apt-get dist-upgrade.
And as far as friendly reminders go, I should’ve given myself an unfriendly reminder beforehand, as it’s not the first time…
UPDATE: Fuck nvidia 545. All my homies hate nvidia 545. 535 4 lyf!
BTW, nixos allows you to easily roll back to a previous generation on boot in case an update breaks something.
Just sayin’ 😁
So does Fedora Silverblue for the record.
It is a damn impressive feature to realize you just broke your install and are able to say “no problem”.
Silverblue gang!
deleted by creator
Si does tumbleweed, two-clicks rollback from the GRUB menu
Woohoo
Opensuse’s aeon is gold
Friendly reminder: just don’t buy nvidia
Many already have nvidia before they start with linux. I’m still on my 1060 from 2018.
Another problem is that most manufacturers of laptops directed at the Linux crowd, for some reason I will never understand, insist on punching Nvidia hardware instead of AMD/ATI. How does that help?
I had one before, then 2060, then 2080 and finally 6800 (current one), how is your nvidia experience right now compared to 2018? Any better?
I started with linux begin 2019. I just use xorg so I don’t know about wayland problems. I think a long time ago nvidia broke once and I switched to nvidia-dkms and it has worked fine since until recently where a mesa update broke xorg but I don’t think that has to do with nvidia. Getting CUDA to work might be trouble though (I think I briefly tried once).
Unfortunately for those of us that use Cuda features, AMD just really isn’t that viable of an alternative. Anyone who’s had to deal with ROCM can attest to this…
As soon as AMD is competitive in price/performance in my region. I won’t give them money just becauae they have open source drivers.
All these laptops make a circle… All these laptops make a circle… ;)
deleted by creator
I slept on BTRFS for years, until getting it by default in OpenSuse. I’m never going back. Snapshots and dedupes are killer features.
I’m on Debian now, and I get an automatic before and after snapshot with every apt install.
How are you taking the snapshot automatically?
Honestly, I’m not 100% sure how to set it up, because I let Spiral Linux handle this for me. Spiral Linux is basically a Debian spin, not a full distro. It installs Debian and preconfigures some reasonable defaults, like BTRFS, backported kernels, virtualization support, and some other niceties. The end result is a pure Debian system, operating solely on Debian repos.
If you’re already running Debian, or want to install from stock Debian media, there are instructions from various places. The gist is that the
snapperpackage automatically sets this up, though you might need to manually configure some subvolumes. Here is a very detailed guide that starts from zero. It is mildly opinionated so it’s a little hard to tell what the minimal config would be: https://medium.com/@inatagan/installing-debian-with-btrfs-snapper-backups-and-grub-btrfs-27212644175fSearch for the part where they mention apt.conf.d.
Ah okay well I appreciate the response anyways. I’m also struggling to figure how to snapshot my /home since I put it in a different partition during install. Timeshift is “unable to see it”.
Tumbleweed does it, comes preconfigured out of the box. TBH I’m trying to get the same on Arch & fail. The snapshot-before-change are easy enough, but reverting is where I fail.
Have you tried to manually specify the subvolume id (sudo btrfs subvolume list /) of the snapshot you want to restore to in /etc/fstab?
When I was distro hopping I believe Garuda Linux took snapshots and was easily able to restore a few times. Maybe you can reverse engineer what they’ve done???
I’m running Nobara right now for my gaming setup but I’m half tempted to try TW.
When u realize you dont even use a backup software rn
I just copy my whole-ass home dir to an external drive every so often like a caveman.
I just make sure I don’t have any important data.
Which you should absolutely do even if you snapshot the eff out of your system. What about hardware failure, eh? Can’t snap that nvidia shit can you?
Remember to check your three "B"s; your balls, your breasts and your backups.
I’m using this from now on
Or you opt for an operating system that lers you roll-back whole generations for when such a thing happens (GNU Guix for the win!)
Timeshift with BTRFS kicks ass. I have mine set for daily snapshots, retained for a week. Only the changes between snapshots are stored, so the extra disk usage is minimal, and easily justified by the peace of mind in case of fuck-ups or broken updates.
Just make sure you back up your stuff to a second disk❤️
Only the changes between snapshots are stored, so the extra disk usage is minimal
If you want to use a similar approach for backups, Borgbackup is a pretty nice piece of software. I have two backups of my most important files: One on my NAS at home, and one “in the cloud” on a storage VPS (ends up way cheaper than using S3, B2 or anything like that).
Which storage VPS have you selected? I’m looking at Hetzner atm.
I’ve got one with HostHatch that’s 10TB of space for $10/month. It was an offer they had during Black Friday 2020. They had a similar offer during Black Friday 2023 but I think it was around $20/month, paid yearly.
I live in the San Francisco Bay Area and my storage server is in Los Angeles, which is around 10ms round-trip ping time from my home internet connection.
Hetzner is good too. They have relatively cheap “storage boxes” that are a shared environment rather than a VPS. You don’t get proper SSH access, but they do support FTPS, SFTP, Samba, Borgbackup, Restic, rclone, rsync and WebDAV. https://www.hetzner.com/storage/storage-box
Borgbackup encrypts the backups, so the host won’t be able to actually view your backups.
Linux noobie here. Any tips on what to search for for instructions on how to do this? Bonus points if it has a GUI and is easy to use.
Timeshift. It has an easy to understand GUI that doesn’t really need much of an intro: You create snapshots of your system files and configs that can be restored if/when you bungle it up.
Timeshift works best if you use BTRFS for your root partition because snapshots can be taken instantly. I have mine setup to automatically take a snapshot every day.
Thanks, all setup!
Never used it myself and i am still a linix noob but what is the restore process if your OS isnt bootable?
If their like a rescue environment you boot into or something?
There are many approaches, depending on what broke. In my case the system was fine, just xorg being completely borked. So I logged into the console and fixed it.
If regular console doesn’t work, something really went bad during boot, for which there’s single-user mode which is kind of similar to safe mode from Windows 98 (I’m sure there’s something similar in newer windows versions).
And of that doesn’t work, there’s the minimalistic rescue shell.
And if that doesn’t work, you can boot from a USB or some other external media and try to fix your system from that, maybe even using chroot to use the system somewhat normally.
As long as your file system is on BTRFS, timeshift is, I O, the easiest backup app for Linux, specially if you’re just getting your feet wet in it.
Why does Timeshift only support btrfs? Is it just a lack of developers? LVM supports snapshots too, even if you’re just using ext4. ZFS supports snapshots too.
Timeshift supports rsync snapshots. No btrfs needed :)
Thanks for letting me know. It’s never worked on any FS other than BTRFS for me, I’ll give it another shot.
Is it dumb that I only backup my docs and anything else I think is important? I can rebuild fairly quickly if something would happen. I ask since I know that people backup a variety of their directories
If you’re comfortable, you’re fine. Anything more would just be to speed up the rebuild, so it’s less important if you don’t mind taking the time.
I will never install a Linux desktop without a snapshotting root filesystem ever again. Nvidia driver updates, /boot getting too full during kernel or driver updates, a bad update of pipewire half a year ago, and more I can’t remember. Was always able to boot to previous snapshot of the OS, and address whatever it was. Some ZFS here, some BTRFS there… and my small fleet of Linux desktops are as easy to recover as any immutable OS. Better even, because snapshots allow me to pull individual items or things between states easily, too.
Am on LMDE6 with an ancient Nvidia card. Because I’ve had to resort to using the Nvidia OEM driver installer (which can be a pain to use), installed Xorg updates lurk quietly until a full reboot at which point they generally cause offloading of GPU tasks to the CPU instead because it hasn’t figured things out properly.
Timeshift has been useful at least twice in getting me back to a less stressed system.
I think I have a procedure figured out now though (documented here for posterity even if it helps no-one today):
-
Make a Timeshift snapshot just in case
-
Install the pending Xorg update
-
Reboot so it’s fully active
-
Check to see if GPU tasks are being offloaded to the CPU by doing something graphics intensive and noting temperatures or usage%. If not, a miracle has occurred and continuing isn’t needed.
-
sudoremove the execute permission on/usr/bin/Xorgso that it can’t immediately be restarted by subsystems designed to protect the average Mint user from command lines and consoles. -
Kill Xorg
-
Log in through a console, via Ctrl+Alt+F1 or similar if not dumped to one by killing Xorg.
-
Re-install the Nvidia OEM driver
-
sudoput the aforementioned execute permission back on -
Repeat steps 2 and 3 and hope that this time the GPU is doing the work.
Reboots ought to be replaceable by running specific commands, but I haven’t gone deep enough into things to know the right things to do there. Reboots are quick and easy enough.
Obvious intermediate steps include not doing anything else important during this and saving important work before starting.
e.g. did you know it’s possible to bookmark all open tabs? Well worth looking into.
-
Neat little guide (Arch, systemd) to set up automatic snapshots when you run a pacman/yay update and the option to chose snapshots in grub during startup. Really useful.
Definitely. I use Timeshift on Linux Mint Debian Edition and set it to take weekly snapshots. Saved my bacon about 2 weeks ago when a kernel update borked my system.
The nvidia 545 drivers are an absolute dumpster fire. Even for beta drivers they are easily the worst drivers I’ve ever used. They claim to fix the vrr gsync bug tho… so as soon as they fix gestures broadly everything else, maybe they’ll be good
I’m on 545 and I have no issues. But I’m also not using Ubuntu…
Maybe it’s the distro that’s the problem not the backup. I mean I’d rather have a distro with smooth updates than one that makes me need snapshots.
What’s even the point with such a distro, ok so I restore previous working state, then what, I can’t do updates anymore? Living in fear of official updates sounds terrible.
So you’re implying there exists a distro that is perfect and never breaks anything? Sounds like denial. Having time shift in place is risk management and says nothing about the distro, which btw all are imperfect and may break eventually. Kinda confused how one can run Linux and be unaware of how complex and fallible ALL software is.
No, I’m implying that official updates breaking the system is insane and should not be accepted as the norm to the point you casually need to use snapshots just to keep your system working.
That’s not an accurate portrayal of anything though. You’re implying risk management means you’re accepting that the system sucks and the only way to keep it running is to have a backup system.
That’s not what anyone is doing. They are acknowledging the flawed nature of software and humanity in general and guarding against the consequences of this. It’s being smart. Name one distro that has never broken with updates? You cannot because there obviously couldn’t exist one.
A major distro breaking your system is the equivalent of a flower pot falling on your head walking down the street. Does it happen? Sure. Do I want to spend my life wearing a motorcycle helmet and looking up all the time? No.
I’m not saying distros can’t crap on you, I’m saying stop tolerating it. Raise a stink or switch distro. There are distros out there where you don’t have to live in constant fear and where nothing happens if you don’t have snapshots.
I do have backups, precisely because shit happens. But there’s a difference between a helmet and health insurance.
Okay what distros are we talking about? What’s an example of an unreliable one and what’s a reliable one?
I think it’s much more complicated than this. Honestly there’s a reason it took decades for Linux distros in general to get as stable as they are today. It’s really really hard to build an operating system.
I think a better analogy for installing updates on linux would be riding a motorcycle. Accidents happen all the time. They’re bound to, be prepared. Just because you can ride for 30 years unscathed doesn’t mean you can take that for granted.
My 2 cents, OpenSUSE Leap is stable as hell. lots of QA happening with their automated testing, and keeping in lockstep with SUSE releases (now sharing same binaries). Every distro upgrade has gone flawlessly, but when I have had the urge to tinker and do stupid things inf config files the built in btrfs snapshots have been a godsend.
What you wrote and what I’m reading about openSuse leap sound excellent. My only concern is support. Just about all the apps and tools I use are well supported on Ubuntu based systems but I don’t recall seeing nearly as much support for rpm based oses, including open suse . What’s your experience? If you go to install just any random software you just heard about, how well does it typically go getting it installed/working?
.deb distros are doomed from the start if you need to use third-party repos (which you do, for a desktop system) because they always end up undermining the stability of the packages from the core repos in the long run.
Try an Arch-based distro, they’re super stable because their compatibility model is more robust, and there are options depending on how much hand-holding you want — ranging from vanilla Arch to Endeavour to Manjaro.
Ya know, this is super interesting you mention Arch. The only person I’ve known IRL who uses and loves Arch champions it hardcore but with the caveat that you have to be okay with things breaking due to the rolling release model. Due to his guidance I have avoided arch specifically. I’ve been running Ubuntu based distros a couple of years and only had issues with updates breaking things like 2 times… Both of which didn’t require a wipe or anything.
Windows?
Lol wtf. Windows has broken in catastrophic ways after updates on several occasions for me. Some of the many reasons I avoid it. But nice GoTcHa tho
I think it’s just dumb to not make a backup before large updates. There’s so many things happening, a lot can go wrong, especially if you have added 3rd party repos and have customized core parts of the system, not just through config files but let’s say you switched to latest kde plasma from the one your distro ships.
And what happens if you have to restore the backup?
You can look up what’s the solution to your problem in peace while everything is still working. If it was a server, all the services are still available, if it was your desktop you don’t have to use a live linux usb that’s without all your configs to find the solutionYou make a good point. Ubuntu gives you so many ways to shoot yourself in the foot that it’s pretty much a given that it will get messed up eventually. So you have to use snapshots.
On Arch based distros the updates just work. I’ve never had to snapshot anything. But having just one single community repo (AUR) contributes to that a great deal.
I don’t like Ubuntu, and I do like Arch’s philosophy. But I think Arch is the more prone to breakage of the two.
Except that time a year and a bit ago where an Arch update broke Grub for a huge number of users.
No distro is immune to breakage.
And a filesystem snapshotting tool would help you restore bootloader how?..
So you agree, Arch can also break by updating.
Of course it can. And your PC can also fall off the desk. I’m saying a snapshot tool is a really poor solution for distro problems, it’s really a bandaid for a problem that shouldn’t exist.
Use a decent distro, take proper backups, and use snapshots for what they were intended — recovering small mistakes with personal files, not for system maintenance.
Use a decent distro
That’s the point – your claim about deb-based distros is just anecdotal.
The example here is Nvidia updates borking the system. I’ve have that happen to me numerous times on Arch-based systems.
I’ve run deb-based distros on some boxes over years of updates with no issues. On the other hand I’ve had updates cause breakages on Arch-based systems pretty much every time I’ve run them.
Which is to say anecdotes are useless, updates can break systems, and being able to immediately roll back to a working system and deal with updating later is a simple, nice thing to have with no downsides.
nvidia drivers have often done weird shit to my system. so, I would probably also lean towards amd in the future.
Yeah that’s why I use Nvidia, because AMD drivers are super stable.
