The root passwords for Leeroy needed to be updated, without having access to the current passwords. Fortunately, this is a well-known problem with lots of software support. Both the host and guest OS root passwords needed to be updated.
Updating the Host
Updating a host root password is easy, as it’s something every good bad admin needs to do sometime in their life. There are lots of posts on it, including https://www.tecmint.com/reset-forgotten-root-password-in-debian/
Changing the password of the non-root account was also easy with the command passwd <theusername>
Updating the Guest
Updating the guest password is much harder.
I first considered using guestfish, which had instructions at links like the following:
https://unixhealthcheck.com/blog?id=424 https://tipsmake.com/use-guestfish-to-reset-the-forgotten-root-password-for-the-virtual-machine-in-qcow2-format-on-kvm
I guess that struck me as too easy, so I kept looking for other ideas. The tool virt-sysprep seemed to be designed to do exactly what I needed it to do (as a subset of its overall functionality), and have a greater likelihood of success (hah). The link that seemed best was https://www.cyberciti.biz/faq/reset-a-kvm-clone-virtual-machines-with-virt-sysprep-on-linux/. In particular, the command virt-sysprep -d testvm --root-password password:MySuperSecureRootPasswordHere seemed very straight-forward. The command executed until 44% and then exited. Usually that means a problem occurred, but after looking at other sample logs online, I think that was normal.
Unfortunately, I found that my lvm disk UUID was no longer discoverable by GRUB. GG.
Despite being caused by virt-sysprep there was little help when searching including that topic. The virt-sysprep execution log showed that the UUID for the disk was left alone (which it should have been due to the difficulty of figuring out where to update all UUID references inside a guest OS). However, something was still messed up.
So, I had to generalize my search to just grub. Fortunately, http://www.linuxandubuntu.com/home/ways-to-rescue-or-recover-grub-menu had well-written information on how to solve the problem. In short, use ls to find the mount, then type set prefix=<an lvm root>/boot/grub, insmod normal, normal. Then after booting, sudo grub-install /dev/vda.
Re-establishing SSH
Leeroy’s http port was forwarded to the guest OS correctly, but ssh was broken. It’s possible that this occurred weeks ago, and I was just now noticing. However, it seems more likely that virt-sysprep removed the ssh keys, since that’s part of what it does (and indeed, upon further inspection, the .ssh directories are missing from the users’ home directories). virt-sysprep also wiped out the server’s ssh keys. The solution was to generate new host keys, which was doable via the instructions at https://webby.land/2017/10/02/generate-new-host-keys-after-sysprep/. More specifically, dpkg-reconfigure openssh-server. Of course I also had to update my ~/.ssh/known_hosts file on the client, but that was trivial
Reflections
Of course the most obvious reflection is that I should have done a better job keeping the passwords. At the time I didn’t have a good system for storing passwords offline, but I’ve developed such a system since, so this is less likely to occur in the future.
Perhaps more interestingly, I’d thought about making a newer backup of the guest image (in fact, it was on my mental todo list). The problem was that I was out of space on the drive, and I didn’t want to go through the trouble of figuring out where the images were or where I’d mounted the extra hard drive (though df -h clued me in immediately once I realized that was the problem I was facing). I guess I expected virt-sysprep to work. In reflection, I should have known better. The upside is that I got to learn some new things and reinforce some old tricks (plus who’s to say whether guestfish would have worked either), and it took less than a couple hours. At the end of it all, that’s not so bad.
Defthm Consulting LLC 