One task that every system administrator should be able to accomplish is resetting a lost root password.
This task is trivial if the administrator is still logged in, either as an unprivileged user but with full sudo access, or as root.
This task becomes slightly more involved when the administrator is not logged in.
Several methods exist to set a new root password.
A system administrator could, for example, boot the system by using a Live CD, mount the root file system from there, and edit /etc/shadow.
This section explores a method that does not require the use of external media.
On Red Hat Enterprise Linux 9, the scripts that run from the initramfs image can be paused at certain points, to provide a root shell, and then continue when that shell exits.
This script is mostly meant for debugging, and also to reset a lost root password.
Starting from Red Hat Enterprise Linux 9, if you install your system from a DVD, then the default kernel asks for the root password when you try to enter maintenance mode.
Thus, to reset a lost root password, you must use the rescue kernel.
To access that root shell, follow these steps:
Reboot the system.
Interrupt the boot-loader countdown by pressing any key, except Enter.
Move the cursor to the rescue kernel entry to boot (the entry with the rescue word in its name).
Press e to edit the selected entry.
Move the cursor to the kernel command line (the line that starts with linux).
Append rd.break. With that option, the system breaks just before the system hands control from the initramfs image to the actual system.
Press Ctrl+x to boot with the changes.
Press Enter to perform maintenance when prompted.
At this point, the system presents a root shell, and the root file system on the disk is mounted read-only on /sysroot.
Because troubleshooting often requires modifying the root file system, you must remount the root file system as read/write.
The following step shows how the remount,rw option to the mount command remounts the file system where the new option (rw) is set.
Because the system has not yet enabled SELinux, any file that you create does not have SELinux context.
Some tools, such as the passwd command, first create a temporary file, and then replace it with the file that is intended for editing, which effectively creates a file without SELinux context.
For this reason, when you use the passwd command with rd.break, the /etc/shadow file does not receive SELinux context.
To reset the root password, use the following procedure:
Remount /sysroot as read/write.
sh-5.1# mount -o remount,rw /sysroot
Switch into a chroot jail, where /sysroot is treated as the root of the file-system tree.
sh-5.1# chroot /sysroot
Set a new root password.
sh-5.1# passwd root
Ensure that all unlabeled files, including /etc/shadow at this point, get relabeled during boot.
sh-5.1# touch /.autorelabel
Type exit twice.
The first command exits the chroot jail, and the second command exits the initramfs debug shell.
At this point, the system continues booting, performs a full SELinux relabeling, and then reboots again.
If your system was installed by deploying and modifying one of the official cloud images instead of using the installer, then some system configuration aspects might differ for the boot process.
The procedure to use the rd.break option to get a root shell is similar to the previously outlined procedure, with some minor changes.
If your system was deployed from a Red Hat Enterprise Linux cloud image, then your boot menu does not have a rescue kernel by default.
However, you can use the default kernel to enter maintenance mode by using the rd.break option without entering the root password.
The kernel prints boot messages and displays the root prompt on the system console.
Prebuilt images might have multiple console= arguments on the kernel command line in the bootloader.
Even though the system sends the kernel messages to all the consoles, the root shell that the rd.break option sets up uses the last console that is specified on the command line.
If you do not get your prompt, then you might temporarily reorder the console= arguments when you edit the kernel command line in the boot loader.
Looking at the logs of previously failed boots can be useful.
If the system journals persist across reboots, then you can use the journalctl tool to inspect those logs.
Remember that by default, the system journals are kept in the /run/log/journal directory, and the journals are cleared when the system reboots.
To store journals in the /var/log/journal directory, which persists across reboots, set the Storage parameter to persistent in the /etc/systemd/journald.conf file.
[root@host ~]#vim /etc/systemd/journald.conf...output omitted... [Journal] Storage=persistent...output omitted... [root@host ~]#systemctl restart systemd-journald.service
To inspect the logs of a previous boot, use the journalctl command -b option.
Without any arguments, the journalctl command -b option displays only messages since the last boot.
With a negative number as an argument, it displays the logs of previous boots.
[root@host ~]# journalctl -b -1 -p errThis command shows all messages that are rated as an error or worse from the previous boot.
To troubleshoot service startup issues at boot time, Red Hat Enterprise Linux 8 and later versions provide the following tools:
By enabling the debug-shell service with the systemctl enable debug-shell.service command, the system spawns a root shell on TTY9 (Ctrl+Alt+F9) early during the boot sequence.
This shell is automatically logged in as root, so that administrators can debug the system when the operating system is still booting.
Disable the debug-shell.service service when you are done debugging, because it leaves an unauthenticated root shell open to anyone with local console access.
Alternatively, to activate the debug shell during the boot by using the GRUB2 menu, follow these steps:
Reboot the system.
Interrupt the boot-loader countdown by pressing any key, except Enter.
Move the cursor to the kernel entry to boot.
Press e to edit the selected entry.
Move the cursor to the kernel command line (the line that starts with linux).
Append systemd.debug-shell. With this parameter, the system boots into the debug shell.
Press Ctrl+x to boot with the changes.
By appending either systemd.unit=rescue.target or systemd.unit=emergency.target to the kernel command line from the boot loader, the system enters into a rescue or emergency shell instead of starting normally.
Both of these shells require the root password.
The emergency target keeps the root file system mounted read-only, while the rescue target waits for the sysinit.target unit to complete, so that more of the system is initialized, such as the logging service or the file systems.
The root user at this point cannot change /etc/fstab until the drive is remounted in a read write state with the mount -o remount,rw / command.
Administrators can use these shells to fix any issues that prevent the system from booting normally, for example, a dependency loop between services, or an incorrect entry in /etc/fstab.
Exiting from these shells continues with the regular boot process.
During startup, systemd spawns various jobs.
If some of these jobs cannot complete, then they block other jobs from running.
To inspect the current job list, administrators can use the systemctl list-jobs command.
Any jobs that are listed as running must complete before the jobs that are listed as waiting can continue.