Bookmark this page

Chapter 3.  Troubleshooting Boot Issues

Abstract

Goal

Identify and resolve issues that can affect a system's ability to boot.

Objectives
  • Repair boot issues on BIOS systems.

  • Repair boot issues on UEFI systems.

  • Identify and resolve service failures that affect the boot process.

  • Reset the root password.

Sections
  • Resolving Boot Loader Issues on BIOS Systems (and Guided Exercise)

  • Resolving Boot Loader Issues on UEFI Systems (and Quiz)

  • Identifying and Resolving Failing Services (and Guided Exercise)

  • Resetting the root Password (and Guided Exercise)

Lab
  • Troubleshooting Boot Issues

Resolving Boot Loader Issues on BIOS Systems

Objectives

  • Repair boot issues on BIOS systems.

Describing the BIOS Firmware Interface

The Basic Input/Output System (BIOS) is a firmware interface that controls the first step of the boot process and provides the lowest-level interface to external devices.

The BIOS checks the system and locates a bootable device with a Master Boot Record (MBR) partition to start the boot process. In Red Hat Enterprise Linux 8, the MBR boot loader use the GRand Unified Boot loader version 2 (GRUB2). The BIOS loads the boot loader that is stored in the MBR into memory. The boot loader then loads and passes control to the GRUB2 boot menu.

Storing the Master Boot Record (MBR)

GRUB2 provides two options to boot from a disk on a BIOS system: Store the first part of the GRUB2 boot loader in the Master Boot Record (MBR) of a disk, or store it in the first sector of a partition that is marked as bootable.

Limitation of Using MBR

Using the MBR has a size limitation. A typical MBR has only 512 bytes and holds the disk's partition table, leaving only 446 bytes for the boot loader. To work around this limitation, GRUB2 can use the MBR gap, also known as the boot track or embedding area, where extra boot modules and files are stored. The MBR gap is the space between the MBR block and the first disk partition.

To work reliably for modern boot loaders, the MBR gap must be at least 31 KiB (63 sectors on a 512-byte sector disk). Disks prepared using anaconda, the RHEL installation utility, typically start the first partition at sector 2048, creating a 1 MiB MBR gap for storing GRUB2 elements.

Booting RHEL 8 BIOS Systems Using GRUB2

The boot loader is the first program executed when the system starts. It finds, loads, and transfers control to an operating system. GRUB2 can boot any compatible operating system or transfer control to other boot loaders for unsupported operating systems.

For systems with an existing operating system, the anaconda installer attempts to automatically detect and configure the boot loader to start another operating system. If the boot loader is not detected, then you can manually configure additional operating systems after finishing the installation. Partition requirements vary, depending on whether a system is BIOS or UEFI. Modern systems may also require GPT partitioning, which handles larger disks and offers more flexible configuration than MBR partitioning while still supporting MBR boot loaders. If a user selects automatic partitioning in anaconda, then the installer creates the required partition type with sufficient MBR gap sizing.

GRUB 2 reads its configuration from the /boot/grub2/grub.cfg file on traditional BIOS-based machines and from the /boot/efi/EFI/redhat/grub.cfg file on UEFI-based machines.

The GRUB2 utilities stores files on a BIOS system in various locations:

  • The /boot/ directory stores the kernel and initial ramdisk.

  • The /boot/grub2/ directory contains configuration files and extension modules.

  • The /boot/grub2/grub.cfg file is the configuration file, with a symbolic link to /etc/grub2.cfg.

  • The /etc/grub.d/ directory contains a helper script to generate the /boot/grub2/grub.cfg file.

  • The /etc/default/grub file contains variables used by the configuration helper script.

  • The /boot/grub2/grubenv file is exactly 1 KiB, and stores variables such as the default or saved boot entry.

Describing the GRUB2 Configuration File

Typically, a system administrator does not manually configure the GRUB2 boot loader. During installation, a new kernel is added to the GRUB2 configuration file automatically. Similarly, the corresponding boot loader entry is removed when a kernel is removed. To temporarily change the GRUB2 menu at boot time, pass parameters to the kernel during startup using the GRUB2 menu. Some parameter options are available in /etc/default/grub file.

GRUB_TIMEOUT

The number of seconds that the GRUB2 menu is displayed before the default entry gets selected automatically.

GRUB_DEFAULT

The default entry that starts whenever a user does not select another entry. GRUB2 entries are counted starting with 0.

GRUB_CMDLINE_LINUX

This variable contains a list of extra kernel command-line options to add to every Linux kernel.

After updating the /etc/default/grub file, run the grub2-mkconfig -o /boot/grub2/grub.cfg command to apply the changes. This command uses the configuration helper scripts and settings to generate a new configuration file. The -o option is used to redirect the command's standard output to the named file.

The Red Hat Enterprise Linux 8 Boot Sequence

Booting Red Hat Enterprise Linux 8 on a physical x86_64 system is a sequential set of tasks. For x86_64 virtual machines, the hypervisor handles some of the hardware-specific steps.

  1. When the machine powers on, the BIOS or UEFI system firmware runs a Power On Self Test (POST) and initializes the critical hardware.

  2. The firmware searches for a bootable device, which is configured in the firmware, by searching for a Master Boot Record (MBR) on each disk.

  3. The firmware searches for a boot loader on the bootable device and passes control to the boot loader. In a RHEL 8 system, the boot loader is the GRand Unified Boot loader version 2 (GRUB2), installed with the grub2-install command.

  4. GRUB2 loads its configuration from the /boot/grub2/grub.cfg file and displays a menu with options to select a kernel to boot.

  5. The boot loader loads the kernel and the initramfs file system from the device into memory after selecting a kernel or when the timeout expires. The initramfs file system contains a minimal but complete set of critical drivers and kernel modules necessary to bootstrap the system.

  6. The boot loader passes control to the kernel, with specified kernel command line options and the memory location of the initramfs file system.

  7. The kernel initializes critical hardware using drivers in the initramfs file system and then executes /sbin/init from the initramfs file system as PID 1. In RHEL 8, /sbin/init is a link to systemd.

  8. The systemd daemon executes all units for the initrd.target and mounts the configured root file system on the /sysroot directory.

  9. The kernel switches (pivots) the root file system from the initramfs file system to the root file system in /sysroot. The systemd daemon re-executes itself with the systemd copy from the root file system.

  10. The systemd daemon activates the default target, which is either passed from the kernel command line or from the target unit linked to /etc/systemd/system/default.target. Systemd starts or stops units to comply with the target's configuration, and solves dependencies between units automatically.

  11. Systemd targets that behave as run levels (with an AllowIsolate=true parameter) will activate its units and display a text-based login prompt or a graphical login screen as the result of the intended system state. The system is now booted and ready for user login.

Updating Boot Loader Configuration Files

The Boot Loader Specification (BLS) manages boot loader configuration without manipulating boot loader configuration files, by defining a scheme and file format.

With BLS, each boot entry references a single configuration file in the drop-in directory. This drop-in architecture extends the configuration without needing to edit or regenerate the main boot loader configuration files.

BLS also uses this concept to configure boot menu entries. Manage the boot loader menu options by adding, removing, or editing individual boot entry files in the drop-in directory. BLS significantly simplifies the kernel installation process and is consistent across multiple architectures.

Generate BLS Configuration Files

  1. Add the following entry in the /etc/default/grub file:

    GRUB_ENABLE_BLSCFG=true
  2. For a legacy BIOS system, generate the /boot/grub2/grub.cfg file.

    [root@host ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
    Generating grub configuration file ...
    done
  3. Generate the /boot/loader/entries.

    [root@host ~]# kernel-install add $(uname -r) /lib/modules/$(uname -r)/vmlinuz

Making Persistent Changes to the GRUB2 Boot Menu

The grubby utility manipulates boot loader-specific configuration files. grubby works as a wrapper for the BLS operations. Use grubby for changing the default boot entry, and for adding or removing arguments in a GRUB2 menu entry.

If grubby is invoked manually without specifying a GRUB 2 configuration file, it defaults to locating grub2.cfg.

Obtain the file name of the default kernel with grubby:

[root@host ~]# grubby --default-kernel
/boot/vmlinuz-4.18.0-305.el8.x86_64

Find the index number of the default kernel:

[root@host ~]# grubby --default-index
0

Use the grubby command to persistently change the default kernel:

[root@host ~]# grubby --set-default /boot/vmlinuz-4.18.0-305.el8.x86_64
The default is /boot/loader/entries/ffffffffffffffffffffffffffffffff-4.18.0-305.el8.x86_64.conf with index 0 and kernel /boot/vmlinuz-4.18.0-305.el8.x86_64

List all the kernel menu entries:

[root@host ~]# grubby --info=ALL
index=0
kernel="/boot/vmlinuz-4.18.0-305.el8.x86_64"
args="ro no_timer_check net.ifnames=0 crashkernel=auto $tuned_params"
root="/dev/vda3"
initrd="/boot/initramfs-4.18.0-305.el8.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (4.18.0-305.el8.x86_64) 8.4 (Ootpa)"
id="ffffffffffffffffffffffffffffffff-4.18.0-305.el8.x86_64"

View the GRUB 2 menu entry for a specific kernel:

[root@host ~]# grubby --info /boot/vmlinuz-4.18.0-305.el8.x86_64
index=0
kernel="/boot/vmlinuz-4.18.0-305.el8.x86_64"
args="ro no_timer_check net.ifnames=0 crashkernel=auto $tuned_params"
root="/dev/vda3"
initrd="/boot/initramfs-4.18.0-305.el8.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (4.18.0-305.el8.x86_64) 8.4 (Ootpa)"
id="ffffffffffffffffffffffffffffffff-4.18.0-305.el8.x86_64"

Reinstalling the GRUB2 Boot Loader on the MBR

If the MBR or the MBR gap is damaged, then reinstall GRUB2 into the MBR to overwrite it. If a system is not running when this issue occurs, the repair must be performed from within a Live CD or the rescue environment available from the anaconda installer. If a user with root permissions has access to a running system with this issue, the solution is to use the grub2-install command.

To handle any scenario, practice booting into a rescue environment to reinstall GRUB2 into the MBR:

  1. Boot from an installation source; such as a RHEL DVD image, netboot CD, or a Preboot eXecution Environment (PXE) provided by a RHEL installation tree.

  2. Append inst.rescue to the boot command line. Alternatively, select Troubleshooting and then the Rescue a Red Hat Enterprise Linux system option from the installation media menu, and press Enter.

  3. From the menu, select option 1 to mount the file system under the /mnt/sysroot directory. To open a shell and obtain a prompt, press Enter.

  4. Use chroot to switch the rescue mode environment to use the mounted root partition (/mnt/sysroot) of the system being repaired.

    sh-4.4# chroot /mnt/sysroot
  5. Reinstall the GRUB2 boot loader on the previously-used block device.

    bash-4.4# /sbin/grub2-install /dev/vda
  6. Reboot the system to verify that the GRUB2 boot loader is now operational.

Describing Early Kdump Support in RHEL 8

Older RHEL versions were unable to capture kernel crashes at early stages in the boot process, because the kdump service's dependencies required it to start later in the process. Without a kernel dump, it is difficult to troubleshooting early boot process problems.

RHEL 8 introduces early kdump support to address this issue. Early kdump support can log kernel crashes during kernel booting, by storing a crash kernel vmlinuz and initramfs inside the initramfs of the normal, booting kernel. Early kdump loads directly into reserved memory (crashkernel) during the early boot stage. Two dracut modules are provided in the kexec-tools package to load the crash kernel and initramfs as early as possible during the boot sequence to be available to capture a crash dump of the normal, booting kernel.

References

systemd(1), grub2-install(1), grub2-mkconfig(1), grubby(8), kernel-install(8), and grub2-editenv(1) man and info pages

For further information, refer to What is early kdump support and how do I configure it? at https://access.redhat.com/solutions/3700611

For further information, refer to 26.4. Making Persistent Changes to a GRUB 2 Menu Using the grubby Tool at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/system_administrators_guide/ch-working_with_the_grub_2_boot_loader#sec-Making_Persistent_Changes_to_a_GRUB_2_Menu_Using_the_grubby_Tool

For further information, refer to G.10.3. Reinstalling the GRUB2 boot loader at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/performing_a_standard_rhel_8_installation/index#reinstalling-the-grub2-boot-loader_using-rescue-mode

For further information, refer to How to generate BLS configuration files under /boot/loaders/entries in Red Hat Enterprise Linux 8? at https://access.redhat.com/solutions/5847011

Revision: rh342-8.4-6dd89bd