Red Hat Enterprise Linux Diagnostics and Troubleshooting
- Section Resolving Boot Loader Issues on BIOS Systems
- SectionObjectives
- SectionDescribing the BIOS Firmware Interface
- SectionStoring the Master Boot Record (MBR)
- SectionBooting RHEL 8 BIOS Systems Using GRUB2
- SectionThe Red Hat Enterprise Linux 8 Boot Sequence
- SectionUpdating Boot Loader Configuration Files
- SectionMaking Persistent Changes to the GRUB2 Boot Menu
- SectionReinstalling the GRUB2 Boot Loader on the MBR
- SectionDescribing Early Kdump Support in RHEL 8
- Guided Exercise: Resolving Boot Loader Issues on BIOS Systems
- Resolving Boot Loader Issues on UEFI Systems
- Quiz: Resolving Boot Loader Issues on UEFI Systems
- Identifying and Resolving Failing Services
- Guided Exercise: Identifying and Resolving Failing Services
- Resetting the root Password
- Guided Exercise: Resetting the root Password
- Lab: Troubleshooting Boot Issues
- Summary
Abstract
| Goal |
Identify and resolve issues that can affect a system's ability to boot. |
| Objectives |
|
| Sections |
|
| Lab |
|
The Basic Input/Output System (BIOS) is a firmware interface that controls the first step of the boot process and provides the lowest-level interface to external devices.
The BIOS checks the system and locates a bootable device with a Master Boot Record (MBR) partition to start the boot process. In Red Hat Enterprise Linux 8, the MBR boot loader use the GRand Unified Boot loader version 2 (GRUB2). The BIOS loads the boot loader that is stored in the MBR into memory. The boot loader then loads and passes control to the GRUB2 boot menu.
GRUB2 provides two options to boot from a disk on a BIOS system: Store the first part of the GRUB2 boot loader in the Master Boot Record (MBR) of a disk, or store it in the first sector of a partition that is marked as bootable.
Using the MBR has a size limitation. A typical MBR has only 512 bytes and holds the disk's partition table, leaving only 446 bytes for the boot loader. To work around this limitation, GRUB2 can use the MBR gap, also known as the boot track or embedding area, where extra boot modules and files are stored. The MBR gap is the space between the MBR block and the first disk partition.
To work reliably for modern boot loaders, the MBR gap must be at least 31 KiB (63 sectors on a 512-byte sector disk). Disks prepared using anaconda, the RHEL installation utility, typically start the first partition at sector 2048, creating a 1 MiB MBR gap for storing GRUB2 elements.
The boot loader is the first program executed when the system starts. It finds, loads, and transfers control to an operating system. GRUB2 can boot any compatible operating system or transfer control to other boot loaders for unsupported operating systems.
For systems with an existing operating system, the anaconda installer attempts to automatically detect and configure the boot loader to start another operating system. If the boot loader is not detected, then you can manually configure additional operating systems after finishing the installation. Partition requirements vary, depending on whether a system is BIOS or UEFI. Modern systems may also require GPT partitioning, which handles larger disks and offers more flexible configuration than MBR partitioning while still supporting MBR boot loaders. If a user selects automatic partitioning in anaconda, then the installer creates the required partition type with sufficient MBR gap sizing.
GRUB 2 reads its configuration from the /boot/grub2/grub.cfg file on traditional BIOS-based machines and from the /boot/efi/EFI/redhat/grub.cfg file on UEFI-based machines.
The GRUB2 utilities stores files on a BIOS system in various locations:
The
/boot/directory stores the kernel and initial ramdisk.The
/boot/grub2/directory contains configuration files and extension modules.The
/boot/grub2/grub.cfgfile is the configuration file, with a symbolic link to/etc/grub2.cfg.The
/etc/grub.d/directory contains a helper script to generate the/boot/grub2/grub.cfgfile.The
/etc/default/grubfile contains variables used by the configuration helper script.The
/boot/grub2/grubenvfile is exactly 1 KiB, and stores variables such as the default or saved boot entry.
Typically, a system administrator does not manually configure the GRUB2 boot loader. During installation, a new kernel is added to the GRUB2 configuration file automatically. Similarly, the corresponding boot loader entry is removed when a kernel is removed. To temporarily change the GRUB2 menu at boot time, pass parameters to the kernel during startup using the GRUB2 menu. Some parameter options are available in /etc/default/grub file.
-
GRUB_TIMEOUT The number of seconds that the GRUB2 menu is displayed before the default entry gets selected automatically.
-
GRUB_DEFAULT The default entry that starts whenever a user does not select another entry. GRUB2 entries are counted starting with
0.-
GRUB_CMDLINE_LINUX This variable contains a list of extra kernel command-line options to add to every Linux kernel.
After updating the /etc/default/grub file, run the grub2-mkconfig -o /boot/grub2/grub.cfg command to apply the changes. This command uses the configuration helper scripts and settings to generate a new configuration file. The -o option is used to redirect the command's standard output to the named file.
Booting Red Hat Enterprise Linux 8 on a physical x86_64 system is a sequential set of tasks. For x86_64 virtual machines, the hypervisor handles some of the hardware-specific steps.
When the machine powers on, the BIOS or UEFI system firmware runs a Power On Self Test (POST) and initializes the critical hardware.
The firmware searches for a bootable device, which is configured in the firmware, by searching for a Master Boot Record (MBR) on each disk.
The firmware searches for a boot loader on the bootable device and passes control to the boot loader. In a RHEL 8 system, the boot loader is the GRand Unified Boot loader version 2 (GRUB2), installed with the
grub2-installcommand.GRUB2 loads its configuration from the
/boot/grub2/grub.cfgfile and displays a menu with options to select a kernel to boot.The boot loader loads the kernel and the
initramfsfile system from the device into memory after selecting a kernel or when the timeout expires. Theinitramfsfile system contains a minimal but complete set of critical drivers and kernel modules necessary to bootstrap the system.The boot loader passes control to the kernel, with specified kernel command line options and the memory location of the
initramfsfile system.The kernel initializes critical hardware using drivers in the
initramfsfile system and then executes/sbin/initfrom theinitramfsfile system asPID 1. In RHEL 8,/sbin/initis a link tosystemd.The
systemddaemon executes all units for theinitrd.targetand mounts the configured root file system on the/sysrootdirectory.The kernel switches (pivots) the root file system from the
initramfsfile system to the root file system in/sysroot. Thesystemddaemon re-executes itself with thesystemdcopy from the root file system.The
systemddaemon activates the default target, which is either passed from the kernel command line or from the target unit linked to/etc/systemd/system/default.target. Systemd starts or stops units to comply with the target's configuration, and solves dependencies between units automatically.Systemd targets that behave as run levels (with an
AllowIsolate=trueparameter) will activate its units and display a text-based login prompt or a graphical login screen as the result of the intended system state. The system is now booted and ready for user login.
The Boot Loader Specification (BLS) manages boot loader configuration without manipulating boot loader configuration files, by defining a scheme and file format.
With BLS, each boot entry references a single configuration file in the drop-in directory. This drop-in architecture extends the configuration without needing to edit or regenerate the main boot loader configuration files.
BLS also uses this concept to configure boot menu entries. Manage the boot loader menu options by adding, removing, or editing individual boot entry files in the drop-in directory. BLS significantly simplifies the kernel installation process and is consistent across multiple architectures.
Add the following entry in the
/etc/default/grubfile:GRUB_ENABLE_BLSCFG=true
For a legacy BIOS system, generate the
/boot/grub2/grub.cfgfile.[root@host ~]#
grub2-mkconfig -o /boot/grub2/grub.cfgGenerating grub configuration file ... doneGenerate the
/boot/loader/entries.[root@host ~]#
kernel-install add $(uname -r) /lib/modules/$(uname -r)/vmlinuz
The grubby utility manipulates boot loader-specific configuration files. grubby works as a wrapper for the BLS operations. Use grubby for changing the default boot entry, and for adding or removing arguments in a GRUB2 menu entry.
If grubby is invoked manually without specifying a GRUB 2 configuration file, it defaults to locating grub2.cfg.
Obtain the file name of the default kernel with grubby:
[root@host ~]# grubby --default-kernel
/boot/vmlinuz-4.18.0-305.el8.x86_64Find the index number of the default kernel:
[root@host ~]# grubby --default-index
0Use the grubby command to persistently change the default kernel:
[root@host ~]# grubby --set-default /boot/vmlinuz-4.18.0-305.el8.x86_64
The default is /boot/loader/entries/ffffffffffffffffffffffffffffffff-4.18.0-305.el8.x86_64.conf with index 0 and kernel /boot/vmlinuz-4.18.0-305.el8.x86_64List all the kernel menu entries:
[root@host ~]# grubby --info=ALL
index=0
kernel="/boot/vmlinuz-4.18.0-305.el8.x86_64"
args="ro no_timer_check net.ifnames=0 crashkernel=auto $tuned_params"
root="/dev/vda3"
initrd="/boot/initramfs-4.18.0-305.el8.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (4.18.0-305.el8.x86_64) 8.4 (Ootpa)"
id="ffffffffffffffffffffffffffffffff-4.18.0-305.el8.x86_64"View the GRUB 2 menu entry for a specific kernel:
[root@host ~]# grubby --info /boot/vmlinuz-4.18.0-305.el8.x86_64
index=0
kernel="/boot/vmlinuz-4.18.0-305.el8.x86_64"
args="ro no_timer_check net.ifnames=0 crashkernel=auto $tuned_params"
root="/dev/vda3"
initrd="/boot/initramfs-4.18.0-305.el8.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (4.18.0-305.el8.x86_64) 8.4 (Ootpa)"
id="ffffffffffffffffffffffffffffffff-4.18.0-305.el8.x86_64"If the MBR or the MBR gap is damaged, then reinstall GRUB2 into the MBR to overwrite it. If a system is not running when this issue occurs, the repair must be performed from within a Live CD or the rescue environment available from the anaconda installer. If a user with root permissions has access to a running system with this issue, the solution is to use the grub2-install command.
To handle any scenario, practice booting into a rescue environment to reinstall GRUB2 into the MBR:
Boot from an installation source; such as a RHEL DVD image, netboot CD, or a Preboot eXecution Environment (PXE) provided by a RHEL installation tree.
Append
inst.rescueto the boot command line. Alternatively, selectTroubleshootingand then theRescue a Red Hat Enterprise Linux systemoption from the installation media menu, and pressEnter.From the menu, select option
1to mount the file system under the/mnt/sysrootdirectory. To open a shell and obtain a prompt, press Enter.Use
chrootto switch the rescue mode environment to use the mounted root partition (/mnt/sysroot) of the system being repaired.sh-4.4#
chroot /mnt/sysrootReinstall the GRUB2 boot loader on the previously-used block device.
bash-4.4#
/sbin/grub2-install /dev/vdaReboot the system to verify that the GRUB2 boot loader is now operational.
Older RHEL versions were unable to capture kernel crashes at early stages in the boot process, because the kdump service's dependencies required it to start later in the process. Without a kernel dump, it is difficult to troubleshooting early boot process problems.
RHEL 8 introduces early kdump support to address this issue. Early kdump support can log kernel crashes during kernel booting, by storing a crash kernel
vmlinuz and initramfs inside the initramfs of the normal, booting kernel. Early kdump loads directly into reserved memory (crashkernel) during the early boot stage. Two dracut modules are provided in the kexec-tools package to load the crash kernel and initramfs as early as possible during the boot sequence to be available to capture a crash dump of the normal, booting kernel.
References
systemd(1), grub2-install(1), grub2-mkconfig(1), grubby(8), kernel-install(8), and grub2-editenv(1) man and info pages
For further information, refer to What is early kdump support and how do I configure it? at https://access.redhat.com/solutions/3700611
For further information, refer to 26.4. Making Persistent Changes to a GRUB 2 Menu Using the grubby Tool at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/system_administrators_guide/ch-working_with_the_grub_2_boot_loader#sec-Making_Persistent_Changes_to_a_GRUB_2_Menu_Using_the_grubby_Tool
For further information, refer to G.10.3. Reinstalling the GRUB2 boot loader at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/performing_a_standard_rhel_8_installation/index#reinstalling-the-grub2-boot-loader_using-rescue-mode
For further information, refer to How to generate BLS configuration files under /boot/loaders/entries in Red Hat Enterprise Linux 8? at https://access.redhat.com/solutions/5847011