Bookmark this page

Guided Exercise: Configuring Kernel Crash Dumps

Configure and test kernel crash dumps.

Outcomes

You should be able to configure the kdump service to send a crash dump to a remote server.

As the student user on the workstation machine, use the lab command to prepare your system for this exercise.

[student@workstation ~]# lab start kernel-crashdump

This command confirms that your systems are reachable.

Instructions

You are tasked to configure the servera system to send crash dumps to serverb. Use the scp command to send crash dumps. Store crash dumps in the /var/crash directory on serverb.

  1. Log in to servera and switch to the root user.

    [student@workstation ~]$ ssh student@servera
    ...output omitted...
    [student@servera ~]$ sudo -i
    [sudo] password for student: student
    [root@servera ~]#
  2. Enable and start the kdump service.

    [root@servera ~]# systemctl enable --now kdump
    Created symlink from /etc/systemd/system/multi-user.target.wants/kdump.service to /usr/lib/systemd/system/kdump.service.
    [root@servera ~]# systemctl status kdump
    ● kdump.service - Crash recovery kernel arming
       Loaded: loaded (/usr/lib/systemd/system/kdump.service; disabled; vendor preset: enabled)
       Active: active (exited) since Wed 2021-11-10 16:26:46 EST; 3s ago
      Process: 1439 ExecStart=/usr/bin/kdumpctl start (code=exited, status=0/SUCCESS)
     Main PID: 1439 (code=exited, status=0/SUCCESS)
    
    Nov 10 16:26:45 servera.lab.example.com systemd[1]: Starting Crash recovery kernel arming...
    Nov 10 16:26:46 servera.lab.example.com kdumpctl[1439]: kdump: kexec: loaded kdump kernel
    Nov 10 16:26:46 servera.lab.example.com kdumpctl[1439]: kdump: Starting kdump: [OK]
    Nov 10 16:26:46 servera.lab.example.com systemd[1]: Started Crash recovery kernel arming.
  3. Modify the /etc/kdump.conf file to send crash dumps to the /var/crash directory on serverb.

    1. Set serverb.lab.example.com as the SSH target.

      ssh root@serverb.lab.example.com
    2. Set /root/.ssh/kdump_id_rsa as the SSH key.

      sshkey /root/.ssh/kdump_id_rsa
    3. Modify the core_collector line to use scp. Use this line to replace any existing core_collector lines.

      core_collector scp
    4. Set the path parameter to the /var/crash directory.

      path /var/crash
  4. Generate and remotely install the SSH key on serverb. If prompted, enter redhat as the password.

    [root@servera ~]# kdumpctl propagate
    kdump: WARNING: '/root/.ssh/kdump_id_rsa' doesn't exist, using default value '/root/.ssh/kdump_id_rsa'
    kdump: Generating new ssh keys...
    kdump: done.
    /bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/kdump_id_rsa.pub"
    The authenticity of host 'serverb.lab.example.com (172.25.250.11)' can't be established.
    ECDSA key fingerprint is SHA256:NJAyJMx8B2AeIYHRnVLAuJ1XZwblomyOKowyfTwGrTY.
    Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
    /bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
    /bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
    root@serverb.lab.example.com's password: redhat
    
    Number of key(s) added: 1
    
    Now try logging into the machine, with:   "ssh 'root@serverb.lab.example.com'"
    and check to make sure that only the key(s) you wanted were added.
    
    kdump: /root/.ssh/kdump_id_rsa has been added to ~root/.ssh/authorized_keys on serverb.lab.example.com
  5. Restart the kdump service to implement the configuration file changes.

    [root@servera ~]# systemctl restart kdump
    [root@servera ~]# kdumpctl status
    kdump: Kdump is operational
  6. Trigger a system crash to test the kdump configuration.

    1. Enable SysRq functions by setting the value in the /proc/sys/kernel/sysrq file to 1.

      [root@servera ~]# echo 1 > /proc/sys/kernel/sysrq
    2. Trigger a system crash by setting the value in the /proc/sysrq-trigger file to c.

      [root@servera ~]# echo c > /proc/sysrq-trigger

      Note

      The SSH session hangs when the system crash is triggered. Press any key to return to the workstation system.

  7. Log in to serverb and switch to the root user.

    [student@workstation ~]$ ssh student@serverb
    ...output omitted...
    [student@serverb ~]$ sudo -i
    [sudo] password for student: student
    [root@serverb ~]#
  8. Verify that the crash dump is present in the /var/crash directory and view its contents.

    1. List the contents of the /var/crash directory.

      [root@serverb ~]# ll /var/crash
      total 0
      drwxr-xr-x. 2 root root 67 Nov 10 16:44 172.25.250.10-2021-11-10-16:43:54
    2. View the contents of the crash dump.

      [root@serverb ~]# ll /var/crash/172.25.250.10-2021-11-10-16\:43\:54/
      total 1977448
      -rw-r--r--. 1 root root      52465 Nov 10 16:44 kexec-dmesg.log
      -r--------. 1 root root 2024812544 Nov 10 16:44 vmcore
      -rw-r--r--. 1 root root      39176 Nov 10 16:44 vmcore-dmesg.txt
  9. Return to workstation as the student user.

    [root@serverb ~]# exit
    [student@serverb ~]$ exit
    [student@workstation ~]$

Finish

On the workstation machine, use the lab command to complete this exercise. This is important to ensure that resources from previous exercises do not impact upcoming exercises. Wait for servera to finish rebooting before executing the command.

[student@workstation ~]$ lab finish kernel-crashdump

Revision: rh342-8.4-6dd89bd