Red Hat Enterprise Linux Diagnostics and Troubleshooting

Instructions

The servera machine has a corrupted RPM database, causing YUM to fail to install the httpd package. Resolve the issue and verify that packages can install successfully.

1. Log in to the servera machine and switch to the root user.

   [student@workstation ~]$ ssh student@servera
   ...output omitted...
   [student@servera ~]$ sudo -i
   [sudo] password for student: student
   [root@servera ~]#

2. Use the yum command to attempt to install the httpd package, and view the resulting error. Abort the package installation.

   [root@servera ~]# yum install httpd
   error: rpmdb: damaged header #2 retrieved -- skipping.
   ...output omitted...
   Is this ok [y/N]: n
   Operation aborted.

3. Check the system and YUM logs to find related error messages. No error messages are logged for this issue in either location.

   [root@servera ~]# tail /var/log/messages
   Oct 13 02:02:06 server systemd-logind[912]: New session 9 of user root.
   Oct 13 02:02:06 server systemd[1536]: Reached target Paths.
   Oct 13 02:02:06 server systemd[1536]: Starting D-Bus User Message Bus Socket.
   Oct 13 02:02:06 server systemd[1536]: Reached target Timers.
   Oct 13 02:02:06 server systemd[1536]: Listening on D-Bus User Message Bus Socket.
   Oct 13 02:02:06 server systemd[1536]: Reached target Sockets.
   Oct 13 02:02:06 server systemd[1536]: Reached target Basic System.
   Oct 13 02:02:06 server systemd[1536]: Reached target Default.
   Oct 13 02:02:06 server systemd[1536]: Startup finished in 25ms.
   Oct 13 02:02:06 server systemd[1]: Started User Manager for UID 0.

   [root@servera ~]# tail /var/log/dnf.log
   ...output omitted...
   
   2021-10-13T02:02:17-0400 INFO Total download size: 2.0 M
   2021-10-13T02:02:17-0400 INFO Installed size: 5.4 M
   2021-10-13T02:02:44-0400 ERROR Operation aborted.
   2021-10-13T02:02:44-0400 DDEBUG Cleaning up.

4. Query an arbitrary package and check again for error messages. The output does not display any helpful information to troubleshoot the issue.

   [root@servera ~]# rpm -q redhat-release
   redhat-release-8.4-0.6.el8.x86_64

5. Query the RPM database to list the package metadata content. By default, both STDOUT and STDERR display on the screen. By redirecting STDOUT to the /dev/null file, only error messages are displayed.

   [root@servera ~]# rpm -qa > /dev/null
   error: rpmdbNextIterator: skipping h#       2 region trailer: BAD, tag 758394368 type 757093744 offset -1684108389 count 544501536

6. Use the lsof command to ensure that no rpmdb locks are present.

   [root@servera ~]# lsof | grep /var/lib/rpm

7. Remove the RPM database index files.

   [root@servera ~]# rm /var/lib/rpm/__db*
   rm: remove regular file '/var/lib/rpm/__db.001'? y
   rm: remove regular file '/var/lib/rpm/__db.002'? y
   rm: remove regular file '/var/lib/rpm/__db.003'? y

8. Back up the RPM database files to the root user's home directory before attempting a repair.

   [root@servera ~]# tar cjvf rpmdb-$(date +%Y%m%d-%H%M).tar.bz2 /var/lib/rpm
   tar: Removing leading `/' from member names
   /var/lib/rpm/
   /var/lib/rpm/Requirename
   /var/lib/rpm/Name
   /var/lib/rpm/Basenames
   /var/lib/rpm/Group
   /var/lib/rpm/Packages
   /var/lib/rpm/.dbenv.lock
   /var/lib/rpm/Providename
   /var/lib/rpm/Conflictname
   /var/lib/rpm/Obsoletename
   /var/lib/rpm/Triggername
   /var/lib/rpm/Dirnames
   /var/lib/rpm/Installtid
   /var/lib/rpm/Sigmd5
   /var/lib/rpm/Sha1header
   /var/lib/rpm/Filetriggername
   /var/lib/rpm/Transfiletriggername
   /var/lib/rpm/Recommendname
   /var/lib/rpm/Suggestname
   /var/lib/rpm/Supplementname
   /var/lib/rpm/Enhancename
   /var/lib/rpm/.rpm.lock

9. Verify the integrity of the RPM database Packages file. The verification fails.

   [root@servera ~]# cd /var/lib/rpm
   [root@servera rpm]# /usr/lib/rpm/rpmdb_verify Packages
   rpmdb_verify: BDB0682 Page 32: bad prev_pgno 0 on overflow page (should be 31)
   rpmdb_verify: BDB0683 Page 32: overflow item incomplete
   rpmdb_verify: Packages: BDB0090 DB_VERIFY_BAD: Database verification failed
   BDB5105 Verification of Packages failed.

10. Rename the Packages file to Packages.broken. Use the rpmdb_dump and rpmdb_load commands to dump the valid packages and reload them into a new Packages file.

    [root@servera rpm]# mv Packages Packages.broken
    [root@servera rpm]# /usr/lib/rpm/rpmdb_dump Packages.broken | /usr/lib/rpm/rpmdb_load Packages

11. Again, verify the integrity of the RPM database Packages file. Ensure that verification is successful.

    [root@servera rpm]# /usr/lib/rpm/rpmdb_verify Packages
    BDB5105 Verification of Packages succeeded.

12. Rebuild the RPM database index files.

    [root@servera rpm]# rpm --rebuilddb
    error: rpmdbNextIterator: skipping h#       2 region trailer: BAD, tag 689992815 type 544105843 offset -1952801887 count 1937141087

13. Query the RPM database again to check if any error messages appear. The output should not display any errors.

    [root@servera rpm]# rpm -qa > /dev/null

14. Attempt to install the httpd package. Verify that no errors are displayed this time. Because this step is included only to test the RPM database for errors, abort the package installation.

    [root@servera rpm]# yum install httpd
    Last metadata expiration check: 0:08:13 ago on Thu 14 Oct 2021 10:28:03 AM EDT.
    Dependencies resolved.
    
    ...output omitted...
    
    Install  9 Packages
    
    Total download size: 2.0 M
    Installed size: 5.4 M
    Is this ok [y/N]: n
    Operation aborted.

15. Return to workstation as the student user.

    [root@servera rpm]# exit
    [student@servera ~]$ exit
    Connection to servera closed.
    [student@workstation ~]$