Red Hat Enterprise Linux Diagnostics and Troubleshooting

Gather information about the failing service.

1. Log in to servera and switch to the root user.

   [student@workstation ~]$ ssh student@servera
   ...output omitted...
   [student@servera ~]$ sudo -i
   [sudo] password for student: student
   [root@servera ~]#

2. Determine the status of the web service.

   The service is in a failed state. The /usr/sbin/httpd binary file could not start.

   [root@servera ~]# systemctl status httpd
   ● httpd.service - The Apache HTTP Server
      Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
      Active: failed (Result: exit-code) since Tue 2021-11-16 17:58:38 EST; 40s ago
        Docs: man:httpd.service(8)
     Process: 1953 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=203/EXEC)
    Main PID: 1953 (code=exited, status=203/EXEC)
   
   Nov 16 17:58:38 servera.lab.example.com systemd[1]: Starting The Apache HTTP Server...
   Nov 16 17:58:38 servera.lab.example.com systemd[1]: httpd.service: Main process exited, code=exited, status=203/EXEC
   Nov 16 17:58:38 servera.lab.example.com systemd[1]: httpd.service: Failed with result 'exit-code'.
   Nov 16 17:58:38 servera.lab.example.com systemd[1]: Failed to start The Apache HTTP Server.

3. Inspect the httpd binary file.

   The binary file is missing executable privileges.

   [root@servera ~]# ls -la /usr/sbin/httpd
   -rw-r--r--. 1 root root 579968 Jan 27  2021 /usr/sbin/httpd

4. Check which package provided the httpd binary.

   [root@servera ~]# rpm -qf /usr/sbin/httpd
   httpd-2.4.37-39.module+el8.4.0+9658+b87b2deb.x86_64

5. Check the status of the package.

   [root@servera ~]# rpm -qV httpd
   .M.......    /usr/sbin/apachectl
   .M.......    /usr/sbin/fcgistarter
   .M.......    /usr/sbin/htcacheclean
   .M.......    /usr/sbin/httpd
   .M.......    /usr/sbin/rotatelogs
   .M.......    /usr/sbin/suexec

Repair the failing service.

1. Restore the permissions of the files in the package.

   [root@servera ~]# rpm --setperms httpd

2. Check the status of the package again.

   [root@servera ~]# rpm -qV httpd

3. Restart the web service.

   [root@servera ~]# systemctl restart httpd

4. Verify that the web service is running.

   [root@servera ~]# systemctl status httpd
   ● httpd.service - The Apache HTTP Server
      Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
      Active: active (running) since Tue 2021-11-16 18:01:19 EST; 6s ago
        Docs: man:httpd.service(8)
    Main PID: 3066 (httpd)
      Status: "Started, listening on: port 80"
       Tasks: 213 (limit: 11049)
      Memory: 27.2M
      CGroup: /system.slice/httpd.service
              ├─3066 /usr/sbin/httpd -DFOREGROUND
              ├─3067 /usr/sbin/httpd -DFOREGROUND
              ├─3068 /usr/sbin/httpd -DFOREGROUND
              ├─3069 /usr/sbin/httpd -DFOREGROUND
              └─3070 /usr/sbin/httpd -DFOREGROUND
   
   Nov 16 18:01:19 servera.lab.example.com systemd[1]: Starting The Apache HTTP Server...
   Nov 16 18:01:19 servera.lab.example.com systemd[1]: Started The Apache HTTP Server.
   Nov 16 18:01:19 servera.lab.example.com httpd[3066]: Server configured, listening on: port 80

5. View the default page that the web server returned.

   The server provides the default Apache2 page, not the required <h1>Home</h1> page. The required pages are on the /dev/vdb1 device.

   [root@servera ~]# curl localhost
   ...output omitted...

Gather information about the required web files and the /dev/vdb1 device.

1. List the contents of the /var/www/html directory.

   [root@servera ~]# ls -l /var/www/html
   total 0

2. List the file system of the /var/www/html directory.

   [root@servera ~]# df -T /var/www/html
   Filesystem     Type 1K-blocks    Used Available Use% Mounted on
   /dev/vda3      xfs   10371052 2308320   8062732  23% /

3. Mount the correct file system.

   An error is expected.

   [root@servera ~]# mount /var/www/html
   mount: /var/www/html: mount(2) system call failed: Structure needs cleaning.

4. Determine the device file system type.

   [root@servera ~]# lsblk -f /dev/vdb1
   NAME FSTYPE LABEL UUID                                 MOUNTPOINT
   vdb1 xfs          73f56278-a55b-4e25-beba-436a880cb0c5

Repair the corrupted device and mount the required files.

1. Perform a read-only repair of the file system.

   [root@servera ~]# xfs_repair -n /dev/vdb1
   ...output omitted...
   Invalid inode number 0x75bcd15
   xfs_dir_ino_validate: XFS_ERROR_REPORT
   Metadata corruption detected at 0x561f28d32dd0, inode 0x80 data fork
   couldn't map inode 128, err = 117, can't compare link counts
   No modify flag set, skipping filesystem flush and exiting.

2. Repair the file system.

   [root@servera ~]# xfs_repair /dev/vdb1
   ...output omitted...
   Phase 6 - check inode connectivity...
           - resetting contents of realtime bitmap and summary inodes
           - traversing filesystem ...
           - traversal finished ...
           - moving disconnected inodes to lost+found ...
   disconnected inode 137, moving to lost+found
   Phase 7 - verify and correct link counts...
   done

3. Attempt to mount the device again.

   [root@servera ~]# mount /var/www/html

4. Inspect the contents of the /var/www/html directory.

   New files are present; however, the index.html file is still missing.

   [root@servera ~]# ls -l /var/www/html
   total 12
   -rw-r--r--. 1 root root 15 Feb 24  2016 about.html
   drwxr-xr-x. 2 root root 16 Feb 24  2016 lost+found
   -rw-r--r--. 1 root root 18 Feb 24  2016 products.html
   -rw-r--r--. 1 root root 17 Feb 24  2016 support.html

5. Inspect the lost+found directory.

   [root@servera ~]# ls -la /var/www/html/lost+found
   total 4
   drwxr-xr-x. 2 root root 16 Feb 24  2016 .
   drwxr-xr-x. 3 root root 79 Feb 24  2016 ..
   -rw-r--r--. 1 root root 14 Feb 24  2016 137

6. Move the orphaned file to index.html in the directory.

   [root@servera ~]# mv /var/www/html/lost+found/137 /var/www/html/index.html

7. Run the curl command against the web server.

   A correct message is displayed.

   [root@servera ~]# curl localhost
   <h1>Home</h1>

8. Return to workstation as the student user.

   [root@servera ~]# exit
   [student@servera ~]$ exit
   [student@workstation ~]$