Red Hat Enterprise Linux Diagnostics and Troubleshooting

Successfully solving troubleshooting problems requires not only evaluating current software and hardware behavior, but also discovering previous events that occurred, especially those that relate to the reported failure or issue. Current system behavior is evaluated with diagnostic commands, performance tools, and live monitoring, as discussed in later chapters. Previous events are discovered by viewing the system's journals and log files, as discussed in this section.

Journals and log files contain messages that are received from applications, services, and the kernel. Messages document events, errors, and configuration parameter changes as they occur. The following methods exist to store message information:

The journald service writes binary-structured, memory-based log files, referred to as journals, that are viewed with the journalctl command. The journals are indexed for faster performance, and the binary format is more secure than plain text log files. The systemd-journald service manages the journals, rotating them automatically to limit their storage space and to reduce the need for maintenance.

Messages that journald obtains can be forwarded to rsyslog by using a memory-based socket file in /run/systemd/journal/. Current rsyslog implementations use an integrated module to access messages that the systemd journal generates instead of by monitoring the socket file.

[user@host ~]$ journalctl
-- Logs begin at Tue 2021-09-14 22:51:26 EDT, end at Wed 2021-09-15 02:01:01 EDT. --
Sep 14 22:51:26 localhost kernel: Linux version 4.18.0-305.el8.x86_64 (mockbuild@x86-vm-07.build.eng.bos.redhat.com) >
Sep 14 22:51:26 localhost kernel: Command line: BOOT_IMAGE=(hd0,gpt3)/boot/vmlinuz-4.18.0-305.el8.x86_64 root=/dev/vd>
...output omitted...

[user@host ~]$ journalctl /dev/vda
-- Logs begin at Tue 2021-09-14 22:52:15 EDT, end at Wed 2021-09-15 02:11:18 EDT. --
...output omitted...
Sep 14 22:52:17 localhost kernel: virtio_blk virtio2: [vda] 20971520 512-byte logical blocks (10.7 GB/10.0 GiB)

[user@host ~]$ journalctl -b _SYSTEMD_UNIT=httpd.service
-- Logs begin at Tue 2021-09-14 22:52:15 EDT, end at Wed 2021-09-15 02:14:43 EDT. --
Sep 15 02:02:06 servera.lab.example.com httpd[5874]: Server configured, listening on: port 80

[user@host ~]$ journalctl -b _SYSTEMD_UNIT=httpd.service _PID=5874
-- Logs begin at Tue 2021-09-14 22:52:15 EDT, end at Wed 2021-09-15 02:15:28 EDT. --
Sep 15 02:02:06 servera.lab.example.com httpd[5874]: Server configured, listening on: port 80

[user@host ~]$ journalctl --list-boots
 0 00552ebbabad42c3a439a303138914ee Tue 2021-09-14 22:52:15 EDT—Wed 2021-09-15 02:19:28 EDT
 0 30cd7eb9116d4078a1f11c0fbeac8082 Wed 2021-09-29 00:26:09 EDT-Wed 2021-09-29 00:22:47 EDT

[user@host ~]$ journalctl --boot 00552ebbabad42c3a439a303138914ee _SYSTEMD_UNIT=httpd.service
-- Logs begin at Tue 2021-09-14 22:52:15 EDT, end at Wed 2021-09-15 02:20:28 EDT. --
Sep 15 02:02:06 servera.lab.example.com httpd[5874]: Server configured, listening on: port 80

The rsyslog service acts as a message processing hub. Applications can use system calls to send messages directly to rsyslog. The rsyslog service uses the imjournal input module to retrieve messages continuously from journald, and also forwards messages that it receives directly to journald by using the omjournal output module.

Messages that are sent to rsyslog include the facility and level fields, which are then used for sorting the messages into log files in the /var/log/ directory. The facility value indicates where the message originated, and the level value indicates the severity of the message. These facilities, among some others, send messages to rsyslog:

Some applications maintain their own log files instead of using the rsyslog service. Those applications might need a custom message structure or might process their messages differently from rsyslog or journald. Typically, these log files are in subdirectories of /var/log. For example, the Apache Web Server on a RHEL server saves log messages to files under the /var/log/httpd directory. Samba is another service that uses private log files, with a default file location under the /var/log/samba directory.

Many commands and services can increase the logging detail that they generate while running by including a log level option when they start. Examples of log level options include -v, -vv, -vvv, and --debug, which can be included in their startup configuration files. Refer to the documentation for individual services where you want to increase the verbosity or logging levels.

The audit system security log file, at /var/log/audit/audit.log, contains audit event objects that relate to possible SELinux access denials. Since the file is securely stored in binary, inspect these audit records with a search utility. Use the ausearch tool to query the audit logs for SElinux events.

To resolve SELinux denials, first analyze the root cause as stored in the audit log. Use the sealert command from the policycoreutils-python-utils and setroubleshoot-server packages to help in resolving the problem. SELinux denials might be caused by an incorrect SELinux label, context, Boolean, or port number.

SELinux context issues also occur when a service uses non-standard directories. For example, use of a web server with the non-standard /home/user/myweb directory requires that directory to be set with the correct SELinux label.

The semanage fcontext command stores a specified SELinux context with the directory location in the SELinux database.

[root@host ~]# semanage fcontext -a -t httpd_sys_content_t "/home/user/myweb"

The restorecon command looks up a specified directory in the database, retrieves the configured context for that directory, and applies the context on the directory.

Use the following command to apply the context change:

[root@host ~]# restorecon -R -v /home/user/myweb

SELinux Booleans allow exceptions to runtime policy that can be enabled or disabled without restarting a service. In this example, enabling this Boolean enables access to the home directories through a local web server, which is normally a policy behavior that is allowed to web servers.

[root@host ~]# setsebool httpd_enable_homedirs on

Many services are allowed by policy to run only on specific port numbers. The semanage port command allows the service to operate through a non-standard port. In this example, the default port of a web service is changed to use port 9876.

[root@host ~]# semanage port -a -t http_port_t -p tcp 9876