Bookmark this page

Chapter 3.  Analyze and Store Logs

Abstract

Goal

Locate and accurately interpret system event logs for troubleshooting purposes.

Objectives
  • Describe the basic Red Hat Enterprise Linux logging architecture to record events.

  • Interpret events in the relevant syslog files to troubleshoot problems or to review system status.

  • Find and interpret entries in the system journal to troubleshoot problems or review system status.

  • Configure the system journal to preserve the record of events when a server is rebooted.

  • Maintain accurate time synchronization with Network Time Protocol (NTP) and configure the time zone to ensure correct time stamps for events that are recorded by the system journal and logs.

Sections
  • Describe System Log Architecture (and Quiz)

  • Review Syslog Files (and Guided Exercise)

  • Review System Journal Entries (and Guided Exercise)

  • Preserve the System Journal (and Guided Exercise)

  • Maintain Accurate Time (and Guided Exercise)

Lab
  • Analyze and Store Logs

Describe System Log Architecture

Objectives

  • Describe the basic Red Hat Enterprise Linux logging architecture to record events.

System Logging

The operating system kernel and other processes record a log of events that happen when the system is running. These logs are used to audit the system and to troubleshoot problems. You can use text utilities such as the less and tail commands to inspect these logs.

Red Hat Enterprise Linux uses a standard logging system that is based on the syslog protocol to log the system messages. Many programs use the logging system to record events and to organize them into log files. The systemd-journald and rsyslog services handle the syslog messages in Red Hat Enterprise Linux 9.

The systemd-journald service is at the heart of the operating system event logging architecture. The systemd-journald service collects event messages from many sources:

  • System kernel

  • Output from the early stages of the boot process

  • Standard output and standard error from daemons

  • Syslog events

The systemd-journald service restructures the logs into a standard format and writes them into a structured, indexed system journal. By default, this journal is stored on a file system that does not persist across reboots.

The rsyslog service reads syslog messages that the systemd-journald service receives from the journal when they arrive. The rsyslog service then processes the syslog events, and records them to its log files or forwards them to other services according to its own configuration.

The rsyslog service sorts and writes syslog messages to the log files that do persist across reboots in the /var/log directory. The service also sorts the log messages to specific log files according to the type of program that sent each message and the priority of each syslog message.

In addition to syslog message files, the /var/log directory contains log files from other services on the system. The following table lists some useful files in the /var/log directory.

Table 3.1. Selected System Log Files

Log fileType of stored messages
/var/log/messages Most syslog messages are logged here. Exceptions include messages about authentication and email processing, scheduled job execution, and purely debugging-related messages.
/var/log/secure Syslog messages about security and authentication events.
/var/log/maillog Syslog messages about the mail server.
/var/log/cron Syslog messages about scheduled job execution.
/var/log/boot.log Non-syslog console messages about system startup.

Some applications do not use the syslog service to manage their log messages. For example, the Apache Web Server saves log messages to files in a subdirectory of the /var/log directory.

References

systemd-journald.service(8), rsyslogd(8), and rsyslog.conf(5) man pages

For more information, refer to the Troubleshooting Problems Using Log Files section in the Red Hat Enterprise Linux 9 Configuring Basic System Settings guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/configuring_basic_system_settings/index

Revision: rh134-9.3-5fd2368