RH124 - ch11s05

After completing this section, you should be able to find and interpret entries in the system journal to troubleshoot problems or review system status.

Finding Events

The systemd-journald service stores logging data in a structured, indexed binary file called the journal. This data includes extra information about the log event. For example, for syslog events this includes the facility and the priority of the original message.

Important

In Red Hat Enterprise Linux 8, the /run/log directory stores the system journal by default. The contents of the /run/log directory get cleared after a reboot. You can change this setting, and how to do so is discussed later in this chapter.

To retrieve log messages from the journal, use the journalctl command. You can use this command to view all messages in the journal, or to search for specific events based on a wide range of options and criteria. If you run the command as root, you have full access to the journal. Regular users can also use this command, but might be restricted from seeing certain messages.

[root@host ~]# journalctl
...output omitted...
Feb 21 17:46:25 host.lab.example.com systemd[24263]: Stopped target Sockets.
Feb 21 17:46:25 host.lab.example.com systemd[24263]: Closed D-Bus User Message Bus Socket.
Feb 21 17:46:25 host.lab.example.com systemd[24263]: Closed Multimedia System.
Feb 21 17:46:25 host.lab.example.com systemd[24263]: Reached target Shutdown.
Feb 21 17:46:25 host.lab.example.com systemd[24263]: Starting Exit the Session...
Feb 21 17:46:25 host.lab.example.com systemd[24268]: pam_unix(systemd-user:session): session c>
Feb 21 17:46:25 host.lab.example.com systemd[1]: Stopped User Manager for UID 1001.
Feb 21 17:46:25 host.lab.example.com systemd[1]: user-runtime-dir@1001.service: Unit not neede>
Feb 21 17:46:25 host.lab.example.com systemd[1]: Stopping /run/user/1001 mount wrapper...
Feb 21 17:46:25 host.lab.example.com systemd[1]: Removed slice User Slice of UID 1001.
Feb 21 17:46:25 host.lab.example.com systemd[1]: Stopped /run/user/1001 mount wrapper.
Feb 21 17:46:36 host.lab.example.com sshd[24434]: Accepted publickey for root from 172.25.250.>
Feb 21 17:46:37 host.lab.example.com systemd[1]: Started Session 20 of user root.
Feb 21 17:46:37 host.lab.example.com systemd-logind[708]: New session 20 of user root.
Feb 21 17:46:37 host.lab.example.com sshd[24434]: pam_unix(sshd:session): session opened for u>
Feb 21 18:01:01 host.lab.example.com CROND[24468]: (root) CMD (run-parts /etc/cron.hourly)
Feb 21 18:01:01 host.lab.example.com run-parts[24471]: (/etc/cron.hourly) starting 0anacron
Feb 21 18:01:01 host.lab.example.com run-parts[24477]: (/etc/cron.hourly) finished 0anacron
lines 1464-1487/1487 (END) q

The journalctl command highlights important log messages: messages at notice or warning priority are in bold text while messages at the error priority or higher are in red text.

The key to successfully using the journal for troubleshooting and auditing is to limit journal searches to show only relevant output.

By default, journalctl -n shows the last 10 log entries. You can adjust this with an optional argument that specifies how many log entries to display. For the last five log entries, run the following journalctl command:

[root@host ~]# journalctl -n 5
-- Logs begin at Wed 2019-02-20 16:01:17 +07, end at Thu 2019-02-21 18:01:01 +07. --
...output omitted...
Feb 21 17:46:37 host.lab.example.com systemd-logind[708]: New session 20 of user root.
Feb 21 17:46:37 host.lab.example.com sshd[24434]: pam_unix(sshd:session): session opened for u>
Feb 21 18:01:01 host.lab.example.com CROND[24468]: (root) CMD (run-parts /etc/cron.hourly)
Feb 21 18:01:01 host.lab.example.com run-parts[24471]: (/etc/cron.hourly) starting 0anacron
Feb 21 18:01:01 host.lab.example.com run-parts[24477]: (/etc/cron.hourly) finished 0anacron
lines 1-6/6 (END) q

Similar to the tail -f command, the journalctl -f command outputs the last 10 lines of the system journal and continues to output new journal entries as they get written to the journal. To exit the journalctl -f process, use the Ctrl+C key combination.

[root@host ~]# journalctl -f
-- Logs begin at Wed 2019-02-20 16:01:17 +07. --
...output omitted...
Feb 21 18:01:01 host.lab.example.com run-parts[24477]: (/etc/cron.hourly) finished 0anacron
Feb 21 18:22:42 host.lab.example.com sshd[24437]: Received disconnect from 172.25.250.250 port 48710:11: disconnected by user
Feb 21 18:22:42 host.lab.example.com sshd[24437]: Disconnected from user root 172.25.250.250 port 48710
Feb 21 18:22:42 host.lab.example.com sshd[24434]: pam_unix(sshd:session): session closed for user root
Feb 21 18:22:42 host.lab.example.com systemd-logind[708]: Session 20 logged out. Waiting for processes to exit.
Feb 21 18:22:42 host.lab.example.com systemd-logind[708]: Removed session 20.
Feb 21 18:22:43 host.lab.example.com sshd[24499]: Accepted publickey for root from 172.25.250.250 port 48714 ssh2: RSA SHA256:1UGybTe52L2jzEJa1HLVKn9QUCKrTv3ZzxnMJol1Fro
Feb 21 18:22:44 host.lab.example.com systemd-logind[708]: New session 21 of user root.
Feb 21 18:22:44 host.lab.example.com systemd[1]: Started Session 21 of user root.
Feb 21 18:22:44 host.lab.example.com sshd[24499]: pam_unix(sshd:session): session opened for user root by (uid=0)
^C
[root@host ~]#

To help troubleshoot problems, you might want to filter the output of the journal based on the priority of the journal entries. The journalctl -p takes either the name or the number of a priority level and shows the journal entries for entries at that priority and above. The journalctl command understands the debug, info, notice, warning, err, crit, alert, and emerg priority levels.

Run the following journalctl command to list journal entries at the err priority or higher:

[root@host ~]# journalctl -p err
-- Logs begin at Wed 2019-02-20 16:01:17 +07, end at Thu 2019-02-21 18:01:01 +07. --
..output omitted...
Feb 20 16:01:17 host.lab.example.com kernel: Detected CPU family 6 model 13 stepping 3
Feb 20 16:01:17 host.lab.example.com kernel: Warning: Intel Processor - this hardware has not undergone testing by Red Hat and might not be certif>
Feb 20 16:01:20 host.lab.example.com smartd[669]: DEVICESCAN failed: glob(3) aborted matching pattern /dev/discs/disc*
Feb 20 16:01:20 host.lab.example.com smartd[669]: In the system's table of devices NO devices found to scan
lines 1-5/5 (END) q

When looking for specific events, you can limit the output to a specific time frame. The journalctl command has two options to limit the output to a specific time range, the --since and --until options. Both options take a time argument in the format "YYYY-MM-DD hh:mm:ss" (the double-quotes are required to preserve the space in the option). If the date is omitted, the command assumes the current day, and if the time is omitted, the command assumes the whole day starting at 00:00:00. Both options take yesterday, today, and tomorrow as valid arguments in addition to the date and time field.

Run the following journalctl command to list all journal entries from today's records.

[root@host ~]# journalctl --since today
-- Logs begin at Wed 2019-02-20 16:01:17 +07, end at Thu 2019-02-21 18:31:14 +07. --
...output omitted...
Feb 21 18:22:44 host.lab.example.com systemd-logind[708]: New session 21 of user root.
Feb 21 18:22:44 host.lab.example.com systemd[1]: Started Session 21 of user root.
Feb 21 18:22:44 host.lab.example.com sshd[24499]: pam_unix(sshd:session): session opened for user root by (uid=0)
Feb 21 18:31:13 host.lab.example.com systemd[1]: Starting dnf makecache...
Feb 21 18:31:14 host.lab.example.com dnf[24533]: Red Hat Enterprise Linux 8.0 AppStream (dvd)    637 kB/s | 2.8 kB     00:00
Feb 21 18:31:14 host.lab.example.com dnf[24533]: Red Hat Enterprise Linux 8.0 BaseOS (dvd)       795 kB/s | 2.7 kB     00:00
Feb 21 18:31:14 host.lab.example.com dnf[24533]: Metadata cache created.
Feb 21 18:31:14 host.lab.example.com systemd[1]: Started dnf makecache.
lines 533-569/569 (END) q

Run the following journalctl command to list all journal entries ranging from 2019-02-10 20:30:00 to 2019-02-13 12:00:00.

[root@host ~]# journalctl --since "2019-02-10 20:30:00" \
--until "2019-02-13 12:00:00"
...output omitted...

You can also specify all entries since a time relative to the present. For example, to specify all entries in the last hour, you can use the following command:

[root@host ~]# journalctl --since "-1 hour"
...output omitted...

Note

You can use other, more sophisticated time specifications with the --since and --until options. For some examples, see the systemd.time(7) man page.

In addition to the visible content of the journal, there are fields attached to the log entries that can only be seen when verbose output is turned on. Any displayed extra field can be used to filter the output of a journal query. This is useful to reduce the output of complex searches for certain events in the journal.

[root@host ~]# journalctl -o verbose
-- Logs begin at Wed 2019-02-20 16:01:17 +07, end at Thu 2019-02-21 18:31:14 +07. --
...output omitted...
Thu 2019-02-21 18:31:14.509128 +07...
    PRIORITY=6
    _BOOT_ID=4409bbf54680496d94e090de9e4a9e23
    _MACHINE_ID=73ab164e278e48be9bf80e80714a8cd5
    SYSLOG_FACILITY=3
    SYSLOG_IDENTIFIER=systemd
    _UID=0
    _GID=0
    CODE_FILE=../src/core/job.c
    CODE_LINE=826
    CODE_FUNC=job_log_status_message
    JOB_TYPE=start
    JOB_RESULT=done
    MESSAGE_ID=39f53479d3a045ac8e11786248231fbf
    _TRANSPORT=journal
    _PID=1
    _COMM=systemd
    _EXE=/usr/lib/systemd/systemd
    _CMDLINE=/usr/lib/systemd/systemd --switched-root --system --deserialize 18
    _CAP_EFFECTIVE=3fffffffff
    _SELINUX_CONTEXT=system_u:system_r:init_t:s0
    _SYSTEMD_CGROUP=/init.scope
    _SYSTEMD_UNIT=init.scope
    _SYSTEMD_SLICE=-.slice
    UNIT=dnf-makecache.service
    MESSAGE=Started dnf makecache.
    _HOSTNAME=host.lab.example.com
    INVOCATION_ID=d6f90184663f4309835a3e8ab647cb0e
    _SOURCE_REALTIME_TIMESTAMP=1550748674509128
lines 32239-32275/32275 (END) q

The following list gives the common fields of the system journal that can be used to search for lines relevant to a particular process or event.

_COMM is the name of the command
_EXE is the path to the executable for the process
_PID is the PID of the process
_UID is the UID of the user running the process
_SYSTEMD_UNIT is the systemd unit that started the process

More than one of the system journal fields can be combined to form a granular search query with the journalctl command. For example, the following journalctl command shows all journal entries related to the sshd.service systemd unit from a process with PID 1182.

[root@host ~]# journalctl _SYSTEMD_UNIT=sshd.service _PID=1182
Apr 03 19:34:27 host.lab.example.com sshd[1182]: Accepted password for root from ::1 port 52778 ssh2
Apr 03 19:34:28 host.lab.example.com sshd[1182]: pam_unix(sshd:session): session opened for user root by (uid=0)
...output omitted...

Note

For a list of commonly used journal fields, consult the systemd.journal-fields(7) man page.

References

journalctl(1), systemd.journal-fields(7), and systemd.time(7) man pages

For more information refer to the Troubleshooting problems using log files section in the Red Hat Enterprise Linux 8 Configuring basic system settings Guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/index#troubleshooting-problems-using-log-files_getting-started-with-system-administration

Discuss Red Hat System Administration I

Go to community

RH124 Chapter 15 Lab 3: Install zsh Package

VictoriaB97

16 sty 2024

Hello,I am trying to finish up Lab 3 of the Chapter 15 Comprehensive Review Labs in RH124, and when I type in sudo dnf install zsh for the 6th specification of the lab, I get an error message basically saying that it failed to download metadata for the repo 'ansible-2.9-for-rhel-9-x86_64-rpms', and that they could not download repomd.xml. It provided a 404 status code.Can someone please let me know what the problem is, and if there is a current workaround? Thanks.

378

Red Hat System Administration I | Guided Exercise: Explain and Investigate RPM Software Pack

AbeHiero

10 lis 2023

I cannot start the lab with command: lab start software-rpm. I get this error:ModuleNotFoundError: No module named 'rhcsa-classroom'Script software-rpm not in course library rhcsa-classroom Anyone else see this?

172

Red Hat System Administration I (RH124)

Haley_Ruccio

19 lip 2023

The first of two courses covering the core system administration tasks needed to manage Red Hat Enterprise Linux serversRed Hat System Administration I (RH124) is designed for IT professionals without previous Linux system administration experience. The course provides students with Linux administration competence by focusing on core administration tasks. This course also provides a foundation for students who plan to become full-time Linux system administrators by introducing key command-line concepts and enterprise-level tools. This course is the first of a two-course series that takes a computer professional without Linux system administration knowledge to become a fully capable Linux administrator. These concepts are further developed in the follow-on course, Red Hat System Administration II (RH134).This course is based on Red Hat® Enterprise Linux 9.0.Course content summaryIntroduce Linux and the Red Hat Enterprise Linux ecosystem.Run commands and view shell environments.Manage, organize, and secure files.Manage users, groups and user security policies.Control and monitor systemd services.Configure remote access using the web console and SSH.Configure network interfaces and settings.Manage software using DNFAudience for this courseThe primary persona is a technical professional with current or pending responsibilities as a Linux enterprise or cloud system administrator.This course is popular in Red Hat Academy, and is targeted at the student who is planning to become a technical professional using Linux.Prerequisites for this courseBasic technical user skills with computer applications on some operating systems are expected.Take our free assessment to gauge whether this offering is the best fit for your skills.Technology considerationsThere are no special technical requirements for this course.This course is not designed for bring your own device (BYOD).Internet access is not required but is recommended so that students can research and browse Red Hat online resources.

336

Revision: rh124-8.2-df5a585