RH199 - ch11s07

By default, the system journals are kept in the /run/log/journal directory, which means the journals are cleared when the system reboots. You can change the configuration settings of the systemd-journald service in the /etc/systemd/journald.conf file to make the journals persist across reboot.

The Storage parameter in the /etc/systemd/journald.conf file defines whether to store system journals in a volatile manner or persistently across reboot. Set this parameter to persistent, volatile, auto, or none as follows:

persistent: stores journals in the /var/log/journal directory which persists across reboots.
If the /var/log/journal directory does not exist, the systemd-journald service creates it.
volatile: stores journals in the volatile /run/log/journal directory.
As the /run file system is temporary and exists only in the runtime memory, data stored in it, including system journals, do not persist across a reboot.
auto: if the /var/log/journal directory exists, then systemd-journald uses persistent storage, otherwise it uses volatile storage.
This is the default action if the Storage parameter is not set.
none: do not use any storage. All logs are dropped but log forwarding will still work as expected.

The advantage of persistent system journals is that the historic data is available immediately at boot. However, even with a persistent journal, not all data is kept forever. The journal has a built-in log rotation mechanism that triggers monthly. In addition, by default, the journals are not allowed to get larger than 10% of the file system it is on, or leave less than 15% of the file system free. These values can be tuned for both the runtime and persistent journals in /etc/systemd/journald.conf. The current limits on the size of the journal are logged when the systemd-journald process starts. The following command output shows the journal entries that reflect the current size limits:

[user@host ~]$ journalctl | grep -E 'Runtime|System journal'
Feb 25 13:01:46 localhost systemd-journald[147]: Runtime journal (/run/log/journal/ae06db7da89142138408d77efea9229c) is 8.0M, max 91.4M, 83.4M free.
Feb 25 13:01:48 remotehost.lab.example.com systemd-journald[548]: Runtime journal (/run/log/journal/73ab164e278e48be9bf80e80714a8cd5) is 8.0M, max 91.4M, 83.4M free.
Feb 25 13:01:48 remotehost.lab.example.com systemd-journald[548]: System journal (/var/log/journal/73ab164e278e48be9bf80e80714a8cd5) is 8.0M, max 3.7G, 3.7G free.
Feb 25 13:01:48 remotehost.lab.example.com systemd[1]: Starting Tell Plymouth To Write Out Runtime Data...
Feb 25 13:01:48 remotehost.lab.example.com systemd[1]: Started Tell Plymouth To Write Out Runtime Data.

Note

In the grep above, the pipe (|) symbol acts as an or operator. That is, grep matches any line containing either the Runtime string or the System journal string from the journalctl output. This fetches the current size limits on the volatile (Runtime) journal store as well the persistent (System) journal store.

Configuring Persistent System Journals

To configure the systemd-journald service to preserve system journals persistently across reboot, set Storage to persistent in the /etc/systemd/journald.conf file. Run the text editor of your choice as the superuser to edit the /etc/systemd/journald.conf file.

[Journal]
Storage=persistent
...output omitted...

After editing the configuration file, restart the systemd-journald service to bring the configuration changes into effect.

[root@host ~]# systemctl restart systemd-journald

If the systemd-journald service successfully restarts, you can see that the /var/log/journal directory is created and contains one or more subdirectories. These subdirectories have hexadecimal characters in their long names and contain *.journal files. The *.journal files are the binary files that store the structured and indexed journal entries.

[root@host ~]# ls /var/log/journal
73ab164e278e48be9bf80e80714a8cd5
[root@host ~]# ls /var/log/journal/73ab164e278e48be9bf80e80714a8cd5
system.journal  user-1000.journal

While the system journals persist across reboot, you get an extensive number of entries in the output of the journalctl command that includes entries from the current system boot as well as the previous ones. To limit the output to a specific system boot, use the -b option with the journalctl command. The following journalctl command retrieves the entries limited to the first system boot:

[root@host ~]# journalctl -b 1
...output omitted...

The following journalctl command retrieves the entries limited to the second system boot. The following argument is meaningful only if the system has been rebooted at least twice:

[root@host ~]# journalctl -b 2

The following journalctl command retrieves the entries limited to the current system boot:

[root@host ~]# journalctl -b

Note

When debugging a system crash with a persistent journal, it is usually required to limit the journal query to the reboot before the crash happened. The -b option can be accompanied by a negative number indicating how many prior system boots the output should include. For example, journalctl -b -1 limits the output to only the previous boot.

References

systemd-journald.conf(5), systemd-journald(8) man pages

For more information refer to the Troubleshooting problems using log files section in the Red Hat Enterprise Linux 8 Configuring basic system settings Guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/index#troubleshooting-problems-using-log-files_getting-started-with-system-administration

Revision: rh199-8.2-3beeb12