Red Hat Enterprise Linux Diagnostics and Troubleshooting

Implementing a central log host requires configuring the rsyslog service on the systems where the log messages originate and on the central log host that receives the messages. On the central log host, the rsyslog service is configured to accept log messages from the originating hosts.

To configure the rsyslog service on the central log host to accept remote logs, uncomment either the TCP or UDP reception lines in the modules section in the /etc/rsyslog.conf file.

For UDP reception:

module(load="imudp")
input(type="imudp" port="514")

or for TCP reception:

module(load="imtcp")
input(type="imtcp" port="514")

In the example syntax, the module object loads the input modules for TCP and UDP, and the input object configures them. Although TCP provides more reliable delivery of remote log messages, UDP is supported by more operating systems and networking devices.

The default rules in /etc/rsyslog.conf accommodate message logging on a single host, sorting and bundling messages by facility. For example, mail messages are stored in /var/log/maillog while messages generated by the crond daemon are stored in /var/log/cron to easily locate each type of message.

Although sorting of messages by facility is ideal on a single host, it produces an undesirable result on a central log host because it causes messages from different remote hosts to be mixed together. On a central log host, it might be optimal for log messages from remote systems to remain separated. This separation is achieved by using the template function of rsyslog.

Templates are defined in /etc/rsyslog.conf by using the template() statement and are used to generate dynamic log files. A template() statement must contain a name and a type parameter. Different template types use different syntax, but still have the same functionality. Template parameters depend on the template type. Templates can dynamically construct log file names and paths based the properties of a log message.

For example, to route cron syslog messages from different systems to different files on a central log host, use a template to generate dynamic log directories based on the hostname property of each log message, as in this example:

template (name="HostTemplate" type="string" string="/var/log/loghost/%HOSTNAME%/cron.log")

Alternatively, use the list type parameter to specify a consistent and preferred syntax:

template(name="HostTemplate" type="list") {
    constant(value="/var/log/loghost/")
    property(name="hostname")
    constant(value="/cron.log")
    }

A template can be referenced by name in a rule:

cron.* action(type="omfile" DynaFile="HostTemplate")

This example rule indicates that log messages of the cron facility type, of any severity level, should be handled by the action object. The action object uses the File Output Module, omfile, to write to the template that the DynaFile parameter indicates.

The following template example generates dynamic log file names. Remote log messages are sorted by originating host name and facility values by referencing the hostname and syslogfacility-text properties. Log messages are written to the dynamically generated log file names.

template (name="HostTemplate" type="string" string="/var/log/loghost/%HOSTNAME%/%syslogfacility-text%.log")
*.* action(type="omfile" DynaFile="HostTemplate")

After activating syslog message reception and creating the rules for log separation by host, restart the rsyslog service to implement the changes. In addition, configure appropriate UDP or TCP firewall rules to permit incoming syslog traffic, and reload firewalld to implement those rules.

[root@loghost ~]# systemctl restart rsyslog
[root@loghost ~]# firewall-cmd --add-port=514/udp --permanent
[root@loghost ~]# firewall-cmd --add-port=514/tcp --permanent
[root@loghost ~]# firewall-cmd --reload

Newly created log files might not be included in the log host's current log rotation schedule, which can cause those new logs to grow too large. Include the new log files in the log rotation by adding a new entry to the list of log files in the /etc/logrotate.d/syslog configuration file.

/var/log/loghost/*/*.log

After configuring the central log host to accept incoming remote messages, configure the rsyslog service on originating systems to send logs remotely to the central log host. Add entries to the rules section in the /etc/rsyslog.conf file to indicate the facility, severity, and an action object to specify how to communicate with the central log host.

For example, to send messages with info or higher severity that are sent to loghost.example.com via UDP, use this entry:

*.info action(type="omfwd" target="loghost.example.com" port="514" protocol="udp")

The action object uses the Forwarding Output Module, omfwd, to send syslog messages to the central log host.

To send all messages to loghost.example.com via TCP, use the following entry:

*.* action(type="omfwd" target="loghost.example.com" port="514" protocol="tcp")

After adding rules, restart the rsyslog service. Use the logger command to send a test message, which can optionally specify the facility and severity to test.

[root@logclient ~]# logger "Test from logclient"

Check the logs on the remote server to ensure that the message was received.

Sessions that are recorded with the web console tool can be exported to a central log host for centralization. Use the systemd export option to forward encrypted journal entries over the network with HTTP/HTTPS and to write journal files from the serialized journal content.

This option is integrated into systemd, by installing the systemd-journal-remote package.

[root@host ~]#  yum install systemd-journal-remote

Create a temporary directory where the journal output with all its entities are exported into a file.

[root@host ~]#  mkdir /tmp/dir
[root@host ~]#  journalctl -o export | /usr/lib/systemd/systemd-journal-remote -o /tmp/dir/example.journal -

In the central log host, create a directory to store the files of the nodes with terminal session recordings and copy or rsync them securely to the directory.

[root@host ~]#  mkdir /var/log/journal/remote/

To playback the recorded sessions, use the tool included in the tlog package.

[root@host ~]#  tlog-play -r journal -M TLOG_REC=<your-unique-host-id>

Automate the installation of the terminal session recording tool with the RHEL tlog system role. Sessions are sent to the central log host for further analysis.