Bookmark this page

Guided Exercise: Acting on System Journal Events

Configure and test an Ansible Rulebook that triggers when it receives an event from the local system journal.

Outcomes

  • Create and test a rulebook that uses the ansible.eda.journald event source plug-in.

  • Create a rule that uses a variable that is defined in a file.

As the student user on the workstation machine, use the lab command to prepare your environment for this exercise, and to ensure that all required resources are available:

[student@workstation ~]$ lab start intro-journald

Instructions

In this activity, Event-Driven Ansible receives events from the local system journal on the machine running the rule engine.

Note

The ansible.eda.journald event source is a useful example for learning purposes because it does not require any special configuration and is straightforward to use.

However, it has some practical limitations because it only reads the system journal from the machine running the rule engine and rulebook. If you want to monitor remote journals, it might be easier to have a log aggregation server pass relevant events to Event-Driven Ansible by using a different event source from the ansible.eda collection, such as alertmanager, kafka, or webhook.

  1. Examine the system journal.

    1. From the workstation machine, use the journalctl command with the _COMM=useradd match argument to filter for log messages related to added users:

      [student@workstation ~]$ sudo journalctl _COMM=useradd
      [sudo] password for student: student
      ...output omitted...
      Apr 05 15:39:43 workstation useradd[8349]: new group: name=new_user, GID=1003
      Apr 05 15:39:43 workstation useradd[8349]: new user: name=new_user, UID=1003, GID=1003, home=/home/>
      Apr 05 15:39:43 workstation useradd[8349]: add 'new_user' to group 'adm'
      Apr 05 15:39:43 workstation useradd[8349]: add 'new_user' to group 'systemd-journal'
      Apr 05 15:39:43 workstation useradd[8349]: add 'new_user' to shadow group 'adm'
      Apr 05 15:39:43 workstation useradd[8349]: add 'new_user' to shadow group 'systemd-journal'
      ...output omitted...

      Although your output is slightly different, you should see that a useradd command (process ID 8349 in the example output) resulted in six separate log messages. Both a user and a group named new_user was created, and then the user was added to supplementary groups.

    2. Press q to exit the journalctl command.

    3. Display additional details about the added user. Use the journalctl command with the -o verbose and _PID options. Replace the process ID in the following command with the process ID displayed in your previous command (such as 8349). The order of the output lines in your command might be different from the following:

      [student@workstation ~]$ sudo journalctl -o verbose _PID=8349
      Mon 2024-04-08 17:54:02.380124 UTC [s=0ba832d396704481a3a541ae085ecb37;i=a6a;...
          _TRANSPORT=syslog
          PRIORITY=6 1
          SYSLOG_FACILITY=10 2
          _GID=0
          _CAP_EFFECTIVE=1ffffffffff
          _SELINUX_CONTEXT=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
          _BOOT_ID=cb4bc4cd42154e2e9cd2b859295b8eee
          _MACHINE_ID=0acf3cdf85734e43b4d2e1f5cad9b757
          _HOSTNAME=workstation
          _RUNTIME_SCOPE=system
          _UID=0 3
          _SYSTEMD_USER_SLICE=-.slice
          _AUDIT_LOGINUID=1001
          _SYSTEMD_OWNER_UID=1001
          _SYSTEMD_SLICE=user-1001.slice
          _AUDIT_SESSION=3
          _SYSTEMD_CGROUP=/user.slice/user-1001.slice/session-3.scope
          _SYSTEMD_SESSION=3
          _SYSTEMD_UNIT=session-3.scope
          _SYSTEMD_INVOCATION_ID=7ac4ab32f34e4cfc906129440a3caebe
          SYSLOG_TIMESTAMP=Apr  8 17:54:02
          SYSLOG_IDENTIFIER=useradd 4
          SYSLOG_PID=8349
          MESSAGE=new group: name=new_user, GID=1003 5
          _PID=8349
          _COMM=useradd 6
          _EXE=/usr/sbin/useradd
          _CMDLINE=/sbin/useradd -G systemd-journal adm -m new_user 7
          _SOURCE_REALTIME_TIMESTAMP=1712598842380124
      ...output omitted...

      1

      Log events with a priority value of 6 are for informational messages.

      2

      Log events with a syslog facility value of 10 are for security and authorization messages.

      3

      The command that generated the log event was effectively performed as the root user.

      4

      Log events related to the useradd command use the useradd identifier.

      5

      The actual message for this log event indicates that the new_user group was created.

      6

      The values for the SYSLOG_IDENTIFIER and _COMM keys are often the same.

      7

      This line contains the complete command that triggered the log event.

      Important

      When used in rule conditions, the ansible.eda.journald event source plug-in converts the journal key names to lowercase. For example, if you want to check the value of the SYSLOG_IDENTIFIER journal key, then your condition must check the value of the event.journald.syslog_identifier event key.

    4. Press q to exit the journalctl command.

  2. Update the /home/student/intro-journald/user_events.yml rulebook so that it restricts the number of events that it receives and the existing rule triggers when a log entry matches the useradd command.

    1. Change to the /home/student/intro-journald project directory:

      [student@workstation ~]$ cd ~/intro-journald
      [student@workstation intro-journald]$
    2. The lab command installed the ansible.eda collection. List the installed collections available to the /home/student/intro-journald directory:

      [student@workstation intro-journald]$ ansible-galaxy collection list
      
      # /home/student/intro-journald/collections/ansible_collections
      Collection               Version
      ------------------------ -------
      ansible.eda              1.4.5
      ...output omitted...
    3. Use a text editor, such as VS Code, to update the user_events.yml rulebook. Change the match parameter for the ansible.eda.journald source. Rather than checking all journal events, your updated rulebook only checks events related to security and authorization.

      ---
      - name: Display system journal events
        hosts: all
      
        sources:
          - name: Watch the system journal
            ansible.eda.journald:
              match: "SYSLOG_FACILITY=10"
      ...output omitted...
    4. Update the user_events.yml rulebook to add a two second delay for the ansible.eda.journald source. Because some commands can generate more than one log message, adding a delay can ensure that the event source plug-in receives all messages.

      ---
      - name: Display system journal events
        hosts: all
      
        sources:
          - name: Watch the system journal
            ansible.eda.journald:
              match: "SYSLOG_FACILITY=10"
              delay: 2
      ...output omitted...
    5. Update the user_events.yml rulebook so that the rule triggers when the event matches the useradd command. The rule uses the debug action to display the journal message associated with the event.

      The complete rulebook contains the following content:

      ---
      - name: Display system journal events
        hosts: all
      
        sources:
          - name: Watch the system journal
            ansible.eda.journald:
              match: "SYSLOG_FACILITY=10"
              delay: 2
      
        rules:
          - name: Display event information
            condition: event.journald._comm == "useradd"
            action:
              debug:
                var: event.journald.message

      Note

      Although this exercise displays the event journal message in the terminal window, if you were writing your own rulebook you might do something else, such as send the event information as an email.

  3. Test the user_events.yml rulebook to verify that the rule triggers on the expected condition.

    1. Use the ansible-rulebook command to run the user_events.yml rulebook. Specify the inventory file with the -i option:

      [student@workstation intro-journald]$ ansible-rulebook -r user_events.yml \
      -i inventory
    2. Open a second terminal window and create the admin_test user:

      [student@workstation ~]$ sudo useradd admin_test
      [sudo] password for student: student
    3. Return to the first terminal window. Although your output is slightly different, you should see the following two log events after a few seconds:

      ** 2024-03-12 16:05:34.319492 [debug] ******************************************
      event.journald.message: new group: name=admin_test, GID=1004
      *******************************************************************************
      
      ** 2024-03-12 16:05:36.321435 [debug] *****************************************
      event.journald.message: new user: name=admin_test, UID=1004, GID=1004, home=/home/admin_test, shell=/bin/bash, from=/dev/pts/1
      *******************************************************************************

      Note

      The single useradd command generated two log events.

      • The first event message indicates that the command created the admin_test group.

      • The second event message indicates that the command created the admin_test user (whose primary group ID matches the group ID listed in the first message).

    4. Press Ctrl+C in the first terminal to stop the running rulebook.

  4. Update the user_events.yml rulebook so that it triggers on commands defined in the vars/identity_commands.yml variable file.

    1. Examine the contents of the provided vars/identity_commands.yml variable file. This file contains commands related to adding, modifying, and deleting users and groups.

      ---
      identity_commands:
        - useradd
        - usermod
        - userdel
        - groupadd
        - groupmod
        - groupdel
        - passwd
    2. Update the user_events.yml rulebook so that the rule triggers when the event matches any of the commands defined in the identity_commands variable. Update the condition line to match the following:

      ...output omitted...
        rules:
          - name: Display event information
            condition: event.journald._comm in vars.identity_commands
      ...output omitted...
  5. Test the user_events.yml rulebook to verify that the rule triggers on the expected condition.

    1. Use the ansible-rulebook command to run the user_events.yml rulebook. Specify the inventory file with the -i option and include the variable file using the -e option:

      [student@workstation intro-journald]$ ansible-rulebook -r user_events.yml \
      -i inventory -e vars/identity_commands.yml
    2. In the second terminal window, delete the previously created admin_test user:

      [student@workstation ~]$ sudo userdel -r admin_test
      [sudo] password for student: student
    3. Return to the first terminal window. Although your output is slightly different, you should see the following log events after a few seconds:

      ** 2024-03-12 16:08:20.207531 [debug] *****************************************
      event.journald.message: delete user 'admin_test'
      *******************************************************************************
      
      ** 2024-03-12 16:08:22.188933 [debug] *****************************************
      event.journald.message: removed group 'admin_test' owned by 'admin_test'
      *******************************************************************************
      
      ** 2024-03-12 16:08:24.191235 [debug] *****************************************
      event.journald.message: removed shadow group 'admin_test' owned by 'admin_test'
      *******************************************************************************

      Note

      To visually separate the output generated by different events, you can press Enter a few times.

    4. In the second terminal window, install the httpd package:

      [student@workstation ~]$ sudo dnf install httpd
      ...output omitted...
      Is this ok [y/N]: y
      ...output omitted...
    5. Return to the first terminal window. Although your output is slightly different, you should see the following log events after a few seconds.

      Installing the httpd package created the apache group and the apache user:

      ** 2024-03-12 16:10:45.759367 [debug] *****************************************
      event.journald.message: group added to /etc/group: name=apache, GID=48
      *******************************************************************************
      
      ** 2024-03-12 16:10:47.759260 [debug] *****************************************
      event.journald.message: group added to /etc/gshadow: name=apache
      *******************************************************************************
      
      ** 2024-03-12 16:10:49.762141 [debug] *****************************************
      event.journald.message: new group: name=apache, GID=48
      *******************************************************************************
      
      ** 2024-03-12 16:10:51.764876 [debug] *****************************************
      event.journald.message: new user: name=apache, UID=48, GID=48, home=/usr/share/httpd, shell=/sbin/nologin, from=none
      *******************************************************************************

      You have demonstrated that your rulebook functions correctly for two of the commands defined by the identity_commands variable.

    6. Close the second terminal window.

    7. Press Ctrl+C in the first terminal to stop the running rulebook.

Finish

On the workstation machine, use the lab command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises.

[student@workstation ~]$ lab finish intro-journald

Revision: do274-2.4-65daa25