Bookmark this page

Guided Exercise: Investigating File-system Changes with AIDE

Use AIDE and Audit to detect changes on a file system, and use Audit tools to identify the cause of those changes.

Outcomes

  • Monitor the /etc/passwd and /etc/shadow files with AIDE.

  • Set Linux Audit rules to monitor the files.

  • Change the files, and then use Audit tools to identify which user and process made the changes.

As the student user on the workstation machine, use the lab command to prepare your environment for this exercise, and to ensure that all required resources are available.

[student@workstation ~]$ lab start aide-investigating

Instructions

  1. From the workstation machine, log in to the servera machine as the student user.

    [student@workstation ~]$ ssh student@servera
    ...output omitted...
    [student@servera ~]$
  2. Change to the root user. Use student as the password.

    [student@servera ~]$ sudo -i
    [sudo] password for student: student
    [root@servera ~]#
  3. Edit the /etc/aide.conf file to direct AIDE to use the CONTENT_EX group definition for monitoring for any changes to the /etc/shadow and /etc/passwd files. Comment out the other selection lines in the configuration file. Usually, you would not modify these lines from your /etc/aide.conf file. The other selection lines are commented out to speed up AIDE scans for the purpose of this exercise.

    ...output omitted...
    # Extended content + file type + access.
    CONTENT_EX = sha512+ftype+p+u+g+n+acl+selinux+xattrs
    
    # Some files get updated automatically, so the inode/ctime/mtime change
    # but we want to know when the data inside them changes
    DATAONLY =  p+n+u+g+s+acl+selinux+xattrs+sha512
    
    # Next decide what directories/files you want in the database
    
    #/boot       CONTENT_EX
    #/opt        CONTENT
    
    /etc/passwd CONTENT_EX
    /etc/shadow CONTENT_EX
    ...output omitted...
  4. Initialize the AIDE database with the results of a baseline scan.

    [root@servera ~]# aide --init
    Start timestamp: 2023-10-16 01:36:16 -0400 (AIDE 0.16)
    AIDE initialized database at /var/lib/aide/aide.db.new.gz
    
    Number of entries:	52938
    
    ---------------------------------------------------
    The attributes of the (uncompressed) database(s):
    ---------------------------------------------------
    
    /var/lib/aide/aide.db.new.gz
      MD5      : Srt/ufvcsTXSVth6usbGXQ==
      SHA1     : FPUaTNXkq5x72FlK2FwXZOF417k=
      RMD160   : DV3dBmQyHM+eo99C+XHEeCWSCgA=
      TIGER    : 6A1ky0dKHLmjGlL7LA6HcRTKSiNksyvN
      SHA256   : NKZfQFk8b5mxL3PbsLrpq4CLFsuN+6C1
                 wi7LpMEkzBA=
      SHA512   : +7YUEfIp7mgWeDdh1ItQE1B+fHkBicAJ
                 asJ4VbX5jCIriPwqvcPy3f9QwZiU8ULS
                 e2lj+XiIVTqVE4p3UwscmQ==
    
    
    End timestamp: 2023-10-16 01:36:52 -0400 (run time: 0m 36s)
  5. Rename the new AIDE database file from /var/lib/aide/aide.db.new.gz to /var/lib/aide/aide.db.gz so that AIDE uses the newly generated file as the current database.

    [root@servera ~]# mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
  6. Determine the current status of the machine's file systems.

    AIDE reports no changes to the files and directories that it monitors, because you have not changed any files or directories after initializing the AIDE database.

    [root@servera ~]# aide --check
    Start timestamp: 2023-10-16 01:38:48 -0400 (AIDE 0.16)
    AIDE found NO differences between database and filesystem. Looks okay!!
    
    Number of entries:	52938
    
    ---------------------------------------------------
    The attributes of the (uncompressed) database(s):
    ---------------------------------------------------
    
    /var/lib/aide/aide.db.gz
      MD5      : Srt/ufvcsTXSVth6usbGXQ==
      SHA1     : FPUaTNXkq5x72FlK2FwXZOF417k=
      RMD160   : DV3dBmQyHM+eo99C+XHEeCWSCgA=
      TIGER    : 6A1ky0dKHLmjGlL7LA6HcRTKSiNksyvN
      SHA256   : NKZfQFk8b5mxL3PbsLrpq4CLFsuN+6C1
                 wi7LpMEkzBA=
      SHA512   : +7YUEfIp7mgWeDdh1ItQE1B+fHkBicAJ
                 asJ4VbX5jCIriPwqvcPy3f9QwZiU8ULS
                 e2lj+XiIVTqVE4p3UwscmQ==
    
    
    End timestamp: 2023-10-16 01:39:20 -0400 (run time: 0m 32s)
  7. AIDE reports files that have changed, but not what or who changed them. You can add Audit rules to monitor your files so that when AIDE reports a change, you can look in the Audit log to determine who or what might have caused that change.

    Add a persistent Audit watch rule to generate Audit log entries whenever there is an attempt to read, write, execute, or change an attribute of the /etc/shadow file. Use group_watch as the filter key on the Audit rule.

    [root@servera ~]# echo "-w /etc/shadow -p rwxa -k group_watch" \
        >> /etc/audit/rules.d/audit.rules
  8. Apply the changes for the newly added Audit rules to take effect.

    [root@servera ~]# augenrules --load
    ...output omitted...
  9. List the Audit rules and verify that the newly added Audit rule is currently in effect.

    [root@servera ~]# auditctl -l
    -w /etc/shadow -p rwxa -k group_watch
  10. Change the password from student to redhat for the student user on the servera machine. This update causes a change in the /etc/shadow file.

    [root@servera ~]# passwd student
    Changing password for user student.
    New password: redhat
    BAD PASSWORD: The password is shorter than 8 characters
    Retype new password: redhat
    passwd: all authentication tokens updated successfully.
  11. Check the status of the machine's file systems that are monitored by AIDE.

    The output of the command should report that the SHA512 checksum (and therefore the content) of the /etc/shadow file has changed. Use the ausearch command to view the activity that caused the change in the /etc/shadow file.

    The values of the SHA512 checksums that you see in the following output might differ from what you see in your system.

    [root@servera ~]# aide --check
    Start timestamp: 2023-10-16 01:49:28 -0400 (AIDE 0.16)
    AIDE found differences between database and filesystem!!
    
    Summary:
      Total number of entries:	52939
      Added entries:		1
      Removed entries:		0
      Changed entries:		3
    ...output omitted...
    ---------------------------------------------------
    Changed entries:
    ---------------------------------------------------
    
    f   ...    .C... : /etc/audit/audit.rules
    f   ...    .C... : /etc/audit/rules.d/audit.rules
    f   ...    .C... : /etc/shadow
    
    ---------------------------------------------------
    Detailed information about changes:
    ---------------------------------------------------
    
    ...output omitted...
    
    File: /etc/shadow
      SHA512   : azDXLAUX+i9BVvvyk5zzJx+VmnQ/fDJO | 00YQkoRwN2epjEA8Il9df4SAZhfpe2rM
                 6BXU+zLxtROQ6JylzE0t/8XoVE3fVw3u | KZoIaSDEvVpplAYuHVuWywvcnx0tipU/
                 yXSxhBpX+2OKwCXgVX9x8A==         | rHXbHn8S14hLtZuF+CYXCw==
    
    ...output omitted...
  12. Use the ausearch command to search the Audit log for further investigation of the changes to the /etc/shadow file that AIDE reported. Use the group_watch key that is used by your watch rule to filter for the events that are relevant to the /etc/shadow file. Use the ausearch command with its -i option to convert numbers for UIDs and time stamps into names and more human-readable text.

    The values of the timestamp and the inode that you see in the following output might differ from what you see in your system.

    [root@servera ~]# ausearch -i -f /etc/shadow -k group_watch
    ...output omitted...
    type=PROCTITLE msg=audit(10/16/23 01:48:26.698:354) : proctitle=passwd student
    type=PATH msg=audit(10/16/23 01:48:26.698:354) : item=0 name=/etc/shadow inode=8391344 dev=fc:04 mode=file,000 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:shadow_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
    type=CWD msg=audit(10/16/23 01:48:26.698:354) : cwd=/root
    type=SYSCALL msg=audit(10/16/23 01:48:26.698:354) : arch=x86_64 syscall=openat success=yes exit=4 a0=AT_FDCWD a1=0x7f842c67c12d a2=O_RDONLY a3=0x0 items=1 ppid=27023 pid=27126 auid=student uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=9 comm=passwd exe=/usr/bin/passwd subj=unconfined_u:unconfined_r:passwd_t:s0-s0:c0.c1023 key=group_watch
    ...output omitted...

    There are several events recorded by the Audit log. Each event is separated by four dashes in the output, and the event number should appear in the output after the final colon in the time stamp (the msg field in each Audit record). This is Audit event 354.

    This event consists of four records: PROCTITLE, PATH, CWD, and SYSCALL.

    • The PROCTITLE record shows the passwd student command that ran on the /etc/shadow file shown in the PATH record.

    • The PATH record shows that the /etc/shadow file was changed and provides information about the file such as its inode number, the major and minor numbers of the device its file system is on, its ownership and permissions, that it is a regular file, and more information.

    • The CWD record gives the current working directory that shows the command was run from the /root directory.

    • The SYSCALL record indicate that this event is logging an attempt to open the /etc/shadow file in read-only mode as shown in the a2=O_RDONLY item representing the third argument to the system call (counting from zero). The access was successful (success=yes). The user was on the pts/0 pseudoterminal (possibly a graphical terminal window or ssh login) at the time. The uid and euid fields indicate that the process was running as the root user. However, the auid field indicates that the user that ran this command originally logged in as the student user.

      This event is not the Audit event that caused the change to the /etc/shadow file, because in this event the passwd command opens the file as read-only. However, this event might be part of the sequence of Audit events that led to the change. You need to look further at the output to find the correct event.

  13. Further down in the output of the ausearch command, another event occurred at almost the same time, which looks similar to the following output.

    [root@servera ~]# ausearch -i -f /etc/shadow -k group_watch
    ...output omitted...
    type=PROCTITLE msg=audit(10/16/23 01:48:26.725:358) : proctitle=passwd student
    type=PATH msg=audit(10/16/23 01:48:26.725:358) : item=4 name=/etc/shadow inode=8393250 dev=fc:04 mode=file,000 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:shadow_t:s0 nametype=CREATE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
    type=PATH msg=audit(10/16/23 01:48:26.725:358) : item=3 name=/etc/shadow inode=8391344 dev=fc:04 mode=file,000 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:shadow_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
    type=PATH msg=audit(10/16/23 01:48:26.725:358) : item=2 name=/etc/nshadow inode=8393250 dev=fc:04 mode=file,000 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:shadow_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
    type=PATH msg=audit(10/16/23 01:48:26.725:358) : item=1 name=/etc/ inode=8388736 dev=fc:04 mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
    type=PATH msg=audit(10/16/23 01:48:26.725:358) : item=0 name=/etc/ inode=8388736 dev=fc:04 mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
    type=CWD msg=audit(10/16/23 01:48:26.725:358) : cwd=/root
    type=SYSCALL msg=audit(10/16/23 01:48:26.725:358) : arch=x86_64 syscall=rename success=yes exit=0 a0=0x7f842c67c38e a1=0x7f842c67c12d a2=0x558ebaa4ed53 a3=0x7 items=5 ppid=27023 pid=27126 auid=student uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=9 comm=passwd exe=/usr/bin/passwd subj=unconfined_u:unconfined_r:passwd_t:s0-s0:c0.c1023 key=group_watch
    ...output omitted...

    The PROCTITLE record indicates that the passwd student command was run.

    The PATH records indicate that the /etc/shadow file that had the 8391344 inode was deleted (item 3). In addition, the 8393250 inode was renamed from the /etc/nshadow file to the /etc/shadow file (items 2 and 4).

    The CWD record gives the current working directory, and shows that the command was run from the /root directory.

    The SYSCALL record indicates that a rename system call was successfully executed. The passwd command was executed by the root user, who originally logged in as the student user, by running the /usr/bin/passwd command on the pts/0 terminal. The passwd command creates the /etc/nshadow temporary file that contains the changes, and then replaces the previous shadow password file with the temporary file. This is the event that changed the /etc/shadow file.

  14. Return to the workstation machine as the student user.

    [root@servera ~]# logout
    [student@servera ~]$ logout
    Connection to servera closed.
    [student@workstation ~]$

Finish

On the workstation machine, use the lab command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises.

[student@workstation ~]$ lab finish aide-investigating

Revision: rh415-9.2-a821299