Red Hat Security: Linux in Physical, Virtual, and Cloud
Investigate causes of file-system changes that are reported by AIDE by using Linux Audit tools.
Advanced Intrusion Detection Environment (AIDE) detects changes on a computer's file systems, and reports to administrators about specific changes. However, one limitation of AIDE is that it does not tell you what caused the changes.
For example, you can configure AIDE to report changes to the /etc/passwd and /etc/shadow files, but AIDE does not provide details such as who made the changes, at what time, or which tool they used.
This is where Audit rules can help you to get the details that you need about what happened.
The Linux Audit system monitors files for activity that might cause a change, and it logs useful information about the time of the activity, and the process and user that are involved. The Audit system logs these changes, but it is not straightforward to set up Audit to report on key events by itself.
You can solve this problem by combining AIDE and Audit. By configuring both systems to monitor key files and directories, you can use AIDE to report periodically on unexpected changes in your systems. Then you can review the Audit log from the last AIDE report before the change to help identify what activity might be responsible.
Note
One limitation of this approach is that AIDE does not tell you in real time that there has been a change. It only detects changes when a check is run.
If you are monitoring for changes to the contents of files, then an AIDE check can be expensive in terms of system resources. Setting the frequency of AIDE checks too high can increase the load on the system in CPU time and disk I/O. Likewise, large amounts of Audit logging, especially by using system call rules, can have an impact on system performance.
For this approach to be effective, you must configure both AIDE and Audit to monitor the same files. Audit records activity that affects a file; AIDE detects and reports that there has been a change to a file. As a result, you must set up the AIDE database and watch rules on files in advance to collect the relevant information for detection and investigation.
Start by making sure that AIDE is properly configured.
Edit the /etc/aide.conf file to configure AIDE to monitor the correct files, initialize the AIDE database, and set up the cron job to periodically run AIDE checks and to send the report to the system administrators.
The following example shows the configuration of the /etc/aide.conf file to monitor the changes of all the files in the /usr directory and its subdirectories.
The default /etc/aide.conf file settings set a regular selection line to monitor the /usr directory for changes to content, permissions, and ownership:
CONTENT_EX = sha256+ftype+p+u+g+n+acl+selinux+xattrs
...output omitted...
/usr/ CONTENT_EXYou must also set an Audit rule to monitor the /usr directory recursively for attempts to write files or change permissions.
One method is to set a file-system rule in a .rules file in the /etc/audit/rules.d directory.
The following example shows a minimal rule to set a watch for changes to the /usr directory:
-w /usr -p wa -k usr-modification
Important
If you do not need to collect information for auditing purposes, you should probably avoid doing so. This reduces extraneous information in the logs, reduces the size of the logs, and might improve performance, depending on the structure of your rules.
In the previous watch example, the -p rwxa option is not set.
This option creates a log entry every time something opens a file in the /usr directory to read or execute it.
Opening a file in the usr directory occurs often and the information is less relevant for this analysis.
Additional information could make your analysis simpler, and you might care about that information for other reasons.
But additional information could also make it harder for you to find the relevant event.
You should carefully consider the tradeoffs of what and how much information you log.
Now you can search for Audit events that are relevant to AIDE reports.
For example, if AIDE reports that the /etc/group file has unexpectedly changed, then you can use the ausearch command to look for relevant Audit logs:
[root@host]# ausearch -i -f /etc/group
...output omitted...Use the -f option to select events relevant to the file that you are investigating.
Use the -i option to help interpret the Audit event, such as by converting numbers to more human-readable names.
Note
In some cases you might want to omit -i to see the uninterpreted Audit log.
For example, if there is a misconfiguration where multiple UID numbers map to the same username, the raw log might be useful.
To further narrow down the output of the Audit search to events of your interest, you can use the -ts option to search for events from the date and time of the last AIDE report that did not report a change.
[root@host ~]# ausearch -ts -i -f /etc/group
...output omitted...
type=PROCTITLE msg=audit(10/15/23 23:48:59.672:158) : proctitle=groupadd test-group1
type=PATH msg=audit(10/15/23 23:48:59.672:158) : item=0 name=/etc/group inode=8393479 dev=fc:04 mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:passwd_file_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(10/15/23 23:48:59.672:158) : cwd=/root
type=SYSCALL msg=audit(10/15/23 23:48:59.672:158) : arch=x86_64 syscall=openat success=yes exit=4 a0=AT_FDCWD a1=0x7f3b467bbe0c a2=O_RDONLY|O_CLOEXEC a3=0x0 items=1 ppid=1221 pid=26503 auid=student uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=3 comm=groupadd exe=/usr/sbin/groupadd subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=group_watch
...output omitted...Depending on the Audit rules that you have set, you might have several Audit events to look through. Next, you need to determine which of those events are involved in the change.
You must review the Audit events that the ausearch command finds to determine what happened.
There might be multiple Audit events involved in sequence.
If you are logging other activity on the files that are involved, then some irrelevant events might be logged that are not involved in whatever changed that file.
Alternatively, there might be events before the actual change that were triggered by the same command that made the change.
For this reason, you need to carefully review the Audit events to understand what happened.
Often, the logged event is a system call to the kernel.
The events might help you understand what arguments the system call takes so that you can better interpret the SYSCALL Audit record to identify the relevant event.
To illustrate this scenario, examine two different Audit events involving the /etc/group file.
Assume that an AIDE report states that the modification time and contents of the /etc/group file have changed.
The following example shows one Audit event:
...output omitted... type=PROCTITLE msg=audit(10/16/23 00:01:26.949:344) :proctitle=touch /etc/grouptype=PATH msg=audit(10/16/23 00:01:26.949:344) : item=1name=/etc/groupinode=8622249 dev=fc:04 mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:passwd_file_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(10/16/23 00:01:26.949:344) : item=0 name=/etc/ inode=8388736 dev=fc:04 mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=CWD msg=audit(10/16/23 00:01:26.949:344) : cwd=/home/student type=SYSCALL msg=audit(10/16/23 00:01:26.949:344) : arch=x86_64syscall=openatsuccess=noexit=EACCES(Permission denied)a0=AT_FDCWD a1=0x7ffdc8d99687 a2=O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK a3=0x1b6 items=2 ppid=26589 pid=26731 auid=student uid=student gid=student euid=student suid=student fsuid=student egid=student sgid=student fsgid=student tty=pts1 ses=5 comm=touch exe=/usr/bin/touch subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=group_watch ...output omitted...
The SYSCALL record for this event indicates that the openat system call is being used to open the file in O_WRONLY mode.
The syscall=openat field indicates the system call used, and the a2 field contains the third argument to the system call, counting from zero.
The openat(2) man page indicates that O_WRONLY mode in that location is a flag for write-only mode.
Review the man page for the system call that you are analyzing to determine which arguments contain what data.
One of the PATH records for this event indicates that the /etc/group file is involved.
The PROCTITLE record for this event suggests that touch /etc/group was the command that triggered this event, which should update the modification time of the /etc/group file, but should not affect the file's contents.
The metadata update would still require write access to the file's inode.
However, carefully examine the SYSCALL record again.
This system call failed as indicated by the success=no and exit=EACCES(Permission denied) records.
This is not the right Audit event.
The following example is a second Audit event:
...output omitted... type=PROCTITLE msg=audit(10/16/23 00:04:25.441:363) :proctitle=vigrtype=PATH msg=audit(10/16/23 00:04:25.441:363) :item=4name=/etc/groupinode=8622254dev=fc:04 mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:passwd_file_t:s0 nametype=CREATE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(10/16/23 00:04:25.441:363) : item=3 name=/etc/group inode=8622249 dev=fc:04 mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:passwd_file_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(10/16/23 00:04:25.441:363) :item=2name=/etc/group.editinode=8622254dev=fc:04 mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:passwd_file_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(10/16/23 00:04:25.441:363) : item=1 name=/etc/ inode=8388736 dev=fc:04 mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(10/16/23 00:04:25.441:363) : item=0 name=/etc/ inode=8388736 dev=fc:04 mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=CWD msg=audit(10/16/23 00:04:25.441:363) : cwd=/root type=SYSCALL msg=audit(10/16/23 00:04:25.441:363) : arch=x86_64syscall=renamesuccess=yesexit=0 a0=0x7ffc431008c0 a1=0x56270e8e4440 a2=0x0 a3=0x0 items=5 ppid=26740 pid=26762 auid=student uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts1 ses=5 comm=vigrexe=/usr/sbin/vipwsubj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=group_watch ...output omitted...
The output shows the PROCTITLE record, five PATH records, a CWD record, and a SYSCALL record.
The SYSCALL record shows that this event was a rename system call that was successful (success=yes), which means the file was renamed.
(That system call is documented by the rename(2) man page.)
Examine the PATH records, paying close attention to the inode numbers and the objtype fields.
The 8622254 inode has two records: item=2 and item=4.
The inode had the /etc/group.edit name in the item=2 record, but that was deleted.
The inode was created with the /etc/group name in the item=4 record, which looks like the file that was renamed.
In addition, inode 8622254 had the /etc/group name and was deleted.
This file is the original /etc/group file that is being overwritten.
Reviewing the SYSCALL record again, the file was renamed by the root (uid=root) user.
But, the user originally logged in as the student (auid=student) user and changed their real UID somehow.
The command used was vigr, which executed the /usr/sbin/vipw command.
The /usr/sbin/vigr file is a symlink to the /usr/sbin/vipw file.
The command was run from terminal pts/1 (tty=pts1), which is a pseudoterminal (most likely a remote login session or graphical terminal window).
This is all useful data for further investigation.
Important
This event certainly changed the contents, modification time, and inode belonging to the /etc/group file, so this is an event of interest.
Remember, there might be more than one event of interest in the period of time.
You should investigate any events that the ausearch command helps you find.
The following table lists some record types that you might see when determining the cause of a file change.
Table 7.1. Common Audit Record Types
| Record type | Description |
|---|---|
CWD
| The current working directory that is relevant to this event. |
PATH
| Information about the path to a file that is involved in this event.
Watch in particular for its objtype field, which records why this record is involved in an event, including a SYSCALL record.
There might be multiple PATH records in the same event, for files that are involved for different reasons. |
PROCTITLE
| The complete command line that triggered this event. |
SYSCALL
| The system call that was made to the kernel that triggered this event.
Particular system calls are likely to be involved in a file change event, such as the open(), fchmod(), and setxattr() calls.
This record is also most likely to have the most information about which user and process is involved in the operation. |
The following table lists some useful fields that you might see in an Audit record.
Table 7.2. Audit Record Fields
| Field | Description |
|---|---|
a0
| First argument of the system call. |
a1
|
Second argument of the system call. This is particularly relevant with the |
a2
|
Third argument of the system call. The |
a3
| Fourth argument of the system call. |
auid
| The Audit UID. This is the user ID that was used to log in to the system initially.
It stays the same even after the user uses the su command or some other command to change their real UID. |
cwd
| The current working directory that is relevant to this event. |
euid
| The effective UID of the user who started the analyzed process. |
egid
| The effective GID of the group that the process has for permission checks. |
proctitle
| The process title.
The record shows the full command line along with the arguments of the executed command.
This record is encoded in hexadecimal notation. Use -i option of the ausearch command to decode it. |
syscall
| The system call that is sent to the kernel. |
success
| Shows whether the system call was successful. |
tty
| The terminal or pseudoterminal that is controlling the process. If the command was run as part of an interactive session and the user had several in progress, then this field can be used to help identify which session was used to run the command. |
uid
| The real UID.
The user ID that started the process.
This might not be the same as the initial login ID of the user.
For example, a user might use the su command to switch from one user to another, which changes their real UID.
Likewise, if the process was started by the setuid or setgid executables, the effective UID or GID of the process might be different. |
References
auditctl(8), ausearch(8), audit.rules(7), and auditd.conf(5) man pages
For more information, refer to the Checking Integrity with AIDE chapter in the Red Hat Enterprise Linux 9 Security Hardening guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/security_hardening/checking-integrity-with-aide_security-hardening
For more information, refer to the Understanding Audit Log Files chapter in the Red Hat Enterprise Linux 9 Security Hardening guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/security_hardening/auditing-the-system_security-hardening#understanding-audit-log-files_auditing-the-system
