Red Hat Security: Linux in Physical, Virtual, and Cloud

Objectives

After completing this section, students should be able to investigate causes of file system changes reported by AIDE by using Linux Audit tools.

Combining AIDE and Audit

AIDE is a useful tool that you can use to detect changes on a computer's file systems. It reports to administrators about specific changes. One of its limitations, however, is that it does not tell you what caused the changes.

For example, you can configure AIDE to report changes to /etc/passwd and /etc/shadow, but that does not produce details such as who made the changes, at what time, using which tool. This is where Audit rules can help you to get the details that you need about what happened.

The Linux Audit system can monitor files for activity that might cause a change, and log a lot of useful information about the time of the activity and the process and user involved. The Audit system logs these changes, but it is not as simple to set it up to report on key events by itself.

The combination of AIDE and Audit can be a useful way to solve this problem. By configuring both systems to monitor key files and directories, you can use AIDE to report periodically on unexpected changes that have been made to your systems. Then you can look at the audit log since the last AIDE report before the change to help identify what activity might be responsible.

Configuring AIDE and Audit

For this approach to be effective, you need to configure both AIDE and Audit to monitor the same files. Audit records activity affecting a file; AIDE detects and reports that there has been a change to a file. This means that you have to set up the AIDE database and watch rules on files in advance to collect the relevant information for detection and investigation.

Start by making sure that AIDE is properly configured. Edit the /etc/aide.conf file to configure AIDE to monitor the correct files, initialize the AIDE database, and set up the cron job to periodically run AIDE checks and send the report to the system administrators.

For example, what if you want to monitor all the files in /usr and its subdirectories for changes? You could use the default settings in /etc/aide.conf that set a regular selection line to monitor all the files in /usr for changes to content, permissions, and ownerships:

CONTENT_EX = sha256+ftype+p+u+g+n+acl+selinux+xattrs
...output omitted...
/usr/    CONTENT_EX

You also need to set an audit rule to monitor the /usr directory recursively for attempts to write files or change permissions. The easiest way to do this is to set a file system rule in a .rules file in /etc/audit/rules.d.

The following example shows a minimal rule to set a watch for changes to the /usr directory:

-w /usr -p wa -k usr-modification

Investigating File System Changes

Now you can search for audit events that are relevant to AIDE reports. For example, if AIDE is reporting that the /etc/group file has unexpectedly changed, you can use the following ausearch command to look for relevant audit logs:

[root@demo]# ausearch -i -f /etc/group

Use the -f option to select events relevant to the file that you are investigating. Use the -i option to help interpret the audit event, converting numbers to more human-readable names.

To further narrow down the output of the audit search to events of interest, you can use the -ts option to search for events since the date and time of the last AIDE report that did not report a change.

[root@demo]# ausearch -i -f /etc/group -ts "08/07/2018" "09:00:00"

Depending on the audit rules you have set, you may have quite a few audit events to look through. Next, you need to determine which of those events are actually involved in the change.

Interpreting Audit Events

You need to review the audit events that ausearch finds to determine what happened. There may be multiple audit events involved in sequence. If you are logging other activity on the files involved, there may be irrelevant events in the log that are not involved in whatever changed that file. Alternatively, there may be events prior to the actual change that were triggered by the same command that actually made the change.

For this reason, you need to carefully look at the audit events in order to understand what happened. Frequently, the event logged was a system call to the kernel. It helps if you understand what arguments the system call takes so that you can better interpret the SYSCALL audit record in the event to determine if it is relevant. This is often documented in man pages in section 2 of the system manual. Pay attention to the other records for the event, because they usually include useful information.

To illustrate this, look at two different audit events involving /etc/group. Assume that an AIDE report states that the modification time and contents of the /etc/group file have changed.

One audit event looks like this:

type=PROCTITLE msg=audit(08/13/2018 22:45:30.698:2432) : proctitle=touch /etc/group
type=PATH msg=audit(08/13/2018 22:45:30.698:2432) : item=1 name=/etc/group inode=33593020 dev=fd:00 mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:passwd_file_t:s0 objtype=NORMAL
type=PATH msg=audit(08/13/2018 22:45:30.698:2432) : item=0 name=/etc/ inode=33554497 dev=fd:00 mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:etc_t:s0 objtype=PARENT
type=CWD msg=audit(08/13/2018 22:45:30.698:2432) :  cwd=/home/student
type=SYSCALL msg=audit(08/13/2018 22:45:30.698:2432) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x7ffc044cb6c3 a1=O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK a2=0666 a3=0x7ffc044c94e0 items=2 ppid=2328 pid=32015 auid=student uid=student gid=student euid=student suid=student fsuid=student egid=student sgid=student fsgid=student tty=pts1 ses=4 comm=touch exe=/usr/bin/touch subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=group_watch

The SYSCALL record indicates that this is an attempt to open a file in write mode: syscall=open and a1=O_WRONLY. One of the PATH records indicates that /etc/group is involved. The PROCTITLE record suggests that this was a touch /etc/group command, which should update the modification time, but it should not affect file contents. The metadata update would still require write access to the file's inode.

However, look carefully at the SYSCALL record again. This call failed: success=no and exit=EACCES(Permission denied). This is not the right audit event.

Here is a second audit event:

type=PROCTITLE msg=audit(08/13/2018 22:40:54.899:2429) : proctitle=vigr
type=PATH msg=audit(08/13/2018 22:40:54.899:2429) : item=4 name=/etc/group inode=33593020 dev=fd:00 mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:passwd_file_t:s0 objtype=CREATE
type=PATH msg=audit(08/13/2018 22:40:54.899:2429) : item=3 name=/etc/group inode=33578559 dev=fd:00 mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:passwd_file_t:s0 objtype=DELETE
type=PATH msg=audit(08/13/2018 22:40:54.899:2429) : item=2 name=/etc/group.edit inode=33593020 dev=fd:00 mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:passwd_file_t:s0 objtype=DELETE
type=PATH msg=audit(08/13/2018 22:40:54.899:2429) : item=1 name=/etc/ inode=33554497 dev=fd:00 mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:etc_t:s0 objtype=PARENT
type=PATH msg=audit(08/13/2018 22:40:54.899:2429) : item=0 name=/etc/ inode=33554497 dev=fd:00 mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:etc_t:s0 objtype=PARENT
type=CWD msg=audit(08/13/2018 22:40:54.899:2429) :  cwd=/root
type=SYSCALL msg=audit(08/13/2018 22:40:54.899:2429) : arch=x86_64 syscall=rename success=yes exit=0 a0=0x7ffc89437360 a1=0x55fc040a7100 a2=0x0 a3=0x65726373662f7274 items=5 ppid=2633 pid=32003 auid=student uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts1 ses=4 comm=vigr exe=/usr/sbin/vipw subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=group_watch

There is a PROCTITLE record, five PATH records, a CWD record, and a SYSCALL record.

The SYSCALL record shows this was a rename(2) operation that was successful (success=yes). That means a file was renamed.

Look at the PATH records, paying close attention to the inode numbers and the objtype fields. It looks like inode 33593020 has two records: item=2 and item=4. It had the name /etc/group.edit in item 2 but that was deleted. It was created with the name /etc/group in item 4. That looks like it was the file that was renamed. In addition, inode 33578559 had the name /etc/group and was deleted. That must be the original /etc/group file being overwritten.

Looking at the SYSCALL record again, this was done by root (uid=root). But, the user originally logged in as student (auid=student) and has since changed their real UID somehow. The command used was vigr, which actually executed /usr/sbin/vipw. (If you look, /usr/sbin/vigr turns out to be a symlink to vipw). The command was run from terminal pts/1 (tty=pts1), which is a pseudoterminal; most likely a remote login session or graphical terminal window. This is all useful data for further investigation.

Some record types that you might see when determining the cause of a file change are listed in the following table.

The following table lists some useful fields that you might see in an audit record.