Red Hat Security: Linux in Physical, Virtual, and Cloud
Audit events that are recorded in the /var/log/audit/audit.log file include a lot of information in a condensed format.
A single event might log multiple Audit records of different types to the log as separate messages.
Each of these records might include several fields of information about the logged event.
The following example shows several Audit records that are associated with a single Audit event as it might appear in the /var/log/audit/audit.log file, recorded in an unprocessed raw format by the auditd daemon.
type=SYSCALLmsg=audit(1371716130.596:28708): arch=c000003esyscall=2success=yes exit=4 a0=261b130 a1=90800 a2=e a3=19 items=1 ppid=2548 pid=26131auid=1000uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="aureport" exe="/sbin/aureport" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023key="audit-access"type=CWD msg=audit(1371716130.596:28708): cwd="/root" type=PATH msg=audit(1371716130.596:28708): item=0 name="/var/log/audit" inode=17998 dev=fd:01 mode=040750 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:auditd_log_t:s0
The previous output displays three Audit records, which are all part of Audit event 28708.
The
type=SYSCALLrecord indicates thetypefield. Each Audit record has a record type, sometimes called a message type, which is reflected by thetypefield that starts each record. This record is aSYSCALLrecord.The
msg=audit(1371716130.596:28708)record indicates themsgfield. Themsgfield gives the time stamp for this record and the event ID. The number before the colon (in this case,1371716130.596) is the time stamp in the number of seconds since the epoch, 00:00 UTC on January 1, 1970. You can convert epoch time to local time by using thedate --date=@command. The number after the colon (<epoch-time>28708) is the Audit event number, which the record shares with the other records for this event.The
syscall=2record indicates thesyscallfield. Thetypeof the first record isSYSCALL, which indicates information about a system call that was made to the kernel. Thissyscallfield indicates the number of the system call that was made (not its name). The mapping of system call numbers to names can vary between processor architectures, which is one reason why it can be challenging to correctly interpret a log that is in raw format without help. You can use theausearchcommand to provide this help.The
auid=1000record indicates theauidfield. Theauidfield records the Audit UID of the user that triggered this event. This is the UID of the initial account that was used to log in to this machine by the user that triggered this event, even if they used thesudoorsucommands to become another user.The
key="audit-access"record indicates thekeyfield. Thekeyfield is an identifier that you can use when searching for events, similar to a tag. You can set keys in your custom Auditing rules to make it easier to filter for certain types of events.
The Audit system includes the ausearch command, a powerful tool for searching Audit logs.
You can use the ausearch command to search for and filter various types of events.
The ausearch command can translate numeric values into more readable values, such as user names or system call names, so that you can interpret those events more easily.
You can use the -i option to interpret the log records and translate numeric values into names.
This option is useful when you have raw log files.
The -if option in the following example takes a saved raw log file as an argument, so that ausearch can analyze the raw log file rather than the current Audit log on the system.
[root@host ~]# ausearch -i -if ./raw_audit.log
...output omitted...
----
type=PROCTITLE msg=audit(10/11/23 14:14:52.183:421) : proctitle=autrace /bin/ls /tmp
type=SYSCALL msg=audit(10/11/23 14:14:52.183:421) : arch=x86_64 syscall=exit_group a0=EXIT_SUCCESS a1=0xfffffffffffffe98 a2=0xe7 a3=0x7ffcdd36a200 items=0 ppid=1438 pid=1440 auid=student uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=8 comm=ls exe=/usr/bin/ls subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)You can use the --raw option to print raw log entries without separators between events.
This option is useful if you have other tools that can parse the raw log format.
The -r short option is equivalent.
In the following example, the -p option looks for Audit log entries that are recorded by a process with a PID of 1440.
[root@host ~]# ausearch -p 1440 --raw
...output omitted...
type=SYSCALL msg=audit(1697048092.183:420): arch=c000003e syscall=3 success=yes exit=0 a0=2 a1=fbad2006 a2=7f366a7f69e0 a3=b items=0 ppid=1438 pid=1440 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=8 comm="ls" exe="/usr/bin/ls" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)ARCH=x86_64 SYSCALL=close AUID="student" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=PROCTITLE msg=audit(1697048092.183:420): proctitle=61757472616365002F62696E2F6C73002F746D70
type=SYSCALL msg=audit(1697048092.183:421): arch=c000003e syscall=231 a0=0 a1=fffffffffffffe98 a2=e7 a3=7ffcdd36a200 items=0 ppid=1438 pid=1440 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=8 comm="ls" exe="/usr/bin/ls" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)ARCH=x86_64 SYSCALL=exit_group AUID="student" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=PROCTITLE msg=audit(1697048092.183:421): proctitle=61757472616365002F62696E2F6C73002F746D70A single event might generate multiple, related Audit log records.
You can use the -a option to show all records for a particular event based on the event ID.<EVENT-ID>
[root@host ~]# ausearch -a 233
...output omitted...
----
time->Wed Oct 11 14:08:48 2023
type=USER_LOGIN msg=audit(1697047728.122:233): pid=1363 uid=0 auid=1000 ses=8 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login id=1000 exe="/usr/sbin/sshd" hostname=? addr=172.25.250.9 terminal=/dev/pts/0 res=success'The message type describes the category of an Audit record.
There are many message types, such as ADD_USER, LOGIN, USER_CMD, VIRT_CONTROL, and many more.
By filtering for a specific message type, you can find information about a specific event, even if you do not know the exact details of the event.
Note
You can find a list of all Audit message types in the RHEL Audit System Reference article at https://access.redhat.com/articles/4409591
You can use the -m option to show all events that include a record with a specific message type.
The <MESSAGE-TYPE>--message long option is equivalent.
For example, if you suspect that an unauthorized login event occurred, you might filter on the LOGIN or USER_LOGIN message types to see all login events.
[root@host ~]# ausearch -m LOGIN
...output omitted...
----
time->Wed Oct 11 14:08:47 2023
type=LOGIN msg=audit(1697047727.989:227): pid=1367 uid=0 subj=system_u:system_r:init_t:s0 old-auid=4294967295 auid=1000 tty=(none) old-ses=4294967295 ses=9 res=1You can use the -f option to search for all events that are related to a specific file name.
The <FILENAME>--file long option is equivalent.
[root@host ~]# ausearch -f /bin/ls
...output omitted...
----
time->Wed Oct 11 14:24:07 2023
type=PROCTITLE msg=audit(1697048647.148:432): proctitle=61757472616365002F62696E2F6C73002D6C002F746D70
type=PATH msg=audit(1697048647.148:432): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=8389810 dev=fc:04 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1697048647.148:432): item=0 name="/bin/ls" inode=1192 dev=fc:04 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1697048647.148:432): cwd="/root"
type=EXECVE msg=audit(1697048647.148:432): argc=3 a0="/bin/ls" a1="-l" a2="/tmp"
type=SYSCALL msg=audit(1697048647.148:432): arch=c000003e syscall=59 success=yes exit=0 a0=7fffa43d17b1 a1=7fffa43d0b30 a2=7fffa43d0b50 a3=7f1b0c7b13e0 items=2 ppid=1450 pid=1452 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=8 comm="ls" exe="/usr/bin/ls" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)Audit can label an event with a key that you can use in searches.
You can use the -k option to search for all events that are labeled with a specific key.
For example, if you have several Audit rules that track changes to identity files, then you can configure Audit to label the created entries with a key such as <KEY>identity.
You can find each of these entries by using the -k identity option.
This way, you can find only the entries that relate to the labeled rules.
You can use the --start option to search for events that occurred after a specific start date and start time.
If you do not specify a starting time, then the search assumes midnight.
If you omit the starting date, then the search assumes the current day.
The time format depends on your current locale setting.
Other values that you can use include [start-date] [start-time]recent (the past ten minutes), this-week, this-month, and this-year.
You can use the --end option to search for events that occurred before a specific date and time with the same time format syntax as the --start option.
Other options exist to help you search for events based on user, terminal, virtual machine, or other identifiers.
For a complete list of options refer to the ausearch(8) manual page.
The following ausearch command returns an interpreted version of event 28708 from the previous example:
[root@host ]#ausearch -i -a 28708---- type=PATHmsg=audit(07/31/2023 10:15:30.596:28708) : item=0 name=/var/log/audit inode=17998 dev=fd:01 mode=dir,750 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:auditd_log_t:s0 type=CWDmsg=audit(07/31/2023 10:15:30.596:28708) : cwd=/root type=SYSCALLmsg=audit(07/31/2023 10:15:30.596:28708) : arch=x86_64 syscall=open success=yes exit=4 a0=261b130 a1=90800 a2=e a3=19 items=1 ppid=2548 pid=26131 auid=student uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=1 comm=aureport exe=/sbin/aureport subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=audit-access
Each Audit event is separated in the interpreted output by four dashes.
This output displays the PATH, CWD, and SYSCALL event types.
The PATH record is a file involved in this event.
The file is named /var/log/audit (name=/var/log/audit), and is inode 17998 (inode=17998).
The file is on the file system on a device with major/minor numbers 253,1 (dev=fd:01, the device numbers are in hexadecimal format).
By looking in the /dev directory with the ls -l command, you can see that the /dev/dm-1 device has those numbers and is associated with a logical volume.
The file is a directory with octal permissions 750 (mode=dir,750), owned by the root user and the root group (ouid=root ogid=root), with the SELinux type auditd_log_t (obj=system_u:object_r:auditd_log_t:s0).
The CWD record is the current working directory that is associated with the process that triggered this event, in this case the /root directory.
The SYSCALL record is the system call that triggered this event.
The open() system call (syscall=open) was used to successfully (success=yes) open the file that is specified by the PATH record (the /var/log/audit directory).
This call was done by a process with the 26131 PID (pid=26131).
The call was started by the /sbin/aureport executable (exe=/sbin/aureport) and was run with the root effective UID (euid=root) and a unconfined_t SELinux domain (subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023) by the root user (uid=root).
The command was run on the pts/0 pseudoterminal (tty=pts0), probably a graphical terminal window or remote login session.
The user originally logged in as the student user (auid=student) and has somehow become the root user since then.
This record has the audit-access key set on it, to make its event easier to find with the ausearch command (key=audit-access).
Instead of reading individual Audit messages, you can use the aureport command to get a quick overview of Audit messages or more detailed reports on specific types of events.
When you run the aureport command without any options, the command returns an overview of how many different types of events are present in the logs.
When you specify search options (mostly the same options as those for the ausearch command), the command displays a list of all events that match the search criteria.
Common options include the -i option to interpret results, and the --summary option to condense the list into a summary.
You can also create reports for specific types of events, such as a login report with the --login option, or an executable name report with the --executable option.
When you use the ausearch --raw command to search for specific events recorded in the Audit log files, you can provide the unformatted search results as input to the aureport command to generate formatted reports.
Note
Two specialized tools also exist: aulast and aulastlog.
These tools replace the last and lastlog tools respectively, but they parse the Audit logs instead of the /var/log/wtmp and /var/log/btmp files.
To investigate the system calls that are performed by a process, you specify the process as an argument of the autrace command.
The autrace command requires that you remove any custom Auditing rules.
When execution finishes, the autrace command clears those rules, and then provides an example ausearch command to investigate those events.
This technique is useful for troubleshooting or investigating programs of interest.
The following example traces the date command:
[root@host ~]#autrace /bin/dateWaiting to execute: /bin/date Thu Jul 31 11:38:46 CEST 2023 Cleaning up... Trace complete. You can locate the records with 'ausearch -i -p 26472' [root@host ~]#ausearch --raw -p 26472 | aureport --file -iFile Report =============================================== # date time file syscall success exe auid event =============================================== 1. 07/31/2023 11:38:46 /bin/date execve yes /bin/date student 29158 2. 07/31/2023 11:38:46 /etc/ld.so.preload access no /bin/date student 29161 3. 07/31/2023 11:38:46 /etc/ld.so.cache open yes /bin/date student 29162 4. 07/31/2023 11:38:46 /lib64/librt.so.1 open yes /bin/date student 29166 5. 07/31/2023 11:38:46 /lib64/libc.so.6 open yes /bin/date student 29173 6. 07/31/2023 11:38:46 /lib64/libpthread.so.0 open yes /bin/date student 29181 7. 07/31/2023 11:38:46 /usr/lib/locale/locale-archive open yes /bin/date student 29208 8. 07/31/2023 11:38:46 /etc/localtime open yes /bin/date student 29213
Warning
The autrace command requires you to remove any active rules before you run the command.
This might cause you to miss events from other processes that those existing rules would record.
If the Audit rules are locked in place, then the autrace command cannot unload the existing rules and does not work.
References
ausearch(8), aureport(8), and autrace(8) man pages
For more information about Audit options, refer to the RHEL Audit System Reference article at https://access.redhat.com/articles/4409591
For more information, refer to the Auditing the System chapter in the Red Hat Enterprise Linux 9 Security Hardening guide at https://access.redhat.com/documentation/es-es/red_hat_enterprise_linux/9/html-single/security_hardening/index#auditing-the-system_security-hardening
