Red Hat Enterprise Linux Diagnostics and Troubleshooting
Abstract
| Goal |
Identify and resolve issues related to security subsystems. |
| Objectives |
|
| Sections |
|
| Lab |
|
Security Enhanced Linux (SELinux) implements Mandatory Access Control (MAC). Every process and system resource has a security label called an SELinux context. An SELinux context, sometimes known as an SELinux label, is an identifier that abstracts the system-level details and focuses on the security properties of the entity. The SELinux policy uses these contexts in a series of rules that define how processes can interact with each other and the various system resources. By default, the policy does not allow any interaction unless a rule explicitly grants access.
When SELinux blocks an action from happening, such as when a web server tries to write to /etc/shadow, the action is logged with the auditd daemon. The audit messages are in /var/log/audit/audit.log where they are searched with the ausearch command. Administrators can use the ausearch command to focus only on messages of interest.
[root@host ~]#ausearch-m avc-ts recent
--- time->Sun Nov 7 21:01:00 2021 type=PROCTITLE msg=audit(1636336860.921:117): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=SYSCALL msg=audit(1636336860.921:117): arch=c000003e syscall=4 success=no exit=-13 a0=7fd8e400e898 a1=7fd8ebffe810 a2=7fd8ebffe810 a3=1 items=0 ppid=5497 pid=5501 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1636336860.921:117): avc: denied { getattr } for pid=5501 comm="httpd" path="/etc/shadow" dev="vda3" ino=16803119 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:shadow_t:s0 tclass=file permissive=0
| The |
| The |
| Typically, three lines are shown for every SELinux denial. The |
| The |
| The |
Further fields on the type=AVC line identify the name of the process for which an action was denied, the file system relative path, the file system's device, the inode of the target file, and the full SELinux contexts of both the process that attempts access and the file, process, or network port that is accessed.
Some frequent actions are always denied and do not need to be tracked. The SELinux policy has a dontaudit rule, so that the action is still blocked, but not logged.
Use the sesearch --dontaudit command from the setools-console package to list the active dontaudit rules.
If a dontaudit rule is suspected as the cause of an issue, then all dontaudit rules can be temporarily disabled with the semanage dontaudit off command. The dontaudit rules remain disabled until the semanage dontaudit on command is run.
Important
The dontaudit rules exist for frequent actions that are always blocked and do not impact system behavior. However, keeping the dontaudit feature disabled fills the audit log rapidly, and impedes locating more important denial events.
When SELinux blocks an action, first look in the /var/log/audit/audit.log file for information about a denial. Use the ausearch command to query the audit logs.
If auditd is running, but the output of the ausearch command does not have any matches, then use the journalctl command to inspect the messages from the systemd journal.
[root@host ~]# journalctl -t setroubleshoot
-- Logs begin at Sun 2021-11-07 19:19:56 EST, end at Sun 2021-11-07 22:34:13 EST. --
Nov 07 21:01:05 host.example.com setroubleshoot[6414]: AnalyzeThread.run(): Cancel pending alarm
Nov 07 21:01:06 host.example.com setroubleshoot[6414]: SELinux is preventing /usr/sbin/httpd from getattr access on the file /etc/shadow. For complete SELinux messages run: sealert -l 2e7830e4-aed5-4ff6-b242-47d104ebc9ec
Nov 07 21:01:06 host.example.com setroubleshoot[6414]: SELinux is preventing /usr/sbin/httpd from getattr access on the file /etc/shadow.
***** Plugin catchall (100. confidence) suggests *****
If you believe that httpd should be allowed getattr access on the shadow file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do allow this access for now by executing:
# ausearch -c 'httpd' --raw | audit2allow -M my-httpd
# semodule -X 300 -i my-httpd.pp
...output omitted...You can diagnose and fix many simple SELinux denials, such as mislabeled files in /var/www/html, by inspecting /var/log/audit/audit.log and running the restorecon command in the files and directories to correct the labels. Other denials are more difficult to diagnose and fix. The sealert command works in two ways:
Passing it the UUID of a denial found in the system logs, such as
sealert -l fdfe9d90-557b-4ad0-a1a3-a6d09e065523.Parsing all denial messages in a file, such as
sealert -a /var/log/audit/audit.log.
In both cases, the output is the same as setroubleshootd writes to the system logs.
Important
The setroubleshoot daemon might suggest creating a policy module with the audit2allow command. In most cases, it is not the preferred solution. Build custom policy modules only when the impact of the new policy module rules is fully understood.
Most SELinux issues fall into the following scenarios:
Using nonstandard locations for service data.
Switching from
disabledtoenforcingmode.Setting Booleans incorrectly.
Using nonstandard network ports for services.
Each scenario has common characteristics that assist troubleshooting.
- Using nonstandard locations for service data
The
targetedpolicy provides default file contexts for all RHEL services, including secondary data locations. For example, the default locations for web server data at/var/wwwand secondary locations under/srv/*/wwware recognized and set with the correct SELinux contexts when using therestoreconcommand.Use the
semanage fcontextcommand to list and manage the stored mappings between file names and standard contexts.When a nonstandard data location is used, configure SELinux to recognize the location. For example, to use
/mysites/sitea/wwwas a document root for a web server, first create a mapping in the SELinux database. This example configures thehttpd_sys_content_tcontext on all files under the/mysites/sitea/wwwdirectory.[root@host ~]#
semanage fcontext -a -t httpd_sys_content_t '/mysites/sitea/www(/.*)?'Then, restore correct settings recursively to the
/mysites/sitea/wwwdirectory, with the proper contexts obtained from the database.[root@host ~]#
restorecon -Rv /mysites/sitea/www- Switching from
disabledtoenforcingmode. When a system is running with SELinux disabled, any program that creates files does not set the file's SELinux context, because the SELinux policy module is not loaded and therefore cannot provide or verify the context. Programs such as
sed, other text editors, and NetworkManager do not edit files in place, but create a temporary file and then move that file to the original file name.When SELinux is enabled after the system is run in disabled mode, SELinux does not trust that the system is secure, because it does not immediately know if files exist without their required SELinux context. To resolve this problem, a system must be relabeled. Relabeling is the process of checking and updating every file with the proper SELinux file context as stored in the
semanagedatabase.If a system with SELinux enabled has unlabeled files, then normal, confined services cannot access those files with SELinux policies. Unlabeled files display the
unlabeled_ttype. If critical files, such as for user authentication, are unlabeled, then RHEL might not boot, because the files are insecure and untrusted.To force a relabel, create an empty file called
/.autorelabel, and then reboot. When detected during the boot process, the system relabels all files, deletes the /.autorelabel file, and then restarts the boot process. This process confirms that all files on a running system with SELinux enabled are properly protected by labels.- Setting Booleans incorrectly
The confined services in RHEL 8 are limited by their SELinux policy, but have optional features that can be enabled with SELinux Booleans. For example, by default the
httpdservices can serve content only from their main web server directory. Moreover, thehttpdservices cannot create outgoing network connections, by default.Specific Booleans exist for
httpduse cases such as connecting to various databases,memcacheduse, FTP servers, and LDAP services. One Boolean allows thehttpdservice to permit all network connections, if another Boolean does not cover a specific use case. Another Boolean allows thehttpdservice to permit users to access their home directory from a web browser withhttp.Use the
semanage boolean --listcommand to query the list of Booleans, including their current state, default state, and description. Use thegetseboolandsetseboolcommands to query and set the Booleans individually.Important
Use the
setsebool -Pflag to set Boolean values that persist across reboots. Otherwise, non-default Boolean settings are lost when SELinux or the system is stopped.- Using nonstandard network ports for services
By default, the
targetedpolicy allows confined services to listen only on a predefined set of network ports. For example, thehttpddaemon is allowed to bind only to ports that are labeled as eitherhttp_port_torhttp_cache_port_t.Use the
semanagecommand to label an unlabeled port.[root@host ~]#
semanage port -a -t http_port_t -p tcp 8001With a port label policy, services cannot listen on ports with a conflicting type. For example,
sshdcannot listen on port443/tcp, which is labeled for use by thehttpdservice. Although not normally recommended, you can create policies that allow non-standard port bindings.
The targeted SELinux policy is the default policy on RHEL 8 and includes with many types, rules, and Booleans to cover all of the services that are distributed in RHEL. To learn about the supported types, policy rules, and available Booleans, install the setools-console package. This package provides various tools to help with SELinux troubleshooting, including seinfo and sesearch.
The seinfo -t and seinfo -b commands list all types and Booleans. Passing a type name with the -t option displays that type's details and configured aliases.
[root@host ~]# seinfo -t httpd_sys_content_t
Types: 1
httpd_sys_content_tThe seinfo command determines which port types are associated with a specific network port:
[root@host ~]# seinfo --portcon=443
Portcon: 4
portcon sctp 1-511 system_:object_r:reserved_port_t:s0
portcon tcp 1-511 system_u:object_r:reserved_port_t:s0
portcon tcp 443 system_u:object_r:http_port_t:s0
portcon udp 1-511 system_u:object_r:reserved_port_t:s0The sesearch command searches within the defined policy rules, to indicate which rules a Boolean enables. For example, use sesearch` to view all allow rules enabled by the httpd_can_connect_ldap Boolean:
[root@host ~]# sesearch --allow -b httpd_can_connect_ldap
allow httpd_t ldap_port_t:tcp_socket name_connect; [ httpd_can_connect_ldap ]:TrueThis rule states that processes of the type httpd_t can connect to TCP sockets of the type ldap_port_t.
References
Further information is available in the Using SELinux chapter in the System Design Guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/system_design_guide/index#using_selinux
For more information, refer to the Auditing the system chapter in the Security Hardening Guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/security_hardening/index#auditing-the-system_security-hardening
semanage(8), sesearch(1), getsebool(8), setsebool(8), sealert(8), setroubleshootd(8), seinfo(1), and sesearch(1) man pages.
