Bookmark this page

Chapter 9. Troubleshooting Security Issues

Abstract

Goal	Identify and resolve issues related to security subsystems.
Objectives	Identify and repair SELinux issues. Identify and repair user authentication and authorization issues. Identify and repair LDAP, Kerberos, and SSSD identity management issues.
Sections	Repairing SELinux Issues (and Guided Exercise) Identifying Authentication Issues (and Guided Exercise) Resolving Identity Management Issues (and Guided Exercise)
Lab	Troubleshooting Security Issues

Repairing SELinux Issues

Objectives

Identify and repair SELinux issues.

SELinux Logging

Security Enhanced Linux (SELinux) implements Mandatory Access Control (MAC). Every process and system resource has a security label called an SELinux context. An SELinux context, sometimes known as an SELinux label, is an identifier that abstracts the system-level details and focuses on the security properties of the entity. The SELinux policy uses these contexts in a series of rules that define how processes can interact with each other and the various system resources. By default, the policy does not allow any interaction unless a rule explicitly grants access.

Figure 9.1: SELinux decision making flow

When SELinux blocks an action from happening, such as when a web server tries to write to /etc/shadow, the action is logged with the auditd daemon. The audit messages are in /var/log/audit/audit.log where they are searched with the ausearch command. Administrators can use the ausearch command to focus only on messages of interest.

[root@host ~]# ausearch -m avc -ts recent  
---
time->Sun Nov  7 21:01:00 2021
type=PROCTITLE msg=audit(1636336860.921:117): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44 
type=SYSCALL msg=audit(1636336860.921:117): arch=c000003e syscall=4 success=no exit=-13 a0=7fd8e400e898 a1=7fd8ebffe810 a2=7fd8ebffe810 a3=1 items=0 ppid=5497 pid=5501 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null) 
type=AVC msg=audit(1636336860.921:117): avc:  denied  { getattr } for  pid=5501 comm="httpd" path="/etc/shadow" dev="vda3" ino=16803119 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:shadow_t:s0 tclass=file permissive=0

	The `-m avc` option directs `ausearch` to display only Access Vector Cache (AVC) messages, the message type that is associated with SELinux denials.
	The `-ts recent` option specifies to show messages starting from 10 minutes ago. Other indications, such as `today`, `yesterday`, and `this-week` are recognized, as well as actual times.
	Typically, three lines are shown for every SELinux denial. The `type=PROCTITLE` line displays the full command line that triggered the Audit event. The field is encoded in hexadecimal notation so that the user cannot influence the Audit log parser. Use the `ausearch` command with the `--interpreter` option to convert hexadecimal values into human-readable equivalents. The `2F7573722F7362696E2F6874747064002D44464F524547524F554E44` value is interpreted as `/usr/sbin/httpd -DFOREGROUND`.
	The `type=SYSCALL` line has useful event information, such as the actual, original, and effective user ID of the calling process and the action that caused the denial, typically a system call, and the actual denial.
	The `type=AVC` line is the actual denial. The `audit(1636336860.921:117)` part lists the time when this message was passed to the audit subsystem, in seconds since the epoch. Use the `date --date=@timestamp` command to convert the time to the local time zone. The remainder of the line is the list of denied actions. Common actions include `read` for reading, `open` for opening, and `getattr` for requesting extended file information.

Further fields on the type=AVC line identify the name of the process for which an action was denied, the file system relative path, the file system's device, the inode of the target file, and the full SELinux contexts of both the process that attempts access and the file, process, or network port that is accessed.

Using dontaudit Rules to Limit Audit Events

Some frequent actions are always denied and do not need to be tracked. The SELinux policy has a dontaudit rule, so that the action is still blocked, but not logged.

Use the sesearch --dontaudit command from the setools-console package to list the active dontaudit rules.

If a dontaudit rule is suspected as the cause of an issue, then all dontaudit rules can be temporarily disabled with the semanage dontaudit off command. The dontaudit rules remain disabled until the semanage dontaudit on command is run.

Important

The dontaudit rules exist for frequent actions that are always blocked and do not impact system behavior. However, keeping the dontaudit feature disabled fills the audit log rapidly, and impedes locating more important denial events.

SELinux Troubleshooting Tools

When SELinux blocks an action, first look in the /var/log/audit/audit.log file for information about a denial. Use the ausearch command to query the audit logs.

If auditd is running, but the output of the ausearch command does not have any matches, then use the journalctl command to inspect the messages from the systemd journal.

[root@host ~]# journalctl -t setroubleshoot
-- Logs begin at Sun 2021-11-07 19:19:56 EST, end at Sun 2021-11-07 22:34:13 EST. --
Nov 07 21:01:05 host.example.com setroubleshoot[6414]: AnalyzeThread.run(): Cancel pending alarm
Nov 07 21:01:06 host.example.com setroubleshoot[6414]: SELinux is preventing /usr/sbin/httpd from getattr access on the file /etc/shadow. For complete SELinux messages run: sealert -l 2e7830e4-aed5-4ff6-b242-47d104ebc9ec
Nov 07 21:01:06 host.example.com setroubleshoot[6414]: SELinux is preventing /usr/sbin/httpd from getattr access on the file /etc/shadow.

        *****  Plugin catchall (100. confidence) suggests   *****

        If you believe that httpd should be allowed getattr access on the shadow file by default.
        Then you should report this as a bug.
        You can generate a local policy module to allow this access.
        Do allow this access for now by executing:
        # ausearch -c 'httpd' --raw | audit2allow -M my-httpd
        # semodule -X 300 -i my-httpd.pp
...output omitted...

You can diagnose and fix many simple SELinux denials, such as mislabeled files in /var/www/html, by inspecting /var/log/audit/audit.log and running the restorecon command in the files and directories to correct the labels. Other denials are more difficult to diagnose and fix. The sealert command works in two ways:

Passing it the UUID of a denial found in the system logs, such as sealert -l fdfe9d90-557b-4ad0-a1a3-a6d09e065523.
Parsing all denial messages in a file, such as sealert -a /var/log/audit/audit.log.

In both cases, the output is the same as setroubleshootd writes to the system logs.

Important

The setroubleshoot daemon might suggest creating a policy module with the audit2allow command. In most cases, it is not the preferred solution. Build custom policy modules only when the impact of the new policy module rules is fully understood.

Common SELinux issues

Most SELinux issues fall into the following scenarios:

Using nonstandard locations for service data.
Switching from disabled to enforcing mode.
Setting Booleans incorrectly.
Using nonstandard network ports for services.

Each scenario has common characteristics that assist troubleshooting.

Using nonstandard locations for service data

The targeted policy provides default file contexts for all RHEL services, including secondary data locations. For example, the default locations for web server data at /var/www and secondary locations under /srv/*/www are recognized and set with the correct SELinux contexts when using the restorecon command.

Use the semanage fcontext command to list and manage the stored mappings between file names and standard contexts.

When a nonstandard data location is used, configure SELinux to recognize the location. For example, to use /mysites/sitea/www as a document root for a web server, first create a mapping in the SELinux database. This example configures the httpd_sys_content_t context on all files under the /mysites/sitea/www directory.

[root@host ~]# semanage fcontext -a -t httpd_sys_content_t '/mysites/sitea/www(/.*)?'

Then, restore correct settings recursively to the /mysites/sitea/www directory, with the proper contexts obtained from the database.

[root@host ~]# restorecon -Rv /mysites/sitea/www

Switching from disabled to enforcing mode.

When a system is running with SELinux disabled, any program that creates files does not set the file's SELinux context, because the SELinux policy module is not loaded and therefore cannot provide or verify the context. Programs such as sed, other text editors, and NetworkManager do not edit files in place, but create a temporary file and then move that file to the original file name.

When SELinux is enabled after the system is run in disabled mode, SELinux does not trust that the system is secure, because it does not immediately know if files exist without their required SELinux context. To resolve this problem, a system must be relabeled. Relabeling is the process of checking and updating every file with the proper SELinux file context as stored in the semanage database.

If a system with SELinux enabled has unlabeled files, then normal, confined services cannot access those files with SELinux policies. Unlabeled files display the unlabeled_t type. If critical files, such as for user authentication, are unlabeled, then RHEL might not boot, because the files are insecure and untrusted.

To force a relabel, create an empty file called /.autorelabel, and then reboot. When detected during the boot process, the system relabels all files, deletes the /.autorelabel file, and then restarts the boot process. This process confirms that all files on a running system with SELinux enabled are properly protected by labels.

Setting Booleans incorrectly

The confined services in RHEL 8 are limited by their SELinux policy, but have optional features that can be enabled with SELinux Booleans. For example, by default the httpd services can serve content only from their main web server directory. Moreover, the httpd services cannot create outgoing network connections, by default.

Specific Booleans exist for httpd use cases such as connecting to various databases, memcached use, FTP servers, and LDAP services. One Boolean allows the httpd service to permit all network connections, if another Boolean does not cover a specific use case. Another Boolean allows the httpd service to permit users to access their home directory from a web browser with http.

Use the semanage boolean --list command to query the list of Booleans, including their current state, default state, and description. Use the getsebool and setsebool commands to query and set the Booleans individually.

Important

Use the setsebool -P flag to set Boolean values that persist across reboots. Otherwise, non-default Boolean settings are lost when SELinux or the system is stopped.

Using nonstandard network ports for services

By default, the targeted policy allows confined services to listen only on a predefined set of network ports. For example, the httpd daemon is allowed to bind only to ports that are labeled as either http_port_t or http_cache_port_t.

Use the semanage command to label an unlabeled port.

[root@host ~]# semanage port -a -t http_port_t -p tcp 8001

With a port label policy, services cannot listen on ports with a conflicting type. For example, sshd cannot listen on port 443/tcp, which is labeled for use by the httpd service. Although not normally recommended, you can create policies that allow non-standard port bindings.

Understanding SElinux Rules

The targeted SELinux policy is the default policy on RHEL 8 and includes with many types, rules, and Booleans to cover all of the services that are distributed in RHEL. To learn about the supported types, policy rules, and available Booleans, install the setools-console package. This package provides various tools to help with SELinux troubleshooting, including seinfo and sesearch.

The seinfo -t and seinfo -b commands list all types and Booleans. Passing a type name with the -t option displays that type's details and configured aliases.

[root@host ~]# seinfo -t httpd_sys_content_t

Types: 1
   httpd_sys_content_t

The seinfo command determines which port types are associated with a specific network port:

[root@host ~]# seinfo --portcon=443

Portcon: 4
   portcon sctp 1-511 system_:object_r:reserved_port_t:s0
   portcon tcp 1-511 system_u:object_r:reserved_port_t:s0
   portcon tcp 443 system_u:object_r:http_port_t:s0
   portcon udp 1-511 system_u:object_r:reserved_port_t:s0

The sesearch command searches within the defined policy rules, to indicate which rules a Boolean enables. For example, use sesearch` to view all allow rules enabled by the httpd_can_connect_ldap Boolean:

[root@host ~]# sesearch --allow -b httpd_can_connect_ldap
allow httpd_t ldap_port_t:tcp_socket name_connect; [ httpd_can_connect_ldap ]:True

This rule states that processes of the type httpd_t can connect to TCP sockets of the type ldap_port_t.

References

Further information is available in the Using SELinux chapter in the System Design Guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/system_design_guide/index#using_selinux

For more information, refer to the Auditing the system chapter in the Security Hardening Guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/security_hardening/index#auditing-the-system_security-hardening

semanage(8), sesearch(1), getsebool(8), setsebool(8), sealert(8), setroubleshootd(8), seinfo(1), and sesearch(1) man pages.

Discuss Red Hat Enterprise Linux Diagnostics and Troubleshooting

Go to community

Welcome to the Red Hat Enterprise Linux Diagnostics and Troubleshooting (RH342) group!

cschunke

26 wrz 2023

We are excited to launch a space dedicated to the Red Hat Training course Red Hat Enterprise Linux Diagnostics and Troubleshooting! To gain the most value from this group - click the "Join Group" button in the upper right hand corner of the group home page.We encourage group members to collaborate in this group to discuss topics, ask questions, share best practices and tips, provide course feedback, and share their accomplishments as it relates to RH342.Read more about Red Hat Enterprise Linux Diagnostics and Troubleshooting here.

445

Revision: rh342-8.4-6dd89bd

Click Create to build all of the virtual machines needed for the classroom lab environment. This may take several minutes to complete. Once created the environment can then be stopped and restarted to pause your experience.
When a lab is created, click Start to run all of the virtual machines in the classroom.
Click Stop to stop all the virtual machines from running. This will not delete your lab.
If you Delete your lab, you will remove all of the virtual machines in your classroom and lose all of your progress.

Virtual machine actions

Click Start to power on the virtual machine.
Click Shutdown to gracefully shut down the virtual machine, preserving disk contents.
Click Power off to forcefully shut down the virtual machine, while still preserving disk contents.
Click Open console to connect to the system console of the virtual machine in a new browser.

Auto-stop timer

The Red Hat Learning Subscription entitles you to set allotment of lab time.
To help conserving your allotted time, the lab environment uses automatic timers to stop or destroy your lab environment when the timer expires.

Click the Auto-stop button [+] to extend the time you would like to spend with the labs.
Click the Auto-destroy button [+] to add day(s) to the auto-destroy timer.

Auto-stop has a maximum of 11 hours, and auto-destroy has a maximum of 14 days.
Be careful to keep the timers set while you are working, so that your environment doesn't shut down unexpectedly.
We also suggest not to set the auto-timers unnecessarily high, which could waste your lab time allotment