Red Hat Security: Identity Management and Authentication
When approaching a troubleshooting problem, you must collect various pieces of information before you form a hypothesis. Not all this information might be readily available. For example, sometimes you might need to configure a service to increase its logging level, or you might need to configure the system to store information that would normally be discarded upon reboot.
Always verify the system logs first, both on the IdM server and the IdM clients. By default, a Red Hat Enterprise Linux 9 system uses two logging services for the system logs:
systemd-journald, which is configured to only keep logs in memory.rsyslogd, which gets messages sent to it bysystemd-journald(and others) and stores them on disk.
You can use the journalctl command to view messages in the system journal.
By default, it shows the full contents of the system journal, but you can use options and filters to specify the content to display.
The following table outlines the options to view system logs for troubleshooting problems with the IdM or SSSD services:
| Option | Description |
|---|---|
journalctl -ef
| Jumps to the end of the journal (-e), and enables follow mode (-f).
This keeps the journal open on the screen, displaying new messages as they come in.
This option is especially helpful for viewing the system behavior when running a command or starting a service. |
journalctl _SYSTEMD_UNIT=ipa.service
| Displays all messages generated by the ipa.service systemd unit. |
journalctl -u ipa.service
| Displays all messages generated by, and about, the ipa.service systemd unit. |
journalctl -p emerg..err
| Displays all messages in the journal with a priority in the range emerg up to and including err. |
Many potential problems occur during the reconfiguration of IdM components.
To find a potential cause of problems, you can review log files in the /var/log/ directory and its subdirectories on the IdM server:
| File | Description |
|---|---|
/var/log/ipaserver-install.log
| Contains the installation log for the IdM server. |
/var/log/ipareplica-install.log
| Contains the installation log for the IdM replica. |
/var/log/ipaclient-install.log
| Contains the installation log for the IdM client. |
/var/log/httpd/
| Contains logs for the Apache web server used by IdM. |
/var/log/pki/
| Contains logs related to the PKI operations, IdM CA, and KRA. |
/var/log/dirsrv/
| Contains logs related to the Directory Server instance used by the IdM server. |
When users report a problem when logging in to their systems using credentials stored and maintained by IdM, the first places to look are the local SSSD service logs.
SSSD provides a tool called sssctl, which can be especially useful for troubleshooting.
The sssctl tool uses the SSSD InfoPipe responder to send information from SSSD over the system bus.
Add ifp to the service option of your /etc/sssd/sssd.conf SSSD configuration file and restart the sssd service:
[sssd]
services = nss, sudo, pam, ssh, ifp[root@client ~]# systemctl restart sssdYou can then use the sssctl config-check command to analyze the SSSD configuration files.
This command validates the /etc/sssd/sssd.conf and /etc/sssd/conf.d/* files.
The sssctl config-check command performs the following checks on the configuration files:
It verifies that the files have the correct permissions. File ownership must be set to
root:rootand the permissions to 0600.It reviews the content of the
/etc/sssd/conf.d/directory. The file names must use the.confextension, and they cannot be hidden files (files that start with a dot).The configuration file is reviewed for typographical errors in section and option names.
Options are reviewed for correct placement in the appropriate sections of the configuration file.
The following example provides sample output with multiple errors in the SSSD configuration file and the wrong file type in the /etc/sssd/conf.d directory:
[root@client ~]#sssctl config-checkIssues identified by validators: 2[rule/allowed_sections]: Section [sUdo] is not allowed. Check for typos. [rule/allowed_domain_options]: Attribute 'offline_temounts' is not allowed in section 'domain/example.com'. Check for typos.Messages generated during configuration merging: 1File .hidden-config-file.conf did not match provided patterns. Skipping. Used configuration snippet files: 0
SSSD uses a number of different log files in the /var/log/sssd directory.
Additional authentication failures and reasons for the failure might be in the /var/log/secure log file.
Should you need more information about the potential problem, you can increase the default log level by setting the debug_level parameter for each section in the sssd.conf file.
The following example illustrates setting the parameter to the maximum output value of 9:
[domain/example.com] id_provider = ipa ipa_server =srv, idmserver.example.com ipa_domain = example.com ipa_hostname = client.example.com auth_provider = ipa chpass_provider = ipa access_provider = ipa cache_credentials = True ldap_tls_cacert = /etc/ipa/ca.crt krb5_store_password_if_offline = Truedebug_level = 9
You can also increase the debug level to the maximum output level when SSSD is running.
To do this, use the sss_debuglevel 9 command.
Kerberos issues usually present in two ways: users cannot obtain a ticket-granting ticket (TGT) to log in to servers, and users or services cannot authenticate to services.
Refer to the following list to start troubleshooting login issues:
Are the times between all parties synchronized? A time difference of more than five minutes causes Kerberos to fail.
Does a
kinitcommand for a user work from all configured machines? If it works from some machines but not others, then there is a problem with the authentication configuration, which can typically be fixed with the SSSD family of tools. If it does not work from any machine, then there might be an issue with the KDC itself, or with the Kerberos configuration files on the machine.usernameDoes the
default_realmsetting, and the mapping of hostnames to machines in thedomain_realmsection of/etc/krb5.conf, match the actual values used in your environment?Are the DNS records for Kerberos correct? Is the DNS server reachable?
Are the
krb5_realmandkrb5_serversettings in/etc/sssd/sssd.confcorrect?The
/var/log/securelog file and thejournalctlcommand contains messages regarding failed logins; do they point to a likely cause?
When users can log in but cannot authenticate to certain services, or service-to-service authentication, such as NFSv4, then you might need to review other conditions to determine the problem.
Can the services read their keytab? For SSH and NFS and other similar services, you can review the
/etc/krb5.keytabfile. The actual file name depends on the service involved.Is the keytab using the most recent version of the password? Keytabs can be inspected with the
klist -ekcommand. The KVNO column shows the version of the password stored. Whenever a principal is added to a keytab, a new random password is generated automatically.FILENAMEAre the correct principals present in the keytab?
Do services agree on how Kerberos should be used? For example, an NFS export that specifies
sec=krb5pcan only be mounted when the client specifies a matchingkrb5psecurity level in its mount options.Are all needed helper services running? For example, an NFS server needs the
nfs-secure-server.serviceservice, and an NFS client needs thenfs-secure.serviceservice when using Kerberos-enabled NFSv4.
When users report problems with requesting Kerberos tickets, start troubleshooting with debug messages from the kinit command:
[user@client ~]$ KRB5_TRACE=/dev/stdout kinit adminMake sure that the server entries in /etc/hosts file are correct.
Verify that the DNS client configuration in the /etc/resolv.conf file points to the correct DNS server and has the correct search domain.
Use the host command to run DNS queries for both the client hostname and IP address, as well as the IdM server hostname and IP address.
On the IdM server, ensure that all required services are running.
You can do this by using the ipactl status command.
Specifically for Kerberos-related problems, ensure that the krb5kdc and dirsrv services are running.
On the IdM server, review the /var/log/krb5kdc.log Key Distribution Center (KDC) log file.
Make sure that DNS discovery is enabled in the /etc/krb5.conf file.
The dns_lookup_kdc parameter must be set to true.
Use dig to query the SRV records related to Kerberos:
[user@client ~]$dig -t SRV _kerberos._tcp.example.com...output omitted... _kerberos._tcp.example.com. 86400 IN SRV 0 100 88 idm.example.com. ...output omitted... [user@client ~]$dig -t SRV _kerberos._udp.example.com...output omitted... _kerberos._tcp.example.com. 86400 IN SRV 0 100 88 idm.example.com. ...output omitted...
You can also verify that the IdM services are running on a specific machine by using the curl command.
You do not need to authenticate to get an IdM service response; you can just fetch the headers of the response and review the output.
In the following example, the idmreplica01 machine returns a 401 error code, which means that the service is running but it could not authenticate the request.
The command for the idmreplica02 machine request fails completely because the service is not running or it is unreachable:
[user@host ~]$curl -I https://idmreplica01.example.com/ipaHTTP/1.1 401 Unauthorized Date: Wed, 12 Jul 2023 23:32:24 GMT Server: Apache/2.4.53 (Red Hat Enterprise Linux) OpenSSL/3.0.1 mod_auth_gssapi/1.6.3 mod_wsgi/4.7.1 Python/3.9 WWW-Authenticate: Negotiate X-Frame-Options: DENY Content-Security-Policy: frame-ancestors 'none' Last-Modified: Thu, 04 Nov 2021 13:12:28 GMT Accept-Ranges: bytes Content-Length: 2719 Content-Type: text/html; charset=UTF-8 [user@host ~]$curl -I https://idmreplica02.example.com/ipacurl: (7) Failed to connect to idmreplica02.example.com port 443: Connection refused
When troubleshooting Sudo with SSSD, Red Hat recommends that you start by enabling the debug logging feature in the /etc/sudo.conf configuration file.
Enabling the debug feature allows you to gather additional information about SSSD and Sudo in the logs.
Debugging is disabled by default; to enable it, add the following lines to the /etc/sudo.conf file:
...output omitted... Debug sudo /var/log/sudo_debug_file.logall@debug Debug sudoers.so /var/log/sudo_debug_file.logall@debug ...output omitted...
After it is enabled, run the sudo command as the user that reported the problem.
The log file is automatically created and provides additional information that can help you determine the cause of the problem, such as:
Additional information about the user and the environment when running the
sudocommandThe data source of the Sudo rules
The number of rules that SSSD returned
A report about matching rules and those that are designed to match but do not
Always enable SSSD debugging when investigating problems.
The /var/log/sssd/sssd domain log file provides additional information, such as:_DOMAIN_NAME.log
The number of rules returned by SSSD
A list of rules downloaded by SSSD
Filters used to download the rules
Whether the rules are stored successfully in cache
References
For more information, refer to the Eliminating Typographical Errors in Local SSSD Configuration chapter in the Configuring Authentication and Authorization in RHEL guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/configuring_authentication_and_authorization_in_rhel/index#eliminating-typographical-errors-in-local-SSSD-configuration_restricting-domains-for-pam-services-using-sssd
For more information, refer to the Troubleshooting Authentication with SSSD in IdM chapter in the Configuring Authentication and Authorization in RHEL guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/configuring_authentication_and_authorization_in_rhel/index#assembly_troubleshooting-authentication-with-sssd-in-idm_restricting-domains-for-pam-services-using-sssd
