Red Hat Enterprise Linux Diagnostics and Troubleshooting
Enterprise computing environments typically consist of hundreds or thousands of interconnected computer systems. These systems run many applications and services, and have many local and remote users who access these systems daily.
Historically, the user accounts, application permissions, and security policies were configured with a diverse set of protocols and utilities, such as the Network Information Service (NIS), Kerberos, Lightweight Directory Access Protocol (LDAP), sudo, and TCP Wrappers. Each of these tools was locally managed and separately administered.
Identity management solutions address this administrative ordeal by creating and configuring users, systems, services, and policies with centralized storage and a single set of tools that is built on existing, native Linux protocols and technologies.
Red Hat Identity Management (IdM) implements identity and access management with central authentication management of identities and security mechanisms, fine-grained access control policies, integrated Public Key Infrastructure (PKI) services, two-factor authentication (2FA) support, cross-realm Kerberos trusts with Active Directory (AD), and a feature for direct client connections to AD.
Although the IdM toolset includes all of these services, you can select to install only the services that you require. Some services can be distributed to additional, external servers for scaling, and be integrated with the IdM domain.
The Kerberos and LDAP services are always installed and are integral to every IdM server. The ipa command uses Kerberos and LDAP to search for user information. The ipa command is an integrated security information management solution that includes a web interface and command-line administration tools for managing identity data.
For example, to search for user information, use the kinit command to obtain a Ticket Granting Ticket (TGT), and then use the ipa command to gather user information.
Obtain a TGT:
[user@host ~]$kinit idmuserPassword for idmuser@LAB.EXAMPLE.COM: [user@host ~]$klistTicket cache: KCM:1000 Default principal: idmuser@LAB.EXAMPLE.COM Valid starting Expires Service principal 11/21/2021 23:17:35 11/22/2021 23:17:33 krbtgt/LAB.EXAMPLE.COM@LAB.EXAMPLE.COM
Gather user information:
[user@host ~]$ ipa user-find idmuser
--------------
1 user matched
--------------
User login: idmuser
First name: idmuser
Last name: idm
Home directory: /home/idmuser
Login shell: /bin/sh
Principal name: idmuser@LAB.EXAMPLE.COM
Principal alias: idmuser@LAB.EXAMPLE.COM
Email address: idmuser@example.com
UID: 546600020
GID: 546600020
Account disabled: False
----------------------------
Number of entries returned 1
----------------------------Lightweight Directory Access Protocol (LDAP) is a protocol for accessing, querying, and updating directory services. It evolved out of the need to simplify the X.509 directory services protocol and related services.
LDAP organizes objects, such as user accounts, and their related attributes, such as passwords and full name, in a tree. The attributes that are used on an object depend on the associated object classes with that object. In a directory, the same object classes and attributes are associated with all objects that represent the same thing, such as a user or a computer.
LDAP is an essential part of common identity management solutions, such as Red Hat Identity Management, Red Hat Directory Server, and Microsoft Active Directory.
Kerberos provides authentication services and a Single Sign-On (SSO) service to users and services. Kerberos assumes that individual machines on the network are trusted but that the network itself is insecure. Therefore, never transmit passwords across the network.
A typical Kerberos transaction involves three parties, where two parties who want to conduct transactions, but do not have a trusted relationship, use a third-party intermediary who is trusted by both parties to provide and verify a secure introduction. The Kerberos Key Distribution Center is the trusted third party for a user who wants to begin transactions with a service.
The Kerberos Key Distribution Center (KDC). The KDC server knows the passwords for all users and services. A principal (a unique identifier for tickets), name, and password identify each user and service.
The user who wants to authenticate. This user must know their password.
The service that the user wants to authenticate with. Because services cannot interactively type their passwords, a file named
keytabstores the service's password.- Initial authentication
When a user with a Kerberos principal configuration logs in, they are authenticated and provided a Ticket Granting Ticket (TGT). This TGT permits the user to obtain service tickets without having to enter their password again, for the duration of the TGT (typically a few hours). The TGT is obtained automatically when interactively logging in to a machine that uses Kerberos, or by using the
kinitcommand-line tool.Obtaining a TGT follows these steps:
At login, the system requests from the KDC a TGT for the user.
The KDC responds with a new TGT, which is encrypted with the user's password.
If the TGT can be decrypted with the user's typed password, then the login succeeds and the TGT is stored.
- Authentication with a Ticket
While a user has a valid TGT, they can connect to other Kerberos-enabled services without typing a password.
Authentication happens according to these steps:
The client requests from the KDC the service ticket for the service that they want to access.
The KDC responds with two copies of the same service ticket: one is encrypted with the user's TGT, and the other is encrypted with the password for the service.
The client decrypts the TGT-encrypted version, and uses the decrypted ticket to encrypt a timestamp.
The client sends to the service the encrypted timestamp and the encrypted ticket with the service's password.
The service decrypts the service ticket, knowing that it can come only from the KDC, and uses it to decrypt the timestamp.
If the timestamp decryption succeeds and is less than two minutes old, then the user is considered to be authenticated.
- A note on principals
In the simplest form, a user's principal name uses the
username@REALMformat, for examplevimes@EXAMPLE.COM. User names typically match an LDAP or local system user, and the realm typically maps to DNS domains as the organization defines.Computer and service principals take a slightly different format, such as
service/hostname@REALM. For example:host/host.example.com@EXAMPLE.COM,nfs/host.example.com@EXAMPLE.COM, orhttp/www.example.com@EXAMPLE.COM. The service type determines the name of the service. Authentication services such as SSH and telnet usehost/; NFS services usenfs/; and web services usehttp/. Other services often use unique prefixes.
The System Security Services Daemon (SSSD) is a system service to access remote directories and authentication mechanisms. It connects a local SSSD client system to an external back-end provider system for remote identity and authentication services. Example remote services include an LDAP directory, an Identity Management (IdM) or Active Directory (AD) domain, or a Kerberos realm.
When a user authenticates to a service, the SSSD daemon handles the authentication process:
SSSD connects the client to an identity store to retrieve authentication information.
SSSD uses the obtained authentication information to create a local cache of users and credentials on the client.
Users on the local system can then authenticate with the stored user accounts in the external back-end system. SSSD does not create user accounts on the local system. Instead, it uses the identities from the external data store and lets the users access the local system.
SSSD can provide caches for several system services, such as Name Service Switch (NSS) or Pluggable Authentication Modules (PAM).
SSSD capabilities:
- Reduce the load on identity and authentication servers
When requesting information, SSSD clients contact SSSD, which checks its cache. SSSD contacts the servers only if the information is not available in the cache.
- Utilize offline authentication
SSSD optionally keeps a cache of user identities and credentials that are retrieved from remote services. In this setup, users can successfully authenticate to resources even if the remote server or the SSSD client is offline.
- Implement single user accounts to improve consistency of the authentication process
With SSSD, you do not need to maintain a central account and a local user account for offline authentication.
Remote users often have multiple user accounts. For example, to connect to a virtual private network (VPN), remote users have one account for the local system and another for the VPN system.
Remote users can connect to network resources simply by authenticating to their local machine, thanks to caching and offline authentication. SSSD then maintains their network credentials.
The most common LDAP issues can be categorized:
- Network connectivity
LDAP uses TCP port 389 for both clear text and TLS-encrypted connections. Older LDAP versions supported TCP port 636 for SSL-encrypted connections. If a firewall is blocking port access, or if an older client attempts to connect to port 636/TCP on a server that supports only STARTTLS on port 389/TCP, then connections fail.
- Mismatched security settings
LDAP servers can set restrictions on how clients connect and authenticate. Some servers might refuse to connect with clients that do not use TLS encryption, or might require either simple or SASL (Simple Authentication and Security Layer) authentication before responding to queries. You can use combinations of these requirements on individual actions. For example, a server can allow only TLS-encrypted connections for queries and require authentication for updates.
- Wrong search base
Clients typically avoid searching the entire LDAP data tree, and limit queries to the branches that store the requested data. For example, a login program would query the part of the tree with user accounts, not the part with computer accounts. If the search base is incorrectly configured, then searches might be slow or not return the needed information.
Administrators can use the getent password command to troubleshoot user information issues.
Note
When using SSSD for user information and authentication, the getent passwd command lists only local users, not all available users. For testing, add the name of a known network account to test whether network users resolve correctly. For example, getent passwd ldapuser.
The two most common Kerberos issues are that users cannot obtain a TGT and fail to log in, and that users or services cannot authenticate to services.
To troubleshoot login issues, an administrator can verify basic configuration:
Are the times synchronized between all parties? A time drift of more than two minutes causes all Kerberos validations to fail.
Does a
kinitfor that user work from the same machine? Ifusernamekinitdoes work, then verify that the PAM configuration files are correctly configured. PAM issues are typically fixed by using theauthselectfamily of tools. Ifkinitdoes not work, then verify that the KDC service is working, and the Kerberos configuration files on the machine.If DNS is used to store the Kerberos records, then are the
SRVrecords configured correctly?The
/var/log/securelopg file and thejournalctlcommand can provide messages regarding failed logins.
When users can log in, but cannot authenticate to certain services, or service-to-service authentication such as NFSv4 fails, additional configuration must be verified.
Can the services read their keytab file? Services such as SSH and NFS use
/etc/krb5.keytab, while other services might use custom keytab file names.Is the keytab using the most recent version of the password? Keytabs can be inspected with the
klist -ekcommand. The<FILENAME>KVNOcolumn shows the version of the stored password. Whenever a principal is added to a keytab, a new, random password is generated automatically.Users with access to the
kadminKerberos KDC management tool (orkadmin.localforrooton the KDC) can use thegetprinccommand from within<PRINCIPAL>kadminto view the password version as recognized by the KDC.Are the correct principals present in the keytab?
Do services agree on how to use Kerberos? For example, an NFS export that specifies
sec=krb5pcan be mounted only when the client also specifies the same security level.Are all needed helper services running? For example, an NFS server needs the
nfs-secure-serverservice, and an NFS client needs thenfs-secureservice when using Kerberos-enabled NFSv4.
On an IdM client machine, SSSD stores a cache of user identities and credentials that are retrieved from the IdM server. SSSD also authenticates online against LDAP or Kerberos and applies an access and password policy to the user who attempts to log in to services. Troubleshooting SSSD issues might involve the IdM server, or the IdM client, or both.
Common command-line tools for troubleshooting SSSD issues include the sssctl command to obtain information about SSSD status, active servers, domains, and cached objects. Another useful tool is the sss_cache command, which invalidates records in SSSD cache to target specific trouble areas more easily.
If you experience issues when attempting to authenticate as an IdM user to an IdM server, then enable detailed debug logging in the SSSD service on the server. Then, gather logs of attempts to retrieve information about the user.
You need the root password to run the sssctl command and restart the SSSD service.
Enable detailed SSSD debug logging on the IdM server.
[root@idmserver ~]# sssctl debug-level 6Invalidate objects in the SSSD cache for the user who is experiencing authentication issues. Invalidating the cache ensures that you do not bypass the LDAP server and retrieve information that SSSD already cached.
[root@idmserver ~]# sss_cache -u idmuserRemove older SSSD logs to minimize the troubleshooting data set.
[root@idmserver ~]# sssctl logs-removeAttempt to switch to a user with authentication problems. Use the
datecommand to generate timestamps before and after the attempt. These timestamps narrow the scope of the data set.
[root@idmserver ~]# date; su idmuser; dateReview SSSD logs for information about the failed request.
[root@idmserver ~]# cat /var/log/sssd/sssd_example.com.logIf you experience issues when attempting to authenticate as an IdM user to an IdM client:
Verify that you can retrieve user information on the IdM server.
Ensure that you have
rootuser access to run thesssctlcommand and restart the SSSD service.On the IdM client, open the
/etc/sssd/sssd.conffile in a text editor. Add theipa_serveroption to the[domain]section of the file and set it to an IdM server. This step avoids the IdM client autodiscovering other IdM servers, thus limiting this test to one client and one server.
[domain/example.com] ipa_server = server.example.com
Restart the SSSD service to load the configuration changes.
[root@idmclient ~]# systemctl restart sssdEnable detailed SSSD debug logging.
[root@idmclient ~]# sssctl debug-level 6Invalidate objects in the SSSD cache for the user with authentication issues, so that you do not bypass the LDAP database and retrieve information that SSSD already cached.
[root@idmclient ~]# sss_cache -u idmuserRemove older SSSD logs to minimize the troubleshooting data set.
[root@idmclient ~]# sssctl logs-removeAttempt to switch to a user with authentication problems. Use the
datecommand to generate timestamps before and after the attempt. These timestamps narrow the scope of the data set.
[root@idmclient ~]# date; su idmuser; dateReview SSSD logs for information about the failed request.
[root@idmclient ~]# cat /var/log/sssd/sssd_example.com.logReferences
sssctl(8), sss_cache(8), sssd.conf(5), krb5.conf(5), ipa(1), and kadmin(1) manual pages.
For further information, refer to Configuring and Managing Identity Management guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_and_managing_identity_management/index
