Bookmark this page

Chapter 8. Troubleshooting Application Issues

Abstract

Goal	Identify and resolve application issues with debugging tools.
Objectives	Identify library dependencies for installed software. Identify applications that have memory leaks. Debug an application with standard RHEL tools. Troubleshoot a containerized application or service.
Sections	Resolving Library Dependencies (and Guided Exercise) Debugging Memory Leaks (and Guided Exercise) Debugging Application Execution (and Guided Exercise) Troubleshooting Containerized Applications (and Guided Exercise)
Lab	Troubleshooting Application Issues

Resolving Library Dependencies

Objectives

Identify library dependencies for installed software.

Using Shared Libraries in Applications

Most Linux applications are dynamically linked against the shared libraries that they use. Shared libraries are built so that their functions can be mapped into an application's memory when the application is executed. Because the required code is provided by the library at runtime, it is not copied into the application itself.

The dynamic linking of applications with shared libraries has many benefits:

Application binaries are smaller due to the external shared code.
Libraries can be shared simultaneously with many running programs.
Shared libraries make it easier to fix bugs in the library code, without requiring application rebuilds.

Linking against Shared Libraries

When an application is built, it is linked against the shared libraries that provide the functions that it uses. The compiler checks the libraries for required symbols and verifies that they exist.

Identifying information about each required library is embedded in the executable. This information can be the absolute path name of the library, but it is more commonly the DT_SONAME field of the shared library. The DT_SONAME field includes the name and version of the shared library, so that the application uses the correct version at runtime. The ln linker with the -soname option` sets this field when creating the shared library. The objdump command includes this field when displaying shared library information.

[user@host ~]$ objdump -p /usr/lib64/libpthread-2.28.so | grep SONAME
  SONAME               libpthread.so.0

The shared library uses a symbolic link that is named with the DT_SONAME field.

[user@host ~]$ ls -l /usr/lib64/libpthread*
-rwxr-xr-x. 1 root root 320704 Mar  5  2021 /usr/lib64/libpthread-2.28.so
lrwxrwxrwx. 1 root root     18 Mar  5  2021 /usr/lib64/libpthread.so.0 -> libpthread-2.28.so

When an application executes, the runtime linker identifies the required shared libraries and maps them into the program's memory space. On Red Hat Enterprise Linux 8, the /lib64/ld-linux-x86-64.so.2 library is the default, 64-bit version of the runtime linker. The glibc.x86_64 package provides the runtime linker. The 32-bit version of the glibc library provides the 32-bit version of the runtime linker when it is installed.

[user@host ~]$ rpm -qlp glibc-2.28-151.el8.i686.rpm  | grep ld-linux
/lib/ld-linux.so.2

The runtime linker uses the embedded DT_SONAME names in the application to determine which library versions to use. The runtime linker searches the following items, in order, to determine which library version to use:

The specified directories in the LD_LIBRARY_PATH environment variable. This step is ignored for setUID and setGID applications. This environment variable is often used when developing or testing an application.

The /etc/ld.so.cache cache file. This file contains a compiled list of previously found candidate libraries. The ldconfig -p command prints the list of libraries that are mapped in /etc/ld.so.cache.

[user@host ~]$ ldconfig -p
566 libs found in cache `/etc/ld.so.cache'
	p11-kit-trust.so (libc6,x86-64) => /lib64/p11-kit-trust.so
	libzstd.so.1 (libc6,x86-64) => /lib64/libzstd.so.1
	libz.so.1 (libc6,x86-64) => /lib64/libz.so.1
	libyaml-0.so.2 (libc6,x86-64) => /lib64/libyaml-0.so.2
	libyajl.so.2 (libc6,x86-64) => /lib64/libyajl.so.2
	libxtables.so.12 (libc6,x86-64) => /lib64/libxtables.so.12
...output omitted...

The directories in the default library paths. The 64-bit shared library loader points to 64-bit versions of the default locations, /lib64 and then /usr/lib64. The 32-bit runtime linker searches in turn the /lib and the /usr/lib directories.

[user@host ~]$ strings /lib64/ld-linux-x86-64.so.2  | grep '^/'
/usr/libH9
/t	E
/t	E
/usr/libI9
/uzH
/var/tmp
/var/profile
/lib64/
/usr/lib64/
/etc/suid-debug
/%x %s
/proc/self/exe
/etc/ld.so.cache
/proc/sys/kernel/osrelease
/dev/full
/dev/null
/etc/ld.so.preload

The ldd command displays the shared libraries that are required by the specified application at runtime. Each library's DT_SONAME name is displayed, followed by the path name to the library.

[user@host ~]$ ldd /usr/sbin/httpd
	linux-vdso.so.1 (0x00007ffe4f7bd000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f5994c55000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f5994a2b000)
	libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f59946e4000)
	libaprutil-1.so.0 => /lib64/libaprutil-1.so.0 (0x00007f59944b6000)
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f599428d000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f5994052000)
	libapr-1.so.0 => /lib64/libapr-1.so.0 (0x00007f5993e18000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5993bf8000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f59939f4000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f599362f000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f59933ab000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f5995154000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f59931a3000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f5992f7c000)
	liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f5992d5f000)
	libcap.so.2 => /lib64/libcap.so.2 (0x00007f5992b59000)
	libmount.so.1 => /lib64/libmount.so.1 (0x00007f59928ff000)
	libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007f59925e1000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f59923c9000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f59921c1000)
	libblkid.so.1 => /lib64/libblkid.so.1 (0x00007f5991f6e000)
	libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f5991d4d000)

Note

The absolute path name of a library is embedded in a program when the shared library does not have a DT_SONAME field. In the previous example, the /lib64/ld-linux-x86-64.so.2 reference points to the runtime linker itself.

Troubleshooting Library Dependency Issues

Library dependency problems can occur on a RHEL system when third-party software is installed with a method other than yum or rpm. Also, using the --force or --nodeps options with rpm can cause library dependency issues.

Library dependency issues are easy to identify. When an application references an unavailable shared library, the runtime linker displays a distinctive error message. The runtime linker displays the name of the first library that it cannot find, and then exits the application with an exit status of 127.

[user@host ~]$ application
application: error while loading shared libraries: library.so.X: cannot
 open shared object file: No such file or directory
[user@host ~]$ echo $?
127

An application might be missing more libraries than the runtime linker indicates. The ldd command displays all of the shared libraries that an application uses. If a library is missing, then the ldd command displays it as not found.

[user@host ~]$ which application
/usr/bin/application
[student@host ~]$ ldd /usr/bin/application
        linux-vdso.so.1 =>  (0x00007ffcbeba9000)
...output omitted...
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fcbd3e7d000)
        library1.so.2 => not found
        library2.so.0 => not found
        libbz2.so.1 => /lib64/libbz2.so.1 (0x00007fcbd409a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fcbd4aa7000)

If shared libraries are missing, then install them. RPM packages include shared library dependency information in the package's metadata.

[user@host ~]$ rpm -q --requires httpd | grep pthread
libpthread.so.0()(64bit)
libpthread.so.0(GLIBC_2.2.5)(64bit)

The yum provides command identifies the package that provides the specified shared library.

[user@host ~]$ yum provides '*/lib/libpthread.so.0'
Updating Subscription Management repositories.
glibc-2.28-151.el8.i686 : The GNU libc libraries
Repo        : rhel-8-for-x86_64-baseos-rpms
Matched from:
Filename    : /lib/libpthread.so.0
...output omitted...

If the shared library is not packaged and distributed by Red Hat, then contact the third-party vendor to obtain it. When a shared library is installed or removed, run the ldconfig command to update the runtime linker cache, /etc/ld.so.cache, with the changed information.

[user@host ~]$ rpm -q --scripts libnl3
postinstall program: /sbin/ldconfig
postuninstall program: /sbin/ldconfig

References

ld(1), ldconfig(8), ld.so(8), ldd(1), nm(1), and objdump(1) man pages

Discuss Red Hat Enterprise Linux Diagnostics and Troubleshooting

Go to community

Welcome to the Red Hat Enterprise Linux Diagnostics and Troubleshooting (RH342) group!

cschunke

26 wrz 2023

We are excited to launch a space dedicated to the Red Hat Training course Red Hat Enterprise Linux Diagnostics and Troubleshooting! To gain the most value from this group - click the "Join Group" button in the upper right hand corner of the group home page.We encourage group members to collaborate in this group to discuss topics, ask questions, share best practices and tips, provide course feedback, and share their accomplishments as it relates to RH342.Read more about Red Hat Enterprise Linux Diagnostics and Troubleshooting here.

445

Revision: rh342-8.4-6dd89bd

Click Create to build all of the virtual machines needed for the classroom lab environment. This may take several minutes to complete. Once created the environment can then be stopped and restarted to pause your experience.
When a lab is created, click Start to run all of the virtual machines in the classroom.
Click Stop to stop all the virtual machines from running. This will not delete your lab.
If you Delete your lab, you will remove all of the virtual machines in your classroom and lose all of your progress.

Virtual machine actions

Click Start to power on the virtual machine.
Click Shutdown to gracefully shut down the virtual machine, preserving disk contents.
Click Power off to forcefully shut down the virtual machine, while still preserving disk contents.
Click Open console to connect to the system console of the virtual machine in a new browser.

Auto-stop timer

The Red Hat Learning Subscription entitles you to set allotment of lab time.
To help conserving your allotted time, the lab environment uses automatic timers to stop or destroy your lab environment when the timer expires.

Click the Auto-stop button [+] to extend the time you would like to spend with the labs.
Click the Auto-destroy button [+] to add day(s) to the auto-destroy timer.

Auto-stop has a maximum of 11 hours, and auto-destroy has a maximum of 14 days.
Be careful to keep the timers set while you are working, so that your environment doesn't shut down unexpectedly.
We also suggest not to set the auto-timers unnecessarily high, which could waste your lab time allotment