Red Hat Enterprise Linux Diagnostics and Troubleshooting
Software applications generally depend on other libraries, configuration files, or services that the runtime environment provides. The traditional runtime environment for a software application is a physical host or virtual machine, and application dependencies get installed as part of the host.
A container is a set of one or more processes that are isolated from the rest of the system. This figure describes the difference between applications that run as containers and applications that run on the host operating system.
The isolation process takes advantage of certain features of the Linux kernel:
- Namespaces
A namespace isolates specific system resources that are usually visible to all processes. Inside a namespace, only processes that are members of that namespace can see those resources. Namespaces can include resources such as network interfaces, the process ID list, mount points, IPC resources, and the system's host name information.
- Control groups (cgroups)
Control groups partition sets of processes and their children into groups to manage and limit the resources that they consume. Control groups place restrictions on the amount of system resources that processes might use. Those restrictions keep one process from using too many resources on the host.
- SELinux
Security-Enhanced Linux (SELinux) is a mandatory access control system for processes. Linux kernel uses SELinux to protect processes from each other and to protect the host system from its running processes. Processes run as a confined SELinux type that has limited access to host system resources.
From the Linux kernel perspective, a container is a process with restrictions. Instead of running a single binary file, a container runs an image. An image is a file-system bundle that contains all the required dependencies to execute a process, including files in the file system, installed packages, available resources, running processes, and kernel modules.
Running containers use an immutable view of the image, so that multiple containers can reuse the same image simultaneously.
Container images must be locally available for the container runtime to execute them, but the images are usually stored and maintained in an image repository. An image repository is a public or private service to store, search, and retrieve the images. Image repositories provide remote access, image metadata, image authorization, and image version control.
Podman is an open source tool for managing containers and container images and interacting with image registries. It offers the following key features:
It uses the specified image format of the Open Container Initiative (OCI) at https://opencontainers.org/. Those specifications define a standard, community-driven, non-proprietary image format.
Podman stores local images in a local file system. Doing so avoids unnecessary client/server architecture or running daemons on a local machine.
Podman follows the same command patterns as the Docker CLI.
The podman command provides subcommands to create, manage, and update containers, and to get information about both running and stopped containers for use in container debugging.
Red Hat distributes container images primarily from two locations: registry.access.redhat.com, with no authentication required, and registry.redhat.io, which requires an account for authentication. Although both registries contain essentially the same container images, images that require a subscription are available only from registry.redhat.io.
To retrieve content from an authenticated registry, log in to the registry with a Customer Portal, Red Hat Developer, or Registry Service Account credentials. Use podman login to authenticate to registry.redhat.io.
[root@host ~]#podman login registry.redhat.ioUsername:redhat-account-usernamePassword:********Login Succeeded!
To test whether you have functional basic authentication, query the API with the curl command. A successful authentication attempt results in HTTP 200 OK.
[root@host ~]# curl -Lv -u redhat-account-username:******** "https://sso.redhat.com/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&client_id=curl&scope=repository:rhel:pull"
* Trying 104.75.112.205...
* TCP_NODELAY set
* Connected to sso.redhat.com (104.75.112.205) port 443 (#0)
...output omitted...
* SSL certificate verify ok.
* Server auth using Basic with user 'redhat-account-username'
> GET /auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&client_id=curl&scope=repository:rhel:pull HTTP/1.1
> Host: sso.redhat.com
> Authorization: Basic cmhuLXN1cHBvcnQtYWNhbGxlamE6TXVsdDF2NGMh
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 200 OK
...output omitted...To consume container images from registry.redhat.io in shared environments, it is recommended to create and use a Registry Service Account, also referred to as an authentication token, instead of using an organization's or an individual's Customer Portal credentials.
Note
To create an authentication token, refer to the Creating Registry Service Accounts section of the Red Hat Container Registry Authentication article at https://access.redhat.com/RegistryAuthentication.
To test basic authentication with an authentication token, store the token as a system variable and reference it with a curl command. A successful authentication attempt results in HTTP 200 OK and a JSON object.
[root@host ~]#export TOKEN="[root@host ~]#account_number|name"export SECRET="[root@host ~]#token_value"curl -u $TOKENID:$SECRET "https://sso.redhat.com/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&client_id=curl&scope=repository:rhel:pull"{"token":"_tokenvalue_","access_token":"_tokenvalue_", "expires_in":300,"issued_at":"2021-10-19T04:18:16Z"}
When the basic authentication test is successful, use the access token to test connectivity with the registry itself.
[root@host ~]#curl -Lv -u $TOKENID:$SECRET "https://sso.redhat.com/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&client_id=curl&scope=repository:rhel:pull" | python3 -m json.tool% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 * Trying 104.75.112.205... * TCP_NODELAY set * Connected to sso.redhat.com (104.75.112.205) port 443 (#0) ...output omitted... < HTTP/1.1 200 OK ...output omitted... 100 2168 100 2168 0 0 3269 0 --:--:-- --:--:-- --:--:-- 3269 * Connection #0 to host sso.redhat.com left intact { "token": "eyJhbGciOiJSUzI1NiJ9...", "access_token": "eyJhbGciOiJSUzI1NiJ9...", "expires_in": 300, "issued_at": "2021-10-19T04:39:52Z" }
Podman can mount a host directory inside a running container, as in a standard volume mount. In this way, the containerized application uses the host directories as the container storage, like a normal application's use of network shared storage. When the container exits and is removed, the container's directory is unmounted, and the source directory and contents remain on the host, allowing new containers to mount the directory and access the contained data.
Directory configuration involves:
Configuring the ownership and permissions of the directory.
Setting the appropriate SELinux context.
Podman uses the SELinux container_file_t context type to control which files on the host system a container is allowed to access.
To mount a host directory to a container, add the --volume (or -v) option, and specify the host directory path and the container storage path, separated by a colon.
[root@host ~]#podman run -d --name mydb -v /home/user/dbfiles:/var/lib/mysql:Z-e MYSQL_USER=user -e MYSQL_PASSWORD=redhat -e MYSQL_DATABASE=inventoryregistry.redhat.io/rhel8/mariadb-103:1-102
Podman subcommands manage a container environment, determining the container health, viewing system information, and monitoring Podman events.
The healthcheck subcommand verifies the health or readiness of the process that runs inside the container. A health check consists of the following components, of which only the first one is required and the others are optional scheduling parameters:
- Command
Executes a command inside the target container and waits for the exit code to determine the container health.
- Retries
Defines the number of consecutive failed health checks that must occur before the container is marked as unhealthy. A successful health check resets the retry counter.
- Interval
Specifies the time interval between each run of the health check command. With small intervals, the system spends much time with running health checks. With large intervals, it is difficult to catch timeouts.
- Start-period
Describes the elapsed time between the container starting and when it ignores health check failures.
- Timeout
Specifies the time interval in which the check must complete before it is considered as failed.
To define a container health check, add the command that confirms the health of the container to run inside the container along with the other processes in the image.
[user@host ~]$podman run --name test-httpd --health-cmd='curl http://localhost:8080 || exit 1' --health-interval=0 -dt -p 8080:8080 registry.redhat.io/rhel8/httpd-248a60...20fa
Verify the health of the running container with the podman healthcheck run command, which invokes the health command that you added to the container.container_name
[user@host ~]$ podman healthcheck run test-httpd
healthyPodman subcommands can monitor and display the events that occur in a container environment. Each event includes a timestamp, a type, a status, and when applicable, a name and an image.
The event logging mechanism is configured in the /usr/share/containers/containers.conf file. The default mechanism is file, and can be changed to journald to send events to the system journal. Using a value of none disables Podman event reporting.
To view all events, verify journald as the logging mechanism and run the podman events command as a privileged user.
[root@host ~]#grep logger /usr/share/containers/containers.confevents_logger = "journald" [root@host ~]#podman eventsOct 19 12:45:24 host.example.com podman[3415]: 2021-10-19 12:45:24.108408331 -0400 EDT m=+0.063967269 container create 9af5...d8ef (image=registry.redhat.io/rhel8/httpd-24:latest, name=test-httpd, ...output omitted... Oct 19 12:45:24 host.example.com podman[3415]: 2021-10-19 12:45:24.080958781 -0400 EDT m=+0.036517732 image pull registry.redhat.io/rhel8/httpd-24 Oct 19 12:45:24 host.example.com podman[3415]: 2021-10-19 12:45:24.251704702 -0400 EDT m=+0.207263649 container init 9af5...d8ef (image=registry.redhat.io/rhel8/httpd-24:latest, name=test-httpd, ...output omitted...
This example adds a filter to show only create events that occurred in the last five minutes.
[root@host ~]# podman events --since 5m --filter event=create
2021-10-19 12:45:24.108408331 -0400 EDT container create 9af5...d8ef (image=registry.redhat.io/rhel8/httpd-24:latest,
...output omitted...Note
The event output length varies depending on the type of event and the container image's configuration.
A rootless container runs as a non-root user, but runs processes and interacts with files as a privileged user with the concept of a user namespace. A user namespace is a Linux facility where a non-root user can map a range of UIDs and GIDs on the host system to a different range of UIDs and GIDs within the namespace.
With rootless containers, the running user must have the mapped ID ranges that are listed in the /etc/subuid and /etc/subgid files.
[root@host ~]#cat /etc/subuiduser:100000:65536 [root@host ~]#cat /etc/subgiduser:100000:65536
The unshare subcommand creates a namespace to run a specified program, instead of sharing the current namespace. With podman unshare, you are leaving your normal namespace, and working in a new Podman user namespace.
View the mappings by displaying the /proc/self/uid_map and /proc/self/gid_map files from inside the user namespace, by using the podman unshare command as the non-root user.
[user@host ~]$podman unshare cat /proc/self/uid_map0 3267 1 1 100000 65536 [user@host ~]$podman unshare cat /proc/self/gid_map0 3267 1 1 100000 65536
Use the podman generate systemd command to create a systemd unit file for a specified service container. The systemd unit automatically starts, stops, enables, or disables a containerized service on the host system.
[user@host ~]$ podman generate systemd --name web --files --new
/home/user/container-web.serviceMove the unit file to the user's systemd configuration directory.
[user@host ~]$ mv container-web.service /home/user/.config/systemd/user/Inspect the unit file to verify the service's systemd structure and parameter settings.
[user@host ~]$ cat /home/user/.config/systemd/user/container-web.service
# container-web.service
# autogenerated by Podman 3.3.1
# Wed Nov 17 20:41:44 CEST 2021
[Unit]
Description=Podman container-web.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers
[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm -f %t/%n.ctr-id
ExecStart=/usr/bin/podman run --cidfile=%t/%n.ctr-id --sdnotify=conmon --cgroups=no-conmon --rm -d --replace --name web -p 8080:8080 registry.access.redhat.com/ubi8/httpd-24
ExecStop=/usr/bin/podman stop --ignore --cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all
[Install]
WantedBy=multi-user.target default.targetTo manage the containerized service, use the systemctl command.
[user@host ~]$ systemctl --user [start, stop, status, enable, disable] container_nameAfter modifying the unit file to change the service configuration, implement the change by requesting systemd to reload its configuration.
[user@host ~]$ systemctl daemon-reloadPodman can inspect logs in running containers to assist with troubleshooting. Podman ignores custom application-specific logs and assumes that containerized applications send all log output to STDOUT.
A container is a process tree from the perspective of the host operating system. When Podman starts a container, it redirects the container's STDOUT and STDERR, saving them to disk in the container's ephemeral storage. The container's log files can be displayed with the podman logs command. Until deleting the container, the container logs remain, because the container instance resides in memory and its ephemeral storage still exists. Deleting a container also deletes its ephemeral storage, making the instance logs unavailable.
[user@host ~]$ podman logs test-httpd
[Tue Oct 19 16:45:24.626604 2021] [:notice] [pid 1:tid 140604915785152] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.0.2.100. Set the 'ServerName' directive globally to suppress this message
[Tue Oct 19 16:45:24.709771 2021] [ssl:warn] [pid 1:tid 140604915785152] AH01909: 10.0.2.100:8443:0 server certificate does NOT include an ID which matches the server name
[Tue Oct 19 16:45:24.715306 2021] [core:notice] [pid 1:tid 140604915785152] AH00094: Command line: 'httpd -D FOREGROUND'
10.0.2.100 - - [19/Oct/2021:18:22:18 +0000] "GET / HTTP/1.1" 403 4481 "-" "curl/7.61.1"
[Tue Oct 19 18:22:24.120915 2021] [autoindex:error] [pid 59:tid 140603983611648] [client 127.0.0.1:50552] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html) found, and server-generated directory index forbidden by Options directive
...output omitted...Podman writes all output to STDOUT, and by default uses an error log level. Podman supports the debug, info, warn, error, fatal, and panic standard logging levels.
[user@host ~]$ podman logs --log-level info test-httpd
INFO[0000] podman filtering at log level info
INFO[0000] Setting parallel job count to 7
=> sourcing 10-set-mpm.sh ...
=> sourcing 20-copy-config.sh ...
=> sourcing 40-ssl-certs.sh ...
---> Generating SSL key pair for httpd...
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.0.2.100. Set the 'ServerName' directive globally to suppress this message
[Tue Oct 19 16:45:24.625273 2021] [ssl:warn] [pid 1:tid 140604915785152] AH01909: 10.0.2.100:8443:0 server certificate does NOT include an ID which matches the server name
[Tue Oct 19 16:45:24.626589 2021] [:notice] [pid 1:tid 140604915785152]+ ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/) configured.
...output omitted...References
Podman
Use of Podman in a Rootless Environment
https://github.com/containers/podman/blob/master/docs/tutorials/rootless_tutorial.md
For further information, refer to the Red Hat Enterprise Linux 8 Building, running, and managing containers Guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/building_running_and_managing_containers/index
