Bookmark this page

Chapter 8.  Monitor and Manage Linux Processes

Abstract

Goal

Evaluate and control processes that run on a Red Hat Enterprise Linux system.

Objectives
  • Determine status, resource use, and ownership of running programs on a system, to control them.

  • Use Bash job control to manage multiple processes that were started from the same terminal session.

  • Use commands to kill and communicate with processes, define the characteristics of a daemon process, and stop user sessions and processes.

  • Define load average and determine resource-intensive server processes.

Sections
  • Process States and Lifecycle (and Quiz)

  • Control Jobs (and Guided Exercise)

  • Kill Processes (and Guided Exercise)

  • Monitor Process Activity (and Guided Exercise)

Lab
  • Monitor and Manage Linux Processes

Process States and Lifecycle

Objectives

  • Determine status, resource use, and ownership of running programs on a system, to control them.

Definition of a Process

A process is a running instance of a launched, executable program. From the moment that a process is created, it consists of the following items:

  • An address space of allocated memory

  • Security properties, including ownership credentials and privileges

  • One or more execution threads of program code

  • A process state

The environment of a process is a list of information that includes the following items:

  • Local and global variables

  • A current scheduling context

  • Allocated system resources, such as file descriptors and network ports

An existing parent process duplicates its own address space, which is known as a process fork, to create a child process structure. Every new process is assigned a unique process ID (PID) for tracking and security purposes. The PID and the parent's process ID (PPID) are elements of the new process environment. Any process can create a child process. All processes are descendants of the first system process, systemd, on a Red Hat system.

Figure 8.1: Process lifecycle

Through the fork routine, a child process inherits security identities, previous and current file descriptors, port and resource privileges, environment variables, and program code. A child process can then execute its own program code.

Normally, a parent process sleeps when the child process runs, and sets a wait request to be signaled when the child completes. After the child process exits, it closes or discards its resources and environment, and leaves a zombie resource, which is an entry in the process table. The parent, which is signaled to wake when the child exits, cleans the process table of the child's entry, and it frees the last resource of the child process. The parent process then continues with its own program code execution.

Describe Process States

In a multitasking operating system, each CPU (or CPU core) can be working on one process at a time. As a process runs, its immediate requirements for CPU time and resource allocation change. Processes are assigned a state, which changes as circumstances dictate.

The following diagram and table describe Linux process states in detail.

Figure 8.2: Linux process states

Table 8.1. Linux Process States

NameFlagKernel-defined state name and description
Running R TASK_RUNNING: The process is either executing on a CPU or waiting to run. The process can be executing user routines or kernel routines (system calls), or be queued and ready when in the Running (or Runnable) state.
Sleeping S TASK_INTERRUPTIBLE: The process is waiting for some condition: a hardware request, system resource access, or a signal. When an event or signal satisfies the condition, the process returns to Running.
D TASK_UNINTERRUPTIBLE: This process is also sleeping, but unlike the S state, does not respond to signals. It is used only when process interruption might cause an unpredictable device state.
K TASK_KILLABLE: Same as the uninterruptible D state, but modified to allow a waiting task to respond to the signal to kill it (exit completely). Utilities often display Killable processes as the D state.
I TASK_REPORT_IDLE: A subset of state D. The kernel does not count these processes when calculating the load average. It is used for kernel threads. The TASK_UNINTERRUPTIBLE and TASK_NOLOAD flags are set. It is similar to TASK_KILLABLE, and is also a subset of state D. It accepts fatal signals.
Stopped T TASK_STOPPED: The process is stopped (suspended), usually by being signaled by a user or another process. The process can be continued (resumed) by another signal to return to running.
T TASK_TRACED: A process that is being debugged is also temporarily stopped and shares the T state flag.
Zombie Z EXIT_ZOMBIE: A child process signals to its parent as it exits. All resources except for the process identity (PID) are released.
X EXIT_DEAD: When the parent cleans up (reaps) the remaining child process structure, the process is now released completely. This state cannot be observed in process-listing utilities.

Importance of Process States

When troubleshooting a system, it is important to understand how the kernel communicates with processes, and how processes communicate with each other. The system assigns a state to every new process. The S column of the top command or the STAT column of the ps command shows the state of each process. On a single CPU system, only one process can run at a time. It is possible to see several processes with an R state. However, not all processes are running consecutively; some of them are in waiting status.

[user@host ~]$ top
PID USER  PR  NI    VIRT    RES    SHR S  %CPU  %MEM   TIME+    COMMAND
2259 root 20   0  270856  40316   8332 S   0.3   0.7   0:00.25  sssd_kcm
   1 root 20   0  171820  16140  10356 S   0.0   0.3   0:01.62  systemd
   2 root 20   0       0      0      0 S   0.0   0.0   0:00.00  kthreadd
...output omitted...
[user@host ~]$ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...output omitted...
root         2  0.0  0.0      0     0 ?        S    11:57   0:00 [kthreadd]
student   3448  0.0  0.2 266904  3836 pts/0    R+   18:07   0:00 ps aux
...output omitted...

Use signals to suspend, stop, resume, terminate, or interrupt processes. Processes can catch signals from the kernel, other processes, and other users on the same system. Signals are discussed later in this chapter.

Listing Processes

The ps command is used for listing detailed information for current processes.

  • User identification (UID), which determines process privileges

  • Unique process identification (PID)

  • Amount of used CPU and elapsed real time

  • Amount of allocated memory

  • The process stdout location, which is known as the controlling terminal

  • The current process state

Important

The Linux version of the ps command supports the following option formats:

  • UNIX (POSIX) options, which can be grouped and must be preceded by a dash

  • BSD options, which can be grouped and must not be used with a dash

  • GNU long options, which are preceded by two dashes

For example, the ps -aux command is not the same as the ps aux command.

The common ps command aux option displays all processes including processes without a controlling terminal. A long listing (lax options) provides more detail, and gives faster results by avoiding username lookups. The similar UNIX syntax uses the -ef options to display all processes. In the following examples, scheduled kernel threads are displayed in brackets at the top of the list.

[user@host ~]$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.1  0.2 171820 16140 ?        Ss   16:47   0:01 /usr/lib/systemd/systemd ...
root           2  0.0  0.0      0     0 ?        S    16:47   0:00 [kthreadd]
root           3  0.0  0.0      0     0 ?        I<   16:47   0:00 [rcu_gp]
root           4  0.0  0.0      0     0 ?        I<   16:47   0:00 [rcu_par_gp]
root           6  0.0  0.0      0     0 ?        I<   16:47   0:00 [kworker/0:0H-events_highpri]
...output omitted...
[user@host ~]$ ps lax
F   UID     PID    PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY  TIME COMMAND
4     0       1       0  20   0 171820 16140 -      Ss   ?    0:01 /usr/lib/systemd/systemd ...
1     0       2       0  20   0      0     0 -      S    ?    0:00 [kthreadd]
1     0       3       2   0 -20      0     0 -      I<   ?    0:00 [rcu_gp]
1     0       4       2   0 -20      0     0 -      I<   ?    0:00 [rcu_par_gp]
1     0       6       2   0 -20      0     0 -      I<   ?    0:00 [kworker/0:0H-events_highpri]
...output omitted...
[user@host ~]$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root           1       0  0 16:47 ?        00:00:01 /usr/lib/systemd/systemd ...
root           2       0  0 16:47 ?        00:00:00 [kthreadd]
root           3       2  0 16:47 ?        00:00:00 [rcu_gp]
root           4       2  0 16:47 ?        00:00:00 [rcu_par_gp]
root           6       2  0 16:47 ?        00:00:00 [kworker/0:0H-events_highpri]
...output omitted...

By default, the ps command with no options selects all processes with the current user's effective user ID (EUID), and selects processes that are associated with the terminal that is running the command. Zombie processes are listed with the exiting or defunct label.

You can use the ps command --forest option to display the processes in a tree format so you can view relationships between parent and child processes.

The default output of the ps command is sorted by process ID number. At first glance, the output might appear to use chronological order, but the kernel reuses process IDs, so the order is less structured than it appears. Use the ps command -O or --sort options to sort the output. The display order matches that of the system process table, which reuses table rows when processes die and spawn.

References

info libc signal (GNU C Library Reference Manual)

  • Section 24: Signal Handling

info libc processes (GNU C Library Reference Manual)

  • Section 26: Processes

ps(1) and signal(7) man pages

Revision: rh124-9.3-770cc61