Saturday, December 10, 2016

Oracle Linux – finding the executable of a process

One of the important things when using and administering an Oracle Linux instance (or any other distribution for that matter) is to understand what is going on within your system. One of the things to understand is what is running on the Linux instance. Even more important is to ensure that you are constantly aware and you are in full control of what is running so you can detect things in case they are running and should not be running so you can detect anomalies. However, before you can look into detecting anomalies in running processes you need to have an understanding of how to look at what is running on your system.

Commonly when people like to know what is running on a Oracle Linux system they use the top command or they use the ps command. the ps command is to report a snapshot of the current processes on your system. An example of the ps command is shown below taken from one of my temporary test servers;

[root@localhost 2853]# ps -ef | grep root
root         1     0  0 Nov26 ?        00:00:01 /sbin/init
root         2     0  0 Nov26 ?        00:00:00 [kthreadd]
root         3     2  0 Nov26 ?        00:00:02 [ksoftirqd/0]
root         5     2  0 Nov26 ?        00:00:00 [kworker/0:0H]
root         6     2  0 Nov26 ?        00:00:00 [kworker/u:0]
root         7     2  0 Nov26 ?        00:00:00 [kworker/u:0H]
root         8     2  0 Nov26 ?        00:00:00 [migration/0]
root         9     2  0 Nov26 ?        00:00:00 [rcu_bh]
root        10     2  0 Nov26 ?        00:00:35 [rcu_sched]
root        11     2  0 Nov26 ?        00:00:05 [watchdog/0]
root        12     2  0 Nov26 ?        00:00:00 [cpuset]
root        13     2  0 Nov26 ?        00:00:00 [khelper]
root        14     2  0 Nov26 ?        00:00:00 [kdevtmpfs]
root        15     2  0 Nov26 ?        00:00:00 [netns]
root        16     2  0 Nov26 ?        00:00:00 [bdi-default]
root        17     2  0 Nov26 ?        00:00:00 [kintegrityd]
root        18     2  0 Nov26 ?        00:00:00 [crypto]
root        19     2  0 Nov26 ?        00:00:00 [kblockd]
root        20     2  0 Nov26 ?        00:00:00 [ata_sff]
root        21     2  0 Nov26 ?        00:00:00 [khubd]
root        22     2  0 Nov26 ?        00:00:00 [md]
root        24     2  0 Nov26 ?        00:00:00 [khungtaskd]
root        25     2  0 Nov26 ?        00:00:05 [kswapd0]
root        26     2  0 Nov26 ?        00:00:00 [ksmd]
root        27     2  0 Nov26 ?        00:00:00 [fsnotify_mark]
root        38     2  0 Nov26 ?        00:00:00 [kthrotld]
root        39     2  0 Nov26 ?        00:00:00 [kworker/u:1]
root        40     2  0 Nov26 ?        00:00:00 [kpsmoused]
root        41     2  0 Nov26 ?        00:00:00 [deferwq]
root       187     2  0 Nov26 ?        00:00:00 [scsi_eh_0]
root       190     2  0 Nov26 ?        00:00:00 [scsi_eh_1]
root       252     2  0 Nov26 ?        00:00:06 [kworker/0:1H]
root       305     2  0 Nov26 ?        00:00:00 [kdmflush]
root       307     2  0 Nov26 ?        00:00:00 [kdmflush]
root       373     2  0 Nov26 ?        00:00:46 [jbd2/dm-0-8]
root       374     2  0 Nov26 ?        00:00:00 [ext4-dio-unwrit]
root       474     1  0 Nov26 ?        00:00:00 /sbin/udevd -d
root      1845     2  0 Nov26 ?        00:00:00 [jbd2/sda1-8]
root      1846     2  0 Nov26 ?        00:00:00 [ext4-dio-unwrit]
root      1893     2  0 Nov26 ?        00:00:00 [kauditd]
root      2100     2  0 Nov26 ?        00:01:08 [flush-252:0]
root      2282     1  0 Nov26 ?        00:00:00 auditd
root      2316     1  0 Nov26 ?        00:00:00 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
root      2445     1  0 Nov26 ?        00:00:00 cupsd -C /etc/cups/cupsd.conf
root      2477     1  0 Nov26 ?        00:00:00 /usr/sbin/acpid
root      2490  2489  0 Nov26 ?        00:00:00 hald-runner
root      2556     1  0 Nov26 ?        00:00:08 automount --pid-file /var/run/autofs.pid
root      2664     1  0 Nov26 ?        00:00:00 /usr/sbin/mcelog --daemon
root      2686     1  0 Nov26 ?        00:00:00 /usr/sbin/sshd
root      2812     1  0 Nov26 ?        00:00:02 /usr/libexec/postfix/master
root      2841     1  0 Nov26 ?        00:00:00 /usr/sbin/abrtd
root      2853     1  0 Nov26 ?        00:00:04 crond
root      2868     1  0 Nov26 ?        00:00:00 /usr/sbin/atd
root      2935     1  0 Nov26 ?        00:00:00 /usr/sbin/certmonger -S -p /var/run/certmonger.pid
root      2981     1  0 Nov26 tty1     00:00:00 /sbin/mingetty /dev/tty1
root      2983     1  0 Nov26 tty2     00:00:00 /sbin/mingetty /dev/tty2
root      2985     1  0 Nov26 tty3     00:00:00 /sbin/mingetty /dev/tty3
root      2987     1  0 Nov26 tty4     00:00:00 /sbin/mingetty /dev/tty4
root      2989     1  0 Nov26 tty5     00:00:00 /sbin/mingetty /dev/tty5
root      2996     1  0 Nov26 tty6     00:00:00 /sbin/mingetty /dev/tty6
root      2999   474  0 Nov26 ?        00:00:00 /sbin/udevd -d
root      3000   474  0 Nov26 ?        00:00:00 /sbin/udevd -d
root      5615  2686  0 Nov30 ?        00:00:07 sshd: root@pts/0
root      5620  5615  0 Nov30 pts/0    00:00:01 -bash
root      9739  5620  0 09:59 pts/0    00:00:00 ps -ef
root      9740  5620  0 09:59 pts/0    00:00:00 grep root
root     16808     2  0 Nov28 ?        00:00:00 [kworker/0:0]
root     17683     1  0 Nov30 ?        00:00:06 /usr/sbin/httpd
root     19810     2  0 Nov28 ?        00:04:08 [kworker/0:2]
root     20820     1  0 Nov28 ?        00:16:47 /usr/bin/consul agent -config-dir=/etc/consul.d
root     21102     1  0 Nov28 ?        00:05:02 /usr/bin/vault server -config=/etc/vault.d
[root@localhost 2853]#

As you can see this provides quite a good insight into what is running and what is not. However, it is not fully showing you all the details you might want to see. For example, we see that some of the lines show the exact path of the executable that is running under this process. For example mingetty (minimal getty for consoles). we can zoom in to mingetty with a grep as shown below;

[root@localhost 2489]# ps -ef |grep mingetty
root      2981     1  0 Nov26 tty1     00:00:00 /sbin/mingetty /dev/tty1
root      2983     1  0 Nov26 tty2     00:00:00 /sbin/mingetty /dev/tty2
root      2985     1  0 Nov26 tty3     00:00:00 /sbin/mingetty /dev/tty3
root      2987     1  0 Nov26 tty4     00:00:00 /sbin/mingetty /dev/tty4
root      2989     1  0 Nov26 tty5     00:00:00 /sbin/mingetty /dev/tty5
root      2996     1  0 Nov26 tty6     00:00:00 /sbin/mingetty /dev/tty6
root      9815  5620  0 10:04 pts/0    00:00:00 grep mingetty
[root@localhost 2489]#

If we look at the above we can be fairly sure that the executable for mingetty is located at /sbin/mingetty . However, if we start looking at the results of other lines this is not always that clear. As an example the HAL daemon hald (which is a just good example in this case). hald  is  a daemon that maintains a database of the devices connected to the system system in real-time. The daemon connects to the D-Bus system message bus to provide an API that applications can use to discover, monitor and invoke operations on devices.

[root@localhost 2489]# ps -ef|grep hald
68        2489     1  0 Nov26 ?        00:00:18 hald
root      2490  2489  0 Nov26 ?        00:00:00 hald-runner
68        2532  2490  0 Nov26 ?        00:00:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
root      9864  5620  0 10:09 pts/0    00:00:00 grep hald
[root@localhost 2489]#

If we look clearly at the above we can learn a number of things. For once, the hald-addon-acpi is a child process of hald-runner and hald-runner is a child process of hald. we can also see that both hald and hald-addon-acpi are running under UID 68 which is the default UID for hald. However, what we are not able to see is what the actual executable is that is runnign between hald.

To find out the exact executable of hald we can find out by going to the /proc directory and than go to the subdirectory which is in line with the pid of the process. In our case this is /proc/2489 which is the directory which holds all the information about process 2489, our hald process. In this dirctory we will find a lot of interesting information;

[root@localhost /]# cd /proc/2489
[root@localhost 2489]# ls
attr        coredump_filter  fdinfo    mem         numa_maps      root       stat
auxv        cpuset           io        mountinfo   oom_adj        sched      statm
cgroup      cwd              latency   mounts      oom_score      schedstat  status
clear_refs  environ          limits    mountstats  oom_score_adj  sessionid  syscall
cmdline     exe              loginuid  net         pagemap        smaps      task
comm        fd               maps      ns          personality    stack      wchan
[root@localhost 2489]#

Even though all the files and directories within a process /proc/pid diretcory are interesting our goal was to find out what the actual running process behind pid 2489 from user UID 68 was. To find out we have to look at the exe which is a symbolic link. So we can do a ls -la command or in case we want this to be part of a bash script to find things out we can use the readlink command.

The simple ls command will be able to tell us in a human readabile manner what the executable is for this pid.

[root@localhost 2489]# ls -la exe
lrwxrwxrwx. 1 root root 0 Dec  3 10:03 exe -> /usr/sbin/hald
[root@localhost 2489]#

Even thought this is great and we just have been able to find out what the executable file of a pid is in case it is not listed in the output of ps we might want to include this in some bash script. The most easy way is using the readlink command which will provide the below;

[root@localhost 2489]# readlink exe
/usr/sbin/hald
[root@localhost 2489]#

Making sure you understand a bit more on how to drill into the information of what us running on your system will help you debug issues quicker and make sure you can implement more strict security and monitoring rules on your Oracle Linux systems. 

No comments: