Oracle Linux - keep an eye on share libs

When running large clusters of Linux servers, you tend to start to look at different things. When running a large number of Linux servers all dedicated to the same task or taskset you might become interested to find out which shared libraries are used on all systems and as a second question, which nodes do every now and than use shared libraries not used by the majority of the nodes. The question, why is a specific node using a shared library that is not used by any other node is a second question, monitoring and detecting is the first part.

You can use a outlier detection on a large dataset containing a time serie of libraries used by systems. For example, if you would be able to capture the data and store this in elastic you could use Kibana and machine learning to do trend analysis and outlier detection to find out if a specific Linux machine in our "farm" is using a specific library that is not in line with all the other machines.

Capturing could be done, as an example, by executing the below example command:

[root@localhost tmp]# awk '/\.so/{print $6}' /proc/*/maps | sort -u
/lib64/ld-2.12.so
/lib64/libattr.so.1.1.0
/lib64/libaudit.so.1.0.0
/lib64/libc-2.12.so
/lib64/libcap-ng.so.0.0.0
/lib64/libcap.so.2.16
/lib64/libcom_err.so.2.1
/lib64/libcrypt-2.12.so
/lib64/libdb-4.7.so
/lib64/libdbus-1.so.3.4.0
/lib64/libdl-2.12.so
/lib64/libfipscheck.so.1.1.0
/lib64/libfreebl3.so
/lib64/libgcc_s-4.4.7-20120601.so.1
/lib64/libgssapi_krb5.so.2.2
/lib64/libk5crypto.so.3.1
/lib64/libkeyutils.so.1.3
/lib64/libkrb5.so.3.3
/lib64/libkrb5support.so.0.1
/lib64/liblber-2.4.so.2.10.3
/lib64/libldap-2.4.so.2.10.3
/lib64/libm-2.12.so
/lib64/libnih-dbus.so.1.0.0
/lib64/libnih.so.1.0.0
/lib64/libnsl-2.12.so
/lib64/libnspr4.so
/lib64/libnss_dns-2.12.so
/lib64/libnss_files-2.12.so
/lib64/libpam_misc.so.0.82.0
/lib64/libpam.so.0.82.2
/lib64/libpcre.so.0.0.1
/lib64/libplc4.so
/lib64/libplds4.so
/lib64/libpthread-2.12.so
/lib64/libresolv-2.12.so
/lib64/librt-2.12.so
/lib64/libselinux.so.1
/lib64/libtinfo.so.5.7
/lib64/libutil-2.12.so
/lib64/libwrap.so.0.7.6
/lib64/libz.so.1.2.3
/lib64/rsyslog/imklog.so
/lib64/rsyslog/imuxsock.so
/lib64/rsyslog/lmnet.so
/lib64/security/pam_cracklib.so
/lib64/security/pam_deny.so
/lib64/security/pam_env.so
/lib64/security/pam_keyinit.so
/lib64/security/pam_limits.so
/lib64/security/pam_loginuid.so
/lib64/security/pam_namespace.so
/lib64/security/pam_nologin.so
/lib64/security/pam_rootok.so
/lib64/security/pam_selinux.so
/lib64/security/pam_sepermit.so
/lib64/security/pam_succeed_if.so
/lib64/security/pam_unix.so
/lib64/security/pam_xauth.so
/usr/lib64/libcrack.so.2.8.1
/usr/lib64/libcrypto.so.1.0.1e
/usr/lib64/libhesiod.so.0.0.0
/usr/lib64/libnss3.so
/usr/lib64/libnssutil3.so
/usr/lib64/libsasl2.so.2.0.23
/usr/lib64/libsmime3.so
/usr/lib64/libssl3.so
/usr/lib64/libssl.so.1.0.1e
/usr/lib64/sasl2/libanonymous.so.2.0.23
/usr/lib64/sasl2/libsasldb.so.2.0.23
/usr/libexec/sudoers.so
[root@localhost tmp]# 

If you have a process taking this snapshot of shared library use on a semi-regular interval you will get a good insight in the use of shared libraries in general on your server farm. Having this in place and adding machine learning and outlier detection you can have a system identify strange behaviour on one or more nodes. Additionally, it might help you to improve the base image of your operating system deployed by identifying shared libraries that could potentially be removed or might be in need of an upgrade.