Tuesday, March 20, 2018

Oracle Linux - Local Vault token cache

Vault is more and more seen in modern day infrastructure deployments. HashiCorp Vault secures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and other secrets in modern computing. Vault handles leasing, key revocation, key rolling, and auditing. Through a unified API, users can access an encrypted Key/Value store and network encryption-as-a-service, or generate AWS IAM/STS credentials, SQL/NoSQL databases, X.509 certificates, SSH credentials, and more.

When using Vault from Hashicorp on your Oracle Linux infrastructure you might have noticed that there is no logout option. You can authenticate yourself against vault and from that moment on you can request all information from Vault that you need (and entitled to see). When starting with Vault and building your scripting you might wonder how you "break" the connection again.

In effect, the connection is build every time you do a request against vault and the authentication with a token is done based upon a local cache of the token. If you want to ensure that after you executed the steps needed against vault all tokens are removed you will have to remove the token which is placed in a local cache.

In the case of vault the local cache is a clear text file stored in your home directory as shown below:

[root@docker tmp]# ls -la ~/.vault-token 
-rw------- 1 root root 36 Mar 19 15:49 /root/.vault-token
[root@docker tmp]# 

Even though some improvement requests have been raised to add a logout like function to the Vault CLI the response from the developers from HashiCorp has been that they are not intending to build this into the CLI due to the fact that removing the .vault-token file has the same effect.

In effect the developers from Vault are correct in this and it has the same effect even though it might be a more understandable way of doing things with an option in the CLI. A reminder for everyone who is using Vault, if you are done, ensure that you remove the .vault-token cache file so you are sure nobody will be able to abuse the key to gain access to information they are not entitled to see. 

Sunday, March 18, 2018

Oracle MySQL - test MySQL with Docker

First things first, I am totally against running any type of Docker container that will hold persistent data in any way or form. Even to the point that I like to make the statement that mounting external storage to a container which will hold the persistent data is a bad thing. Some people will disagree with me, however, in the current state of Docker I am against it. Docker should run stateless services and should in no way be depending on persistent data which is directly available (in any form) in the container itself. Having stated this, this post is about running databases in a container, while databases are one of the best examples of persistent storage.

The only exception I make on the statement of not having persistent storage in a container is volatile testing environments. If you have a testing environment you intend to only use for a couple of hours, using a container to serve a database is not a bad thing at all. What you need to remember, if your container stops, all your data is gone.

Getting started with MySQL in Docker
To get started with MySQL in a Docker container you first have to pull it from the Docker registry. You can pull the official container image from Docker as shown in the example below which is done on Oracle Linux:

[root@docker ~]# docker pull mysql
Using default tag: latest
latest: Pulling from library/mysql
2a72cbf407d6: Pull complete 
38680a9b47a8: Pull complete 
4c732aa0eb1b: Pull complete 
c5317a34eddd: Pull complete 
f92be680366c: Pull complete 
e8ecd8bec5ab: Pull complete 
2a650284a6a8: Pull complete 
5b5108d08c6d: Pull complete 
beaff1261757: Pull complete 
c1a55c6375b5: Pull complete 
8181cde51c65: Pull complete 
Digest: sha256:691c55aabb3c4e3b89b953dd2f022f7ea845e5443954767d321d5f5fa394e28c
Status: Downloaded newer image for mysql:latest
[root@docker ~]# 

Now, this should give you the latest evrsion of the MySQL container image. You can check this witht  the docker images command as shown below:

[root@docker ~]# docker images | grep mysql
mysql        latest       5195076672a7        4 days ago          371MB
[root@docker ~]#

Start MySQL in Docker
To start MySQL you can use the below command as an example. As you can see this is a somewhat more extended command than you might see on the Docker page for MySQL.

docker run --name testmysql -e MYSQL_ROOT_PASSWORD=verysecret -p 3306:3306 --rm -d mysql

What I have added in the above example is that I map the internal port 3306 to an external port 3306. If you run multiple instances of MySQL you will need to change the external port numbers. I also added --rm to ensure the docker image is not persisted in any way or form as soon as you stop it.

After starting the container you should be able to find it with a docker ps command:

[root@docker ~]# docker ps |grep mysql
5d8f8bac45a1        mysql        "docker-entrypoint..."   8 minutes ago     Up 8 minutes        0.0.0.0:3306->3306/tcp  testmysql
[root@docker ~]# 

Use databases in Docker?
As already stated, and actually the reason I wrote this post, you should not run anything in a container where you will need to have persistent storage available within the container itself. Databases are a good example of this. Based upon that statement you should not run a database in a container. Having stated that, if you can live with the fact you might lose all your data (for example in a quick test setup) there is nothing against running a database in a container.

Just make sure you don't do it with your production data (please....).


Sunday, March 11, 2018

Oracle Linux - keep an eye on share libs

When running large clusters of Linux servers, you tend to start to look at different things. When running a large number of Linux servers all dedicated to the same task or taskset you might become interested to find out which shared libraries are used on all systems and as a second question, which nodes do every now and than use shared libraries not used by the majority of the nodes. The question, why is a specific node using a shared library that is not used by any other node is a second question, monitoring and detecting is the first part.

You can use a outlier detection on a large dataset containing a time serie of libraries used by systems. For example, if you would be able to capture the data and store this in elastic you could use Kibana and machine learning to do trend analysis and outlier detection to find out if a specific Linux machine in our "farm" is using a specific library that is not in line with all the other machines.

Capturing could be done, as an example, by executing the below example command:

[root@localhost tmp]# awk '/\.so/{print $6}' /proc/*/maps | sort -u
/lib64/ld-2.12.so
/lib64/libattr.so.1.1.0
/lib64/libaudit.so.1.0.0
/lib64/libc-2.12.so
/lib64/libcap-ng.so.0.0.0
/lib64/libcap.so.2.16
/lib64/libcom_err.so.2.1
/lib64/libcrypt-2.12.so
/lib64/libdb-4.7.so
/lib64/libdbus-1.so.3.4.0
/lib64/libdl-2.12.so
/lib64/libfipscheck.so.1.1.0
/lib64/libfreebl3.so
/lib64/libgcc_s-4.4.7-20120601.so.1
/lib64/libgssapi_krb5.so.2.2
/lib64/libk5crypto.so.3.1
/lib64/libkeyutils.so.1.3
/lib64/libkrb5.so.3.3
/lib64/libkrb5support.so.0.1
/lib64/liblber-2.4.so.2.10.3
/lib64/libldap-2.4.so.2.10.3
/lib64/libm-2.12.so
/lib64/libnih-dbus.so.1.0.0
/lib64/libnih.so.1.0.0
/lib64/libnsl-2.12.so
/lib64/libnspr4.so
/lib64/libnss_dns-2.12.so
/lib64/libnss_files-2.12.so
/lib64/libpam_misc.so.0.82.0
/lib64/libpam.so.0.82.2
/lib64/libpcre.so.0.0.1
/lib64/libplc4.so
/lib64/libplds4.so
/lib64/libpthread-2.12.so
/lib64/libresolv-2.12.so
/lib64/librt-2.12.so
/lib64/libselinux.so.1
/lib64/libtinfo.so.5.7
/lib64/libutil-2.12.so
/lib64/libwrap.so.0.7.6
/lib64/libz.so.1.2.3
/lib64/rsyslog/imklog.so
/lib64/rsyslog/imuxsock.so
/lib64/rsyslog/lmnet.so
/lib64/security/pam_cracklib.so
/lib64/security/pam_deny.so
/lib64/security/pam_env.so
/lib64/security/pam_keyinit.so
/lib64/security/pam_limits.so
/lib64/security/pam_loginuid.so
/lib64/security/pam_namespace.so
/lib64/security/pam_nologin.so
/lib64/security/pam_rootok.so
/lib64/security/pam_selinux.so
/lib64/security/pam_sepermit.so
/lib64/security/pam_succeed_if.so
/lib64/security/pam_unix.so
/lib64/security/pam_xauth.so
/usr/lib64/libcrack.so.2.8.1
/usr/lib64/libcrypto.so.1.0.1e
/usr/lib64/libhesiod.so.0.0.0
/usr/lib64/libnss3.so
/usr/lib64/libnssutil3.so
/usr/lib64/libsasl2.so.2.0.23
/usr/lib64/libsmime3.so
/usr/lib64/libssl3.so
/usr/lib64/libssl.so.1.0.1e
/usr/lib64/sasl2/libanonymous.so.2.0.23
/usr/lib64/sasl2/libsasldb.so.2.0.23
/usr/libexec/sudoers.so
[root@localhost tmp]# 

If you have a process taking this snapshot of shared library use on a semi-regular interval you will get a good insight in the use of shared libraries in general on your server farm. Having this in place and adding machine learning and outlier detection you can have a system identify strange behaviour on one or more nodes. Additionally, it might help you to improve the base image of your operating system deployed by identifying shared libraries that could potentially be removed or might be in need of an upgrade. 

Oracle Linux - check shared library version

As we have seen in a previous post, you can quickly see which shared library files are used by an executable under Oracle Linux by a specific executable. In our example we found that libidn.so.11 and libc.so.6 are both used by ping under Oracle Linux. We extracted that information by using the elfread command. Even though this is providing some information it is not telling you the exact version that is used at present. In case you do need to know which exact version is used you will have to dig a bit deeper into the system.

In a previous example we used the below command on Oracle Linux to find out the shared library files for the ping command.

[root@localhost tmp]# readelf -d /bin/ping | grep 'NEEDED'
 0x0000000000000001 (NEEDED)             Shared library: [libidn.so.11]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
[root@localhost tmp]# 

If you like to have a bit more information on the version that is used (for example libidn.so.1) you can use ldconfig. ldconfig creates the necessary links and cache to the most recent shared libraries found in the directories specified on the command line, in the file /etc/ld.so.conf, and in the trusted directories, /lib and /usr/lib (on some 64-bit architectures such as x86-64, lib and /usr/lib are the trusted directories for 32-bit libraries, while /lib64 and /usr/lib64 are used for 64-bit libraries).

This means that ldconfig can provide a lot of information than we might want. In the below example (only first couple of lines) we do a verbose output of ldconfig.

[root@localhost tmp]# ldconfig -v
ldconfig: /etc/ld.so.conf.d/kernel-4.1.12-61.1.28.el6uek.x86_64.conf:6: duplicate hwcap 1 nosegneg
/lib:
/lib64:
 libattr.so.1 -> libattr.so.1.1.0
 libplc4.so -> libplc4.so
 libdevmapper.so.1.02 -> libdevmapper.so.1.02
 libdevmapper-event-lvm2.so.2.02 -> libdevmapper-event-lvm2.so.2.02
 libplds4.so -> libplds4.so
 libnss_hesiod.so.2 -> libnss_hesiod-2.12.so
 libfreeblpriv3.so -> libfreeblpriv3.so
 libselinux.so.1 -> libselinux.so.1
 libz.so.1 -> libz.so.1.2.3
 libdevmapper-event.so.1.02 -> libdevmapper-event.so.1.02
 libip6tc.so.0 -> libip6tc.so.0.0.0-1.4.7
 libmount.so.1 -> libmount.so.1.1.0
 libgpg-error.so.0 -> libgpg-error.so.0.5.0
 liblvm2app.so.2.2 -> liblvm2app.so.2.2

Now, if we want to have some information on libidn.so.11 we can do so by adding grep to the command via a pipe to only have the information we want.

[root@localhost tmp]# ldconfig -v | grep libidn.so.11
ldconfig: /etc/ld.so.conf.d/kernel-4.1.12-61.1.28.el6uek.x86_64.conf:6: duplicate hwcap 1 nosegneg
 libidn.so.11 -> libidn.so.11.6.1
[root@localhost tmp]# 

As you can see from the above output, libidn.so.11 is linked to libidn.so.11.6.1 which is the actual version used (via linking) by the ping executable we used as an example.

Oracle Linux - find shared libraries using readelf

A library is a file containing compiled code from various object files stuffed into a single file. It may contain a group of functions that are used in a particular context. For example, the ‘pthread’ library is used when thread related functions are to be used in the program. Shared Libraries are the libraries that can be linked to any program at run-time. They provide a means to use code that can be loaded anywhere in the memory. Once loaded, the shared library code can be used by any number of programs. So, this way the size of programs(using shared library) and the memory footprint can be kept low as a lot of code is kept common in form of a shared library.

In some cases you want to understand which shared libraries are used by a specific executable file. We take as an example the ping executable as it is available on most systems. To give the complete picture, we are running Oracle Linux and use the below version of ping:

[root@localhost tmp]# uname -a
Linux localhost 4.1.12-61.1.28.el6uek.x86_64 #2 SMP Thu Feb 23 20:03:53 PST 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost tmp]# ping -V
ping utility, iputils-sss20071127
[root@localhost tmp]# 

Now, a number of options are available to find which shared libraries are available. In this example we use the readelf way of doing things. The readelf command displays information about one or more ELF format object files. The options control what particular information to display.

In the below example we use readelf on Oracle Linux to find out which shared library files are used by the ping command as an example

[root@localhost tmp]# readelf -d /bin/ping

Dynamic section at offset 0x8760 contains 22 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libidn.so.11]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x1d68
 0x000000000000000d (FINI)               0x6d28
 0x000000006ffffef5 (GNU_HASH)           0x260
 0x0000000000000005 (STRTAB)             0xdc0
 0x0000000000000006 (SYMTAB)             0x3d0
 0x000000000000000a (STRSZ)              1074 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x208a78
 0x0000000000000002 (PLTRELSZ)           1320 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x1840
 0x0000000000000007 (RELA)               0x1348
 0x0000000000000008 (RELASZ)             1272 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x12c8
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0x11f2
 0x000000006ffffff9 (RELACOUNT)          44
 0x0000000000000000 (NULL)               0x0
[root@localhost tmp]# 

As you can see, we have two shared library files; libidn.so.11 andlibc.so.6 in this case. As you can see the readelf command gave a lot more information as well, in case you do not want to have those lines you can use a simple pipe to some commands to ensure you have a more clean output.

[root@localhost tmp]# readelf -d /bin/ping | grep 'NEEDED'
 0x0000000000000001 (NEEDED)             Shared library: [libidn.so.11]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
[root@localhost tmp]# 

The above example showcases the more clean version to check which shared library files are used.