Friday, July 07, 2017

Oracle Linux - reading CPU flags from /proc/cpuinfo

Most people using Linux take the CPU they have for granted. Commonly the only question is, how fast is that CPU handling my code. However, in some cases, especially when doing lower level development work or doing more advanced Oracle Linux system tuning it is good to have a bit more insight into your physical CPU. The first place to look for more information on the CPU is /proc/cpuinfo which might give a wealth of additional information.

Below is an example of the content of my /proc/cpuinfo (running Oracle Linux in a Virtualbox image on a MacBook Pro):

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model  : 78
model name : Intel(R) Core(TM) i5-6267U CPU @ 2.90GHz
stepping : 3
cpu MHz  : 2903.998
cache size : 4096 KB
physical id : 0
siblings : 2
core id  : 0
cpu cores : 2
apicid  : 0
initial apicid : 0
fpu  : yes
fpu_exception : yes
cpuid level : 22
wp  : yes
flags  : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm 3dnowprefetch rdseed clflushopt
bugs  :
bogomips : 5807.99
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model  : 78
model name : Intel(R) Core(TM) i5-6267U CPU @ 2.90GHz
stepping : 3
cpu MHz  : 2903.998
cache size : 4096 KB
physical id : 0
siblings : 2
core id  : 1
cpu cores : 2
apicid  : 1
initial apicid : 1
fpu  : yes
fpu_exception : yes
cpuid level : 22
wp  : yes
flags  : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm 3dnowprefetch rdseed clflushopt
bugs  :
bogomips : 5807.99
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

As you can see, a lot of flags are mentioned, those might be interesting to understand the capabilities of your processor in more detail. In our case the following flags are mentioned:  fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm 3dnowprefetch rdseed clflushopt.

fpu
Onboard FPU (floating point support). designed to carry out operations on floating point numbers. Typical operations are addition, subtraction, multiplication, division, square root, and bitshifting.

vme
Virtual 8086 mode enhancements. In the 80386 microprocessor and later, virtual 8086 mode (also called virtual real mode, V86-mode or VM86) allows the execution of real mode applications that are incapable of running directly in protected mode while the processor is running a protected mode operating system. It is a hardware virtualization technique that allowed multiple 8086 processors to be emulated by the 386 chip; it emerged from the painful experiences with the 80286 protected mode, which by itself was not suitable to run concurrent real mode applications well

de
Debugging Extensions (CR4.DE). A control register CR is a processor register which changes or controls the general behavior of a CPU or other digital device. Common tasks performed by control registers include interrupt control, switching the addressing mode, paging control, and coprocessor control. CR4 is used in protected mode to control operations such as virtual-8086 support, enabling I/O breakpoints, page size extension and machine check exceptions. CR4 bit 3 controls the DE (Debugging Extension), if set, enables debug register based breaks on I/O space access.

pse
Page Size Extensions (4MB memory pages). PSE is a feature of x86 processors that allows for pages larger than the traditional 4 KiB size. It was introduced in the original Pentium processor, but it was only publicly documented by Intel with the release of the Pentium Pro.

tsc
Time Stamp Counter (RDTSC). is a 64-bit register present on all x86 processors since the Pentium. It counts the number of cycles since reset. The instruction RDTSC returns the TSC in EDX:EAX. In x86-64 mode, RDTSC also clears the higher 32 bits of RAX and RDX. Its opcode is 0F 31.

msr
Model-Specific Registers (RDMSR, WRMSR). MSR is any of various control registers in the x86 instruction set used for debugging, program execution tracing, computer performance monitoring, and toggling certain CPU features.

pae
Physical Address Extensions (support for more than 4GB of RAM). Physical Address Extension (PAE), sometimes referred to as Page Address Extension, is a memory management feature for the x86 architecture. PAE was first introduced by Intel in the Pentium Pro, and later by AMD in the Athlon processor.[2] It defines a page table hierarchy of three levels, with table entries of 64 bits each instead of 32, allowing these CPUs to directly access a physical address space larger than 4 gigabytes

mce
Machine Check Exception. A Machine Check Exception (MCE) is a type of computer hardware error that occurs when a computer's central processing unit detects a hardware problem. Modern versions of Microsoft Windows handle machine check exceptions through the Windows Hardware Error Architecture. On Linux, a process (such as klogd[2]) writes a message to the kernel log and/or the console screen (usually only to the console when the error is non-recoverable and the machine crashes as a result)

cx8
CMPXCHG8 instruction (64-bit compare-and-swap). Compares the 64-bit value in EDX:EAX (or 128-bit value in RDX:RAX if operand size is 128 bits) with the operand (destination operand). If the values are equal, the 64-bit value in ECX:EBX (or 128-bit value in RCX:RBX) is stored in the destination operand. Otherwise, the value in the destination operand is loaded into EDX:EAX (or RDX:RAX). The destination operand is an 8-byte memory location (or 16-byte memory location if operand size is 128 bits). For the EDX:EAX and ECX:EBX register pairs, EDX and ECX contain the high-order 32 bits and EAX and EBX contain the low-order 32 bits of a 64-bit value. For the RDX:RAX and RCX:RBX register pairs, RDX and RCX contain the high-order 64 bits and RAX and RBX contain the low-order 64bits of a 128-bit value. This instruction encoding is not supported on Intel processors earlier than the Pentium processors.

apic
Onboard APIC. Advanced Programmable Interrupt Controller (APIC) is a family of interrupt controllers. As its name suggests, the APIC is more advanced than Intel's 8259 Programmable Interrupt Controller (PIC), particularly enabling the construction of multiprocessor systems. It is one of several architectural designs intended to solve interrupt routing efficiency issues in multiprocessor computer systems.

sep
SYSENTER/SYSEXIT. Executes a fast call to a level 0 system procedure or routine. SYSENTER is a companion instruction to SYSEXIT. The instruction is optimized to provide the maximum performance for system calls from user code running at privilege level 3 to operating system or executive procedures running at privilege level 0.

mtrr
Memory Type Range Registers. Memory type range registers (MTRRs) are a set of processor supplementary capabilities control registers that provide system software with control of how accesses to memory ranges by the CPU are cached. It uses a set of programmable model-specific registers (MSRs) which are special registers provided by most modern CPUs. Possible access modes to memory ranges can be uncached, write-through, write-combining, write-protect, and write-back. In write-back mode, writes are written to the CPU's cache and the cache is marked dirty, so that its contents are written to memory later.

pge
Page Global Enable (global bit in PDEs and PTEs). the page global enable (PGE) flag in the register CR4 and the global (G) flag of a page-directory or page-table entry can be used to prevent frequently used pages from being automatically invalidated in the TLBs on a task switch or a load of register CR3. Bit 7 in CR4 is used to set PGE, if set, address translations (PDE or PTE records) may be shared between address spaces.

mca
Machine Check Architecture. Machine Check Architecture (MCA) is an Intel mechanism in which the CPU reports hardware errors to the operating system. Intel's Pentium 4, Intel Xeon, P6 family processors as well as the Itanium architecture implement a machine check architecture that provides a mechanism for detecting and reporting hardware (machine) errors, such as: system bus errors, ECC errors, parity errors, cache errors, and translation lookaside buffer errors. It consists of a set of model-specific registers (MSRs) that are used to set up machine checking and additional banks of MSRs used for recording errors that are detected.

cmov
CMOV instructions (conditional move) (also FCMOV). FCMOV is a floating point conditional move opcode of the Intel x86 architecture, first introduced in Pentium Pro processors. It copies the contents of one of the floating point stack register, depending on the contents of EFLAGS integer flag register, to the ST(0) (top of stack) register. There are 8 variants of the instruction selected by the condition codes that need be set for the instruction to perform the move. Similar to the CMOV instruction, FCMOV allows some conditional operations to be performed without the usual branching overhead. However, it has a higher latency than conditional branch instructions.[2] Therefore, it is most useful for simple yet unpredictable comparison or conditional operations, where it can provide substantial performance gains. The instruction is usually used with the FCOMI instruction or the FCOM-FSTSW-SAHF idiom to set the relevant condition codes based on the result of a floating point comparison.

pat
Page Attribute Table. The page attribute table (PAT) is a processor supplementary capability extension to the page table format of certain x86 and x86-64 microprocessors. Like memory type range registers (MTRRs), they allow for fine-grained control over how areas of memory are cached, and are a companion feature to the MTRRs.

pse36
36-bit PSEs (huge pages). PSE-36 (36-bit Page Size Extension) refers to a feature of x86 processors that extends the physical memory addressing capabilities from 32 bits to 36 bits, allowing addressing to up to 64 GB of memory. Compared to the Physical Address Extension (PAE) method, PSE-36 is a simpler alternative to addressing more than 4 GB of memory. It uses the Page Size Extension (PSE) mode and a modified page directory table to map 4 MB pages into a 64 GB physical address space. PSE-36's downside is that, unlike PAE, it doesn't have 4-KB page granularity above the 4 GB mark.

clflush
Cache Line Flush instruction. Invalidates the cache line that contains the linear address specified with the source operand from all levels of the processor cache hierarchy (data and instruction). The invalidation is broadcast throughout the cache coherence domain. If, at any level of the cache hierarchy, the line is inconsistent with memory (dirty) it is written to memory before invalidation. The source operand is a byte memory location. The availability of CLFLUSH is indicated by the presence of the CPUID feature flag CLFSH (bit 19 of the EDX register, see Section , CPUID-CPU Identification). The aligned cache line size affected is also indicated with the CPUID instruction (bits 8 through 15 of the EBX register when the initial value in the EAX register is 1).

mmx
Multimedia Extensions. MMX is a single instruction, multiple data (SIMD) instruction set designed by Intel, introduced in 1997 with its P5-based Pentium line of microprocessors, designated as "Pentium with MMX Technology".It developed out of a similar unit introduced on the Intel i860, and earlier the Intel i750 video pixel processor. MMX is a processor supplementary capability that is supported on recent IA-32 processors by Intel and other vendors.

fxsr
FXSAVE/FXRSTOR, CR4.OSFXSR. Operating system support for FXSAVE and FXRSTOR instructions. If set, enables SSE instructions and fast FPU save & restore. bit 9 in CR4 is used to control fxsr.

sse
Intel SSE vector instructions. Streaming SIMD Extensions (SSE) is an SIMD instruction set extension to the x86 architecture, designed by Intel and introduced in 1999 in their Pentium III series of processors shortly after the appearance of AMD's 3DNow!. SSE contains 70 new instructions, most of which work on single precision floating point data. SIMD instructions can greatly increase performance when exactly the same operations are to be performed on multiple data objects. Typical applications are digital signal processing and graphics processing.

sse2
SSE2 (Streaming SIMD Extensions 2), is one of the Intel SIMD (Single Instruction, Multiple Data) processor supplementary instruction sets first introduced by Intel with the initial version of the Pentium 4 in 2001. It extends the earlier SSE instruction set, and is intended to fully replace MMX. Intel extended SSE2 to create SSE3 in 2004. SSE2 added 144 new instructions to SSE, which has 70 instructions. Competing chip-maker AMD added support for SSE2 with the introduction of their Opteron and Athlon 64 ranges of AMD64 64-bit CPUs in 2003.

ht
Hyper-threading (officially called Hyper-Threading Technology or HT Technology, and abbreviated as HTT or HT) is Intel's proprietary simultaneous multithreading (SMT) implementation used to improve parallelization of computations (doing multiple tasks at once) performed on x86 microprocessors. It first appeared in February 2002 on Xeon server processors and in November 2002 on Pentium 4 desktop CPUs.[4] Later, Intel included this technology in Itanium, Atom, and Core 'i' Series CPUs, among others.

syscall
SYSCALL (Fast System Call) and SYSRET (Return From Fast System Call). SYSCALL invokes an OS system-call handler at privilege level 0. It does so by loading RIP from the IA32_LSTAR MSR (after saving the address of the instruction following SYSCALL into RCX). (The WRMSR instruction ensures that the IA32_LSTAR MSR always contain a canonical address.)

nx
Execute Disable. The NX bit, which stands for No-eXecute, is a technology used in CPUs to segregate areas of memory for use by either storage of processor instructions (code) or for storage of data, a feature normally only found in Harvard architecture processors. However, the NX bit is being increasingly used in conventional von Neumann architecture processors, for security reasons.

rdtscp
Read Time-Stamp Counter and Processor ID. Loads the current value of the processor’s time-stamp counter (a 64-bit MSR) into the EDX:EAX registers and also loads the IA32_TSC_AUX MSR (address C000_0103H) into the ECX register. The EDX register is loaded with the high-order 32 bits of the IA32_TSC MSR; the EAX register is loaded with the low-order 32 bits of the IA32_TSC MSR; and the ECX register is loaded with the low-order 32-bits of IA32_TSC_AUX MSR. On processors that support the Intel 64 architecture, the high-order 32 bits of each of RAX, RDX, and RCX are cleared.

lm
Long Mode (x86-64: amd64, also known as Intel 64, i.e. 64-bit capable). Long mode is the mode where a 64-bit operating system can access 64-bit instructions and registers. 64-bit programs are run in a sub-mode called 64-bit mode, while 32-bit programs and 16-bit protected mode programs are executed in a sub-mode called compatibility mode. Real mode or virtual 8086 mode programs cannot be natively run in long mode.

constant_tsc
TSC ticks at a constant rate. Recent Intel processors include a constant rate TSC (identified by the kern.timecounter.invariant_tsc sysctl on FreeBSD or by the "constant_tsc" flag in Linux's /proc/cpuinfo). With these processors, the TSC ticks at the processor's nominal frequency, regardless of the actual CPU clock frequency due to turbo or power saving states. Hence TSC ticks are counting the passage of time, not the number of CPU clock cycles elapsed.

rep_good
rep microcode works well.

nopl
The NOPL (0F 1F) instructions

xtopology
cpu topology enum extensions

nonstop_tsc
TSC does not stop in C states. NONSTOP_TSC acts in conjunction with CONSTANT_TSC. CONSTANT_TSC indicates that the TSC runs at constant frequency irrespective of P/T- states, and NONSTOP_TSC indicates that TSC does not stop in deep C-states.

pni
SSE-3 (“Prescott New Instructions”). SSE3, Streaming SIMD Extensions 3, also known by its Intel code name Prescott New Instructions (PNI), is the third iteration of the SSE instruction set for the IA-32 (x86) architecture. Intel introduced SSE3 in early 2004 with the Prescott revision of their Pentium 4 CPU. In April 2005, AMD introduced a subset of SSE3 in revision E (Venice and San Diego) of their Athlon 64 CPUs. The earlier SIMD instruction sets on the x86 platform, from oldest to newest, are MMX, 3DNow! (developed by AMD, but not supported by Intel processors), SSE, and SSE2.

pclmulqdq
Perform a Carry-Less Multiplication of Quadword instruction — accelerator for GCM). Carry-less Multiplication (CLMUL) is an extension to the x86 instruction set used by microprocessors from Intel and AMD which was proposed by Intel in March 2008 and made available in the Intel Westmere processors announced in early 2010. One use of these instructions is to improve the speed of applications doing block cipher encryption in Galois/Counter Mode, which depends on finite field GF(2k)) multiplication, which can be implemented more efficiently with the new CLMUL instructions than with the traditional instruction set. Another application is the fast calculation of CRC values, including those used to implement the LZ77 sliding window DEFLATE algorithm in zlib and pngcrush.

ssse3
Supplemental SSE-3. SSSE3 was first introduced with Intel processors based on the Core microarchitecture on 26 June 2006 with the "Woodcrest" Xeons. SSSE3 has been referred to by the codenames Tejas New Instructions (TNI) or Merom New Instructions (MNI) for the first processor designs intended to support it. SSSE3 contains 16 new discrete instructions as a supplement on SSE-3

cx16
CMPXCHG16B. al words. This is useful for parallel algorithms that use compare and swap on data larger than the size of a pointer, common in lock-free and wait-free algorithms. Without CMPXCHG16B one must use workarounds, such as a critical section or alternative lock-free approaches.

sse4_1
SSE4.1 instruction set. These instructions were introduced with Penryn microarchitecture, the 45 nm shrink of Intel's Core microarchitecture. Support is indicated via the CPUID.01H:ECX.SSE41[Bit 19] flag.

sse4_2
SSE4.2 instruction set. SSE4.2 added STTNI (String and Text New Instructions), several new instructions that perform character searches and comparison on two operands of 16 bytes at a time. These were designed (among other things) to speed up the parsing of XML documents. It also added a CRC32 instruction to compute cyclic redundancy checks as used in certain data transfer protocols. These instructions were first implemented in the Nehalem-based Intel Core i7 product line and complete the SSE4 instruction set. Support is indicated via the CPUID.01H:ECX.SSE42[Bit 20] flag.

x2apic
The xAPIC was introduced with the Pentium 4, while the x2APIC is the most recent generation of the Intel's programmable interrupt controller, introduced with the Nehalem microarchitecture. The major improvements of the x2APIC address the number of supported CPUs and performance of the interface. The x2APIC now uses 32 bits to address CPUs, allowing to address up to 232 − 1 CPUs using the physical destination mode. The logical destination mode now works differently and introduces clusters; using this mode, one can address up to 220 − 16 processors. The x2APIC architecture also provides backward compatibility modes to the original Intel APIC Architecture (introduced with the Pentium/P6) and with the xAPIC architecture (introduced with the Pentium 4).

movbe
Move Data After Swapping Bytes instruction. Performs a byte swap operation on the data copied from the second operand (source operand) and store the result in the first operand (destination operand). The source operand can be a general-purpose register, or memory loca-tion; the destination register can be a general-purpose register, or a memory location; however, both operands can not be registers, and only one operand can be a memory location. Both operands must be the same size, which can be a word, a doubleword or quadword. The MOVBE instruction is provided for swapping the bytes on a read from memory or on a write to memory; thus providing support for converting little-endian values to big-endian format and vice versa. In 64-bit mode, the instruction's default operation size is 32 bits. Use of the REX.R prefix permits access to additional registers (R8-R15). Use of the REX.W prefix promotes operation to 64 bits. See the summary chart at the beginning of this section for encoding data and limits.

popcnt
Return the Count of Number of Bits Set to 1 instruction (Hamming weight, i.e. bit count). These instructions operate on integer rather than SSE registers, because they are not SIMD instructions, but appear at the same time and although introduced by AMD with the SSE4a instruction set, they are counted as separate extensions with their own dedicated CPUID bits to indicate support. Intel implements POPCNT beginning with the Nehalem microarchitecture and LZCNT beginning with the Haswell microarchitecture. AMD implements both beginning with the Barcelona microarchitecture. Population count (count number of bits set to 1). Support is indicated via the CPUID.01H:ECX.POPCNT[Bit 23] flag.

aes
Advanced Encryption Standard Instruction Set (or the Intel Advanced Encryption Standard New Instructions; AES-NI) is an extension to the x86 instruction set architecture for microprocessors from Intel and AMD proposed by Intel in March 2008. The purpose of the instruction set is to improve the speed of applications performing encryption and decryption using the Advanced Encryption Standard (AES).

xsave
Save Processor Extended States: also provides XGETBY,XRSTOR,XSETBY. Performs a full or partial save of processor state components to the XSAVE area located at the memory address specified by the destination operand. The implicit EDX:EAX register pair specifies a 64-bit instruction mask. The specific state components saved correspond to the bits set in the requested-feature bitmap (RFBM), which is the logical-AND of EDX:EAX and XCR0.

avx
Advanced Vector Extensions. Advanced Vector Extensions (AVX) are extensions to the x86 instruction set architecture for microprocessors from Intel and AMD proposed by Intel in March 2008 and first supported by Intel with the Sandy Bridge processor shipping in Q1 2011 and later on by AMD with the Bulldozer processor shipping in Q3 2011. AVX provides new features, new instructions and a new coding scheme.

rdrand
Read Random Number from hardware random number generator instruction. RDRAND (previously known as Bull Mountain) is an instruction for returning random numbers from an Intel on-chip hardware random number generator which has been seeded by an on-chip entropy source. RDRAND is available in Ivy Bridge processors and is part of the Intel 64 and IA-32 instruction set architectures. AMD added support for the instruction in June 2015. The random number generator is compliant with security and cryptographic standards such as NIST SP 800-90A, FIPS 140-2, and ANSI X9.82. Intel also requested Cryptography Research Inc. to review the random number generator in 1999 and 2012, which resulted in two published papers: The Intel Random Number Generator in 1999, and Analysis of Intel's Ivy Bridge Digital Random Number Generator in 2012.

hypervisor
This flag is set by the hypervisor to indicate that your machine is running as a virtual machine on a hypervisor. Even though the presence of the flag is a good indicator that your machine is in fact a virtual machine running on a hypervisor you should be careful when building logic upon the presence of this flag. It is the decision of the hypervisor to push the flag or not. This means that if the flag is not pushed by the hypervisor, and is not present, the machine can still be a virtual machine instead of a bare-metal machine.

lahf_lm 
Load AH from Flags (LAHF) and Store AH into Flags (SAHF) in long mode.

abm
Advanced Bit Manipulation. ABM is only implemented as a single instruction set by AMD; all AMD processors support both instructions or neither. Intel considers POPCNT as part of SSE4.2, and LZCNT as part of BMI1. POPCNT has a separate CPUID flag; however, Intel uses AMD's ABM flag to indicate LZCNT support (since LZCNT completes the ABM).

3dnowprefetch
3DNow prefetch instructions. 3DNow! is an extension to the x86 instruction set developed by Advanced Micro Devices (AMD). It adds single instruction multiple data (SIMD) instructions to the base x86 instruction set, enabling it to perform vector processing, which improves the performance of many graphic-intensive applications. The first microprocessor to implement 3DNow was the AMD K6-2, which was introduced in 1998. When the application was appropriate this raised the speed by about 2-4 times. However, the instruction set never gained much popularity, and AMD announced on August 2010 that support for 3DNow would be dropped in future AMD processors, except for two instructions (the PREFETCH and PREFETCHW instructions). The two instructions are also available in Bay-Trail Intel processors.

rdseed
The RDSEED instruction. Non-deterministic random bit generator compatible with NIST SP 800-90B & C (drafts)

clflushopt
CLFLUSHOPT instruction. With the CLFLUSHOPT instruction, a store buffer only needs to be flushed if it holds data from the same cache line that the CLFLUSHOPT is accessing.   Store buffers holding any other addresses can continue to "cache" their stored data until some other mechanism forces them to push that data to the L1 Data Cache (where it becomes visible to all agents in the coherence fabric).

Reading and understanding the above flags will give you a good insight into the capabilities of your processor and how Oracle Linux is seeing the processor. Additionally, when you are writing low level code for processes that need to interact relatively direct with the hardware (or virtual hardware) it is of vital essence to understand what the options are at your disposal and what the machine is capable of doing. 

Oracle Linux – understanding Linux process memory mapping with pmap

Every operating system uses memory for running processes. In most cases you do not worry about how memory is used, how it is mapped and what the internal processes are that are governing the memory. However, in cases where you start to run into issues with performance and memory looks to be one of the root causes your system is not performing the way you expect it the time has come to take a deep dive into the Linux internals.

One of the tools which can be used to gain more understanding of how memory is used in somewhat more detail is pmap. You can use pmap to gain more insight in the memory usage of one or multiple processes based upon a specified PID.

Redis memory usage example:
In this example we will dive a bit into the memory consumption of a Redis instance running on Oracle Linux. To find out the current PID used by Redis we execute a ps command to find it as shown below.

[root@localhost proc]# ps -ef| grep redis-server | head -1
redis     1100     1  0 Jul04 ?        00:01:36 /usr/sbin/redis-server /etc/redis.conf
[root@localhost proc]#

now that we know that the redis-server is running with PID 1100 we can start to dive into the memory consumption using pmap. The pmap command takes the PID as input to display the desired information. An example of a details view pmap command is shown below, the –x flag is used to ensure pmap shows the details view instead of the more compact version.

[root@localhost proc]# pmap -x 1100
1100:   /usr/sbin/redis-server /etc/redis.conf
Address           Kbytes     RSS   Dirty Mode   Mapping
0000000000400000     312     312       0 r-x--  redis-server
000000000064d000       8       8       8 rw---  redis-server
000000000064f000      84      84      84 rw---    [ anon ]
00000000021ff000     132       4       4 rw---    [ anon ]
00007f55223fe000       4       0       0 -----    [ anon ]
00007f55223ff000    8192       8       8 rw---    [ anon ]
00007f5522bff000       4       0       0 -----    [ anon ]
00007f5522c00000    8192    2048    2048 rw---    [ anon ]
00007f5523400000    4096    2048    2048 rw---    [ anon ]
00007f5523c00000    4096    2048    2048 rw---    [ anon ]
00007f55242c7000    1576    1112       0 r-x--  libc-2.12.so (deleted)
00007f5524451000    2048       0       0 -----  libc-2.12.so (deleted)
00007f5524651000      16      16      16 r----  libc-2.12.so (deleted)
00007f5524655000       8       8       8 rw---  libc-2.12.so (deleted)
00007f5524657000      16      12      12 rw---    [ anon ]
00007f552465b000      92      84       0 r-x--  libpthread-2.12.so (deleted)
00007f5524672000    2048       0       0 -----  libpthread-2.12.so (deleted)
00007f5524872000       4       4       4 r----  libpthread-2.12.so (deleted)
00007f5524873000       4       4       4 rw---  libpthread-2.12.so (deleted)
00007f5524874000      16       4       4 rw---    [ anon ]
00007f5524878000       8       8       0 r-x--  libdl-2.12.so (deleted)
00007f552487a000    2048       0       0 -----  libdl-2.12.so (deleted)
00007f5524a7a000       4       4       4 r----  libdl-2.12.so (deleted)
00007f5524a7b000       4       4       4 rw---  libdl-2.12.so (deleted)
00007f5524a7c000     524      60       0 r-x--  libm-2.12.so (deleted)
00007f5524aff000    2044       0       0 -----  libm-2.12.so (deleted)
00007f5524cfe000       4       4       4 r----  libm-2.12.so (deleted)
00007f5524cff000       4       4       4 rw---  libm-2.12.so (deleted)
00007f5524d00000     128     128       0 r-x--  ld-2.12.so (deleted)
00007f5524f14000      16      16      16 rw---    [ anon ]
00007f5524f1e000       4       4       4 rw---    [ anon ]
00007f5524f1f000       8       8       8 r----  ld-2.12.so (deleted)
00007f5524f21000       4       4       4 rw---  ld-2.12.so (deleted)
00007f5524f22000       4       4       4 rw---    [ anon ]
00007ffc44e66000     132      20      20 rw---    [ stack ]
00007ffc44fa4000       8       0       0 r----    [ anon ]
00007ffc44fa6000       8       4       0 r-x--    [ anon ]
ffffffffff600000       4       0       0 r-x--    [ anon ]
----------------  ------  ------  ------
total kB           35904    8076    6368
[root@localhost proc]#

The above output can be used to gain more insight into the way Redis is using memory. As we used the –x flag in the previous example we did have a detailed view. In case you want to see the offset and the device ID you can use the –d flag as show in the example below:

[root@localhost proc]# pmap -d 1100
1100:   /usr/sbin/redis-server /etc/redis.conf
Address           Kbytes Mode  Offset           Device    Mapping
0000000000400000     312 r-x-- 0000000000000000 0fb:00001 redis-server
000000000064d000       8 rw--- 000000000004d000 0fb:00001 redis-server
000000000064f000      84 rw--- 0000000000000000 000:00000   [ anon ]
00000000021ff000     132 rw--- 0000000000000000 000:00000   [ anon ]
00007f55223fe000       4 ----- 0000000000000000 000:00000   [ anon ]
00007f55223ff000    8192 rw--- 0000000000000000 000:00000   [ anon ]
00007f5522bff000       4 ----- 0000000000000000 000:00000   [ anon ]
00007f5522c00000    8192 rw--- 0000000000000000 000:00000   [ anon ]
00007f5523400000    4096 rw--- 0000000000000000 000:00000   [ anon ]
00007f5523c00000    4096 rw--- 0000000000000000 000:00000   [ anon ]
00007f55242c7000    1576 r-x-- 0000000000000000 0fb:00001 libc-2.12.so (deleted)
00007f5524451000    2048 ----- 000000000018a000 0fb:00001 libc-2.12.so (deleted)
00007f5524651000      16 r---- 000000000018a000 0fb:00001 libc-2.12.so (deleted)
00007f5524655000       8 rw--- 000000000018e000 0fb:00001 libc-2.12.so (deleted)
00007f5524657000      16 rw--- 0000000000000000 000:00000   [ anon ]
00007f552465b000      92 r-x-- 0000000000000000 0fb:00001 libpthread-2.12.so (deleted)
00007f5524672000    2048 ----- 0000000000017000 0fb:00001 libpthread-2.12.so (deleted)
00007f5524872000       4 r---- 0000000000017000 0fb:00001 libpthread-2.12.so (deleted)
00007f5524873000       4 rw--- 0000000000018000 0fb:00001 libpthread-2.12.so (deleted)
00007f5524874000      16 rw--- 0000000000000000 000:00000   [ anon ]
00007f5524878000       8 r-x-- 0000000000000000 0fb:00001 libdl-2.12.so (deleted)
00007f552487a000    2048 ----- 0000000000002000 0fb:00001 libdl-2.12.so (deleted)
00007f5524a7a000       4 r---- 0000000000002000 0fb:00001 libdl-2.12.so (deleted)
00007f5524a7b000       4 rw--- 0000000000003000 0fb:00001 libdl-2.12.so (deleted)
00007f5524a7c000     524 r-x-- 0000000000000000 0fb:00001 libm-2.12.so (deleted)
00007f5524aff000    2044 ----- 0000000000083000 0fb:00001 libm-2.12.so (deleted)
00007f5524cfe000       4 r---- 0000000000082000 0fb:00001 libm-2.12.so (deleted)
00007f5524cff000       4 rw--- 0000000000083000 0fb:00001 libm-2.12.so (deleted)
00007f5524d00000     128 r-x-- 0000000000000000 0fb:00001 ld-2.12.so (deleted)
00007f5524f14000      16 rw--- 0000000000000000 000:00000   [ anon ]
00007f5524f1e000       4 rw--- 0000000000000000 000:00000   [ anon ]
00007f5524f1f000       8 r---- 000000000001f000 0fb:00001 ld-2.12.so (deleted)
00007f5524f21000       4 rw--- 0000000000021000 0fb:00001 ld-2.12.so (deleted)
00007f5524f22000       4 rw--- 0000000000000000 000:00000   [ anon ]
00007ffc44e66000     132 rw--- 0000000000000000 000:00000   [ stack ]
00007ffc44fa4000       8 r---- 0000000000000000 000:00000   [ anon ]
00007ffc44fa6000       8 r-x-- 0000000000000000 000:00000   [ anon ]
ffffffffff600000       4 r-x-- 0000000000000000 000:00000   [ anon ]
mapped: 35904K    writeable/private: 25012K    shared: 0K
[root@localhost proc]#

Additional flags that can be used are:
-q  quiet; less header/footer info
-V  show the version number
-A  limit results to the given range

As from version 3.3.4 also the flag –XX is introduced to provide even more information however this is not detailed in this post.

Reading pmap output:
Now that we understand how to get output from pmap the more important question is how to read and interpret this data so we can give it meaning.  The values returned by pmap are listed below, which values are returned is depending on the flags provided during execution of the pmap command.

Address
The address is to provide the start address of map.

Kbytes
The size of the page in kbytes. Depending on the flag given you will get a sum in the output of the pmap command.

RSS
RSS or resident set size which can be used to calculate how much of the memory is actually using main memory. The difference between the total amount of memory and the RSS values is what is not in memory. The difference of the occupied memory exists in the swap space or file system, either because some parts of the occupied memory were paged out, or because some parts of the executable were never loaded.

Dirty
dirty pages (both shared and private). This can be usefull when finding out which parts of the complete memory stack are responsible for the total set of dirty pages in your memory. For shared file-backed mappings, dirty pages can be written back to the underlying file if the kernel feels it has to make some room in RAM or that there are too many dirty pages.

Mode
Each map can have multiple modes set, reading the mode of a map will give you more insight and hints of what the use and role is of the map. The mode representation provided by pmap differs slightly from the actual kernel flags. Using the below list as reference provides a good insight in the meaning of a mode flag.

  • r: if set, the map is readable
  • w: if set, the map is writable
  • x: if set, the map contains executable code
  • s: if set, the map is shared. While the kerel maintains two flags for stating the shared or private mode of a map the pmap representation only shows s for shared while the kernel itself uses s for shared and p for private.
  • R: if set, the map has no swap space reserved (MAP_NORESERVE flag of mmap)

Offset
Offset into the file, this can be used in combination with the address to find the exact location.

Device
Device provides you the major:minor notation of the device used for storing the map.

Mapping
Each map has a mapping (type) associated with it. The following mappings can be observed when looking at pmap output;

  • Anon - non-file backed memory, Anonymous memory: Memory not relating to any named object or file within the file system is reported as [ anon ].
  • Stack – memory allocation reserved for the process stack
  • “file backed” – file backed maps will provide the name of the file that is mapped. You can observe the “(deleted)” statement associated with a file backed page. In effect the term is misleading as the file is not deleted as the term indicates, it is unlinked and the map can be deleted. The file will (in most cases) still be present. Also, you will notice that executable file (file backed maps with a mode x flag) will have multiple entries in the nmap output. Reason for this is that executable files are build up out of different segments who are mapped in different ways.


Wednesday, July 05, 2017

Oracle Linux – finding your CPU speed

People used to know the speed of the CPU in their server, mainly because they actively purchased the hardware they were using to run their systems. Now that the move to the cloud is becoming more and more standard this information is not always known. Cloud providers provide you Infrastructure as a Service and you can define the number of vCPU’s and other things however the speed of this specific CPU is not always known upfront.

When you are deploying high performance systems on the cloud it is good to understand what the speed is of the vPCU’s you have. In Oracle Linux, and in other distributions, you can relative easily find the speed of you cpu as it is noted down in /proc/cpuinfo

[vagrant@localhost ~]$ cat /proc/cpuinfo | grep MHz | uniq
cpu MHz  : 2903.998
[vagrant@localhost ~]$

The above example shows how to get the required information which is stored in /proc/cpuinfo where we use uniq to prevent values to appear multiple times. In general all the CPU’s have the same speed. In theory, this is not a given, however, in practice this is almost always the case and all you CPU’s will have the same speed. In case it is not you will see two entries and you will have to find out which CPU is having which speed.

Monday, June 26, 2017

Oracle Linux - bash check for HTTP response code

Bash is one of the most used scripting languages on Linux systems to undertake numerous checks on the system itself and towards other systems. Reason for this, it is available on all Linux hosts and it is very easy to script.

In some cases you might need to be able to know the HTTP status code from a remote HTTP server. A ping command might not do the full trick as it only tells you if the host is responsive to a network ping and getting the whole page might not be a good idea because an error page might be interpreted as a valid response. A better way is to check the HTTP response code that is returned.

In general HTTP resonse codes can be classified as follows:
1xx - informational reponses
2xx - success responses
3xx - redirect responses
4xx - client errors
5xx - server errors

This means that getting a 2xx response will indicate that the server is responding in a good fashion. On all other responses you might want to add some additional actions. Main questions is, how to get only the response code by using a bash script. The below example will do so:

curl -Lw '%{http_code}' -s -o /dev/null -I www.google.com

The above command will return 200 as a response code. In case we would have used a invalid domain name we would have received a error code. The above example is relative simple however will require curl to be present on the system. In case you do not want curl on your system you will have to find an alternative method to execute the check. 

Monday, June 12, 2017

Oracle Data Visualization Cloud Service - upload files

Within the Oracle Cloud portfolio Oracle has positioned the Oracle Data Visualization Cloud Service as the tool to explore your data, visualize it and share your information with other people within the enterprise. The Oracle Data Visualization Cloud Service can be used as a part of a data democratization strategy within an enterprise to provide all users access to data where and whenever they need it. The concept of data democratization is discussed in another blogpost on this blog.

Currently the Oracle Data Visualization Cloud Service provides two main ways of getting data in the Oracle Data Visualization Cloud Service. One is by connecting it to a oracle database source, for example located in the Oracle database cloud service, another is by uploading a file. Primarily CSV and XLSX files are supported for uploading data.

In most cases it is not a best practice to upload files as the data is relative static and is not connected to a live datasource as you would have with a database connection. However, in some cases it can be a good way to get data in. Examples are; users who add their own content and do not have the means to connect an Oracle database or relative static data.

Example data from DUO
In the example below we add a relative static piece of data to the Oracle Data Visualization Cloud Service. This is a year over year report of people dropping out of schools in the Netherlands. The data is per year, per location and per education type and is freely available as open-data from the DUO website. You can locate the file for reference here.

Loading a data file
When a user wants to load data into the Oracle Data Visualization Cloud Service the most easy way to do so from and end-user perspective is to use the GUI. Loading data includes a limited number of steps.

1) within the Data Sources section navigate to "Create" - " Data Source" Here you can select the type file by default. When selected you are presented with the option to select a file on your local file system.


2) the next step, after the file is uploaded, is to verify and if needed modify the definition of the uploaded data.


3) after this step is completed you will find the file ready for you use in your data sources as shown below.

In effect, those are the only actions needed by a user to add data to the Oracle Data Visualization Cloud Service.

Tuesday, June 06, 2017

Oracle Cloud - The value of edge computing

A term you currently see coming up is edge computing, edge computing is the model where you push computations and intelligence to the edge of the network or the edge of the cloud. Or, in the words of techtarget.com ; “ Edge computing is a distributed information technology (IT) architecture in which client data is processed at the periphery of the network, as close to the originating source as possible. The move toward edge computing is driven by mobile computing, the decreasing cost of computer components and the sheer number of networked devices in the internet of things (IoT).

Analysts state that edge computing is a technology trend with a medium business impact which we will see surfacing in 2017.


Even though the business impact is generally seen as medium it provides a large technology wise benefit and understanding the concepts of edge computing can be of vital importance, especially when you are developing IoT solutions or geographically distributed systems that rely on machine 2 machine communication

The high-level concept
From a high-level point of view edge computing states that computations should be done as close as possible to the location where the data is generated. This implies that raw data should not be send to a central location for computations, the basic computations should rather be done at the collection point.

As an example, if you would do license plate recognition to decide if a security gate would open for a certain car you can architect this in a couple of ways. The “traditional” way of doing this is having a very light weight system which would take a number of pictures as soon as a car would trigger the camera. The pictures would then be send to a central server where a license plate recognition algorithm would extract the information from the picture and compare the result against a database to make a decision to open the security gate or not.

Architecting the same functionality with edge computing would involve a number different steps. In the edge computing model the car would trigger the camera and the pictures would be taken. A small computer, possibly embedded within the camera, would run the license plate recognition algorithm and only the result would be send to a REST API to check if the gate should be opened or should remain closed.

The benefit of edge computing in this case is that you would have a lot less data which needs to be communicated between the camera and the central server. Instead of sending a number of high resolution photos or even a video stream you only have to communicate a JSON object containing the license plate information.  By doing so you can limit the amount of computing power needed at the central location and at the same time improve the speed of the end user experience.

A more data intensive example
The example of the license plate recognition is a good illustration of the concept, a bigger scale and more data intensive example could be an example using smart devices.

Such an example could be industrial (or home use) equipment which relies on the data collected by a set of sensors to make decisions. If we would take an industrial example this could be a smart skid in factory responsible for ensuring that a number of liquid storage containers are always filled to a certain extend and are always at a certain temperature and mix.


Such a skid as described above involves a large set of sensors as well as a large set potential actions on valves, pumps and heating equipment. Traditionally this was done based upon an industrial PLC in a disconnected manner where it was not possible to centrally monitor and manage the skid.

Certain architecture blueprints state that the sensor data should be collected more centrally to ensure a more centralized management and monitoring solution. The result of this is that all data is being send to a central location where computations are done on the received data. The resulting actions are being communicated back again. The result of this is that a lot of data needs to be communicated back and forward and a loss in communication can result is a preventive shutdown of a skid.

In an edge computing architecture, the skid would be equipped with a local computing power solution which would take care of all the computations that are in other cases done in a central location. All decision making would be done locally on the edge of the network  and a subset of data and a log of the actions being undertaken would be send to a central command and control server where the remote skid can be monitored and human intervention could be triggered.

In this model the loss of connectivity would not result in a preventive shutdown, the operations would be continuing for a much longer time given the operational parameters that the edge computer holds.

Oracle Cloud and MQTT
As already mentioned in the example of the remote skid, a way to communicate the data in a IoT fashion is using MQTT. MQTT stands for Message Queue Telemetry Transport, It is a publish/subscribe, extremely simple and lightweight messaging protocol, designed for constrained devices and low-bandwidth, high-latency or unreliable networks. The design principles are to minimize network bandwidth and device resource requirements whilst also attempting to ensure reliability and some degree of assurance of delivery. These principles also turn out to make the protocol ideal of the emerging “machine-to-machine” (M2M) or “Internet of Things” world of connected devices, and for mobile applications where bandwidth and battery power are at a premium.

On this blog we already discussed MQTT in combination with Oracle Linux and the Mosquitto MQTT message broker.

To facilitate the growing amount of IoT devices and the principle of edge computing which is relying on MQTT communication Oracle has included MQTT in the Oracle Cloud. Oracle primarily positions MQTT in combination with the Oracle IOT Cloud service in the form of a MQTT bridge. The Oracle IoT Cloud Service MQTT Bridge is a software application that must be installed and configured on Oracle Java Cloud Service in order to enable devices or gateways to connect to Oracle IoT Cloud Service over the MQTT communication protocol.


Within the Oracle cloud you see the MQTT bridge as a solution to connect remote devices to the Oracle IoT cloud via the MQTT protocol. The MQTT bridge receives the MQTT traffic and "translates" it to HTTPs calls which communicate with the Oracle IoT cloud Service.

In conclusion
As already outlined in the above examples, processing a large part of the computations at the edge of the network and implementing the principles of edge computing will drastically reduce the amount of computing power and storage capacity you need in the Oracle Public Cloud. In many cases, you can rely on MQTT communication or HTTPS communication where you call a REST API.

By pushing a large part of the computations to the edge your remote systems and devices become more reliable, even in cases where network communication is not always a given, and the resulting services become faster. 

Sunday, June 04, 2017

Oracle Linux - Using Consul for DNS based service discovery

Consul, developed by hashicorp,  is a solution for service discovery and configuration. Consul is completely distributed, highly available, and scales to thousands of nodes and services across multiple datacenters. Some concrete problems Consul solves: finding the services applications need (database, queue, mail server, etc.), configuring services with key/value information such as enabling maintenance mode for a web application, and health checking services so that unhealthy services aren't used. These are just a handful of important problems Consul addresses.

When developing microservices it is important that as soon as new instance of a microservice comes online it is able to register itself to a central registry, this process we call service registration. As soon as an instance of a microservice is registered at the central registry it can be used in the load balancing mechanism. If a call to a service is to be initiated to a microservice the service needs to be discovered via the central registry, the process is called service discovery.

In effect two ways are common for service discovery. One is based upon an API discovery model where the calling service discovers the service by executing a call to the service registry based upon a HTTP REST API call and receiving an endpoint which can be used. Commonly an URL based upon a IP and a port number.

The other common way is using a DNS based lookup against a service registry. The effect of doing a DNS based lookup is that you need to ensure that all the instances of a service are always running on the same port on all instance. Enforcing the same port number might be somewhat limiting in cases where your port number for each service instance can vary.

Using Consul on Oracle Linux
In an earlier blogpost I already outlined how to install Consul on Oracle Linux. In this post I will provide a quick inisght in how you can configure it on Oracle Linux. We build upon the installation done in the mentioned post.

We take the example of a service where we have two instances for, the name of the service is web and in effect is nothing more than a simple nginx webserver running in a production instance. Every time we call the service we want to discover the call by using a DNS lookup, we do however want to have this balanced, meaning we want to have a different IP being returned from the DNS server.

Configure a service in consul
We configure the service "web" manually in this case by creating a JSON file in /etc/consul.d and ensure we have the information about the two instances in this JSON file. An example of the file is shown below

{
 "services": [{
   "id": "web0",
   "name": "web",
   "tags": ["production"],
   "Address": "191.168.1.10",
   "port": 80
  },
  {
   "id": "web1",
   "name": "web",
   "tags": ["production"],
   "Address": "191.168.1.11",
   "port": 80
  }
 ]
}

As you can see, we have one name, "web" with two ID's; "web0" and "web1". The id's are used to identify the different instances of the service web. as you can see they have both a port noted next to the address. Even though it is good practice to have this in the configuration file it will not be used in the response from the internal consul DNS service as DNS will only return the addresses and not the ports.

Discover the service via Consul DNS
If we want to discover the service we can have our code to undertake a lookup against the DNS server. If we have configured the underlying Oracle Linux instance to have the Consul server in your /etc/resolv.conf file it will happen almost automatically. It will be important to make sure the ordering of your DNS servers is done correctly to improve resolving speed.

In effect all services configured in Consul will be by default part of .service.consul which will mean that if we want to do a DNS resolving for the web service we will have to do a resolve for web.service.consul. In the below example I have consul running on my localhost at port 8600 and I use dig to explicitly force dig to resolve it at this DNS server. As stated, if you configure it correctly you do not have to explicitly call it and you should be able to do a DNS resolve as you always do.

[root@localhost consul.d]#
[root@localhost consul.d]# dig +noall +answer @127.0.0.1 -p 8600 web.service.consul 
web.service.consul. 0 IN A 191.168.1.10
web.service.consul. 0 IN A 191.168.1.11
[root@localhost consul.d]#

As you can see from the above example I will get the two addresses returned from the Consul DNS server.

Consul and load-balancing
As microservices are commonly build up out of a number of instances of the same service we do want to ensure that load-balancing is done. We can already see from the dig example above that there are two instances. However, having them always in the same order returned will not ensure that the load is balanced over the two instances.

Consul will by default do a load-balancing and will return the IP's in a different order by rotating them in the DNS response. In the below example you can see that this is done when we call the DNS server a couple of times.

[root@localhost consul.d]#
[root@localhost consul.d]# dig +noall +answer @127.0.0.1 -p 8600 web.service.consul 
web.service.consul. 0 IN A 191.168.1.10
web.service.consul. 0 IN A 191.168.1.11
[root@localhost consul.d]#
[root@localhost consul.d]# dig +noall +answer @127.0.0.1 -p 8600 web.service.consul 
web.service.consul. 0 IN A 191.168.1.11
web.service.consul. 0 IN A 191.168.1.10
[root@localhost consul.d]#

In conclusion
If you are running a microservices based IT footprint and you are using Oracle Linux you can read in the referenced article how to install Consul to do service discovery and registration. Consul supports both API based as well as DNS based discovery. If you have your service instances always on the same and pre-defined port using DNS is a very good option to use for your service discovery process.


Oracle Cloud - Data democratization by using REST API’s

The idea of helping everybody to access and understand data is known as data democratization. Data democratization means breaking down silos and providing access to data when and where it is needed at any given moment.  By striving to have full data democratization within the enterprise is actually taking the step to a data driven company and nurture data driven decision making.

The general idea of data democratization and providing access to everyone in the company to use it is a very simple idea, the realization of this idea is however a very complex one in many cases. This is especially true in organically grown companies who have, over time, grown their IT footprint. In general, this includes a large set of legacy applications who do not by nature support integration that well.

However, the fact that an enterprise has a large set of legacy applications should not hold back the ambition to change to a more data driven enterprise. Moving to a more data driven enterprise, democratization of data and base decisions on actual data is a huge benefit for enterprise. Additionally, it is the starting point of integrating other systems and drive business in new and disruptive ways to keep the advantage over competitors.

Getting started
To get started with data democratization the first step is to start finding your data and classify the data sources. The below pointers can be of importance when evaluating the data.
- Data location : where is the data located, how easily can it be accessed
- Data ownership : which department owns the data
- Data confidentiality : How confidential is this data
- Data privacy : is there privacy related data in the set
- Data value : what is the monetary value of the data
- Data alignment : how well aligned is the data with other sources
 
Taking the above questions into mind when classifying all data this will give you a route of action per dataset. I will help you to identify how to handle each data-source, how to classify it and to integrate it. It also helps you to prioritize it.

Moving to the cloud
When moving to a data democratization model, this might be a turning point in how you look at IT and it might be good moment to consider the use of cloud. When trying to integrate and store a large set of data you can select, as an example, the Oracle Cloud to house the data you make available for all your users.

This is not necessarily meaning that you have to move the actual systems to the Oracle Cloud. One can think of a model where the backend systems remain in your current datacenter or cloud and you move / sync your data and the changes to the Oracle Cloud where you unlock them to the users using REST API’s and portals in the form of a data shop.

Opening up with a data shop
The concept of a data shop is the way to get started with data democratization. A data shop is a self-service portal where users can gain access to all the data that you have liberated. It provides users the option to get access to REST API’s or to, as an example, the Oracle Data Visualization Cloud Service, which can show data already included in graphs and other visualization.

As with a real shop, a large number of “products” are available. Some are for the standard users in the form of pre-defined dashboards and reports and some users will require the data in a rawer format to make and share their om reports and analysis.

Making it easy
Making it easy for data consumers to use the data is actually two folded. You will have two types of consumers, the tech consumers and the non-tech consumers. The tech consumers will require REST API’s to gain access to the data and undertake all the actions they need and think are valuable. The other type of consumers are the non-tech users. For non-tech users the REST API approach might be too difficult to master and they will need a more simple way to gain access to the liberated data.

After you moved your data to the cloud , as a first step in the process you will have to ensure that the data is accessible, via REST API’s and also via standard dashboards. Oracle is providing a growing number of options in the Oracle Public Cloud to do both. You can use standard visualization and data exploration tooling to your users within the cloud which have a relative low learning curve and people can start with right away. An example of this is the Oracle Data Visualization Cloud Service.


Oracle is also providing API functionality, even though the services Oracle provide standard from within the database and with some of the cloud services it might very well be beneficial to consider building your own REST API implementation while leveraging both the Oracle Compute Cloud Service with Oracle Linux instances and the Oracle Container Cloud Service.

Putting it all together
With data democratization, you open up your data, break the silo way of architecture and provide your users the option to analyze the data and make use of a active and up-to-date collection of data in one single place, the data shop. Moving to the cloud and leveraging the cloud is a technical solution to make this happen. Moving to the cloud is not the goal for data democratization. 

Wednesday, May 24, 2017

Oracle Linux - capture context switching in Linux

Before we dive into the subject, context switching is normal, the Linux operating system needs context switching to function, no need to worry. The question is, if it is normal why would you like to monitor it? Because, as with everything, normal behavior is accepted however behavior which gets out of bounds will cause an issue. With context switching, you are ok in expecting a certain number of context switching at every given moment in time, however, when the number of context switches get out of hand this can result in slow execution of processes for the users of the system.

Definition of context switching
The definition to context switching given by the Linux Information Project is as follows: “A context switch (also sometimes referred to as a process switch or a task switch) is the switching of the CPU (central processing unit) from one process or thread to another. A process (also sometimes referred to as a task) is an executing (i.e., running) instance of a program. In Linux, threads are lightweight processes that can run in parallel and share an address space (i.e., a range of memory locations) and other resources with their parent processes (i.e., the processes that created them).

A context switch comes with a cost, it takes and capacity to undertake the context switch. Meaning, if you can prevent a context switch this is good and will help in the overall performance of the systems. In effect, context switching comes in two different types, voluntary context switches and non-voluntary context switches.

Voluntary context switches
When running a process can decide to initiate a context switch, if the decision is made by the code itself we talk about a voluntary context switch (voluntary_ctxt_switches). This can be for example that you voluntarily give up your execution time by calling sched_yield or you can put a process to sleep while waiting for some event to happen.

Additionally, a voluntary context switch will happen when your computation completes prior to the allocated timeslice expires.

All acceptable when used in the right manner and when you are aware of the costs of a context switch.

non-voluntary context switches
Next to the voluntary context we have the non-voluntary context switches (nonvoluntary_ctxt_switches). A non-voluntary context switch happens when a process becomes unresponsive, however, it also happens when the task is not completed within the given timeslice. When the task is not completed in the given timeslice the state will be saved and a non-voluntary context switch happens.

Prevent context switching
When trying to develop high performance computing solutions you should try to, at least, be aware of context switching and take it into account. Even better try to minimize the number of voluntary context switches and try to find the cause of every non-voluntary context switch.
As context switching comes with a cost you want to minimize this as much as possible, and when a non-voluntary context switch happens the state needs to be saved and the task is placed back in the scheduler queue needing to wait again for a execution timeslice. This makes the overall performance of your system slow down and the specific code you have written becomes even more slow.

Check proc context switches
When working on Linux, we are using Oracle Linux in this example however this applies for most systems, you can check information on context switches by looking into the status which can be located at /proc/{PID}/status in the below example we check for the voluntarty and non-voluntary context switches of pid 25334.

[root@ce /]#
[root@ce /]# cat /proc/25334/status | grep _ctxt_
voluntary_ctxt_switches: 687
nonvoluntary_ctxt_switches: 208
[root@ce /]#

As you can see the number of voluntary context switches is (at this moment) 687 and the number of non-voluntary context switches is 208. This is a quick and dirty way of determining the number of context switches that a specific PID has had at a specific moment.

Monitor context switches
You can monitor your systems for context switching. Even though you are able to do so, you will need a good case to do it. Even though it provides information on your system in most cases and deployments there is no real need to monitor the number of context switches constantly. Having stated that, there are also a lot of cases where monitoring context switching can be vital for ensuring the health of your server and/or compare nodes in a wide cluster.

A quick and dirty way of monitoring your context switches is by taking a sample. For example you could take a sample of the average number of context switches for all processes on you Linux instance that execute a context switch in the sample timeframe.

The below example script takes a 10 second sample of the context switches and provide the output of only the relevant data for this we use the pidstat command which can be installed by installing the sysstat package which is available on the Oracle Linux YUM repository.

pidstat -w 2 1 | grep Average | grep -v pidstat | sort -n -k4 | awk '{ if ($2 != "PID") print "ctxt sample:" $2" - "  $3 " - " $4 " - "  $5}'

The full example in our case looks like the one below:

[root@ce tmp]# pidstat -w 2 1 | grep Average | grep -v pidstat | sort -n -k4 | awk '{ if ($2 != "PID") print "ctxt sample:" $2" - "  $3 " - " $4 " - "  $5}'
ctxt sample:12 - 0.50 - 0.00 - watchdog/0
ctxt sample:13 - 0.50 - 0.00 - watchdog/1
ctxt sample:15 - 0.50 - 0.00 - ksoftirqd/1
ctxt sample:18 - 3.00 - 0.00 - rcuos/1
ctxt sample:2183 - 1.00 - 0.00 - memcached
ctxt sample:2220 - 1.00 - 0.00 - httpd
ctxt sample:52 - 1.00 - 0.00 - kworker/1:1
ctxt sample:56 - 1.50 - 0.00 - kworker/0:2
ctxt sample:7 - 14.00 - 0.00 - rcu_sched
ctxt sample:9 - 11.50 - 0.00 - rcuos/0
[root@ce tmp]#

to understand the output we have to look at how pidstat normally provides the output. The below is an example of the standard pidstat output:

[root@ce tmp]# pidstat -w 2 1
Linux 4.1.12-61.1.28.el6uek.x86_64 (testbox7.int)  05/23/2017  _x86_64_ (2 CPU)

03:24:37 PM       PID   cswch/s nvcswch/s  Command
03:24:39 PM         3      0.50      0.00  ksoftirqd/0
03:24:39 PM         7     14.43      0.00  rcu_sched
03:24:39 PM         9      9.45      0.00  rcuos/0
03:24:39 PM        18      3.98      0.00  rcuos/1
03:24:39 PM        52      1.00      0.00  kworker/1:1
03:24:39 PM        56      1.49      0.00  kworker/0:2
03:24:39 PM      1557      0.50      0.50  pidstat
03:24:39 PM      2183      1.00      0.00  memcached
03:24:39 PM      2220      1.00      0.00  httpd

Average:          PID   cswch/s nvcswch/s  Command
Average:            3      0.50      0.00  ksoftirqd/0
Average:            7     14.43      0.00  rcu_sched
Average:            9      9.45      0.00  rcuos/0
Average:           18      3.98      0.00  rcuos/1
Average:           52      1.00      0.00  kworker/1:1
Average:           56      1.49      0.00  kworker/0:2
Average:         1557      0.50      0.50  pidstat
Average:         2183      1.00      0.00  memcached
Average:         2220      1.00      0.00  httpd
[root@ce tmp]#

As you can see from the “script” we print $2, $3, $4 and $5 for all average data where S2 is not “PID”. This gives us all the clear data. In our case the columns we show are the following:

$2 – the PID
$3 – number of voluntary context switches in the given sample time
$4 – number of non-voluntary context switches in the given sample time
$5 – the command name


How to use the monitor data
Collecting data, collecting sample data via monitoring is great, however when not used it is worthless and has to justify the costs of running the collector. As collecting the number of context switches has a cost you need to make sure you really need the data. A couple of ways you can use the data are described below and given a potential value in your maintenance and support effort.

Case 1 - Node comparison
This can be useful when you want to compare nodes in a wider cluster. Checking the number of context switches will be part of a wider set of checks and taking sample data. The number of context switches can be a good datapoint in the overall comparison of what is happening and what the difference between nodes is.

Case 2 - Version comparison
This can be a good solution in cases where you often have new version (builds / deployments) of code to your systems and want to track subtle changes in behavior of how the systems are working in a subtle manner.

Case 3 – Outlier detection
Outlier detection to detect subtle changes in the way the system is behaving over time. You can couple this to machine learning to detect changes over time. The number of context switches changing over time can be an indicator of a number of things and can be a pointer for a deeper investigation to tune your code.

Case 4 – (auto) scaling
Detecting the number of context switches, in combination with other datapoints can be input for scaling the number of nodes up and down. This in general is coupled with CPU usage, transaction timing and others. Adding context switching as an additional datapoint can be very valuable.

The site reliability engineering way
When applying the above you can adopt this in your SRE (site reliability engineering) strategy as one of the inputs to monitor your systems, automatically detect trends and prevent potential issues and feedback to developers on certain behaviour of the code in real production deployments.

Tuesday, May 09, 2017

Oracle Linux - Installing dtrace

When checking the description of dtrace for Oracle Linux on the Oracle website we can read the following: "DTrace is a comprehensive, advanced tracing tool for troubleshooting systematic problems in real time.  Originally developed for Oracle Solaris and later ported to Oracle Linux, it allows administrators, integrators and developers to dynamically and safely observe live systems for performance issues in both applications and the operating system itself.  DTrace allows you to explore your system to understand how it works, track down problems across many layers of software, and locate the cause of any aberrant behavior.  DTrace gives the operational insights that have long been missing in the data center, such as memory consumption, CPU time or what specific function calls are being made."

Which sounds great, and to be honest using dtrace helps enormously in finding and debugging issues on your Oracle Linux system in cases you need to go one level deeper than you would normally go to find an issue.

Downloading dtrace
If you want to install dtrace one way you can do this is by downloading the files from the oracle website, you can find the two RPM's at this location.

Installing dtrace
when installing dtrace you might run into some dependency issues that are not that obvious to resolve. Firstly they have a dependency on each other. This means you will have to install the RPM files in the right order. You can see this below;

[root@ce vagrant]# rpm -ivh dtrace-utils-0.5.1-3.el6.x86_64.rpm 
error: Failed dependencies:
 cpp is needed by dtrace-utils-0.5.1-3.el6.x86_64
 dtrace-modules-shared-headers is needed by dtrace-utils-0.5.1-3.el6.x86_64
 libdtrace-ctf is needed by dtrace-utils-0.5.1-3.el6.x86_64
 libdtrace-ctf.so.1()(64bit) is needed by dtrace-utils-0.5.1-3.el6.x86_64
 libdtrace-ctf.so.1(LIBDTRACE_CTF_1.0)(64bit) is needed by dtrace-utils-0.5.1-3.el6.x86_64
[root@ce vagrant]# rpm -ivh dtrace-utils-devel-0.5.1-3.el6.x86_64.rpm 
error: Failed dependencies:
 dtrace-modules-shared-headers is needed by dtrace-utils-devel-0.5.1-3.el6.x86_64
 dtrace-utils(x86-64) = 0.5.1-3.el6 is needed by dtrace-utils-devel-0.5.1-3.el6.x86_64
 libdtrace-ctf-devel > 0.4.0 is needed by dtrace-utils-devel-0.5.1-3.el6.x86_64
 libdtrace-ctf.so.1()(64bit) is needed by dtrace-utils-devel-0.5.1-3.el6.x86_64
 libdtrace.so.0()(64bit) is needed by dtrace-utils-devel-0.5.1-3.el6.x86_64
 libdtrace.so.0(LIBDTRACE_PRIVATE)(64bit) is needed by dtrace-utils-devel-0.5.1-3.el6.x86_64
[root@ce vagrant]# 

As you can see, you also have a number of other dependencies. The most easy way to resolve this is to simply use YUM to install both RPM's from your local machine and leverage the power of YUM to install the rest of the dependencies. For this we will use the yum localinstall dtrace-utils-* command.

Now we can quickly check if dtrace is indeed installed by executing the dtrace command without any specific option. You should see the below on your terminal:

[root@ce vagrant]# dtrace
Usage: dtrace [-32|-64] [-aACeFGhHlqSvVwZ] [-b bufsz] [-c cmd] [-D name[=def]]
 [-I path] [-L path] [-o output] [-p pid] [-s script] [-U name]
 [-x opt[=val]] [-X a|c|s|t]

 [-P provider [[ predicate ] action ]]
 [-m [ provider: ] module [[ predicate ] action ]]
 [-f [[ provider: ] module: ] func [[ predicate ] action ]]
 [-n [[[ provider: ] module: ] func: ] name [[ predicate ] action ]]
 [-i probe-id [[ predicate ] action ]] [ args ... ]

 predicate -> '/' D-expression '/'
    action -> '{' D-statements '}'

 -32 generate 32-bit D programs and ELF files
 -64 generate 64-bit D programs and ELF files

 -a  claim anonymous tracing state
 -A  generate driver.conf(4) directives for anonymous tracing
 -b  set trace buffer size
 -c  run specified command and exit upon its completion
 -C  run cpp(1) preprocessor on script files
 -D  define symbol when invoking preprocessor
 -e  exit after compiling request but prior to enabling probes
 -f  enable or list probes matching the specified function name
 -F  coalesce trace output by function
 -G  generate an ELF file containing embedded dtrace program
 -h  generate a header file with definitions for static probes
 -H  print included files when invoking preprocessor
 -i  enable or list probes matching the specified probe id
 -I  add include directory to preprocessor search path
 -l  list probes matching specified criteria
 -L  add library directory to library search path
 -m  enable or list probes matching the specified module name
 -n  enable or list probes matching the specified probe name
 -o  set output file
 -p  grab specified process-ID and cache its symbol tables
 -P  enable or list probes matching the specified provider name
 -q  set quiet mode (only output explicitly traced data)
 -s  enable or list probes according to the specified D script
 -S  print D compiler intermediate code
 -U  undefine symbol when invoking preprocessor
 -v  set verbose mode (report stability attributes, arguments)
 -V  report DTrace API version
 -w  permit destructive actions
 -x  enable or modify compiler and tracing options
 -X  specify ISO C conformance settings for preprocessor
 -Z  permit probe descriptions that match zero probes
[root@ce vagrant]# 

All ready and set to start with dtrace on your Oracle Linux instance. As an addition, you will also have to install the below mentioned packages for your specific machine:

yum install dtrace-modules-`uname -r`

Oracle Linux - using pstree to find processes

When checking which processes are running on your Oracle Linux instance you can use the ps command. Most likely the ps command is the most likely the most used command to find processes, and for good reasons as it is very easy to use. However, when you want some more insight and a more easy view what is related to what the pstree option can be very useable.

pstree shows running processes as a tree. The tree is rooted at either pid or init if pid is omitted. If a user name is specified, all process trees rooted at processes owned by that user are shown. pstree visually merges identical branches by putting them in square brackets and prefixing them with the repetition count. As an example you can see the below standard output of pstree without any additional options specifief;

[root@ce tmp]#
[root@ce tmp]#  pstree
init─┬─VBoxService───7*[{VBoxService}]
     ├─acpid
     ├─crond
     ├─dhclient
     ├─httpd───10*[httpd]
     ├─java───26*[{java}]
     ├─java───47*[{java}]
     ├─java───35*[{java}]
     ├─java───29*[{java}]
     ├─java───46*[{java}]
     ├─memcached───5*[{memcached}]
     ├─6*[mingetty]
     ├─ntpd
     ├─2*[rsyslogd───3*[{rsyslogd}]]
     ├─2*[sendmail]
     ├─slapd───5*[{slapd}]
     ├─sshd───sshd───sshd───bash───sudo───su───bash───pstree
     └─udevd───2*[udevd]
[root@ce tmp]#
[root@ce tmp]# 

As you can see in the above example httpd is between brackets and 10 is mentioned. Which means that 10 which indicates that more processes are running as httpd. Below is shown a part of the full tree (removed the lower part for readability):

[root@ce tmp]# pstree -p
init(1)─┬─VBoxService(1182)─┬─{VBoxService}(1185)
        │                   ├─{VBoxService}(1186)
        │                   ├─{VBoxService}(1187)
        │                   ├─{VBoxService}(1188)
        │                   ├─{VBoxService}(1189)
        │                   ├─{VBoxService}(1190)
        │                   └─{VBoxService}(1191)
        ├─acpid(1130)
        ├─crond(1275)
        ├─dhclient(984)
        ├─httpd(3612)─┬─httpd(3616)
        │             ├─httpd(3617)
        │             ├─httpd(3618)
        │             ├─httpd(3619)
        │             ├─httpd(3620)
        │             ├─httpd(3621)
        │             ├─httpd(3622)
        │             ├─httpd(3623)
        │             ├─httpd(5020)
        │             └─httpd(5120)
        ├─java(3993)─┬─{java}(3996)
        │            ├─{java}(3997)
        │            ├─{java}(3998)

This shows the main process (PID 3612) and all other processes that are forked from this process. Using pstree is primarily (in my opinion) to support you when doing some investigation on a machine and is not by default the best tool to use when scripting solutions on Oracle Linux. Having stated that, it is a great tool to use.

Monday, May 08, 2017

Oracle Linux - get your external IP in bash with ifconfig.me

Developing code that helps you in automatic deployments can be a big timesaver. Repeating tasks for installing servers, configuring them and deploying code on them is something which is more and more adopted by enterprises as part of DevOps and continuous integration and continuous deployment methods. When you use scripting for automatic deployment of your code in your own datacenter the beauty is that you fairly well know how the infrastructure looks and you have a fairly good view on how, for example, your machine will be accessible from the outside world. For example, if you deploy a server that has an external IP address on the outside of the network edge you should be able to determine this relatively easy even in cases where this IP is not the IP of your actual machine.

If you however provide scripting which you distribute you will not be able to apply the logic you might apply in your own network. For this you need some way to find out the external IP address. And, as stated, this can be something totally different than the IP which the machine actually has from your local operating system point of view.

The people at ifconfig.me have done some great work by providing a quick service to resolve this problem. ifconfig.me provide a service that will provide you all the information need (and more) in a manner that is easily included in bash scripting.

As an example, in case you would need your external IP to use in a configuration in your Oracle Linux deployment you could execute the below command:

[root@ce tmp]# curl ifconfig.me
172.217.17.46
[root@ce tmp]#

(do note, this is not my IP as I do not use a google webhost as one of my test machines). As you can see this is relative easy and to provide an example of how you could include this in a bash script you can review the below code snippet:

#!/bin/bash

 myIp=`curl -s ifconfig.me`

 echo $myIp

And, even though this is a very quick and easy solution to a problem you could face when you try to automate a number of steps while scripting the ifconfig.me provides more options. A number of options to get information from the "external" view are available and can all be found at the ifconfig.me website. However most important one is the ability to do a curl to ifconfig.me/all.json which will return a JSON based response with all the information in it. This makes it parsable. And to make it more easy, Oracle has included jq in the YUM repository which makes parsing JSON even more easy. An example of the JSON response from ifconfig.me is shown below (again.... using a fake google webhost and not my own private information.

{
 "connection": "",
 "ip_addr": "172.217.17.46",
 "lang": "",
 "remote_host": "ams16s29-in-f46.1e100.net",
 "user_agent": "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2",
 "charset": "",
 "port": "54944",
 "via": "",
 "forwarded": "",
 "mime": "*/*",
 "keep_alive": "",
 "encoding": ""
}

Thursday, May 04, 2017

Oracle Linux - find files after yum installation

Installing software on Oracle Linux is relative easy using the yum command. The downside of the ease of installing is that you do have a lot of files being "dumped" on the filesystem without keeping clear track of what is exactly installed where. Keeping your filesystem clean and understanding what ends up in which location is vital for maintaining a good working Linux instance.

A couple of options are available to keep track of what is installed where and provide you with a list of files which shows where things ended up on the filesystem.

First option is making use of the repoquery utility. Where repoquery is part of the yum-utils package you will have to ensure that you have installed repo-utils. Basically you can check this by checking if you have the repoquery utility (which is a good hint) or you can check by using the yum command as shown below:

[root@ce ~]# yum list installed yum-utils
Loaded plugins: security
Installed Packages
yum-utils.noarch                                                                                    1.1.30-40.0.1.el6                                                                                    @public_ol6_latest
[root@ce ~]#

If you have the repoquery utility you an use it to find out which files are installed into which location. An example of this is shown below where we check what is installed and where it is installed when we did the installation of yum-util 

[root@ce ~]# 
[root@ce ~]# repoquery --installed -l yum-utils
/etc/bash_completion.d
/etc/bash_completion.d/yum-utils.bash
/usr/bin/debuginfo-install
/usr/bin/find-repos-of-install
/usr/bin/needs-restarting
/usr/bin/package-cleanup
/usr/bin/repo-graph
/usr/bin/repo-rss
/usr/bin/repoclosure
/usr/bin/repodiff
/usr/bin/repomanage
/usr/bin/repoquery
/usr/bin/reposync
/usr/bin/repotrack
/usr/bin/show-changed-rco
/usr/bin/show-installed
/usr/bin/verifytree
/usr/bin/yum-builddep
/usr/bin/yum-config-manager
/usr/bin/yum-debug-dump
/usr/bin/yum-debug-restore
/usr/bin/yum-groups-manager
/usr/bin/yumdownloader
/usr/lib/python2.6/site-packages/yumutils
/usr/lib/python2.6/site-packages/yumutils/__init__.py
/usr/lib/python2.6/site-packages/yumutils/__init__.pyc
/usr/lib/python2.6/site-packages/yumutils/__init__.pyo
/usr/lib/python2.6/site-packages/yumutils/i18n.py
/usr/lib/python2.6/site-packages/yumutils/i18n.pyc
/usr/lib/python2.6/site-packages/yumutils/i18n.pyo
/usr/sbin/yum-complete-transaction
/usr/sbin/yumdb
/usr/share/doc/yum-utils-1.1.30
/usr/share/doc/yum-utils-1.1.30/COPYING
/usr/share/doc/yum-utils-1.1.30/README
/usr/share/doc/yum-utils-1.1.30/yum-util-cli-template
/usr/share/locale/da/LC_MESSAGES/yum-utils.mo
/usr/share/man/man1/debuginfo-install.1.gz
/usr/share/man/man1/find-repos-of-install.1.gz
/usr/share/man/man1/needs-restarting.1.gz
/usr/share/man/man1/package-cleanup.1.gz
/usr/share/man/man1/repo-graph.1.gz
/usr/share/man/man1/repo-rss.1.gz
/usr/share/man/man1/repoclosure.1.gz
/usr/share/man/man1/repodiff.1.gz
/usr/share/man/man1/repomanage.1.gz
/usr/share/man/man1/repoquery.1.gz
/usr/share/man/man1/reposync.1.gz
/usr/share/man/man1/repotrack.1.gz
/usr/share/man/man1/show-changed-rco.1.gz
/usr/share/man/man1/show-installed.1.gz
/usr/share/man/man1/verifytree.1.gz
/usr/share/man/man1/yum-builddep.1.gz
/usr/share/man/man1/yum-config-manager.1.gz
/usr/share/man/man1/yum-debug-dump.1.gz
/usr/share/man/man1/yum-debug-restore.1.gz
/usr/share/man/man1/yum-groups-manager.1.gz
/usr/share/man/man1/yum-utils.1.gz
/usr/share/man/man1/yumdownloader.1.gz
/usr/share/man/man8/yum-complete-transaction.8.gz
/usr/share/man/man8/yumdb.8.gz
[root@ce ~]# 
[root@ce ~]# 

This will help you to keep track of what is installed in which location and can support in ensuring you have a clean system.