Oracle Linux - SSH slow login

When you run a lot of test installations of Oracle Linux, like I do on my laptop and my home Oracle VM installation, you do not have them all configured in DNS. When hopping from Linux machine to Linux machine using SSH you do often have the situation that there is a long time between the moment you enter your username and the moment you are asked for the password. Reason for this is that the SSH deamon by default will try to do a DNS lookup to retrieve the machine name from DNS that you use to login to. When running Oracle Linux in a operational environment where you most likely need to have an audit trail this is absolutely a good way of working. However, in case you run multiple lab an play machines in your local environment this is not needed and the wait between the moment you enter your username and the moment you are asked for the password is quickly becoming an annoyance.

To change the behaviour of the SSH deamon you will need to change the configuration file /etc/ssh/sshd_config. You have to ensure that UseDNS no is included in the file. Standard deployment of Oracle Linux is that UseDNS yes is commented out of the configuration. The default behavior is already set to yes so you have to explicitly include "UseDNS no". To esure that the settings are applied restart the service after you made the changes to the file.

[root@localhost ~]# service sshd restart
Redirecting to /bin/systemctl restart  sshd.service
[root@localhost ~]#

Oracle VM - Anti-Affinity Groups

Using Oacle VM to virtualize machines and run them as virtual machines is a great way to reduce the number of physical machines you need to run a large estate of virtual machines. In general cloud administrators should not be worried about the exact physical machine a virtual machine is started. Oracle VM will select, based upon an algorithm on which physical machine the virtual machine will start. However, in some cases you should very much need to worry about the fact where a virtual machine will start. Especially that you do not want a virtual machine started on the same machine as some other specific virtual machines are running.

An example of this is when you run a high availability cluster of databases on virtual machines. What you want to prevent is that all nodes of the cluster run on the same physical hardware. Obvious reason for this is that you do not want your entire cluster to fail in case one physical box fails. For a long time you where required to ensure this manually or use custom scripting. However, within Oracle VM server you now have the Anti-Affinity groups option.

Virtual machines placed in the same Anti-Affinity group will not be started on the same physical hardware. Meaning, if you have a cluster of n virtual machines, all hosting a node of the same database cluster you can place them into one Anti-Affinity group and the algorithm responsible for selecting a physical machine to start the virtual machine will take this into account.

The above image shows a Anti-Affinity group named MyAAGroup which holds 3 virtual machines all members of database cluster. By placing them into the same Anti-Affinity group you will have the insurance that they will never be started on the same physical hardware and by doing so you will honor the Maximum Availability Architecture principles in this specific area. 

Oracle Linux - change IO scheduler

When installing Oracle Linux you will be equipped with a I/O scheduler which is perfectly usable for a database. This is not surprising as Oracle is from origin a database vendor. Input/output (I/O) scheduling is the method that computer operating systems use to decide in which order the block I/O operations will be submitted to storage volumes. I/O scheduling is sometimes called disk scheduling. However, even though the fact that you get a good scheduler there might be a need to change the default scheduler for some reason. A number of reasons can be thought of, depending on the type of I/O your performance can improve by selecting a different scheduler.

The below image shows the overall view on the Linux Storage Stack which includes the scheduler within the Block Layer. This image shows the full stack which includes more components then only the I/O scheduler.

In case you want to check what the current I/O scheduler is for a specific device you can do this by using the following command (for example for sda):

cat /sys/block/sda/queue/scheduler

This will show you the current scheduler that is used. For example, the output could be the one shown in the example below:

[root@demo1 etc]# cat /sys/block/sda/queue/scheduler
noop [deadline] cfq
[root@demo1 etc]# 

In case you need to change this there are some differences in how to do this. To activate a different scheduler, for example change it to cfq you can cat the new scheduler in by using the below commands:

[root@demo1 etc]#
[root@demo1 etc]# cat /sys/block/sda/queue/scheduler
noop [deadline] cfq
[root@demo1 etc]# cat cfq > /sys/block/sda/queue/scheduler
[root@demo1 etc]# cat /sys/block/sda/queue/scheduler
noop deadline [cfq]
[root@demo1 etc]#

As you can see this has changed the scheduler to cfq and no longer deadline is selected as the standard I/O scheduler. To make things persistent you have to change the grub configuration. When you are using grub2 the process is a bit different from using grub which is still in most standing Linux implementations. When using grub2 you have to edit the default grub2 file which is located at /etc/default/grub here you will have to add the new scheduler to GRUB_CMDLINE_LINUX which could look like the example below:

GRUB_CMDLINE_LINUX="crashkernel=auto  vconsole.font=latarcyrheb-sun16 rd.lvm.lv=ol/swap rd.lvm.lv=ol/root vconsole.keymap=us rhgb quiet"

If we, for example, like to make cfq persitant we have to change the line into the example below by add elevator=cfq to it:

GRUB_CMDLINE_LINUX="crashkernel=auto  vconsole.font=latarcyrheb-sun16 rd.lvm.lv=ol/swap rd.lvm.lv=ol/root vconsole.keymap=us rhgb quiet elevator=cfq"

This only has placed the new information into the defaults file and not yet into the grub2.cfg file where it is needed during boot. To ensure this is added to the grub2.cfg file you have run grub2-mkconfig and ensure the output is directed to /boot/grub2/grub.cfg as shown in the example below.

[root@demo1 default]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-123.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-123.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.8.13-35.3.1.el7uek.x86_64
Found initrd image: /boot/initramfs-3.8.13-35.3.1.el7uek.x86_64.img
Warning: Please don't use old title `Oracle Linux Server, with Unbreakable Enterprise Kernel 3.8.13-35.3.1.el7uek.x86_64' for GRUB_DEFAULT, use `Advanced options for Oracle Linux Server>Oracle Linux Server, with Unbreakable Enterprise Kernel 3.8.13-35.3.1.el7uek.x86_64' (for versions before 2.00) or `gnulinux-advanced-8f652ccf-3540-4549-9a5c-1d126e882d35>gnulinux-3.8.13-35.3.1.el7uek.x86_64-advanced-8f652ccf-3540-4549-9a5c-1d126e882d35' (for 2.00 or later)
Found linux image: /boot/vmlinuz-0-rescue-782e1cbce43c4c9d8829bd4addd5f09d
Found initrd image: /boot/initramfs-0-rescue-782e1cbce43c4c9d8829bd4addd5f09d.img
done
[root@demo1 default]#

If we now reboot the machine and check again what the scheduler is that is applied on sda we can see that cfq has been selected as the default scheduler for this device:

[root@demo1 ~]# cat /sys/block/sda/queue/scheduler
noop deadline [cfq]
[root@demo1 ~]#

When working with a grub bootloader you can directly change the scheduler in /etc/grub.conf however, withthe introduction of grub2 this is no longer an option and you need to take the above mentioned steps to change the I/O scheduler in Oracle Linux. 

Oracle Enterprise Manager query table space sizes

Oracle Enterprise Manager provides you the ideal solution to manage a large number of targets. All information about the targets, for example Oracle databases, is stored in the Oracle Enterprise Manager Repository database. What makes it interesting is that you can query the database with SQL and get information out of it quickly, showing you exactly what you need.

In the below example we do query the total size of the Oracle database tablesize per database. The query provides a list of all databases that are registered as a target in OEM in combination with the name of the server it is running on and the total size of the table space.

SELECT
      HOST_NAME,
      TARGET_NAME,
      round(SUM(TABLESPACE_SIZE)/1024/1024,0) AS "DB_TABLESPACE_USED"
FROM 
    MGMT$DB_TABLESPACES
GROUP BY
        target_name,
        host_name
ORDER BY 
        host_name, 
        target_name

The code is also available on github where you can find a larger collection of scripts. This scripting repository will be updated continuously so everyone is able to make use of the scripts.

Dual node SSH tunnel with putty

When connecting to a remote Linux server over SSH you have the option to create a tunnel from the remote server back to your local workstation. This can be very handy in case you, for example, need to map the port from the remote server to a localhost port on your workstation. For example, if the only allowed connection is SSH to the server and port 1521 is running on the server for the database you will not be able to remotely connect to port 1521 directly. You can use a tunnel over port 22 (ssh) and create a tunnel within this session to your local machine so you can connect to localhost:1521 and communicate (via the SSH tunnel) with the database.

The above use is quite straight forward, when using a Linux workstation creating a tunnel is quite straightforward, when using Windows with putty this is also done quite easy with creating a tunnel profile in putty. It gets more interesting when you have the below shown configuration.

In this situation you have a windows laptop which is only able to connect to the “jump server” via SSH. However, when you like to make use of Oracle SQL developer and connect to the database on the database server you will not be able to connect directly on port 1521 or create a direct tunnel between your workstation and port 1521 via a tunnel.

You will need to create a tunnel between your workstation to the “jump server” and from the “jump server” to the database server. This is in essence a double hop tunnel. To arrange this take the following steps:

• Configure on your windows workstation a putty tunnel where the source will be 45678 and the destination is localhost:45678  (see screenshot below)
• Connect with this configuration from your workstation to the “jump server”.
• Execute the following command while on the “jump server” shell: ssh -L 45678:database-server root@database-server
• While on your workstation connect Oracle SQL Developer to localhost: 45678

This should enable you to use Oracle SQL Developer locally by making use of a dual hop SSH tunnel to the database server via the “jump server”.

Exadata check IB cables

One of the things that helps make the Exadata perform at the speed it is performing is the fact that the connections between the compute nodes and the storage nodes is based upon Infiniband . In some cases also other, external components, are connected to the Exadata by making use of Infiniband . Infiniband is an intergrated and vital part of Exadata. The below presentation gives a quick introduction into the Infiniband cabling of a full rack Exadata and how you can connect other Oracle Engineered systems to a Exadata Infiniband  fabric.

Oracle infiniband cabling for Oracle appliances from Johan Louwers

In normal situations all cables should be present in an Exadata. However, in some cases due to some reason a cable might have been unplugged. As datacenters are often not near the location where engineers are it can be handy to have the ability to check the state of the cables from the commandline without the need to be physically present in the datacenter. The below bash script enables you to check the state of the cables on both the compute as well as the storage servers.

for ib_cable in `ls /sys/class/net | grep ^ib`; do 
  printf "$ib_cable: "; cat /sys/class/net/$ib_cable/carrier; 
done 

The output will tell you if the cable is present per Infiniband interface. A 1 indicates that a cable is found a 0 indicates that no cable is found.

Oracle EM12C querie virtual machines

Oracle Enterprise Manager, partially in combination with oracle VM manager is able to monitor and manage your Oracle VM landscape and the virtual machines that are deployed on this. One of the advantages of Oracle Enterprise Manager is that all the information associated with known targets is that it is stored in a database. This means that with some simple SQL statements you are able to query information, in the below sample code we do a simple query on the Oracle Enterprise Manager database to get information about the virtual machines we have deployed on Oracle Enterprise Manager in combination with the location where they are in the cluster.

This query can be very handy in case you need to make a quick impact analysis and are in need to know in which datacenter, in which pool, on which physical server specific virtual machines are deployed.

SELECT 
   v_ovm_vm.ovm_display_name         AS VM_NAME,
   v_ovm_vm.kernel_ver               AS VM_KERNEL,
   v_ovm_serverpool.ovm_display_name AS VMSERVER_POOL,
   v_ovm_zone.ovm_display_name       AS VMSERVER_ZONE,
   v_ovm_server.OVM_DISPLAY_NAME     AS VMSERVER_SERVER
FROM 
   MGMT$VT_VM_SW_CFG v_ovm_vm,
   MGMT$VT_VSP_CONFIG v_ovm_serverpool,
   MGMT$VT_ZONE_CONFIG v_ovm_zone,
   MGMT$VT_VS_SW_CFG v_ovm_server
WHERE 
   v_ovm_vm.vsp_uuid = v_ovm_serverpool.vsp_uuid
   AND v_ovm_serverpool.zone_uuid = v_ovm_zone.zone_uuid
   AND v_ovm_vm.VS_UUID = v_ovm_server.vs_uuid
   ORDER BY 3,4,5,1

The code is also available on github where you can find a larger collection of scripts. This scripting repository will be updated continuously so everyone is able to make use of the scripts.