Monday, December 07, 2015

IOT infrastructure with Oracle Linux and Mosquitto

MQTT stands for MQ Telemetry Transport. It is a publish/subscribe, extremely simple and lightweight messaging protocol, designed for constrained devices and low-bandwidth, high-latency or unreliable networks. The design principles are to minimize network bandwidth and device resource requirements whilst also attempting to ensure reliability and some degree of assurance of delivery. These principles also turn out to make the protocol ideal of the emerging “machine-to-machine” (M2M) or “Internet of Things” world of connected devices, and for mobile applications where bandwidth and battery power are at a premium.

We see that a large number of IOT and intelligent sensor manufacturers are making use of MQTT to send information from the edge of the network to a centralized location within an infrastructure. For example, sensors are sending out readings using MQTT to a centralized broker, a number of other services have subscribed to the broker to receive readings (payloads) for one or more topics. When building an IOT infrastructure based upon MQTT the role of the message broker is vital. Understanding of the MQTT protocol is also vital for making the correct decisions.

The above architecture blueprint shows on how you can position a MQTT broker in your landscape. The implementation is done based upon the Mosquitto an Open Source MQTT v3.1/v3.1.1 Broker which we deploy on an Oracle Linux 7.1 distribution. The decision for Oracle Linux is driven due to the fact that it is a stable enterprise grade Linux distribution and due to the fact that we do interact a lot with other products from Oracle in this example. Having a lot of Oracle products (which is not necessary) in this landscape gives you the ability to monitor and manage the entire landscape with Oracle Enterprise Manager. Oracle Enterprise manager can play the role as central monitoring solution over all components, for example the Oracle Big Data Appliance as well as the Oracle Exadata database machine and the other Oracle components. And in the same single monitoring solution the Oracle Linux Mosquitto server is integrated. This provides an easy way of maintaining all components without the need to develop custom scripting and lowers the overall TCO and improves the ROI.

When deploying a solution which is, or will/might become a important part of your business, it is advisable to ensure you select enterprise grade components. Selecting Oracle Linux in combination with Oracle Enterprise Manager for monitoring and possible other Oracle hardware and software components is a good practice.

Positioning Mosquitto
Mosquitto is an open source (BSD licensed) message broker that implements the MQ Telemetry Transport protocol versions 3.1 and 3.1.1. MQTT provides a lightweight method of carrying out messaging using a publish/subscribe model. This makes it suitable for "machine to machine" messaging such as with low power sensors or mobile devices such as phones, sensors, embedded computers or microcontrollers. In essence the components that will form a large part of the internet of things (IOT) are making use of the MQTT protocol and Mosquitto is a vital part as the broker.

Understanding the MQTT protocol
The MQTT protocol is based upon a publish and subscribe model. Within this model messages are published to a broker (in our case Mosquitto), subscribers receive messages with a topic they have subscribed to. A “topic” is an important and central piece within the MQTT protocol, a topic can be seen as the descriptor of a message payload.
For example, if your message payload contains the temperature coming from a sensor you want to ensure everybody knows what it is, where it is coming from, etc etc. As an example, if you have an office building where every room contains temperature sensor your “topic” could be: /building01/floor04/room25/tempsensor if you subscribe to this topic as a subscriber the broker will send you all the message payload for this topic.

However, unless this is your specific office space, it is not very useful, you might want for example, want to have all temperature readings of the fourth floor, in this case you can subscribe on the topic /building01/floor04/#/tempsensor by using a wildcard. Adding in every room another type of sensor, you can reuse the topic setup and replace for those sensors tempsensor with humsensors. This enables you to subscribe, for example, to all sensors on the fourth floor by using /building01/floor04/#/#

Understanding topics is vital and critical to be able to build architect a correct topic model which can be used in the optimal way and can be easily extend. If the topic architecture is created in a suboptimal way it will be an enourmous task (depending on the number of sensors) to correct this at a later stage. Taking the correct amount of time to develop a topic model is worth the investment.

The MQTT protocol has a number of message packets that are form the backbone of the MQTT protocol. Prime messages are:

  • Publish
    • Message payload send by a publisher to the broker
  • Subscribe
    • Message from a client (subscriber) to subscribe on a specific topic
  • Suback
    • Message from the broker as a response (conformation) back on a subscription request
  • Unsubscribe
    • Message from a client (subscriber) to unsubscribe on a specific topic
  • Unsuback
    • Message from the broker as a response (conformation) back on a unsubscribe request

Building a Oracle Linux & Mosquitto test server
For reasons of stability and due to the fact that Mosquitto will be used often in enterprise grade deployments we will use Oracle Linux as the Linux distribution of choice.  To be able to install Mosquitto on Oracle Linux (release 7.1) you can do a number of things. You can download the source code and compile it yourself or you can use yum to install everything you need. When you want to go down the easy path and install Mosquitto on Oracle Linux by making use of yum you have to realize that Oracle is not provding a RPM for it, meaning it will not be in the standard yum repository that is provided by Oracle. You can however use a specific centos repository. With the below command you will download the repository file and ensure it will be placed in /etc/yum.repos.d

wget -O /etc/yum.repos.d/mqtt.repo

If you now do a check on your repositories you should be able to see the mqtt (Mosquitto) repository and it should be active:

[root@localhost ~]# yum repolist all | grep mqtt
home_oojah_mqtt               mqtt (CentOS_CentOS-7)             enabled:      8
[root@localhost ~]#

If we now want to install Mosquitto we could directly use a yum install, however to be sure that the mqtt repository is available we can first do a quick yum list as shown below:

[root@localhost ~]# yum list mosquitto
Loaded plugins: langpacks
Available Packages
mosquitto.x86_64          1.4.4-2.1     home_oojah_mqtt
[root@localhost ~]#

As can be seen from the example we have a mosquitto.x86_64 version 1.4.4-2.1 available within the home_oojah_mqtt repository. Installing it can now be done with a quick yum install as shown below:

 [root@localhost ~]# yum install mosquitto
Loaded plugins: langpacks
Resolving Dependencies
  Running transaction check
  Package mosquitto.x86_64 0:1.4.4-2.1 will be installed
  Processing Dependency: uuid for package: mosquitto-1.4.4-2.1.x86_64
  Running transaction check
  Package uuid.x86_64 0:1.6.2-26.el7 will be installed
  Finished Dependency Resolution

Dependencies Resolved

 Package               Arch               Version                   Repository                   Size
 mosquitto             x86_64             1.4.4-2.1                 home_oojah_mqtt             102 k
Installing for dependencies:
 uuid                  x86_64             1.6.2-26.el7              ol7_latest                   54 k

Transaction Summary
Install  1 Package (+1 Dependent package)

Total download size: 156 k
Installed size: 346 k
Is this ok [y/d/N]: y
Downloading packages:
(1/2): uuid-1.6.2-26.el7.x86_64.rpm                                            |  54 kB  00:00:01
warning: /var/cache/yum/x86_64/7Server/home_oojah_mqtt/packages/mosquitto-1.4.4-2.1.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 49e1d0b1: NOKEY
Public key for mosquitto-1.4.4-2.1.x86_64.rpm is not installed
(2/2): mosquitto-1.4.4-2.1.x86_64.rpm                                          | 102 kB  00:00:06
Total                                                                  23 kB/s | 156 kB  00:00:06
Retrieving key from
Importing GPG key 0x49E1D0B1:
 Userid     : "home:oojah OBS Project "
 Fingerprint: bdf4 d371 5b8d c145 d583 46e9 f8c8 d6db 49e1 d0b1
 From       :
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : uuid-1.6.2-26.el7.x86_64                                                           1/2
  Installing : mosquitto-1.4.4-2.1.x86_64                                                         2/2
  Verifying  : mosquitto-1.4.4-2.1.x86_64                                                         1/2
  Verifying  : uuid-1.6.2-26.el7.x86_64                                                           2/2

  mosquitto.x86_64 0:1.4.4-2.1

Dependency Installed:
  uuid.x86_64 0:1.6.2-26.el7

[root@localhost ~]#

After installation of Mosquitto you can check if a service has been defined and check the status by executing a systemctl command, shown below:

[root@localhost ~]# sysctl status mosquitto
sysctl: cannot stat /proc/sys/status: No such file or directory
sysctl: cannot stat /proc/sys/mosquitto: No such file or directory
[root@localhost ~]# systemctl status mosquitto
mosquitto.service - LSB: Mosquitto MQTT broker
   Loaded: loaded (/etc/rc.d/init.d/mosquitto)
   Active: inactive (dead)
[root@localhost ~]#

As you can see, sysctl recognizes Mosquitto however Mosquitto has not been started yet. To start this you can execute systemctl start mosquitto and the next time you do a status lookup you will notice a lot more is running within Oracle Linux:

[root@localhost ~]# systemctl status mosquitto
mosquitto.service - LSB: Mosquitto MQTT broker
   Loaded: loaded (/etc/rc.d/init.d/mosquitto)
   Active: active (running) since Tue 2015-11-03 14:38:45 EST; 5s ago
  Process: 2099 ExecStart=/etc/rc.d/init.d/mosquitto start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/mosquitto.service
           ΓΆΓΆ2101 /usr/sbin/mosquitto -d -c /etc/mosquitto/mosquitto.conf

Nov 03 14:38:45 localhost.localdomain systemd[1]: Starting LSB: Mosquitto MQTT broker...
Nov 03 14:38:45 localhost.localdomain mosquitto[2099]: [121B blob data]
Nov 03 14:38:45 localhost.localdomain mosquitto[2099]: 1446579525: Config loaded from /etc/mosqui...f.
Nov 03 14:38:45 localhost.localdomain mosquitto[2099]: 1446579525: Opening ipv4 listen socket on ...3.
Nov 03 14:38:45 localhost.localdomain mosquitto[2099]: 1446579525: Opening ipv6 listen socket on ...3.
Nov 03 14:38:45 localhost.localdomain systemd[1]: Started LSB: Mosquitto MQTT broker.
Hint: Some lines were ellipsized, use -l to show in full.
[root@localhost ~]#

In esscence you have a running Mosquitto service on your Oracle Linux server. The downside of this installation is that you do not “really” have the tools to test what is working and to play around with your fresh installation. If you want to test Mosquitto you will also have to install the mosquitto-clients tools by executing a yum install mosquitto-clients. As an example you can now test to the Mosquitto service like shown below where we subscribe to the topic test/mqtt

[root@localhost ~]# mosquitto_sub -d -t test/mqtt
Client mosqsub/2239-localhost. sending CONNECT
Client mosqsub/2239-localhost. received CONNACK
Client mosqsub/2239-localhost. sending SUBSCRIBE (Mid: 1, Topic: test/mqtt, QoS: 0)
Client mosqsub/2239-localhost. received SUBACK

If you want to see the full working of the pub/sub mechanism you can keep the above statement running in a terminal and open a second terminal. The first (above shown) command will act as the subscriber to the topic test/mqtt. In the second we will be pubishing information to Mosquitto and if everything works as expected the subscriber should receive from Mosquitto what the publisher sends to Mosquitto. An example of sending a test message as a publisher on the topic test/mqtt with the payload “test123”

[root@localhost ~]# mosquitto_pub -d -t test/mqtt -m "test123"
Client mosqpub/2411-localhost. sending CONNECT
Client mosqpub/2411-localhost. received CONNACK
Client mosqpub/2411-localhost. sending PUBLISH (d0, q0, r0, m1, 'test/mqtt', ... (7 bytes))
Client mosqpub/2411-localhost. sending DISCONNECT
[root@localhost ~]#
As can be seen from the subscriber side we now receive the message:
[root@localhost ~]# mosquitto_sub -d -t test/mqtt
Client mosqsub/2384-localhost. sending CONNECT
Client mosqsub/2384-localhost. received CONNACK
Client mosqsub/2384-localhost. sending SUBSCRIBE (Mid: 1, Topic: test/mqtt, QoS: 0)
Client mosqsub/2384-localhost. received SUBACK
Subscribed (mid: 1): 0
Client mosqsub/2384-localhost. received PUBLISH (d0, q0, r0, m0, 'test/mqtt', ... (7 bytes))

The above examples are all working on localhost by default, if you want to publish (or subscribe) to a remote machine you can use the –h option to specify the host. Make sure you have port 1883 (the default port of mqtt) is open and not blocked. As an example of a remote subscription:

mosquitto_sub -h -d -t test/mqtt

No comments: