Push API solutions for open and linked data

The internet has been growing at a rapid speed which was most likely not foreseen by a lot of people in the beginning. When Tim Berners-Lee added HTML and HTTP to the internet mix and by doing so opened the possibilities for the world wide web as we know it today he most likely was also not foreseeing what this would become, not even in his most wildest dreams.

As we are now becoming more and more used to the daily influence of the internet and the possibilities that it is offering us we are also trying to find ways on how to improve to the current state of it. We have had the web2.0 revolution which came down to a more social and a more interactive internet where people could more easily add to the content instead of only consuming it as a reader of web-pages. Many see the next big paradigm shift for the internet in the Semantic Web. The idea of the Semantic Web includes a lot of components from open-data, linked data, micro-formats and the internet of things / the internet of everything till big-data concepts. It all comes down to how we think about data and how data will play a role in the next version of the web.

In the below video you see what the vision from Tim Brenners-Lee is on the next version of the internet which he posed during one of his talks at TED.


 What it comes down to is the desire to provide data in an open format. Not necessarily in a nice and flashy website or a closed website however rather in the pure and raw format. This in an open format like XML or any other usable format.

By unlocking this data there will be directly involved the unlocking the potential of a lot of new options. When people do have access to raw data you can create new applications and come to interesting insights. currently the most data is still in closed silo's and not accessible to the public or not accessible in an open way that you can use it in a way you can build new applications around it. When the world starts with adopting the open-data model and the linked data model there will be a sprawl of new options coming to the public.

Even though all the above mentioned things are coming our way and will change the way the internet is used and seen by the public one important architectural part is missing in my opinion, or rather, is not given the attention it deserves. The missing, or rather under appreciated part is the role that push interfaces and API's will play in this new revolution. If we look at many of the new solutions, models and formats they are all based, or primarily based, upon a pull architecture.

If you draw a high level architectural representation of the current state in which applications are commonly are developed you will get something similar like the drawing below.

The custom application could for example be a web application that tracks flight information from a major airport and shows this to visitors of the website. The website itself is represented as the "Presentation layer" in the above diagram. Every time someone is logging in to the website the code will query for the most recent flights in the local database. However, to ensure that the local database contains this information it needs to be filled with data that comes from the "data owner". What you commonly see is that there is a scheduled program to download the basic information from the "data owner" data API and it is stored in the local database. The moment a user clicks on a flight the web application code will do a "online" query to the "data owner" to get the more detailed and latest information about this specific flight. 

In essence there is nothing wrong with this model and it is serving its purpose. However there are more and better solutions to do this which makes the development much more easy and ensures that applications are becoming faster, less prone to issues and less resource intensive on both the "custom application" side as well as the "data owner" side. 

A more favorable way of doing this is by making use of a push API rather then a pull API. In the below diagram the same solution is shown however no making use of a push API initiated by the data owner. 

The implementation of this model makes a lot of sense as it is removing a lot of unneeded communication between the customer application(s) and the data owner. Also missing an update is less likely to happen and new information, in our example flight information, is in real-time available to all who custom applications that have subscribed to the push interface from the data owner. 

In the above implementation, as soon as a new flight is known at the data owner it will send out a message to all the subscribed applications. By doing so the data owner knows that all the other applications are now aware of the new flight and might, if needed, query additional information via the "direct remote data access" method to the more traditional data access API. In some cases the real-time push API will remove the need for a more traditional API however in most cases there will still be a need for this. 

The hurdle for implementing this model is the need to develop standardization for communication between the data owner and the custom applications. Ideally the communication between both parties is arranged based upon the HTTP protocol and JSON or XML messages are exchanged between the 2 parties as shown in the image below. 

From a technical point of view it is not hard to implement the communication from both the "data owner" side for a push data API nor is it for the receiving side complicated to create a data update receiver who will receive and process the data send by the data owner. 

The biggest hurdle in this process is to ensure that the construction of the message is correct and contains all the needed information that the receiver of the receivers might need and that this is done in a functional format that is not only for a single receiving application usable but for a large set of external applications. If we take for example a look at the work done in the field of ontology for annotating offerings and other aspects of e-commerce on the Web by GoodRelations or the work done by schema.org you will see that developing a good standard is something that is not to be thought of lightly. However the below illustration might show the need for such standardization of message formats on both a more functional level and a technical level.

Commonly a custom application, for example a web portal showing flight information, is not making use of a single source. If, for example, you like to build a web portal showing all the incoming and outgoing flights in real-time or near real-time for the city of London you have to work with 11 different data providers. London is having ten "normal" airports, Luton, Stansted, Northolt, Stapleford, Heathrow, London city, Biggin Hill, Blackbushe, Farnborough and Gatwick. They also have a helicopter only airport named Battersea.

From a technical point of view it would be a great benefit if all those airports do make use of the same way of providing a push API interface to all push the information in the same manner. This way the developers of the custom applications that will connect to it will only have to develop a single data update receiver and simply subscribe to all the 11 airport push interfaces instead of developing 11 different receivers. Next to this the data providers, in this case the airports, can make use of an already developed standard data format and can potentially decide that they will develop a push interface together so they can split costs.

From a functional point of view a standard makes sense because you will know exactly what data you will get and which format. From a functional point of view this is not the technical format but rather the functional format. For example, if the data owner is also sending out weather information per flight, it would be good if all data owners do this in the same manner and all the information is in the same functional format. Meaning, temperature is for all data owners in Celsius or it is for all data owners, who comply to the standard, in Fahrenheit. This way you always know from a functional point of view what the data means and how to interpret this.

In conclusion, push API implementations will fuel the next generation of the internet in my mind and they will ensure that the web becomes (even) more real-time and data rich. The flow of data will help sprawl new initiatives and will help create new companies and great projects. For this however there will be a need to ensure that standards will be developed, not only on a technical implementation level, however even more important on a functional level where the standards will define which information in which format will be provided to the data consumers. 

Indoor navigation with the Apple iBeacon framework

Already for a long time companies are trying to tackle the issue of indoor navigation. Already in 2011 I wrote a short post about the upcoming market of indoor navigation and what the people at Nokia research lab where doing is this area. Unfortunately, even though a lot of people do see a market potential in this, there is not much to report in this field and not many really adopted products or protocols have been developed by the industry. Now a new attempt to tackle the issue of indoor navigation is coming into play.

with the release of Apple IOS 7 and the software development kit Apple has introduced the option to the development community to make use of BLE or Bluetooth Low Energy.

"The iBeacon works on Low Energy Bluetooth (BLE), also known as Bluetooth 4.0 or Bluetooth Smart. In a real life scenario it would be more of a location-aware, context-aware, pervasive small wireless sensor beacon that could pinpoint your location in a store: iBeacons could send you notifications of items around you that are on sale or items you may be looking for, and it could enable payments at the point of sale (POS) where you don’t need to remove your wallet or card to make a payment. It could be a possible Near Field Communication (NFC) competitor."

Even though this technology is thought to be a pure Apple technology by some people this is a wrong assumption. iBeacons can be pruchased from a number of companies who develop them and all comply to the same standards. iBeacon is nothing more then a framework where some hardware vendors are developing the hardware beacons for. The most known developer of beacons is estimote however beacons are developed by for for example roximity.

An explanatory video from Estimote can be seen here;

The options of what can be done with the iBeacon solutions are numerous. In the above video some examples are mentioned. For example personalized prizing of items in a store which might differ from person to person based upon their membership of the stores loyalty program. Also an option is to show the number of likes for a product and product reviews.

However a large number of other options and solutions that can be created are possible, think for example indoor navigation on an airfield where someone is directed to the lounge or gate where this person should be. Also a personal plan for someone visiting an exhibition, some exhibitions can be extremely large and navigating the floors and be able to see all the vendors and conference speeches can be extremely difficult.

If you take for example the annual Oracle OpenWorld conference in the Moscone center in San Fransisco. It consists out of a number of exhibition floors with a large number of vendors and also it is a conference with a extreme large number of speeches. Getting from conference room to conference room and not missing anything can be hard. Currently visitors can select a track and they are guided and helped by a program. Adding to this program an APP that will help you get to the correct room on time and help you, for example, keep track of where all your colleagues are so you can meet up for coffee between talks would be a great addition. Oracle OpenWorld is just an example and a indoor navigation framework could be offered by the conference center owners as a service to the conference that is using their location.

When machine learning goes wrong

Human emotions and understanding human emotions in text and speech is one of the most challenging fields in computer science at the moment. The ability to understand what a human means by a certain text or a certain sentence is challenging. Humans are able to pick up the subtle things like sarcasm or humor while developing an algorithm to understand this has proven to be a challenge. Being able to do so, having a system that understands this will provide the option to have a much more human like interaction with technology and will provide options to have machines make decisions based upon feelings of real people.

One other part that is difficult is slang, street language, the language used by certain groups of people and the meaning that it has. This was already adressed for a bit in the video from Michael Karasick, VP and Director at IBM research, in my post big data sentiment analysis.

Now IBM has tried to have Watson understand slang language. Watson is an artificial intelligence computer system capable of answering questions posed in natural language, developed in IBM's DeepQA project by a research team led by principal investigator David Ferrucci. Watson was named after IBM's first president, Thomas J. Watson. The machine was specifically developed to answer questions on the quiz show Jeopardy!. In 2011, Watson competed on Jeopardy against former winners Brad Rutter, and Ken Jennings. Watson received the first prize of $1 million.



"In order to boost Watson’s aptitude with everyday lingo, its software engineers began teaching it the Urban Dictionary, that massive online depository of current slang. There was just one small problem: Watson couldn’t differentiate “clean” terms from profanity. It’s one thing when a supercomputer uses “OMG,” but quite another when it starts cursing like a Quentin Tarantino character—hopefully not in the middle of a “Jeopardy” episode, although that could prove memorable for everyone involved." From slashdot

This shows that understanding slang and everyday lingo is difficult however even more difficult is having a algorithm to understand when it is appropriate to use it and when not. This research field will be very interesting in the upcoming years and will prove to be challenging however, when mastered it can be a big addition to how we communicate with technology. 

Commercial drone usage in 2015

Disappointed about the future, disappointed that we do not have the type of cities that we where envisioning back in the 80s? Wonderin why we do not have swarms of droids flying over our cities at this moment running all kind of errands? Well, one of the things that is holding this drone thing back is the regulation by the US FAA (Federal Aviation Administration). However that might change very rapidly, the FAA announced that it will be allowing unmanned aircraft systems starting in 2015. You can read all about this announcement in this FAA news item.

It is already rumored that companies like FedEx are looking into drone systems which will deliver packages to certain locations. That means that in the future you might see a FedEx quadrocopter flying to your house with a package for you instead of a delivery truck pulling up to your front door. Also one of the things FedEx is looking into is transport flights between main transportation hubs by UAV's, you can read some more about this on the diydrones website. Below is a impression Boeing / cranfield concept UAV.

Next to transportation a new field is opening up for security drones. Secom is developing small security drones to patrol your property and make sure you have a view everywhere and a constant patrol. Having drones (on wheels or flying) to safeguard remote sites is something we probably will see more and more in the future.  You can rent a drone from Secom for $58 per month so that means it will become available to a large number of people and companies which will help the adoption a lot.

Having the new rules of the FAA in place by 2015 will possibly take us a step close to the city of the future where we see lots and lots of drones taking away some of the work of humans. The question will remain if this is a positive or a negative things for humanity and employment rates.

Mitchell Joachim's Eco-City Nick Kaloterakis

Social media analysis

IBM is running a project called Smarter Planet in which they combine a lot of the new and upcoming technologies and ways of thinking and put them to use to craft a view of how to build a better world and a smarter planet. As the tech company IBM is they do touch the subject of social media and big data. In the minds of a lot of companies social media and the capability of harvesting information from social media will be a valuable asset for the future. A future where big data is analyses on the fly and where we can understand what humans are saying and meaning, where we can interpret the feelings and emotions and have technology interact on this.

Secondly we will have a more instrumented, interconnected and intelligent way of doing things and analyze things where we can combine human expressions like a tweet or blogpost in combination with traffic and weather. All this combined will give is a good view of the state of a person or a group of persons.

One of the things that will play a big role in this big data future is social media analysis, where are we in the social media world, what are our connections, who are our opinion providers and who are our opinion consumers. Having the option to harvest and give meaning to a enormous amount of data will open up a whole new way of doing traditional business and will provide new business opportunities. New companies and ways of doing commerce will see the light of day.



In the above presentation, Graham Mackintosh, IBM social media analytics expert explains why social media analytics is so important and how it can be used to derive insights from the growing level of data that we generate through social media.

Human emotion algorithm for social media

A couple of post ago I toughed the subject of big data sentiment analysis and how the field of sentiment analysis will be the next frontier in data analysis. Sentiment analysis will help developers build applications and analysis tools to improve the machine to human interaction and make technology play a more integrated and more natural way of helping humanity.

Feelings and computer algorithms are a field in which we do not have that much experience at this moment and a lot of people do feel that we will not be able to concur all the fields of emotions. Some people feel that we should not try to concur all the fields of computerized emotions and digitized emotions. In the below video Intel futurist Brian David Johnson talks with Dr. Hava Tirosh-Samuelson, Director of Jewish Studies at ASU. Dr. Hava talks about transhumanism, our future and the fact that to her opinion not all emotions can be captured or should be captured.


According to some it will be possible to capture the human emotion and extract the human emotion from a sentence or a work of art however it will be impossible to give a machine emotions. Giving a machine emotion will most likely be one of the most hardest parts of computer science in the upcoming time however first we will have to be able to capture and extract human emotions. This will provide us a much better way to interact with humans from a machine point of view. Some projects are working on this, some are working on collecting emotions and some try to make meaning out of the harvested emotions. One of the graphically most appealing emotion harvesting projects is done by Jonathan Harris an Sep Kamvar at wefeelfine.org.The wefeelfine.org project scans the internet continualsiy to find feelings based upon blogposts containing a sentence "I feel". On the project page you can see how the world is feeling at this moment.

 Below you can see a talk done by Jonathan Harris at TED.com.



As already stated capturing human emotions from social media en machine interaction and giving meaning to it will be one of the next big things a lot of companies will try to resolve. Having a algorithm to understand human emotions can be implemented in a lot of applications and will provide a whole new era of how we work with and interact with technology. The challenge in this is that this cannot be done by computer researchers alone, a lot of people and a lot of fields of science will have to interact to only begin to imagine the complexity of crafting such a algorithm.