Tuesday, November 26, 2013

Use cURL in combination with My Oracle Support

When interacting with Oracle support it is quite common that you would like to, or asked to, upload files to the oracle support site. This can be for example be a logfile or the output from a diagnostic script. Generally most people download the files to their workstations and then upload it the the Oracle support site. however, in some cases you would like to be able to directly upload the file from the server to the Oracle support website.

You can now use cURL to upload files to transport.oracle.com. cURL is commonly available on most Linux and UNIX systems. "cURL is a computer software project providing a library and command-line tool for transferring data using various protocols. The cURL project produces two products, libcurl and cURL. It was first released in 1997."

To upload a file to a specific service request you can use the below command to do so:

curl –T  –u “” ftps://transport.oracle.com/issue//

It is advised to not include the password in your curl command and you will be promoted for it. For more and detailed information you can refer to MOS document: 1547088.2

Create new target properties in Oracle Enterprise Manager

When using Oracle Enterprise Manager you have the option to store some information on the target in OEM as target properties. Up to recently I was under the assumption that this was a fixed list of target properties. One of the things I was hoping for was that this list would be expandable so you can add more target properties when needed for a business purpose. For example you might want to have a field to register your business owner, security officer, if a system should be compliant to PCI DSS and other things. 

Initially I was unable to find a way and Oracle support reported that this indeed was not an option. However, after Oracle development has reviewed the enhancement request I filed they came up with a solution that is already in place. It turns out you can add additional target property fields however not from the GUI, you can only add them by using emcli. 

The command to use to add additional target properties for a target in Oracle Enterprise Manager is to add_target_property. By using this command you can extend the number of fields per target type. 

For example, if you wanted to extend all targets of the type "oracle_database" with a new target property to store the name of the system owner you could do so be executing the below command;
emcli add_target_property -target_type="oracle_database" -property="Owner Name"

In case you wanted to add the owner property to ALL targets you could do so by executing the below command:
emcli add_target_property -target_type="*" -property="Owner"

having this option will enable you to extend the options you have to store information on a target level in Oracle Enterprise Manager. 

Monday, November 25, 2013

Big Data and cars 14.5 billion dollar in 2020

IHS Automotive has just released a report stating that the concept of connected cars will represent billions for the automotive industry. A raw estimate is that the automobile data systems will represent approximately 14.5 billion dollar in 2020. As a sidenote IHS researches note that this is only the amount that is directly related to the direct automotive industry however not included in this forecast is the revenue for companies and activities who have a spin off from this data. Inlcuding this it is estimated by IHS that this could run between $16 billion and $80 billion.

IHS expect that the following four categories will be of the most benefit directly for the automotive industry;

  • Diagnostics 
  • Location 
  • User Experience (UX)/Feature Usage
  • Advanced Driver Assistance System (ADAS)/Autonomy data

The below graph shows the buildup of data for the automotive industry between the year 2010 and 2020 and it shows a rapid growth that is predicted;

Even though this are impressive numbers it is in no comparison to what the actual revenue could be when we take into account the non-direct automotive revenue. This will not only bring new revenue to already existing companies however it will also fuel new companies to build new solutions and services. Interesting question to ask is however, who is the owner of the data collected. Most likely this will be the car manufacturer as this is somehow to be inserted within a contract with the owner. However, in an ideal world the owner of the car should be the owner of the data and should be able to opt for sharing this data, most likely anonymously, witth the rest of the world.

If the automotive industry could come up with a way that all car data could be shared openly accessible and free this could potentially even more fuel new innovations and could provide a hughe momentum for the adoption of systems like this.

Deploying Ceph and OpenStack with Juju

A lot of people working on building private or public clouds will be familiar with OpenStack amd will use it or at least have adopted some of the technological thinking behind OpenStack. Less know might be juju and CEPH.  Akash Chandrashekar is working as a solution engineer at Canonical which is the organization behind Ubuntu Linux and he gives a very clear explanation how your can build clouds by using OpenStack, juju and CEPH.

To give some highlevel background before watching the video on the 3 components; OpenStack, juju and CEPH.

OpenStack: OpenStack, a cloud-computing project, aims to provide an infrastructure as a service (IaaS). It is free and open-source software released under the terms of the Apache License. The project is managed by the OpenStack Foundation, a non-profit corporate entity established in September 2012 to promote OpenStack software and its community.

juju: Juju (formerly Ensemble) is a service orchestration management tool developed by Canonical Ltd.. It is an open-source project hosted on Launchpad released under the Affero General Public License (AGPL). Juju concentrates on the notion of service, abstracting the notion of machine or server, and defines relations between those services that are automatically updated when two linked services observe a notable modification. This allows for services to very easily be up and down scaled through the call of a single command. For example, a web service described as a Juju charm that has an established relation with a load balancer can be scaled horizontally with a single juju "add-unit" call, without having to worry about re-configuring the load-balancer to declare the new instances: the charm's event based relations will take care of that.

CEPH: Ceph is a free software storage platform designed to present object, block, and file storage from a single distributed computer cluster. Ceph's main goals are to be completely distributed without a single point of failure, scalable to the exabyte level, and freely-available. The data is replicated, making it fault tolerant.



In case you want to view the slides about the presentation deploying openstack and CEP with Juju at your own speed and comfort please find the slides below as they are shared on slideshare.




MYSQL using numbers in VARCHAR fields

Recently someone asked me to help out with some coding parts for a startup they where beginning. Some of the things they needed help with where infrastructure related and some where more in data management and some query modeling and optimization on an already existing datamodel done in a MySQL database.

This gave me some options to start exploring again some things in MySQL which is not my standard database to develop in and use as I am more focusing on Oracle databases. Part of the fun was trying to find out why some queries where not working as expected, main reason for this was that the person who originally designed the datamodel had a love for the VARCHAR datatype.

In the database we do have a table named ovsocial_inst which holds an ID column named inst_m_id. For some reason the original datamodel developer created the column as a VARCHAR even though it is only holding numbers. Now some funny effects do happen when you try to sort.

When you execute the following query:

SELECT 
      inst_m_id
FROM 
    ovsocial_inst
ORDER BY 
        inst_m_id

you will get a result something like:

inst_m_id
1
10
2
3
4
5
6
7
8
9
This is somehow a little strange as long as you do not realize that the values of inst_m_id are treated as text. When you consider it to be text everything makes sense and the query is doing exactly what you ask it to do. However, we do not want it to behave in this manner, we do want it to treat the numbers as numbers even though they are stored in a VARCHAR column. To do so in a sort we can use the following query which converts the VARCHAR into an unsigned.

SELECT 
      inst_m_id
FROM 
    ovsocial_inst
ORDER BY
        convert(inst_m_id,unsigned) ASC

Now you will get a result as you expect;

inst_m_id
1
2
3
4
5
6
7
8
9
10

Now imagine another scenario. we know that the table ovsocial_inst will be relative small so to assign a new ID to a record we would like to query the table for the max inst_m_id + 1 and we would like to use that value to insert a new record. When you do not consider the fact that the values are written to the table in a VARCHAR manner this process will work until you have 10 records in your table. Reason for this is that if you have 9 records in your table the highest value (the latest in the sort) is 9. This means that the new ID is 10 (9+1). When we hit 10 records something strange will happen. when we hit 10 records the highest value or at least the latest in the sort will be 9. This results in a new ID of 9+1 instead of 10+1.

When your ID field inst_m_id would have been a proper column for holding numbers you would use a query like the one below;

SELECT 
      (max(inst_m_id)+1) as new_id
FROM 
    ovsocial_inst

however, the above will give the issue as soon as you hit more then 10 records. To prevent this you will need to construct your query like the one below;

SELECT 
      max(convert(inst_m_id,unsigned))+1 as new_id
FROM 
    ovsocial_inst

In this way you will ensure that it keeps working even if you hit the 10 marker.

Sunday, November 10, 2013

Oracle engineered big data analytics solution

Big data is playing a major part in the future Oracle strategy for all good reasons and we do see that Oracle is adopting and connecting with big data solutions. Oracle is still one of the major players in the database world and for all good reasons they want to assure a place in the future data market space. To be able to do so they will have to adopt and integrate with big data solutions. Even though some people claim that big data is a hype it is holding some vert valuable points. For example the fact that data is growing in such rapid acceleration that (some) companies have to adopt different and new technologies to handle the amount of data that is available to them.

A major part of the Oracle strategy around big data is focussing on how to funnel big-data into subsets which can be loaded into a data warehouse so it can be analysed. As can be seen from the below example everything is to be funnelled into a data warehouse in this approach and then OBIEE or in-database analytics can be used to analyse this subset of the original acquired data.


Within this strategy a couple of options to implement this are available. For once you can use a full Oracle engineered systems approach as shown in the image below or you could go for a solution in which you design and create parts yourself. In the below image you see how the Oracle Big Data Appliance is used to acquire and organize the data that is collected from a number of sources. From the Oracle Big Data Appliance the pre-processed data is stored within databases inside an Oracle Exadata engineered system. Finally analysts will connect to the Oracle Exalytics machine to make use of their standard Oracle BI tools to query the subsets stored within the Oracle Exadata.


One of the advantages of the above outlined solution is that the systems are designed to work together and the time to create a working solution from a infrastructure perspective is short. Also the systems are designed to provide you with a optimised performance. Also the communication between the components is handled by infiniband, for more information on how to connect exadata and exalytics together with infiniband please do refer to my presentation on this subject on slideshare.

JSON validation

When working with API's and webservices JSON is becoming more and more the standard used for "packaging" data. When you are looking into how API's are sending back data to you it will become evident that JSON is winning over XML.

JavaScript Object Notation, is an open standard format that uses human-readable text to transmit data objects consisting of key:value pairs. It is used primarily to transmit data between a server and web application, as an alternative to XML.

Although originally derived from the JavaScript scripting language, JSON is a language-independent data format, and code for parsing and generating JSON data is readily available in a large variety of programming languages.

The JSON format was originally specified by Douglas Crockford, and is described in RFC 4627 and ECMA-404. The official Internet media type for JSON is application/json. The JSON filename extension is .json.

Issue is that when working with JSON it is very easy to have a small mistake somewhere in your JSON file which will break you entire code or makes your code unresponsive during parsing. If you for example use the json_decode function in PHP and your JSON file is having an issue it will simply return NULL as the value of the variable you used in conjunction with the json_decode function. Some debugging can be done using json_last_error() however a good thing to do first is to check if you JSON file is correct formatted.

Thanks to the guys from arc90 lab you can make use of the JSONLint. JSONLint is an online JSON validator which you can also use under the MIT license and which can be downloaded from github.

Friday, November 01, 2013

Drive visitors to your B2C website

When you are operating a business 2 consumer website, or any other form of website for that matter you do want to drive people to your website. When operating a B2C service you do tend to the more consumer based social networks to help you in this approach. In general the believe is that, when you like to spread the word of your service, you need to interact on facebook and twitter. Specially twitter is seen as a driving force and solution to interact with customers and it is generally thought that it would drive visitors to your website.

Below graph is created by Statista and shows what drives traffic to your website.



Not very much surprising is that facebook is a large contributor in driving traffic to your website. What is extreme interesting is that Pinterest is driving a lot of traffic to the publishers websites and that this is a lot more then what is coming from twitter. Issue with the above numbers is that they are not plotted on type of website. Twitter might for example be used much more in B2B and in tech. websites.

Reading the above numbers of how pinterest is driving traffic to websites it is good to understand what the demographic breakdown is of the pinterest user community.


The above demographic data from the pinterest community shows that 80% of the users is female and that the largest age group is 25 - 34. Not surprising is that 50% of the users do have children.

So, what is this meaning. Pinterest is most likely not on the top of your social networks you think you should interact on to drive more customers to your website. However, you should benchmark your target audience to the pinterest community. If you, for example, operate a B2C fashion website which focuses on female clothing you very much should focus on this. If you do want to focus on this there are a number of things you should keep in mind to make this a success.

Have a strategie;
Ensure you have a clear strategy of who you want to engage, how you would like to interact with users on Pinterest. Ensure that you have a clear step by step approach on what you will publish when and how you will ensure that users will start spreading your pictures.

Have a recognizable image style;
When people see an image it would be good if they recognize on the first glimpse that it is coming from your company. It might however not be the best practice to have your logo or you name in full over the image. A identifying style is better and making the images look great / look cool or look funny will make it that people are more willing to re-pin them on their own pinterest boards.

Make sure you can measure;
Having your images on pinterest is nice, however, at some point in time you do want to know if your images and your work is resulting in sales and what the effect is. Ensuring you have the correct techniques in place to capture this and to analyse it will be key of tune your strategy. With good reporting you will be able to find out what is working and what is not and how to change things to get more success.