Oracle CX - act upon a negative customer experience

Customer experience is becoming more and more important both in B2B as well as in B2C. When a customer is having a bad experience with your brand and/or the service you are providing the changes that this customer will purchase something or will be a returning customer in the future is becoming very unlikely. Additionally, if the customer is having a good customer experience however is lacking the emotional binding with your brand or product the changes that he will become a brand advocate is becoming unlikely.

The challenge companies are facing is that it becomes more and more important to ensure the entire customer experience and the customer journey is perfect and at the same time an emotional binding is created. As a large part of the customer journey is being moved to the digital world a good digital customer journey is becoming vital for ensuring success for your brand.

As more and more companies invest heavily in the digital customer journey it is, by far, not enough to ensure you have a good working and attractive looking online shopping experience.  To succeed one will have to go a step further than the competition and ensure that even a negative experience can be turned into a positive experience.

The negative experience
The below diagram from koobr.com showcases a customer journey which has a negative experience in it. Is shows that the customer wanted to purchase an item and found that the item was not in stock.

This provides two challenges; the first challenge is how to ensure the customer will not purchase the required items somewhere else and the second challenge is how to ensure we turn a negative experience into a positive experience.

Turning things positive
In the example for koobr.com a number of actions are taken on the item not in stock issue.

• Company sends offer for their website
• Company emails when item is in stock
• Company tweets when item is in stock

This all takes the assumption that we know who the customer is or that we can get the user to reveal who he is. In case we do not know the customer, we can display a message stating that the customer can register for an alert when the items is in stock and as soon as the item is in stock a discount will be given. The promise for a discount on the item in the future also helps to make sure the customer will not purchase somewhere else.

Making a connection
The way you can contact the customer when the item is back in stock depends on the fact if we know who the customer is and which contact details we have from this customer. If we assume that we know who this customer is we can provide a discount specific for this customer only or provide another benefit.

The default way of connecting with a customer in a one on one manner is sending out an email to the mail address we have registered for this customer. A lot of other methods are however available and depending on the geographical location and demographic parameters better options can be selected.

As an example;

• A teenage girl might be more triggered if we send her a private message via Facebook messenger.
• A young adult male in Europe might be more triggered if we send a private message via WhatsApp.
• A young adult female in Asia might be more triggered if we use WeChat
• A Canadian male might want to receive an email as a trigger to be informed about an item that is back in stock
• A senior citizen might be more attracted if a phone call is made to inform him that the item is back in stock. 

Only depending on email and a generic tweet on twitter will provide some conversion however much less conversion than might be achieved when taking into account more demographic parameters and multiple channels.

Keep learning
One of the most important parts of a strategy as outlined above is that you ensure your company keeps learning and ensures that every action as well as the resulting reaction are captured. In this case, no reaction is also an action. Combining constant monitoring of every action and reaction and a growing profile of your individual customer as well as the entire customer base provides the dataset upon which you can define the best action to counteract a negative experience as well as ensuring a growing emotional bonding between your customer and your brand.

Integrate everything
When building a strategy like this it needs to be supported by a technology stack. The biggest mistake a company can make is building a solution for this strategy in isolation and have a new data silo. Customers are not interested in which department handles which part of the customer journey, the outside view is that of the company as one entity and not as a collection of departments.

Ensuring that your marketing department, sales department, aftercare department, web-care department and even your logistical department and financial department make use of a single set of data and add new information to this dataset is crucial.

To ensure this the strategy needs to make use of an integrated solution, an example of such an integrated solution is the Oracle Cloud stack where for example the Oracle Customer experience social cloud solution is fully integrated with Oracle marketing, services, sales and commerce.

Even though this might be the ideal situation and provides a very good solution for a greenfield implementation a lot of companies will not start in a greenfield, they will adopt a strategy like this in an already existing ecosystem of different applications and data stores.

This means that breaking down data silos within your existing ecosystem and ensuring that they provide a unified view of your customer and all actions directly and indirectly related to the customer experience is vital.

In conclusion
Creating a good customer experience for your customers and building an emotional relationship between customer and brand is vital. Nurturing this is very much depending on the demographical parameters for each individual customer and a good customer experience as well as building a relationship requires having all data available and capturing every event.

Adopting a winning strategy will involve more than selecting a tool, it will require identifying all available data, all data that can potentially be captured and ensuring it is generally available to select the best possible action.

Implementing a full end to end strategy will be a company wide effort and will involve all business departments as well as the IT department. 

Oracle DEV – Automated testing with Selenium

When developing a solution, and more specific a web application in this example, part of your CI/CD process should be automated testing. Ensuring automated testing will save a lot of time and money and will support a fail fast principle where developers are made aware of issues in the application in a very early stage of the development process.

Part of a fail fast strategy is developing in a business driven development or test driven development manner and ensure that developers will develop automated tests to validate every part of the application. The growing set of tests can be executed every time the CI/CD automation triggers the automated testing.

Selenium for testing
One of the test automation tools commonly used is Selenium. Selenium is capable of running a web browser and interact as a user would interact with the system while checking every assertion defined in the test code.

Selenium example with XML output
The below example showcases a very small testcase which will execute two tests and which has been tested using a Oracle Linux instance to run the Selenium code. The difference between a standard selenium test and the below example is that it ensures that the test report is generated as XML and is stored in a default location.

Storing the test reports in XML will enable you to combine and report multiple testcases into a single test report while building individual tests instead of one large testcase. Selenium supports multiple development languages to define your testcases, the below example is a Python based testcase.

import unittest
import xmlrunner
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

# set some generic variables used within the wider test scripting
geckodriver = '/usr/local/lib/selenium/drivers/geckodriver'
options = webdriver.FirefoxOptions()
options.add_argument('-headless')

class PythonOrgSearch(unittest.TestCase):

    def setUp(self):
        self.driver = webdriver.Firefox(executable_path=geckodriver, firefox_options=options)

    def test_search_in_python_org(self):
        driver = self.driver
        driver.get("http://www.python.org")
        self.assertIn("Python", driver.title)
        elem = driver.find_element_by_name("q")
        elem.send_keys("pycon")
        elem.send_keys(Keys.RETURN)
        assert "No results found." in driver.page_source

    def testCaseFindTitle(self):
        driver = self.driver
        driver.get("http://www.python.org")
        self.assertIn("Python", driver.title)
        elem = driver.find_element_by_name("q")
        elem.send_keys("pycon")
        elem.send_keys(Keys.RETURN)
        assert "No results found." not in driver.page_source

    def tearDown(self):
        self.driver.close()

if __name__ == "__main__":
    unittest.main(
        testRunner=xmlrunner.XMLTestRunner(output='test-reports'),
        failfast=False, buffer=False, catchbreak=False)

Executing the test
If you execute the above test you will notice that one case will fail and one will succeed (at this very moment). A standard execution looks like the example below:

[root@testnode12]# python 7seltest.py 

Running tests...
----------------------------------------------------------------------
.F
======================================================================
ERROR [15.376s]: test_search_in_python_org (__main__.PythonOrgSearch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "7seltest.py", line 23, in test_search_in_python_org
    assert "No results found." in driver.page_source
AssertionError

----------------------------------------------------------------------
Ran 2 tests in 27.462s

FAILED (errors=1)

Generating XML reports...
[root@testnode12]# 

As an addition you will find the XML report in the ./test-reports directory for future references and to enable you to parse the individual XML reports into a single report. 

Oracle Cloud – add new nodes to your loadbalancer automatically

Services that requiring you to balance requests over multiple backend services are very common. Moving away from monolith based applications and building more smaller components increases the need for good load balancing. Adding to this that within a cloud environment the number of instances can scale up and down whenever required makes it a requirement for a loadbalancer to quickly adopt to the scaling up and scaling down of nodes.

When you design your solution right the configuration of your loadbalancer should automatically adopt changes in your landscape. As an example, if a new instance of a service is created it should automatically result in the fact that this node is added to the loadbalancer configuration. Additionally, when removing an instance this should result in the fact that this instance is removed from the loadbalancer in a graceful manner.

Oracle Cloud Load balancers
As part of the cloud offerings Oracle provides a load balancing service. When deploying your cloud based applications you can leverage the Oracle Cloud load balancing service and ensure that it spans multiple availability domains.

The below diagram showcases a simple deployment with a high available load balancer service available in two availability domains which will balance the load over six machines who are spread over three availability domains

The above diagram showcases a simple implementation of the Oracle load balancer service in the Oracle cloud. You can use this blueprint as a starting point for building your more complex and sophisticated deployments for enterprise deployments in the Oracle cloud. 

Design for automation

The intention of your designs and architecture should be to support full automation of the loadbalancing process. The processes of adding and removing an instance of your backend services should never result in manual actions which need to be performed by a human. 

Taking this into account when you design your solution will give you a good starting point to ensure your solution is elastic and capable of reacting to changes in the landscape without any additional effort. 

When building the logic for your bootstrapping to ensure automated registration and de-registration of your backend nodes you can leverage the APIs from the Oracle Load Balancer. The two main choices you have are using either the REST APIs or the SDK to call out to the loadbalancer from your bootstrap logic. 

The REST APIs provide you an easy way to work with the loadbalancer in a programmatic manner, this will enable you to call the API endpoints with any programming language you like. As an example, you could use bash scripting under Oracle Linux for your bootstrap process and call out to the API endpoints to register and de-register your nodes. More information on the APIs can be found at this page. 

The SDK(s) for Oracle Cloud infrastructure, including the Oracle Cloud loadbalancer are provided in a number of programming languages such as Java, Go, Python and Ruby. As an example you can read the documentation of the Python SDK used to work with the Oracle Cloud Loadbalancer at this page. 

Bootstrap your services

A big part of ensuring that your solution is capable of automatically register and de-register instances at the Oracle Cloud Load Balancer is ensuring the bootstrapping of your instance is done right. This is especially of importance when you need to balance load over custom components that run on a virtual environment in the Oracle Compute Cloud. 

The importance in this is to ensure your services, virtual machine images, have a good bootstrapping. The bootstrapping should ensure both the registration and de-registration of an instance with the loadbalancer. 

The importance in registration and de-registration is that you take the service into account and not the running operating system. This means that part of your bootstrap will require checking if the service itself is up and running, only after the service is available you can register the service at the loadbalancer. Having the operating system up and running and not the service will result in requests being routed to the new node while it is not able to serve requests yet.

Design concept positioning

The concept of adding and removing instances automatically to your load balancer can be named as “Automatic load balancing registration” and could form a part of your enterprise architecture blueprint library. Including the concept in your enterprise architecture library and ensuring all your deployments are done in this manner will ensure a mature and unified way of working across all your solutions. 

Oracle Development – Proof your bug first

When building applications, or more specific, when resolving bugs in your application the initial reaction of a lot of developer is to resolve the bug first thing. Another approach is to first proof your bug is indeed a tangible bug by writing a test script to proof it. Even though you might be natural inclined to quickly resolve the bug a lot of value can be found in proving it first.

By applying a test-driven development approach the first thing you will do is write a short test script that will inform you that the bug is present. After that you can write the fix for the bug and re-run the test script. Now, with the bug resolved, the test script should inform you that everything is working as expected.

The value in in this approach is that, with every bug you resolve, you will add to the set of tests for your application. When done right you should have an automated test step in your CI/CD pipeline. This means that every time you build, test and deploy a new release of your application the application will be tested for every bug ever found in your application and make sure that it has not occurred again.

Test new functions
Implementing an automated testing strategy in combination with test driven development is not only applicable when resolving bugs, it should also be part of your new functionality development cycle. Every time you develop a new functionality it should be accompanied with a test script.

The test stage in your pipeline
As stated, when you have implemented your CI/CD strategy correctly and made sure that you have a test step in your pipeline every execution of your pipeline should result in all tests being executed automatically.

When the execution of tests is a manual process by developers or even worse a process that needs to be requested with a specific test team the changes are high testing is happening less frequently than it should be.

By ensuring that testing is an automated part and your CI/CD pipeline enforces the execution fo all tests makes sure that you know that your application is tested over and over again to make sure all functionality is available and that bugs that have been resolved in the past do not surface again.

Testing your web application with Oracle Linux
If you develop a web application the use of tools like Selenium can be extremely valuable.  Selenium allows you to run tests fully automatically and has support for multiple browsers.

When writing your selenium test it is good to know that Selenium is fully working on Oracle Linux and that you can use this in a headless mode. The benefit of the headless mode is that you can run virtual Oracle Linux machines and have them execute the Selenium tests without the need to have a graphical user interface running within your Oracle Linux instance.

The below code snippet shows a test case for Selenium written in Python capable of running with FireFox on an Oracle Linux instance without the need to have a graphical user interface.

options = webdriver.FirefoxOptions()
options.add_argument('-headless')

One of the models to run your Selenium test quickly is to use an Oracle Linux based Docker container and have the test running within the context of this container. There will be no need for any graphical user interface and due to the way the Oracle Linux base image for Docker containers is provided the size of your Docker image will be minimal.

Oracle Linux based container testing
As part of your strategy to run your Selenium tests in a headless Oracle Linux Docker container you could consider the following;

Develop a simple Selenium image based upon the Oracle Linux Docker base image. The simple Selenium image should be able to download Selenium tests from a central repository and execute them. This means that you can keep your Selenium image relative simple and you will not have a need to rebuild the image every time you add a test to your test set.

Every time your pipeline will invoke the testing stage you Oracle Linux based container will download the latest version of a test as part of the bootstrap process.

One thing to remember is that you need to ensure that your container will report the test results to a central location to ensure you will have all the test results after the container has stopped.

Why innovators need to eat frogs

Innovation is a long process of tackling very complex problems and often problems nobody has been trying to tackle before. Throughout my career I have been working at a number of hard innovation projects, building IOT like solutions way before IOT was a known phrase and developing cryptographic solutions to allow secure communication with chips on government ID cards. In all those cases I followed the mantra of Google X without knowing it, even before Google was a real company and Google X existed.

The mantra at Google X, the moonshot department of Google where people tackle really hard problems is #MonkyFirst. The idea behind this is that if you want a monkey to recite Shakespeare on a pedestal you do not start with building the pedestal. Everyone can build a pedestal; a lot of people have been building a pedestal before Training the monkey is the hard part. If you are unable to train the monkey to recite Shakespeare there is no need to build the pedestal at all.

In short, try to tackle the hard problem first before you spend time and money on tackling the parts of your project that you know will not be that hard.

The mantra #MonkeyFirst is also stated by Mark Twain in a bit of a different form while having the same meaning. Mark Twain wrote; “eat a live frog first thing in the morning and nothing worse will happen to you for the rest of the day”.

Companies start to eat frogs
In companies around the world the common thing to do is to build the pedestal first and after that try to train the monkey. One of the reasons for it is that people tend to desire quick satisfaction and within companies there is commonly a tendency that management want to see tangible results fast. Building the pedestal is something that can quickly be done and it will show results towards management. Showing what you have accomplished is a more convenient message to tell than providing a long list on why it is so hard to train a monkey.

However, if you are unable to train the monkey there is no reason to build a pedestal. Eating the frog or #MonkeyFirst ties a bit into the “fail fast, fail early, fail cheap" concept.

As resources can only be spend once it is in the best interest of a company to make sure that you fail early in a project. If it turns out that you are unable to train the monkey before you spend resources on building a, then useless, pedestal you have saved burning resources without getting a usable output from it.

Say no to your inner self
It is a natural thing to try to get instant satisfaction, it is a natural thing to build the pedestal first and see what you have achieved. However, it is a wiser thing to try and train the monkey first. On a personal level it is difficult to say no to the natural tendency. Within an enterprise it is equally hard to change the mindset to aim for instant satisfaction. Changing the mindset within an enterprise might even be harder than changing your own inner mindset.

However, changing to a culture of eating frogs is very beneficial for enterprises how strive for innovation. Eating frogs will save valuable time and money and ensures that the focus is on projects that have a higher rate of providing a success. 

Oracle dev – microservice secure bootstrapping for shared secrets

When building microservices, to be more precise when building microservices in containers at one point in time you will start hitting the problem of secure bootstrapping and the handling of shared secrets. This blogpost aims to provide some insights into possible design patterns for handling this problem.
Problem outline

When building microservices or functions which will run from a container you want to provide them as little configuration as possible while at the same time you want them to allow to be configured dynamically at the time of startup. To make the problem a bit more interesting, you want to be able to scale up and down containers dynamically. This problem is directly visible when you are building functions in, for example, Oracle project Fn. A function is in effect a docker container which will only live for a very short amount of time.
s
We did mention configuration we should actually split configuration into two parts; one part is actual configuration and one part is secrets. Configuration is stored in a central configuration store (for example consul from Hashicorp) and secrets are stored in a secret management solution (for example vault from Hashicorp). Making a distinct split between secrets and configuration is a best practice.

The simple bootstrap
Let’s say that you need to build a microservice which needs to call another service within your landscape you might potentially need the following two things to allow your microservice to call the other microservice; (1) a URL and (2) a shared secret for authentication.

A simple way to resolve this is to ensure your container or function will have a bootstrap routine which will call the configuration store in a static manner to acquire the needed configuration and the shared secret from a configuration store. Static manner means that your microservice or function has the URL and possibly a key baked into the deployment which allows it to connect to the configuration store, this is shown in the example below.

The above works and already provide a relative better implementation than building in all configuration and keys needed for microservice 1 to communicate with microservice 2.

Bootstrap with service registry 
The below model is a bit more complex, however, it is also providing some more options. In the below diagram all steps are done over HTTPS, you can however implement step 1 also with a socket connect from your docker container or you can select an IP which is only available within the Docker internal network and is not accessible from outside.

 The reason you might want to secure the “service registry” service from the outside world is because it is key to gaining access to everything else.  In the above model the following steps are executed:

1. The bootstrap of microservice-1 registers at the service registry and receives a key and URL for the configuration store
2. The service registry informs the configuration store about the new service that has been booted and which random secret it will use to connect. 
3. Microservice-1 connects to the configuration store while only providing the secret it received from the service registry. Based upon this secret the configuration store knows which configuration and secrets to return to the service. Additionally, it creates a random secret for this specific instance of microservice-1 to be used to communicate with microservice-2
4. Microservice-1 calls microservice-2 while using the configuration and the key it receives from the configuration store.
5. Microservice-2 receives a call from microservice-1 and verifies the secret with the configuration store. 

The benefit of this model is that only a service which is started within Docker can access the service registry and by enforcing this only a service starting within docker can acquire a key to communicate with the dynamic configuration store. 

Oracle Fn - build your first Go function

In this guide we showcase how easy it is to build a single function using Oracle Fn based upon Go.

The first step in developing a function is to initialize a function with the fn command. Below example shows we init a new function which will be using Go as the programming language and the trigger to call the function will be a http call. We call the function myCoolFunction

[root@projectfn devstuff]# fn init --runtime go --trigger http myCoolFunction
Creating function at: /myCoolFunction
Function boilerplate generated.
func.yaml created.
[root@projectfn devstuff]#

As soon as the initalization is complete we can see that we have a new directory called myCoolFunction which has a number of files inside

[root@projectfn devstuff]# ls -la myCoolFunction/
total 16
drwxr-xr-x. 2 root root  73 Oct  5 18:48 .
drwxr-xr-x. 8 root root  90 Oct  5 18:48 ..
-rw-r--r--. 1 root root 469 Oct  5 18:48 func.go
-rw-r--r--. 1 root root 193 Oct  5 18:48 func.yaml
-rw-r--r--. 1 root root 127 Oct  5 18:48 Gopkg.toml
-rw-r--r--. 1 root root 505 Oct  5 18:48 test.json
[root@projectfn devstuff]# 

If we look at the files the two most important files are the func.go and the func.yaml file. The func.go file will contain the function logic and the func.yaml will contain the configuration of the function.

If we look at the content of func.go we can see that this is a very simple hello world example written in Go which will respond in the form of a JSON response file " Hello World" or in case you provide a JSON payload file with a name it will respond with a "Hello name". In effect the example is very simple and very handy to quickly test if your function is OK and can be used before you start coding your own logic into it.

package main

import (
 "context"
 "encoding/json"
 "fmt"
 "io"

 fdk "github.com/fnproject/fdk-go"
)

func main() {
 fdk.Handle(fdk.HandlerFunc(myHandler))
}

type Person struct {
 Name string `json:"name"`
}

func myHandler(ctx context.Context, in io.Reader, out io.Writer) {
 p := &Person{Name: "World"}
 json.NewDecoder(in).Decode(p)
 msg := struct {
  Msg string `json:"message"`
 }{
  Msg: fmt.Sprintf("Hello %s", p.Name),
 }
 json.NewEncoder(out).Encode(&msg)
}

The content of the func.yaml file will help in the configuration of the function and how it is, for example, accessible externally on what endpoint.

schema_version: 20180708
name: mycoolfunction
version: 0.0.1
runtime: go
entrypoint: ./func
format: json
triggers:
- name: mycoolfunction-trigger
  type: http
  source: 

  Now we have to build and deploy the function. What will happen in the background is that a docker container is build and that the application, the function and the trigger is registered within Fn so it can be called. As we have not defined any application name we will call this application mycoolapp. The command required and the result is shown in the example below.

[root@projectfn myCoolFunction]# fn --verbose deploy --app mycoolapp --local
Deploying mycoolfunction to app: mycoolapp
Bumped to version 0.0.4
Building image mycoolfunction:0.0.4 
FN_REGISTRY:  FN_REGISTRY is not set.
Current Context:  No context currently in use.
Sending build context to Docker daemon  6.144kB
Step 1/10 : FROM fnproject/go:dev as build-stage
 ---> fac877f7d14d
Step 2/10 : WORKDIR /function
 ---> Using cache
 ---> 910b06b938d1
Step 3/10 : RUN go get -u github.com/golang/dep/cmd/dep
 ---> Using cache
 ---> f6b396d6e1fa
Step 4/10 : ADD . /go/src/func/
 ---> 35a944c2ad0f
Step 5/10 : RUN cd /go/src/func/ && dep ensure
 ---> Running in 8ef4cfb23602
Removing intermediate container 8ef4cfb23602
 ---> 75991cccc0b0
Step 6/10 : RUN cd /go/src/func/ && go build -o func
 ---> Running in 5d38abb76d94
Removing intermediate container 5d38abb76d94
 ---> 87f20cf4d16d
Step 7/10 : FROM fnproject/go
 ---> 76aed4489768
Step 8/10 : WORKDIR /function
 ---> Using cache
 ---> 1629c0d58cc1
Step 9/10 : COPY --from=build-stage /go/src/func/func /function/
 ---> Using cache
 ---> ac97ccf6b37f
Step 10/10 : ENTRYPOINT ["./func"]
 ---> Using cache
 ---> 5c61704790e4
Successfully built 5c61704790e4
Successfully tagged mycoolfunction:0.0.4

Updating function mycoolfunction using image mycoolfunction:0.0.4...

In effect, this is the only thing you need to do to get your first function up and running. To test if it is really working we can call the function as shown below and we will get the result back as expected.

[root@projectfn myCoolFunction]# curl http://192.168.56.15:8080/t/mycoolapp/mycoolfunction-trigger
{"message":"Hello World"}

Oracle Cloud - Elastic Search set number of replicas for shards

We see more and more that customers leverage Elastic Search within modern application deployments. Recently we have been experimenting with Elastic Search in the Oracle Cloud. The initial setup was a relative simple cluster setup of a number of nodes. When doing the first resilience test we found out that we where finding that we degraded the cluster state and lost data when turning off a virtual machine in the Oracle Compute Cloud. The reason for this was that we did not set the replication for shards in the correct manner. Ensuring you have set sharding correctly is vital to ensure your Elastic Search Cluster in the Oracle Cloud is resilient against node failure.

Building a cluster in the cloud
When building a Cluster in the Oracle Compute Cloud based upon Oracle Linux this is in effect nothing different from building an Elastic Cluster in any other cloud or in your own datacenter. This holds that the below is applicable to any installation you do.

The general idea when building an Elastic Search Cluster is that you ensure that you have multiple nodes with multiple roles working as one while ensuring that the cluster is capable of loosing a number of its member nodes and will still continue functioning. The main roles within the cluster configuration of Elastic Search are data nodes, master nodes and client nodes.

when configuring your cluster you have to ensure that you are capable of loosing one node and still be able to provide the services to your customers. One of the things we overlooked while building the initial cluster was the level of replica shards.

Use replica shards
When you store an index in Elastic Search this can be broken down into multiple shards. The shards can be distributed over multiple machines (nodes) in the cluster. The idea behind it is two folded. Firstly it helps you to distribute load and secondly it helps you to ensure data is on more than one node at the same time to ensure that the data is available when a node is removed from the cluster.

To ensure the optimal use for both distributing operations as well as ensuring you have a replica of a shard to mitigate against failure you will have to ensure you have a replicated shard. The below image shows this on a high level:

As you can see in the above example we have two shards for a single index, P0 and P1. However, we also have a replica shard for both R0 and R1. In case node 1 is failing for some reason the data is still available for the cluster in the form of shard R0. In this example we have a replication of 1, you can however set the replication much higher. The level of replication will depend on the level of risk you want to take combined with the level of compute distribution you want to achieve against the costs you are willing to have.

As storage is relative cheap in the Oracle Compute Cloud it is advisable to set the replication higher than 1 to ensure you have 2 or more replication shards.

Setting the replication level
You can set the replication level per index in Elastic Search. This will help you to ensure you have more shards for data which is accessed frequently or where you need a higher level of availability. Setting the replication level is done with the parameter number_of_replicas as shown below:

PUT /my_index/_settings
{
  "number_of_replicas": 1
}

Conclusion
using shard replication will help you protect against failure of a datanode in your elastic search cluster and it will improve the performance. As storage is relative cheap in the Oracle Cloud it is advisable to ensure you have set number_of_replicas to 2 or higher.

Oracle Jet - security by obfuscation - Do not use it

Obfuscation “the action of making something obscure, unclear, or unintelligible” is often used when trying to secure an application. The idea behind obfuscation is to make the technical working of an application so unclear to attackers that it will become very hard to figure out the actual working.  Even though obfuscation might look like a good measure it provides no real security against people who intend to understand the real working of an application. A good example of security by obfuscation is URL obfuscation.

When applying URL obfuscation as a security measure a common practice is to obfuscate the URL parameters in such a way that they are not easily understandable by someone who has not build the application.

A recent example I found is the below URL which ends with details?id=QEREUS where at first instance it appears that the ID is a randomly generated ID. When having a randomly generated ID this will make it harder to build a script which will access the details site. However, in this specific example the result of the URL is a JSON file, when examining the JSON file another ID was shown. In this case the ID in the JSON file was A01049 while the ID in the URL was QEREUS

By examining a number of different detail pages quickly a pattern emerges, for example, all URL IDs started with a Q and only used a specific subset of the alphabet. In this specific example the actual ID, shown in the JSON file, was an A followed by a sequential number where the developer has chosen to use a substitution method to obfuscate the actual ID. The below list shows the selected substitution method:

• 0 - E
• 1 - R
• 2 - T
• 3 - Y
• 4 - U
• 5 - I
• 6 - O
• 7 - P
• 8 - A
• 9 – S

When you take a good look at the letters and take a good look at a QWERTY keyboard layout you can figure out why the developer has selected this set of letters to substitute the numbers.

Why people use obfuscation. 
There are very good reasons why one would like to make it hard for externals to guess ID parameters. When you are able to guess the ID parameter it might become very easy to write a script to scrape a website for information. When looking at more modern web based applications, for example Oracle JET based applications, a javascript will call a REST API, this means that the user will be able to gain access to pure JSON (or XML) based information.

In essence there is nothing wrong with external people accessing this information however there might be very good reasons why you do not want to over-promote the use of the pure JSON data outside of the context of your Oracle JET based application.

A better way to do obfuscation
Besides implementing true randomization and more advanced security than obfuscation there are much better ways to do obfuscation in your URLs. An example would be for example simple encryption would work much (much much much) better. The below examples are rijndael-256 encrypted IDs with a base64 encoding

• A01049 - fiCOcGED9SDiYe9du0XUIu1tsHQNGwWVK9uvI755+fg=
• A01050 - cQbgwi1mnsYloonV4ZhqPMo3C2ie0+ilmsZFr3mBb3A= 
• A01050 - U0xs5e4QP6OIclZBCSLuf34WFNRqs3lbtBgkmMiRvkc=

Opposed to the number substitution shown below for the same IDs:

• A01049 - QEREUS
• A01050 - QEREIE
• A01051 - QEREIR 

Should you use obfuscation?
my personal opinion; no. Main reason for saying no is that it provides a false sense of security. It gives you the idea you are safe while someone who is determined will figure it out at one moment in time. There are better and more structural ways of doing things like preventing people from overly active scrape your website. However, if obfuscation is a part of a wider set of security implementations you should think about a very good way of doing obfuscation and not simply rely on a solution like substitution as it will take a very short amount of time for someone to figure out the substitution algorithm

Oracle Linux - security hardening - CIS control 1.1.2

As part of ensuring you deploy Oracle Linux 7 in a secure way the CIS benchmark van provide a good guidance. Following the CIS benchmark will ensure that most of the important security hardening topics will be considered. As with most general guidelines, the Oracle Linux 7 CIS benchmark, not all will apply on your specific situation. Having stated that, it is good to consider all the points mentioned in the benchmark and apply them with a comply or explain model.

Within this series of posts we will go through all the Oracle Linux 7 CIS benchmark controls and outline them a bit more than might have been done on the actual CIS benchmark.

control : Set nodev option for /tmp Partition

The rationale given by the CIS benchmark is ; Since the /tmp filesystem is not intended to support devices, set this option to ensure that users cannot attempt to create block or character special devices in /tmp.

In more detail, if the nodev option is not set it would allow for mounting the /tmp filesystem on a device. In this sense, a device could be for example a USB drive attached to the machine. When, if this is not prevented, a USB drive is added to the machine a device node will be created under /dev which can be used to mount /tmp on. The nodev option will ensure that this is prevented.  The official man page reads the following on nodev : “Do not interpret character or block special devices on the file system.”

The CIS benchmark documentation provides the below command as a way to verify that the nodev option is given. In reality two possible options are provided however, I feel the one below is providing the most assurance that it is actually actively implemented in the right way.

mount | grep "[[:space:]]/tmp[[:space:]]" | grep nodev 

A more extensive version of this check which will provide a pass/fail response is shown below:

#!/bin/bash
mount | grep "[[:space:]]/tmp[[:space:]]" | grep nodev | wc -l&> /dev/null
if [ $? == 0 ]; then
   echo "fail"
else
   echo "pass"
fi

This provides a easier way to implement an automated check if you want to incorporate this in a wider check for your Oracle Linux installation.

Oracle Linux - security hardening - CIS control 1.1.1

As part of ensuring you deploy Oracle Linux 7 in a secure way the CIS benchmark van provide a good guidance. Following the CIS benchmark will ensure that most of the important security hardening topics will be considered. As with most general guidelines, the Oracle Linux 7 CIS benchmark, not all will apply on your specific situation. Having stated that, it is good to consider all the points mentioned in the benchmark and apply them with a comply or explain model.

Within this series of posts we will go through all the Oracle Linux 7 CIS benchmark controls and outline them a bit more than might have been done on the actual CIS benchmark.

control : Create Separate Partition for /tmp

The rational behind this control is : Since the /tmp directory is intended to be world-writable, there is a risk of resource exhaustion if it is not bound to a separate partition. In addition, making /tmp its own file system allows an administrator to set the noexec option on the mount, making /tmp useless for an attacker to install executable code. It would also prevent an attacker from establishing a hardlink to a system setuid program and wait for it to be updated. Once the program was updated, the hardlink would be broken and the attacker would have his own copy of the program. If the program happened to have a security vulnerability, the attacker could continue to exploit the known flaw.

What this in effect means is: that if an attacker would be able to flood /tmp with “junk” data it could lead to a situation where your system disks are full and the operating system is unable to function in the way it should. Additionally, as /tmp is a place most users will be able to write data to an attacker could also write scripts and code to it. As it provides a place to store code it can be abused from that point of view. If you allow users to write to the /tmp space however disallow them to execute code that is stored in /tmp this takes away this specific risk of code execution for code stored in /tmp.

The CIS benchmark provides the below standard code to verify if you have a separate /tmp in place:

grep "[[:space:]]/tmp[[:space:]]" /etc/fstab

Even though this check works the below might be a bit smarter and will provide a pass or fail result based upon the check:

#!/bin/bash

grep "[[:space:]]/tmp[[:space:]]" /etc/fstab | wc -l&> /dev/null
if [ $? == 0 ]; then
   echo "fail"
else
   echo "pass"
fi

In effect the above will do the same as the check promoted in the CIS benchmark document, however, it might be easier to include in a programmatic check. In case /tmp is not a separate file system it will return a fail on this control.

Oracle WebLogic - prevent users from bypassing the SAML authentication

The Oracle Critical Patch Update Advisory of July 2018 provides an advisory on patching Weblogic to resolve two SAML related security issues. Both issues have been registered as a CVE (Common Vulnerabilities and Exposures) entry. For more information you can refer to CVE-2018-2998 and CVE-2018-2933 who provide additional information as well as the Oracle website where you can find more information about the issues found and the way to mitigate against them.

At current the CVE database provides most insight on CVE-2018-2998:

Vulnerability in the Oracle WebLogic Server component of Oracle Fusion Middleware (subcomponent: SAML). Supported versions that are affected are 10.3.6.0, 12.1.3.0, 12.2.1.2 and 12.2.1.3. Easily exploitable vulnerability allows low privileged attacker with network access via HTTP to compromise Oracle WebLogic Server. Successful attacks of this vulnerability can result in unauthorized update, insert or delete access to some of Oracle WebLogic Server accessible data as well as unauthorized read access to a subset of Oracle WebLogic Server accessible data. CVSS 3.0 Base Score 5.4 (Confidentiality and Integrity impacts). CVSS Vector: (CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:N).

Both issues have been found by Denis Andzakovic from Pulse Security. Denis gives more insight on the Pulse Security website on the inner workings and additional details of the exact issue. In effect the issues results in the option to bypass the SAML authentication and by inserting an XML comment, an attacker can coerce the WebLogic SAML Service Provider to log in as another user. When an XML comment is added inside a NameID tag, the WebLogic server only processes the string after the comment. Adding the XML comment does not invalidate the SAML assertion’s signature.

Oracle has provided a solution for the issue found by Denis Andzakovic, the July CPU page provides additional information on how to patch your systems. As an addition the Oracle support website provides document 2421480.1 "Recommendations for Configuring Assertion Signatures in WebLogic Server" with additional information.