Posted 02nd December 2014 by APSIS

Small data – the needle in the big data haystack

The story of Big Data seems to be entering its third stage now. We had the great theoretical excitement over the possibilities that Big Data could create. Followed by the period of experimentation as many companies invested in the likes of Hadoop and hastily threw together as much data as they could find across the organisation. Today, however, many companies are trying to move Big Data from a costly experiment that’s siloed in a ‘Data Scientist’ team, to an integrated part of the business process. If you want evidence for this, check out LinkedIn’s recent move.
Looking forward to 2015, the key to Big Data and its use within organisations might come from the small, actionable data that is the result of Big Data. Not Big Data itself.
Currently, most companies would agree that they have too much data for their systems to handle. In a recent Econsultancy report, 72% of digital marketers cited fragmented data as a problem in their organisation. Often the data is held within different technologies, databases, teams and departments across the business.
Bringing all of this data together, analysing it, mining it, applying algorithms etc. will hopefully result in useful insight (I’m not belittling the vast amount of effort and expertise that goes into this, I just cant cover all that in this blog!). This insight can assist with strategic decisions around marketing, product development, operational efficiency, sales strategy, etc. etc. But using insight to inform strategic decisions is a world away from using data to power an experience with one individual, in real time, as they interact with a brand across different channels and devices.
To speak to an individual you need small data. Small data simply being the minimal amount of information that you need to speak to the consumer in THIS instant. Wherever they may be interacting with you. This represents a very different challenge to a traditional Big Data challenge – which is ‘how do I find one system that can handle all of my data’. Suddenly the challenge is ‘how do I get the right information to the system that is communicating with George Benson on his Android device in Oxford Street right NOW’.
Big Data can create some fantastic insights at a customer level but for the value of that insight to be realised across the organisation it needs to be easier to use that insight in the multitude of different systems that companies are already using (and this link is just the marketing based systems. Not to mention sales and customer service).
I am not the only one talking about this concept and APSIS is not the only platform that is trying to address this problem. The term Data-as-a-Service (DaaS) is becoming used more often. It’s a fancy way of saying ‘right data, right time, right place’.
This post was written by Andy Walker