The Data Gold Rush has Started!

Peter Gatt
Jun 9, 2017

Gold was first discovered in Australia in Ballarat in late August 1851 on the rise above Canadian Creek at the base of "Poverty Point", which was later renamed to "Golden Point". Reports in the local paper, the Geelong Advertiser, instigated the first rush to the Ballarat Diggings. New mining techniques never before seen such as the "canary" sniffer and sifting tools found their way to the front line in Canadian Creek, and the accessibility of the new technology inspired ordinary people to dig in their own back yards, hoping to make their fortunes.
Fast forward 170 years and history is repeating itself, only this time the "gold" in your own back yard is the data and systems which organisations have been running and maintaining for years. The big data and analytic drive we're seeing today in the IT industry is allowing organisations to uncover and realise the hidden value of their existing data assets.

Why is everyone telling you to start looking at data and analytics?
So why now? Well, historically, only the larger companies and enterprises could afford expensive infrastructure and tooling to analyse their data in near real time and provide amazing insights for themselves and their clients. But now, with the evolution of cloud services such as those offered by AWS, all organisations - small and large - now have the ability to take advantage of powerful and scaleable infrastructure and tools. This gives them the ability to analyse Big Data feeds and large amounts of data in real time.
You might not know it, but you may have a fortune in data gold in your own back yard.
So what sort of data is valuable? There are many articles which articulate some of the cheap yet valuable data which most organisations store and don't take advantage of. In fact, you could well have a fortune in unrealised value locked up in your data today. A very simple example is that of Sales and Marketing data: being able to analyse how your customers buy can often reflect new and improved sales strategies.
"I just found that my customer has come to my site via <source>, so if I take out a targeted campaign with <source>, I'm likely to drive an increase in leads and potential revenue"
The obvious data wizard in the marketing space is Google which completes the sentence above as one of their offerings. In fact, Google Adwords has been key to Google's global success. But who else is doing it too difficult for just anyone to do? Certainly not. Almost every retailer out there is analysing customer behaviours and buying patterns, and then (either periodically or sometimes in near real time) using this data to alter the suggested sell to customers' shopping carts.
But there are also many other forms of accessible and affordable enterprise data sets which can be taken advantage of that you may not know exist. Read about our case study with Powercor where we worked together to build a Big Data and Analytics Platform which enabled them to transform their digital business for customers and engineers.
But surely this can't be done without a massive cash injection and huge projects requiring significant investment in data excavation tools? Not at all! Those days are let me show you how:

Panning for Gold

In ways which are very similar to the concept of panning for gold, you need to pick up some data sets that you think may contain valuable data and metaphorically sift it through your business and marketing teams. These teams have an uncanny knack of understanding the value of data which isn't immediately apparent to others. This is something the Vibrato team has witnessed many times throughout our Data Platform workshops. You can learn a little more about them here.
To get started, you really need a "place to play" and enough permission or de-sensitised data so that you can share your findings with your marketing and business teams. But even then, where do you look? In our experience with many customers, these are the key areas where Marketing Insights and Business teams tend to find data gold: 
  • Fraud Detection
  • Loss Prevention
  • Buying Trends
  • Unified User/Customer
  • Sentiment Analysis
There are plenty of other areas too, and Vibrato is confident that if you get in touch with us <here>, we'll be able organise a workshop and work with you to find the right places to start panning for your own data gold. 
But as is often the case, experimentation is key. Get started with some simple data sets and your own laptop. It's the equivelant of taking home a bucket of dirt from a quarry and sifting through things with a stick - break up the rocks and clumps in different ways and see whether you can find something interesting.
Remember, you can also pan for gold in a river, which we liken to investigating and correlating events which come from data streams such as logs or alerts. Often just doing a simple count on various events that occur in logs can quickly help you detect some data gold that didn't know you had. It's once you've found these little fragments that you're ready for the next step... 

Selling a Nugget

Once you've found the most promising data seam and you've decided that you might just know where to set up your mining rig, it's time to get some funding. Rigs can be expensive if they're set up permanently in the wrong spot with years of work put into deep foundations and support structures which are too rigid to move. But guess what; you're in luck! With the power of the cloud enabling elastic and scalable on-demand resources, you have the opportunity to set up your rig in such a way that it can eventually be reinforced if it needs to become permanent. OK, mining analogies aside, we're talking about establishing a cloud data initiative which isn't throw-away and can deliver consistent Return On Investment. 
My recommendation is that you build a Proof of Value (PoV), not Concept (PoC). I can't even recall the countless times I've heard "Proof of Concept" thrown around or even acted upon in our new cloud world which so often leads to a dead-end solution. Back to mining analogies, if you were setting up a gold sifting operation, would you let the gold fall out of the sift and be lost in dirt on the other side? Certainly not... so why not set up the means to collect the gold, process it and make a return on your investment? In line with this analogy, I recommend finding a couple of Proofs of Value data processing and analysis solutions which you can get running in production. Build a Minimal Viable Product (MVP) data mining operation which still allows for a future robust framework. 
Once you've established the sort of data gold you're likely to find, you'll be in a position to assess the potential Return on Investment of extracting that same gold. This needs to be calculated as a priority, as it will be this tangible financial return which will enable you to get business permission to invest further in your mining expedition. Every mining propector has a financial backer and you should too - having demonstrated the potential value of your data, you've got a strong business case to secure additional funding. 

Plot your mining expedition and expansion

If you're still reading, stop and go and find your own nugget of data gold.
But in case you're interested in the full story and you're serious about making a fortune from data, then please keep reading. It's at this point in your journey where you're going to need to advertise your findings. I would start with "Gold Rush in <insert company name>" as being as a good a presentation title as ever. However, If you do this, you're going to have all sorts of data requests (opportunistic miners) and plot expansion requests (new data sets) in a short space of time, so you really need to be prepared for the influx. 
Develop a Data Platform so that you can take control of your data and automate the construction of more mining practices.
Vibrato has a wealth of experience to work with you and help establish a robust Data Platform which will take data operations to the next level. It should be automated and allow for an ease in raw data set collections (Data Lake) as well as collect all of your long-term structured data ready for heavy algorithms to be run (DataWarehouse). You can see an example of these sorts of lake implementations for AWS below.
What's next? It's important that you work with your Business and Marketing teams to build a roadmap for your mining expeditions. Having a healthy list of twenty to thirty data-driven solutions will truly empower your organisation to unearth the full potential of your data seam. With the initial return on investment and gold harvest, you should have enough capital and business support to invest in developing your very own data quarry. 


Now that you've got some success and capital behind you, as well as a map of the surrounding areas which have more valuable minerals scattered throughout, you're ready to establish a quarry. What makes a quarry a little different to your normal operations?
  • Deep Core Earth Excavators - These are the equivelant of your big data pipes and possible kinesis streams
  • Fast Processing - In technical terms, you're going to need some BigData and transformation capabilites such as EMR, Spark Clusters and Machine Learning techniques.
  • Long-Term Storage - You may need BigData Lake and DataWarehousing capabilities.
 And don't forget: 
  • Having sold some gold and having a healthy treasure map of your back yard, it's time to excavate.
  • Building an enterprise-grade Data Platform is not an easy task, but with careful planning and innovation you can succeed in building your own Data Quarry.
  • You'll need some heavy lifting capabilities from time to time, so make sure you've planned for on-demand and just-in-time scaling for when that BIG nugget comes along.


Just remember:

  • It's worth panning for your gold in your own back yard, even a small nugget can be worth a fortune to your business
  • Starting with a Proof of Value is simple, and AWS make it easy to get started.
  • Get started today with these simple reference architectures, but if you're after a tried and trusted platform, Vibrato is happy to work with you to plot your mining expedition
  • For the enterprise clients out there, we'd love to chat to you about building your very own quarry - we call it a Data Platform!
  • Please contact us about our 3-day workshop and chat to our clients today about how they've found the program. It's easy, just fill out the form below:

 Register for a Data Platform Workshop!