AI and Big Data 2017-11-09T09:15:38+00:00

The Next Big Trend: Big Data + AI

AI and Big Data, Business Insights, Alphago, Ford, Microsoft

From 2013 to 2017, Big Data was the popular buzzword in tech. As computing power improved and data became more accessible with the digitization of the internet, technology was primed for advancements in processing large amounts of data.  Pitches, whitepapers, and water cooler conversations included buzzwords such as Big Data, Hadoop, and Spark.  It was an exciting time because engineers were finally solving data volume challenges by leveraging distributed technologies to efficiently process terabytes of data.

Come 2016, we started to see a shift.  The topic of conversation started to shift towards Artificial Intelligence (AI).  Companies were now more focused on using this data to make intelligent decisions through AI.  At the forefront of AI was machine learning algorithm Deep Learning, a neural network algorithm inspired by the human brain.  Tech companies were making significant strides with this unsupervised algorithm, one of which came in 2016, when Google’s AlphaGo beat a world champion in the ancient Chinese game of Go.  This feat was the first of its kind.  The ability to beat a world champion in Go was a big milestone because of the sheer number of possibilities per move, making effective pre-programming impossible without AI.   

Now AI is making headline news on a daily frequency as every industry is being revolutionized. Andrew Ng, a leader in the AI field and co-founder of, makes parallels between AI and electricity, stating that just as electricity has transformed society a hundred years ago, so will AI.  

In the next few years, we will see a growing prevalence of Big Data and AI technologies working together to deliver AI with not only precision, but with speed.  As AI becomes more mainstream, there will be greater focus to deploy Spark and Storm topologies to deliver intelligent decisions within milliseconds.  This is also known as real time streaming.  Big tech companies such as Google and Amazon have been doing this for many years with search and personalization, but this will become more mainstream than exception.

The applications of real time streaming are everywhere.  The most obvious is website personalization, where companies can create affinity profiles for each customer and serve personalized content real time.  Or in fraud detection, where firms process data real time to flag unusual behavior and stop fraudulent transactions before they happen.  Or in financial lending, where financial lenders can accurately calculate the risk of each applicant real time to make predictive decisions on loans.  The breadth of uses cases are nearly limitless, however the fundamental concept is all the same, which is the ability to accurately predict with speed.

In the next few years, expect to see companies focusing on developing and deploying technologies that harness both Big Data and AI to delivery accuracy and speed.  AI will transform every nook and cranny of society with the help of Big Data technologies as a delivery mechanism.  It is an exciting time that we live in, as there are many opportunities for us to leverage these technologies and improve our standard of living and prosper economically.  And as for companies, it will be the early adopters that will benefit the most.  

Eric Kim
Eric KimFounder of, ML/Data Scientist | Marketing Analytics Leader