Big Data

Everything we do in the digital realm - from surfing the Web

 to sending an e-mail to conducting a credit card transaction

   to, yes, making a phone call - creates a data trail. And if that 

   trail exists, chances are someone is using it - or will be soon.

                                                                   Douglas Rushkoff

 ………………..

A revolution that compares with the impact of the Internet is changing the way that business, politics, health, education – almost everything – is being conducted. It is pervasive to the extent that everyone knows that it’s there, but no one can do anything to stop encroachment.

Data Revolution

The term "Big Data" was coined in 2008 and caught on quickly as a blanket term for any collection of large and complex data sets that are difficult to handle using traditional data processing. Everything that surrounds everybody at all times generates data. Every digital process and social media exchange produces it: messages, updates, images posted to social networks; readings from sensors; GPS signals from cell phones. Enormous streams of data are tied to people, activities, and locations. ⁽¹⁾

Data arrives from multiple sources at high speed, huge volume and variety, often unstructured and unwieldy. But there’s a huge amount of signal in the noise, simply waiting to be used. Big-data analytics brings decision-making that is at once simpler and more powerful. ⁽²⁾

It is not the sheer quantities of data that is revolutionary; the revolution is that something can now be done with the data. It does not require more storage or computational capacity, but rather improved statistical methods that can be used to solve problems thousands of times faster than conventional computer methods.  New techniques of data analysis add astonishing new insights and value. ⁽³⁾

Old-style Data Processing Obsolete

Today there is burgeoning ability to crunch vast collections of information analyze it almost instantly and draw conclusions that are often very surprising. Most commercial transactions and events are transformed into searchable formats to find correlations that could never have been known before.

The structured databases that stored most corporate information until recently are not suited to storing and processing big data – the results are woefully inadequate. So, large computer banks, and large data-processing staffs, are quickly becoming obsolete; processing power is shifting to the Cloud, and new data-intensive approaches are quickly becoming much more economical.

Big Data Applications

Familiar applications of big data include “recommendation engines” such as those used by Netflix and Amazon to offer purchase suggestions based on prior interests of specific customer compared to millions of others. ⁽⁴⁾

Consider the emergence and growth of Amazon. Once shopping moved online, the understanding of customers increased dramatically. Online retailers could track not only what customers bought, but also what else they looked at; how they navigated through the site; how much they were influenced by promotions, reviews, and page layouts; and similarities across individuals and groups.

Soon Amazon developed algorithms to predict what products individual customers would like – algorithms that performed better every time the customer responded to or ignored a recommendation. Traditional retailers simply couldn’t access this kind of information, let alone act on it in a timely manner. It’s not surprising that Amazon keeps putting so many brick-and-mortar retailers out of business.

Machine learning

The super-abundance of new data, in turn, accelerates advances. Machine-learning algorithms learn from data and the more data, the more the machines learn. ⁽⁵⁾

Take Siri, the talking, question-answering application in iPhones. Apple bought Siri in 2010, and kept feeding it more data. Now, with people supplying millions of questions, Siri is an increasingly adept personal assistant, offering reminders, weather reports, restaurant suggestions and answers to an expanding universe of questions.

Just recently, Amazon Web Services (AWS) unveiled its first product for machine learning - simply called Amazon Machine Learning - to make it easier for AWS developers to extract value from the troves of transactional and operational data their hosted systems collect.

The big data revolution is far more powerful than the analytics that were used in the past. Management can be more precise than ever before, with better predictions and smarter decisions. Areas that have been dominated by intuition can now utilize rigorous data insights.

Research & Government Applications

In the public realm, there are all kinds of applications: finding associations between air quality and health; or using genomic analysis to speed the breeding of crops like rice for drought resistance; allocating police resources by predicting where and when crimes are most likely to occur. Do you remember the futuristic movie, “Minority Report” where a special police unit is able to arrest murderers before they commit their crimes?

At the 2012 World Economic Forum in Davos, Switzerland, Big Data was a major topic and was declared data a new class of economic asset, like currency or gold. ⁽⁶⁾ The potential for channeling huge amounts of data into actionable information that can be used to identify needs & provide services for the benefit of low-income populations. There was a call for concerted action to ensure that big data helps the individuals and communities who create it. 

Big Data in Political Campaigns

The goal of political campaigns is to maximize the probability of victory. Every activity in a campaign is evaluated by how many votes it can generate and at what cost. To make this cost–benefit analysis, campaigns need accurate predictions about the preferences of voters, their expected behaviors, and their responses to campaign outreach. For instance, efforts to increase voter turnout are counterproductive if the campaign mobilizes people who support the opponent.

Over the past six years, campaigns have become increasingly reliant on analyzing large and detailed datasets to create the necessary predictions. While the adoption of these new analytic methods has not radically transformed how campaigns operate, the improved efficiency gives data-savvy campaigns a competitive advantage. This has led the political parties to engage in a race to leverage ever-growing volumes of data to create votes. The techniques used as recently as a decade or two ago by political campaigns to predict the tendencies of citizens appear extremely rudimentary by current standards. ⁽⁷⁾

Competitive Advantage

Analyzing “big data” is becoming a key competitive advantage, generating waves of productivity growth, innovation and consumer surplus. Every business will have to grapple with the implications. The increasing amount and detail of information captured by enterprises, the rise of multimedia and social media and the Internet of Things will fuel exponential growth. McKinsey Research reports that Big Data is now an important factor of production, along with labor and capital. ⁽⁸⁾

The use of big data will become a key basis of competition and growth. Every company must take big data seriously. Most industries will leverage data-driven strategies to innovate, compete, and capture value from wide-ranging, deep and real-time information. ⁽⁸⁾

Big Data Problems

By combining the power of modern computing with the plentiful data of the digital era, Big Data promises to solve virtually any problem just by crunching the numbers. But, precisely because of its popularity and growing use, we need to be levelheaded about what big data can, and cannot, do.  A NY Times Op-ed points out several fallacies and trends that tend develop significant inaccuracies. ⁽⁹⁾

Several issues will need to be addressed to capture the full potential of big data. Policies related to privacy, security, intellectual property, and even liability will need to be re-evaluated in the big data world.

Says Wired magazine, science has a problem in not doing nearly enough to encourage and enable the sharing, analysis and interpretation of the vast swatches of data that individual researchers are collecting. If more credit were given to open sharing of research data, scientific progress would accelerate. ⁽¹⁰⁾

Talent Shortage

To exploit the data flood, the McKinsey Global Institute projects that the United States needs 140,000 to 190,000 more workers with “deep analytical” expertise and 1.5 million more data-literate managers, whether retrained or hired. Clearly there will be a shortage of talent necessary for organizations to take advantage of big data.

Organizations need not only to put the right talent and technology in place but also structure workflows and incentives to optimize the use of big data.

Good Book

In this excellent book on Big Data, two leading experts explain what big data is, how it will change our lives, and what we can do to protect ourselves from its hazards. Big Data is the first big book about the next big thing. ⁽¹¹⁾ Read it.

References:

1. Wikipedia – Big Data: http://goo.gl/Dic8ug
2. NY Times – The Age of Big Data: http://goo.gl/iQMYSH
3. Why Big Data Is a Big Deal: http://goo.gl/coeKMF
4. Big Data: The Management Revolution: http://goo.gl/Cy3zzX
5. Here comes Amazon Machine Learning: http://goo.gl/KaX3jH
6. WEF Davos – Big Data, Big Impact: http://goo.gl/ru7oRC
7. Changed Political Campaign With Big Data: http://goo.gl/0wMXGV
8. The next frontier for Innovation: http://goo.gl/r2KKqf
9. Problems With Big Data: http://goo.gl/L2hPtM
10. Science’s Big Data Problem: http://goo.gl/UlhyiI
11. Book – Revolution That Will Transform: http://goo.gl/dv6JGH

Jim Pinto

Carlsbad, CA.

USA

12 April 2015