Monday, April 27, 2015


In the long run, we shape our lives, and we shape ourselves.
The process never ends until we die.
 And the choices we make are ultimately our own responsibility.
                                                                        Eleanor Roosevelt

Choice implies judging the merits of multiple options and making a decision – selecting one or more of them. Selecting none is a choice – which is indecision. If the decision is complex, then reasoning, instinct and feeling become more intertwined.

Simple choices might include what to eat for dinner or what to wear – choices that have relatively low-impact. More complex choices might involve what job to select, making vows to a life partner, picking candidates in election, etc. – choices based on multiple influences and having greater or longer-lasting consequences.

Too many choices

Choice seduces the modern consumer everywhere and all the time – coffee can be tall, short, decaf, flavored, spiced, hot or iced, caramel flan, frappuccino. Any fast-food restaurant has a plethora of choices that confuse visitors from other countries. Being in line adds stress, which often leads to no response – one visiting relative called it “instant indecision”. Is so much choice a good thing?

Every day, everyone faces lots of decisions, large and small – what to eat for lunch, whether to change careers or pursue a new romantic relationship. 

How does the human brain decide? A new study (1) suggests that it relies on two separate networks to do so: one that determines the overall value – the risk versus reward – of individual choices. The other guides ultimate behavior.

Choice Paradox

In his TED talk on the “Paradox of Choice”, psychologist Barry Schwartz discusses a central principle of western societies: freedom of choice. He thinks that choice does not provide freedom, but paralyzes. It does not cause happiness, but dissatisfaction. Simpler times, with fewer choices, are appealing to many. (2)

Daniel McFadden, economist at University of California, Berkeley, suggests that most people find too many options troubling because of the “risk of misperception and miscalculation, of misunderstanding the available alternatives, of misreading one's own tastes, of yielding to a moment's whim and regretting it afterwards”, combined with “the stress of information acquisition”. Indeed, the expectation of indecision can prompt panic and a failure to choose at all. (3)

In a paper published in The American Journal of Psychiatry, in 1970, Lipowski wrote, “I maintain that it is specifically the overabundance of attractive alternatives, aided and abetted by an affluent and increasingly complex society, that leads to conflict, frustration, unrelieved appetitive tension, more approach tendencies and more conflict—a veritable vicious cycle.” That cycle, in turn, likely had “far-reaching and probably harmful effects on the mental and physical health of affected individuals.” (4)

Having too many options demands too much effort to make a sensible decision: better to avoid the decision, or have somebody else pick for you. The vast majority of shoppers in a Californian grocery store faced with 24 varieties of something simply choose not to buy any. The French have a saying: “Trop de choix tue le choix” (too much choice kills the choice).

Psychologists say that having more choices raises expectations too high, making even a good decision feel bad. The potential for regret about the options not taken is greater in the face of multiple choices – resulting in confusion, indecision, panic, regret, anxiety.

Unpopular choices

Often, making a choice is not easy. There are few unanimous choices - 100% agreement in anything is rare.

Change, by definition, requires us to embrace new (or contrary) opinions. When change is necessary, new ideas must be introduced, and new ideas are almost always met with confrontation on some level.

As a leader, it gets really hard to make the unpopular choices. It’s especially hard when a company owner is called upon to make hard, but necessary, decisions such as announcing layoffs and reducing benefits. It’s hard to decide between success and reputation, friendships, and family. It’s likely that someone will be hurt, or unhappy – but that cannot stop the hard choices.  One has to have the self-confidence and the “intestinal fortitude” to make tough choices.

Leaders must recognize that friendships can't get in the way of making the right choice. Good people recognize this need and respect it. Teams don't work well when their leader makes only popular choices.

It’s even harder to go against the popular trend when one is just a regular employee. I remember that Action Instruments, the company I founded in San Diego, gave employees who’d been at the company for at least a year the choice to buy stock ownership. Most employees were enthusiastically in favor and we were known as an employee-owned company. But one person refused, saying, “Ownership implies responsibility, which I don’t need.”  I respected his motivation and accepted his candid admission. I’m happy to note that this person stayed with the company and kept being promoted for his excellent work.

Free Will

Now, let’s dig in deeper into the psychological aspects of making choices. Where does free will come in when making a choice? Is behavior (choice) pre-determined, or does it include free will? (5)

It’s commonly believed that humans have free will. Conscious reflection on the available choices is the best way to achieve objectives; and learn from mistakes. Calculation, strategy, organization etc. are interpreted as key elements to help make specific choices in particular situations.

Free Will Theological Views

A benevolent and all-knowing god always chooses the path that causes the most good, which means there is no real choice to be made, no free will. But, god lets evil exist, which is a required side effect of free will. See my blog – Creation Allegory. (6)

In Christian theology God, by virtue of his foreknowledge, knows what factors will influence individual choices, and by virtue of his omnipotence he controls those factors. This becomes especially important for the doctrines relating to salvation and predestination.

According to Islamic doctrine, free will is the main factor for man's accountability in his actions throughout life, which will be counted on the Day of Judgment because they are man’s own and not God's.

On this subject, I won’t presume to pontificate, but will merely summarize what major monotheistic religions believe. Follow the provided web links to probe deeper. (7)


This is the principle that all events, including human action, are ultimately determined by causes external to the will. Some philosophers have taken determinism to imply that individual human beings have no free will and cannot be held morally responsible for their actions. (8)

Many think that it “feels” like we have free will. If so, then at what point does it assert itself? At what point does free will change the way our thought processes progress? As neurons fire and trigger cascading events, at what point does it free will suddenly stop, or suddenly start? Is there a point in the cycles of our brains' processes where "free will" determines the outcome? (9)

Some philosophers think that circumstances, in line with the strict determinism of physics and biochemistry, predetermine all choices and therefore, free will is an illusion.

Some approaches in psychology see the source of determinism as being outside the individual, a position known as environmental determinism. For example, it’s been shown that children with violent parents will in turn become violent parents through observation and imitation. Others see it from coming inside i.e., in the form of unconscious motivation or genetic determinism – biological determinism. (10) These are forms of determinism.

Modern quantum physics shows that the universe is not deterministic at the atomic scale, which has profound effects on microscopic processes. At the macro level, the classical deterministic physical laws apply. Philosophers and scientists are still debating whether or not determinism applies in the macro cosmos. (11)

Fatalism is the special form of determinism where every event in the future is fated to happen. This does not require that any causal laws or higher powers be involved. Que sera, sera.

The movie, “What the Bleep do we know?” is a 2004 film that combines documentary-style interviews, computer-animated graphics, and a narrative that discusses the connection between quantum physics and consciousness. This was followed by a substantially changed, extended DVD version in 2006, called “Down the Rabbit Hole”. I confess that I’ve viewed these many times. For those with an interest in both science and philosophy, it’s well worth viewing. (12)

Let’s Engage

I’m asking you to engage! Don’t just read passively. Answer the following questions, directly in the blog. Plus insert your comments and ideas.

  1. When simple choices are made, do you like to decide? Or just let things happen?
  2. For key choices in your life (job, partner, home, move, divorce), did you choose?
  3. Do you make your choices? Or do they “just happen”?
  4. Are you happy because you choose to be happy? Do you choose to be unhappy?
  5. Do you think that some of your bad choices led to good things later? Or, vice versa?
  6. Does your free will govern your choices? Or are they pre-determined?
  7. What determines your choices: Chance, Destiny, God, or Fate? Choose one.



1.     Making Choices: How Your Brain Decides:
2.     TED talk – Paradox of Choice:
3.     The tyranny of choice – You choose:
4.     When It’s Bad to Have Good Choices:
5.     The Choice Is Yours: The Fate Of Free Will:
6.     Pinto Blog – Creation Allegory:
7.     Free will in theology:
8.     The Illusion of Choice – Free Will and Determinism:
9.     Freewill and Determinism in Psychology:
10.  Free Will Vs. Free Choice:
11.  Youtube Video - Determinism VS Freewill:
12.  Film – What the Bleep do we know? 

Jim Pinto
Carlsbad, CA.
27 April 2015

Saturday, April 11, 2015

Big Data

Everything we do in the digital realm - from surfing the Web
 to sending an e-mail to conducting a credit card transaction
   to, yes, making a phone call - creates a data trail. And if tha
   trail exists, chances are someone is using it - or will be soon.

                                                                   Douglas Rushkoff

A revolution that compares with the impact of the Internet is changing the way that business, politics, health, education – almost everything – is being conducted. It is pervasive to the extent that everyone knows that it’s there, but no one can do anything to stop encroachment.

Data Revolution

The term "Big Data" was coined in 2008 and caught on quickly as a blanket term for any collection of large and complex data sets that are difficult to handle using traditional data processing. Everything that surrounds everybody at all times generates data. Every digital process and social media exchange produces it: messages, updates, images posted to social networks; readings from sensors; GPS signals from cell phones. Enormous streams of data are tied to people, activities, and locations. (1)

Data arrives from multiple sources at high speed, huge volume and variety, often unstructured and unwieldy. But there’s a huge amount of signal in the noise, simply waiting to be used. Big-data analytics brings decision-making that is at once simpler and more powerful. (2)

It is not the sheer quantities of data that is revolutionary; the revolution is that something can now be done with the data. It does not require more storage or computational capacity, but rather improved statistical methods that can be used to solve problems thousands of times faster than conventional computer methods.  New techniques of data analysis add astonishing new insights and value. (3)

Old-style Data Processing Obsolete

Today there is burgeoning ability to crunch vast collections of information analyze it almost instantly and draw conclusions that are often very surprising. Most commercial transactions and events are transformed into searchable formats to find correlations that could never have been known before.

The structured databases that stored most corporate information until recently are not suited to storing and processing big data – the results are woefully inadequate. So, large computer banks, and large data-processing staffs, are quickly becoming obsolete; processing power is shifting to the Cloud, and new data-intensive approaches are quickly becoming much more economical.

Big Data Applications

Familiar applications of big data include “recommendation engines” such as those used by Netflix and Amazon to offer purchase suggestions based on prior interests of specific customer compared to millions of others. (4)

Consider the emergence and growth of Amazon. Once shopping moved online, the understanding of customers increased dramatically. Online retailers could track not only what customers bought, but also what else they looked at; how they navigated through the site; how much they were influenced by promotions, reviews, and page layouts; and similarities across individuals and groups.

Soon Amazon developed algorithms to predict what products individual customers would like – algorithms that performed better every time the customer responded to or ignored a recommendation. Traditional retailers simply couldn’t access this kind of information, let alone act on it in a timely manner. It’s not surprising that Amazon keeps putting so many brick-and-mortar retailers out of business.

Machine learning

The super-abundance of new data, in turn, accelerates advances. Machine-learning algorithms learn from data and the more data, the more the machines learn. (5)

Take Siri, the talking, question-answering application in iPhones. Apple bought Siri in 2010, and kept feeding it more data. Now, with people supplying millions of questions, Siri is an increasingly adept personal assistant, offering reminders, weather reports, restaurant suggestions and answers to an expanding universe of questions.

Just recently, Amazon Web Services (AWS) unveiled its first product for machine learning - simply called Amazon Machine Learning - to make it easier for AWS developers to extract value from the troves of transactional and operational data their hosted systems collect.

The big data revolution is far more powerful than the analytics that were used in the past. Management can be more precise than ever before, with better predictions and smarter decisions. Areas that have been dominated by intuition can now utilize rigorous data insights.

Research & Government Applications

In the public realm, there are all kinds of applications: finding associations between air quality and health; or using genomic analysis to speed the breeding of crops like rice for drought resistance; allocating police resources by predicting where and when crimes are most likely to occur. Do you remember the futuristic movie, “Minority Report” where a special police unit is able to arrest murderers before they commit their crimes?

At the 2012 World Economic Forum in Davos, Switzerland, Big Data was a major topic and was declared data a new class of economic asset, like currency or gold. (6) The potential for channeling huge amounts of data into actionable information that can be used to identify needs & provide services for the benefit of low-income populations. There was a call for concerted action to ensure that big data helps the individuals and communities who create it. 

Big Data in Political Campaigns

The goal of political campaigns is to maximize the probability of victory. Every activity in a campaign is evaluated by how many votes it can generate and at what cost. To make this cost–benefit analysis, campaigns need accurate predictions about the preferences of voters, their expected behaviors, and their responses to campaign outreach. For instance, efforts to increase voter turnout are counterproductive if the campaign mobilizes people who support the opponent.

Over the past six years, campaigns have become increasingly reliant on analyzing large and detailed datasets to create the necessary predictions. While the adoption of these new analytic methods has not radically transformed how campaigns operate, the improved efficiency gives data-savvy campaigns a competitive advantage. This has led the political parties to engage in a race to leverage ever-growing volumes of data to create votes. The techniques used as recently as a decade or two ago by political campaigns to predict the tendencies of citizens appear extremely rudimentary by current standards. (7)

Competitive Advantage

Analyzing “big data” is becoming a key competitive advantage, generating waves of productivity growth, innovation and consumer surplus. Every business will have to grapple with the implications. The increasing amount and detail of information captured by enterprises, the rise of multimedia and social media and the Internet of Things will fuel exponential growth. McKinsey Research reports that Big Data is now an important factor of production, along with labor and capital. (8)

The use of big data will become a key basis of competition and growth. Every company must take big data seriously. Most industries will leverage data-driven strategies to innovate, compete, and capture value from wide-ranging, deep and real-time information. (8)

Big Data Problems

By combining the power of modern computing with the plentiful data of the digital era, Big Data promises to solve virtually any problem just by crunching the numbers. But, precisely because of its popularity and growing use, we need to be levelheaded about what big data can, and cannot, do.  A NY Times Op-ed points out several fallacies and trends that tend develop significant inaccuracies. (9)

Several issues will need to be addressed to capture the full potential of big data. Policies related to privacy, security, intellectual property, and even liability will need to be re-evaluated in the big data world.

Says Wired magazine, science has a problem in not doing nearly enough to encourage and enable the sharing, analysis and interpretation of the vast swatches of data that individual researchers are collecting. If more credit were given to open sharing of research data, scientific progress would accelerate. (10)

Talent Shortage

To exploit the data flood, the McKinsey Global Institute projects that the United States needs 140,000 to 190,000 more workers with “deep analytical” expertise and 1.5 million more data-literate managers, whether retrained or hired. Clearly there will be a shortage of talent necessary for organizations to take advantage of big data.

Organizations need not only to put the right talent and technology in place but also structure workflows and incentives to optimize the use of big data.

Good Book

In this excellent book on Big Data, two leading experts explain what big data is, how it will change our lives, and what we can do to protect ourselves from its hazards. Big Data is the first big book about the next big thing. (11) Read it.


  1. Wikipedia – Big Data:
  2. NY Times – The Age of Big Data:
  3. Why Big Data Is a Big Deal:
  4. Big Data: The Management Revolution:
  5. Here comes Amazon Machine Learning:
  6. WEF Davos – Big Data, Big Impact:
  7. Changed Political Campaign With Big Data:
  8. The next frontier for Innovation:
  9. Problems With Big Data:
  10. Science’s Big Data Problem:
  11. Book – Revolution That Will Transform:

Jim Pinto
Carlsbad, CA.
12 April 2015