Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

January 05 2012

Understanding randomness is a double-edged sword

The Drunkard's Walk coverLeonard Mlodinow's "The Drunkard's Walk: How Randomness Rules our Lives" is a great book on an important subject. As data scientists know, random phenomenon are everywhere, and humans don't understand them well. We're not wired to understand them well. This book is a huge help, and will be a relief to anyone who's heard people say "I don't believe in global warming because last winter we got a lot of snow," or some load of crap like that. The book is well written, there's a lot of storytelling, and the storytelling is fun and interesting. Along the way Mlodinow gives coherent explanations of Bayes' theorem, the Monty Hall problem (offering the simplest correct explanation I've ever seen), the origins of statistics and more. If you want an excellent non-mathematical introduction to probabilistic thinking, this is the book to get. (If you want the mathematics, this book studiously avoids equations. Get William Feller's "An Introduction to Probability Theory and Its Applications" for the deeper material.)

But there's always a but. But, but, but ...

I have two problems with "The Drunkard's Walk." They've been nagging me ever since I finished.

First, Mlodinow spends a lot of time debunking the notion of "hot streaks." He's right, and that's important: most hot streaks in sports and elsewhere can be adequately explained by randomness. Randomness is inherently streaky and clumpy; it's not just a smooth gray. In fact, if you get something that looks smooth and "random," it's almost certainly not random. So far, so good. But — when he moves from Roger Maris' record-breaking season to portfolio managers picking hot stocks, there's a fundamental asymmetry.

With Maris, the author starts with the long-term batting average. We're not just "flipping coins"; we're flipping a weighted coin, a coin that happens to land with the "home run" side facing up a lot more frequently than it would if I were in the batter's box. That's all well and good. If I faced a season's worth of professional baseball pitching, I daresay I wouldn't get a single hit, let alone any home runs. But — and this is important — he doesn't do the same for the stock pickers, book acquisition editors, or Hollywood movie execs that he talks about. For them, it's just flipping coins. And it's one thing to say that, if you just flipped coins for 10 years, you'd have a 75% chance of duplicating a great financial manager's performance over some five-year period. It's another thing to imply that the manager's performance is just a matter of luck, not skill. Yes, there is a lot of luck involved, but where's the notion of baseline performance, of long term success or failure, that was the starting point for analyzing Maris' hot year? Maris' hot year may have been a random phenomenon, but it was a random phenomenon in the context of five years hitting more than 20 home runs per season, during which his cumulative batting average was somewhere around .271. What's the stock picker's cumulative batting average? Who are the other financial analysts working at the same level? We never find out. And that's a big part of the story to omit.

Second, Mlodinow frequently forgets one of the most important aspects of the mathematical study of random processes. When we're talking probability and statistics, we're talking about interchangeable events. It's easy to forget this, but as Mlodinow himself points out, there are many, many ways to make important mistakes when you're talking about probability. The important thing about urns with black and white balls is that the balls are the same. (If you don't know about urns, take a probability course or read the book; they're baked into the history of probability theory.) If some of the balls were ovals and some were star-shaped, these probability experiments wouldn't work.

So, back again to the stock pickers, the acquisitions editors, and the Hollywood execs. We agree at some level that all at-bats in baseball are equivalent. This is, of course, an idealization, but it's one we're fairly comfortable with. But all stocks are not the same, all books are not the same, and all movies are not the same. They may be the same within a certain class (energy stocks, cheap romance novels, spy movies). A stock analyst who's good with financials may have nothing to say about manufacturing. But at the high end of the spectrum (literary novels, fine wines, art movies), everything is unique, precisely in a way that Harlequin romances aren't. Probability and statistics are still powerful tools, but you have to be very careful about how you apply them.

Since I'm in the publishing business, I'm particularly annoyed by the story of an editor who, in an experiment, was given a typewritten chapter of a V. S. Naipaul novel that had won a major award. She rejected it. I'm not a fan of Naipaul, so I'm sympathetic. But is that evidence of her editorial skill (or lack thereof), or of random processes? Since we're now in a world where every event is unique we have to ask more questions: What publisher was she working for? Grove Press, which publishes top drawer literary fiction with a tendency toward the avant garde (for whom Naipaul might have been too stodgy)? Or Bantam, which specializes in lightweight beach-side reading? In both cases, a rejection would have been perfectly appropriate. Probability aside, it's a cheap shot to say: "Because this book won a major award, we'd expect editors at a publishing company to accept it. If they don't, that's evidence that publishing is a random process."

Publishing (and movies, and wines, and maybe even stocks) are a different world, and the disagreements are precisely what is important. Modeling disagreement as random fluctuation isn't doing anyone a service. I may dislike Naipaul's fiction, but I hardly see that as a random result. We could ask about the conditional probability that an English major will dislike Naipaul, given that the English major plays piano, has a strong background in electrical engineering and mathematics, and likes Salman Rushdie, and use that to come up with some sort of number. But I'd have no idea what that number means. We're not picking black and white balls out of urns here — or if we are, the balls are of different shapes and sizes.

Am I just going back to the human tendency to build stories where there is nothing but randomness? Am I just refusing to deal with the stark realities of random phenomenon that surround us everywhere? Perhaps. Then again, that's what makes us human. And in the many situations where probability and statistics aren't appropriate tools, such as picking books or movies, then all we have to fall back on is our ability to make stories, our ability to make sense. Where "make" is precisely the most important word in that last sentence.

Strata 2012 — The 2012 Strata Conference, being held Feb. 28-March 1 in Santa Clara, Calif., will offer three full days of hands-on data training and information-rich sessions. Strata brings together the people, tools, and technologies you need to make data work.

Save 20% on registration with the code RADAR20

Postscript

There's an important, but subtle, distinction to be made between events that can be modelled by random processes and events that are actually random processes. Mlodinow makes this same distinction in his discussion of Maris. At one point, he grants that a hot or cold streak could be the result of changes in exercise, eating habits, personal stress, or any number of non-random factors; but since we can't account for these, he rolls them up into randomness. So, essentially he's saying that a hot streak can be modelled as a random process, though it may have an underlying cause that isn't random at all. Say Maris signed a contract to appear on the front of Wheaties boxes, and decided that he might as well eat the stuff. And say that eating Wheaties actually did increase his slugging percentage significantly. If so, betting heavily on Maris during a hot streak might not be such a bad idea, since he's not just hitting well because he happens to be lucky. And if so, I would still bet heavily that Maris' record-breaking year could be modelled as a random process. After all, probability and statistics are very blunt instruments.

Rolling up potentially non-random factors that can't be measured into "randomness" is a common trick, and reasonably acceptable. You can't analyze what you don't know. But it's a trick that worries me. Let's take a situation that I think is similar, but with much more profound consequences. A decade or so ago, it was well-known that Tamoxifen was a useful drug against breast cancer, effective in roughly 80% of all cases. That's equivalent to saying that Tamoxifen has an .800 batting average. You could model Tamoxifen's success by flipping a coin that came up heads 80% of the time.

But more recent research has revealed that Tamoxifen's story isn't random; at least, not random in that way. It's successful almost 100% of the time on patients with certain genetics and almost 0% of the time on other patients. In other words, the randomness is in the stream of incoming patients, not the effects of the drug. That discovery has a huge practical effect on breast cancer treatment. You can do tests to figure out whether treating a patient with Tamoxifen is likely to be successful, or a waste of time. You can also look in a more focused way for treatments that will be effective on the remaining 20%. Even more important: It's my belief that the next generation of medicine will be "personalized." Rather than using drugs that have been successful in broad clinical trials involving thousands of patients, we'll be focusing on drugs that are tuned to an individual's genetic makeup. Is it possible that the drug that would be effective on the 20% of women who don't respond to Tamoxifen has already been discovered and discarded, because its success rate wasn't statistically significant? Is it possible that there's a drug that's 100% effective on only 5%? Or 1%? What methods will we use to evaluate the performance of these drugs?

Understanding randomness is a double-edged sword. Humans are built to create patterns, even when there's nothing going on but random phenomena. Granted, that's an extremely important story, and Mlodinow does an excellent job of telling it. At the same time, we are wired to create stories, and can't afford to let randomness stop us from doing so, particularly when a story that gives a richer understanding of the data is just beyond our grasp. Understanding what is random and what is not (or, more precisely stated, understanding what parts of any processes are really random) is the key. While humans are all too willing to grasp at the straws of a story when there's no story there (just go to any casino), we can also throw out the stories we haven't yet finished because we're convinced there's nothing there. And that's a tragedy.

November 23 2011

February 15 2010

Innovation Lessons in "Start-Up Nation"

One might expect href="http://www.startupnationbook.com/"Start-Up Nation: The Story
of Israel's Economic Miracle
to come from the pen of business
school or economics professors, but the biographies of authors Dan
Senor and Saul Singer reveal policy backgrounds. Both were advisors in
the U.S. Federal Government.

These backgrounds give a clue that Senor and Singer aim beyond
questions of how to be a successful entrepreneur or high-tech
executive. In fact, their book is a serious investigation of the
social, historical, and psychological traits that produce
extraordinarily creative people--and significantly, creative people
who can translate their cranial light-bulbs into technologies with the
potential to change the world.

The book has garnered a fair amount of news coverage, but still not as
much as it deserves, in my opinion. It took me only about three hours
to read, and I highly recommend it as a refreshing--but not
necessarily reassuring--perspective on a country that is profoundly
misunderstood and misrepresented by media outside its diminutive
borders.

In this blog I'll summarize the traits that that the authors find make
Israel a successful incubator for innovation, distinguishing between
traits that other countries can emulate and traits that seem uniquely
embedded in Israel's historical and geographic circumstances.
Finally, as I usually do in these book reviews, I'll lay out three
observations that came to my mind while following the authors'
argument: the importance of hard data, flipping axioms, and the
creative role government can play.

Israel's stunning performance in world markets is beyond argument. The
country lists more companies on NASDAQ than China, India, Japan, South
Korea, Singapore, and all the countries of Europe--combined! Venture
capital flows into the country. It's often noted that technically
trained Israelis emigrate, but less often reported that many of them
return and spectacularly exploit their international connections. And
because investment is based on technical progress instead of clever
financial book-keeping, Israel has weathered the current recession
with some decline in revenues but not a single bank failure.

Start-Up Nation claims to reveal the reasons behind the
success of this country that not so long ago was considered a victim
of quasi-socialist stagnation and crippling costs related to war and
violence. Some of the authors' observations have been made
before--such as the importance of forcing young people to take serious
responsibility in army service, and the value of accumulating highly
trained immigrants from Russia and other places--but these
observations have previously been oversimplified and leave out what
Senor and Singer consider crucial inputs. Start-Up Nation not
only tries to restore the proper balance and show traits from a more
productive perspective, but integrates them into a comprehensive
picture of a churning economy.


Traits that other countries can emulate


Although Israel has special advantages, some of the elements to which
Senor and Singer trace its innovativeness can theoretically be
achieved elsewhere. Briefly, these are:

  • A loyalty to the entire community that goes beyond personal success.
    The authors point out that, for all of Israelis' notorious
    fractiousness, they expend enormous effort helping total strangers.
    All of Israel is a single team, even a single family. (Obviously, this
    family feeling does not extend to non-Jews.) Israeli entrepreneurs who
    give talks abroad often play up the strengths of their country as well
    as their company.

  • A sense of dissatisfaction. To innovate, one must be convinced that
    things are not good enough the way they are now. For Israelis, this
    drive for change has both Biblical and more recent historical roots,
    but technology provides a new arena rewarding hopes for improvement.

  • A Do-It-Yourself approach to technology, which perhaps is one
    manifestation of the afore-mentioned innate dissatisfaction. The
    authors report that equipment purchased by the army is always being
    tinkered with. The same interest in taking things apart and
    jerry-rigging them extends throughout the culture.

  • A culture of challenging authority. The authors point out that this is
    a deep cultural value (and like many before them, trace it partly to
    the Jewish intellectual tradition), one that is particularly hard to
    foster in countries with controlling regimes.

  • A determination to succeed against all odds. Countries that get
    complacent and rest on their laurels--as most observers think North
    Americans are doing--eventually lose their privileged places. The
    authors highlight fascinating stories of Israelis keeping up
    production in the face of war, and of cheerfully taking on seemingly
    impossible challenges.

  • Interdisciplinary agility. Israelis tend to learn many skills--partly
    to survive in the armed forces--and to form companies closely linking
    people with different areas of expertise. In an age where many
    challenges require mashups between disciplines, this imparts a strong
    advantage.

  • A tolerance for failure. Like the Silicon Valley, Israel is a place
    where someone can start a company, manage it through bankruptcy, and
    then pick up to start another company. A single failure, the authors
    say, gives the entrepreneur a high chance of succeeding at the next
    venture. Even in the military, people are rewarded for tackling
    problems with creative intelligence--not so much for the ultimate
    success or failure of the attempt.

  • Providing young people with arenas to exert responsibility. In Israel,
    of course, this arena is its unusually unhierarchical armed forces
    (and people who don't do army service, such as Arabs and the
    ultra-orthodox, miss out on critical experiences). But other countries
    could find other ways to challenge youth in situations where taking
    charge is a must and where results really matter.

  • A fruitful mentoring relationship between venture capitalists and new
    entrepreneurs. Injecting money into new ventures (as so many
    countries do) is not enough; the managers must be guided through the
    shoals of financial, technical, and human resource challenges. Israel
    set up a unique program called href="http://www.yozma.com/overview/">Yozma in 1993 to bring
    together all the necessary elements.

  • Government policies friendly to startups. Israel has a decidedly mixed
    history here. Even after making a historic turn away from government
    control and toward a free market, its environment is most helpful to
    computer and high-tech companies. There are certainly innovations in
    many other areas--notably agriculture--but the authors say these
    fields encounter hampering regulations.

  • A truly open-arms approach to immigrants, who bring not only fresh
    perspectives but a high tolerance for risk. Once again, of course,
    Israel's liberal attitude toward immigrants applies only to Jews (and
    a lot of haggling goes on around deciding who qualifies). Even for
    Jews, it can take a long time to assimilate waves of newcomers and
    turn them into productive employees. But countries that don't make it
    easy to set down roots suffer economically. Short-term foreign workers
    never form the sustainable innovative institutions that can be planted
    by truly committed immigrants.


Traits unique to Israel's history and geography


Israel also possesses unique traits that the Senor and Singer draw in
to explain its entrepreneurial success. No other culture has undergone
thousands of years of persecution culminating in genocide--nor should
that be a necessary price. But it's amazing how Israel has turned its
liabilities into assets, and these lessons can be inspirational for
people suffering in other parts of the world.

  • The country's regional isolation (boycotted by all its neighbors) and
    its small size has forced it to think internationally from the start.
    Israelis are also avid travelers, gathering up experiences that allow
    them to think creatively.

  • Demographic diversity is another trait that has many negative
    repercussions, but that the Israelis have also mined for its
    strengths. New ideas always seem less odd in a culture where one has
    to get along with different types of people.

  • A long intellectual tradition encourages a love of learning that
    extends to worldly accomplishments (although I have to point out that
    the same tradition contains many rabbis who railed against the study
    of mundane things); just as important, it encourages students to
    examine and re-examine every idea from new points of view--effectively
    making them ask constantly what would result if something obvious were
    not true. I will explore this intellectual attitude in the
    section Flipping axioms.

  • The tight-knit society leads to a situation where "everybody knows
    everybody" and one can easily find the right mix of staff for a new
    company. (I wonder whether this cliquishness extends far beyond the
    economically dominant ethnic group--the Ashkenazi Jews, who trace
    their ancestry to Northern and Western Europe.)

  • The constant existential threat posed by attacks and bombings call for
    hair-trigger creativity. It's worth noting here that the authors
    present Israel's response to war and terror in a consistently heroic
    light (along with recognition that they sometimes screw up). But the
    authors note that every engagement is followed by an assessment to
    determine what could have been done better.

  • The small size of the population (further reduced, as I noted earlier,
    by deferrals for the religious and for Arabs) forces each army recruit
    to take on responsibility right away.


The importance of hard data


At this point I will delve into some more subtle threads that I see
running through the book.

The first is the importance of accumulating evidence to build a case
for change. Two incidents from the book illustrate this nicely:

  • Israel's Intel facility, which invented the historic 8088 (and would
    later invent the Core 2 Duo) faced its biggest uphill battle when it
    approached its parent company to propose the principle behind the
    Centrino. Senor and Singer note that Israeli staff flew into Intel
    headquarters insistently to talk about the advantages of the new
    design--and seemed to deal with management resistance by increasing
    their own pressure. But it's important to note that they accumulated
    data to prove the superiority of the design. Mere persistence
    would probably not have carried the day.

  • An Israeli company called Fraud Sciences pitched a system for
    identifying untrustworthy customers and vendors to a manager at
    eBay. The manager was adamantly unwilling to believe that this tiny
    group of developers could do a better job than all of eBay's experts,
    and the founder of Fraud Sciences was not a good salesman. Luckily,
    eBay had a low-risk, low-cost way to test the start-up: eBay would
    give information about one hundred thousand customers to Fraud
    Sciences and compare their analyses to eBay's analyses as well as the
    subsequent history of those customers. Fraud Sciences passed the test
    with flying colors and ended up being bought by eBay. There was no way
    even the most determined cynic could argue with the facts.

We now see data-crunching brought right into the development process,
as with a process called "predictive drug development" practiced by
the Israeli company Compugen.


Flipping axioms


I mentioned before the oddly subversive style of the Jewish
intellectual tradition. I call it flipping an axiom: find the key
trait that holds back change--a trait that no one has challenged
before because it seems unassailably true--and ask "What would happen
if it were false?" This is a kind of non-Euclidean geometry of
thought. Two examples of this thinking at work in Israel include:

  • The Centrino project at Intel, mentioned in the previous section,
    required overturning a tautology accepted throughout a whole industry
    and customer base. The problem they were facing was that increases in
    chip speed led to increases in heat dissipation. Laptops with advanced
    processors could not be designed because the build-up of heat would
    destroy them.

    So the Israeli team asked: what if we sped up processing without
    increasing clock speed? To give an indication what a radical thought
    this was, consider that U.S. Intel management rejected the idea
    because they had build their entire marketing strategy around clock
    speed. Clock speed had become the universal measure for computer
    performance. The Israelis threw out the assumption, found a solution,
    and sold it to management--and the world.

  • Start-Up Nation begins appropriately enough with the
    well-known innovator Shai Agassi, who wants to create a network of
    battery replacement stations to support an infrastructure of electric
    cars. The basic elements of the plan had all been thought up by auto
    manufacturers already and rejected as unfeasible. But Agassi found a
    promising way to cut through the difficulties (although it has not had
    a chance to be tested yet in everyday use).

    The key to Agassi's thinking, as presented by the authors, was to
    assume that cars had to stop using fossil fuels. The world is
    running out of them. Agassi rejected other options such as hydrogen
    and hybrids, so he decided cars must be electric around the
    same time manufacturers decided they must use gasoline
    (although other options such as ethanol are also being explored).
    Taking electric cars as an axiom, Agassi found all the other
    propositions to make cars work.


The creative role government can play


I'll end by exploring the positive role of government in Israel.
Entrepreneurial government is just as important as entrepreneurial
business--but it must do different things. In this regard the Israeli
experience matches up somewhat with the positive role Nandan Nilekani
proposed in Imagining India, which I href="http://radar.oreilly.com/2009/09/computerization-in-nilekanis-i.html">reviewed
a few months ago.

In the early years, the Labor government kept a firm hand on the
economy. The authors suggest that this worked well in the Israel of
the 1950s and 1960s, an underdeveloped country needing obvious
infrastructure investments. When the country reached a higher level
calling for more complex economic relationships, the socialist bent
became dysfunctional. (Many economists have found similar trajectories
in the Soviet Union and Maoist China.) It was time to retract, so to
speak, to permit more creation.

The authors enthusiastically endorse the Chicago School shock
treatment carried out by the right-wing Likud government that took
over in the 1970s. They don't mention the troublesome income gap
created by this turn to the markets, but it does seem to have
unleashed the innovative capitalism that has made Israelis relatively
affluent for their part of the world, and an attractive location to
work for Jews as well as non-Jews from elsewhere.

The government doesn't make direct investments, but to some extent it
chooses winners--hooking them up to external investors--through the
Yozma program mentioned earlier.

Like all governments, Israel promotes R&D through the military.
But it provides on particularly interesting kind of government
incubator: Talpiot, an intensive multi-disciplinary training program
for promising young recruits.

Talpiot looks over the entire eligible high school population and
aggressively encourages applications from the people it believes will
make the most creative problem-solvers. It puts them through basic
training as well as a technical academic program, and assigns them
quickly to real-life problems. The training goes on for forty-one
months, and the recruits are required to stay in the military for six
more years.

Most Talpiot graduates leave the armed forces when their required six
years are up, and this is seen as a strike against the program by its
critics. But the authors point out that these highly trained, highly
practical experts go on to apply their skills in private industry, so
the government has effectively provided an elite corps for the
country.

Israel certainly faces problems that technology and organizational
savvy cannot solve. But its political, economic, and cultural
successes have made it a model for other developing countries--none
more than the Arab nations, who are trying to emulate it in many ways.
Start-Up Nation is surprisingly broad and deep, for such a
short book. I'm persuaded that it has unraveled many of the mysteries
behind Israel's business success. And it does its best, in a very
uncertain world, to suggest how both Israel and other countries could
replicate that success.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!

Schweinderl