Beware the Rareness Illusion When Exploring the Unknown

Here's a great vacation idea. Spend the summer roaming the world in search of the 10 lost tribes of Israel, exiled from Samaria by the Assyrians 2700 years ago (2 Kings 17:6). Or perhaps you'd like to search for Prester John, the virtuous ruler of a kingdom lost in the Orient? Or would you rather trace the gold-laden kingdom of Ophir (1 Kings 9:28)? Or do you prefer the excitement of tracking the Amazons, that nation of female warriors? Or perhaps the naval power mentioned by Plato, operating from the island of Atlantis? Or how about unicorns, or the fountain of eternal youth? The Unknown is so vast that the possibilities are endless.

Maybe you don't believe in unicorns. But Plato evidently "knew" about the island of Atlantis. The conquest of Israel is known from Assyrian archeology and from the Bible. That you've never seen a Reubenite or a Naphtalite (or a unicorn) means that they don't exist?

It is true that when something really does not exist, one might spend a long time futilely looking for it. Many people have spent enormous energy searching for lost tribes, lost gold, and lost kingdoms. Why is it so difficult to decide that what you're looking for really isn't there? The answer, ironically, is that the world has endless possibilities for discovery and surprise.

Let's skip vacation plans and consider some real-life searches. How long should you (or the Libyans) look for Muammar Qaddafi? If he's not in the town of Surt, maybe he's Bani Walid, or Algeria, or Timbuktu? How do you decide he cannot be found? Maybe he was pulverized by a NATO bomb. It's urgent to find the suicide bomber in the crowded bus station before it's too late - if he's really there. You'd like to discover a cure for AIDS, or a method to halt the rising global temperature, or a golden investment opportunity in an emerging market, or a proof of the parallel postulate of Euclidean geometry.

Let's focus our question. Suppose you are looking for something, and so far you have only "negative" evidence: it's not here, it's not there, it's not anywhere you've looked. Why is it so difficult to decide, conclusively and confidently, that it simply does not exist?

This question is linked to a different question: how to make the decision that "it" (whatever it is) does not exist. We will focus on the "why" question, and leave the "how" question to students of decision theories such as statistics, fuzzy logic, possibility theory, Dempster-Shafer theory and info-gap theory. (If you're interested in an info-gap application to statistics, here is an example.)

Answers to the "why" question can be found in several domains.

Psychology provides some answers. People can be very goal oriented, stubborn, and persistent. Marco Polo didn't get to China on a 10-hour plane flight. The round trip took him 24 years, and he didn't travel business class.

Ideology is a very strong motivator. When people believe something strongly, it is easy for them to ignore evidence to the contrary. Furthermore, for some people, the search itself is valued more than the putative goal.

The answer to the "why" question that I will focus on is found by contemplating The Endless Unknown. It is so vast, so unstructured, so, well ..., unknown, that we cannot calibrate our negative evidence to decide that whatever we're looking for just ain't there.

I'll tell a true story.

I was born in the US and my wife was born in Israel, but our life-paths crossed, so to speak, before we were born. She had a friend whose father was from Europe and lived for a while - before the friend was born - with a cousin of his in my home town. This cousin was - years later - my 3rd grade teacher. My school teacher was my future wife's friend's father's cousin.

Amazing coincidence. This convoluted sequence of events is certainly rare. How many of you can tell the very same story? But wait a minute. This convoluted string of events could have evolved in many many different ways, each of which would have been an equally amazing coincidence. The number of similar possible paths is namelessly enormous, uncountably humongous. In other words, potential "rare" events are very numerous. Now that sounds like a contradiction (we're getting close to some of Zeno's paradoxes, and Aristotle thought Zeno was crazy). It is not a contradiction; it is only a "rareness illusion" (something like an optical illusion). The specific event sequence in my story is unique, which is the ultimate rarity. We view this sequence as an amazing coincidence because we cannot assess the number of similar sequences. Surprising strings of events occur not infrequently because the number of possible surprising strings is so unimaginably vast. The rareness illusion is the impression of rareness arising from our necessary ignorance of the vast unknown. "Necessary" because, by definition, we cannot know what is unknown. "Vast" because the world is so rich in possibilities.

The rareness illusion is a false impression, a mistake. For instance, it leads people to wrongly goggle at strings of events - rare in themselves - even though "rare events" are numerous and "amazing coincidences" occur all the time. An appreciation of the richness and boundlessness of the Unknown is an antidote for the rareness illusion.

Recognition of the rareness illusion is the key to understanding why it is so difficult to confidently decide, based on negative evidence, that what you're looking for simply does not exist.

One might be inclined to reason as follows. If you're looking for something, then look very thoroughly, and if you don't find it, then it's not there. That is usually sound and sensible advice, and often "looking thoroughly" will lead to discovery.

However, the number of ways that we could overlook something that really is there is enormous. It is thus very difficult to confidently conclude that the search was thorough and that the object cannot be found. Take the case of your missing house keys. They dropped from your pocket in the car, or on the sidewalk and somebody picked them up, or you left them in the lock when you left the house, or or or .... Familiarity with the rareness illusion makes it very difficult to decide that you have searched thoroughly. If you think that the only contingencies not yet explored are too exotic to be relevant (a raven snatched them while you were daydreaming about that enchanting new employee), then think again, because you've been blinded by a rareness illusion. The number of such possibilities is so vastly unfathomable that you cannot confidently say that all of them are collectively negligible. Recognition of the rareness illusion prevents you from confidently concluding that what you are seeking simply does not exist.

Many quantitative tools grapple with the rareness illusion. We mentioned some decision theories earlier. But because the rareness illusion derives from our necessary ignorance of the vast unknown, one must always beware.

Looking for an exciting vacation? The Endless Unknown is the place to go.