The quest for good explanations

For most of human history, the rate of progress was so slow as to be unnoticeable on the timescale of an individual’s life: the gaps between significant innovations or increases in knowledge were typically measured in centuries, if not millennia. But since the Enlightenment and the Scientific Revolution, we have been in an unprecedented era of rapid and continually accelerating progress, an explosion of knowledge. In his book The Beginning of Infinity, David Deutsch tries to put his finger on the key factor that makes the difference between no-progress and progress.

A central tenet of the Enlightenment was a rejection of authority with regard to knowledge. But rejection of authority on its own is not enough: we need an alternate (better) way of seeking knowledge, without relying on authority. The Scientific Revolution initiated a sustained tradition of criticism, which came to require that a scientific theory be testable: “the theory must make predictions which, if the theory were false, could be contradicted by the outcome of some possible observation.”

Yet testability alone is not sufficient for scientific progress: the prophet who claims the world will end tomorrow, and the gambler who feels his winning streak will continue, are making eminently testable predictions. Testability is not enough because the purpose of science is not mere prediction. In order to solve problems you need explanations: “assertions about what is out there and how it behaves.” A scientific theory says something about the reality that accounts for our observations.

But having a testable, explanatory theory is still not enough. Consider mythical explanations, like the idea that disease is the gods’ punishment for sinners. Such myths do make a testable claim about the underlying reality, but they are nevertheless unscientific (that is, useless for improving our understanding of the world) because they are bad explanations: they are easy to invent and easy to adapt such as to explain anything. If a myth’s predictions fail (a righteous person falls ill), the myth can simply be tweaked (the gods sometimes test the faith of the righteous), or replaced with a different myth (the devil, witchcraft) — and no progress is ever made, no knowledge created. The details of a myth are only loosely connected to the particulars of the phenomena it purports to explain, so there’s no way to judge the relative merits of competing myths and no reason to prefer one over another: at bottom they are all equivalent to “the gods did it.” But an explanation that could easily explain anything actually explains nothing.

Good explanations (like the germ theory of disease), on the other hand, are hard to discover and hard to vary once discovered: the details of a good explanation are intimately connected to the specifics of the phenomena being explained, and every detail plays a functional role. If a previously good explanation is refuted by experiment, it ceases to be a good explanation, and must be dropped; but theories that have passed many stringent tests and are highly constrained by other existing knowledge become extremely good explanations, and our scientific knowledge grows. Seeking good explanations, according to Deutsch, is the key to the scientific frame of mind — which unlocks potentially unlimited progress:

As the physicist Richard Feynman said, ‘Science is what we have learned about how to keep from fooling ourselves.’ By adopting easily variable explanations, the gambler and prophet are ensuring that they will be able to continue fooling themselves no matter what happens. Just as thoroughly as if they had adopted untestable theories, they are insulating themselves from facing evidence that they are mistaken about what is really there in the physical world.

The quest for good explanations is, I believe, the basic regulating principle not only of science, but of the Enlightenment generally. It is the feature that distinguishes those approaches to knowledge from all others, and it implies all those other conditions for scientific progress…

Old ways of thought, which did not seek good explanations, permitted no process such as science for correcting errors and misconceptions. Improvements happened so rarely that most people never experienced one. Ideas were static for long periods… The emergence of science, and more broadly what I am calling the Enlightenment, was the beginning of the end of such static, parochial systems of ideas. It initiated the present era in human history, unique for its sustained, rapid creation of knowledge with ever-increasing reach.