Why bad philosophy is stopping progress in physics

Nature seems to have played us for a fool in the past few decades. Much theoretical research in fundamental physics during this time has focused on the search ‘beyond’ our best theories: beyond the standard model of particle physics, beyond the general theory of relativity, beyond quantum theory. But an epochal sequence of experimental results has proved many such speculations unfounded, and confirmed physics that I learnt at school half a century ago. I think physicists are failing to heed the lessons — and that, in turn, is hindering progress in physics.

Take the 2012 discovery of the Higgs boson using the Large Hadron Collider (LHC) at CERN, Europe’s particle-physics laboratory near Geneva, Switzerland. Peter Higgs and François Englert, two of the theorists who had established the underlying theory almost 50 years earlier, shared the 2013 Nobel Prize in Physics for this feat. The Higgs boson was the last particle of the standard model of particle physics to be discovered, and spectacularly confirmed that model, rather than the dozens of theories beyond it. Meanwhile, the apparent absence of evidence for ‘supersymmetric’ particles in LHC data has disappointed a generation of theoretical physicists who had bet on such particles existing, motivated by speculative theories, including string theory.

The expanding Universe — do ongoing tensions leave room for new physics?

Or take the first direct detection of gravitational waves in 2015. This was a spectacular confirmation of Albert Einstein’s century-old theory of gravity, general relativity. The 2017 physics Nobel went to three physicists who had made decisive theoretical and experimental contributions to the discovery.

That same year, the near-simultaneous detection of gravitational and electromagnetic signals from the fusion of two neutron stars (the event GW170817) improved scientists’ knowledge of the ratio between the speeds of gravitational and electromagnetic waves (which general relativity implies should be the same) by some 14 orders of magnitude. In one fell swoop, this excluded a huge domain of theories beyond general relativity. In a similar vein, the 2020 Nobel Prize in Physics was awarded for work showing that the existence and phenomenology of black holes were in complete agreement with general relativity.

And, finally, take the 2022 physics Nobel, awarded for experiments performed over decades that verified phenomena such as quantum entanglement over great distances, falsifying speculation about theories beyond quantum mechanics. These results are often presented as surprising, but in fact they just confirmed what I was taught in school.

And yet, for decades, every new data point in particle physics or astronomy that seemed to be in some slight tension with established ideas has been announced as a hint of physics ‘beyond’ current theories. Large communities of physicists have used such hints as a springboard for speculations, producing a forest of papers in the process. But these new data have always turned out to be explicable by established physics, by statistical fluctuations — or even by experimental error.

You might think that this repeated ‘no’ to wild speculations beyond our best theories would encourage a certain humility in our methodological attitude. Yet I see little evidence for that among many of my fellow theorists, who remain intent on pursuing the next big theory ‘beyond’ those we have today. Why?

Bad digestion

My hunch is that it is at least partly because physicists are bad philosophers. Scientists’ opinions, whether they realize it or not (and whether they like it or not), are imbued with philosophy. And many of my colleagues — especially those who argue that philosophy is irrelevant — have an idea of what science should do that originates in badly digested versions of the work of two twentieth-century philosophers: Karl Popper and Thomas Kuhn.

From Kuhn comes the idea that new scientific theories are not grounded in previous ones: progress instead comes about through ‘paradigm shifts’, the scientific equivalents of revolutions. Popper, meanwhile, supplies the notion that a theory is scientific only if it is ‘falsifiable’: if it can be proved wrong by empirical evidence. Superficial readings of Popper and Kuhn, I think, have encouraged several assumptions that have misled a good deal of research: one, that past knowledge is not a good guide for the future and that new theories must be fished from the sky; and two, that all theories that have not yet been falsified should be considered equally plausible and in equal need of being tested.

The history of science suggests that such attitudes are wrong-headed. It is difficult, if not impossible, to think of a major advance in fundamental physics that has emerged from arbitrary hypotheses. They have instead come from two sources, both empirical.

The first is new data. Johannes Kepler’s insight in the early seventeenth century that the planets move in elliptical, rather than circular, orbits around the Sun, came from observations of anomalies in the movement of Mars. James Clerk Maxwell’s nineteenth-century equations of classical electromagnetism stemmed from earlier experiments by Michael Faraday and others. The founding idea of quantum mechanics came in around 1900 from studies of atomic spectroscopy. Quantum chromodynamics, the theory of the strong nuclear force that is one of the underpinnings of the standard model of particle physics, emerged from attempts to bring order to a zoo of newly discovered ‘hadron’ particles in the 1950s and 1960s. And so on.

The second source of advances is the study of apparent inconsistencies or incoherencies in established knowledge: taking this knowledge seriously, and trying to make it consistent. A celebrated example is Einstein’s 1905 special theory of relativity. Special relativity was born from the observation that Maxwell’s equations account very well for electromagnetic phenomena, but say that they travel at an absolute velocity: the speed of light. This seemed to be incompatible with a fact understood since investigations by Galileo Galilei in the seventeenth century: that velocity is a relative quantity.

Why even physicists still don’t understand quantum theory 100 years on

Following the naive reading of Popper and Kuhn, Einstein should have explored modifications of Maxwell’s equations or forgone the relativity of velocity. He did the opposite. He trusted past scientific knowledge, kept both Maxwell’s equations and the relativity of velocity, and got rid of the apparent contradiction between the two by assuming that both were right and changing something else: the idea that time and simultaneity are absolute. Einstein made the idea that simultaneity depends on how an observer is moving the foundation of a new theory of relativity.

General relativity, which Einstein introduced in 1915, is a product of a similar strategy. It is based on confidence in fundamental knowledge that had already been acquired: special relativity; Newtonian gravity in the relatively weak fields in which it had so far been tested; and Maxwell’s theory as a model for circumventing a problem with Newton’s gravity, that it seemed to act instantaneously over distances. Gravity, like electromagnetism, could also be assumed to have an absolute speed: the speed of light.

In these and similar cases, progress was made mostly without new data. When, in the sixteenth century, Nicolaus Copernicus explored the possibility that Earth spins and moves around the Sun, rather than the Sun around Earth, he had practically nothing more to lean on than anyone else in astronomy had in the previous millennium. Einstein said that the result of the 1887 Michelson–Morley experiment, which did indeed seem to show that the speed of light is absolute, regardless of who is observing it, was not important to him in deriving special relativity. This assertion is credible, because the Michelson–Morley result is actually irrelevant for deriving special relativity (although it is a validation of the theory). Similarly, the ability of general relativity to explain the anomalous movement of Mercury’s perihelion (its point of closest approach to the Sun), which Newtonian gravity could not, worked as a validation of the new theory, but not as a source.

The fabulous case of classical electromagnetism is a little different. Fresh data from Faraday and others were essential, but Maxwell constructed his final synthesis by looking for consistency with previously discovered laws. The new physics was again born from previous knowledge — from taking it seriously as reliable information about reality.