In the 1930s, Einstein urged theoreticians to broaden the scope of their most basic laws of nature using a method he believed had played a crucial part in helping him deliver his new theory of gravity. He urged his colleagues to focus on the mathematics underlying modern theories, rather than to take inspiration from new experimental discoveries. Einstein believed that physicists could build on the mathematical frameworks of their best theories in ‘natural’ ways.

Independently, Paul Dirac proposed a similar approach – theoreticians should above all seek mathematical beauty when looking for new theories that describe an ever-larger part of nature. For several decades, the mathematical approaches proposed by Einstein and Dirac made little or no impact among their peers.

2. Rock-solid foundations

Most physicists believe that any successful basic theory of nature must incorporate the two foundational theories of twentieth-century physics, quantum mechanics and the basic (special) theory of relativity. Neither has ever been contradicted by experiment. The problem for theoreticians is that the two theories are based on different mathematics, and it is extremely difficult to jam them together. Physicists were able to arrange this, however, in modern quantum field theory. A classic example of a successful quantum field theory is the supremely successful Standard Model of particle physics, which gives an excellent description of all the sub-atomic forces and particles.

Physicists have discovered that combining quantum mechanics and special relativity leads to a Klondike of new mathematics and a slew of new physical ideas. These attempts to combine the two foundational theories have triggered many of the innovations that have led physicists and pure mathematicians to work on topics that are similar or even identical.

Scientists have long sought a theory that unified our understanding of gravity, which shapes the cosmos, and the three forces mainly responsible for the inner working of atoms. Such a unification seemed to be a long way off until the 1980s, when many leading physicists began to develop what some of them (misleadingly) called a ‘theory of everything’. Such a theory must reproduce the successes of Einstein’s (non-quantum) theory of gravity and the Standard Model, the well-established quantum theory of sub-atomic forces. Even with precious few new clues from sub-atomic experiments, theorists developed new mathematical structures for a candidate unified theory. By far the most popular of these candidates was (and is) the string framework, which is consistent with both special relativity and quantum mechanics.

Perhaps the best reason for believing in the potential of the string framework is that it naturally explains why gravity must exist (by contrast, ordinary field theory cannot be reconciled with the existence of gravity). Alas, the theory has yet to make a distinctive, correct prediction about the natural world. Nevertheless, many leading theorists persist enthusiastically with it, mainly because of its internal consistency, its potential and its applications in other parts of science. Many physicists are also encouraged by the many remarkable insights into contemporary mathematics the theory is continuing to supply.

In Dirac’s 1939 lecture ‘On the Relation Between Maths and Physics’, he wrote that ‘as time goes on, it became increasingly clear that the rules that the mathematician finds interesting are the same as those that Nature has chosen.’ In recent decades, these words have begun to look remarkably far-sighted. Not only is mathematics ‘unreasonably effective’ in physics, as Eugene Wigner famously observed, the opposite is also true: physics is unreasonably effective in mathematics. Could this two-sided unreasonableness be leading us to a unified understanding of physics and mathematics, as Dirac proposed?

While theorists await guidance from new experimental information, many are investigating how new mathematics can help us to understand the order at the heart of the universe. As the theorist Nima Arkani-Hamed says, ‘the universe speaks to us not only through experiments, but also through mathematics.’ These investigations, based squarely on quantum mechanics and special relativity, are taking theoreticians to extraordinary places and leading them to question many of our long-held beliefs about the fundamental nature of the universe. Many leading theoretical physicists now believe, based mainly on pure thought, that our current best understanding of space and time is doomed.

Despite the allegations that fundamental theoretical physics is in a woeful state, Graham believes that such pessimism is unjustified: theorists are not lost. Rather, physicists are slowly but surely making progress towards the structure of a fundamental, unified theory of the universe. As Arkani-Hamed says: ‘There has never been a better, more exciting time to be a theoretical physicist.’

Graham began to conceive of ‘The Universe Speaks in Numbers’ in 2013. Initially, he intended to write a brief and simple account of the mysterious relationship between physics and mathematics. It was only after talking with dozens of leading physicists, historians and philosophers that he changed his mind and decided to present the modern mathematics-physics relationship in a broader context. As he says: ‘I want to give some idea of how so many top-class physicists and mathematicians have been led – often to their amazement – to work together.

Above all, Graham says, he wants to give a sense of the excitement leading physicists are experiencing as they are advance towards one of the noblest – and least narcissistic – goals of humanity: to understand what Stephen Hawking described as ‘the mind of God’.