Is QM Spookyness Under Threat

POOR old Einstein. He didn't much like quantum theory, and who can blame him? It's just so... well, peculiar. Nothing in the quantum world exists until you measure it. Certainty melts away. And then there's what Einstein famously called "spooky action at a distance". He didn't like any of it. Yet experiments show quantum mechanics to be the most accurate physics theory in history. Not only does quantum theory make all the right predictions, physicists largely agree that modern experiments, combined with quantum theory's mathematics, leave no room for alternatives. There is no competing theory that banishes the weirdness and embraces a reality that exists independent of our observations of it. The spookiness, it seems, is here to stay.

Or is it? Listen to Joy Christian at the University of Oxford and you may wonder if these grandiose quantum conclusions are really necessary at all. He claims that physicists' supposed proofs of the impossibility of more "realistic" theories rest on false assumptions and so don't prove much at all.

"Contrary to the received wisdom," he says, "quantum theory doesn't rule out the possibility of a deeper theory, even one that might be fully deterministic." Christian's conclusion follows from a relatively simple calculation using alternative mathematics, described in a paper now under review at the journalPhysical Review Letters.

It is a controversial finding, and many theorists disagree with his result, but if Christian is right, perhaps the angst felt by Einstein and others will have been for naught. Whatever the answer, Christian is by no means alone in questioning whether quantum theory is the "final theory" it is often cracked up to be. "A theory that yields 'maybe' as an answer should be recognised as an inaccurate theory," says Gerard 't Hooft of the University of Utrecht in the Netherlands.

For decades, physicists have struggled to unite quantum theory with Einstein's theory of gravity. This difficulty, and lingering inconsistencies at the heart of quantum theory itself, have convinced a growing number of researchers that something deeper is going on behind the scenes - a "pre-quantum" world of certainties and objective realities which, once understood, might reveal how the strange rules of quantum physics emerge from something less strange. A few even think they are starting to see its tantalising outlines.

The proofs of unavoidable quantum weirdness centre on entanglement, the spooky quantum link that Einstein found so distasteful. Entangled pairs of particles such as photons are routinely created in the lab for all kinds of experiments. Send a photon into a "non-linear" crystal and a pair of entangled photons emerge whose characteristics are mysteriously linked. According to quantum theory, it makes no sense to talk about the properties of just one of the entangled photons that appears from the crystal, since all of the information about the photons - such as their "up" or "down" spin states - lies only in their joint properties. Such photons remain connected, even over vast distances.

Quantum theory also asserts that particles have no particular spin before they are observed. Instead, the spin is an indefinite superposition, pointing up and down simultaneously. Only when you measure the spin, do you "force" it into the up or down state.

Do this to an entangled photon and its counterpart responds instantly, even if they are light years apart. If you measure one photon and find its spin pointing "up", you'll find that the other has spin "down". Consequently, quantum theory appears to dictate that what happens in one part of the universe can have instantaneous "non-local" effects in another part, which seems to threaten the basic assumptions of Einstein's special theory of relativity.

Physicists since Einstein have wondered if this peculiarity might only demonstrate a problem with quantum theory itself. Could a better theory account for the linked spins without resorting to this crazy non-locality? One obvious idea is that the particles really do have designated spin all along, but that this is somehow hidden until measured: maybe the spins are created equal and opposite at the moment particles are entangled, and maybe quantum theory just isn't enough to capture these details.

As enticing as this "hidden variables" explanation is, it doesn't seem to work. It was in 1964 that physicist John Bell at Europe's CERN laboratory for particle physics near Geneva, Switzerland, apparently proved that an appeal to such a hypothesis would never bear fruit. Bell imagined an experiment that would send particles from millions of entangled pairs to distant places around the globe, where experimenters would measure their spins. He assumed that some "real", pre-existing properties of the two particles would determine the measurement outcomes. He also assumed that relativity remains intact, so if measurements of entangled particles were made at the same moment, the properties of one particle could not possibly affect its entangled twin quickly enough. From these assumptions, Bell predicted what such experimenters would find.

Remarkably, his results revealed that if his simple assumptions held true then the experimental outcomes would conflict with quantum theory. Many delicate experiments have been carried out since then to test Bell's scenario and, in every case, they have supported quantum theory. Most physicists take these experiments as definitive proof that hidden variables are simply not an option: you must accept that distant, instant influences between particles are possible or give up on any deeper, underlying reality.

Recent experiments have gone further and tried to establish which of the two ideas has to go: locality or realism. They concluded that we have to abandon the idea of an objective reality (New Scientist, 23 June, p 30). All of this rests on the fundamental assumption that Bell's original argument was sound, and most physicists have accepted his conclusions for 40 years. But if Christian is right, they've been overlooking an alternative all this time.

Bell assumed the hidden variables in his argument would be familiar numbers, akin to the value of a velocity or a mass. Such numbers obey the ordinary rules of algebra, including a law that says that the order of multiplication doesn't matter - so that, for example, 2 × 5 equals 5 × 2. This property of multiplication is called commutation. The idea that hidden variables are commuting numbers might seem so basic as to be beyond question, but Christian argues it is important to question this point because mathematicians know that different kinds of variables needn't obey commutative algebra. Take rotations in space, for example. They differ fundamentally from ordinary numbers in one important respect: the order of rotations matters (see Diagram). Rotations do not commute.

In 1843, the Irish mathematician William Rowan Hamilton found a way to capture this non-commuting property in a set of number-like quantities called quaternions. Later, the English mathematician William Clifford generalised Hamilton's quaternions into what modern mathematicians call Clifford algebra, widely considered the best mathematics for representing rotations. So convenient are quaternions that they are commonly used in computer graphics and aviation.

So why is all this important? Christian argues that the existence of this other algebra reveals a weakness at the core of Bell's proof: the only hidden variables Bell considered were ordinary numbers. But ordinary numbers are not the be all and end all. "Why should theorists be obliged to remain unimaginative and consider only commuting numbers in their theories?" Christian says.

Was Bell wrong?

He claims that Bell's argument no longer leads to its impressive conclusion if you allow that hidden variables can have other algebraic properties. Following the logic through, Christian shows that a local, realistic model can actually reproduce everything that quantum theory can. Christian concludes that Bell's theorem is simply not equipped to say whether or not hidden variables are a possible explanation for non-local quantum effects. "When I started out looking at this, it never occurred to me that Bell's theorem might turn out to be wrong," says Christian. "But that's what I found."

Einstein might have been relieved, and it's a shot in the arm for those seeking a deeper reality beyond quantum theory that might be more "reasonable" and akin to classical physics. At this early stage, however, many physicists consider it too good to be true.

Philippe Grangier at the Institute of Optics in Orsay, France, is one of those who believe Christian's claim is unwarranted. "The argument is just too remote from Bell's hypothesis to have anything relevant to say about his theorem," he says. "It might be worth considering as an alternate formulation of quantum mechanics, maybe with a more 'realistic' flavour, but the 'disproof' argument simply makes no sense."

The debate seems likely to continue for some time while researchers puzzle over details. However it turns out, Christian's work reflects a growing willingness among physicists to question whether quantum theory is really the ultimate foundation for theoretical physics. Even those who doubt Christian's conclusion suggest that there's a long way to go to before we truly understand quantum mechanics. "I have no problem thinking that quantum theory is incomplete," says Nicolas Gisin of the University of Geneva.

Twenty years ago, it was heretical even to raise such an idea, but physicists are now questioning quantum theory for a range of reasons. Lee Smolin of the Perimeter Institute in Waterloo, Canada, for one, doubts that physicists can really make headway building a true theory of quantum gravity and space-time before making some serious revisions to quantum theory itself. The inability of theorists to extend quantum theory to the entire universe, he suggests, may imply that it only works for parts of the universe, as an approximation of some deeper reality.

Smolin and Fotini Markopoulou, also at the Perimeter Institute, have been exploring how hints of that deeper theory might emerge from primitive notions of geometry. Their research centres on the concept of loop quantum gravity, in which the smooth picture of space-time is replaced at the Planck scale of around 10^-35 metres by a network of abstract links and nodes. In 2004, the pair found that quantum theory tumbles out as an approximate set of physical laws when you zoom out to the far larger scales of the subatomic world (Physical Review D, vol 70, p 124029). The results are promising, although they stress it's only a start. "There's a lot to fill in before we can say it's a complete model," says Smolin.

't Hooft has been pursuing similar ideas and has proposed that the universe is deterministic at some fundamental level. He suggests that quantum theory may be akin to thermodynamics in the sense that it describes physical systems on average, rather than at a deeper, more detailed level. 't Hooft has constructed various determinsitic theories in which the vacuum of empty space holds the key. To him, the vacuum consists of an enormous class of distinct states that evolve in a deterministic way, details that are ignored by quantum theory. If 't Hooft is right, his idea could explain how randomness arises in quantum theory and why it fails to make specific predictions.

Other researchers, stimulated by long-standing paradoxes of quantum theory, are pursuing experiments that may reveal chinks in it. Markus Aspelmeyer and colleagues at the University of Vienna in Austria have created fragile entangled states between photons and far larger objects, such as small mirrors. Their aim is to explore why we see quantum behaviour on the very small scales, but never seem to find it in everyday objects.

A photon in a quantum superposition can, for example, pass through two narrow slits in a screen at the same time, yet we cannot walk through two doors simultaneously. In practice, researchers assume that superpositions somehow collapse into one specific state or another whenever large objects are involved. For decades, theorists unsatisfied with the vagueness of that explanation have tried to construct a more specific theory. Mathematician Roger Penrose at the University of Oxford suggests that the collapse may be linked to gravity. His idea is that massive objects in a superposition, such as a heavy particle passing through a double slit, would stir up unknown forces that would drive the superposition to collapse rapidly leaving the object in a well-defined place.

Aspelmeyer says that experimental techniques are becoming sensitive enough to test such proposals. His group hopes soon to entangle a mirror weighing less than a microgram with a beam of light in a superposition of states containing different numbers of photons. Upon reflection, the beam would give the mirror a momentum kick and send the mirror into a superposition of positions. This might trigger the kind of collapse predicted by Penrose. Other teams are pursuing similar experiments. "I'd be surprised if one of these groups didn't report some results pretty soon," Aspelmeyer says.

For Stephen Adler at the Institute for Advanced Studies in Princeton, New Jersey, such renewed interest in the deeper foundations of quantum theory is long overdue. For more than two decades, Adler has been quietly developing what he calls "emergent quantum theory" - an idea that builds quantum physics from the bottom up, starting from a hypothetical lower level that obeys classical physics.

Like Smolin and Markopoulou, Adler's scheme assumes a pre-quantum level of physical fields currently unknown to physics. He assumes these have certain basic features and then explores how something like quantum theory might emerge at higher levels. Intriguingly, he's found that it works if his pre-quantum fields have a mixture of both ordinary algebraic properties and non-commuting properties, much like those considered by Christian. What emerges from this hypothetical foundation are basic quantities of quantum field theory, providing a basis for all of quantum theory.

Even more exciting, says Adler, is that fluctuations in his pre-quantum fields lead naturally to the collapse of quantum superpositions. His theory predicts that this will happen all the more rapidly in large objects, just as in the theories of Penrose and others. It is not yet clear whether the assumptions in Adler's research stand up, but the work has been turning heads. "This work is truly ingenious," says physicist Philip Pearle, now retired from Hamilton College in New York. "Is it the long-sought formulation that makes quantum theory understandable? I'd say a definite maybe."

So after decades of physicists bending their minds over the weirdness of the quantum world, it is just possible that its uncertainties and paradoxes may give way to something a little less weird and more definite. Suddenly it's more acceptable to challenge the dogma and to look for a more fundamental, simpler story.

"When I started 30 years ago," Pearle recalls, "almost no one was doing this kind of work. Now lots of people are looking at the standard ideas of physics with new eyes." Surely Einstein would have approved.