Story

Why Large-Scale Experiments are Important: My Perspective

After conducting psychology experiments on a regular scale for 20 years, I came to the conclusion that it was necessary to transition towards conducting large-scale experiments. The following are the reasons why.

My initial perspective on experimental psychology was that it was still in its early stages, and that once it reached full maturity, it would resemble physics to a great extent. A solid cognitive theory should emulate a physical theory, featuring three key characteristics: simplicity for ease of understanding, breadth for applicability to a wide range of phenomena, and precision for making testable predictions. 

Upon entering the field, I quickly realized that there was a gap between the ideal and the reality. While there are many solid findings that possess both simplicity and precision, they lack the necessary breadth to apply to a wide range of phenomena. Conversely, there are some general concepts that have both simplicity and breadth, but their lack of precision is apparent: different operationalizations of the same concept often render these implementations incomparable.

At the time, I did not perceive any fundamental differences between psychology and physics. I harbored a high level of optimism and believed that, with persistent efforts towards constructing better theories and designing refined experiments, creating a theory similar to that of physics would not pose an insurmountable challenge.

Carrying this optimism forward, I dedicated the majority of my early years to developing a unified theory of attention, modeled after those in physics. In physics, Newton explained the falling of an apple and the trajectory of planets using the unified concept of gravity. If a unified theory could account for one aspect of the human mind such as “attention”, it would be hugely exciting. Along with my mentor, Hal Pashler, we developed a Boolean map theory of visual attention. 

What exactly is a Boolean map? Essentially, it is a data format in which one feature is associated with multiple locations, and we propose that it underlies the content of information that can be seen at a given moment. Looking at the Figure below, it's evident that the disks in the two panels occupy the same set of locations since one can perceive multiple locations in parallel. However, to verify that they contain the same set of colors, one would need to check each color individually.

I started working on the Boolean map theory in 2003, published it in 2007, and carried out further investigations until approximately 2012, spanning almost a decade.

However, I eventually reached a point where I could no longer proceed.

The main difficulty lay in two areas. The first was the lack of precision. Like all concept-based theories in psychology, the theory was unavoidably vague, which made it unclear what predictions it would make in a different situation than the one in which it was originally developed. Theoretically, a well-defined concept can be objectively assessed. However, in reality, even the most classic concepts have rarely achieved this ideal objectivity. This is evident in numerous debates where neither side can persuade the other using logical grounds and experimental findings.

By the time Boolean map theory was published in 2007, I was already fully aware of the problem of theoretical vagueness and I was working hard to address it. However, at the time, I remained optimistic and did not anticipate the second, and more fatal problem - the complexity of the data.

In 2008, I told a friend that I intended to spend the next five years conducting a series of studies aimed at concretizing and quantifying the concepts involved in the theory. My goal was to produce a clear set of evidence that would either support the theory or form the basis for a new one. I estimated a 30% chance of the theory being supported, and regardless of the outcome, I believed that a unified theory capable of answering all relevant questions would emerge.

My friend smiled and replied, "What if there isn't a consistent theory that emerges from this evidence?"

To this, I replied, "I consider that outcome to be unlikely. Truth is always grounded in a set of principles, and with a precise method of examination, these principles can be revealed."

In hindsight, I realize how naïve I was back then. By 2012, most of the planned experiments had been completed and their results were already evident. Once the variables were precisely quantified, it became clear that no theory could completely account for the wide-ranging results. Neither my theory nor any of the existing ones seemed to work, and I couldn't come up with any new theories that would be effective.

The result was a major disappointment for me, and I spent years in a daze, contemplating it. After careful consideration, I have come to the realization that my pursuit of a simple theory may have been misguided from the start. The human mind is incredibly complex, and even a relatively narrow aspect of it requires a set of intricate principles to be adequately described.

Naturally, this is not to suggest that the principle of parsimony is incorrect. When two theories offer equally good explanations for the data, the simpler theory should be favored. However, it is imperative that the complexity of a theory aligns with the underlying mechanisms it aims to explain. If these mechanisms are complex, the theory must also be complex. 

When a theory aims to elucidate complex mechanisms, which is often the case in psychology, it encounters an impossible trinity concerning three criteria: simplicity, precision, and breadth. A theory can achieve any two of the three criteria, but it cannot fulfill all three at once. 

After realizing that my original approach was fundamentally unworkable, my enthusiasm waned considerably. However, after some time feeling down, I did not want to abandon my efforts entirely but instead wanted to attempt to develop a more intricate theory that can be both broad and precise.

When working towards developing a complex theory, the main challenge is the gap between the limited amount of available information from experiments and the vast amount of information required to elucidate the complex principles. In order to bridge this gap, I opted to expand the pool of available information by gathering additional data, while simultaneously narrowing down the scope to minimize the amount of required information. All in all, I deemed it optimal to undertake a "systematic exploration of a medium-sized field". 

This involved identifying a research area of medium size, standardizing the existing studies into a unified experimental method, and then repeating the experiments to summarize the rules. I had initially planned around 10 projects, some of which were abandoned after a brief attempt. Ultimately, I was able to complete three of them, but only one produced definitive results, which were subsequently published as the FVS framework.

So, what is in this FVS framework? Researchers utilize various types of stimuli and tasks in the field of attention and perception. However, the relationships between these different stimuli types and tasks remain poorly understood. The FVS framework aims to shed light on both of these aspects (16 stimulus types × 26 tasks) through a sample size of 1744 participants, which is considered large within the standards of experimental psychology. This substantial sample size provides a reasonably clear picture of both of these aspects.

Despite the progress made in the case of the FVS framework, I found that the systematic exploration of a medium-sized field was not as effective overall as I had hoped. Although there were several issues, I believe the underlying reason for all of them was still the inadequacy of data. In other words, while I have made some progress in narrowing the gap between available and required information, I must take more drastic steps to fully close the gap. 

We need more data! A lot of high-quality data specifically designed for research purposes!

As I gradually came to this realization, I coincidentally stumbled upon a similar narrative in physics. It is widely known that Newton is credited with discovering gravity, and many of us are also aware that Kepler formulated the laws of planetary motion prior to Newton. However, what is often overlooked is the crucial role played by another individual, Tycho Brahe, who supplied Kepler with the necessary data. Until then, my attention had been fixated on theorists like Newton and Kepler, with little regard for Tycho Brahe's contribution.

But this time, I came across a statement attributed to Tycho Brahe that caught my attention:

"I have studied all the existing star charts, but none of them agrees with another. There are as many methods for measuring the positions of the celestial bodies as there are astronomers, and those astronomers all disagree with each other. What is needed now is a long-term plan to measure the entire celestial sphere from one location."

Doesn't this resemble the current state of experimental psychology? Tycho Brahe's vast collection of observational data formed the basis for transforming physics from a scattered set of observations to a systematic theoretical framework. Perhaps a similar approach is needed in psychology to build a solid foundation for the field.

This paragraph marked a pivotal moment for me. Although there are certainly benefits to conducting experiments on a regular scale, such as elegance and ease of communication, I have found it necessary to shift my focus towards larger-scale experiments as they provide a more accurate reflection of the truth. Starting in 2020, I dedicated all my efforts to conducting large-scale experiments.

Before undertaking large-scale experiments, I was unaware of anyone else pursuing this type of work. My fear of being different from everyone else played a role in my initial hesitation. However, in recent years, it seems that many others have started moving in the same direction (e.g., Awad, et al., 2018; Peterson, Bourgin, Agrawal, Reichman, & Griffiths, 2021; Agrawal, Peterson, & Griffiths, 2020; see Almaatouq, et al 2023, for a Behavioral and Brain Sciences article).

In a large-scale experiment, we study a topic using data sets that are hundreds of times larger than usual. However, what is more significant is how the experience of conducting research has changed. During my 20 years of running conventional psychological experiments, I always felt like a detective holding a glass of water, searching for clues. Even though it was sometimes easier, I still needed to deliberately look for clues and interpretations. With large-scale experiments, the experience is entirely different. When I began data analysis, a massive amount of information rushed at me like a hurricane, leaving me breathless. The contrast is truly striking.

Will it work this time? Honestly, I am not sure. At present, I believe that this is the right direction, but we have only finished one piece of the puzzle in the whole picture, and I haven't seen any immediate indication of a unified theory on the horizon. Nonetheless, we should strive to pursue endeavors that we deem meaningful, regardless of the seemingly low probability of success. So, I will give it a try anyway.


Liqiang Huang

Last updated on March 31, 2023