Criteria for Assessing Theories

Post date: Sep 06, 2017 5:20:46 PM

What is a Theory?

A theory is an explanation of something. It is typically an explanation of a class of phenomena, rather than a single specific event. Instead of explaining why there is a brown stain on my tie, a theory would explain why men's ties often have brown stains.

Theories are often expressed as chains of causality: this happens because this and that happened just when something else happened and this in turn happened because ... you get the idea!

Theories are sometimes confused with hypotheses, because both seem to consist of statements relating one variable to another. And it's true that some published theories are little more than hypotheses. But good theories are a bit different. Here are some of the differences:

• • theories are more general

  • theories explain why things are related, whereas hypotheses just say they are related

  • theories generate hypotheses; hypotheses are implicit in theories: given a theory, you can deduce hypotheses

  • hypotheses are used to test theories

As discussed in the next section, one way that theories explain is by providing a sense of process or mechanism for how one thing is related to another. This is very important.

Having a sense of process is one attribute or characteristic of a good theory. There are many characteristics that make a theory good. It is not just whether the theory is correct or not. In fact, the correctness of a theory is a very complicated issue, and is not quite as important as you might think.

Correctness of Theories

Unfortunately, we can never prove a theory right. We can (more or less) prove it wrong, but can never prove it right. There are two reasons for this. First, it doesn't matter how many times you test a theory, there is not enough time in the universe to do all possible tests. So even if a theory has survived 100 tests, it could still fail the 101st test. In a way, the situation is the opposite of locating a missing object in a house. If you search for the object in the house and find it, well, it's definite that the object was in the house -- case closed. But if you search and don't find it, that doesn't absolutely mean that the object is not in the house. It could still be there, you just missed it. The same (well, the opposite) is true of theories. If you test a theory and it fails, that's it: it's been disproved [editor's note: not really. maybe the test was faulty]. But if you test it and it passes, that's just one test. There may be other data out there, or other situations, that will disprove it -- you just haven't gotten to them yet.

The second reason you can't prove a theory true is that there is never just one theory that fits the facts. A theory is really just a narrative. A tale that explains. But stories can be told very differently. In a sense, there are always an infinite number of theories that fit the facts. Think for example of Newtonian theories for the motion of bodies -- equations like f = ma. Those theories served us very well for a very long time. But now, we have replaced Newtonian physics with a whole new theory brought to light by Einstein. Was Newton wrong? Not exactly. His theories were correct as far as they went. They predicted the motion of bodies quite well: well, enough, for example to build airplanes that actually fly. Engineers still use Newton's theories to build certain things. But for other things, today we use entirely different equations built on a completely different understanding of the physical universe to do exactly the same thing. The new theory explains additional phenomena that the old theory didn't -- for example, according to Newtonian theory, objects should not change mass as they approach the speed of light (which they do), nor should time slow down.

Good Theories

Good theories have a number of important characteristics, including:

• • mechanism or process

  • generality

  • truth

  • falsifiability

  • simplicity

  • fertility

  • surprise

Mechanism (or Process). A good theory has a sense of movement, a dynamic element. The feeling of understanding that a good theory gives is due mostly to having a sense of process by which one state of affairs leads to another. For example, suppose athletes tend to ask dumb questions in class. A bad theory explains this very simply: athletes are dumb. This is a bad theory on many counts, but one problem is that there is no sense of process by which the quality of the mind is linked to the stupidity of the question. What is the mechanism by which the questions are formed, and how is mind quality related? Contrast this with a much better theory: that athletes have to spend a lot of their time practicing and going to games, and so have less time to study. This has a sense of process: there is only so much time in a day, the more time is spent on sports, the less there is available for study, the less studying the less they will understand what's going on in class, and therefore the less cogent their questions will be. This theory is a chain of causal links, each one small enough that we can readily believe it.

Here's another example. Why do some people steal, hurt people, and spend their lives going in and out of jail? A common type of answer is something bad happened to them as children, or they had bad or missing parents (the old "came from a broken home" idea). We tend to think that whatever bad happens, it is due to something bad. But what exactly is the mechanism by which something bad happening as a child causes them to do bad things themselves? What is it about the way the brain works that one bad thing leads to another bad thing? That's the part we need to specify in order to have a good theory.

Bad theories often just give a name to the cause of something, without actually explaining anything. We are often fooled into accepting these theories because it's been given a name, which makes it seem real or credible. It's like a doctor saying, the reason you are coughing so much and have trouble breathing sometimes is because you have COPD. So you go an look up COPD and discover that it is an administrative diagnosis. Anyone who has two or three of a list of respiratory problems (like asthma, chronic bronchitis, emphysema, etc) is categorized as having COPD. For a different example, suppose we observe that some workers work harder than others. What's the reason for that? Some people will say "motivation". They are motivated. Motivation is an inner drive to do something. But does it really explain anything or does it really just restate the observation? Knowing that working harder is caused by motivation doesn't seem to help us understand anything. It really just brings up the question 'why are some workers more motivated than others?'. In general, explaining things in terms of drives and instincts tends not be very satisfying. 

In general, essentialist theories lack process. Why are women more likely to bring cookies to business meetings? Essentialist answer: women are more nurturing by nature. Why do people ask dumb questions? Essentialist answer: They are dumb people. 

Generality. Good theories are general enough to be applicable to a wide range of individual events, people or situations. Consider the theory that athletes ask dumb questions in our classes because they spend so much time on athletic stuff that they don't have time for school. This is general because it should work for all athletes, not just local ones, and not just for one sport. Furthermore, it can really be applied to any person who has a serious time commitment outside of class, such as musicians. The basic idea of the theory -- the mechanism -- is that people with significant time commitments in other areas will perform less well on the area in question.

Truth. Unfortunately, theories can never been proved to be true. There are two reasons for this. 1) No matter how thoroughly we test the theory against data, there is always the possibility that tomorrow there will be some data that contradicts the theory. Just because the sun has risen everyday since we started checking, doesn't prove a theory that suggests that it will always rise. 2) Theories are just descriptions. There are always other ways to describe the facts that are equally valid. In this sense, truth is not a reasonable concept. All that is available to us is descriptions that are not contradicted by the currently available facts.

Falsifiability. A good theory is falsifiable. If there is no conceivable way to construct an experiment or collect some data that could potentially contradict the theory, the theory is worthless. Suppose you are trying to explain the pattern of heads and tails that come up when you flip a coin 10 times. Your theory is that it comes up heads when an invisible magician wants it to, and tails otherwise. How do you test the theory? If you flip the coin and it comes out heads, that does not contradict the theory because it just means that the magician wanted it to be heads. If you flip the coin and it comes out tails, that does not contradict the theory either, because it just means the magician wanted it to be tails. No matter how the experiment turns out, the data cannot possibly contradict the theory.

Theories like this do not really explain anything. You can't use them to predict outcomes, nor to do things (e.g., to build airplanes that actually fly). A lot of psychological theory comes very close to being non-falsifiable. For example, the general concept that employees in an organization work hard because of something called "motivation", is kind of like saying the coin comes up heads because a magician wants it that way. We can't see motivation. We can only infer its existence by its effects (human behavior). So if a person works hard, we say they were motivated. If they don't work, we say they weren't motivated. Yet we say the reason they work hard or not is because of motivation. This is circular: if they are motivated, then they work hard. If they work hard, they are motivated.

In general, any theory that explains human behavior in terms of human desires is treading on thin ice. In other words, if you study voluntary turnover in organizations and find that people leave organizations because they want to, you haven't really explained anything, and you could never be proven wrong. 

To summarize, there are two ways that theories can fail to be falsifiable: (a) because the data are impossible by their very nature to collect, or (b) because they are circular.

Parsimony refers to the simplicity of a theory -- the avoidance of positing complex relationships when a simpler alternative exists. One reason for preferring parsimony is that nature seems to. Complicated things have more ways of not happening, and less likelihood, therefore, to endure to the present. The other reason is that theories are useless unless they are simple enough for people to understand. Theories are sometimes called models, and the whole idea of a model is that it is a smaller, simpler version of the real thing. Models are meant to pull out the important parts, and leave the unimportant behind. The power of a model can be defined as the proportion reduction in complexity that it affords over nature. Too much detail can obscure the key things. Really complicated models don't actually explain much. The best model possible of the Earth's weather patterns would be obtained by constructing a duplicate Earth and surrounding solar system, exactly the same in every detail. Presumably, it would predict very well. The problem is, the model is as complicated as the thing we were trying to understand in the first place.

An example of parsimony is chance models. Suppose we want to understand why almost all human societies have significant inequality -- that is some people are much richer than others. We could posit a number of special reasons, including supernatural causes like "God wants it that way", but it is important to realize that inequality is what we would expect even if there were no special reasons why it should happen. If we take 100 coconuts and divide them randomly among 10 people, there are only a handful of ways it could come out that would be approximately equal: but there are about 1030 ways to divide them so that there is significant inequality. It's just like keeping your bedroom neat: there is basically one way of distributing all your stuff in the 3-dimensional space you call your room such that you would say 'everything is in its place'. But there are millions of ways that stuff can be arranged such that you would say 'the place is a mess'.

The principle of using parsimony as a criterion for model selection is known as Occam's Razor.

Fertility. A fertile theory is one that generates lots of implications in different areas. Implications are important because (a) they are essentially insights that were not obvious prior to stating the theory, so they represent potentially new knowledge, and (b) they represent possibilities for testing the theory.

To be fertile, a theory pretty much has to be general.

Surprise. A quality of good theories is that they are interesting. This means that they generate non-obvious implications. They lead you to understand things in new ways. Surprise refers to the theory's ability to make non-obvious, unexpected predictions. An example is a theory that explains why certain countries have so many more girls than boys. The theory says that this happens, ironically, where people prefer boys. You see, the probability of having a boy is different in different families -- it's a genetic thing. Now, suppose what people do is keep having babies until they've got more boys than girls, or they have run out of room. So if the first baby is a boy, they stop there. If the first baby is a girl, they have two more kids. If both are boys, they stop there. But if one's a girl, they keep going. The result is that families that have a predisposition to have boys, tend to have small families -- if the first kid's a boy, the stop there. But families that have a predisposition to have girls have enormous families, as they keep trying to get boys. If there were no preference between boys and girls, then there would be no relationship between number of kids and the sex of the kids: large families would be just as likely to contain boys as small families.

This paradoxical result is fun, interesting.

Process of Theorizing

Start with an observation, such as "white people and black people sit at different tables in the cafeteria". Then create an initial explanation. For example, you might try out the idea that people prefer to eat lunch with their own kind.

Now think about your explanation in terms of the qualities of good theory, and try to make it better. For example, to make the theory more general, change "eat lunch with" to the more general "socialize with". Then check the other criteria. One problem with this particular theory is that it lacks a sense of process -- how does it happen that people prefer their own kind? Because it has no sense of process, this theory is little more than a restatement of the initial observation. It's also hard to test. It basically says: people sit at different tables because they want to. So if some people don't sit with their own kind, it must be because they didn't want to -- they didn't have that preference for their own kind. Another problem with this theory is that it is not fertile. It does not generate interesting implications. The best you could do is predict that, at  other social events, blacks and whites will self-segregate.

A model with a little more sense of process and explanation is: "People tend to do what they've done before. So if whites grew up socializing with whites, then they will continue to socialize with whites, and the same for blacks. People's earliest experience is with their families, who are typically the same color as they are." This theory generates implications much more readily. For example, it suggests that kids adopted at a young age by families of a different color will prefer to socialize with people of that color, rather than their own. It also implies that kids growing up in racially mixed school systems should not show as much preference for their own kind.

Theorizing is an iterative process of creation, criticism, and re-creation. It is also an art. Good theories are beautiful, and the process of creating this beauty is what art is all about.

For more detail on the process of theorizing, click here.