31 Human Reactivity

(Modified on 2026-08-11)
3,486 words

It is to the highest degree probable that the subject’s…general attitude of mind is that of ready complacency and cheerful willingness to assist the investigator in every possible way by reporting to him those very things which he is most eager to find, and that the very questions of the experimenter…suggest the shade of reply expected.

— Pierce (1908)

An omnipresent validity issue for human measures is bias—ways in which the human participants (or researchers) can systematically distort the measurement, either intentionally or accidentally. Human measures are all extremely reactive: our physiology, behavior, and thoughts are affected by being measured. Wearing an EEG cap; being filmed; being interviewed; even just entering a room called a “laboratory”—all of these things affect how a person behaves. Even relatively unobtrusive measurement in non-laboratory settings can affect people. For example, if you set up mics to record a rehearsal for research purposes, the musicians won’t play or act quite the same way they would when nobody is recording them.

Response bias is generally a bigger problem for self-report and behavioral measures, compared to physiological measures. However, even physiological measures can be reactive. A classic example is the polygraph—or so called “lie-detector”—test, which is actually based on a topical measure discussed above, galvanic skin response (GSR). Polygraph tests have many validity problems (they aren’t considered valid evidence in any court!): Many people, even when being honest, react to being polygraphed by getting nervous, which can appear (physiologically) just like deceptive nervousness. Worse, some people have shown that they can fool a polygraph by consciously manipulating their own metabolic state—for example, by pinching themselves to feel pain, or thinking of frightening or relaxing thoughts. In other words, GSR is not a good operational definition for “lying.”

Much of the measurement reactivity we see with human participants is pretty random—which means it might still “average out” without affecting our result. However, some measurement reactivity can definitely introduce systematic bias, which we’ll call response bias. This can mean participants consciously, or unconsciously, changing their responses or behaviors in response to the measurement process. The most significant, and omnipresent, problems with human research are social-desirability bias and demand characteristics.

Demand Characteristics

Demand characteristics arise when a participant specifically reacts to (what they believe to be) the ideas of your research. Any human research participant will tend to form their own idea about what is going on in a study. They may guess what the experiment is about, and then choose to respond in a way that corresponds to whether they agree or disagree with what they think is the purpose. Demand characteristics often add undesirable confounds to a study, so it is important to be able to identify and minimize possible demand characteristics.

Demand characteristics do not merely arise because a participant thinks about the experiment. Demand characteristics can also be entirely unconscious. One of the most powerful examples of a demand characteristic is the placebo effect. The simple expectation that the placebo will improve one’s situation can lead to the placebo actually improving one’s situation.

Cooperation…or not

Cooperation bias is the most dangerous form of demand characteristics.

Experimental participants are often eager to be perceived by researchers as cooperative (Orne 1962). Thus, participants are apt to respond according to what they believe the experimenter wants to hear. This is known as cooperation bias. Cooperation bias is the most dangerous form of demand characteristics because it may lead us to make Type 1 errors: (false) claims that our hypothesis is supported. Research suggests that cooperation bias differs between cultures: For example, research suggests that people from Asian cultures tend to behave in a more cooperative manner than people from American culture (Nisbett 2003).

Apart from attempting to please the experimenter, people often respond in ways that favor positive rather than negative responses. This is referred to as acquiescence bias. When asked whether they agree or disagree with various statements, some respondents will favor the “yes” response over the “no” response, even if it causes them to behave in a contradictory fashion. For example, at different points in a long questionnaire, a person may agree with both of the following statements (Messick and Jackson 1961):

I prefer a cup of coffee to a glass of water.
I prefer a glass of water to a cup of coffee.

Demand characteristics are not always positive. Some participants may be hostile to research ideas or hypotheses and may change their behavior to try to disprove (what they believe to be) the hypothesis.

Imagine that we were interested in studying how the definition of “vocal quality” is different between rappers and singers. Perhaps we even hypothesize something specific, like that a “good singing voice” has a lower spectral centroid and higher harmonicity than a “good rapping voice.” To test our hypothesis, we have participants listen to a variety of R&B and hip-hop and ask them to rate the quality of the vocals.

Here’s the danger: Some participants might think that we are trying to prove that rap is “not really music.” If they are fans of hip hop, they might be hostile and decide to mark all the rapped vocals as “good voices,” and not the sung vocals. In contrast, other participants might be prejudiced against rap, and they might label all the sung vocals as high quality and the rapped vocals as low quality. Either, way the problem is not that the participants are cooperative or antagonistic, but that they aren’t responding naturally the way we want them to. The thing we want to measure (vocal quality) is not what we are really measuring (attitude towards rap), so the validity of our operational measure is questionable.

A Story (David Huron)

In 1990, I carried out an experiment to test whether listeners are prejudiced against women composers. I expected that people would tend to view women composers less favorably than male composers. I had student participants listen to excerpts of recordings of relatively obscure modernist works. For each work, I had reproduced old program notes that described aspects of the pieces. I had made slight changes to the program notes so that they would subtly indicate whether the composer was male, female, or of indeterminate gender. I was concerned about demand characteristics, so I made special efforts to ensure that the gender indicators were quite subtle. For example, consistent with typical concerts, most of the excerpts were assigned to male genders.

The experiment was a between-subjects design in which one group heard certain passages assigned to certain genders, whereas the other group was led to the impression that the composers of the same passages were of the opposite gender. Would listeners rate a musical passage as being of less quality because the composer was a woman? Were men more likely than women to exhibit such a bias?

The results were quite clear. The implied sex of the composer made no difference whatsoever on the assessed quality of the music. I was frankly surprised by the results. A couple of weeks after the end of the experiment, I happened to interact with one of my participants in the hallway. Our discussion turned to the experiment. I asked him what he thought was the purpose of the experiment. Subsequent conversations with other student participants confirmed what the first student had told me:

It seemed pretty obvious that you were trying to determine whether we were prejudiced against women composers.

All of the students I talked to had correctly inferred what I was up to, and they had made sure not to answer in a way that appeared sexist.

Dealing with Reactivity

How can we deal with participants’ reactivity? The main approaches are concealment, deception—both of which require special ethical/legal permission from our Institutional Review Board (IRB)—, and the use of implicit measures. Note that human reactivity is so sensitive, that even these techniques are no guarantee that reactivity is controlled!

Concealment

Concealment can be easy enough in exploratory research or natural experiments—for example, observing public data online. Field experiments might be concealed as well: A classic example of such an approach is found in Simha Arom’s (Voisin, Arom, and Léothaud 1997) experiments with scales in the Central African Republic. Arom was interested to learn which tuning system was preferred by his African xylophone musicians. He constructed a xylophone-like instrument that was actually a MIDI synthesizer with several different tuning systems. Arom was eager to know what the musicians preferred, without them being influenced by his presence. So he set up a video camera near the instrument and left the musicians to play with the instruments without them being aware that they were being recorded. On the video, two musicians are recorded as saying “It’s all the same” and “There’s no difference [between the different tuning systems].”

Deception

Concealment may be obvious, but would we want to deceive participants? The main idea is concealing the intent of our measurement protocol or study. The goal is to avoid correlations between participants’ reactivity and our variables of interest. If a participant doesn’t know what we are really measuring, or why, many forms of response bias are avoided.

What does deception look like? One approach is to embed the true measures alongside other fake measures. For example, you could have a somebody wear an EEG cap which appears to be recording, but which isn’t really. You could then video this person and inspect the video for the behaviors you are really interested in (foot tapping, etc.). Wearing the EEG cap would draw the participant’s attention aware from your real measure. Similarly, a standard approach in survey design is to include “lure questions” which obscure the true intent of the study. Finally, you can simply tell the participants the experiment is about something else.

Implicit Measures

Some kinds of operational measures are less susceptible than others to demand characteristics. In particular, self-reports are the easiest to be confounded by demand characteristics—a participant can simply report any answer they want. Rather than relying on explicit self reports, we can look for implicit measures: behaviors or responses that measure what we are interested in indirectly. There are several types of implicit measures, but one of the best is reaction-time (speed of response).

A simple example is provided by the affective priming method. Frank Ragozzine (2011) carried out a simple experiment where he flashed words on a screen and asked participants to respond as quickly as possible to whether the word was a happy word or a sad word. This task is easy; The words are very clearly happy or sad. For example, sad words might include glum, down, sorrow, blue and blah. Happy words might include smile, jolly, pleased, jumping and sunny. The participant simply had to press one of two buttons as quickly as possible—either the happy-word button or the sad-word button.

Now consider what happens when we play either a major or minor chord immediately before the word appears on the screen. What Ragozzine found was that playing a major chord improved the reaction speed for happy words but reduced the speed for sad words. At the same time, he found that playing a minor chord reduced the speed of reaction for happy words and improved the reaction speed for sad words. In other words, the chords facilitated performance when they conformed to common major-happy or minor-sad associations, but interfered with performance when they didn’t match these associations. What’s nice about this method is that the task is simply too fast for people to overthink their answers.

In general, researchers prefer these implicit observations over explicit measures. Implicit methods reduce the impact of what a person believes—and instead focus on how they behave. The most revealing kinds of behaviors are those that are spontaneous and unconscious. When a person taps their foot or smiles, it’s likely that they’re enjoying the music—whatever they might say. As researchers, then, we’re always on the look-out for spontaneous, easily observable behaviors that are not strongly regulated by conscious thought.

Here is a self report question:

“Are you prejudiced against women or black people?”

There is little point in asking questions like this explicitly, because virtually no one will admit that they are prejudiced. However, our behaviors may indicate that we have underlying predilections and biases, even if we won’t admit it.

A useful technique for identifying a person’s preconceptions is the implicit association test (IAT). Suppose that you are interested in whether people think that Native Americans are viewed as less “American” than white Americans. First, you begin with a series of photographs of people. Half of the photographs are of people who are obviously white and half are of people who are obviously native Americans. The task is simply to respond as quickly as possible to whether the faces are white or native-American. Next, we give you a task consisting of place names. Here the participant is asked to respond as quickly as possible to whether the place names are foreign or American. Again, the task is straightforward with places such as Seattle, France, Ohio, Russia, Miami, and Oslo. These tasks establish a person’s baseline reaction-times for identifying “white,” “native,” “foreign,” and “American.”

Next, the two tasks are joined together: Press one key if image is either a white person or an American place name; press a different key if the image is either a native-American person or a foreign place name. Finally, the reverse pairings are made: press one key if the image is either of a white person or a foreign place name; press a different key if the image is either a native-American or an American place name.

In this task, if a person tends to unconsciously think of Native Americans as less American than a white person, then they will tend to exhibit faster responses when Native+foreign are linked, and slower responses when Native+American are linked. Similarly, they will tend to exhibit faster responses when white+American are linked, and slower responses when white+foreign are linked.

Do people tend to associate blacks with crime? Arabs with terrorism? Reggae music with drugs? The implicit association test has been widely used to investigate a wide range of associations and unconscious prejudices. You can take some of these tests yourself at https://implicit.harvard.edu/implicit/demo/. Our actions can betray attitudes that our conscious selves might find quite uncomfortable.

Not all implicit tasks are as dramatic as the implicit association test. You may simply do things like ask indirect questions which avoid making the intention of the study obvious. For example, our PhD student John McNamara has been studying the psychological phenomenon of “flow,” which many video-game players experience. How could John measure “flow”? He could simply ask people if they experience flow. However, there are several potential issues:

Participants might not know what “flow” means.
Participants may be unable to accurately judge their own “flow” state
Participants may be biased—for example, if they don’t like the game experience, they might be biased in their responses.

An indirect, implicit approach could circumvent all of these problems. At the beginning of the experiment, John could ask participants to put their phones away in their bag, to avoid distraction. Then, after playing a game for a while, he might casually ask them what time they think it is. If a participant has been drawn into a “flow” state, they will underestimate how long they have been playing. So the difference between perceived time and actual time could be an indirect measure of “flow.”

Be suspicious of the capacity of participants to accurately introspect and honestly report what they’re experiencing.

Once again, self-report really is the most important source of behavioral observations for studying music. However, there are times when we need to be suspicious of the capacity of participants to accurately introspect and honestly report what they’re experiencing., and implicit measures might be the answer. The best approach is to look for converging evidence, using a mix of implicit and explicit methods and seeing if they agree.

Performance Tasks

Another approach that is (less) susceptible to demand characteristics is to use measurement protocols with very specific performance criteria—i.e., clear criteria for success, failure, and/or accuracy. Basically, we might ask “can a person do X?” or “how well can they do X?” Reaction time (mentioned above) is an example, but there are many other possible approaches: for example, you could ask a participant to sing a major third interval, measure the interval they sing, and classify it as in-tune or out-of-tune. Performance tasks tend to be relatively free from demand characteristics because participants can’t easily give “fake” responses.

Let’s say you hypothesize that musical training makes you better at doing arithmetic. You could compare how many arithmetic problems musicians and non-musicians can correctly complete in ten minutes. The musician participants might like this hypothesis—it’s nice to think that being a musician makes you smarter than other people—but they can’t force the results to be positive. The only thing they can do is do the task as well as they can. However, if, for some reason, your musician participants were opposed to the hypothesis, they could still “throw” the results by not trying as hard, or giving purposely wrong answers. Thus, performance tasks are still reactive to some extent.

Experimental Design

A stronger approach to avoiding reactivity biases, is through experimental design. Our goal is to avoid correlation between participants’ reactivity and our variables of interest. This means, that what we really need is to conceal those variables.

When designing an inquiry, we decide what conditions to sample and measure. In a human-participant inquiry, the participant themselves are another variable to consider in our design.

Consider an inquiry to determine how genre affects music’s suitability as a study aid. Let’s say we use a “genre”-independent variable with three levels, and we recruit six participants. We have the participants listen to music while studying and we measure their studying effectiveness (somehow). We could cross our genre variable with our participant variable, having each participant listen to five songs in each genre:

	Rock	Jazz	Hip hop
Participant 1	\(N=5\)	\(N=5\)	\(N=5\)
Participant 2	\(N=5\)	\(N=5\)	\(N=5\)
Participant 3	\(N=5\)	\(N=5\)	\(N=5\)
Participant 4	\(N=5\)	\(N=5\)	\(N=5\)
Participant 5	\(N=5\)	\(N=5\)	\(N=5\)
Participant 6	\(N=5\)	\(N=5\)	\(N=5\)

In this design, genre is called a within-subjects variable, because each participant hears more than one level of the genre variable (in this case, all three). However, there is a problem here. If we give our participants playlists that bounce between jazz, rock, and hip-hop, they will likely conclude (guess) that genre comparison is the intent of our study. So the participants know what we are trying to do. Depending on their attitude towards the hypothesis, they may change their behavior to enhance or hinder the results. For example, if they really hate jazz music, they might purposely (or unconsciously) study poorly when listening to jazz.

To avoid potential response bias, we could conceal that your experiment involves genre. To do this, we could not cross our genre and participant variables. Something like this:

	Rock	Jazz	Hip hop
Participant 1	\(N=15\)
Participant 2	\(N=15\)
Participant 3		\(N=15\)
Participant 4		\(N=15\)
Participant 5			\(N=15\)
Participant 6			\(N=15\)

Now genre is a between-subjects variable. Each participant only hears tracks from one genre (one level of the variable). This means, that each participant doesn’t know that genre is a variable we are interested in, and we won’t tell them—in fact, we might throw them off by telling them it’s about something else. If the participant doesn’t know genre is being studied, their behaviors cannot be influenced by the genre variable!

The disadvantage of between-subjects designs is that personality covariates are not (as well) controlled. For example, in the study above, there’s a chance that Participants 1 and 2 just happen to be particularly studious people, or that Participant 6 is a guitarist who gets distracted really easily by listening to rock music. With enough people, these random variations should average out, but still, they are not ideal. The within-subject design (further above) does a better job at controlling for these personality variables. However, in this case, the risk of demand characteristics may outweigh the gain of control.

Social Desirability Bias