36  Working with Human Participants

(Modified on 2026-08-11)
5,810 words

Nearly all music tech research ultimately involves working with human participants in some way. Even projects that are primarily computational frequently begin with human-derived “ground truth” data and end with human user-studies, for the evaluation of the results. Even outside of academia, human-participant research is essential to many businesses for marketing and user-design purposes.

You will note that I always refer to human research “participants.” Traditionally, the word “subject” was more commonly used, but the community is gradually moving away from this word because it has negative connotations. You will certainly encounter people referring to human “subjects” and “human-subject research,” but we should favor the term “participant,” even if it doesn’t roll off the tongue! Nonetheless, there are a few common terms that still refer to “subjects,” for example “human-subject research” and “within-subject design.”

Human-participant research may be conducted in person (in the field, or in the lab) or virtually (by mail, by phone, by app, or online). Obviously, physiological measurement protocols can (generally) only be conducted in the lab; some behavioral tasks cannot be accurately measured online either—for example, we can’t capture reliable reaction time data using browser-based software. In contrast, most self-report measurement protocols can easily be implemented online.

In-lab, in-field, and online approaches have advantages and weaknesses. Online approaches make it easier to recruit participants. By avoiding the unnatural environment of the lab, participants may also give more natural responses—People listen to music on their computers at home, so listening to music in an online study may encourage natural responses (ecological validity). However, online studies introduce many variables that are hard to control: did the participants fall asleep? Did they “cheat” in some way? Were they doing the experiment in a noisy environment (on the subway or in a bar)? Did they listen to the experiment using quality headphones or cheap earbuds? You have no way of knowing! Bringing a participant into your lab allows you to control all these variables precisely. We should also keep in mind that being asked to answer survey questions is never a totally natural thing. The only way to truly observe “natural” human behavior is in field-research, like making observations at a concert venue or measuring public online activity, preferably without the participants’ knowledge.

Human variables

Humans are invaluable sources of data about music and musical experiences. However, we have weaknesses—we are not machines! There are a few of important things to keep in mind when designing human-participant research sessions:

  • Stamina: Participants get tired.
  • Attention: Participants can lose focus.
  • Enthusiasm: Participants can get bored or frustrated.
  • Understanding: Participants can get confused.

To get usable data, it is essential that participants are awake, engaged (preferably enthusiastic), paying attention, and that they understand what they are supposed to be doing. This can be hard to achieve.

Unfortunately, in some cases, you will need to discard data from some participants. This should not generally be done on the basis of the content of their answers, but only based on signs that they are not actually trying to complete their task in good faith, or that they are giving invalid answers for other reasons (like lack of understanding). For example, if you observe during an in-person study that a participant falls asleep (this really happens!) you should discard their data. Even online, you might see signs of bad data: perhaps most participants finish your survey in 7–10 minutes, but one participant takes 40 seconds and another takes seven hours; You should (probably) discard their data. In some cases there are valid, a priori criteria for discarding data: for example, you might only want data from participants who achieve a certain minimum performance level, or participants who have certain skills. Be careful though, reviewers will be wary of researchers discarding data because it doesn’t show what we want it to show—if you discard data because it doesn’t match your predictions, that would be a red flag to reviewers.

Note: You cannot punish participants for giving bad data. For example, if you’ve agreed to pay them, you will pay them no matter what. Also, you should never tell a participant their data is bad and/or being discarded. Participants are understandably discouraged when they learn that their data is unusable.

Recruitment

The hardest part of human-participant research is often recruiting (and motivating) participants. We’d like to sample a large group of participants who well represent an ideal target population. However, unless you have significant funding, practical concerns usually (unfortunately) far outweigh sampling ideals. We typically advertise music studies through professional mailing lists, like the McGill Research in Auditory Perception list, social media, or through print flyers. We can also spread word of our study among our friends and colleagues. These approaches are clearly cases of convenience sampling and should be regarded and presented with a grain of salt: When describing your recruitment process in a paper, remember to acknowledge the imperfections of your sample.

Ideally, we offer to reward participants for their participation, either with monetary payments ($10–$30), gifts, gift-cards, or (in many cases) with course credit. If, as is often the case, we don’t have any funding to offer participants, we can still recruit participants, but it will be harder. Paying participants allows us to get more diverse samples, and also better motivates the participants.

Most research universities have programs where students are required to participate in research to receive academic credit, creating a pool of free research participants. At Georgia Tech, the School of Psychology operates such a pool called Sona—Talk to your research advisor about recruiting participants from Sona, or other similar pools.

Research Sessions

Once recruited, a participant is invited to participate in a measurement session, where data is gathered from them. This could be anything from a five-minute online survey to a five-hour MRI session; typical behavioral and self-report measurement sessions last 20–90 minutes.

For some research, you may need to invite the same participant to engage in multiple sessions, separated by some span of time. For example, memory researchers often ask participants to attend two sessions: a first session where they are exposed to music and a second session, days or weeks later, where their memory of the first session is tested. In the most complex case, educational researchers may observe the same participants numerous times over the course of several months, or even years. This allows them to observe (and even manipulate) their participants as they change and evolve over time. This is called a longitudinal study.

Time

All human-research requires a time commitment from those who participate in the study and it is important to respect their time. When designing a study, don’t squander the patience of your participants by asking too many questions. If the goal is to test a hypothesis, then include only those measurement items that are necessary for testing that hypothesis. Avoid the temptation to add questions or other measurements for vague or nebulous reasons. In the case of exploratory studies there is more latitude to ask questions. Since the purpose of an exploratory study is to invite new ideas and unanticipated observations, it is appropriate to “go fishing”—asking questions simply to see what happens. Nevertheless, keep it as short and as focused as you can.

Generally, to avoid participant fatigue and boredom, sessions should be no more than 20–30 minutes. For online sessions, you are especially likely to lose participants’ interest and attention, so it is better to keep sessions as short as possible, like 5–15 minutes. Note that this consideration also affects recruiting: participants are far more likely to volunteer for a five-minute survey than a twenty-minute survey. If you are able to pay participants, they will be better motivated, so you can do longer sessions; Still, I highly recommend keeping in-person studies shorter than 45 minutes.

You should always let participants know how long their time commitment is 1) during recruitment 2) before starting their session, and 3) during their session. Your recruitment email/flyer/post should indicate something like “the study takes only 25 minutes.” You will then tell the participant this repeatedly at the beginning of their participant session (either in person, or online). Finally, let the participants know how much is left to do as they progress through the session. In the case of printed questionnaires, respondents can see how many pages/questions remain. With web-based surveys, it is helpful to include an indicator of progress: This might be expressed in words (e.g. “Page 3 of 6”) or as a horizontal bar-graph showing how much has been completed and how much remains. In the case of telephone or in-person interviews, you can express this verbally as the interview progresses: e.g. “Just four more questions to go…”

Procedure

The entire process of gathering data from a participant is called the research protocol or procedure—many psych papers will have a section called “Procedure.” Let’s introduce some common terms:

  • Each data point gathered from a participant is referred to as a response.
  • A stimuli is whatever provokes the response: it may be a written question, or audio or visual stimuli.
    • The presentation of the stimuli is the “exposure.”
  • The whole process of being exposed to then responding to a stimuli is a task. In most cases, participants will repeat the same task many times, but with different stimuli.
  • Each specific measurement (specific stimuli, specific response) is an item.
    • Each measurement from each participant in a session is called a trial.
  • Measurement tasks are organized into blocks: organized sets of repeated (similar) tasks.
    • You might have two blocks of 20 stimuli, with a break in between. Typically, each block corresponds to a different experimental condition.
  • Most studies these days are set up on a computer, whether they are in-person or not, through an experimental interface. In some cases, taking responses on paper or verbally may also be done, but is less common—I wouldn’t do it unless there is some reason for it.

For example, we might run an experiment measuring the emotional impact of music with contrasting musical and lyrical valence (major/minor tonality; positive/negative lyrics). The participant’s task would be to listen to a 10-second musical excerpt (exposure) then mark on a scale of 1–9 the emotional valence of the excerpt (their response), all via an experimental interface. This task would be repeated 80 times, each time with a different musical stimuli (80 measurement items). (Each participant would see the same 80 items, but each participant would have their own 80 trials.) We would probably arrange these 80 items into four blocks of 20, with each block representing a different experimental condition: 1) major mode, positive lyrics; 2) major mode, negative lyrics; 3) minor mode, positive lyrics; 4) minor mode, negative lyrics. (This would be a within-subject design.) The blocks could have a little break between them.

We can visualize these various pieces in diagram form:

Notice that, in this diagram, the participants don’t see the same items in the same order (explained below).

Instructions

Confused participants will not provide useful data.

When working with human participants in any capacity, clear instructions are essential—confused participants will not provide useful data. You must clearly explain what you want the participant to do, taking care to define terms as you want them to be understood. If your goal is to measure “groove,” you might give instructions like:

Please rate the “grooviness” of the following recording. In this experiment, “grooviness” is how much the music makes you want to move your body along with it.

In some cases, you might want to leave participants room to interpret what terms mean—if so, tell them that, like:

Please rate the “grooviness” of the following recording, however you interpret the idea of “grooviness.”

Using visual representations of the experimental task (e.g., diagrams) in your instructions can help a lot.

Most experiments should begin with practice trials, from which data is not recorded. This gives each participant a chance to figure out what they are doing and, if in person, ask for clarification. Indeed, one major advantage of doing human-participant research in-person is that you can give the participant an opportunity to ask you questions to clarify the task before they start; with online studies, this isn’t possible, so it’s especially important that your instructions are clear.

To keep participants on track, it can be a good idea to provide new instructions for each block, or simply to reiterate the instructions before each block. If possible, build reminders of the instruction directly into each question. There is no reason to make your participants remember everything, if you can simply remind them as they go!

Timeline

In music research, self-report and behavioral measurement tasks typically consist of exposure to a musical stimuli followed by a response from the participant. The stimuli plays, then after it has stopped playing, the participant responds. This format may be somewhat artificial or unnatural, especially for measuring musical experience: After all, music listening is a dynamic experience—not something that happens after you listen. If stimuli are short, this may not be a big problem; However, imagine if I played you an eight-minute symphony movement then asked you to rate how happy it was on a scale of 1 to 10—That might be difficult! You may have felt different degrees of happiness at different times during the music. What’s worse, you may have forgotten what you felt earlier in the piece, so your response would probably tell me about how happy the piece ended, not how happy it was overall. This effect is called recency bias, and it is a well known problem in psychological measurement; more recent stimuli will dominate responses. Generally, you should keep stimuli short (1–30 seconds) so that they are easier to remember and think about, and so the ending doesn’t dominate responses.

An alternative approach is to try and measure musical experience or behavior in real time, having participants give responses during the exposure (i.e., while the stimuli is playing). You might, for example, ask a participant to listen to music and “click every time you hear a dissonance.” This approach is called continuous response, and is perhaps a more realistic (ecologically valid) measure of real-time experience. However, continuous response tasks are a lot more difficult for participants, and often don’t actually feel as “natural” as they sound: It can be difficult and unnatural to introspect quickly in real time, and the result can be a lot of random noise. Note that continuous responses are often delayed significantly as well, so a participant asked to “click when they hear X” might click 1–4 seconds after they heard X.

A final, middle-ground, approach in music research is progressive exposure, where each participant hears a little bit of the stimuli at a time, stopping to respond after each segment. If for example, you cut that eight-minute symphony movement into sixteen 30-second segments, it could be easier to rate the happiness of each segment, one at a time. Progressive exposure has been widely used in the study of musical expectations—i.e., play a little bit of music and then ask a participant what they expect to hear next.

As you design your measurement task(s), you’ll need to keep the total time of the session in mind. Try to estimate how long it will take for each participant to perform each measurement task; The speed of each task will determine how many data points you can gather. Generally, participants can do simpler (constrained) tasks quicker. For example, participants might be able to do a forced-choice binary task at a rate of ten seconds per item; At this pace, you could gather 180 responses from a participant in 30 minutes. On the other hand, a more elaborate task could take 60–90 seconds, which means you might only get 20–25 data points in a session. There is always a trade-off between quantity and complexity. Keep in mind that participants will respond much more slowly than you would—ask a friend to try the task and see how long it takes them.

Repeated Measures

As we know, we’d like for each data observation to be independent. However, human-participant procedures always involve repeated measures from the same person, so the data is never totally independent. Fortunately, some of this dependence can be accounted for.

Order effects

One very significant source of data dependence is participants’ own memory: when a participant is working on Item 5, they still remember Item 4, and Item 3, etc., so these previous items are variables that influence their response to Item 5. Generally, the most recent items will have the strongest influence (recency bias again). This means that the order that items are presented is a covariate that influences the dependent variable, potentially confounding our claims. We call this the problem of order effects.

To deal with order effects, we always randomize the order of everything we can. Randomize the order of items; randomize the order of blocks; etc. As always, randomization doesn’t magically make these effects go away; rather, it (hopefully) turns the effects into random, unbiased “noise.”

Another approach, is to counterbalance the order of items: for example, half your participants could hear the stimuli in one order, and half in the reverse order. You can also design more elaborate counterbalancing. Counter balancing makes the most sense when the number of things you are randomizing is small. If you have a 100 stimuli with potential order effects, you would randomize the order for each participant. However, if you have four blocks of stimuli, with potential order effects between blocks, randomizing the order of blocks could result in an uneven sample. Why? Because there are only 24 ways to order four blocks of stimuli. If you recruit 48 participants and randomize the order of blocks for each participant, by chance a few of the participants will see the same order, and some orders will not be seen at all. Instead, you could make sure all 24 orderings are used twice, while still assuring that each participant is randomly assigned to an order. To achieve this, you could assign unique numbers to each participant—from 1 to 48—and use their number modulo 24 to pick the order they hear.

The only exception to order randomization would be if the order is actually a research variable we are interested in. In other words, for some research hypotheses order may actually be an independent variable that we are interested in, not “just” a covariate. In this case, we wouldn’t want to randomize order; we’d want to control or otherwise manipulate order. For example, if we are studying what music makes music “relaxing,” we might hypothesize that the relaxing quality of music might be contextual. Consider these three pieces: The Rite of Spring (Stravinsky), Take Five (Dave Brubeck), As the Palaces Burn (Lamb of God).

If I play a participant Take Five first, followed by The Rite of Spring, I’d predict that the relaxing rating of The Rite of Spring would be low. If I play a participant As the Palaces Burn first, followed by The Rite of Spring, I’d predict that the relaxing rating of The Rite of Spring would be (relatively) high. In this scenario, I wouldn’t want to randomize order: I’d need to systematically manipulate the order.

Answer inertia

At the beginning of a session, participants will be unsure about what they are doing, and proceed slowly and thoughtfully. However, as they get used to the task given to them, participants often “get into a groove.” This is generally a good thing, as it means the participant is comfortable with the task, it makes sense to them, and is not too difficult. On the other hand, some times this “groove” can turn into a rut: participants who lose focus may end up giving the same, or similar, responses to many items, without paying sufficient attention. David Huron calls this “answer inertia.”

In the worst case answer-inertia scenario, a frustrated (or lazy) participant may simply give the same answer over and over again. If you see something like this happening, you may need to discard that participant’s data. However, be careful, because even attentive participants may fall into a pattern of responding in similar ways. It may be that your choice of measure was bad: perhaps you are asking them to rate “grooviness,” but none of the stimuli you selected is very groovy. It can also happen if the task is simply too tiring: If you ask somebody to rate perceived “poignancy” on 0–100 scale, a participant might feel like the slider is overkill—what is the difference between a poignant rating of 70 and another of 75? Or 32? It may be that they perceive poignancy as binary: either the music is poignant, or not. In this case, they might fall into a groove of rating each stimuli as either 0 or 100, with maybe an occasional 50 when they aren’t sure. The result is really a three-level ordinal response, not a numeric response. If you really want more fine-tuned responses, you may need different stimuli or different instructions to try to encourage finer-grained responses, or simply to make the session shorter.

One approach to limiting answer inertia is to not to use the same exact question too many times. When working with ordinal scales or sliders, you might randomly (or systematically) switch the order of scales. For example, sometimes the scale is:

  • always–frequently–sometimes–rarely–never

While other times it is:

  • never–rarely–sometimes–frequently–always

This forces the participant to think freshly about each item; Mixing questions in this way will also tend to reduce acquiescence bias. However, this can also be frustrating and annoying, and might lead to mistakes. If a participant doesn’t notice the scale changing, your data will be befuddled!

Be especially careful with measures that have inherent default or neutral positions. For example, if a continuous slider always starts in the same default position (like the middle), participants will be biased towards not moving the slider. A “Select all that apply” option could have a similar bias towards selecting nothing, or selecting only a few. A good technique is to randomize the initial/default value for each trial—for example, sometimes the slider starts at 0, sometimes at -30, sometimes at 72, etc. Another approach is to force participants to make selections, or move a slider, before they can continue. That way, if they want the slider set to zero, they can, but they have to move it away from zero, then back to zero. This prevents participants from being lazy but may also be frustrating, so use it with care!

Another way to avoid inertia is to make sure your experimental manipulations cover a wide variety of levels. If our study of “relaxing” music only used jazz and classical music, we might see participants falling into patterns of giving medium-to-high ratings. If we mix in some extremely unrelaxing music (like Lamb of God) and some extremely relaxing music, we keep the participants on their toes, and stop them from falling into a rut. Generally, we want participants to make use of the whole range of response options we give them.

Interacting with Human Participants

Interacting with human participants involves a combination of legal, moral, scientific, and educational considerations. Four goals should always shape your interaction with participants:

  1. Ensuring the physical and psychological well-being of the participants.
  2. Collecting high quality experimental data with a minimum of confounds.
  3. Providing a learning opportunity for participants.
  4. Providing a learning opportunity for the experimenter about possible unanticipated confounds and insights pertinent to the research.

Whether the participant is paid or unpaid, a student or non-student, it is important that the participant have a meaningful and informative experience. Not all experiments are inherently interesting, but it is possible for a participant to understand how participating in an otherwise boring task might produce research results of great interest and value. If a participant has a negative experience, this indicates a failure of the experiment. Moreover, if a participant has a negative experience, it reflects poorly on our research program, and may make it difficult to recruit participants for future studies.

One of the most egregious assumptions experimenters make is to assume that other people are just like us (i.e., egocentric bias). Not all people experience the world the same way. It is important not to assume that the participant is having the experience you think they are.

Throughout the experiment be vigilant for any signs that your participant is having a negative experience. The physical and mental welfare of the participant is your most important responsibility. If you deem that the participant is at mental or physical risk, or if the participant reports any negative feelings or concerns, stop the experiment (if it is safe to do so). If you inadvertently cause harm to your participant, do not attempt to cover up the problem or make light of it. Covering up a mistake will result in more onerous disciplinary actions than reporting an honest error. Should any untoward episode occur, contact the study’s Principal Investigator (probably our research advisor) immediately. Your advisor will need to submit a form to the GT Institutional Review Board. Reporting such events is mandatory.

Confidence Building

To keep participants engaged, attentive, and enthusiastic, it is important to build their confidence. If you are asking musical questions to someone who is not a musician, or is an amateur, they may feel deferential—thinking they aren’t qualified to respond or make judgements—, leading them to make unnatural responses which they think are what you want, or are “right.” Participants are less likely to be influenced by these “demand characteristics” if you boost their self-confidence. If they are confident, then they are more likely to feel comfortable with their own opinions and judgments, rather than attempting to respond in a way that they think will please you, the researcher.

Confidence-building can be done in five basic ways:

  1. Employ tasks that are relatively easy for the participant. The simplicity of the task raises the self-confidence of the participant by making them aware of their own mastery.

    If you need to ask harder questions (or use harder tasks), at least start with easy tasks. Respondents, especially if participating online, may quickly abandon a session if the initial tasks seems too onerous. A helpful strategy is to place easy questions at the beginning of the questionnaire. For example, asking a person’s age, sex, native language, country of birth, etc. As respondents answer more questions, they will tend to feel committed to the task, and so feel a greater sense of obligation to finish the session. Respondents are more likely to complete a survey if the difficult questions are posed at a time when they feel the questionnaire is nearly complete.

  2. Encourage self-confidence through positive feedback: “You seem to be really good at this task”; “You were selected for this experiment because of your background and expertise”; etc. Never give participants negative feedback. Such feedback may be honest or deceptive. That is, even if a participant is not especially good at the task, such positive feedback will still boost their self-confidence and reduce the likelihood of them looking to echo what they think the experimenter wants.

  3. In lots of behavioral and self-report research, there is no “right” or “wrong” answer—we are genuinely interested in the participant’s unique view, opinion, or experience. Unfortunately, it is very common for participants to be preoccupied with giving the “right” answer If a task doesn’t have “right” answers, you need convince participants of this, by wording the instructions to reinforce this. For example, instead of asking “Which musical passage sounds better?” Ask “Which musical passage do you think sounds better?”

  4. Behaving in a deferential way toward the participant, or reducing the appearance of being an authority figure. A participant who feels they are socially equal to, or superior to the experimenter is more likely to respond honestly. For many participants, if the experimenter is an older male professor dressed in a white lab coat using formal language with a deep confident voice, then participants may be more likely to respond in a way that conforms to what the participant thinks the experimenter wants. Conversely, for many participants, if the experimenter is a younger student dressed casually using colloquial language with a tentative insecure voice, then participants may be more likely to respond honestly.

  5. Employing tasks that are not socially or politically charged. For example, asking “Which of these two tones has a longer duration?” is likely to incur fewer demand characteristics than asking “Which of these musical styles is better?”.

Standard Protocols

When running a human-participant study, there are a number of standard protocols you should follow. I will focus here on the protocols that are relevant for in-person sessions. However, all the principles here can be reflected in the design of online research interfaces.

Preparation

All interfaces (online or in person) should be thoroughly tested all the way through, checking that data is being recorded correctly.

Before running any session be sure you are fully acquainted with the equipment, instructions, and procedures for the experiment. All interfaces (online or in person) should be thoroughly tested all the way through, checking that data is being recorded correctly. Unless the experiment is a minor variant of a previous experiment, you should actually practice running the experiment on another member of your lab. Let me repeat: TEST YOUR INTERFACES ALL THE WAY THROUGH AND MAKE SURE THE DATA IS BEING RECORDED CORRECTLY!

Interaction

When engaging with participants (online, or in-person), be friendly and casual. In the lab, they are entering what they perceive as “your space,” and may feel that they are merely cogs in some impersonal research machine. Interact with your participant to make them feel at ease: Introduce yourself and your role in the process. Address them by name. Ask them how their day is going; ask whether they feel “up” for an experiment. Make sure they are comfortable. Use your interaction to get a sense of whether the participant is nervous, or in a hurry, excited, depressed, or out-of-breath. If the participant has a cold or is otherwise ill, they are not likely to enjoy the experience; and they are not likely to provide good data.

Describe

Once a participant is comfortable, you can begin to tell them about the experiment. Tell them the overarching topic you are researching. Describe the stimuli they will hear, the nature of the task. Remind the participant about the approximate length of the experiment, and that they are free to stop and/or leave at anytime. As you describe the task they will perform, make sure to ask them whether they have any questions. This is a chance to catch misunderstandings before they happen.

Depending on the nature of the research, it may be important to deceive the participant (only with IRB approval), or at least keep some information secret, to avoid biasing their responses. If a participant asks about the specific goal of the experiment—or other information you don’t want to reveal)—you can respond by (1) telling them about the broad goals of this field of research, (2) telling them that you prefer not to reveal the specific theory or hypothesis you are testing, but that (3) you will tell them exactly what you are doing following the end of the experiment.

Instruct

At this point, you can go over the specific instructions for the experiment. Often, these instructions will appear on a computer screen. Since participants often “skim” over any printed instructions, it is prudent to go over the process with them, either reading the instructions aloud or simply describing the process yourself. If you find it useful to embellish the instructions, use the same language for each participant and record the exact words you use.

Ask the participant if they have any questions—most won’t have any questions to ask at this point. If the participant asks a question, make a mental note of the question. Later, include the question in your experimental notes. Such questions can help identify ways to improve the experimental set-up. Participants’ questions may also alert you to possible misinterpretations of the instructions. For example, you may be unaware that there is some ambiguity in the instructions, and that some participants may be responding in a way you hadn’t anticipated. Finally, such questions may later prove helpful in identifying data that should be excluded from the final analysis.

When answering participants’ questions, try to avoid giving them extra insights into the experimental hypothesis or research goals. Be careful in your choice of words; Say as much as needed in order for the participant to feel they understand. If you re-explain the instructions, be wary if the participant says s/he now understands. After repeating an explanation, it is common for a confused participant to claim s/he understands in order not to be perceived as stupid.

Include your response in your experimental notes. Make a mental note of the exact words you used to clarify the instructions. Be accurate. If you said something you shouldn’t have, be honest in reporting.

Don’t hover

In most cases, once the participant has started the session, you don’t need to do anything until they finish. Don’t hover over the participant. Go away and let the participant continue on their own. Unless the experiment explicitly requires you to observe or interact with the participant, you should leave the room or leave the participant alone. But don’t go too far away. Occasionally, a participant will stop the experiment before the end. Sometimes this is for a good reason—such as an equipment failure, or the decision of the participant to leave. In rare cases, a participant will stop to get a drink of water or ask you a question. If the participant wants to continue with the experiment, use your judgment in determining whether the experiment should resume.

Debrief

To check the validity of your research, it is essential to end every research session with an opportunity for participants to provide feedback. This feedback can alert you to unanticipated problems, including demand characteristics. Participant feedback is also an excellent source of ideas for future research. Asking these questions before the experiment might bias participants’ responses, but once the main task is complete, it is safe to ask! You want to know:

  • Did they understand/perform the task the way you wanted/expected them to?
  • Did the participant form an idea of the hypothesis (or research goals)?
  • Did other covariates you didn’t consider affect their responses?

To gather general feedback, you can include a free-response “general comments” text-box at the end of an experimental interface. Better yet, when working with participants in person, you should engage in a short post-session interview, to “debrief” the participant. Sit down with the participant and ask them to describe their experience. Begin with the most general questions:

  • “How did it go?”
  • “Did you find it easy or difficult?”
  • “Did you have a strategy for doing this task?”

The idea is simply to get the participant to talk about their experience. Try to avoid asking leading questions. Be vigilant for comments that suggest that the participant misunderstood the instructions or was using a strategy you hadn’t imagined. If necessary, begin to ask them more direct questions:

  • “What do you think of the experiment?”
  • “What do you think the purpose of the research is?”
  • “Was the task intuitive or difficult?”
  • “How did you use the rating scale?”
  • “What factors affected your responses on the scale?”
  • “Did you have a strategy you were using for this experiment?”
  • “Did you simply follow your intuitions, or did you follow a particular way of doing the task?”
  • etc.

These sorts of questions might alert you to some unanticipated demand characteristics, or other problems.

The value of debriefing is evident in a study by Huron, Kinney & Precoda (2006). The study was intended to test the conjecture that low transpositions of melodies are heard as more aggressive than high transpositions of the same melodies. Each participant was asked to judge how aggressive each melody sounded. In the debriefing, one participant remarked:

Some of the melodies were quite low, and I thought it was obvious that lower melodies would sound more aggressive, so I discounted that and tried to focus on the melody itself, and what would make it sound aggressive.

Although the participant hadn’t inferred the purpose of the experiment, she nevertheless regarded the main effect as obvious, and so answered in a way in which she thought controlled for the very effect we were anticipating. In this case, we used the results of the post-experiment interview to discard that person’s data from the analysis. Note that this was done before looking at any of the data.

Notice that debriefing does not reduce demand characteristics. Instead, debriefing alerts you to the possible existence of demand characteristics. Debriefing is cheap and easy to do.

Final Steps

Every human-participant research session should end with a few last steps.

  • Confess: Occasionally an experiment will involve deception (with approval by the IRB). We must always “confess” the deception, and explain the real purpose of the deception. If a participant is upset by the deception, do not attempt to make light of the situation. If the participant expresses any negative feelings about the deception, inform them that they are free to file a complaint with the IRB.
  • Educate: At an academic institution, education is always part of our mission. Simply participating in research is a learning experience, but it is valuable to take the opportunity to teach the participant more about what you are doing, and why. Tell the participant about your research, and say something about what you expect to find. At this point, you can safely explain any details of the research participants are interested in. Identify the independent and dependent variables. You might describe the different types of stimuli used or describe the methods used. Give the participant an opportunity to ask questions about the purpose and methods of the experiment.
  • Thank: Thank your participant. Tell your participant how important s/he is in the research program, and that we are genuinely grateful for their participation. Finally, ask them not to disclose any information about the session to their friends or class mates, so they don’t bias any future participants.

Mantras

Debrief before belief

At each stage in research, it is important to look back and evaluate: how did it work? Did it work the way we expected? Did we measure what we intended to measure? Did we sample what we intended to sample? It’s also important to ask the opinions of others, not just yourself: your colleagues, your data coders, your research participants. Don’t form your final claims, or craft your final rhetoric (or publish!), before debriefing.