32 Self Report

(Modified on 2026-08-11)
3,249 words

The most common approach to measuring human behavior, perception, attitudes, and knowledge, is by simply asking people what they think or feel. A common example is the questionnaire—a research inquiry that consists of nothing but self-report questions. Each individual question/task on a psychological survey or questionnaire is called an item. However, many human-participant research studies contain elements of self-report mixed with, or among, behavioral and physiological measurements. As mentioned in the Human Measurement chapter, the boundary between “behavioral” tasks and “self report” is a blurry (and unimportant). Most methodological advice here applies regardless of whether the measurement is “purely” self-report, or if it involves a different behavioral task.

The most important, overarching consideration with self-report measures is how much you constrain participants’ responses. There is a continuum of possibilities. On one end of this continuum are free responses, open questions (“fill in the blank”), and full on interviews, where even the questions asked can be guided by the participant. On the other end, we have closed questions (multiple-choice) or ratings on ordinal scales.

Open and Closed Questions

Humans are (generally) capable of pontificating at great length about their experiences. Humans know a lot and are able to express sophisticated knowledge, using natural language. Open questions and free responses invite participants to share this knowledge without imposing (many) preconceptions about the answers. The main advantage of these open questions is that they limit the influence we, the researchers, have on participants’ responses; this keeps us open to new and unanticipated information and stops us from biasing the answers. These approaches are most suited to exploratory work, where we don’t know exactly what we want participants to tell us, but can be used in hypothesis-driven inquiries as well.

Free responses lead to data which is less coherent and reliable.

Unfortunately, responses to free/open questions can be difficult to make sense of or analyze. Giving participants response freedom free responses will lead to data which is less coherent, reliable, and precise. Two different participants may give completely different, unrelated answers to the same question, making generalization difficult. In particular, it can be difficult to test (falsify) specific, precise hypotheses using free responses.

Constrained responses call for simplistic, reductive answers, without nuance.

To get data which is more suitable for analysis and hypothesis testing, we usually constrain self-report responses significantly, by guiding participants’ responses towards the specific information we want them to provide. This inevitably means that we are biasing responses towards what we think responses should look like. Closed (constrained) questions call for simplistic, reductive answers, hiding complexity and reducing the possibility of nuance. If your closed operational definition doesn’t match the construct you aim to measure, or the participant simply doesn’t understand it, their responses may be unpredictable or strange. However, this also means that the data is more consistent and easy to analyze, lending itself to modeling and hypothesis testing.

The most constrained self-report task is a binary choice—choose A or B, Happy or Sad, Yes or No, etc. The more measurement levels we allow our participants, the less constrained the task becomes. Common self-report items might include

Ordinal scales with anywhere between three and fifteen levels.
Categorical list, for example a drop-down menus, with a handful of choices (for example, genres) or dozens of choices (for example, emotion categories).
Continuous rating scales, like sliders ranging from 0–1, or 1–100.

Forced-Choice

Most psychological questionnaire items are what we call forced-choice, meaning the participant must choose between the responses we allow. They cannot skip a question, or say “I don’t know.” Forced-choice items enforce a lot of consistency on the resulting data, which is good.

However, forced-choice questions can be problematic as well: if the choices you provide are insufficient to answer the question you ask—or your participant thinks they are—their responses might be random or guided by unpredictable demand characteristics. If you’re not confident in your choices, you may allow participants the option not to respond to an item; usually by providing a “I don’t know,” “does not apply,” or “neutral” level. You might also include an open-response “Other” option, with an opportunity for the respondent to provide further explanation: E.g.

▢ Option #1 
▢ Option #2 
▢ Other (please explain:) _______

Generally, forced-choice is a good option. It is not uncommon that participants will feel unsure about a forced response, or even feel like they are being forced to answer randomly, but when you analyze their responses you find that they consistently tend in one direction. If there was a “I don’t know” choice, they would use it, but this ultimately makes it harder for you to analyze their results. That being said, in some cases “does not apply” really is the only realistic answer, so you must consider whether you want to include it.

Scales

A common form of constrained response is a discrete ordinal scale with 5–\(\approx 15\) levels, commonly called “Likert scales.” Rensis Likert was a sociologist who developed measurement techniques for sociological research in the 1930s. His approach (Likert 1932) actually involves averaging together data from multiple scales. However, it has become common to call any ordinal response scale a “Likert” or “Likert-type” scale.

Effective use of Likert-type scales is not always as simple as it might seem. In particular, it is not clear whether scale data should be treated as ordinal data or interval data.

Suppose that we want to ask participants the question “How often do you listen to music?”, and measure their response on a five-point scale. Compare the following response items:

always–frequently–sometimes–rarely–never
always–◉–◉–◉–never

The first scale has much more specific labels, which our participants will try to follow. Effectively, these labels make this scale explicitly ordinal: after all, we can’t really claim that “sometimes” is exactly halfway between “always” and “never.” However, by labeling only the two ends, the second scale encourages our participants to treat the steps as being evenly spread between the extremes; For example, the middle dot should represent “halfway between always and never.” Thus, it would be more plausible to claim that responses on the second scale are on an interval scale.

However they are labeled, whether participants consistently and reliably treat Likert-type scales as ordinal or interval scales is not clear. Scale data is often analyzed as if the responses are on an interval scale, but they probably should be treated as ordinal data in most cases, especially if the scale only has five levels. It is more plausible that participants might treat larger scales, with 10+ levels, like ratio scales. Generally, using Likert-type scales with fewer than seven levels should be discouraged.

Here is another consideration: If a response scale has an odd number of levels, this allows the respondent to select a middle or neutral response. This is the reason why most Likert scales involve 3, 5, 7, or 9 positions:

always–◉–◉–◉–never

There are times, however, when it is preferable to force the respondent not to sit on the fence. A good example is found in political polling. A plurality of Americans claim that they are politically independent, not considering themselves committed to either the Democratic or Republican party. However, whenever a person votes, they are ultimately forced to make a choice. What’s more, a majority of self-identified independents actually do consistently support either Democratic or Republican candidates. In a poll with an odd-numbered scale, these participants might select the neutral option, which tells us nothing useful. However, an even number of response points would force them to tip his/her hand, and show which direction they are leaning:

Prefer Democrat–◉–◉–◉–◉–Prefer Republican

In the case of political polling (in the USA, at least) this approach actually leads to better inferences about future elections.

Question Wording

When working with self-report measures, you are leveraging the vast complexity of human cognition and natural language to gather information. However, natural language is extremely subtle and contextual. This means that the exact wording of instructions and questions can have a large influence on how participants respond.

In order for your data to make sense—for your items to measure what you think they are measuring—, your participants need to understand the questions as you intend them and answer the questions in a way which is consistent with your understanding. The more constrained your questions are, the more important this is.

Generally, your questions (and instructions) should use ordinary language, avoiding fancy vocabulary and complex grammar. Aim for an everyday conversational style. Try to avoid questions that make especially refined distinctions: these may be confusing for many respondents. The more clear and precise your questions and instructions, the more constrained and consistent participants’ answers will be.

If you are working with specialist participants (like musicians), you might use some relevant jargon, but only if you are confident all your participants will understand it. It is a good idea to precisely define what you mean, even if working with expert participants: Terms like “beat” and “groove” have many meanings, even among music experts. You might need to define a lot of terms and give examples to clarify your intended meaning.

Many musical terms will be familiar to non-musicians, but their understanding of what they mean might be totally wrong and inconsistent. For example, lay people might have an idea what a “bassline” is, but not actually be able to identify one. Even words like “tempo,” “note,” and “chord” might be unclear to lay people. Working with non-musicians, you’ll generally need to express ideas in plainer language.

Instead of asking if a note is “in key,” you might ask “does this note sound like it fits here?”
Instead of asking if “the tempo is fast or slow,” you might ask “would you tap your foot fast or slow to this?”
Instead of asking if “how much dynamic contrast there is in the music,” you might ask “does the music get louder and quieter a lot”?
Instead of asking if “this chord sounds dissonant,” you might ask “does this moment sound harsh or unpleasant?”
Etc.

Note, in some cases, you might want unconstrained responses, leaving some words up to the participants to interpret. Musical terms like “groove” and “genre” are natural language, not technical terms. Any native English speaker has just as much ownership (and understanding) of these terms as we do. So, if you want to, for example, measure how “punk” a band is, you could simply ask participants to rate the “punkness” of a band, without clarifying the meaning. That is, allow the participant to judge for themselves what “punk” means. This will lead to less consistent responses between participants, but can nonetheless be the best approach in many cases: basically, if superimposing your narrow definition of punk would be worse.

Consider again the experiment I’ve talked about before, where Rosa Westfall and I ask participants to listen to different musical mixes while we manipulated the amount of dynamic range compression used. Our dependent variable was the “quality” of the mix. We didn’t want to bias our participants’ judgements of what a “quality” mix was. We worried that any attempt we made to define “mix quality” would be biased towards our opinions which, as mixing experts, are not representative of the general listening population. Worse, we feared that we might draw attention directly to the changes in compression, which could lead to demand characteristics. So, we simply asked our participants to rate the “quality” of the mix, and allowed them to judge for themselves what that meant. However, we did make clear that the question was about the mix, not other musical attributes like melody or rhythm. So, if the participants didn’t like the tune, they were supposed to ignore that.

Another consideration is, unfortunately, that participants may find it difficult to interpret and understand questions, or tasks, as we’d like them to. For example, we asked our participants not to allow melody/harmony/rhythm/lyrics to influence their judgement of musical mix “quality.” However, as a listener, it can be difficult to isolate different features like this; Non-musicians in particular might not be equipped to isolate different aspects of music. It is very common that different self-report measures that shouldn’t be correlated are. For example, a common approach is to ask participant both for their personal opinion and an “objective” opinion—like I really love Blink-182 but I know they really aren’t that great of a band. What you’ll find is that, even if participants make an honest effort to do this, their responses to these two questions will be highly correlated.

Priming

In research, we must always do what we can to make our work falsifiable, and not biased in favor of the results we want. It is thus important to avoid leading questions that “point” participants to the answer you want. Questions should be formulated in a neutral way that doesn’t favor particular answers. Even subtle differences in word choice can prime (bias) answers.

Imagine that we’d like to study the relationship between musical complexity and popularity across genres. We might ask people how much they appreciate complexity vs simplicity in music. Consider these two possible response items:

Simplistic–◉–◉–◉–◉–◉–Sophisticated
Authentic–◉–◉–◉–◉–◉–Pretentious

The word choices in both of these items are badly biased. The words “sophisticated” and “authentic” generally have positive associations, while “simplistic” and “pretentious” have negative connotations. By choosing one or the other of these scales, we could easily bias participants towards the response we predict. Something like this:

Authentic–◉–◉–◉–◉–◉–Sophisticated

would be better. However, an even better choice would be something like:

Not Complex–◉–◉–◉–◉–◉–Complex

Mixed labels

It is generally a bad practice to label response scales using different concepts for the endpoints. Why? Because many pairs of antonyms relate to concepts that aren’t really total opposites. For example: the following are bad uses:

Bitter–◉–◉ ◉–◉–◉–Sweet
Sad–◉–◉ ◉–◉ ◉–Happy

Sweet is not the opposite of bitter: we can have experiences that are simultaneously bitter and sweet. Similarly, happy is not the opposite of sad. A person may be unhappy (as, for example, when feeling angry), but that doesn’t mean the person feels sad. There are many other such “imperfect antonym” pairs; For example, it is not entirely clear that consonant is the opposite of dissonant.

It is much better to label bipolar scales with a single term, modified by negatives—such as un-, dis-, no, low or less—or amplifiers—more, greater, increasing. For example:

Happy–◉–◉–◉–◉–◉–Unhappy
Less Dissonant–◉–◉ ◉–◉–◉–More Dissonant
No Syncopation–◉–◉–◉–◉–◉–Highly Syncopated

Some bipolar terms may be okay if they are especially clear, such as dark/light (rather than more-dark/less-dark), and warm/cool (rather than cool/less cool). However, you are assuming that they make sense to your participants and that your participants will use the scales in a way that makes sense to you!

After assembling a questionnaire, get in the habit of scrutinizing each of your response items. For each question, ask yourself “who might have trouble answering this question?” For closed questions, you’ll also want to consider “are the choices given appropriate for everybody?” For example, you may have a question about marital status with two choices: married or single. Suppose a person was technically married, but had lived as a single person for the past decade. How should they answer this question? You might consider adding a third category: married, single, or divorced/separated. However, even then a divorced-and-remarried person might wonder whether to answer divorced/separated or married.

After scrutinizing each question, you may nevertheless decide to keep the question as it is. For example, you may conclude that although the question has some ambiguity, a less ambiguous form would be too complicated. Nevertheless, question-scrutinizing is a useful exercise in alerting you to possible problems. Writing down your answers is important because this task will slow you down and help you think carefully about each question.

Converging Questions

Given the many dangers and problems with self-report measures, you may be discouraged from using them. Fear not, these measures are widely used, and for all their problems, generally provide valuable data, and far more cheaply and easily that most other approaches.

The best way to guard against the problems of self-report measures is to use more than one measurement and look for converging results. Suppose, for example, that you want a measure of the degree of musical interest for each respondent. One question might be:

Identify the phrase that best describes your level of musical enjoyment:

▢ I experience relatively little enjoyment from music

▢ I find music mildly enjoyable

▢ I get quite a bit of enjoyment from music

▢ I am a music lover

▢ I am a passionate music nut

A later question might address the level of musical interest differently:

Compared with most people, how would you compare your musical interest?

▢ I am much less interested in music than most people.

▢ I am a little less interested in music than most people.

▢ I have about the same interest in music as most people.

▢ I am a little more interested in music than most people.

▢ I am much more interested in music than most people.

Yet another question might address the level of musical interest using a different approach:

How important is music?

▢ music is absolutely essential

▢ music is important, but not essential

▢ music is one of the many good things in life

▢ music is over rated

The three questions address slightly different things. The first question relates to musical enjoyment; the second question relates to musical interest; and the third question relates to the importance of music. Notice that a person might cogently respond by claiming to find music intensely enjoyable (“a passionate music nut”), have only moderate interest in music (“I have about the same interest in music as most people”), and regard music as ultimately unimportant (“music is over-rated”). This is a theoretically tenable view. Nevertheless, we would predict that the answers to these three questions will tend to correlate: someone who regards him/herself as a music lover, will be more likely to believe he/she enjoy music more than most, and to believe that music is relatively important. Conversely, someone who regards him/herself as having little interest in music, is also apt to believe that he/she is less interested in music than most people and that music is over-rated in importance. To make your research even stronger, you might try to pair your self-report measures with other implicit behavioral or physiological measures, and look for consistency (i.e., convergent validity).

As with all operationalizations, we as researchers can never capture the “true” essence of the theoretical construct we want to measure. All measurements are approximations of the true concept (All models are wrong, but some are less wrong). In the analysis, we would check to see that the various operationalizations correlate positively. If so, we might then combine the results from the pertinent questions into a single measure that we hope captures well the theoretical concept of interest.