35 Free Response

(Modified on 2026-08-11)
6,950 words

All research methods fall on a continuum between qualitative and quantitative. This continuum can be viewed numerous ways:

\(\leftarrow\) More Qualitative	More Quantitative \(\rightarrow\)
\(\leftarrow\) more open	more restricted \(\rightarrow\)
\(\leftarrow\) more broad	more narrow \(\rightarrow\)
\(\leftarrow\) less precise	more precise \(\rightarrow\)
\(\leftarrow\) more complicated	more simplistic \(\rightarrow\)
\(\leftarrow\) more nuanced	more general \(\rightarrow\)
\(\leftarrow\) more exploratory	more hypothesis-driven \(\rightarrow\)

When gathering self-report data from human participants, we often rely on restricted, forced-choice responses. This quantitative approach has a price. Because we ask the same, closed, questions in the same order to every participant, we do not obtain full reports. Instead, the information we obtain from any one person is fragmentary, made up of bits and pieces of attitudes and observations and appraisals.

By increasing the number of possible responses, we can give participants more freedom to express their detailed knowledge. However, even a continuous slider or a drop-down menu with 50 items to choose from is still relatively restricted. There are times when it is appropriate to give participants much more freedom in their responses. Perhaps an empty text-box with instructions to “fill in the blank” or “tell us about your experience.” The participant is then free to talk at length in response to any question, sharing details and nuances, and often tying together disparate ideas that might be “isolated” in more quantitative (reductive) research paradigms.

Free responses are most useful in exploratory research, where we don’t know exactly what we are looking for yet. In these situations, we don’t want our (limited) preconceptions to shape or bias participants’ responses. Questions may be relatively focused, suggesting (relatively) constrained responses, or totally open. For example, we could ask “Do you feel any negative emotions when listening to this music?” or “What does it feel like to listen to this music?” Of course, as with any self-report, the questions we ask can still affect (bias) our participants’ responses; however, the more open the question, the less we influence the answer.

Interviews

An interview is a type of formal conversation that takes place between an interviewer and one or more interviewees. Interviews may be conducted in person or remotely. The “formality” of the interview can vary significantly. One advantage of interviews is that they can be done in extremely informal ways, making them feel more natural to participants, which lends itself to good ecological validity. To enhance this, interviews may take place in an interviewee’s home, place of work, city street, a neutral space, or a special locale—such as in a retail environment, at a concert hall, green room, in a recording studio, etc. Typically, the responses are recorded by the interviewer (with the prior approval of the respondent).

In most research, it is a good idea to have specific interview questions planned in advance. This can assure some consistency, especially if all respondents are asked the same questions. With fixed-questions, an “interview” can even be conducted as a written survey using text-box responses. However, people are much less likely to write at length about their experiences or views, compared to talking about them. One of the disadvantages of fixed-question interviews is the tendency to create a situation in which the respondent is not free to “tell their story.” If the question is too narrow, or doesn’t quite resonate with the participant, they may answer in perfunctory fashion and simply wait for the next question, rather than launching into a detailed description of their experiences, impressions, or views.

The Open Interview

The real strength of interviewing is that it becomes possible to adapt the interview questions based on participants’ answers. If an answer is interesting but short, or unclear, you can ask follow-up questions. If a participant mentions something new you didn’t expect, you can ask them about it. Only through dynamically-generated questions can the “full” qualitative potential of human participant research be unlocked. Especially when the interview has been arranged in advance and is conducted in a comfortable place, the interview becomes a sort of guided conversation that encourages the interviewee to say whatever they feel is important. This is a fully open interview. The open interview is also known by other names, including unstructured interview, free interview, narrative interview, depth interview, and nondirective interview.

The open interview is one of the very best exploratory research methods. Open interviews are widely used in consumer research, in criminal investigations (such as eyewitness testimony), in clinical psychology and psychiatry, in social work, by historians, investigative journalists, sociologists, anthropologists, and ethnomusicologists. Open interviews are valuable for at least three reasons:

First, open interviews invite observations that are unanticipated by the researcher.

Interviewees provide access to first-person “facts” that are otherwise inaccessible. Interviews can draw attention to unique experiences that are beyond the personal experience of the researcher. The interview can draw attention to novel phenomena, and different forms of human experience.
Second, interviews can introduce the researcher to different ways of thinking about a question or phenomenon.

Interviewees are apt to offer different perspectives or unique interpretations. The ideas offered by interviewees inevitably expand the horizons of the researcher. Interviews often provide inspiring new conjectures and theories that stimulate the researcher’s imagination. As Robert Weiss (1995) has noted, “those who do qualitative interview studies invariably wind up knowing a lot about the topic of their study.”
Third, interviews can provide vivid first-person anecdotes.

Charities have long recognized the value of personalizing tragic situations. One might make a charity appeal by distributing statistics, such as evidence that 2.2 million people in a particular region of the world are starving. However, charity appeals are much more effective if one tells the story of a single individual: Maria, an 8-year-old child, who, with her mother, is starving in a modest hut. Although the starvation of millions of people is clearly a more serious problem than the starvation of one or two people, the concrete personalized story is far more compelling to human minds.

A similar phenomenon occurs in research: People often find research results dry and uninteresting. Researchers can make the results of research much more compelling to readers by illustrating various findings through personal anecdotes—the sorts of anecdotes that arise from personal interviews rather than through formal experiments or through impersonal questionnaires. Interviews give the researcher access to potentially evocative narratives—stories that can illustrate a phenomenon in concrete, human terms. Lifeless theoretical accounts can be brought alive through the judicious relaying of narrative examples.

By way of summary, open interviews can lead to (1) new facts, (2) new theories, and (3) vivid illustrations.

Interviews also suffer from a number of problems. The information collected is anecdotal, often impressionistic, and conveyed in a social situation in which the respondent is strongly motivated to be viewed with sympathy or admiration. The “presentation of self” typically looms larger in inter-personal situations (such as interviews) compared with the more impersonal or anonymous questionnaire or experiment. This can make social-desirability bias a major concern.

Emic and Etic perspectives

In anthropology and the social sciences, including ethnomusicology, it is common to simply engage with and interact with people in their natural environment. Formality is avoided; the researcher aims to learn through active participation in the culture. The best-known example of this approach is the participant-observation method commonly used in anthropology and ethnomusicology. For example, to learn about Moroccan Berber music, a researcher might travel to Morocco and study how to play the rebab in a traditional setting with a local master. Similarly, somebody studying Trap music, could go out to clubs around the Atlanta area and observe (and participate) in that musical culture. Many important observations are introspective—recognizing what it feels like to dance, play an instrument, or be disciplined by a master teacher. The researcher observes human activity “from the inside” rather than from the outside.

In anthropology, the “inside” perspective of a member of a culture is referred to as the emic viewpoint. In contrast, the “outsider” perspective is the etic viewpoint. Understanding the emic perspective of people within a culture is very difficult to do without actually participating in that culture. Through most of the 20th century, it was very common to see Western (white) academics engaging in “armchair musicology”—studying foreign musical culture from a purely etic (supposedly objective) perspective. This approach is now regarded to be a colossal scientific (and moral) failure. Even for non-sociological research, we can take this lesson to heart: for example, if an American researcher wants to study music from another culture, collaborating with researchers who have an emic perspective on that culture will make the research seem much stronger.

Recruiting Participants

Interviews typically require an especially large time commitment from participants, and from researchers. Ninety minutes is a common time period for an interview, and many studies interview participants several times. The time commitment for an interview can lead to sampling problems. Sample sizes tend to be small. People who are retired, unemployed, or eager for company are most welcoming to interviewers; In contrast, people who are busy simply cannot afford the time. Consequently, interview methods tend to be biased toward people who have leisure time.

Given these problems, it is common for qualitative researchers to abandon the goal of recruiting a representative sample. Instead, they recruit people who have special knowledge or extensive experience related to some phenomenon. Highly experienced or knowledgeable respondents are commonly referred to as informants, witnesses, experts or research collaborators. Alternatively, the researcher might aim to recruit a heterogeneous group of participants—people who are apt to represent a range of contrasting experiences or perspectives. As an exploratory research method, the main aim of open interviewing is to collect new ideas. Consequently, a representative sample may be less valuable than a heterogeneous sample.

By way of example, a study examining “the concert experience” might focus on interviewing audience members attending classical and popular concerts. However, depending on the aims of the study, a researcher might wish to expand the list of interviewees to including concert promoters, artist managers, publicists, house managers, ticket sellers, scalpers, conductors, musicians, and ushers. Once again, the aim of the sample is not to be representative of average or typical experience, but to be representative of the variability of experience.

In ethnographic work, it is common for the researcher to seek out an expert, sometimes called a “key informant.” This is someone with an unusually high level of expertise or advanced knowledge related to the researcher’s subject of interest. Commonly, this is a master musician or master teacher in some musical tradition. Once again, it is hoped that what is lost in representativeness is made up for by the wealth of nuanced detail and the opportunities to glimpse unanticipated experiences.

In recruiting interviewees, it is important to consider a control group. For example, Ehrensaft (1990) carried out a study of married couples who share parenting. However, Ehrensaft only interviewed couples who share parenting duties, without interviewing a comparison group of couples who do not share parenting. This makes it nearly impossible for Ehrensaft to convincingly support any claims about the special characteristics of couples who share parenting. Unfortunately, it is common for interview-based researchers to overlook the need for a control group (see also Weiss 1995).

Snowball Sampling

In the case of recruiting experts, one must often rely on the experts themselves to help you contact and recruit other experts for interviews. This is referred to as snowball sampling, which is a form of convenience sampling. As you might expect, snowball sampling is likely to introduce bias, since there is a strong likelihood of interviewing individuals who are similar, and there is thus little data independence. In order to reduce this bias, the researcher should aim to follow lines of referral that lead to people who are unfamiliar with earlier interviewees. For example, suppose that Alice recommends Bill, and you interview them both. Then Bill recommends Carol and Daniel. If Alice knows Daniel but not Carol, then it is preferable to interview Carol over Daniel. By following this strategy, it is less likely that the researcher will interview only individuals within a single social network. In ethnographic fieldwork, it is common for the researcher to switch to a different village or locale in order to avoid undue emphasis on a single social group.

Notice that snowball sampling is biased against those who have smaller social networks. Experts who are less sociable are less likely to be known by other experts, and so are less likely to be interviewed.

The Interview Guide

Although open interviews may have few predefined questions, they are normally shaped by an interview guide. The interview guide is a written document that identifies the goals of the interviews, lists the topics of interest, and provides some suggested questions. The guide may or may not be consulted during the interview itself, although it is common toward the end of the interview for the interviewer to refer to the guide to ensure that all of the pertinent topics have been covered.

As with other research approaches, it is important to test the interview guide by conducting some pilot interviews. It may be impossible to employ an appropriate expert for the pilot interview. Nevertheless, even a non-expert interviewee in a pilot interview may help you recognize ways to refine and improve your interview guide.

When the research topic relates to specific activities, like composing or practising, the guide may follow a diachronic sequence. For example, a guide related to the experience of attending a concert might include the following:

Becoming aware of an up-coming concert.
The decision to attend.
Inviting friends, securing tickets.
Travel to the concert.
Pre-concert activities (e.g., dinner).
In the hall before the start.
The concert experience.
Intermission activities.
Post-concert activities.
The next day: discussion with friends. Monday-morning quarter-backing.
Concert memories, reminiscence, assessment.

The guide is intended to help you organize your questions, and to ensure that the participant has thoroughly addressed the various components in a concert experience.

Conducting the Interview

Enter the interview prepared. Confirm the interview appointment in advance and show up a little early. Make sure you are fully familiar with your interview guide, that you have two copies of the consent form, and that your recording gear is operating properly.

The first order of business is to establish a positive and professional rapport with the respondent. This begins with an explanation of the aims of the research, a conversation about the formal consent form, and a discussion about the use of the recording. By clearly stating what is expected, respondents are more likely to focus on the research topic and less likely to digress into unrelated topics

Explicitly solicit the interviewee’s partnership. Tell them that you need their help. If there is a significant difference in social status, it is sometimes helpful to adopt the persona of the student or apprentice in a “teacher-student” or “master-apprentice” relationship. By demonstrating your respect and admiration, you may benefit by more detailed narratives and explanations. By contrast, at other times, an interview may produce the best results by being confrontational—pushing an interviewee to go into greater depth, or showing that you are not willing to accept self-serving glosses.

Ask only one question at a time. Give the respondent time to collect his/her thoughts before answering. Show patience. Don’t ask questions simply because you are curious. All of the questions should relate to the research aims. Questions motivated by personal interest immediately mark the interview as unprofessional. Without a professional manner, respondents will begin to doubt that the contents of an interview will remain confidential, and so will resist providing details they might find embarrassing.

In questionnaire research, it is common to ask simple questions near the beginning of the questionnaire; Such questions might include questions of age, musical training, etc. However, in open interviews, it is best to leave these questions to the end of the interview. Beginning an interview with a series of simple factual questions sets the wrong tone. The interviewee is likely to respond to all of the subsequent questions in a perfunctory, factual manner rather than using a conversational or narrative approach.

Discourage respondents from talking in generalities. You want to hear about their specific experiences. Also, discourage respondents from talking about the experiences of others—like friends or partners. This is second-hand information, and the main attraction of interviews is collecting detailed first-hand accounts. Discourage respondents from offering or discussing theories—unless this is part of the aim of the research. Keep the discussion concrete. For example, instead of asking about their concert-going experiences in general, have them recall their most recent concert-going experience. Phrase your questions to help the respondent focus on specific events: “Can you tell me about your most recent…” “Can you walk me through that event…?”

If the respondent recounts a story that is especially pertinent to the aims of study, don’t hesitate to press for more detail. One of the main advantages of open interviews is the opportunity to obtain such detailed accounts.

Provide “back-channel” feedback with frequent “un-huh,” “yeah,” etc. This demonstrates interest and tells the interviewee that you are not bored. Maintain a respectful and serious demeanor.

In general, let the respondent talk. People are usually more interested in talking about things that are important to them than things that are unimportant. So letting the respondent talk about matters they deem important is apt to be revealing. Moreover, people rarely have the chance to talk at length about their experiences; interviewees will appreciate the opportunity to tell the whole story.

Avoid prompting the respondent with your own words. For example, suppose you ask a question: “Was your argument with the stage manager a clash of personalities.” Don’t be surprised if the respondent then characterizes the argument as resulting from “a clash of personalities.” The words are yours, not the respondent’s, and so it would be inappropriate to quote the respondent as saying “a clash of personalities.” The final narrative should be in their words, not in your words.

Never judge someone’s behavior. Your purpose is to gather information, even if you find the relayed behavior obnoxious or immoral. If you are recording the interview, sometimes the recording gear may become an impediment: the interviewee may be uncomfortable talking about certain topics or people. At times you may need to turn off the recorder. If so, do your best to retain exactly what is said, and then make notes immediately following the interview. Your personal notes don’t have the same legal status as a recording or a transcript of a recording. So this may allow you to gather information about activities that might be considered illegal or socially unacceptable.

You are not the respondent’s friend or therapist. Never offer advice or criticize or judge a person’s actions. Avoid interpreting the respondent’s narrative. If an interviewee becomes emotional, don’t abandon your role as a professional. Be compassionate and sympathetic but maintain a professional image. Offer quiet sympathy, but don’t offer advice, don’t pass judgment, and don’t respond with your own confession. Encourage a sense of collaborative partnership in creating a productive and informative interview.

The cardinal sin of interviewing is to talk about yourself. Resist the temptation to recount your own experiences. You may have your own story that reinforces a story just told by the respondent. In ordinary conversation, it would be appropriate to take turns, and tell your story. However, this is not the purpose of a research interview. Don’t talk about yourself. Don’t tell your own stories. Leave the story-telling to your respondent. The principal role of the interviewer is to listen sympathetically, and to encourage the respondent to speak. Aim to be unobtrusive.

As the interview winds-down, ask the respondent for leads or recommendations about other possible respondents you might recruit for the project. That is, solicit information for snowball sampling. Ask whether the respondent knows other people you have interviewed. This will help you determine the heterogeneity of your sample of interviewees. End the interview by expressing your gratitude. Tell the respondent that you might have some follow-up questions in the future, and ask whether it would be okay to contact them again later.

For informants with considerable knowledge, more than one interview may be appropriate. When a respondent is especially informative, don’t be afraid to ask for more additional meetings. Try to avoid cramming everything into a single interview, unless you know that further interviews are impossible. With more interviews, the respondent is apt to become more comfortable with you. Experienced interviewers know that with more interviews, more is revealed. For example, a person is more likely to admit to illegal activities, or to portray themselves in a less-than-ideal light.

Markers

Robert Weiss, an expert on open interviews, talks about “markers.” A marker is “a passing reference made by a respondent to an important event or feeling state.” (Weiss 1995, l. 1469). For example, the respondent might mention the break-up of their band, or make passing reference to an argument with a conductor. In normal conversation, markers are invitations for the conversation partner to pursue the matter further. Markers usually indicate the respondent is eager to talk about the matter (otherwise it wouldn’t be mentioned).

In ordinary conversation, it is always appropriate to follow up on markers. However, in a research interview, a marker topic may take the interview in a direction that is irrelevant to the aims of the researcher. Here, the interviewer needs to exercise some judgment. If the marker is germane to the research goals, then ask the respondent to expand: “You mentioned the break-up of your band; could you say more about that.” If the subject of the marker is not relevant, then the interviewer should simply let the marker pass. Or, if the interviewer is eager to establish a closer relationship with the respondent, pick-up on the marker, even if the subject is a distraction from the research goals.

Interview Advice

We might summarize the above advice as follows:

Prepare and use an interview guide. Update the guide as you continue doing interviews.
Explain the aims of the research to the respondent.
Only ask one question at a time.
Don’t ask questions out of idle curiosity, or questions that are irrelevant to the research goals. Such questions will make you appear less professional.
Discourage respondents from talking in generalities. Tell the respondent you are interested in specific experiences.
Provide “back-channel” feedback with frequent “un-huh,” “yeah,” etc. This demonstrates interest and encourages the respondent to speak.
Be vigilant for markers that signal topics that are pertinent to the research goals. Don’t pick-up on markers that are irrelevant to the research goals.
Avoid “seeding” the respondent with certain words or phrases. Ensure that the narrative is in their words rather than your words.
Never judge someone’s behavior. Your purpose is to gather information, even if you find the relayed behavior obnoxious or immoral.
Never offer advice. You are not the respondent’s therapist or friend. Simply listen, and offer a sympathetic ear.
Don’t talk about yourself. A major attraction for respondents is the personal attention paid by the interviewer. People feel validated when they receive someone’s uninterrupted attention.
Aim to be unobtrusive.
Ask the respondent for leads or recommendations about other possible respondents you might recruit for the project. That is, solicit information for snowball sampling. 14. End the interview by expressing your gratitude, and asking the respondent whether it would be possible to follow-up with further questions in the future.

Analyzing Free Responses

When it comes time to analyze free response data, the aim of analysis depends on the goals of the interviews. For investigative journalists, detectives and historians, the goal will be to uncover the sequence of events, and to draw attention to the ways in which different informants interpret the events differently. For the clinical psychologist or psychiatrist, the aim of interview analysis might be to formulate a diagnosis and to assess the conditions that might lead to a helpful intervention. For sociologists and political scientists, the aim of analysis might be to chronicle the effects of particular policies. For anthropologists and ethnomusicologists, the aim of analysis might be to glimpse the cultural factors that shape thought and behavior.

The analysis of interviews may be highly structured (such as following the DSM procedures for diagnosing mental disorders), or may be highly informal. Just listening to (or reading transcripts of) the interviews may be highly informative without any analysis. For example, a series of vignettes of people describing their experiences of a hurricane may stand on their own, without the need to interpret or analyze them. An interviewer might choose to tell a story simply based on the overall impressions arising from the interviews.

There are few formal methods for analyzing interviews. However, most formal analyses include the following elements:

Transcribe the recordings
Proofing the transcripts
Breaking up the transcripts into topic-units or vignettes
Sorting the comments into topic categories
Coding the comments within each topic
Collating and summarizing the coded material in each topic

We expand on these analysis elements below.

Labeling topics

After transcribing and proofing the transcripts, it is common to mark-up the transcript by labelling topics of conversation. For example, a researcher might label a transcript segment “dressing before going to a concert.” Most topic labels will emerge in a straightforward manner from the transcript. However, in light of the goals of the research, certain topics will be regarded as important even before you begin your analysis. Many of these topics will already be evident from the interview guide. For example, a series of interviews related to the concert experience might include becoming aware of an up-coming concert, the decision to attend, inviting friends, securing tickets, travel to the concert, pre-concert activities, etc. Label interview segments according to these topics.

As you label the topics of conversation, new topics will emerge from the interviews themselves. For example, concert memorabilia (T-shirts, buttons, posters, etc.) may emerge as a pertinent topic—even though it was not anticipated in the interview guide. In general, be vigilant to allow the emergence of new topic from the interviews themselves.

Sorting and Coding topics

Having labelled the topics, the researcher next typically extracts similar topic materials from various interview transcripts and copies them into topical folders. For example, all of the discussions regarding intermission activities may be collated together.

Once the materials are collated, they may be coded according to some criteria. These criteria may be a priori ideas—with a formalized coding scheme—, or they may emerge from the transcripts themselves. For example, the researcher may become aware that it is common for people to change their concert plans for various reasons. The researcher might then elect to code the material according to the reason for the change. For example, you might code comments as “Decided to pursue an alternative activity because the concert was deemed too expensive.” Or “Decided to pursue an alternative activity because transportation to the concert venue would be difficult.” Or “Decided to pursue an alternative activity because of the failure to find a companion.” etc.

In some cases, the coding may be uncontroversial. However, in other cases, the researcher may be “reading between lines”—inferring something that is not explicit. In order to enhance coding validity, it may be appropriate to recruit an assistant or collaborator who independently codes the transcripts according to the same criteria. The researcher might then only rely on codings that agree between the two coders.

Analyzing Coded Topics

Analysis of interviews is not guaranteed to lead to any coherent conclusions. Sometimes, the researcher can summarize the interviews by simply relaying various anecdotes, without offering any interpretation. At the other extreme, an obvious over-arching pattern may be evident in the interviews. In this case, the researcher may be able to summarize the interviews by proposing or describing a broad theory. More likely, only some aspects of the interviews will admit a generalized interpretation. Coded data might contribute to the plausibility of certain interpretations. For example, a researcher might conclude (on the basis of coded transcripts) that the most common reason for deciding against attending a concert is the failure to interest a companion.

In dealing with descriptions of events, both detectives and historians will hope to be able to use the interviews to assemble a coherent descriptive narrative. If all of the interviews converge on a coherent story, then the task of weaving together the various perspectives offered by different informants should be straightforward. However, if there are discrepancies between informants, then both the historian and the detective will need to draw attention to the different interpretations of what is nominally the same event.

Freelisting and Pile Sorting

The freelist is, I believe, the single most powerful and informative systematic data-collection technique available, because it is easy to use; you don’t need many informants; and it offers a powerful emic snapshot of the cultural domain that you are interested in studying.

— Victor De Munck (Munck 2009, 47)

Freelisting is an exploratory research method intended to give a quick overview of some domain of human thought within a cultural context. It is usually used when a researcher first explores a cultural phenomenon. For example, a researcher might be interested in how people in some culture conceive of foods and food preparation, or a scholar might be interested in how people distinguish different forms of dance.

Freelisting typically involves brief interviews with a relatively small number of informants. The interview consists of posing a question of the form: “What sorts of X are there?” Or: “List all the Xs you know.” Sample freelist instructions might include the following:

List all the colors you know.

What are bad things a person might do?

On what occasions do people sing and dance?

List the kinds of activities people do while listening to music?

List all of the musical instruments you know.

What sorts of conductors are there?

For each interview, the researcher records the listed items in the order in which the informant mentions them. For example, when asked to list all of the musical instruments you know, a person might begin: piano, guitar, violin, trumpet, etc.

Once the lists are acquired, an aggregate list is assembled in which the items are ordered from the most commonly mentioned to the least commonly mentioned. Usually, this aggregate omits those items that were mentioned only once. The items that are culturally important will tend to be those mentioned most frequently.

In general, people tend to begin their lists with the cognitively most salient items. So culturally important items are more likely to appear near the beginning of the list. (For Western musicians, instruments like the contrabassoon and didgeridoo are likely to appear rarely and toward the end of the list.) A second version of the aggregate freelist will be ordered according to the mean rank-order. This can be calculated by numbering each item for each informant, and then averaging the rank numbers. For example, if one person lists the guitar first, and a second person lists the guitar second, then the average rank for this instrument is 1.5.

A complication arises when there are items that might be regarded as synonyms. For example, when asked to identify occasions when music is played, responses might include “Easter,” “Good Friday,” “Easter vigil,” “Easter Sunday,” “Orthodox Easter,” and so on. The question is whether these should all be coded as a single item, or as two or more items. Similarly, one person might enumerate soprano saxophone, alto saxophone, tenor saxophone, etc., whereas another person might simply identify “saxophone.” In order to minimize researcher bias, it may be useful to recruit one or more informants to resolve such situations.

Free-listing results are more trustworthy when there is high inter-informant agreement in the elicited terms. The proportion of shared terms across participants should be reported along with the frequency and rank-order lists. In general, an average inter-informant agreement of 2/3 or more may be considered good or adequate. That is, if the informants typically share two-thirds or more of the listed items, then one can have greater confidence that the lists are representative of a plausible cultural norm.

Freelisting is relatively quick and easy to do. In addition, freelisting emphasizes culturally-grounded, emic conceptions and tends to reduce the influence of researcher bias. However, freelisting also has a number of disadvantages. De Munck (Munck 2009, 66) notes that freelists elicit terms that are at “the top of people’s heads” or on the “tip of their tongues.” With more careful reflection, consultants may change their views, and wish to offer more nuanced distinctions. Once again, freelisting is typically an exploratory method.

Pile Sorting

It is common for freelisting to be followed by pile sorting. Pile sorting is a task in which respondents are asked to sort items into categories. Two types of pile-sorting tasks are usually distinguished:

Constrained pile-sort, in which the informant is told the number of categories,
Open-ended pile-sort, in which the informant is free to determine the number of categories.

Pile sorting can be done with all kinds of items. For example, pile sorting might be done with actual physical objects such as kitchen utensils, fruits and vegetables, bird feathers, or spices. Alternatively, printed cards may be used, such as photographs (e.g., different facial expressions), or written words. Photographs are often easier for a respondent to sort than the actual physical objects. In the case of sorting sounds, a computer display is helpful, where icons representing different sounds can be moved around the screen to form various groups.

It is worth noting that pile sorting is typically time consuming. It is important to recruit participants who have time, and potentially to provide a gift or offer compensation for their participation. Pile sorting can prove to be particularly onerous if there are rather few items, or an especially large number of items to be sorted. If the items used for pile sorting come from an earlier freelisting task, it may be useful to limit the number of items to those mentioned two or more times, or three or more times, etc.

In pile sorting tasks, the temptation is to tell the informant to sort items into “whatever categories you like.” This can lead to arbitrary criteria, such as sorting instruments according to their weight or color. In instructing the informant, it can be useful to explicitly reference culturally normative categories. For example, one might instruct an informant to sort the items “in a way that would make sense to a friend.”

In order to minimize order effects, the items are usually randomized for each participant. Participants are sometimes suspicious that there is some special trick or ulterior motive in the task, so when cards are used, it can be helpful to shuffle the cards in the presence of the participant.

Once the sorting task has been completed it is useful to have the participant describe each of the categories. Finally, encourage the participant to provide a single word or short phrase as a descriptive label for each category.

Hierarchies

The categories people use are often hierarchically organized. For example, a Westerner might divide foods into things you drink and things you eat. Things you eat might be subdivided into vegetable and animal sources. Animal sources might be further subdivided into meats and milk products, and so on. It is often useful to understand the sorts of mental taxonomies that might exist for certain phenomena.

In pile sorting tasks, one approach to better understanding mental taxonomies is to follow the pile sorting by a subjective stepwise clustering task. Suppose an informant has sorted the items into 12 categories. Having recorded the results, the researcher might then instruct the informant as follows.

You have 12 piles here. Suppose I told you that I want you to reduce the 12 to just 11. What two piles are the most similar? What two piles would you put together?

This reduction technique is then successively repeated until only two piles remain. Although this procedure is more time-consuming, it can better reveal latent hierarchies in the mental organization of the categories. In addition, it can reconcile the usual divergence between “lumpers” and “splitters”—people who tend to favor small numbers of groups and those who favor a larger number of groups. Note that it may be useful at each stage to ask the informant to re-label one or more categories.

An advantage of the stepwise clustering approach is that you may be able to better reconcile different pile-sorts by different informants. For example, one informant might produce 18 clusters, another informant might produce 13 clusters, and a third informant might produce 9 clusters. However, when asked to reduce the number of clusters, the researcher might find that at 8 clusters, all three informants are in very close agreement. The differences beyond 8 clusters might represent less culturally salient distinctions.

In a constrained pile-sort, the researcher tells the informant the number of categories for sorting. This approach can be used for hypothesis testing. For example, in previous work, the researcher may have observed a tendency for informants to regard the items as falling into the same five categories. As a test of this observation, the researcher may explicitly ask several informants to sort the items into five categories, without offering any additional instruction. This can be used to test whether the five categories are culturally robust.

Especially when working with people from an unfamiliar culture, it may be difficult for the researcher to infer why a particular item has been assigned to a particular category. In these cases, it can be insightful simply to ask a participant to explain why they have assigned a particular item to one group rather than another.

Pile sorting might be followed by more formal statistical methods. For example, pile sorting might be followed by multi-dimensional scaling (MDS) and/or formal hierarchical clustering (to be discussed later). Both MDS and hierarchical clustering invite the researcher to interpret the meaning of the resulting dimensions or the meaning of each cluster. In order to minimize cultural bias, these interpretations should not be done by the experimenter. It is more appropriate to recruit one or more informants to offer possible interpretations.

De Munck (Munck 1998) used freelisting, pile sorting, and multidimensional scaling to examine the concept of romantic love among Lithuanians. He found that Lithuanians tend to think of love along two dimensions—a negative-positive dimension and a real-fantasy dimension. He then tested this interpretation with a follow-up experiment in which Lithuanians were explicitly asked to rate a series of love-related terms along the two dimensions. He found that Lithuanians tend to view romantic love according to four prototypes: fantasy-good love, fantasy-bad love, real-good love, and real-bad love.

Advantages and Disadvantages of Interviews

In general, empirical researchers have tended to underestimate the value of exploratory interviews. Open interviews provide quick access to rich ideas and interpretations of a range of human experiences. Interviews immerse the researcher in a dense world, full of relationships, contexts, nuanced distinctions, competing interpretations, and (sometimes) fascinating anecdotes. It would take many decades for hypothesis-driven research to achieve the sort of complex picture that emerges from open interviewing. Interviews are apt to generate a lot of ideas with relatively little effort.

Interviews are especially valuable when investigating something new. When an anthropologist visits a novel culture, it is important to be open to whatever it is that is distinctive, unique or surprising about that culture. Of course anthropologists have their own agendas. An anthropologist might be interested in child-rearing practices or religious observances. However, a good anthropologist will be alert to other possibilities. For example, the child-rearing or religious practices might be very similar to those found in other cultures, while the medicinal practices or decorative arts might be extraordinary. A researcher doesn’t know what is important until they are exposed to lots of possibilities. There are times when the research agenda should not be set by the researcher in advance. This is the whole purpose of exploratory research. Open interviews provide one of the very best methods for exposing the researcher to unexpected phenomena.

At the same time, qualitative researchers have tended to overestimate the value of open interviews. First, interviewers rarely question the accuracy of the information provided by an interviewee or informant. Although qualitative researchers acknowledge the problems of validity, they rarely (if ever) verify specific claims by examining independent sources. Interviews are only as good as the accuracy of introspection and memory. Validity is not simply an issue of respondents telling the whole truth. Research on eyewitness testimony is sobering. For example, research by Elizabeth Loftus and her colleagues has established the extraordinary fallibility (and suggestibility) of human memory (Loftus 1979; Loftus, Doyle, and Dysert 2008). Moreover, validity involves more than just avoiding conscious lying. Dozens of psychological experiments have shown that people often have very limited self-knowledge. We often have no insight into our own motivations. We all engage in unconscious self-justification, and freely confabulate spurious explanations for things we don’t understand (e.g., Wilson 2002; Ariely 2009; Kahneman 2012).

Research in social psychology suggests that parallel problems plague the interviewer. For example, research on hiring practices has shown that unstructured interviews with job candidates actually leads to worse hiring decisions than if no interview occurs (Ross and Nisbett 2011). As interviewers, we grossly over-estimate our ability to judge character. Past experience (as documented in the resumé), the ability to write (as documented in an application letter), and confidential letters of recommendation are much more predictive of the future success of a candidate than the impressions gained from an open interview (Ross and Nisbett 2011).

Traditionally, marketing researchers have made the greatest use of interviews in their work. Because of the large amounts of money involved, consumer researchers have been most motivated to “get it right.” Yet in recent years, marketing researchers have become highly suspicious of consumer interviews and focus groups precisely because of the inability of people to accurately introspect (Graves 2010). People are mostly unaware of their own motivations, and will confidently confabulate answers to any question a researcher may pose.

In his otherwise fine book on interviewing methodology, Robert Weiss (1995) offers the following regarding the crucial issue of validity:

Sometimes, we can check on the validity of a respondent’s account by interviewing other respondents. Occasionally, there are records we can look to for corroboration. But for the most part we must rely on the quality of our interviewing for the validity of our material. Ultimately, our best guarantee of the validity of interview material is careful, concrete level, interviewing within the context of a good interviewing partnership.
— Robert Weiss

In short, qualitative researchers don’t have good ways to address the problem of validity. There is a notable tendency among qualitative researchers to avoid discussions about validity.

Because of the close face-to-face interaction, a second problem is that interviews are highly susceptible to investigator bias as well as demand characteristics. Interviews are the virtual antithesis of the double-blind procedure. As a result, confirmation bias is especially problematic. The open interview method has proved especially attractive to those researchers who are motivated by a particular social cause. The more strongly held the political beliefs, the greater the likelihood that the researcher will rely solely on interview methods. Again, qualitative researchers are generally aware of these difficulties, however, little work has been carried out to either measure the magnitude of this bias, or to develop techniques to minimize these confounds.

As noted earlier, people find personal stories more compelling than raw statistics. News stories and documentaries are often more effective when they begin or end with a concrete personal example. Interviews can provide vivid case examples whose influence far exceeds the reporting of representative statistics. A danger is that examples can typically be found that seem to illustrate opposite cases. One can find stories of human heroics as well as human cowardice. One can find stories where seat-belts save lives and stories where wearing a seat-belt causes a person’s death. Good research is needed to resolve which account is the more common or representative. Readers and viewers tend to interpret case studies as typical, and regard them as holding important general lessons. Empirical work using representative samples can benefit by recounting an anecdote that illustrates or exemplifies what is found. However, relaying anecdotes that are contrary to the common or typical experience can amount to propaganda. Ideally, exceptional cases will lead to careful thought about how to promote more good results, or minimize bad results. However, policy based on exceptional cases can sometimes make matters worse.

A third and more onerous problem is that qualitative researchers tend to view open interviews as confirmatory research rather than exploratory research. Ideas that arise from interviews are rarely tested. Recall the problem of double-use data where the same data should not be used to both inspire a hypothesis, and as independent evidence in support of that hypothesis. However, it has been common for qualitative researchers to use narrative anecdotes in precisely this way. Said another way, the researcher who relies exclusively on open interviewing is a researcher who never invites failure. Interviews are important starting points for research. But, for the conscientious researcher, interviews cannot be the end point. Open interviews belong as one part in an explore-then-test research program.