24 Qualitative and Quantitative Research
(Modified on 2026-08-12)
2,019 words
In the remainder of this book, we will spend a significant amount of time and energy talking about and working with numbers. We’ll spend lots of time modeling research variables using numbers and equations—e.g., using quantitative methods. Quantitative methods are incredibly powerful, and useful; However, quantification can also be extremely reductive. As always, we should regard our methods with skepticism. Many musical phenomena are extremely difficult to quantify, and forcing them into a quantified framework may be problematic. Qualitative research methods, which we’ll also explore, should not be forgotten.
A Mutual Disregard
Qualitative and quantitative researchers tend to regard each other with mutual contempt. This conflict is rooted in the differing philosophies of reductionism and holism. Many people, especially in the humanities, can be very hostile to reductive quantification. On the other hand, plenty of scientists and engineers are also dismissive of qualitative, humanistic work, which they regard as “wishy washy,” vague, or pointless.
In Favor of Qualitative Methods
Especially with regard to arts, culture, creativity, and personal expression, quantification can be a bizarre exercise. Can you really rate your musical experience on a scale from 1 to 10? Humanities scholars tend to embrace art and culture in all its messy, holistic complexity. They argue that we can only understand things like “groove” by feeling them, talking about them, and sharing them in a real human context. Inventing a quantified operational definition for “groove” would be contrary to the humanistic spirit.
Indeed, humanists will often complain that empirical researchers are obsessed with numbers and measurement, at the expense of real, nuanced understanding.
Another the criticism of scientific research is that it pretends to be objective, but that our theories and hypotheses are inherently subjective and biased. After all, who chooses which questions to ask? And which hypotheses to test? We do. Especially in the case of cross-cultural research, the questions, theories, conjectures and hypotheses entertained by the researcher already echo the world-view of the researcher rather than the world-view of those of another culture. In short, a priori hypothesis testing is susceptible to a form of egocentric bias.
We should note that there is a historical basis for these concerns. In the 19th and 20th centuries, the scientific and engineering communities were dominated by white men who often cluelessly conflated their own narrow, Western-centric, sexist, and racist viewpoints with “science.” Often the very same “genius” scholars who contributed to enormous scientific advances in physics or engineering, would turn around and present obviously flawed racist/sexist beliefs as if they had the same scientific validity. Using “quantified” metrics, many traditions were “shown” to be “objectively” inferior to great Western art. There was also a general trend to underestimate the complexity of human behaviors; many “scientists” were obsessed with finding simple laws to explain human behavior and culture, equivalent to \(e = mc^2\). Entire generations of otherwise intelligent white men failed to see the many glaring problems in these approaches and conclusions.
In the 1960s–1970s, the post-modern movement arose as a backlash to the racist, imperialist “science” which still dominated academia at the time. Post-modernism was (and is) characterized by a deep, deep suspicion of “science” being applied to human culture, and post-modern thinking is still extremely significant in the humanities, including fields like ethnomusicology. Unfortunately, the post-modernists often failed to see that the problem was not quantification and scientific methods, but that these methods were poorly applied by researchers who were blind to their own racist assumptions.
This last point is important: when humanistic and scientific researchers criticize each other, what they are usually really criticizing are bad research practices. Lack of skepticism; Lack of self-criticism; Lack of clarity. Plenty of scientists are convinced that their work is objective; Plenty of scientific research, especially related to human behavior and culture, is poorly done. Scientists with a shallow understanding of music use weak operational definitions, or make invalid assumptions about music. For example, these days many machine-learning researchers train models using musical “ground truth” datasets that they poorly understand. Conversely, lots of humanistic research is wishy washy—relying on vague, but “smart sounding” rhetoric and buzzwords.
In Favor of Quantitative Methods
Humanists are absolutely right to point out that our choice of hypotheses (and operational definitions) may represent a source of bias, especially when studying another culture. While quantitative researchers acknowledge that a priori hypotheses may exhibit cultural bias, we are also skeptical that any researcher can avoid cultural bias. Spending time in a culture, and claims to sensitivity, do not, in themselves, ensure that the researcher’s ideas and interpretations aren’t culturally biased. Indeed, ethnographic studies within each era (1920s, 1940s, 1960s, 1980s, etc.) resemble each other in the kinds of overarching interpretations and research concerns (e.g., kinship terms, ethnobotany, cultural relativism, globalization, etc.). This suggests that ethnographers are motivated more by intellectual trends in Western anthropology than by the individuality of different cultures.
Showing You Are Wrong
In my (Huron) first year as an undergraduate student, I spent time reading some of the classic feminist writers—including works by Betty Friedan, Germaine Greer, Susan Brownmiller and Gloria Steinem. As a man, I frequently felt defensive when reading these works. From time-to-time I thought various claims were exaggerated or simply wrong. I recall reading one passage which discussed how men tend to dominate mixed conversations. My first response was skeptical. I’d grown up in a society where women were regularly portrayed as “chatterboxes” and “gossips.” I personally knew a couple of women whose monologues were perfectly capable of filling-in whatever air-time was available.
My skepticism led me to perform what may have been my first empirical study. One afternoon, I simply sat in a student lounge eavesdropping on different conversations around me. With pencil and notebook in hand, I watched the second-hand of a wall-clock, and recorded the amount of time men and women spoke in various groups. I also recorded the number of men and women participating in each conversation. After an hour or so, I tallied up the results. There was no question about it. In every mixed group that I monitored, men dominated the conversation by amounts that were disproportionate to their numbers.
It can be surprisingly hard to get people to change their views. Anecdotes can be easily dismissed as “one-off” exceptions to our beliefs. It takes constant battering with evidence, before people will finally acknowledge that their beliefs may be incorrect. Despite the thousands of conversations I’d heard in my life to that point, somehow I’d failed to notice what was obvious to most women. The numbers in my notebook were not consistent with the “chatterbox” image of women. The numbers told me that I had been wrong. Ultimately, the numbers changed my attitude.
In research, the principal benefit of quantitative methods is that they can provide compelling evidence that you’re wrong. The biggest problem with qualitative research is that little or no testing takes place. In particular, ethnographies commonly use the same observations both to formulate an idea and as evidence in support of that idea (double-use data), so their ideas cannot be falsified.
Although the essence of good research is to invite failure, we may be slow to recognize failure when we see it. The humble act of counting (quantification) is the clearest way to invite failure. Recall that we recognize failure by drawing a line in the sand. Drawing a line means that we must have some way of determining on which side of the line the observations lie. Numbers provide a useful tool for defining failure. \(\alpha = .05\), and \(p\) is either below or above, there is no wiggle room. We count and measure, not because of some quantitative obsession, but because counting helps us recognize self-deception. Quantitative methods are valuable checks against our own intellectual enthusiasms.
A Satellite Story
Quantification is also useful, because it can help us find patterns that are invisible to our naked senses.
Counting can help us discover phenomena that might otherwise be invisible. In the early years of the space program, NASA launched a simple satellite that used a radar beam to image the planet’s surface. The satellite produced very accurate readings of the height (elevation) for each point as it orbited the earth thousands of times. The purpose of satellite was to help geographers produce more accurate topographical maps of the earth.
About 3/4 of the earth’s surface is covered by water, so most of the data collected by the satellite was precise elevation measures of a given point on the ocean as the satellite passed overhead. Since the goal of project was to create accurate topographical maps of the landmass, the ocean data was considered useless. Moreover, each time the satellite passed over a particular spot in the ocean, the water level was different (because of the effects of waves and tides).
Nevertheless, some bright scientist asked, “What would happen if, for a given point in the ocean, we averaged together all of the elevation measures?” Repeated measures would tend to erase the effects of the waves and tides. You might think that this would result in a smooth ocean surface. But when the researchers did this, they found that there were regions of the ocean where the average sea level was relatively high and other regions where the sea level was relatively low. It was only possible to “see” these regions because of averaging large amounts of (highly variable) data. The averaging allowed the effects of waves and tides to be eliminated.
Qualitative vs Quantitative
So what’s the difference between qualitative and quantitative research anyway? I would argue that the real distinction to be made is how much, and how, we assume or impose structure on information. Qualitative research focuses on “raw,” messy, complex, holistic information. The more we impose a model on information, the more “quantitative” it becomes.
Qualitative data collection is essentially observing, interviewing, and interacting with humans, preferably in situ. Gathering the “raw data” of human interaction/behavior, using natural language in all its power and messiness. Qualitative research, thus, tends to be exploratory and/or descriptive. Qualitative work may, nonetheless, be guided by a priori hypotheses, ideas, or hunches. From these we can impose some structure on the data through our sampling approach (who do we talk to) and the questions we ask.
Truly qualitative (humanistic) work focuses on freely interpreting information and incorporating observations into a narrative or theory. Techniques for actually “analyzing” qualitative data immediately turn somewhat quantitative. Most qualitative analysis begins with a coding scheme: some kind of process for reducing qualitative information into levels of concrete variable. Basically an operational definition. Coding schemes may be a priori, or may be more dynamically derived through engagement with the raw data. Typically, “coding” is done by humans, but it may be automated. When done by humans, it is best-practice to have multiple raters code the same data, to check for reliability. None of this is fundamentally different than measurement/operational definition in “quantitative” research. It is all a matter of degree. How much structure do we impose, and when?
In the chapters to come, we will see that there are many areas of research methodology where we can choose between using relatively exploratory, “free form,” or “open-ended” measures/models or using restricted, precise, hypothesis-driven measures/models. We already know about the difference between exploratory and hypothesis-driven research. We’ll soon consider other contrasts, like
- Free response vs forced-choice measures in human research;
- Optimization vs hypothesis-testing in statistics;
- Exploratory vs confirmatory analyses;
- Unsupervised vs supervised machine learning;
- and many other examples.
Each of these pairs straddle part of the continuum between “qualitative” and “quantitative” methods. This continuum can be viewed numerous ways:
| \(\leftarrow\) More Qualitative | More Quantitative \(\rightarrow\) |
|---|---|
| \(\leftarrow\) more open | more restricted \(\rightarrow\) : |
| \(\leftarrow\) more broad | more narrow \(\rightarrow\) : |
| \(\leftarrow\) less precise | more precise \(\rightarrow\) : |
| \(\leftarrow\) more complicated | more simplistic \(\rightarrow\) : |
| \(\leftarrow\) more nuanced | more general \(\rightarrow\) : |
| \(\leftarrow\) more exploratory | more hypothesis-driven \(\rightarrow\) : |
Both approaches have their strengths and weaknesses. As conscientious researchers, our goal should be to integrate the best research practices from all sources. Strong music research can incorporate both qualitative and quantitative insights. As music scientists and engineers, we are in a unique position to bridge the gap between qualitative and quantitative study of music.