17 Scientific Methods
(Modified on 2026-08-11)
3,520 words
Once we’ve established a research question, be it descriptive or explanatory, we may conduct a research inquiry to answer it. Each inquiry, or “study,” will involve different methods: procedures we follow to formulate predictions, gather data, analyze results, make conclusions, etc. As part of the humble and transparent rhetoric of science, we must always precisely and honestly account, explain, and justify the methodologies we use in our inquiries.
There is not one “Scientific Method”—each research inquiry needs specific methods designed for it. Still, there are many standard research methodologies, terminologies, and traditions; Part of your education is learning the methodological standards and norms of the (sub)fields that interest you. Methodological standards exist because they are designed to avoid specific known fallacies or logical errors, in particular, to counteract the natural confirmation bias which we all suffer from. Established approaches have been vetted and justified in and by past research. Using standard approaches makes our lives easier because we don’t have to invent (or justify) new methods every time we conduct an inquiry. However, this does not mean we can turn off our own critical thinking or skepticism; standard approaches are never perfect and the standard approach may not be appropriate for our particular research.
In fact, you will find that many methodological standards are arbitrary, and in some cases simplistic or dumb. However, standards—even if not ideal—give us less freedom to make decisions, giving us less “wiggle room” to, for example, tip the scales in our favor. A good example is the statistical “alpha level” \(\alpha = .05\), which is used universally for evaluating statistical tests. There is nothing magical about the number \(.05\). The notion that a \(p = .049\) and \(p = .051\) should be interpreted differently, is kind of silly—as many people will complain/argue. However, if we were free to pick any \(\alpha\) level we want, there would be concern that we, perhaps unconsciously, would choose an \(\alpha\) that favors the results we want. The arbitrary standard \(\alpha = .05\) is a goal-post we cannot move—a line in the sand we cannot cross—, enforcing falsifiability.
The methodology section of a paper is where (most of) the details of our research procedures are presented: operational definitions for all variables, measurement and sampling procedures, experimental design, etc. Unlike the introduction, the methodology section should be dry and formulaic, reporting the appropriate details of your study as is typically done in your (sub)field. For example, the methodology section of a music psychology paper will inevitably have one paragraph reporting how many participants there were and their basic demographics, and another paragraph describing the “interface” that participants used, including details like the model of headphones used.
Remember to write the paper first—by writing your methodology out in advance, it forces you to think through all the details of your project. What’s more, if you wait until after conducting the study to write the methodology, you may forget many details—you’ll also be so burned out at that point, the last you want is to go back and write the boring methodology part!
Validity
When designing a scientific inquiry, our goal is to convince our audience of skeptical scientists that our methods are not flawed by unfounded assumptions, errors, confounds, biases, or logical fallacies. We must convince them that the deductive and/or inductive inferences we make are valid. Validity is the key concern of research methodology: valid methods result in justified knowledge.
There are three broad types of “validity” recognized by scientific methodologists:
- Internal validity
- Construct validity
- External validity
We will learn about each of these in turn over the next few weeks.
Approaches to Scientific Inquiry
We have already discussed the fundamental distinction between hypothesis-driven and exploratory research, and the advantages of combining them in the explore then test paradigm. As we know, hypothesis-driven research is the “gold standard” in convincing scientific rhetoric, because only concrete predictions can be falsified. Unfortunately, hypothesis-driven research is hard. It requires that you have clear (a priori) theoretical ideas about a topic, and that you are confident enough about these theories to use them as the driving force in your research. Hypothesis-driven research is also, by design, highly constrained. What’s more, there is a risk that preconceived theoretical ideas may overly restrict your view of the topic, especially when the area of inquiry hasn’t been explored much before. If your theory is too restricted, it may bias you away from the truth.
Compared to hypothesis-driven work, exploratory research is often easier to prepare; Exploratory research, is by its nature, better suited to description than explanation, and description is (frankly) easier than explanation. As we know, exploratory research ultimately is limited in the rhetorical strength of its arguments, as you cannot prove/disprove propositions based on induction alone. However, it also requires fewer preconceived notions, is absolutely necessary when studying topics with less theoretical background. The principle value of exploratory research is that it can lead us to new (or better) questions, which in turn, can lead to the development of explanatory theories to test.
Observation
As we know, inductive empiricism is essential to science. Most scientific studies, thus, rely on observation and description. “Observation” can mean a lot of things, especially in music technology. Examples of music research observation include:
- Computing features from audio files or extracting genre tags from the Spotify API.
- Asking humans to perform a task or answer questions and noting their performance/responses.
- Interviewing someone, and taking notes.
- Video recording music performers or their audiences.
- Scraping millions of lyrics or social media posts from the web.
Scientific observation is often done in controlled, artificial environments, like within a laboratory. However, the conditions we create in a laboratory may not reproduce the natural behavior of people or other systems in the real world. This is the concern of ecological validity, which is an aspect of external validity. To improve ecological validity, we can also (with proper permissions) gather field observations: for example, observing human behaviors in natural environments, like in public places or concert halls. Observations on online-gaming platforms or social media are also forms of field observation.
Keep in mind that observation and description are never entirely passive—we are never fully neutral (“objective”) observers. As we will discuss in detail later, we decide where, what, and when we observe (sampling). We also decide how we measure and “code” (classify) our observations. (These decisions relate to construct validity, which we’ll read about later.) Thus, even in exploratory observation, the data never “speaks for itself.” Still, exploratory approaches generally project fewer preconceptions onto our observations than in theory-driven work, which is one of their advantages.
Correlation
We are rarely, if ever, interested in describing one phenomenon in isolation. We are typically looking for relationships between things that we observe: perhaps we observe people listening to sad music at the same time that we observe rainy or cold weather. It is this “covariation” or correlation between observations that we are really interested in describing. We say two things are correlated when there is some sort of connection or association between them. For example, music in the minor mode tends to be slower in tempo than music in the major mode; Although there are exceptions to this, in general, there is a correlation between mode and tempo. Observational research is often called “correlational research.”
If we observe that changes in phenomena A correlate with changes in phenomena B, we might postulate that changes in A cause changes in B. However, justifying inductive claims about causation is tricky, a problem known as internal validity. Recall that causation is a key ingredient of explanatory theories: saying that “X causes Y” or “X happens because of Y.” Thus, establishing internal validity is essential to explanatory research.
To support claims about causation, we can consider a few basic arguments:
- Temporal precedence—A always precedes B.
- Positive and negative correlation—the presence of A predicts B, and the absence of A also predicts the absence of B.
- Alternative explanations—there are no other factors which might influence both A and B.
From a rhetorical perspective, we can make a stronger causal argument if we theorize, a priori, that A would affect B before we actually gather data. In other words, we can predict a correlation then gather observations in an attempt to disprove it (this is called a natural experiment). If, on the other hand, we discover a correlation based on entirely exploratory observations, we need to regard any post-hoc claims about causation with healthy skepticism.
Ultimately, pure observation always lacks internal validity. Let’s consider why with a classic (but silly) example:
In a beach town, city officials observe that ice-cream sales are highly correlated with shark attacks. They conclude that eating ice cream must make you more susceptible to shark attacks. They theorize (post-hoc, of course) that the sugar content makes your sweat sweeter, attracting sharks.
Is the city officials’ conclusion reasonable? It could be true, but is weakened by the possibility of a (more plausible) alternative explanation: Both ice cream sales and shark attacks will tend to be higher on hot days (when people are more likely to go swimming and/or want ice cream) and lower on cold days. Temperature, a third variable, causes changes in shark attacks and ice cream sales, but the two are otherwise unrelated. This is known as the third variable problem, which generally destroys the internal validity of claims based purely on correlational observations.
There is another issue with the claimed causal relationship between ice-cream and shark attacks: direction. Why didn’t the city officials think that increased shark attacks cause an increase in ice cream sales? The causation could just have plausibly gone in the other direction, which again is always an issue in correlational work.
Manipulation
Fortunately, there is one compelling method to establish internal validity. Rather than simply observing passively, we intentionally manipulate the world around us. For example, we cause changes in phenomenon A and observe the resulting effect, if any, on B. Scientific inquiries which use manipulation and observation to establish internal validity are called experiments. Through experimental manipulation, we create a scenario where causation, not just correlation, can be falsified.
Let’s consider an example. Let’s say we are interested in how music affects “pro-social” behavior. A common approach to measuring pro-social behavior is to have a researcher (or hired actor) “stumble” and drop books in a public place, and to observe whether or not anyone stops to help them. To conduct an experiment, we might set up to repeat this procedure outside an elevator on a high floor of an office building—“stumbling” in front of people after they step out of the elevator. We could then manipulate the music being played in the elevator(s): for example, we could have classical music in one elevator and no music in the other. We could then compare how often people coming out of each elevator engage in pro-social behavior. By manipulating the music playing in each elevator, we are conducting an experiment. If the amount of pro-social behavior is different between people leaving the two elevators, we can make an argument that the music is causing the change in behavior.
In music tech we often perform experimental manipulations like:
- Changing the features of a live-coding system or electric-acoustic instrument.
- Changing aspects of musical stimuli, like mode, tempo, or genre.
- Changing the wording of questions on questionnaires.
- Changing (meta)parameters or input representation of a machine-learning model.
- Changing the algorithm used to control movements of a robotic arm.
Types of Studies
Based on the ideas presented so far, we now see that scientific inquiries can be classified into four basic categories based on two criteria: 1) is the inquiry hypothesis-driven or exploratory? and 2) is the work experimental or observational?
| Hypothesis-driven | Exploratory | |
|---|---|---|
| Experimental | True experiment | Exploratory experiment |
| Observational | Natural experiment | Observational study |
True experiments are designed to test causal hypotheses, and are thus the strongest rhetorical form in science—allowing us to falsify explanatory theories. The word “experiment” is often used to describe scientific inquiries in general, but only hypothesis-driven studies which engage in experimental manipulations are truly experiments.
In fields like biology, psychology, linguistics, and human-computer interaction, the majority of studies are true experiments, and other forms of inquiry are regarded with skepticism.
Natural experiments are hypothesis driven, but don’t involve any experimental manipulations—it’s actually a misnomer to call them “experiments,” but we do. This means they are falsifiable, but lack internal validity (strong arguments about causality). It is “natural” because we are observing nature, but not interacting with it. Natural experiments are also known as “quasi-experiments.”
Many music information retrieval and computational musicology studies are natural experiments, as well as most sociological “surveys” and questionnaires.
In observational studies, we gather data (observe) and analyze relationships, but don’t manipulate anything. They are also known as “correlational studies.” Observational studies are an essential way of gathering information to motivate research, and are the dominant form of exploratory research. However, they are always regarded with a grain of salt, as any conclusions made from them are necessarily post-hoc.
Lots of machine learning and “data mining” work is observational; large-scale surveys, like The American Soldier study, are also observational studies.
Exploratory experiments involve some kind of experimental manipulation, but have no hypothesis. They are very uncommon in real research—typically, if you are manipulating something, you have some expectation of what will happen, so it’s not exploratory.
In natural English parlance, the word “experiment” often refers to what we’d call an exploratory experiment: fiddle with something and see what happens. However, in actual research, a (true) experiment is prediction-driven.
Experimental Design
Many scientific methodology terms and concepts originate in the design of (true) experiments. However, most ideas/terms also apply to observational and exploratory studies as well;
Unfortunately, we often use experimental terminology even when we are doing non-experimental work. This can be confusing, so be warned!
In scientific studies, we refer to the various phenomenon we observe and measure as variables. A variable is anything you can measure which results in two or more possible measured values. Each value a variable can take is called a level. In many inquiries, we use binary independent variables, with two possible levels: music versus no-music; heavy-metal versus reggae; loud versus quiet; consonant versus dissonant; in-fashion versus out-of-fashion; and so on. However, can have any number of levels, even \(\infty\) levels—but we will talk about that in a later chapter, when we talk about quantifying variables!
Experimental variables
In an experiment, the things we manipulate are called independent variables (IVs). The things which we predict changes in are called dependent variables (DVs). The idea is that the dependent variable(s) “depend” on the values of the independent variables. We hypothesize that changes in the IV cause changes in the DV.
Variations of these terms are not uncommon:
| Independent Variable | Dependent Variable |
|---|---|
| Manipulated variable | Measured variable |
| Explanatory variable | Explained variable |
| Treatment variable | Response variable |
In observational research, we don’t claim that one variable causally “depends” on another. Ideally, we should use terminology that is more appropriate for observational work:
| Variable A | Variable B |
|---|---|
| Predictor variable | Predicted variable |
| Regressor | Regressand |
| Feature | Outcome |
| Input | Output/Target |
However, the experimental terms “independent variable” and “dependent variable” are often used in observational work even though they aren’t really appropriate. Notice that many observational terms (table above) imply a direction (input\(\rightarrow\)output, predictor\(\rightarrow\)predicted, etc.), which may or may not be appropriate. For example, using one variable to predict changes in another is very common—in fact, it’s the core of machine learning and artificial intelligence research. However, in purely correlational work, we don’t know which direction (if any) is appropriate; you’ll find that any variable can be used to predict any other variable and the (correlational) result will be the same.
Effect size
When changes in one variable cause changes in another, we call that an effect. Typically, this will be the independent variable affecting the dependent variable. The magnitude of the effect is called the effect size.
Once again, the terms “effect” and “effect size” are often incorrectly used to describe correlational relationships. In fact, there is no other good terminology I’m aware of. Just make sure to keep it clear in your mind whether you are talking (or reading) about causation or correlation.
Recall the elevator-music \(\rightarrow\) pro-social behavior experiment above. Let’s say we want to conduct another experiment to determine if the genre of the music affects pro-social behavior. To proceed, we need to pick what genres to compare—this is what our actual manipulation will be. We could for example, play baroque music in one elevator and classical music in the other. Alternatively, we could play death metal in one elevator and smooth jazz in the other. Which of these two scenarios might evoke a bigger effect?
When designing an experiment, a key concern is the strength of the manipulation. If the effect size is large, we might be able to measure the effect even with small manipulations. If the effect size is small, subtle manipulations may be too weak to discern (measure) changes in the dependent variable. For an experiment like our pro-social study, the contrast between baroque and classical music might be too subtle to evoke a measurable effect. Contrasting death metal and smooth jazz would be far more likely to get an effect—on the other hand, it might not be very meaningful! Hearing death metal on an elevator would be so unexpected and jarring that it would just confuse people.
My main advice regarding planning our manipulations: don’t err on the side of subtlety! It is very easy to overestimate effect sizes. Most researchers are well-versed and passionate about their research topics. This means that they tend to be interested in and sensitive to subtle details. For example, if you are a classically trained musician, the difference between baroque and classical music might be extremely obvious and glaring to you. However, I can assure you that most lay people would have trouble recognizing the difference!
Let’s think about some other manipulations in music tech research.
- Maybe you need to manipulate the “loudness” of a signal:
- The difference between \(70_{dB\ SPL}\) and \(80_{dB\ SPL}\) will be too subtle—Try \(60_{dB\ SPL}\) and \(100_{dB\ SPL}\) instead.
- Maybe you are manipulating the “quality” of chords:
- The difference between a minor-7th chord and a minor triad might be too subtle—try comparing minor and major chords.
- Maybe you are manipulating the orchestration of a piece:
- A bassoon and a clarinet might be too similar—contrast a bassoon with a violin to see a larger effect.
- Maybe you are manipulating the number of speakers in an ambisonic array:
- Eight speakers vs twelve speakers might not be a big difference: maybe try comparing four speakers to twenty-four.
In some cases, one level of the independent variable can be set to “none” or 0. For example, if you are user-testing a new VR system, you might have a 3D audio variable with two levels: \(\{off, on\}\). Or perhaps, three levels: \(\{off, weak, strong\}\). Always include the “none” level. An experiment with only \(\{weak, strong\}\) is a terrible idea! In the case of speakers arrays it’s probably a good idea to include stereo as a level.
Manipulation Check
It can be a great idea to conduct a manipulation check—a test to see if your experimental manipulation seems to actually evoke any measurable effects. This can be done in a separate “pilot” experiment, where you can try a wide range of manipulations and see what the effect size is before you design your main experiment. A manipulation check can also be achieved by simply including a few really extreme experimental levels to make sure you can observe an effect.
A few years ago, Rosa Westfall and I did an experiment looking at the effect of dynamic range compression on the perception of mix quality. As music production experts with lots of practice listening to mixes, Rosa and I might have thought that the difference between a compression threshold of \(-5_{dB}\) and \(-15_{dB}\) was obvious. However, we made sure to manipulate the threshold as low as \(-24_{dB}\). If you look at our results, it’s clear this was a good idea:
The \(-24_{dB}\) condition was the only condition where there seemed to be any (measurable) effect! If we hadn’t included this extreme level, we would have had no observed effect at all. Notice that we also include the “none” level, \(0_{dB}\).
Experimental Conditions
In many studies, there are multiple variables which might affect (or correlate with) the dependent variables. These effects could be analyzed in isolation, but in many cases it is important to understand the relationships between all the variables. Generally, we want to be able to compare all the combinations of the levels of all our independent variables.
Recall that each variable in a study has two or more levels. When we start working with more variables, there are many possible combinations of levels. Every combination of levels we can measure is called a condition.
To analyze relationships between more than two variables, we need to measure contrasting conditions for each variable. When we contrast levels of two or more variables, we say that those levels of those variables are “crossed.” When all the levels of two or more variables are crossed, we call this a factorial design. A “fully factorial” design includes every combination of levels of all variables. Factorial designs are often described as the product of the number of levels of each variable:
- \(2 \times 2\) design
- \(3 \times 5\) design
- \(2 \times 2 \times 4\) design
- etc.
Obviously, this notation only makes sense for categorical variables (with few levels).
Consider these two variables:
- Favorite genre \(\in \{Rock, Jazz, EDM\}\)
- Musical skill \(\in \{Musician, NonMusician\}\).
This “favorite genre” variable has three levels, while the “musical skill” variable has two. To fully cross these variables, we need to sample from six conditions. We might, for example conduct a human-participant experiment and recruit ten participants for each condition:
| Musician | Non-musician | |
|---|---|---|
| Rock | \(N = 10\) | \(N = 10\) |
| Jazz | \(N = 10\) | \(N = 10\) |
| EDM | \(N = 10\) | \(N = 10\) |
This is a fully factorial \(2 \times 3\) design.
However, if we, for example, can’t find any EDM musicians in our sample, we can’t have a fully factorial design. We might end up with something like this:
| Musician | Non-musician | |
|---|---|---|
| Rock | \(N = 10\) | \(N = 10\) |
| Jazz | \(N = 10\) | \(N = 10\) |
| EDM | \(N = 0\) | \(N = 10\) |
Our “favorite genre” and “musical skill” variables are not fully crossed. Our conclusions about the interaction between genre (EDM) and musicianship would be limited.
Mantras
Validity is the fundamental concept of methodology—but it’s not a badge worn by a method, test, or number. It’s a property of an inference: the same measure can support a valid claim and an invalid one. So don’t ask “is my method valid?” but “is this claim, drawn from it, justified?” That’s what reviewers are really checking, and it’s the thread through the mantras that follow—comparison, confounds, and sampling are all ways a claim can fail.
This is a classic. Ultimately, all induction is based on the observation of correlations. However, these correlations are not necessarily indications of causation.
The only way to decisively argue that causation is to manipulate one variable and observe its effect on other variables.