20 Covariates and Control
(Modified on 2026-08-11)
4,088 words
In 2006, Muhammad Yunus received the Nobel Peace Prize for his work serving the poor of Bangladesh. How did Yunus help alleviate poverty? By starting a bank—Grameen Bank.
When banks loan money, they are eager to know that the person receiving the loan is likely to pay the loan back. For this, banks look for collateral. For example, a farmer might use property as collateral to purchase a new tractor. If the farmer defaults on the loan, then the bank sells the land, recoups their money, and returns the remainder to the borrower. Alternatively, bankers look for evidence of stable income. If the borrower has a job and a history of regular income, then the banker may feel more confident that the loan will be repaid.
Traditionally, banks have been very reluctant to lend to poor people—especially poor people in the developing world. Even though a peasant family may have farmed a certain parcel of land for generations, they often don’t have any official documentation (such as a deed) that testifies to their legal ownership. Without formal documentation of ownership, the land has no value as collateral. Apart from farming, poor people rarely have a history of stable employment that brings in a regular income. Consequently, banks are reluctant to lend to poor people. Ironically, banks are mostly interested in lending money to people who already have money.
Often, poor people can significantly improve their living standard, if only they could invest in something productive. A peasant farmer might be better able to plow the land (and harvest a bigger crop) if he could purchase a buffalo to pull a plow. A woman could start a food stall in a local market if only she could afford a portable gas cooker and a pot. A loan of just $50 or $100 can transform a peasant family from subsistence to entrepreneurial success (Anonymous 2009).
In lieu of collateral, one way to encourage poor borrowers to repay a loan is to make use of social pressure. In the case of Grameen Bank, a promise would be made to successively loan money to each of 10 women in a village collective. The first woman would receive a loan. If she ultimately repaid the loan, then the second woman would receive a loan, and so on. If any woman failed to repay her loan, then the remaining women would be ineligible for their hoped-for loans. If a woman has difficulty repaying her loan, the likelihood is that the other women will either help her out, or bring social pressure to bear on her to fulfill the conditions of the loan. In this way, social pressure replaces the need for collateral, and so makes it possible for a bank to make loans to the poor.
Social pressure also helps to ensure that the loaned money is used for productive purposes. Instead of purchasing luxury items like paying for a fancy wedding, or a status-raising motorcycle, everyone understands that the best purchase is one that increases your income. For this reason, women have been typically favored over men in receiving loans. Men tend to spend the money on themselves, or on items that raise their status rather than improving their productivity.
This phenomenon—called microlending, microfinance or microcredit—appears to have had a major impact on the people who received the loans. By a number of measures, people who received the loans do well. For example, a simple measure of affluence is the number of children in a family who go to school. Impoverished families cannot afford to educate their children. They cannot afford books and tuition; moreover, impoverished families need the children to work the fields, gather wood, etc. Years after receiving their loans, borrowers are, on average, much more likely than non-borrowers to have children in school.
There are innumerable anecdotal success stories in the world of microlending. Grameen Bank estimates that 5% of their clients exit poverty each year. By the 1990s, most developing countries were promoting microlending in rural communities as a way of reducing poverty. This all sounds wonderful, but to make a strong claim that micro-financing is really a successful technique for reducing poverty, we need to apply some skepticism. Can we be confident that these “success stories” reflect a real effect?
In a scientific inquiry, we analyze the correlations between two or more variables, in particular looking at what variables [effect][Inquiry.html#effect-size] a dependent variable that we are interested in. Unfortunately, when researching complex real-world phenomena, like human behavior, there can be many variables which influence our dependent variable(s), and each other.
For example, we might study the dependent variable of “enjoyment” of a musical mix. Perhaps we predict (as Rosa Westfall and I did) that the amount of dynamic range compression will affect this enjoyment variable. But consider this: what other variables might influence a person’s enjoyment of a mix? There may be variables associated with each person: their personality, background/experience, age, gender, etc. For example, some people are generally more positive and open to new experiences and will tend to enjoy mixes regardless; Similarly, a professional mixing engineer will experience a mix differently than a lay person. There are also more dynamic human variables: mood, hunger, energy level, or even an irritating itch might affect a person’s enjoyment of a mix. There are other non-musical variables as well, like the listening environment (lighting, background noise, etc.) which could affect their enjoyment. Finally, there are also other musical variables which will affect their enjoyment: genre, tempo, track-length, tonality, catchiness, etc. Any of these variables might affect our dependent variable, and whats more, many of these variables will interact with each other!
In our hypothetical “mix enjoyment” study, the independent variable we are interested in is compression. All of other potential variables which might affect mix enjoyment are covariates: variables that vary with our independent variable, but are not pertinent to our research question. We sometimes call these sorts of variables extraneous variables or nuisance variables, but “covariate” is the most general term.
There is no clear-cut distinction between an independent/predictor variable and a covariate: The difference is theoretical and rhetorical—What role does the variable play in your theory/model? In our theory about how compression affects our enjoyment of mixes, we didn’t expect that tonality (major vs minor) was relevant. However, tonality might affect the enjoyment of the music, so it is a covariate we needed to consider, even if it’s just a “nuisance” covariate. The point is, that just because a covariate doesn’t interest us doesn’t mean we can ignore it.
For another example, Rosa and I anticipated that genre could interact strongly with compression: for example, compressing hip-hop might improve mix enjoyment but compressing classical music might not. Genre was another covariate that we cannot ignore, and because we expected the effect of genre would interact with the effect of compression, it was especially important that we consider it. Genre was more than just a “nuisance”—it was something that could radically affect our results. In fact, given the (potential) importance of the genre \(\times\) compression interaction, it might have made sense to rethink our whole theory/project with genre as an independent variable in its own right. Thus, the line between a covariate and an independent variable can be a bit blurry at times.
Confounding variables
Imagine that we conducted an experiment where we had people listen to highly compressed hip-hop tracks and completely uncompressed classical music. If we found that our listeners enjoyed the mix in the hip-hop track better, could we conclude that the compression improved the mixes? No. In this scenario, genre is a covariate which is not being accounted for. There is also a third variable—our participants’ listening preferences—which might be interacting with our other variables: Perhaps our participants aren’t classical music fans and tend to prefer hip-hop music overall. In this case, genre (interacting with participants’ preferences) is a confounding variable. The presence of confounding variables sabotages the internal validity of research claims. Failure to consider or account for a potential confounding variable is generally seen as a major methodological blunder. “Confound,” “confounding,” “confounded”—these are the most damning terms used in scientific review.
Note that a confounding variable is a covariate. But not all covariates are confounds. The term “covariate” is neutral, referring to any variable that might correlate with your dependent variable(s), but which may or may not be an issue. A confound is a specific case where the presence of a covariate which you didn’t properly consider damages your ability to make a valid research claim.
Control
To account for the influence of covariates, the best approach is to control them. There are two main ways to do this:
- Manipulating the covariate.
- Sampling data based on the covariate.
The goal of experimental control is to isolate the effect of the variables we are interested in (independent variables) from the variables we aren’t interested in (covariates). In other words, we want to know what effect our independent variables (e.g., compression) have on our dependent variable(s), all else being equal.
In the simplest form, a control variable is kept constant, so it can’t influence the dependent variable. For example, we might conduct our compression experiment using only hip-hop tracks, with varying amounts of compression. By keeping genre constant, we can be assured that it is not inadvertently affecting our dependent variable. By sampling only hip-hop tracks, we’d say we are “controlling for” the effect of genre.
Fixing a variable to a single value is fine when we expect an effect to be purely additive (only a “main effect”). However, as we’ve already said, it seems likely that the genre and compression variables will interact. Thus, by only studying hip-hop we are getting an incomplete picture of the what’s going on. Another approach to “controlling for” genre would be to test a variety of genres, For this to work, we need to fully cross the covariate (genre) with the independent variable we are interested in. This means we need to measure/manipulate all “levels” of compression for each genre. Something like this:
| Hip-Hop | Classical | Rock | Jazz | |
|---|---|---|---|---|
| Compressed | \(N = 10\) | \(N = 10\) | \(N = 10\) | \(N=10\) |
| Not Compressed | \(N = 10\) | \(N = 10\) | \(N = 10\) | \(N=10\) |
This would allow us to see, for example, if compression has different effects in different genres. These are (simplistic) examples of systematic experimental control. This can be hard in many cases; for example, there are a lot more than four genres, so we haven’t really controlled for all possible effects of genre. How many genres (and which ones) would we need to include to be confident that we totally controlled for the effect of genre?
Randomization
In music research, there are often many, many potential covariates. We can’t conceivably control them all in a systematic way—we often don’t even know what they all are! For example, in human research, you can’t possibly control all personal variables—people are too complex. However, with a large enough random sample of people, we can reasonably hope that our sample will include all, or at least many, different levels of these personal covariates. For example, researchers often model human personality with a “neuroticism” variable. In a random sample, some people would have low neuroticism, some would have high neuroticism, some would be in the middle, etc. With a large enough sample, we will get people with many possible neuroticism scores, so the influence of the neuroticism variable would “average out” in our data, effectively controlling for the variable. As we’ll learn later, this sort of reasoning about “control through randomization” is important to [statistical modeling][RandomVariables.html].
Of course, we could try to be systematic and, for example, recruit an equal number of people with different neuroticism scores, like twenty people from each of the low, middle, and high score ranges. But, neuroticism is only one personality variable: if we wanted to also control for introversion or openness—not to mention age, genre, ethnicity or anything else—, systematic control can quickly get out of hand. By randomly sampling enough people, we can get all of these variables controlled (roughly) at the same time. This is why random sampling is essential to achieving experimental control of unknown variables.
We could also apply randomized control to our genre problem, from the previous section. If we randomly sampled a large number of songs from, for example, the iTunes API, we could reasonably hope that we’d get a mix of a bunch of different genres (and sub genres). We wouldn’t get all possible genres, but we’d (probably) get a good mixture, so when we conducted our compression experiment we could hope that the effect of genre would “average out” in the end, controlling for genre. Using this approach, we’d also be controlling for song length, popularity, and many other factors at the same time.
Randomization isn’t just applied in sampling. We can also apply it to control for other extraneous covariates. A good example is order effects in behavioral research. When exposed to a new stimuli, participants’ reactions can be affected a lot by other stimuli that they heard previously. For example, if I play you classical music first then hip-hop, your response to the hip-hop might be different than if we reversed the order. Thus, the “order or presentation” becomes a covariate in its own right, which might confound our results.
In simple cases, we can control for order effects through systematic control: fully cross the “order” variable with our other variable(s). For example, if we are only studying hip-hop and classical music, we could have half of our participants hear “Classical -> Hip-Hop” and half hear “Hip-Hop -> Classical.” This is the best way to do it, if you can! However, if there are more than two levels to work with, the number of possible orders can explode, so systematic control is not feasible. If you are working with seven genres, there over 5000 possible orders! Thus, the standard practice in behavioral experiments is to randomize the order of stimuli; each participant hears the same stimuli in a different, random order.
Randomization can (and should) be applied anywhere possible in experimental designs. In machine learning, we randomly split the data into train/test/validation sets—this randomization is meant control for covariates in the dataset. Generally, we want experimental conditions covering all the levels of our independent variables. All other variables, both known and unknown covariates, should be either systematically or randomly divided between the conditions. Only the independent variables should differ between conditions, isolating their effects (if any).
Control groups
Let’s consider a common scenario we’ve all lived through: suffering from cold symptoms. Once we notice the symptoms, maybe we start to take some antibiotics and, after three days, our symptoms fade away. Can we conclude that the antibiotics contributed to our recovery? In isolation, we can’t. Assessing the effectiveness of our medicine requires that we consider the counterfactual: what would have happened if we hadn’t taken the medicine? Most people who catch a cold feel better in three or four days anyway, even without medication.
When we get sick, there are many covariates that can affect the speed of our recovery: our particular immune health and history (past exposure to similar infections); the nature of the infection (bacterial vs viral); our other behavior (e.g., amount of rest, diet); etc. Even our beliefs about our recovery will affect how fast we recover—the well known placebo effect. These natural, unaided healing processes are covariates that confound any conclusion we can make about the effectiveness of the medicine.
The biggest problem in our scenario is that we’ve only observed one level of our independent variable. The variable is “take antibiotics”: we observed the “yes antibiotics” level but not the “no antibiotics” level. In order to make conclusions about the effectiveness of antibiotics, we need to compare our two antibiotic levels, while controlling for other covariates.
A standard paradigm in medical and user studies is to divide observations (usually people) into a control condition and a treatment condition. (In other research scenarios, we might call them “comparison condition” or “test condition”, etc.) The “control group” is given no antibiotic; the “treatment group” is given the medicine. The key thing is that assignment to the two groups is randomized, so levels of all other covariates get mixed up between the two groups. Ultimately, the two groups should only differ with regard to the relevant independent variable (the antibiotic). Only by having a control group are we able to compare different levels of the independent variable in isolation.
Note that a “control group” is so called because it helps control for numerous covariates. The control group represents a specific level of the independent variable (in this case, the treatment). However, we are not “controlling” the antibiotics treatment, we are controlling for other (possibly unknown) covariates.
The control vs treatment paradigm can also be used in non-medical studies, like user studies. Imagine you work for a company that sells a software synth. Maybe you have come up with a new design for the synth interface, which you predict will make it more fun and usable for musicians. How would you test this hypothesis? You could recruit participants for a user study and ask them. Of course, existing users of your synth may be biased because they are accustomed to the old interface, so you’d control for this covariate by only recruiting people who have never used your synth before. You could then assign participants randomly to either use the old version (control) or the new version (test), and measure how much they like the two versions. Since participants are assigned randomly to the two groups, participant variables (like age, gender, musical experience, etc.) should be equally divided between the two groups, leaving only your independent variable (synth version) different between the two groups.
Examples
After-School Music Class
Consider another scenario: Say we want to investigate the hypothesis that attending after-school musical activities improves grades. One thing we could do is survey many schools and compare the grades of students who take after-school music classes with those who don’t. (This would be an example of a natural experiment.) We’ll end up with an “after-school music class” variable—with two levels: \(\{no, yes\}\)—as our predictor variable, and grade point average (GPA) as our predicted variable. Perhaps we find that grade-point-averages are, on average, \(+0.7\) higher in the \(yes\) group. That seems consistent with our hypothesis. Unfortunately, there are some possible confounding variables to consider. It’s likely that students who sign up for music classes differ on a number of variables from those that don’t. Notably, families that choose to sign-up for after-school music classes may generally place a higher value on education. If so, the variable “family education values” could be causing an increase in GPA, not the music class. Family values are confounding our predictor.
In order for randomized control to work, it has to be, well, random. One of the biggest problems/challenges in any sociological research is self-selection bias—if participants choose to participate, then they become a variable influencing the result. This is a big problem with any mass-distributed survey—the people who take the time to fill out the survey differ in various ways from those who don’t. In order to remove this confound from our after-school music inquiry, we need to make sure that children/families (and experimenters) don’t get to decide who goes into each group. We must assign children randomly to the two groups, assuring that other covariates (parenting, income, personality, etc.) will be controlled for (on average).
OK. So let’s say we assign students to groups randomly, and still observe that kids in the \(yes\) group see better grades. Can we argue that the after-school music class improved grades? Yes…but. There is more to consider. Have we really isolated the variable “music”? What about the variable “after-school class”? It is plausible that the extra attention and structure of attending any after-school program might improve students’ grades. Thus, “after-school class” vs “no after-school class” is a separate variable from the variable “music.” What we’ve really got here is this:
| Class | No class | |
|---|---|---|
| Music | sampled | |
| No music | sampled |
If we want to show that music specifically improves grades, we’d need to measure more conditions, so we can actually isolate the effect of music: We might, for example, assign a third group of kids to an after-school drama class. For a full factorial grid, we’d also want to have a group of kids doing music on their own at home, and a group doing drama at home. Only this way can we fully isolate the effects we are interested in:
| Class | No class | |
|---|---|---|
| Music | sample | sample |
| Drama | sample | sample |
Micro-financing
Consider again the example of a micro-financing, which actually presents a similar scenario to the cold treatment example above.
Among any group of people, including poverty-stricken people, some will do better than others. Visit a village ten years later, and some people have managed to build a new house and send all of their children to school. Others will have slipped deeper into poverty. These variations in success can be attributed to many covariates—natural talent, family connections, lucky encounters (good or bad), etc. In order to make a strong conclusion about the effectiveness of microlending, we need to compare people who received micro-loans to those that did not.
Another issue to consider is a possible confounding variable: Like families signing up for music classes, there may be variables which affect which families received micro-loans. The people who receive micro-loans may be particularly enterprising or social, or have a reputation as being competent and hard-working, which gets the attention of the bank in the first place. At least one study has showed that people who sign-up for microloans are already more eager to send their children to school than those who don’t. These covariates will confound the independent variable: it may be that the people who receive microloans do better overall than those who do not, but only because they are more likely to succeed anyway.
In 2009 and 2010, two research studies were published that included proper randomized control conditions as a way of attempting to determine the effectiveness of microcredit in alleviating poverty. One study (Banerjee et al. 2010) was carried out by researchers from the Poverty Action Lab at MIT. They identified 104 slums in the city of Hyderabad, India. The slums were randomly divided into two groups, each containing 52 slums. In the treatment group, an Indian microfinance firm began making loans. In the control group, the same loan procedure was used to identify individuals who would be eligible for a microloan, but did not receive any loan. This allowed the researchers to compare the number of borrowers who improved their lives to the number of would-be-borrowers who improved their lives. They found that, with the passage of time, there was no difference in the poverty levels for the treatment and control groups.
A second study was carried out in the Philippines by Dean Karlan and Jonathan Zinman (2009). In their study, they modified the computer software used by a microfinance bank to determine who was eligible for a loan. People with the same credit-score were randomly assigned: half received a loan, and the other half were denied a loan (even though they were equally credit-worthy). Karlan and Zinman then monitored the progress of both groups of individuals. Once again, they found that there were no significant differences in the two groups in terms of poverty alleviation.
It bears emphasizing that I’ve only shared the results of two (now old) controlled-randomized studies. Further research is needed. However, these results don’t look especially promising. Muhammad Yunus is undoubtedly a decent and virtuous person with an honest concern for reducing poverty. But his solution doesn’t appear to be as effective as the Nobel Prize committee might have thought.
Epistephobia
An important lesson from these examples is that our enthusiastic embrace of a promising outcome can prevent us from pursuing a dispassionate examination of its effectiveness. As music-lovers and musicians, we’d like to believe that music makes students smarter (or at least, improves their grades). As human beings, we’d like to believe that the Grameen Bank was successful in its mission. Our attitude towards these hypotheses can easily bias us away from making the necessary comparisons. More over, we might feel icky even being skeptical of these programs at all—a case of epistephobia creeping in.
Mantras
Randomness is our enemy—noise, error. However, randomness can also be an ally—a tool to control biases we aren’t aware of. The variables that interest us are always intertwined with numerous covariates, some which we know about, and many that we don’t know about. The only way to account for these covariates is to control them directly, or failing that, sample randomly to control for them indirectly.
Corollary: “Compare, compare, compare.”
To make valid claims it is essential that we compare all levels of our independent variables, and of relevant covariates. Lack of adequate comparison is often a problem, especially for novice researchers. Compare, compare, compare!