Research Mantras
(Modified on 2026-08-06)
269 words
Our forty-eight mantras remind us of the tenants of good research methodology.
There is no magic wand or special method produces objective knowledge; the best we can do is find beliefs we hope are useful models of the Truth. Scientists can look and feel objective—passive grammar, jargon, lots of numbers—without being so. Worse, once we’re confident of our own objectivity, we stop questioning our beliefs, which is the opposite of good science. Own your subjectivity; don’t hide from it.
(Introduced in Chapter 1)
Good scientists don’t claim to prove anything. We are motivated by truth, but we don’t aspire for “proof.”
(Introduced in Chapter 1)
Though induction can not prove propositions, it can disprove them; Thus, we approach research from the perspective of trying to disprove our ideas. If we can’t show that we are not right, then maybe there is a hope that we are “not not right.”
(Introduced in Chapter 1)
There is always error in our knowledge. Our goal is to minimize these errors, not remove them.
There is no way to evaluate knowledge absolutely—we’d have to know Objective Truth to do that. What we can do is evaluate whether one model is more useful than another.
(Introduced in Chapter 1)
The greatest barrier to good research is confirmation bias. We want our research to “succeed,” and there are many ways that we can consciously or unconsciously help the research “succeed”—or appear to succeed. This is why the main concern in scientific methodology is the control/reduction of Type 1 errors—stopping ourselves from making false-positive claims. Our reviewers are on the lookout for ways that we might bias the design, implementation, execution, or interpretation of our research in our favor.
(Introduced in Chapter 1)
Admitting, owning, and quantifying how wrong we might be is what sets good science apart from other sources of knowledge. Counter-intuitively, acknowledging imperfections earns readers’ trust: if a reader spots a flaw we never mention, they assume we never saw it, and our credibility collapses. Name the problem first, then argue—with humility—that the work is still valuable.
(Introduced in Chapter 1)
“Why” is the question that leads to the most interesting and useful answers. (“How” is good too.) The answer to a why question is a theory.
Of course, before asking “why,” make sure the concepts you are asking about are (at least plausibly) valid. If you are making assumptions, do background research to validate those assumptions.
(Introduced in Chapter 2)
When exploring a new topic, the first thing to do is to figure out what the major, important aspects of the topic are. Don’t go off into the weeds if you haven’t first surveyed the forest. Beware that your specific/experience interests and passions, which are personal to you, can blind you to the big picture.
(Introduced in Chapter 2)
Everything is interconnected…especially when it involves human thoughts and behaviors. You can’t explain it all. Don’t try to. Bite off chunks that you can chew.
(Introduced in Chapter 2)
Many research questions might have answers that appear obvious—at least to some people! It can still be worthwhile to explore these questions anyway, to validate the intuition or perhaps to disprove it!
(Introduced in Chapter 3)
Your research doesn’t “speak for itself”—conclusions don’t just fall out of the data. We aren’t writing down the Truth; we’re convincing readers to agree with our claims, which means making arguments for our research’s validity. This shapes the work from the very beginning: at every stage, ask what point you’re making and why it’s valid. If you can’t make that argument, something is wrong—rethink or retool.
(Introduced in Chapter 5)
Skepticism is the most important skill you can develop. Be prepared to question and critically analyze everything—especially yourself, your own ideas, your own questions and, yes, your own skepticism. Self-skepticism is a tool to do the best research: It will help you find problems in your research before you commit them.
(Introduced in Chapter 5)
Aim for modesty, even as you make brilliant points. The strength of science is precisely that its practitioners choose methods that enforce intellectual humility.
When presenting your result, frame them narrowly rather than broadly. Couch your claims. Aim for modesty and competence: over-deliver on evidence while understating your conclusions. Offer more evidence than your readers think necessary, and express your conclusions in a more tentative or circumspect manner than your readers expect. Let the reader form their own impression about the degree to which a result might generalize.
(Introduced in Chapter 5)
Using concise and precise technical terms is a good thing, if it enhances clarity. When in doubt, its better to over-define and explain than to under-explain, to make totally sure your readers/reviewers understand what you are saying.
Define acronyms and jargon. Don’t assume everyone will understand the words you use the way you understand them—explain what you mean by different terms. Don’t be afraid to repeat yourself.
(Introduced in Chapter 5)
The rhetoric of science is the rhetoric of prophecy. Prophecies are compelling because, if they are accurate, they can be useful, and if they are not, they cannot hide. Making concrete, testable (falsifiable) predictions is the bedrock of science.
In contrast, exploratory research is most valuable when mixed with hypothesis-driven research. Don’t fall into the (easy) rut of doing only exploratory work.
(Introduced in Chapter 6)
Falsifying a priori predictions makes for compelling rhetoric—but the hypothesis test is just one example of a broader idea that should inform all research. To claim your conclusion is true, it must be possible for it to be false. So when you start a project, ask: what would it look like for my ideas, data, or claims to be wrong? If they can’t be wrong, you’re probably reasoning in a circle. Put your neck out—take a risk.
Mantra: A null is not a nothing DontFuss
The strongest research claims can be made when we attempt to falsify our hypotheses. This means that, if we are doing science right, we will sometimes get negative results. Negative results are always disappointing, but they are not a professional or personal failure—if you falsified your theory, you are doing research it right. It is essential that we view “negative results” as real, valuable results, and aim to publish and disseminate them. Burying failed predictions in the file drawer distorts the whole literature.
(Introduced in Chapter 6)
The special rhetorical power—and epistemological credibility—of science rests, more than anything else on the idea that we try to disprove our own theories, not prove them! Of course, we don’t really want to disprove our theories, but to make our claims scientifically compelling we must design our experiments to disprove our theories. Being totally objective is never possible, so in practice we design in such a way to slightly “tip the scales” against our theories. This, in part, helps counteract our own natural confirmation biases.
(Introduced in Chapter 6)
In order to prevent ourselves from consciously or unconsciously adapting our theories to match what we observe—“moving the goal post”—we must make the goal post fixed and immovable in advance. We must draw a line in the sand (like \(p < .05\)) in advance, and respect and that line, whatever comes. Even if you have (or think you have) good reasons to adopt a new (post hoc) goal, you must always acknowledge the original goal!
(Introduced in Chapter 7)
We say “hindsight is 20/20”: once something has happened, we feel we understand exactly why—and imagine we could have predicted it (“I knew it all along”). This clarity is an illusion. Uncertain outcomes come to feel inevitable, and an explanation that merely fits what happened feels obvious in retrospect. But “it makes sense in hindsight” is no evidence the explanation is right—which is exactly why a prediction made in advance beats an explanation invented after the fact (Post-hoc is a four-letter word).
(Introduced in Chapter 7)
Humans are gifted at coming up with explanatory accounts of data we have already seen—post-hoc theorizing. These after-the-fact explanations feel as good as a priori ones (our hind-sight bias), but aren’t. What’s worse, we can easily lose track of which of our ideas were post hoc and which a priori, which is why scholars are wary of post-hoc reasoning sneaking in where it shouldn’t be. An important act of self-skepticism and professional humility is clearly labeling post-hoc aspects of your work. We’ll often say things like “we found this pattern in the data, but only in a post-hoc analysis.”
(For non-native speakers: “four-letter word” is a euphemism for an offensive word.)
(Introduced in Chapter 7)
Data can be explored to find patterns and generate new theories, or to test a priori theories, never both! You should’nt use data to devise a theory, then use the same data to test that theory. We can’t train and test on the same data.
(Introduced in Chapter 8)
Scholarship is a community of colleagues and collaborators—not competitors. It’s natural to fear someone will “steal” your ideas, but that fear hurts everyone: if others are independently doing similar work, that’s good—you replicate each other and can collaborate. Learn from others (and cite them), ask questions, share your own experience, and join the community. Even in industry, collaboration is usually a net benefit.
(Introduced in Chapter 10)
As good scientists, we want other researchers to help us by criticising our work, but we don’t want them to hurt our feelings. As good people, we should reciprocate. Criticism matters, but can easily cause ill will and damage your reputation in the community. Remember: All research is flawed, including your own. Always give polite, constructive criticism, even when you hate an idea or its execution. And always find something nice to say! This applies to criticizing yourself, too: don’t beat yourself up.
(Introduced in Chapter 12)
There is no simpler way to improve the quality and efficiency of research than writing the paper first—i.e., thinking through and planning everything in advance. That doesn’t mean it’s easy to make yourself do it! It’s not research until you write it down.
(Introduced in Chapter 13)
Don’t think in terms of “facts”—this is a slippery slope. Always think in term of assumptions. The question is, will your colleagues/readers consider your assumptions valid?
(Introduced in Chapter 15)
Anecdotes are, by their very nature, unrepresentative samples of reality. Anecdotes give us ideas and motivate us, but their value as evidence in an argument is basically nill.
(Introduced in Chapter 16)
Validity is the fundamental concept of methodology—but it’s not a badge worn by a method, test, or number. It’s a property of an inference: the same measure can support a valid claim and an invalid one. So don’t ask “is my method valid?” but “is this claim, drawn from it, justified?” That’s what reviewers are really checking, and it’s the thread through the mantras that follow—comparison, confounds, and sampling are all ways a claim can fail.
(Introduced in Chapter 17)
This is a classic. Ultimately, all induction is based on the observation of correlations. However, these correlations are not necessarily indications of causation.
(Introduced in Chapter 17)
The only way to decisively argue that causation is to manipulate one variable and observe its effect on other variables.
(Introduced in Chapter 17)
Randomness is our enemy—noise, error. However, randomness can also be an ally—a tool to control biases we aren’t aware of. The variables that interest us are always intertwined with numerous covariates, some which we know about, and many that we don’t know about. The only way to account for these covariates is to control them directly, or failing that, sample randomly to control for them indirectly.
(Introduced in Chapter 20)
Corollary: “Compare, compare, compare.”
To make valid claims it is essential that we compare all levels of our independent variables, and of relevant covariates. Lack of adequate comparison is often a problem, especially for novice researchers. Compare, compare, compare!
(Introduced in Chapter 20)
Every study is imperfect in some way. The best way to make a strong research claim is not to make your method/data/analysis absolutely perfect, but to investigate the same topic in two (or more) independent ways. Two different operational measures of a variable; two different experiments; two different evaluation metrics; etc. If multiple separate studies/approaches are consistent with an idea, this provides converging evidence, which makes for a stronger argument than any single piece of evidence.
(Introduced in Chapter 22)
To falsify a hypotheses, we need to make a specific prediction about what we will measure. This means we need precise operational definitions of exactly how (research protocol) we will measure the constructs we make predictions about.
(Introduced in Chapter 22)
The most common criticism of scientific practice is that it is overly reductive: “reducing” the rich tapestry of reality to a handful of simple variables in an equation. This is a reasonable concern! However, as scientists, we should remind ourselves (and our critics) that we don’t believe our reductions are “True”: We reduce complex problems to simpler problems because it is a useful approach. This means that, when we make claims based on our “reduced” ideas, we have to be humble about how broad and meaningful we can generalize.
(Introduced in Chapter 23)
When we define terms, we are not claiming that our definitions are complete or correct. We always remember that our operational definitions—our measurements—are imperfect. We never measure the essential truth of something, only a rough model of it. Your definitions create a limited contract between you and your reader: you agree to understand this term this way, but only for the duration of the paper.
(Introduced in Chapter 23)
Always check the content of your samples, especially when relying on convenience samples. What observations are included in the sample? What observations are excluded from the sample? What observations are missing?
(Introduced in Chapter 26)
Avoid repeated sampling from the same source, related sources, or sources that can influence each other. These are sources of data dependence.
(Introduced in Chapter 29)
At each stage in research, it is important to look back and evaluate: how did it work? Did it work the way we expected? Did we measure what we intended to measure? Did we sample what we intended to sample? It’s also important to ask the opinions of others, not just yourself: your colleagues, our data coders, your research participants. Don’t form your final claims, or craft your final rhetoric (or publish!), before debriefing.
(Introduced in Chapter 36)
Data is not objective; Analyses do not just “pop out” of the data. Data sampling, collection, parsing, munging, analysis…all are influenced by your subjective decisions. Data is interpreted by researchers.
(Introduced in Chapter 44)
It’s not true that “statistics can prove anything,” but it is possible to manipulate data in ways that deceive yourself and others. Create a data analysis plan. Do not exclude outliers, normalize data, set conditions for excluding participants, or introduce post hoc tests without some principled prior reasoning. Be honest in reporting the analyses you carry out. Don’t hide multiple tests.
(Introduced in Chapter 44)
You might think data analysis is about uncovering patterns—but finding patterns you want to find is easy. The real problem is knowing whether a pattern is real. Since science means trying to prove ourselves wrong, the goal of analysis is to show the pattern isn’t there: make an honest, skeptical effort to explain away any correlation you observe before you believe it.
(Introduced in Chapter 44)
Data has problems; Always. Look at (or listen to) the data closely. Search out errors, by exploring random subsets of the data as well as any obvious outliers. Is the data formatted correctly? Is anything missing?
(Introduced in Chapter 45)
Visualizing your analysis is essential to understanding your results and conveying that understanding to your audience (“graph it to grasp it”). Visualizing is an important part of data spot-checking (“plot it to spot it”). Visuals are also one of the best way of judging and conveying effect size: does the effect look meaningful when you put it into a meaningful graph? (“See it to believe it.”)
(Introduced in Chapter 45)
The law of large numbers doesn’t apply to small numbers. Smaller subsets will often have the most extreme values. I your data is divided into subsets, the smallest subsets will likely have the most variability and most extreme averages.
(Introduced in Chapter 46)
Multiple at bats gives you multiple chances to get lucky. It’s not a test if you retake it until you get an A. If you give yourself enough chances, you’ll eventually hit a home run.
Each statistical test increases the likelihood of making a Type I (false positive) error. Repeating an experiment also increases the probability of a Type I error.
(Introduced in Chapter 57)
Skepticism is essential—but don’t let it tip into nihilism. All research is flawed, and all of it balances trade-offs (e.g., simpler measures are more testable but less valid). Pick apart the work you read and your own work—yet remember imperfect research still beats nothing. Don’t be discouraged, or discourage others, just because no solution is perfect.
(Introduced in Chapter 61)
It may feel like being a scholar means doing work: writing, designing, working with data, etc. However, the heart and soul of scholarship is ideas and ideas come from thinking. Schedule time to do nothing but think. When you read research articles, give yourself time to stop and think about what you are reading. Chronicle your thoughts using a research diary. Thinking makes you smarter.
(Introduced in Chapter 61)
Some problems are just plain hard. Some projects drag on far too long. Patience, dedication and stamina can be admirable qualities, but they can also stand in the way of using your time productively. Many projects fail, and that is okay. Move on to the next project, and try to learn from the experience.
(Introduced in Chapter 61)