8 Explore Then Test
(Modified on 2026-08-11)
1,322 words
We have already seen a few “yin and yang” dynamics in science: deduction vs induction; explanation vs description; a priori prediction vs post-hoc explanation; etc. We can relate all these ideas to a methodological paradigm—a research dynamic—we’ll call “explore then test.” We can think of this as “first ask ‘do?’ then ask ‘why?’.”
The explore-then-test research strategy combines an exploratory study with a subsequent study that tests a resulting hypothesis. The exploratory study is used to acquaint the researcher with the phenomenon, and to inspire the researcher to formulate an interpretation or explanatory theory. This theory is then used to generate one or more conjectures. One of the conjectures is refined to one or more testable hypotheses, and these hypotheses are then tested.
Double-Use Data
Recall that whenever we look at a set of data, we can immediately form a number of ideas that might account for any patterns evident in the data. However, these ideas are post hoc not a priori. After looking at the data, it would be wrong to formulate a theory and then to present the data as though it was an independent test of our theory. It is okay to use data to formulate a theory, but it is not okay to use a single set of data both to formulate the theory and to “independently” test the theory. We refer to this as double-use data. The paradigm is “explore then test” not “explore and test”—the explored data and the testing data must be separate!
Normally, we are not interested in where theories come from (the context of discovery). Instead, we are interested in testing theories (the context of legitimation). However, we are right to question a theory if the context of discovery and the context of legitimation are the same.
When you look at some observations you have collected, any theory you form is now post hoc theory. You cannot then claim that your theory was a priori and then use the observations as evidence that tests your theory. Once you look at your data, you cannot pretend that you predicted the data. In short, post hoc theories are not “prophetic.” You cannot use the language of prediction that is the essence of hypothesis testing.
A historical example of double-use data relates to the Theory of Continent Drift in the field of geology. Today, there is very good evidence that the continents behave as “plates” that slide very slowly across the earth’s surface. However, the theory began in an inauspicious manner. If you look at a map of the world, there seems to be a certain similarity between the eastern coastlines of north and south America, and the western coastlines of Europe and Africa. One could easily imagine all four continents pushed together like jigsaw puzzle pieces.
Over the past century, excellent evidence has been assembled that is consistent with this theory. However, this evidence was not available when the theory was first proposed. If we ask “What inspired the theory?” the answer was “Look at how the continents seem to fit together like jigsaw puzzle pieces.” If we then ask “What evidence do we have that’s consistent with the theory?” the answer was “Look at how the continents seem to fit together like jigsaw puzzle pieces.” In other words, the same evidence was used both as the inspiration for the theory, and as evidence for the theory. Notice that this is circular reasoning. The Theory of Continental Drift took time to be accepted precisely because of the lack of independent evidence.
The Reserved Dataset
Suppose that a researcher has administered an exploratory survey questionnaire to 100 people. Examining the surveys, the researcher observes certain correlations. For example, the researcher might discover that people who enjoy listening to sad music score high on questions measuring the personality trait known as “openness.” Notice that the research employs an exploratory approach, and that the discovery is “post hoc.”
Having made this observation, the conscientious researcher would then go ahead and test it. Their newly-developed hypothesis is that “people who enjoy listening to sad music will score high on the openness personality trait.” Our researcher might form a new questionnaire (or even reuse the earlier questionnaire) to collect new data. With this new data, the hypothesis can now be tested as an a priori hypothesis.
In many circumstances, returning to collect a second set of data can prove to be inconvenient or costly. For example, an ethnomusicologist may have difficulty obtaining funding to return to a remote location for a second round of data collection. In these situations, a clever strategy is to keep a so-called reserved dataset.
Suppose that our researcher had originally collected survey questionnaire data from 200 people (rather than 100). Before examining any of the data, the researcher randomly selects 100 surveys and stores them out-of-reach. (The researcher simply avoids looking at these 100 surveys.) This will be the “reserved dataset.” Examining the surveys, the researcher observes the correlation between enjoyment of sad music and “open” personalities. As before, this is a “post hoc” discovery. It would be nice to test this as a proper a priori hypothesis. In effect, the reserved dataset is equivalent to gathering questionnaire data for 100 new participants. (It’s like collecting the data before we need it.) So the researcher can then test the hypothesis using the reserved dataset.
Notice that the reserved dataset allows the researcher to carry out an exploratory study (whose purpose is to inspire theories and conjectures), followed by proper a priori hypothesis testing, while collecting data only once. This method is especially valuable when the researcher knows that it will be difficult to arrange a second round of data collection.
Reserved datasets, also called “test sets” or “validation sets,” are widely used in machine-learning and other highly exploratory research paradigms. This injects falsifiability into work that is otherwise “just” exploratory.
Hypothesislessness
- Hypothesis: A claim whose truth or falsity could, in theory, be tested (but not proven) by making one or more observations.
- Hypothesisless: The state of being without a hypothesis.
- Hypothesislessness: A persistent and sometimes chronic disease where a researcher is perpetually without a hypothesis.
— David Huron
In general, there is considerable attraction to pursuing descriptive or exploratory research. It is relatively easy to invent and design exploratory/descriptive research—just pick a topic and start gathering data. In contrast, coming up with the theories and hypotheses is hard. It also makes you feel more personally vulnerable—what if my theory is dumb? What if my hypothesis makes no sense? Who am I to propose to explain something? Gathering data never looks dumb!
You may feel that proposing a theory, or making a specific prediction, feels cocky and arrogant—the antithesis of the careful scholar. Wouldn’t it be more “objective” and humble to just gather the data and simply see what we find? On the contrary, boldness is the requirement for intellectual humility. The humble scholar is the one who is prepared to be wrong—especially when he or she is shown to be wrong in public. It is a mistake to believe that the researcher who proposes no theory or hypothesis is the humblest of scholars. The best research aims to test explanations by inviting failure. This means inventing theories, making predictions, and putting them out there for everyone to see them crash and burn.
Some scholars just don’t like being wrong. These scholars will tend to focus exclusively on descriptive and exploratory studies. The researcher who advances theories/analyses/stories only after making observations is a researcher who always appears omniscient. Their work is never falsified; their theories are never really tested. We will (and should) always regard their work with an extra grain of salt.
Many researchers fall into a trap of conducting only exploratory research, because it is easier and safer. Make sure to include both hypothesis-driven and exploratory projects in your research program. Avoid hypothesislessness.
Mantras
We’ve got two new mantras to remember:
Data can be explored to find patterns and generate new theories, or to test a priori theories, never both! You shouldn’t use data to devise a theory, then use the same data to test that theory. We can’t train and test on the same data.