6.4 Internal Validity
A study’s internal validity is the degree to which its cause-and-effect conclusions are correct; in other words, it relates to the researcher’s ability to conclude that no other variables influenced the result. Internal validity can be threatened in several ways.
The presence of a confounding variable could render an experiment’s conclusion invalid. A confounding variable is a variable that impacts the response, but that the modeler did not account for in the experimental design. For instance, suppose a marketer working for a major toy retailer tried a new type of in-store ad display, and launched the new display style in the first week of November. Suddenly, he notices that sales of the toy included in the new display jump up 60%, and they stay at this elevated level for several weeks. Triumphantly, he spins in a circle, dancing joyfully, imagining the year-end bonus that he can now expect.
But did the sales increase happen because of the new display style? Maybe. But perhaps the annual bump in toy sales that occurs each November, and lasts through the first three weeks of December, is really responsible. In this case, it is highly likely that the seasonality of toy sales is a confounding variable.
If subjects are not chosen for treatment and control groups randomly, an experiment’s internal validity could be jeopardized. As noted in the AwesomeSauce example in the previous section, non-random assignment to the groups is likely to result in self-selection bias.
Another internal validity threat comes from attrition. Attrition occurs when subjects leave the experiment before their outcome can be observed. The longer an experiment runs, the greater the risk of subjects’ attrition.
Yet another internal validity threat comes from the Hawthorne Effect. The Hawthorne Effect impacts an experiment when subjects’ knowledge of the treatment could impact the outcome. For instance, suppose a researcher wanted to know whether a Professor who drank less coffee than usual prior to class would still use the words “like”, “ really”, and “you know” as often during a lecture. If the Professor knows that the experiment is occurring, and knows that the researcher is monitoring the lecture, counting each utterance of these filler words, that could certainly impact the Professor’s behavior –he might use the words less frequently, because he is now so self-conscious about this. Alternatively, he might use the words more frequently, because his mind is so focused on these words. In a “double-blind” experiment, neither the subjects nor the researchers know whether any particular subject belongs to the treatment or the control group. This way, the subjects will not be influenced by their awareness of their group status. Furthermore, this prevents another problem – if the researchers are aware of which subjects are in the control group, and which are in the treatment group, they might behave differently towards the subjects as a result. That different behavior could, in turn, impact the experiment results.