When teaching the meaning of p values and null hypothesis testing, I recognize that (1) most of my students have been exposed to this before and (2) they still do not clearly understand. This is quite normal in my experience. My goal is to lean on what they have previously been exposed to and flesh it out.
Before the class begins, I find one student – usually out in the hallway – and surreptitiously give them a card (e.g., 7 of Hearts) and ask that they conceal it until I ask them to draw a (different) card and then show the former to everyone. After the class begins we discuss null hypothesis testing. I explain that the null hypothesis is the boring hypothesis, a hypothesis of no effect. To drive this home, I ask them how the English would pronounce the final score of a soccer game which ended 0-0, to which a student inevitably replies “nil-nil.” So, a null hypothesis is a hypothesis of nothing. The p value, then is the probability of obtaining a given result if you assume that the null hypothesis is true. If the p value is small enough, then we reject the null hypothesis, effectively concluding that “the observed effect is so unlikely to be due to chance that I reject the null hypothesis and instead must find an alternate hypothesis.”
Now the fun begins. We begin to discuss how extreme a result is required before we reject the null hypothesis. At this point, I break out my deck of cards and ask “suppose one of you draws a card. What would it take for you to believe that I was doing more than just guessing what card you drew?” I find my confederate student and ask them to draw a card. With a little delay to give them a chance to switch cards, I ask them to show the card to everyone except me. I then ask “would you reject the idea that I was merely guessing if …
- … I successfully guessed the color of the card?
- … the suit of the card?
- … the rank of the card but not the suit?
- … the exact card drawn – both rank and suit?”
Usually they say that they are willing to reject the idea that I am guessing if I can get the rank or both the rank and suit correct. After a pregnant pause and some false starts, I finally stammer out “7 of Hearts.” The room falls silent for a few moments before one or more students blurt out “you cheated!” I ask them how and they say something like “they’re all 7 of Hearts” or “you saw the card.” I then explain that this is exactly how hypothesis testing works. They have just rejected the null hypothesis of mere guessing because the observed event – correctly guessing the card – was incredibly unlikely to be due to chance. In fact, my guess had a chance of 1 in 52 or p = .019 and is therefore less than our typical threshold for rejecting a null hypothesis, p < .05. I find that students remember this demonstration and become a bit more confident in interpreting statistical tests for a long time afterwards.