Mindless Statistics succintly shows the history of how NHST (null hypothesis significance testing) came to be, and how it actually is a degenerate hybrid of different approaches suggested first by Fisher, and then by Neyman and Person. It presents the 'null ritual' as why conclusions from psychology, or any non-experimental science, shouldn't be taken at face value:
In nonexperimental settings with large sample sizes, the probability of rejecting the null hypothesis of nil group differences in favor of a directional alternative is about 0.50.
(Meehl’s conjecture was shown to hold in a simulation by Waller in The fallacy of the null hypothesis in soft psychology.)
Chapter 4 presents a showcase of (mis)understanding of NHST. Can you assess the truthiness of the 6 statements below? Don't worry, 90% of professors who did not teach statistics, and 80% who did, endorsed >= 1 false statement.
Suppose you have a treatment that you suspect may alter performance on a certain task. You compare the means of your control and experimental groups (say 20 subjects in each sample). Further, suppose you use a simple independent means t-test and your result is significant (t = 2.7, d.f. = 18, p = 0.01). Please mark each of the statements below as “true” or “false.” “False” means that the statement does not follow logically from the above premises. Also note that several or none of the statements may be correct.
You have absolutely disproved the null hypothesis (that is, there is no difference between the population means).
You have found the probability of the null hypothesis being true.
You have absolutely proved your experimental hypothesis (that there is a difference between the population means).
You can deduce the probability of the experimental hypothesis being true.
You know, if you decide to reject the null hypothesis, the probability that you are making the wrong decision.
You have a reliable experimental finding in the sense that if, hypothetically, the experiment were repeated a great number of times, you would obtain a significant result on 99% of occasions.
Overall, a recommended read, with the exception of chapter 6, which seems a bit out of place with its play on Freudian Superego/Ego/Id.