In hypothesis testing, we use statistical methods to determine the level of evidence for or against a hypothesis. In this process, we may make two types of errors: Type I and Type II.
- Type I error: When we reject a null hypothesis that is actually true, we make a Type I error. This is also known as a false positive. It occurs when we set a low level of significance (e.g. Ξ±β=β0.05) and the evidence against the null hypothesis seems strong, so we reject the null hypothesis even though it is true. The probability of making a Type I error is equal to the level of significance that is chosen.
- Type II error: When we fail to reject a null hypothesis that is actually false, we make a Type II error. This is also known as a false negative. It occurs when we set a high level of significance and the evidence against the null hypothesis seems weak, so we fail to reject the null hypothesis even though it is false. The probability of making a Type II error is denoted as Ξ² and it depends on the sample size, the level of significance, and the effect size of the alternative hypothesis.
To illustrate this more clearly, letβs consider an example. Suppose we are testing a null hypothesis that a coin is fair (i.e. equally likely to land on heads or tails). We conduct a hypothesis test at a significance level of Ξ±β=β0.05 by flipping the coin 100 times and counting the number of times it lands on heads. If we get 60 heads, we would reject the null hypothesis and conclude that the coin is biased towards heads. However, if the true probability of getting heads is actually 0.5 (i.e. the null hypothesis is true), then our decision to reject the null hypothesis would be a Type I error.
On the other hand, if we repeat the experiment with the same null hypothesis but a sample size of 10 flips and get only 2 heads, we may fail to reject the null hypothesis and conclude that the coin is fair. However, if the true probability of getting heads is actually 0.8 (i.e. the alternative hypothesis is true), then our decision to not reject the null hypothesis would be a Type II error.
In conclusion, Type I and Type II errors are both possible outcomes of a hypothesis test and the probabilities of making each kind of error depend on the level of significance, sample size, and effect size of the alternative hypothesis. It is important to choose an appropriate significance level and to be aware of the possibility of both types of errors when interpreting the results of a hypothesis test.