Answer

Is a normality test worth running on your data?

Inspired by a question on Cross Validated · September 21, 2026

distributionshypothesis testingnormality

What a normality test really answers

Tests such as Shapiro-Wilk, Anderson-Darling or Lilliefors start from the assumption that the data are exactly normal, then ask whether the sample looks too different from that to be chance. A small p-value tells you the departure is detectable. It says nothing about whether the departure is big enough to matter for your analysis.

Real data are almost never exactly normal. Measurements are rounded, bounded, or a blend of slightly different groups. So the null hypothesis is false before you collect a single observation, and the only open question is whether your sample is large enough to notice.

Why sample size flips the answer

That is the heart of the argument that these tests are close to useless as gatekeepers:

• Large samples: power is high, so even tiny, harmless departures produce small p-values. At the same time, methods like the t-test and ordinary regression become less sensitive to non-normality as the sample grows, because averages behave more and more like normal variables.
• Small samples: power is low, so a real problem, such as a strong skew, can pass unnoticed. This is exactly when the method's assumptions matter most.
• Either way, the test result and the practical consequence move in opposite directions.

There is a second issue. If you run a test first and choose your analysis based on the outcome, the error rates of the final procedure are no longer what the textbook promises.

A simulation you can run

The code below compares truly normal data with data from a t distribution with 15 degrees of freedom, which is only mildly heavy-tailed (its excess kurtosis is about 0.55) and would look normal on nearly any plot. It counts how often Shapiro-Wilk rejects at the 5% level.

set.seed(2024)

# Mildly non-normal data: t distribution with 15 df (slightly heavy tails)
# vs. genuinely normal data. Shapiro-Wilk rejection rates at alpha = 0.05.
reject_rate <- function(n, rgen, reps = 500) {
  mean(replicate(reps, shapiro.test(rgen(n))$p.value < 0.05))
}

sizes <- c(20, 100, 1000, 5000)
res <- data.frame(
  n         = sizes,
  normal    = sapply(sizes, reject_rate, rgen = rnorm),
  t15       = sapply(sizes, reject_rate, rgen = function(n) rt(n, df = 15))
)
print(res)

# The same tiny departure, judged by a Q-Q plot correlation instead of a p-value
x <- rt(5000, df = 15)
cat("Q-Q correlation, t15 sample of 5000:",
    round(cor(sort(x), qnorm(ppoints(5000))), 4), "\n")
cat("Shapiro-Wilk p-value on that sample:",
    signif(shapiro.test(x)$p.value, 3), "\n")

In one seeded run with 500 repeats per cell, the rejection rate for the mildly heavy-tailed data was 9.4% at n = 20, 13.8% at n = 100, 56.8% at n = 1000 and 99.8% at n = 5000. For truly normal data it stayed near the expected 5% (between 3.4% and 5.4%). The single sample of 5000 had a Shapiro-Wilk p-value of about 2.3e-09, yet its Q-Q correlation was 0.9983, meaning the points hug the line almost perfectly. These figures come from one seeded run and will vary a little with a different seed.

Notice that the distribution never changed. Only the sample size did, and the test went from missing it to declaring it almost certain.

What to do instead

1. Plot the data. A normal Q-Q plot (qqnorm() and qqline() in R) or a histogram shows the size and shape of any departure, which is what you care about.
2. Ask what your method needs. For a t-test or a regression coefficient, moderate non-normality in a decent sample is rarely a problem, while outliers and strong skew in a small sample can be. For prediction intervals, tail behavior matters directly.
3. Check the residuals, not the raw outcome, when fitting a regression model. The normality assumption concerns the errors, not the response variable.
4. Use robust or resampling methods (a bootstrap, a rank-based test, or a model with a better-suited distribution) when a real departure is visible and it affects your question.
5. Report effect sizes for departures such as skewness or excess kurtosis if you need a number, rather than a bare p-value.

When a normality test is still fine

A test is not forbidden. Used as one piece of evidence next to a plot, it can back up what you see. It is also reasonable in quality-control or simulation work where you truly want to check a generator or a process against a specified normal model. In R, shapiro.test() accepts samples of 3 to 5000 values, which is a practical hint that it was designed for modest sample sizes.

The habit to drop is the two-step recipe of running a test, then switching methods automatically depending on the p-value. Decide based on the plot, the sample size, and the sensitivity of your method.

Related tools and guides

• Which statistical test should I use?
• The Data Audit Checklist to Run First
• Normality test (Wikipedia)
• What does kurtosis tell you about your data?

More answered questions

• What does kurtosis actually tell you about your data?

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.