What all three are asking

Kolmogorov–Smirnov, Anderson–Darling and chi-square answer versions of the same question: if the sample really came from this fitted distribution, how unusual would the disagreement we observe be?

That framing carries a consequence worth stating before any of the details. A large p-value is not evidence that the model is correct. It means the sample is too small, or the disagreement too mild, to rule the model out. Failing to reject is a weak conclusion, and it gets weaker as the sample shrinks.

All three also share a subtler problem. Their standard critical values assume the distribution was specified before seeing the data. In fitting, the parameters are estimated from the very sample being tested, which makes the model fit better than it should and the p-values optimistic. The tests are still useful for comparing candidates against each other; they are less trustworthy as absolute verdicts.

Kolmogorov–Smirnov: the largest single gap

KS takes the empirical cumulative distribution of the sample, the fitted cumulative distribution, and measures the largest vertical distance between them.

Because it reports a maximum, one location determines the whole statistic. And because the empirical CDF is constrained near 0 and 1 at the ends, that location is almost always near the middle of the distribution.

  • Sensitive to: shifts in location, disagreement in the body of the distribution.
  • Blind to: the tails. A model can be badly wrong about extreme values and still post an excellent KS statistic.
  • Use it for: a fast check that the bulk of the distribution is right.

Anderson–Darling: distance weighted toward the tails

AD integrates the squared distance between the two cumulative functions across the whole range, weighting each point by 1 / (F(x)(1 − F(x))). That weight grows without bound as F(x) approaches 0 or 1, so disagreement in the tails counts for far more than disagreement in the middle.

This makes it the sharper of the two for most practical work, and the one to prefer whenever extreme values matter.

  • Sensitive to: tail behaviour, and to shape errors anywhere.
  • Blind to: less than the others, which is why it usually rejects first.
  • Use it for: risk work, capacity planning, anything where the tail is the point.

Chi-square: comparing counts in bins

Chi-square divides the range into bins, compares observed counts against the counts the model predicts, and sums the normalised squared differences.

Binning is both its weakness and its unique strength. It is the only one of the three that applies naturally to discrete data, where values repeat and an empirical CDF is a step function with few steps. But for continuous data the result depends on a choice you make rather than on the data alone: change the number of bins and the statistic changes.

  • Sensitive to: systematic misfit across broad regions; usable on discrete data.
  • Blind to: structure finer than the bin width. Needs adequate expected counts per bin — the usual rule of thumb is at least five.
  • Use it for: discrete fits, and as a cross-check on continuous ones.

They disagree, and the disagreement is informative

From a real fit of 63 continuous distributions to 500 lognormal observations (full run here):

Distribution KS AD χ² Rank
inverse_gaussian 0.0352 0.5356 25.5446 1
lognormal 0.0305 0.3296 22.4568 2
burr_4p 0.0266 0.2124 19.6533 6

Three tests, three statistics, and the ordering they imply is not the ordering the ranking gives — because the ranking sorts by information criteria instead. burr_4p wins on all three tests and places sixth, having been charged for its four parameters.

Read together, the three tests tell you where models differ. Read individually, any one of them can be satisfied by a model that is wrong somewhere it does not look.

The number that matters more than any single test

In that same run, 20 of the 63 fitted distributions were rejected by none of the three tests at the 95% level.

Passing is not a distinction. It is the normal outcome for any candidate with roughly the right shape, because the tests are asking whether there is enough evidence to rule the model out — and with n = 500, for models that differ only in their tails, there is not.

This also raises a multiple-comparison problem that the tests were never designed to handle. Evaluate 63 candidates at the 95% level and a handful will survive on chance alone. A list of models that “passed” is closer to a list of models the sample cannot eliminate.

Practical reading order

  1. Check support first. A model that puts probability mass on impossible values is disqualified, whatever its p-values say.
  2. Read Anderson–Darling first among the three. It is the most sensitive and, in most applied settings, sensitive in the right place.
  3. Use KS as a body check. A good AD with a poor KS points at a problem in the middle of the distribution.
  4. Use chi-square for discrete data, and record the bin count alongside the result — it is not reproducible without it.
  5. Look at a Q–Q plot. All three statistics compress a whole function into one number. The plot shows where the model departs, and in which direction, which is the information you need to decide whether the departure matters.
  6. Do not stop at “not rejected.” Ask which model is defensible for the process, not which survived.

Phitter reports all three, with test statistic, critical value, p-value and the reject decision for each, in df_sorted_distributions. That the three are shown side by side is deliberate: the comparison between them carries more information than any one column.

Primary references