What a goodness-of-fit result means
A goodness-of-fit test evaluates disagreement between observations and a specified model under particular assumptions. A p-value is not the probability that the model is correct. Non-rejection can occur because the model is adequate for the question, because the sample cannot reveal the discrepancy, or because the test is poorly calibrated for the setting.
Phitter reports Kolmogorov–Smirnov, Anderson–Darling and chi-square for continuous fits, and KS and chi-square for discrete fits. Read unavailable results separately from non-rejections.
Kolmogorov–Smirnov
KS measures the largest distance between empirical and theoretical cumulative distributions. It is generally more sensitive in the centre than in the tails. Its usual continuous, fully specified calibration does not directly cover parameters fitted from the same data or discrete observations. See NIST’s KS reference.
Anderson–Darling
AD weights cumulative discrepancies more heavily near the tails. Its critical values depend on the distribution and treatment of estimated parameters; NIST describes these requirements. Tail weighting can be useful, but it does not make AD universally most powerful or guarantee that it will reject first.
Chi-square
Chi-square compares observed and expected counts in bins or categories. It can be used for continuous or discrete data, with suitable grouping and expected counts. Record the bins and parameter treatment: changing them can change the result. A statistic computed from sparse expected counts needs particular care.
Use the test’s assumptions and your modelling question to decide which diagnostic matters. Do not choose a test solely because it produces the most favourable result.
Separate diagnostics from the ranking
In the reproduced lognormal example, the leading models have these rounded statistics:
| Distribution | KS | AD | χ² | Rank |
|---|---|---|---|---|
| inverse_gaussian | 0.0352 | 0.5356 | 25.5446 | 1 |
| lognormal | 0.0305 | 0.3296 | 22.4568 | 2 |
| burr_4p | 0.0266 | 0.2124 | 19.6533 | 7 |
Burr has smaller statistics than the other two displayed models, yet ranks below them. All three have three non-rejections; BIC breaks the tie, with AIC and SSE as subsequent criteria. These rows do not establish that Burr has the smallest statistics in the entire catalogue.
The run returned 63 fitted candidates: 20 with at least one non-rejected test, including 11 with all three tests not rejected, at the nominal 5% level. df_not_rejected_distributions represents the group of 20. Counting that table as models passing every test gives the wrong result.
Account for parameter estimation
A test against a model specified before observing the sample differs from a test after fitting its parameters to that sample. For suitable continuous problems, a parametric bootstrap can simulate and refit samples before comparing statistics; SciPy documents this procedure. Do not assume the default Phitter diagnostics perform this additional calibration.
When searching many candidates, the nominal significance level alone cannot predict how many false models will survive. Selection, power and calibration all matter. Avoid describing the surviving list as a set of verified distributions.
Use the results together
Review the data and possible values first. Read each test with its assumptions, then inspect empirical CDF and Q–Q plots to locate discrepancies. Compare the probabilities or quantiles needed by your application. Report plausible alternatives if the decision changes with the model. Reading a ranking demonstrates this process.