A ranking does not identify the generating distribution
In the reproduced example, 500 observations were generated from a lognormal distribution. Phitter 1.0.4 returned 63 fitted candidates. Its first three rows, rounded, were:
| Rank | Distribution | AIC | BIC | KS | AD | χ² p |
|---|---|---|---|---|---|---|
| 1 | inverse_gaussian | 1976.32 | 1984.75 | 0.0352 | 0.5356 | 0.0832 |
| 2 | lognormal | 1976.43 | 1984.86 | 0.0305 | 0.3296 | 0.1678 |
| 3 | inverse_gamma_3p | 1978.62 | 1991.27 | 0.0281 | 0.3105 | 0.3054 |
The generating family came second. A finite sample can be approximated by several families, and the criterion used to order them need not favour the generating family in every sample.
Understand the sorting rule
Phitter first prioritises candidates with more tests that do not reject them. Continuous fits break ties by lower BIC, then AIC, then SSE. Discrete fits break ties by SSE and do not report AIC or BIC. The web results therefore should not be described as a table sorted only by fitting error.
BIC and AIC penalise model complexity differently. Compare them on the same observations and compatible likelihoods; comparing values from different datasets is not meaningful. In this example the first two models have equal parameter counts and an AIC gap of about 0.11. That small absolute gap provides little separation. Dividing it by the total AIC is not a useful measure of support.
The lognormal has lower KS, AD and chi-square statistics than the first row. That is compatible with the sorting rule: both models have three non-rejections, and the information criterion breaks the tie. A small statistic, a large p-value and a low information criterion are different kinds of evidence.
Count non-rejections correctly
Of the 63 fitted candidates, 20 have at least one test that does not reject them; 11 have no rejection among all three tests at the nominal 5% level. df_not_rejected_distributions reports the first group. Its name does not mean that every test accepted every listed model.
A non-rejection is not a probability that the model is correct. Its interpretation depends on test calibration and power. Searching many candidates also creates selection effects, but the significance level does not tell you how many false models will survive. Keep the full outcomes and review the assumptions behind the tests.
Check the quantities your decision needs
The fitted inverse Gaussian and lognormal have medians of 3.2595 and 3.2841, and 95th percentiles of 8.0596 and 7.9617. Their predictions for typical observations are close in this example.
Their estimated probabilities of exceeding 20 are approximately 0.000195 and 0.000396. The sample maximum was 16.4879, so this comparison concerns extrapolation. The difference may matter for a simulation concerned with rare overruns even if it is unimportant for describing the median.
Turn the ranking into a modelling decision
- Check the data preparation, units, possible values and whether the observations represent a stable process.
- Review several plausible candidates and their parameters. Quantify probability assigned to impossible outcomes before deciding whether an approximation is acceptable.
- Inspect cumulative and Q–Q plots to locate discrepancies. Scalar statistics do not show where the discrepancy occurs.
- Compare the probabilities or quantiles your application actually uses, and report sensitivity to the model choice.
- Preserve the software versions, options and complete ranking so another person can repeat the assessment.
For a simulation, carry plausible alternatives forward into a sensitivity analysis. If conclusions change substantially with the fitted family, that uncertainty belongs in the result.