How to read a ranking without trusting the first row

In a real fit on lognormal data, the top-ranked model was not the lognormal — and the model that was had better test statistics.

A ranking that gets the answer wrong

Five hundred values were drawn from a lognormal distribution. Phitter fitted 63 continuous distributions to them. The result, sorted as the library returns it:

Rank	Distribution	AIC	KS	AD	χ² p
1	inverse_gaussian	1976.32	0.0352	0.5356	0.0832
2	lognormal	1976.43	0.0305	0.3296	0.1678
3	inverse_gamma_3p	1978.62	0.0281	0.3105	0.3054

The true model came second. The full run is documented in a reproducible fit, seed included, so this is not a hypothetical.

Nothing malfunctioned. The ranking answered the question it was asked; that question was not “which distribution generated this sample?”

The gap that decides first place

The AIC difference between first and second is 0.11, on values of about 1976 — a relative difference of 0.006%. Refit on a different sample from the same process and the order would very likely swap.

The convention in model selection is that an information-criterion difference below roughly 2 gives essentially no grounds for preferring one model over another. A gap of 0.11 is not a narrow win. It is a tie that the sort had to break somehow.

So the first question to ask of any ranking is not what is on top but by how much. A table sorted by a continuous criterion always produces a first row, whether or not the data justify one.

The winner scores worse on every test

Compare the two top rows on the goodness-of-fit statistics — lower is better for all three:

	inverse_gaussian	lognormal
Kolmogorov–Smirnov	0.0352	0.0305
Anderson–Darling	0.5356	0.3296
Chi-square	25.5446	22.4568

The second-place model is closer to the data by all three measures. It ranks below because the sort criterion is not any of them.

Information criteria such as AIC and BIC combine likelihood with a penalty for parameter count. Goodness-of-fit statistics measure distance between the empirical and fitted distributions. These are different questions, and they can disagree — as they do here. A ranking has to pick one; knowing which one it picked is the difference between reading the table and misreading it.

The effect is clearer further down. burr_4p has the best raw Kolmogorov–Smirnov statistic (0.0266) and the best Anderson–Darling (0.2124) of any model in the run. It ranks sixth, because BIC charges it for four parameters. Whether that penalty is appropriate depends on what you are doing — it usually is, but it is a modelling choice, not a fact about the data.

“Passes the tests” is not a filter

Of the 63 distributions fitted, 20 were rejected by none of the three tests at the 95% level.

That is the number that should change how the table is read. Passing is not evidence of being right; with n = 500 it is the normal condition for any model with roughly the correct shape. The tests are answering “is there enough evidence to rule this out?” and the answer, for a third of the catalogue, is no.

There is a second problem hiding in the same number. Running 63 tests at the 95% level means that even if every candidate were wrong, some would survive by chance. A ranking presents this as a shortlist of successes. It is closer to a list of models the sample is too small to eliminate.

What the ranking is actually good for

It reduces 63 candidates to a handful worth examining. That is a real service and it is the whole of the service. The decision comes from questions the table does not answer:

Is the support right? A model that assigns probability to impossible values is disqualified regardless of rank. Negative durations, fractional counts, values above a physical maximum.
Are the parameters plausible? In the run above the lognormal recovered mu=1.1891, sigma=0.5384 against true values of 1.2 and 0.55. Estimated parameters that make no sense for the process are a stronger signal than any score.
Where does the model disagree? A Q–Q plot shows the location and direction of the departure. Two models with near-identical statistics can fail in opposite tails.
Does the difference matter for your use? This is the decisive question, and it is entirely outside the table.

When the tie matters and when it does not

The top two models in this run are close to indistinguishable across the observed range. At the median they differ by 0.7%; at the 95th percentile by 1.2%.

In the far tail they separate. The estimated probability of exceeding 20 is 0.0002 under the inverse Gaussian and 0.0004 under the lognormal — a factor of two, on an event neither model saw, since the sample maximum was 16.49.

If you are summarising typical repair time, the choice between them is immaterial and you should not spend an afternoon on it. If you are sizing a buffer against rare overruns, the choice is the entire analysis and the ranking has not helped you make it. Fitting for Monte Carlo covers what to do in the second case.

A working rule

Treat the first row as a candidate, not a conclusion. Read the top group rather than the top line — every model within about 2 AIC of the leader. Check support and parameter plausibility before you check scores. Then ask what your model is for, and whether the models in that group disagree about it. If they do not, take any of them. If they do, the disagreement is your result, and reporting a single winner conceals it.