Distance-KNN

This study compared the tabular classification performance of 4 general-purpose LLMs and distance-weighted KNN (Distance-KNN) across 19 tabular classification datasets.

The experiments used a controlled-variable design: for each test sample, the same retriever was used to obtain neighbors, and the exact same neighbors were used respectively as demonstrations for the LLM and as input to Distance-KNN.

At every retrieval pool size setting, Distance-KNN had the highest average accuracy.

From the global average, LLM reasoning did not surpass distance-weighted voting (Distance-KNN) on the tabular classification task.

Under the premise of low label consensus, some LLMs can surpass Distance-KNN in the near-neighbor and medium-distance neighborhoods, but in the farthest-distance group all LLMs lag on average.

Supplementary experiments on the number of demonstrations, prompt wording, retrieval representation and label-balanced demonstration selection largely preserved the main conclusions: the KNN average advantage is robust, and LLM gains are concentrated in low-consensus, near-distance scenarios.

For engineers building tabular classification systems, when retrieval quality is guaranteed and neighbor labels are consistent, Distance-KNN is a simpler and on-average more accurate baseline, and one should not default to assuming LLM reasoning is better.

Distance-KNN is a training-free classical instance-based learning algorithm that performs tabular classification through distance-weighted voting and is often used as a strong baseline.