Hyperparameter Method Comparison

Addendum to LossLandscapeProbe — Evaluating the LLP meta-model's cheap-probe strategy against standard HP search methods across four datasets.

Experiment Design

For each dataset in {CIFAR-10, MNIST, Fashion-MNIST, SVHN}, we compare five hyperparameter selection methods, all training the same FlexibleCNN architecture (3-layer CNN, ~32 channels, dropout) for 50 epochs:

Method	How It Works	Search Budget
LLP Meta-Model	20 cheap probes (10% data, 5 epochs each) → train RF surrogate → predict best config → full training	~200s probe overhead
Fixed Default	Adam, lr=0.001, channels=32, dropout=0.2 — no search at all	0s
Random Search	20 random configs, train each fully, pick best	20 × full training
Hyperband	Optuna RandomSampler + HyperbandPruner, 30 trials	30 trials (pruned)
BOHB	Optuna TPESampler + HyperbandPruner, 30 trials	30 trials (pruned)

Primary metric: Total wall-clock seconds (search overhead + final training) to reach within 95% of the best accuracy found by any method. The meta-model is a supervised surrogate (Random Forest) trained via an active-learning-style loop on the target dataset itself — not cross-dataset transfer.

Results Summary

Dataset	LLP Acc	Default Acc	Best (Search)	LLP Cost	Default Cost	Search Cost	LLP vs Default
CIFAR-10	82.2%	84.0%	85.0%	541s	347s	3,765s	−1.8%
MNIST	97.4%	99.3%	99.6%	607s	409s	5,671s	−1.9%
Fashion-MNIST	91.1%	91.5%	93.1%	606s	408s	3,774s	−0.4%
SVHN	92.1%	92.1%	93.8%	962s	567s	6,565s	≈ 0%

"Search Cost" shows the cheapest search method (Hyperband) for each dataset. Random Search was consistently 1.5–2× more expensive.

CIFAR-10

The hardest dataset in the pool. LLP predicted SGD with lr=0.024, momentum=0.42, channels=40. This trained successfully (unlike the earlier cross-dataset attempt) but fell 1.8% short of the default Adam config.

Key finding: LLP cost 541s vs Hyperband 3,765s (7× cheaper) but accuracy gap of 1.8% to the default and 2.7% to Hyperband's best.

MNIST

The easiest and most well-studied dataset. Every method exceeds 97%, and the default Adam lr=0.001 is near-optimal at 99.3%. This is not a meaningful test of the meta-model's value — the default config has been effectively optimised by the entire ML community over decades.

Fashion-MNIST

A harder variant of MNIST (clothing items instead of digits). The accuracy gap narrows to just 0.4% behind the default — the closest result after SVHN.

SVHN

The largest and most “real-world” dataset (street-view house numbers). LLP effectively tied the default at 92.1% — a gap of just 0.03%. This is the strongest result for the meta-model.

Key finding: On the most complex dataset, LLP matched the default within 0.03% at 962s total cost — while Hyperband needed 6,565s to achieve 93.6% (only 1.5% better).

Analysis

Verdict: Promising but Limited

The formal kill criterion fires as FAIL — the LLP meta-model does not beat the fixed default on accuracy for any dataset. However, the nuanced picture is more encouraging:

What works

Cost efficiency: LLP is 5–12× cheaper than Hyperband, BOHB, and Random Search in total wall-clock time.
Reasonable configs: The RF surrogate consistently predicts configs that train successfully and reach 95–98% of the best-found accuracy.
Scaling trend: The gap narrows on harder datasets (1.9% on MNIST → 0.4% on Fashion-MNIST → 0.03% on SVHN), suggesting the meta-model adds more value when the default config is less optimal.

What doesn't work

Can't beat Adam lr=0.001: This turns out to be an exceptionally strong default for SimpleCNN across all four datasets. The meta-model's predictions (often SGD with higher learning rates) are reasonable but not superior.
Probe overhead: The ~200–400s spent on cheap probes makes LLP more expensive than simply running the default, even though it's far cheaper than full search methods.
Small probe sample: With only 20 probe configs, the RF has limited data to learn from. Increasing the budget could improve predictions.

Future directions

Equal-budget comparison: Give the LLP meta-model the same compute budget as Hyperband (~3,700s) — run 200+ probes or iterate the predict-probe loop — and see if it can match or beat search methods.
Hybrid approach: Use cheap probes to narrow the HP space, then run Hyperband within that reduced space. Combines LLP's speed for coarse filtering with Hyperband's thoroughness for fine-tuning.
Novel datasets: Test on datasets where no established default exists. The value of cheap probing should be highest when you genuinely don't know what config to use.