LDA-HSI — Wordification Design-Space Platform
A research platform that asks which representation a hyperspectral topic model should see: 19 wordification recipes × 4 topic-model backbones × Q∈{8,16,32}, scored on a 12-axis battery across six public scenes, with a live web app and API.
Business Context
Hyperspectral sensing is increasingly available, but choosing how to turn raw spectra into a topic-model corpus is treated as an afterthought — when in fact it can flip which method appears to win. Without a systematic comparison, a practitioner has no principled basis for picking a representation, and published results can hinge on an unexamined preprocessing choice.
Strategic Value
The defensible headline is a finding about methodology, not a single accuracy record: the wordification choice materially changes the conclusions, and there is no universal winner — there are two poles and a non-discriminating axis. V8 (NFINDR endmembers) is the portable, reproducible recipe (highest cross-backbone topic–label NMI 0.431, +0.034 over rank two; reseed reliability ≈0.96, stable across Q) — what you reach for when the backbone is uncertain. V20 (MI-weighted bands) is the LDA Q-scaling peak (F-7 NMI climbs to 0.563 at Q=32; it trails V12 at Q=8 then the ranking inverts to lead by +0.030 at Q=32) — what you reach for when LDA is fixed and Q≥16 is affordable. V11 is a backbone specialist (wins HDP and ETM, collapses under LDA). And F-1 classification accuracy does not discriminate — every recipe lands ~0.86–0.92 — so the platform refuses to headline an accuracy win. It is fully reproducible (1140-cell Q=8 grid, 100% populated; ~1726 committed artefacts; 133/133 API smoke per deploy) and surfaced through a 28-tab interactive web app, a public Q-extension API, and a P3/P4/P5 manuscript set.
The Challenge
In hyperspectral topic modelling almost all attention goes to the model and almost none to the representation fed into it. But a topic model is only as good as its vocabulary — and the open question is whether the wordification choice actually changes the conclusions, and which recipe to trust when.
Our Approach
Treat spectral variability as a corpus, then sweep the full design space: 19 wordification recipes (V1–V20, V16 reserved) in 7 families × 4 topic-model backbones (LDA, HDP, ProdLDA, ETM) × quantisation Q ∈ {8,16,32}, each combination scored on a 12-axis evaluation battery (coherence, topic–label NMI, seed stability, reliability, diversity, counterfactual robustness, cross-scene transfer, spatial coherence, endmember and LLM-judge baselines) with a hierarchical-Bayesian dominance test per axis. Run over six public labelled scenes plus spectral libraries, unmixing benchmarks, HIDSAG mineral subsets and MSI field data.
Key Performance Indicators
| KPI | Baseline | Result | Impact |
|---|---|---|---|
| Design space swept | 1 recipe, 1 backbone (the field default) | 19 recipes × 4 backbones × Q∈{8,16,32} | Representation choice made measurable |
| Portable leader (cross-backbone) | no portability analysis | V8 — F-7 NMI 0.431 (+0.034), reliability ≈0.96 | Use when backbone is uncertain |
| LDA Q-scaling peak | V20 trails V12 at Q=8 | ranking inverts; V20 leads +0.030 at Q=32 | Use when LDA fixed, Q≥16 |
Architecture
lda hsi platform
Which Wordification Matters?
LDA-HSI is the current state of the hyperspectral topic-modelling line that began as a 2022 conference paper. It treats spectral variability as a corpus — pixel spectra become documents of quantised spectral tokens — and asks the question the original paper only gestured at: which “wordification” should a topic model actually see, and does that choice change the conclusions? The offline experiment grid is the product; the public web app is a validated projection of its outputs.
The Design Space
Nineteen wordification recipes (V1–V20, V16 reserved) span seven families — band intensities, wavelet/derivative responses, absorption & endmember fractions, learnt codebooks, manifold coordinates, spatial regions, and label-aware MI weights. Each is run across four topic-model backbones (LDA, HDP, ProdLDA, ETM) and three quantisation levels (Q ∈ {8,16,32}), scored on a 12-axis evaluation battery with a hierarchical-Bayesian dominance test per axis. The Q=8 base grid is 1140 cells, 100% populated.
Datasets — Indian Pines Is Only the Headline
Where the 2022 work used a few small private mineral sets, the platform spans a deliberately broad surface so a representation’s win has to hold across sensors and scene types: six public labelled scenes (Indian Pines as the headline, plus Salinas, Salinas-A, Pavia University, KSC, Botswana), public spectral libraries (USGS splib07, ECOSTRESS), unmixing benchmarks (Samson, Jasper Ridge, Urban), the HIDSAG mineral subsets (the bridge back to the geometallurgy origin), and MicaSense MSI field samples.
The Verdict: Two Poles, No Leaderboard
V8 (NFINDR endmembers) is the portable recipe — highest topic–label NMI averaged across all four backbones (0.431, +0.034 over rank two) and reliable across reseeds (≈0.96). V20 (MI-weighted bands) is the LDA Q-scaling peak — its F-7 NMI climbs to 0.563 at Q=32; it trails V12 at Q=8, then the ranking inverts and V20 leads by +0.030 at Q=32. V11 is a backbone specialist (wins HDP and ETM, collapses under LDA). And F-1 classification accuracy does not discriminate — every recipe sits within ~0.86–0.92 — so no recipe is headlined on accuracy. An earlier “triple-axis win” framing was walked back after an internal audit (F-1 ties; V20 ties V12 on F-2 coherence); the surviving claim is the narrow, true one above.
Live, Reproducible
A React/Vite web app exposes a 28-tab interactive workspace + benchmarks; a public Q-extension API serves the topic-count trajectories; ~1726 reproducible artefacts back every figure, with 133/133 API smoke on each deploy. Companion manuscripts (P3 nineteen-recipe sweep, P4 backbone factorial, P5 interpretability) are in preparation. Live at lda-hsi.fasl-work.com.
Technology Stack
Visual assets for this project are not publicly available.