Lab · a small transformer on the cyclone record

Can a small transformer read a cyclone's track and say how strong it will be tomorrow?

An analysis request in the standard format (docs/analysis-request-format.md, v0.1). TerraPulse tc dex, tracks fetched 2026-09-12. Written and run 2026-09-15.

1. Question

Given only the last four days of a tropical cyclone's track, tokenised into discrete symbols, can a two-layer causal transformer predict the wind speed 24 hours ahead better than persistence, a linear trend, and a lookup table? And does it match the feature-engineered network from the Tiny Forecaster on the rapid-intensification question, without being told what a trend is?

A yes means the track's own history carries a forecastable signal that a sequence model can find without hand-built features, which is the premise of every foundation-model-for-weather pitch in miniature. A no means the hand-built trends were doing the work, and a sequence model on this little data is a worse way to spend the same information. Either answer is useful for deciding how the lab treats sequence data from here.

2. Data

Export date 2026-09-12. The manifest carries no coordinates; every fix comes from the per-storm dossier. Wind is present on 58% of fixes across the whole archive and on 93% of synoptic fixes from 1980 on, which is the share the tokeniser reports as non-missing.

3. Label

At each grid step, the wind token four steps (24 h) later: the wind in knots rounded to the nearest 5 kt, so 38 classes from 0 to 185 kt. A step with no wind now, or no wind 24 h later, has no label and is masked out of the loss and the metrics. The label may use the future; the inputs at that step may not.

Two derived questions are scored from the same predicted distribution: the expected wind in knots (sum of bin centre times probability), and the probability of rapid intensification, the mass on bins at or above the current wind plus 30 kt. The RI base rate on the evaluated test steps is 5.8%, close to the Tiny Forecaster's 5.96% (that table also required a pressure value, which this one does not).

4. Features, which here means the tokeniser

The transformer sees no numbers. Every grid step becomes five integer tokens from five fixed vocabularies, and the five embeddings are summed. The edges are chosen a priori, not fitted, so the tokeniser cannot leak the split.

Why these. Wind is the label's own history and the only field a persistence forecast needs. Motion is direction and speed in one token, so a recurving storm reads as a change of symbol. Latitude band stands in for sea temperature and shear climatology; month and basin for season and region. Nothing about pressure, which the Tiny Forecaster used, because the point is to test the track alone.

The grid. Fixes at non-synoptic hours (3-hourly interpolations, landfall specials) are dropped. Each storm is placed on a 6-hourly grid from its first synoptic fix to its last; a missing grid step becomes a missing wind token and a missing motion token, and the next step's motion is also missing because it has no previous position. Longitude differences are wrapped at ±180 so a dateline crossing does not become a 359 degree jump.

Windows and padding. Each storm is left-padded with 12 pad steps and cut into windows of 16 with stride 4. Every real step therefore appears at positions 12 to 15 of exactly one window, with 12 steps (72 h) of context where the storm is old enough and pad tokens where it is not. Metrics are computed only at those four positions, so no step is scored twice and a storm's first day is scored with the same rules as its tenth. Training uses all 16 positions with the causal mask, which is four times more supervision per window than the scored positions alone.

What the tokeniser throws away. Anything finer than 5 kt, any motion finer than half a degree per six hours, and all wind values above 185 kt (none exist). Wind tokens above 150 kt (bin 30) are under 0.1% of training steps each; the model will not learn them, and the metrics treat those steps like any other.

5. Split

By storm year, so a storm's windows never straddle sets and the test years are unseen.

Same boundaries as the Tiny Forecaster.

6. Method

A decoder-only transformer in tinygrad, written from nn.Linear, nn.LayerNorm, nn.Embedding and Tensor.scaled_dot_product_attention with the causal flag. Width 64, 4 heads, 2 pre-norm blocks with a 256-wide GELU feed-forward, final norm, a 38-way head; 112,102 parameters. Adam at 1e-3, batch 64 windows, cross-entropy on the wind-token-ahead at every unmasked position, best validation epoch kept. No dropout, no weight decay, no learning-rate schedule. Seeds 0 and 1 for the full model; seed 0 for the wind-only ablation. Code: lab/tokens.py, lab/tc_transformer.py.

Baselines scored on the same steps: persistence (wind now), linear trend (wind now plus the change over the previous 24 h, persistence where that is missing), a unigram table (training frequency of the target bin) and a bigram table (target bin given the current wind bin, Laplace smoothed), both from the training years only.

Two tinygrad notes. sparse_categorical_crossentropy(ignore_index=-1) zeroes the masked positions but its mean still divides by every position, so with 29% of positions masked the reported loss is 0.71 of the true one and the gradient is scaled the same way. The script renormalises by hand. And TinyJit needs fixed shapes, so prediction batches are padded to 256 and trimmed.

7. Results, test years 2020 to 2026, 15,406 scored steps

MAE is on the expected wind (probability-weighted bin centre); the argmax bin is about 0.4 kt worse. Seed 0 was run twice and reproduced to every printed digit, so the seed governs the run. Seed spread on the full model is about 0.1 kt of MAE and 0.008 of AUC.

By basin, full model, seed 0:

By current intensity, full model: depressions under 34 kt 5.8 kt against 7.3 for persistence (5,770 steps); tropical storms 10.5 against 12.3 (6,389); category 1 and 2 hurricanes 18.0 against 22.7 (2,114); category 3 and up 18.1 against 24.8 (1,133). The gain over persistence grows with intensity, from 1.5 kt to 6.7 kt, because strong storms are the ones that change fastest.

Training curves: validation loss bottomed at epoch 2 for the full model (2.284) and epoch 4 for wind-only (2.309), then rose while training loss kept falling. 112,102 parameters on 30,767 windows overfits within three passes; the best-epoch rule is doing real work. The whole script, two full fits plus the ablation plus scoring, took 36 minutes on one CPU core.

8. What it means, caveats, next

Yes, the track alone carries signal and the transformer finds it. A two-layer model reading nothing but 5 kt wind symbols cuts the 24-hour error by 16% against persistence and reaches an RI AUC of 0.836; the four context tokens add a further 0.4 kt and 0.01 of AUC. That is within 0.01 of the Tiny Forecaster's feature-engineered network, which also saw pressure and was told what a 6, 12 and 24 hour trend is. The transformer was told nothing of the kind and had to learn the trend from the symbols.

The linear trend is a bad forecast, which the tokens knew. Extrapolating the last 24 hours makes the error worse than doing nothing (14.9 against 12.8 kt), because intensification runs end. The bigram table, which cannot see a trend at all, already beats persistence slightly by knowing that strong storms tend to weaken and weak ones to strengthen (regression to the mean). The transformer's edge over the bigram is exactly the part of the history that the table cannot hold: the shape of the last two days.

Caveats.

• Steps from one storm are correlated, so 15,406 is an overcount of the effective sample. Basin-level differences of 0.02 in AUC are not established.
• Wind values mix agencies and averaging conventions; the model learns the archive's habits, including West Pacific winds that run high, which is also the basin with the largest error.
• The RI comparison to the Tiny Forecaster is on different rows (that table needed a pressure value and used the 3-hourly fixes too) with a different base rate (5.8% against 5.96%), so "within 0.01" is indicative, not a matched test.
• No environmental inputs. Operational skill comes from shear, ocean heat and humidity, none of which is in the track.
• The 5 kt tokeniser cannot express a forecast finer than its bins, and the 38-way head spends capacity on bins above 150 kt that never occur in the test years.

Next, in order of expected value.

1. Pressure as a sixth token (10 mb bins) to make the Tiny Forecaster comparison fair, then a matched-rows scoring.
2. Multi-horizon heads (6, 12, 24, 48 h) from the same trunk; the tokeniser already carries every horizon.
3. Dropout or weight decay and a longer window (32 steps); the model is capacity-rich and data-poor, and the fix for that is regularisation, not depth.
4. Attention maps on a handful of storms to see whether the model looks at the last 24 hours or further back, which is the cheapest interpretability this architecture offers.

Follow-up, same day: v2 with pressure, regularisation and four horizons

Items 1 to 3 of the list above, run as one change. Code: lab/tokens2.py, lab/tc_transformer2.py. What changed: a sixth token for pressure (10 mb bins from 870 mb, 16 bins plus missing and pad; 17% of real steps have no pressure), dropout 0.1 on the embeddings and after each attention and feed-forward block, AdamW with weight decay 0.01, and one trunk with four heads predicting the wind token 6, 12, 24 and 48 hours ahead, trained on the mean of the four masked cross-entropies. Eight epochs, seed 0, 120,664 parameters, 17 minutes. Same tokeniser rules, windows and scored positions, so the 24-hour numbers are directly comparable to v1.

At 24 hours, pressure and regularisation changed nothing measurable: MAE 10.28 against v1's 10.33, RI AUC 0.842 against 0.845 (seed 1 of v1 gave 0.853, so this is inside the spread). What the regularisation did change is the training curve: validation loss kept falling to epoch 6 instead of turning at epoch 2, so the model is no longer capacity-limited by overfitting, it is limited by what the track carries.

The horizon curve is the useful new result. The edge over persistence is 4% at 6 hours, 14% at 12, 20% at 24 and 23% at 48. At 6 hours the 5 kt tokens are too coarse to say much more than "the same", and most fixes are reported in 5 kt steps anyway. At 48 hours the model is worth almost 5 kt.

The matched comparison, which was the point. 15,395 of the 15,406 scored test steps also appear in the Tiny Forecaster's table, keyed by storm and time, and the two labels agree on every one of them. On those rows:

The network retrained here scores 0.858 on all 28,883 of its own test rows (0.859 in the original note; a seed difference), and 0.848 on the 6-hourly subset, so the matched rows are slightly harder than its full set. The gap between the two models on identical rows is 0.006, well inside seed spread, and the average of the two beats either, which says they make partly different mistakes. A feature-engineered network and a token transformer reading the same track land in the same place, and the honest reading is that the track has about 0.85 of AUC in it for this task, whichever way you ask.

Per basin, v2 at 24 h. East Pacific 9.1 kt against 12.6 persistence and AUC 0.889; West Pacific 11.7 against 15.1, 0.822; North Atlantic 9.9 against 11.8, 0.863; South Indian 10.1 against 11.7, 0.818. The ordering matches v1 within 0.02.

Still open. Item 4, attention maps, and a longer window. The page lab/kiosks/tiny-forecaster.html now carries a transformer section with the horizon chart, the matched table, the vocabularies and the basin table (lab/page_data.py folds the JSON in and rebuilds).