Lab · reproduction of an outside note

Reproduction: Does a tornado warning's shape predict whether it verifies?

Independent rerun of the 2026-09-15 note, done 2026-09-15 in the kiosks lab against TerraPulse exports cached 2026-09-12. Script: lab/false_alarm.py. Nothing from the author's run was reused except the note itself; the feature code, join and model were rewritten from its description and its code block.

Verdict: reproduced. Every reported number lands within rounding or within the model's own seed-to-seed spread. The conclusions stand as written. Two things in the note are underspecified and one finding deserves a second look; all three are below.

Data

Rated tornadoes with coordinates: 71,912 (2,272 preliminary 2026 reports with no rating excluded, as the note's "rated" implies).

Results, test set

Seed spread on the MLP is about 0.007, so the differences between the three ablations and the full model (0.01, 0.01, 0.03) are at the edge of what one seed can resolve. The ordering matches the note. The logistic model is stable to 0.001 across seeds and beats the MLP by the same small margin the note found, which supports the "it is linear" reading.

Verified rate by predicted quintile, low to high: note 11, 17, 20, 24, 30; this run 11, 16, 20, 23, 30.

Permutation importance, AUC drop when one column is shuffled on the test set:

Raw rates:

66 offices have 300 or more warnings in the period.

Where the note was silent, and what this run assumed

These did not change the answer, but a second reproducer could pick differently, so they belong in the note.

1. Units for the shape features. Aspect and compactness need a perimeter and a bounding box in consistent units. This run projects each polygon locally (longitude scaled by the cosine of its mean latitude, 111.32 km per degree) before measuring. In raw degrees the aspect ratio is stretched by up to 30% at northern offices. Both features are noise either way, so it did not matter here; it would for a paper that reported them.
2. Vertex count is taken as the exterior ring's coordinates minus the closing point. Every polygon in this export is a simple Polygon, so no multipolygon rule was needed.
3. Solar hour is UTC hour plus longitude over 15, with no equation of time. The note says "local solar hour" without the formula.
4. The label bracket. Touchdown time between start_utc and end_utc inclusive, inside the polygon by shapely's within (a point exactly on the boundary does not count). end_utc is the actual end after any cancellation, which is what the note's exclusion list implies: the label may use post-issuance information, the features may not.
5. Target-encoding smoothing of 20 pseudo-warnings is in the code block but not the text. Offices unseen in training (none here) fall back to the train base rate.
6. Weight initialisation. The note seeds tinygrad and numpy but the code constructs the network before seeding, so the seed governs the batch order and not the initial weights. This run re-initialises the layers under the seed. Either way lands within 0.005.

Worth a second look

Duration's raw effect is area in disguise. The note's point 2 reads the duration bins (16% to 32%) as forecaster commitment. In the model, initial duration has a permutation importance of 0.002 while area has 0.051, and the two are strongly related: rank correlation 0.41, median area 561 km² for warnings of 30 minutes or less against 1,339 km² for 60 minutes or more. Cross-tabulating the two:

Area climbs inside every duration bin. Duration does nothing inside the smallest quartile and adds about 14 points inside the largest. So the "60 minute warning verifies twice as often" line is mostly the size of the polygons that get 60 minutes, and the interaction (duration matters only when the box is big) is the one piece of non-linear structure the MLP could have used and apparently did not. This sharpens rather than contradicts the note's policy point: the thing that predicts verification is area, and area is what the 2012 policy shrank.

One bad initial expiry. LIX-TO-W-2-2011 carries an initial expiry 63 days after issuance (2011-01-01 to 2011-03-05). It is one row in 50,469 and survives standardisation harmlessly, but a duration feature should be clipped, and the dex row is worth a fix.

The mechanical area caveat is checkable now. The note's main open question is how much of the area effect is a bigger net catching more points. The tor export carries length_miles for rated tornadoes, so a partial answer needs no dossier fetch: restrict verification to tornadoes with a path under 2 miles (near-points) and see whether the area gradient survives. Not done here because it changes the label, and this run's job was to match the note.

Environment

tinygrad 0.14.0 on the CPU backend with clang present, shapely 2.1.2, pandas 2.2, one core. Join plus five models plus three extra seeds in 102 seconds. Tensor.binary_crossentropy_logits exists in 0.14.0 with a reduction argument, as the note's code assumes.