Field notes · July 2026
Verification
The fire model was pointing the wrong way
We built a scorekeeper for our own wildfire risk maps. It found a data leak, a dead input, and a weather term that ranked fire backwards. None of it was visible from the map.
For most of its life, our wildfire risk model looked right. It produced maps with red where you would expect red: the historic burn scars, the dry slopes, the places that make the news. It fused a Bayesian prior built from years of satellite fire detections with a forecast hazard index built from temperature, humidity, wind and soil moisture. By the standards of the genre, a perfectly respectable model.
Nobody had ever checked whether it pointed at the right places.
That is not a criticism of us in particular. It is the normal condition of hazard mapping. A risk map is a plausible-looking object, and plausibility is exactly the quality that makes it hard to audit. Red over the Cerros Orientales looks correct. Red over the Colorado Front Range looks correct. Who is going to say otherwise?
So we built something to say otherwise.
The scorekeeper
The idea is embarrassingly simple. Take the map the model would have made three weeks ago, using only the data available at that moment. Then look at where fires actually broke out in the twenty-one days that followed. Did the model's hot pixels burn?
One number summarises the answer: the AUC. It measures ranking. At 0.5 the model orders pixels no better than a coin flip. At 1.0 it ranks every burned pixel above every unburned one. It is exactly what a civil protection officer implicitly needs. Not "what is the probability", but "which places do I send the crew to first".
And there is a third region of that scale that we did not expect to spend so much time in. Below 0.5. A score of 0.35 is not a weak model. It is a model with real information in it, pointed the wrong way. Flip its sign and you get 0.65.
Reading the scale. Below the null line, the signal is not missing. It is reversed.
The dashed circle is the mirror: what an inverted score is worth once you flip its sign. A model at 0.35 knows a great deal about where fire goes. It just has the arrow reversed.
Three things it caught immediately
It caught the model reading the answer sheet
Our legacy model scored 0.975, which is nearly perfect and therefore nearly certainly wrong. Its fire-history layer drew on satellite detections from a rolling three-year window ending today, and that window contained the very fires it was being tested against. A student marking their own exam with the answer key open. Once every input was rebuilt to end strictly before the test window opened, 0.975 became 0.62.
It caught a dead input
The "where fires happened before" layer was reading from a near-real-time satellite feed that retains about two months of data, while our code asked it for the years 2019 through 2025. It got nothing back. The model had been silently fusing against a blank layer for months. The maps still looked fine. That is the entire point: the maps always look fine.
It caught the model pointing backwards
In Colorado the forecast hazard term scored 0.33. The reason turned out to be almost funny. The index measures dryness, and Colorado's driest ground is the eastern plains: shortgrass steppe with essentially nothing to burn. The fires are up in the wetter, forested mountains. The model was faithfully ranking aridity, and in Colorado aridity is an anti-map of fuel.
We added a term for standing biomass, fuel load rather than mere fuel presence, and the inversion went away. Then Portugal produced something we did not expect at all.
Absolute weather is a map of elevation
In the Viseu district of Portugal, the fire-history prior scored 0.760. Strong. The forecast weather term scored 0.350, inverted, and inverted hard. And here is the part that should worry anyone who builds fusion models.
When the two were combined the way our Bayesian engine combines them, the result was 0.373.
The fusion took a good prior and destroyed it. It did not average two signals into something mediocre. It took a 0.76 and turned it into a 0.37, because mixing in a term that points the wrong way does not dilute your signal. It reverses it.
The mechanism is the same one we found in Colorado, wearing different clothes. Inside a fire-season area, on a hot September day, everywhere is hot and dry. The only real spatial variation left in an absolute weather field is the lapse rate: temperature falls with altitude. So "hottest pixel" quietly means "lowest pixel". And in Iberia the hot lowlands are farmland, while the fires burn in the cooler, forested uplands.
An absolute fire-weather index, mapped across mountainous terrain, is a disguised elevation model.
The fix follows directly. Instead of asking "is this pixel hot", ask "is this pixel hotter and drier than it normally is at this time of year". Rank every pixel against its own climatology. That anomaly is comparable across elevation in a way that raw temperature is not.
Viseu, Portugal. Same prior, same fusion, one substitution.
The anomaly does not locate fire. On its own it is a coin flip. It simply stops actively lying, and that is enough for the prior to survive contact with it.
Weather tells you when. History tells you where.
That last line is the whole architecture, and we had been getting it wrong.
We were asking a single map to answer two different questions, then grading it on only one. Fire risk factorises. There is a spatial component, susceptibility: where can fire happen at all? Fuel, terrain, fire history, people. It changes slowly, and a map is the right object for it. And there is a temporal component, danger: how unusual are conditions right now? That is weather, and on any given day it is essentially uniform across a region.
Scoring a temporal signal with a spatial metric is a category error. It is why our weather term looked useless or worse. We kept demanding the one thing it cannot do, which is locate.
So we score the two separately now. Susceptibility gets an AUC, as a map should. Danger gets collapsed to one number per week and tested against the question it can actually answer: do the dangerous weeks burn more?
Two regimes, opposite signs
Set the regions side by side and the contrast is sharp.
| Surface | Viseu, PT | Colorado, US |
|---|---|---|
| Fire-history prior | 0.760 | 0.520 |
| Legacy multiplicative model | 0.699 | 0.693 |
| Weather, absolute | 0.350 | 0.662 |
| Ignition pressure: roads, people, clearing | 0.604 | 0.422 |
Two things stand out. The fire-history prior is excellent in Portugal and worthless in Colorado. In a small, densely settled Iberian district with a long ignition record, fire recurs where it has recurred. Across a vast western state, it does not, at least not at this horizon.
And ignition pressure splits the regions cleanly. In Portugal, proximity to roads and people predicts fire. In Colorado it anti-predicts it. Which is coherent, and not subtle: Iberian fires start where people are, and Colorado's start in remote wilderness, away from them.
That has a hard consequence for anyone shipping a global fire model. The same predictor changes sign between regions. A single set of fusion weights cannot be right in both places. Ours was tuned by hand, using intuition, and the intuition turned out to be regionally parochial without our knowing it.
Two hypotheses we killed
We want to be explicit about the ones that did not survive, because the ones that die are the reason to trust the ones that live.
The reburn hypothesis
At one point Portugal's prior came back at 0.130: strongly inverted, predicting almost perfectly where fire would not be. We had an elegant explanation ready. Portugal burned enormously between 2017 and 2022; ground that burned recently has spent its fuel; fire moves on to what is left. Recent scars repel new fire. It is ecologically plausible, it is in the literature, and we were ready to write it up.
It was one dry June window. When we measured the actual fire season from the burn record and re-anchored the test inside it, the sign flipped back and the effect vanished. The story had been built on a single test window that fell outside the season.
Our own regime index
We built a diagnostic meant to predict which failure mode a landscape is in, and to set the fusion weights accordingly. We wrote its prediction into the model's console output, in falsifiable form, before running the test.
It failed. In Colorado it predicted the prior would win: weather won and the prior sat at chance. In Portugal it predicted weather would win: the prior won decisively and weather was inverted. Zero for two. We keep it in the code, printed and clearly labelled as falsified, because a diagnostic that fails a public test is more useful than one that quietly disappears.
What this is really an argument for
Not for our model. Our model, on this evidence, is a weather index in Colorado, a fire-history map in Portugal, and a fusion engine that was actively destroying signal in at least one of them.
It is an argument for the scorekeeper.
Every one of these findings was invisible from the map. The leak, the dead input, the inverted term, the fusion that ate its own prior, the two hypotheses that died. All of them. The maps looked correct throughout, and some of the wrongest maps looked the most convincing, because a saturated red blob over a historic burn scar is exactly what a viewer expects to see.
The lesson is not that our model was bad. It is that an unverified risk map cannot be known to be good, and the field ships unverified risk maps constantly, into decisions about where to pre-position crews and whom to evacuate.
The scorekeeper is not a nice-to-have sitting beside the model. It is the thing that makes the model a scientific object rather than a picture.
Coda: the region we actually care about
Capybara exists for the Bogotá region: the Cerros Orientales, the Sabana, the dry slopes around Suesca. So we ran the timing test there. Does the weather anomaly rise in the weeks that burn?
Across 120 weekly windows and six fire seasons, the rank correlation was 0.031. Not significant. Zero, to a good approximation.
Weather does not tell you when the Sabana burns. That is a null result with teeth, and it points at something we already half knew: around Bogotá, fire is not a meteorological process at all. It is an agricultural and social one. People burn when they decide to burn, on the rhythms of pasture renewal and land clearing, not when the atmosphere says so.
Which means the layers we compute only to count victims, the roads, the population, the clearing front, may be the ones that should be locating the hazard. The spatial half of that test is running now.
Note: Only a member of this blog may post a comment.