Beth Tellman, “The Benchmark-Reality Chasm for Geospatial Foundation Models in Flood Applications”

Abstract: Are Geospatial Foundation Models (GeoFMs) meaningfully improving flood maps that would enable better flood adaptation and response? While evaluations from my Social Pixel Lab show marginally better quantitative scores (mIoU) compared to U-Nets trained from scratch, these gains often fail to translate into real-world applications outside the "lab". We find it difficult to replicate high benchmark accuracy from global datasets like Sen1Floods11 and FloodPlanet (covering Sentinel-1, Sentinel-2, and Planetscope) to operational settings for end-users. End-users require seamless maps across thousands of km2 in hazy or cloudy conditions—scales orders of magnitude larger than the 5-10km2 "chips" used in standard benchmarking. I’ll show examples from the Rio Grande Valley, Texas FLUJOS (Flood Justice Utilizing Satellite Observations-https://assets.rgvflood.arizona.edu/ ) used by community-based organizations to recent efforts in Michigan to support GLISA- a Great Lakes regional climate hub government-university partnership.

We are finding that "high-scoring" GeoFMs often qualitatively fail to capture critical spatial nuances which are highly relevant for flood water near buildings and infrastructure. For stakeholders like insurers, who prize temporal consistency of inundated area over pixel-perfect spatial accuracy, the standard IoU (Intersection Over Union) metric is fundamentally insensitive to the errors that matter most for financial transactions. I’ll show an alternative validation approach used in a commercial context by Floodbase, which validates models using a "convergence of evidence" —integrating news media, rainfall and streamflow observations, and insurance claims. I conclude with a call to stoke ideas about how to benchmark GeoFMs and machine learning more broadly to bridge the chasm between today’s benchmarking standards and flood-relevant decision-making reality.

Bio: Beth is an Assistant Professor at the University of Wisconsin-Madison, in the Nelson School of Environmental Studies whose research addresses the causes and consequences of global environmental change on people, with a focus on flood risk and land use change. She engages in multiple disciplines and methods to “socialize the pixel” or understand the social processes behind environmental change captured in satellite image pixels and leverage satellite data to improve human well-being. She is a co-founder and Chief Science Officer of Floodbase, a public benefit corporation that leverages remote sensing to build flood parametric insurance products. Beth co-founded an NGO in Mexico, Umbela, to promote transformation from Global South Perspectives. Beth is passionate about co-producing knowledge with actors outside academia to achieve social impact.