ARFBench: Benchmarking Multimodal Time Series Reasoning for Software Incident Response

Time series question-answering (TSQA), in which we ask natural language questions to infer and reason about properties of time series, is a promising yet underexplored capability of foundation models. In this work, we present ARFBench, a novel TSQA benchmark that evaluates the understanding of multimodal foundation models on time series anomalies prevalent in software incident data. ARFBench consists of 750 questions across 142 time series and 5.38M data points from 63 production incidents sourced exclusively from internal telemetry at an observability platform vendor. We evaluate leading proprietary and open-source LLMs, VLMs, and time series FMs on ARFBench and observe large room for improvement, as the leading model (GPT-5) achieves a 53.2% F1. We next demonstrate the promise of specialized multimodal approaches. We develop a novel TSFM + VLM hybrid prototype which we post-train on a small set of synthetic and real data, yielding the top overall F1 among evaluated models, a 7.1 percentage point improvement. The benchmark is available at \url{https://huggingface.co/datasets/XXXXXX/ARFBench}.