Physics-Guided Multimodal Multi-Agent Learning for Intelligent Transportation Systems

Intelligent transportation systems (ITS) require reliable coordination among multiple vehicles operating under heterogeneous behaviors and time-varying traffic conditions. Prior approaches face a recurring trade-off: physics-based controllers provide interpretability and constraint handling but can be brittle under model mismatch, whereas data-driven policies adapt to complex scenarios but often lack safety guarantees and transparent decision logic. We propose a hierarchical, physics-guided framework that separates semantic coordination from continuous control execution. At the regional level, a large language model (LLM) generates discrete, human-interpretable coordination directives (e.g., yielding priority and target gaps) from multimodal observations. At the vehicle level, each directive is realized by a physics-informed controller augmented with a learned residual policy, where the residual is constrained and safety-filtered to preserve feasibility and closed-loop robustness. Multimodal fusion via vision–language models (VLMs) supports context-aware coordination by combining visual cues with textual traffic descriptors and temporal signals. In highway merging simulations, the proposed framework improves traffic throughput by 23% and reduces collision rates by 31% relative to classical and learning-based baselines, indicating that semantic coordination and physics-grounded execution can be combined without sacrificing safety-critical control requirements.