The rapidly growing number and variety of Large Language Models (LLMs)present significant challenges in efficiently selecting the appropriate LLM fora given query, especially considering the trade-offs between performance andcomputational cost. Current LLM selection methods often struggle to generalizeacross new LLMs and different tasks because of their limited ability to leveragecontextual interactions among tasks, queries, and LLMs, as well as their depen-dence on a transductive learning framework. To address these shortcomings, weintroduce a novel inductive graph framework, named as GraphRouter, whichfully utilizes the contextual information among tasks, queries, and LLMs to en-hance the LLM selection process. GraphRouter constructs a heterogeneousgraph comprising task, query, and LLM nodes, with interactions represented asedges, which efficiently captures the contextual information between the query’srequirements and the LLM’s capabilities. Through an innovative edge predictionmechanism, GraphRouter is able to predict attributes (the effect and cost ofLLM response) of potential edges, allowing for optimized recommendations thatadapt to both existing and newly introduced LLMs without requiring retraining.Comprehensive experiments across three distinct effect-cost weight scenarios haveshown that GraphRouter substantially surpasses existing routers, delivering aminimum performance improvement of 12.3%. In addition, it achieves enhancedgeneralization across new LLMs settings and supports diverse tasks with at least a9.5% boost in effect and a significant reduction in computational demands. Thiswork endeavors to apply a graph-based approach for the contextual and adaptiveselection of LLMs, offering insights for real-world applications.