In the development of anti-cancer drugs, a major scientific challenge is disentangling the complex interplay between high-dimensional genomics data derived from patient tumor samples, the organ of origin of the tumor, the drug targets associated with the specified treatments, and the ensuing treatment response. Furthermore, to realize the aspirations of precision medicine in identifying and adjusting treatments for patients depending on the therapeutic response, there is a need for building tumor dynamics models that can integrate the longitudinal tumor size measurements with multimodal, high-throughput data. In this work, we take a step towards enhancing personalized tumor dynamics predictions by proposing a heterogeneous graph encoder that utilizes a bipartite Graph Convolutional Neural networks(GCNs) combined with Neural Ordinary Differential Equations (Neural-ODEs). We apply the methodology to a large collection of patient-derived xenograft (PDX) data, spanning a wide variety of treatments (as well as their combinations) and tumor organs of origin. We first show that the methodology is able to discover a tumor dynamic model that significantly improves upon an empirical model in current use. Additionally, we show that the graph encoder is able to effectively incorporate multimodal data to enhance tumor predictions. Our findings indicate that the methodology holds significant promise and offers potential applications in pre-clinical settings.