Efficient clinical trial recruitment and personalized treatment depend on the ability to predict future disease progression from medical images. However, often there is a lack of well-defined biomarkers that can predict future disease development and a wide inter-subject variation in disease progression speed. We address these issues in the context of predicting the onset of late dry Age-related Macular Degeneration (dAMD) from retinal OCT scans. To model the CDF of future dAMD onset, we propose jointly training an AMD stage classifier with a Neural-ODE that predicts the future disease trajectory. A temporal ordering is imposed that inversely relates the distance from the decision hyperplane of the classifier to the time-to-conversion. Furthermore, we ensure intra-subject temporal consistency by incorporating pairs of longitudinal scans from the same eye during training. Our method is evaluated on a longitudinal dataset comprising 235 eyes (3,534 OCT scans), including 40 converters. The results demonstrate the efficacy of our approach, achieving an average eye-level AUROC of 0.83 in predicting conversion within the next 6,12,18 and 24 months, outperforming several popular survival analysis methods.