The solution of partial differential equations (PDEs) by deep neural networkstrained to satisfy the differential operator has become increasingly popular. Whilethese approaches can lead to very accurate approximations, they tend to be over-confident and fail to capture the uncertainty around the approximation. In thiswork, we propose a Bayesian treatment to the deep Galerkin method (Sirignano &Spiliopoulos, 2018), a popular neural approach for solving parametric PDEs. Inparticular, we reinterpret the deep Galerkin method as the maximum a posterioriestimator corresponding to a likelihood term over a fictitious dataset, leading thusto a natural definition of a posterior. Then, we propose to model such posterior viathe Laplace approximation, a fast approximation that allows us to capture mean-ingful uncertainty in out of domain interpolation of the PDE solution and in lowdata regimes with little overhead, as shown in our preliminary experiments.