Nuclear and particle physicists seek to understand the structure of matter at the smallest scales through numerical simulations of lattice Quantum Chromodynamics (LQCD) performed on the largest supercomputers available. Multi-scale techniques have the potential to dramatically reduce the computational cost of such simulations, if a challenging parameter regression problem matching physics at different resolution scales can be solved. Simple neural networks applied to this task fail because of the dramatic inverted data hierarchy that this problem displays, with orders of magnitude fewer samples typically available than degrees of freedom per sample. Symmetry-aware networks that respect the complicated invariances of the underlying physics, however, provide an efficient and practical solution. Further efforts to incorporate invariances and constraints that are typical of physics problems into neural networks and other machine learning algorithms have potential to dramatically impact studies of systems in nuclear, particle, condensed matter, and statistical physics.