Training controllers via reinforcement learning (RL) in simulation has emerged as a powerful approach for synthesizing robust and agile robotic behaviors evaluated in reality. We push the envelope of the simulation training paradigm by exposing problems encountered when learning agile behaviors requiring dynamic coordination between many joints, such as in the whole-body control of an arm-mounted quadruped robot. We found that training athletic whole-body control behaviors from scratch often fails, and the sim-to-real gap is greatly pronounced, especially on commodity hardware using complex-to-model harmonic drive actuators with limited sensing. We propose general solutions to overcome these issues that bypass tedious reward design schemes: (i) leveraging a pre-trained whole-body controller as a robust foundation that can be fine-tuned with RL for the target task; and (ii) a framework for modeling complex actuation mechanisms without requiring access to torque sensors. Along with several other design decisions that we elaborate, we achieve highly dynamic whole-body control behaviors such as ball throwing, lifting heavy weights, and sled pulling.