Training convolutional neural networks (CNNs) directly from RGB pixels hasenjoyed overwhelming empirical success. But can more performance be squeezedout of networks by using different input representations? In this paper we proposeand explore a simple idea: train CNNs directly on the blockwise discrete cosinetransform (DCT) coefficients computed and available in the middle of the JPEG codec. We modify libjpeg to produce DCT coefficients directly, modify a ResNet-50 network to accommodate the differently sized and strided input, andevaluate performance on ImageNet. We find networks that are both faster and moreaccurate, as well as networks with about the same accuracy but 1.77x faster thanResNet-50.