We investigate Predictive Coding Networks (PCNs) by analyzing their performance under different choices of activation functions. We link existing theoretical work on the convergence of simple PCNs to a concrete, toy example of a network - simple enough to explicitly discuss the fixed points in its training stage. We show that using activation functions that are popular in mainstream machine learning, such as the ReLU, does not guarantee the minimization of the empirical risk during training. We show non-convergence on an illustrative toy example and significant accuracy loss in classification tasks on common datasets when using ReLU compared to other activation functions.