While deep learning has achieved excellent performance in many various tasks, because of its black-box nature, it is still a long way from being widely used in the safety-critical task like healthcare tasks. For example, it suffers from poor explainability problem and is vulnerable to be attacked both in the training and testing time. Yet, existing works mainly for local explanations lack global knowledge to show class-wise explanations in the whole training procedure. In this talk, I will introduce our effort on visualizing a global explanation in the input space for every class learned in the training procedure. Our solution finds a representation set that could demonstrate the learned knowledge for each class, which could provide analyse on the model knowledge in different training procedures. We also show that the generated explanations could lend insights into diagnosing model failures, such as revealing triggers in a backdoored model.