Understanding deep learning with activation pattern diagrams

The growing demand for machine learning tools to solve hard tasks, from natural language processing to image understanding, recently shifted the attention to understand and possibly to explain the behaviour of deep learning. Deep neural networks represent today the state-of-the-art in many applications that have been shown to be solved by data-driven approaches. However, they are also well known for their complexity, which hinders the interpretation of their functioning. To address this issue, researchers have lately focused either on understanding the optimization algorithms or on extracting information from a trained model; in this context we propose the Activation Pattern Diagram (APD) as a new tool to analyse neural networks by mainly focusing on the input data. The APD is a graphical representation of how a dataset is learned by a neural network with piecewise linear activation functions, such as the ReLU activation. By analysing the evolution of the diagram during the training procedure, the APD sheds light on the learning process and how data influences it. Additionally, we introduce a way to plot the APD to help the visualization and interpretation of the diagram.