Towards the evolutionary assessment of neural transformers trained on source code

In recent years, deep learning have become popular for solving tasks in a wide range of domains. With this growing diffusion, combined with architectures becoming increasingly complex and sophisticated, understanding how deep models make their predictions is now a crucial and challenging research issue. In this work, we are interested in analysing the behaviour of networks that deal with source code, and we address the problem along two experimental directions: in the first one, we study the activations of the neurons of a transformer trained in the detection of software vulnerabilities so as to identify if (and, eventually, where) some human understandable concepts emerge in the network. In the second one, we generate programs by applying a grammar-based evolutionary algorithm with a fitness function that favours individuals which stimulate (or weaken) the activations in neurons where given concepts majorly emerge. We then study the output of the network on sets of such evolved programs, i.e. how the evolutionary pressure along the direction of a concept affects the prediction. We finally discuss how this combination of evolutionary algorithms with fitness functions derived from the neural activations can be effective for explaining the decision process of deep models, and we suggest further research directions.