Skip to yearly menu bar Skip to main content

Workshop: Deep Learning for Code

Code Summarization: Do Transformers Really Understand Code?

Ankita Sontakke · Manasi Patwardhan · Lovekesh Vig · Raveendra Kumar Medicherla · Ravindra Naik · Gautam Shroff


Recent approaches for automatic code summarization rely on fine-tuned transformer based language Models often injected with program analysis information. We perform empirical studies to analyze the extent to which these models understand the code they attempt to summarize. We observe that these models rely heavily on the textual cues present in comments/function names/variable names and that masking this information negatively impacts the generated summaries. Further, subtle code transformations which drastically alter program logic have no corresponding impact on the generated summaries. Overall, the quality of the generated summaries even from State-Of-The-Art models is quite poor, raising questions about the utility of current approaches and datasets.

Chat is not available.