AN ADDITIVE ABLATION STUDY ON AUTOMATED CODE GENERATION: INSIGHTS FROM ROBERTABART_X
Keywords:
Automated Code Generation, Ablation Study, Transformer Models, RoBERTaBART_X, Attention Mechanisms, Retrieval-Augmented Generation.Abstract
Automated Code Generation (ACG) remains a trending research domain used to facilitate software development; however, previous studies have shown that ACG models often struggle with long-term context and exhibit poor domain adaptation and functional precision. As a result of this, we proposed a modular hybrid transformer, RoBERTaBART model augmented by Natural Language Processing (NLP) methods to address these limitations. We perform an ablation study on RoBERTaBART_X, a hybrid transformer that combines the context power of RoBERTa with the generation capability of BART. The following modules such as Task-Adaptive Pretraining (TAPT), Domain-Specific Data Augmentation (DA), Retrieval Augmented Generation (RAG), FlashAttention, and Sparse Attention were further utilized to enhance the hybrid model. Furthermore, the impact and the benefits of each module as well as the overall combination of the enhancements were carried out on CoNaLa, Django, and
CodeSearchNet datasets. It was shown that every module complements each other, with RoBERTaBART-X outperforming other variant models across all evaluation metrics, particularly in CodeBLEU and syntax validity. These results show that the hybrid modeling, retrieval, and iterative correction are key to significantly improving automated code generation performance.
