GenePrune : Automated Pruning of Large Language Models for Code using Genetic Algorithm

Large Language Models (LLMs) for code generation exhibit remarkable capabilities but face deployment challenges due to their high computational and memory demands. Traditional pruning methods, often based on static heuristics like magnitude-based weight pruning, fail to effectively balance sparsity and performance, particularly for structured tasks such as code generation. To address this, we propose GenePrune, a novel genetic algorithm-based pruning framework that optimizes pruning masks for pre-trained Code LLMs without requiring costly retraining. GenePrune iteratively refines pruning configurations through evolutionary operations such as crossover and mutation, guided by a fitness function that balances model sparsity and task-specific performance. Experiments on open-source models like CodeT5 demonstrate that GenePrune achieves superior pruning efficiency, significantly reducing model size while maintaining high BLEU scores for code generation tasks. Our results highlight GenePrune as a promising approach for efficient LLM compression, with potential applications in optimizing inference speed and deployment in resource-constrained environments.