Fine-Tuning Qwen 2.5 7B for Python Code Generation using LoRA

Abstract

Motivation & Problem Statement

The advent of large language models has revolutionized code generation, promising to enhance developer productivity dramatically. However, fine-tuning large pre-trained LLMs for specific tasks often demands substantial computational resources.

The Challenge

• Resource Constraints: Full fine-tuning of 7B+ parameter models requires significant GPU memory and compute time
• Task Specialization: General-purpose LLMs need adaptation for domain-specific code generation
• Quality vs. Efficiency: Balancing model performance with computational efficiency
• Evaluation Complexity: Code correctness requires more than lexical similarity metrics

Our Solution: LoRA

Parameter-Efficient Fine-Tuning (PEFT) methods, specifically LoRA (Low-Rank Adaptation), offer a viable solution by:

• Injecting trainable rank decomposition matrices into transformer architecture
• Significantly reducing trainable parameters (typically 0.1-1% of original model)
• Maintaining or improving performance compared to full fine-tuning
• Enabling fine-tuning on consumer-grade hardware

Key Contributions

Methodology

Model Architecture

LoRA Hyperparameters

Dataset

flytech/python-codes-25k: A curated dataset of Python code snippets enabling the model to learn syntax, structure, and common programming patterns specific to Python.

Evaluation Metrics

Experimental Results Comprehensive

Initial Hyperparameter Search (1 Epoch, 625 Steps)

Systematic exploration of 30+ LoRA configurations revealed important trade-offs between evaluation metrics and functional correctness:

Key Findings from Initial Search

Extended Training Results (Up to 3175 Steps)

Selected configurations were trained for multiple epochs to analyze learning dynamics and identify optimal stopping points:

Critical Observations

Generated Code Samples Functional

The following samples demonstrate the model's capability to generate syntactically correct and functionally accurate Python code:

Sample 1: Factorial Function

def factorial(n):
    if n == 0 or n == 1:
        return 1
    else:
        return (n * factorial(n-1))

Analysis: Correctly implements recursive factorial with proper base cases for 0 and 1.

Sample 2: Reverse String Function

def reverse_string(string):
    reversed_str = ""
    for char in string:
        reversed_str = char + reversed_str
    return reversed_str

Analysis: Iterative approach building reversed string character by character, demonstrating clear string manipulation logic.

Sample 3: Prime Number Check

def is_prime(n):
    if n <= 1:
        return False
    for i in range(2, n):
        if n % i == 0:
            return False
    return True

Analysis: Standard primality test algorithm with correct edge case handling (n ≤ 1).

Sample 4: Binary Search

def binary_search(array, target):
    left = 0
    right = len(array) - 1
    
    while left <= right:
        middle = (left + right) // 2
        if array[middle] == target:
            return middle
        elif array[middle] < target:
            left = middle + 1
        else:
            right = middle - 1
    
    return False

Analysis: Correctly implements iterative binary search with proper pointer management and termination conditions.

Sample 5: Longest Common Substring (Advanced)

def longest_common_substring(str1, str2):
    # Create 2D matrix for dynamic programming
    matrix = [[0]*(len(str2) + 1) for _ in range(len(str1) + 1)]
    max_length = 0
    end_idx_str1 = 0
    
    # Fill matrix using dynamic programming
    for i in range(1, len(str1) + 1):
        for j in range(1, len(str2) + 1):
            if str1[i-1] == str2[j-1]:
                matrix[i][j] = matrix[i-1][j-1] + 1
                if matrix[i][j] > max_length:
                    max_length = matrix[i][j]
                    end_idx_str1 = i
            else:
                matrix[i][j] = 0
    
    # Extract longest common substring
    if max_length == 0:
        return ""
    else:
        start_idx_str1 = end_idx_str1 - max_length
        return str1[start_idx_str1:end_idx_str1]

Analysis: Sophisticated implementation using dynamic programming with proper matrix initialization, filling logic, and substring extraction. Demonstrates model's capability with complex algorithms.

Discussion & Analysis

Key Insights

Detailed Performance Analysis

Configuration r=1, α=2: Despite highest eval loss (0.5500), achieved exceptional 93.34% execution rate in initial sweep and maintained 86.67% at extended training. Suggests minimal parameter adaptation sufficient for functional code generation.

Configuration r=8, α=16: Emerged as optimal balance—90.00% execution rate with reasonable eval loss (0.3946) at step 1270. This configuration provides sufficient expressiveness without overfitting.

Configuration r=32, α=64: Achieved lowest eval loss (0.1895) but execution rate dropped to 56.67%, indicating potential overfitting to training data patterns rather than learning generalizable code structures.

Implications for Code Generation

• Execution Rate as Primary Metric: For code generation tasks, functional correctness (execution success) should be weighted more heavily than lexical similarity metrics
• Early Stopping Strategy: Monitor execution rate on validation set; stop when it plateaus or begins declining, even if loss continues improving
• Conservative Hyperparameters: Smaller LoRA ranks may be preferable for code generation compared to text generation tasks
• Dataset Quality: Model learns patterns from training data—high-quality, diverse code examples crucial for generalization

Implementation Details

Training Configuration

Model: Qwen 2.5 7B
Dataset: flytech/python-codes-25k
Training Steps: 625 (1 epoch) to 3175 (5 epochs)
Batch Size: Adaptive based on GPU memory
Learning Rate: Adaptive with LoRA α/r scaling
Optimizer: AdamW
Target Modules: Query, Key, Value projections in attention layers

LoRA Architecture

For attention weight matrix W ∈ R^(d×k):
  W' = W + BA
where:
  B ∈ R^(d×r) - Down-projection matrix (trainable)
  A ∈ R^(r×k) - Up-projection matrix (trainable)
  r << min(d,k) - Rank (hyperparameter)
  
Effective parameter reduction:
  Original: d × k parameters
  LoRA: (d + k) × r parameters
  Ratio: ~1-2% for r=8, d=4096, k=4096