Open-Source LLMs for Technical Q&A
Lessons from StackExchange: can open-source LLMs — Solar, CodeLlama, Mistral, Qwen2, StarCoder2, LLaMA3 — reach expert-level technical answers without a proprietary model? RAG improves every model we tested, and Solar-10.7B with RAG and fine-tuning comes closest to accepted expert answers — a promising, cost-effective direction.
1University of California, Davis
Numbers
At a Glance
Abstract
Overview
In the rapidly evolving domain of software engineering, large language models are increasingly leveraged to automate developer support. Open-source LLMs have grown competitive with proprietary models such as GPT-4 and Claude-3, without the associated financial and accessibility constraints.
This study investigates whether state-of-the-art open-source LLMs — Solar-10.7B, CodeLlama-7B, Mistral-7B, Qwen2-7B, StarCoder2-7B, and LLaMA3-8B — can generate responses to technical queries that align with those crafted by human experts.
Leveraging retrieval-augmented generation (RAG) and targeted fine-tuning, we evaluate these models across critical performance dimensions, such as semantic alignment and contextual fluency. Our results show that Solar-10.7B, particularly when paired with RAG and fine-tuning, most closely replicates expert-level responses — offering a scalable and cost-effective alternative to commercial models.
This vision paper highlights the potential of open-source LLMs to enable robust and accessible AI-powered developer assistance in software engineering.
Data. From the public StackExchange Data Dump (May 2014 – December 2023, 73,560 posts), we randomly sampled 41,000 posts and split them into a 1,000-question test set, a 20,000-pair RAG database, and a 20,000-pair fine-tuning set. Only posts with an accepted answer were kept, and the accepted answer serves as the expert reference. All runs use temperature 0 and the same vanilla prompt.
The study asks three questions: RQ1 — can local LLMs answer technical questions in a base setting? RQ2 — how much does RAG improve them? RQ3 — does fine-tuning within the RAG framework help further?
Framework
Six Models Under the Microscope
Solar-10.7B
Best model in the base setting and with RAG, and the largest RAG gain (cosine 0.56 → 0.64). The only model fine-tuned.
CodeLlama-7B
Meta’s code-specialist model. Close behind Solar in the base setting (0.54 cosine, 0.54 BERTScore).
Mistral-7B
Weakest in the base setting (0.32 cosine, 0.41 BERTScore), but RAG lifted its cosine to 0.57, a large relative gain.
Qwen2-7B
Alibaba’s multilingual model. Competitive in the base setting; 0.60 cosine with RAG.
StarCoder2-7B
Code-focused model from Hugging Face and ServiceNow. Competitive in the base setting; 0.58 cosine with RAG.
LLaMA3-8B
Meta’s open 8B model. Close behind Solar in the base setting (0.55 cosine); 0.61 cosine with RAG.
Pipeline
The RAG Pipeline
QA pairs from the RAG database are embedded with all-MiniLM-L6-v2, indexed with LangChain, and stored in FAISS. Each developer question is embedded the same way and the top 5 matches are retrieved. An embedding-distance filter (threshold 0.5) drops retrieved pairs that are not close enough to the question, so only relevant context reaches the LLM. The filtered context plus the question go to the model, and its answer is compared against the accepted expert reply.
Fine-tuning
LoRA Fine-Tuning Setup
Because of resource limits, only the best-performing model from the base and RAG settings, Solar-10.7B, was fine-tuned. It was LoRA-fine-tuned on the 20,000-pair training set (q_proj layer, 4-bit quantization with bfloat16, early stopping), using the 1,000-question test set for validation, then evaluated inside the same RAG pipeline.
Results
Who Comes Closest to Human Experts?
| Model | Base | + RAG | + RAG + Fine-tuning | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Cosine | BERT | METEOR | Cosine | BERT | METEOR | Cosine | BERT | METEOR | |
| CodeLlama-7B | 0.54 | 0.54 | 0.22 | 0.54 | 0.55 | 0.25 | not fine-tuned | ||
| Solar-10.7B | 0.56 | 0.54 | 0.23 | 0.64 | 0.59 | 0.31 | 0.65 | 0.63 | 0.61 |
| Mistral-7B | 0.32 | 0.41 | 0.11 | 0.57 | 0.55 | 0.26 | not fine-tuned | ||
| StarCoder2-7B | 0.50 | 0.52 | 0.20 | 0.58 | 0.56 | 0.28 | not fine-tuned | ||
| Qwen2-7B | 0.53 | 0.53 | 0.22 | 0.60 | 0.57 | 0.29 | not fine-tuned | ||
| LLaMA3-8B | 0.55 | 0.53 | 0.23 | 0.61 | 0.58 | 0.30 | not fine-tuned | ||
Table 1 from the paper. Bold marks the best score in each base and RAG column. Only Solar-10.7B was fine-tuned.
- RQ1 (base). Solar-10.7B leads, with LLaMA-3 and CodeLlama close behind; Qwen2 and StarCoder2 are competitive; Mistral underperforms. METEOR is low across the board (0.11–0.23).
- RQ2 (RAG). Every model improves. Solar gains the most in absolute terms (cosine 0.56 → 0.64), and Mistral, the weakest base model, jumps from 0.32 to 0.57.
- RQ3 (fine-tuning). Fine-tuning Solar within RAG mainly raises METEOR (0.31 → 0.61) and BERTScore (0.59 → 0.63); cosine similarity barely moves (0.64 → 0.65).
Takeaway. Solar-10.7B, especially with RAG and fine-tuning, aligned most closely with accepted expert answers. As a vision paper, this suggests open-source LLMs optimized with RAG and fine-tuning are a promising, cost-effective alternative for SE tasks.
Limitations. The datasets are relatively small, only six models in the 7–11B range were evaluated, fine-tuning was applied to a single model, and all experiments used a basic prompt without chain-of-thought or few-shot techniques.
Origin
From Class Project to SEET ’25
The paper grew out of ECS 260: Software Engineering at UC Davis (Winter 2024, taught by Prof. Vladimir Filkov). The initial team was Muhammad Hassnain, Zeerak Babar, Nafiz Imtiaz Khan, Prem Gorde and Disha. After the final submission, the project was extended with RAG and fine-tuning; the extended version was accepted to SEET 2025 (Software Engineering for Emerging Technologies) and published in Springer’s CCIS, vol. 2725.
Cite
BibTeX
@inproceedings{Babar2026OpenSourceLLMs,
author = {Babar, Zeerak and Khan, Nafiz Imtiaz and
Hassnain, Muhammad and Filkov, Vladimir},
title = {Open-Source LLMs for Technical Q\&A:
Lessons from StackExchange},
booktitle = {SEET---Software Engineering for Emerging Technologies},
year = {2025},
pages = {615--626},
publisher = {Springer, Cham},
series = {Communications in Computer and Information Science},
volume = {2725},
editor = {Hussain, Shahid and Khan, Arif Ali and
Abdul Basit Ur Rahim, Muhammad and
Khan, Saif Ur Rehman},
isbn = {978-3-032-08977-9},
doi = {10.1007/978-3-032-08977-9_41},
url = {https://doi.org/10.1007/978-3-032-08977-9_41},
}