SEET ’25 · Springer CCIS v2725 · DOI 10.1007/978-3-032-08977-9_41

Open-Source LLMs for Technical Q&A

Lessons from StackExchange: can open-source LLMs — Solar, CodeLlama, Mistral, Qwen2, StarCoder2, LLaMA3 — reach expert-level technical answers without a proprietary model? RAG improves every model we tested, and Solar-10.7B with RAG and fine-tuning comes closest to accepted expert answers — a promising, cost-effective direction.

Zeerak Babar1 · Nafiz Imtiaz Khan1 · Muhammad Hassnain1 · Vladimir Filkov1

1University of California, Davis

Numbers

At a Glance

6
Open-Source LLMs
Solar · CodeLlama · Mistral · Qwen2 · StarCoder2 · LLaMA3
RAG
Retrieval Pipeline
20k StackExchange QA pairs in a FAISS store
LoRA
Fine-Tuning
applied to the best model only (Solar-10.7B)
3
Metrics
Cosine Similarity · BERTScore · METEOR
Solar-10.7B
Best Match
0.65 cosine · 0.63 BERT · 0.61 METEOR with RAG + FT
Local
Deployment
open models run locally via Ollama

Abstract

Overview

In the rapidly evolving domain of software engineering, large language models are increasingly leveraged to automate developer support. Open-source LLMs have grown competitive with proprietary models such as GPT-4 and Claude-3, without the associated financial and accessibility constraints.

This study investigates whether state-of-the-art open-source LLMs — Solar-10.7B, CodeLlama-7B, Mistral-7B, Qwen2-7B, StarCoder2-7B, and LLaMA3-8B — can generate responses to technical queries that align with those crafted by human experts.

Leveraging retrieval-augmented generation (RAG) and targeted fine-tuning, we evaluate these models across critical performance dimensions, such as semantic alignment and contextual fluency. Our results show that Solar-10.7B, particularly when paired with RAG and fine-tuning, most closely replicates expert-level responses — offering a scalable and cost-effective alternative to commercial models.

This vision paper highlights the potential of open-source LLMs to enable robust and accessible AI-powered developer assistance in software engineering.

Data. From the public StackExchange Data Dump (May 2014 – December 2023, 73,560 posts), we randomly sampled 41,000 posts and split them into a 1,000-question test set, a 20,000-pair RAG database, and a 20,000-pair fine-tuning set. Only posts with an accepted answer were kept, and the accepted answer serves as the expert reference. All runs use temperature 0 and the same vanilla prompt.

The study asks three questions: RQ1 — can local LLMs answer technical questions in a base setting? RQ2 — how much does RAG improve them? RQ3 — does fine-tuning within the RAG framework help further?

Framework

Six Models Under the Microscope

1

Solar-10.7B

Best model in the base setting and with RAG, and the largest RAG gain (cosine 0.56 → 0.64). The only model fine-tuned.

2

CodeLlama-7B

Meta’s code-specialist model. Close behind Solar in the base setting (0.54 cosine, 0.54 BERTScore).

3

Mistral-7B

Weakest in the base setting (0.32 cosine, 0.41 BERTScore), but RAG lifted its cosine to 0.57, a large relative gain.

4

Qwen2-7B

Alibaba’s multilingual model. Competitive in the base setting; 0.60 cosine with RAG.

5

StarCoder2-7B

Code-focused model from Hugging Face and ServiceNow. Competitive in the base setting; 0.58 cosine with RAG.

6

LLaMA3-8B

Meta’s open 8B model. Close behind Solar in the base setting (0.55 cosine); 0.61 cosine with RAG.

Pipeline

The RAG Pipeline

RAG pipeline architecture: a developer question is embedded via all-MiniLM-L6-v2 and semantic-searched against a FAISS vector store of QA-pair embeddings; the top-K results are passed through an embedding-distance filter, then the filtered context plus the question are sent to the LLM, which produces the answer.
Figure 1 from the paper — architecture of the proposed RAG pipeline.

QA pairs from the RAG database are embedded with all-MiniLM-L6-v2, indexed with LangChain, and stored in FAISS. Each developer question is embedded the same way and the top 5 matches are retrieved. An embedding-distance filter (threshold 0.5) drops retrieved pairs that are not close enough to the question, so only relevant context reaches the LLM. The filtered context plus the question go to the model, and its answer is compared against the accepted expert reply.

Fine-tuning

LoRA Fine-Tuning Setup

Fine-tuning overview: a base LLM is fine-tuned using LoRA on a 20,000-QA-pair training set to produce a fine-tuned LLM, which is validated across each epoch on a 1,000-QA-pair test set. Hyper-parameters: batch size 8, Paged AdamW optimizer, learning rate 2.5e-5, 500 training steps, q_proj fine-tuned layer, early stopping.
Figure 2 from the paper — overview of the fine-tuning process and selected hyper-parameters.

Because of resource limits, only the best-performing model from the base and RAG settings, Solar-10.7B, was fine-tuned. It was LoRA-fine-tuned on the 20,000-pair training set (q_proj layer, 4-bit quantization with bfloat16, early stopping), using the 1,000-question test set for validation, then evaluated inside the same RAG pipeline.

Results

Who Comes Closest to Human Experts?

Model Base + RAG + RAG + Fine-tuning
Cosine BERT METEOR Cosine BERT METEOR Cosine BERT METEOR
CodeLlama-7B 0.54 0.54 0.22 0.54 0.55 0.25 not fine-tuned
Solar-10.7B 0.56 0.54 0.23 0.64 0.59 0.31 0.65 0.63 0.61
Mistral-7B 0.32 0.41 0.11 0.57 0.55 0.26 not fine-tuned
StarCoder2-7B 0.50 0.52 0.20 0.58 0.56 0.28 not fine-tuned
Qwen2-7B 0.53 0.53 0.22 0.60 0.57 0.29 not fine-tuned
LLaMA3-8B 0.55 0.53 0.23 0.61 0.58 0.30 not fine-tuned

Table 1 from the paper. Bold marks the best score in each base and RAG column. Only Solar-10.7B was fine-tuned.

Takeaway. Solar-10.7B, especially with RAG and fine-tuning, aligned most closely with accepted expert answers. As a vision paper, this suggests open-source LLMs optimized with RAG and fine-tuning are a promising, cost-effective alternative for SE tasks.

Limitations. The datasets are relatively small, only six models in the 7–11B range were evaluated, fine-tuning was applied to a single model, and all experiments used a basic prompt without chain-of-thought or few-shot techniques.

Origin

From Class Project to SEET ’25

The paper grew out of ECS 260: Software Engineering at UC Davis (Winter 2024, taught by Prof. Vladimir Filkov). The initial team was Muhammad Hassnain, Zeerak Babar, Nafiz Imtiaz Khan, Prem Gorde and Disha. After the final submission, the project was extended with RAG and fine-tuning; the extended version was accepted to SEET 2025 (Software Engineering for Emerging Technologies) and published in Springer’s CCIS, vol. 2725.

Cite

BibTeX

@inproceedings{Babar2026OpenSourceLLMs,
  author    = {Babar, Zeerak and Khan, Nafiz Imtiaz and
               Hassnain, Muhammad and Filkov, Vladimir},
  title     = {Open-Source LLMs for Technical Q\&A:
               Lessons from StackExchange},
  booktitle = {SEET---Software Engineering for Emerging Technologies},
  year      = {2025},
  pages     = {615--626},
  publisher = {Springer, Cham},
  series    = {Communications in Computer and Information Science},
  volume    = {2725},
  editor    = {Hussain, Shahid and Khan, Arif Ali and
               Abdul Basit Ur Rahim, Muhammad and
               Khan, Saif Ur Rehman},
  isbn      = {978-3-032-08977-9},
  doi       = {10.1007/978-3-032-08977-9_41},
  url       = {https://doi.org/10.1007/978-3-032-08977-9_41},
}