# Open-Source LLMs for Technical Q&A: Lessons from StackExchange > Open-Source LLMs for Technical Q&A: Lessons from StackExchange (SEET 2025). Babar, Khan, Hassnain, Filkov — UC Davis. - Authors: Zeerak Babar, Nafiz Imtiaz Khan, Muhammad Hassnain, Vladimir Filkov (University of California, Davis) - Venue: SEET: Software Engineering for Emerging Technologies (Springer CCIS vol. 2725), 2025 - DOI: https://doi.org/10.1007/978-3-032-08977-9_41 - Paper PDF: https://muhammad-hassnain.github.io/open-source-llms.github.io/assets/pdf/llm.pdf - Code: https://github.com/zeerakb1/llm-developers-evaluation - Project page: https://muhammad-hassnain.github.io/open-source-llms.github.io/ ## Page content Open-Source LLMs for Technical Q&A SEET ’25 · Springer CCIS v2725 · DOI 10.1007/978-3-032-08977-9_41 Open-Source LLMs for Technical Q&A Lessons from StackExchange: can open-source LLMs — Solar, CodeLlama, Mistral, Qwen2, StarCoder2, LLaMA3 — reach expert-level technical answers without a proprietary model? RAG improves every model we tested, and Solar-10.7B with RAG and fine-tuning comes closest to accepted expert answers — a promising, cost-effective direction. Zeerak Babar1 · Nafiz Imtiaz Khan1 · Muhammad Hassnain1 · Vladimir Filkov1 1University of California, Davis Read Paper Springer Code Slides Numbers At a Glance 6 Open-Source LLMs Solar · CodeLlama · Mistral · Qwen2 · StarCoder2 · LLaMA3 RAG Retrieval Pipeline 20k StackExchange QA pairs in a FAISS store LoRA Fine-Tuning applied to the best model only (Solar-10.7B) 3 Metrics Cosine Similarity · BERTScore · METEOR Solar-10.7B Best Match 0.65 cosine · 0.63 BERT · 0.61 METEOR with RAG + FT Local Deployment open models run locally via Ollama Abstract Overview In the rapidly evolving domain of software engineering, large language models are increasingly leveraged to automate developer support. Open-source LLMs have grown competitive with proprietary models such as GPT-4 and Claude-3, without the associated financial and accessibility constraints. This study investigates whether state-of-the-art open-source LLMs — Solar-10.7B, CodeLlama-7B, Mistral-7B, Qwen2-7B, StarCoder2-7B, and LLaMA3-8B — can generate responses to technical queries that align with those crafted by human experts. Leveraging retrieval-augmented generation (RAG) and targeted fine-tuning, we evaluate these models across critical performance dimensions, such as semantic alignment and contextual fluency. Our results show that Solar-10.7B, particularly when paired with RAG and fine-tuning, most closely replicates expert-level responses — offering a scalable and cost-effective alternative to commercial models. This vision paper highlights the potential of open-source LLMs to enable robust and accessible AI-powered developer assistance in software engineering. Data. From the public StackExchange Data Dump (May 2014 – December 2023, 73,560 posts), we randomly sampled 41,000 posts and split them into a 1,000-question test set, a 20,000-pair RAG database, and a 20,000-pair fine-tuning set. Only posts with an accepted answer were kept, and the accepted answer serves as the expert reference. All runs use temperature 0 and the same vanilla prompt. The study asks three questions: RQ1 — can local LLMs answer technical questions in a base setting? RQ2 — how much does RAG improve them? RQ3 — does fine-tuning within the RAG framework help further? Framework Six Models Under the Microscope 1 Solar-10.7B Best model in the base setting and with RAG, and the largest RAG gain (cosine 0.56 → 0.64). The only model fine-tuned. 2 CodeLlama-7B Meta’s code-specialist model. Close behind Solar in the base setting (0.54 cosine, 0.54 BERTScore). 3 Mistral-7B Weakest in the base setting (0.32 cosine, 0.41 BERTScore), but RAG lifted its cosine to 0.57, a large relative gain. 4 Qwen2-7B Alibaba’s multilingual model. Competitive in the base setting; 0.60 cosine with RAG. 5 StarCoder2-7B Code-focused model from Hugging Face and ServiceNow. Competitive in the base setting; 0.58 cosine with RAG. 6 LLaMA3-8B Meta’s open 8B model. Close behind Solar in the base setting (0.55 cosine); 0.61 cosine with RAG. Pipeline The RAG Pipeline Figure 1 from the paper — architecture of the proposed RAG pipeline. QA pairs from the RAG database are embedded with all-MiniLM-L6-v2, indexed with LangChain, and stored in FAISS. Each developer question is embedded the same way and the top 5 matches are retrieved. An embedding-distance filter (threshold 0.5) drops retrieved pairs that are not close enough to the question, so only relevant context reaches the LLM. The filtered context plus the question go to the model, and its answer is compared against the accepted expert reply. Fine-tuning LoRA Fine-Tuning Setup Figure 2 from the paper — overview of the fine-tuning process and selected hyper-parameters. Because of resource limits, only the best-performing model from the base and RAG settings, Solar-10.7B, was fine-tuned. It was LoRA-fine-tuned on the 20,000-pair training set (q_proj layer, 4-bit quantization with bfloat16, early stopping), using the 1,000-question test set for validation, then evaluated inside the same RAG pipeline. Results Who Comes Closest to Human Experts? Model Base + RAG + RAG + Fine-tuning Cosine BERT METEOR Cosine BERT METEOR Cosine BERT METEOR CodeLlama-7B 0.54 0.54 0.22 0.54 0.55 0.25 not fine-tuned Solar-10.7B 0.56 0.54 0.23 0.64 0.59 0.31 0.65 0.63 0.61 Mistral-7B 0.32 0.41 0.11 0.57 0.55 0.26 not fine-tuned StarCoder2-7B 0.50 0.52 0.20 0.58 0.56 0.28 not fine-tuned Qwen2-7B 0.53 0.53 0.22 0.60 0.57 0.29 not fine-tuned LLaMA3-8B 0.55 0.53 0.23 0.61 0.58 0.30 not fine-tuned Table 1 from the paper. Bold marks the best score in each base and RAG column. Only Solar-10.7B was fine-tuned. RQ1 (base). Solar-10.7B leads, with LLaMA-3 and CodeLlama close behind; Qwen2 and StarCoder2 are competitive; Mistral underperforms. METEOR is low across the board (0.11–0.23). RQ2 (RAG). Every model improves. Solar gains the most in absolute terms (cosine 0.56 → 0.64), and Mistral, the weakest base model, jumps from 0.32 to 0.57. RQ3 (fine-tuning). Fine-tuning Solar within RAG mainly raises METEOR (0.31 → 0.61) and BERTScore (0.59 → 0.63); cosine similarity barely moves (0.64 → 0.65). Takeaway. Solar-10.7B, especially with RAG and fine-tuning, aligned most closely with accepted expert answers. As a vision paper, this suggests open-source LLMs optimized with RAG and fine-tuning are a promising, cost-effective alternative for SE tasks. Limitations. The datasets are relatively small, only six models in the 7–11B range were evaluated, fine-tuning was applied to a single model, and all experiments used a basic prompt without chain-of-thought or few-shot techniques. Origin From Class Project to SEET ’25 The paper grew out of ECS 260: Software Engineering at UC Davis (Winter 2024, taught by Prof. Vladimir Filkov). The initial team was Muhammad Hassnain, Zeerak Babar, Nafiz Imtiaz Khan, Prem Gorde and Disha. After the final submission, the project was extended with RAG and fine-tuning; the extended version was accepted to SEET 2025 (Software Engineering for Emerging Technologies) and published in Springer’s CCIS, vol. 2725. Cite BibTeX @inproceedings{Babar2026OpenSourceLLMs, author = {Babar, Zeerak and Khan, Nafiz Imtiaz and Hassnain, Muhammad and Filkov, Vladimir}, title = {Open-Source LLMs for Technical Q\&A: Lessons from StackExchange}, booktitle = {SEET---Software Engineering for Emerging Technologies}, year = {2025}, pages = {615--626}, publisher = {Springer, Cham}, series = {Communications in Computer and Information Science}, volume = {2725}, editor = {Hussain, Shahid and Khan, Arif Ali and Abdul Basit Ur Rahim, Muhammad and Khan, Saif Ur Rehman}, isbn = {978-3-032-08977-9}, doi = {10.1007/978-3-032-08977-9_41}, url = {https://doi.org/10.1007/978-3-032-08977-9_41}, } Open-Source LLMs for Technical Q&A · University of California, Davis · ← Muhammad Hassnain’s homepage