TY - GEN
T1 - Engineering RAG Systems for Real-World Applications
T2 - Euromicro Conference on Software Engineering and Advanced Applications
AU - Hasan, Md Toufique
AU - Waseem, Muhammad
AU - Kemell, Kai Kristian
AU - Khan, Ayman Asad
AU - Saari, Mika
AU - Abrahamsson, Pekka
N1 - Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.
PY - 2025
Y1 - 2025
N2 - Retrieval-Augmented Generation (RAG) systems are emerging as a key approach for grounding Large Language Models (LLMs) in external knowledge, addressing limitations in factual accuracy and contextual relevance. However, there is a lack of empirical studies that report on the development of RAG-based implementations grounded in real-world use cases, evaluated through general user involvement, and accompanied by systematic documentation of lessons learned. This paper presents five domain-specific RAG applications developed for real-world scenarios across governance, cybersecurity, agriculture, industrial research, and medical diagnostics. Each system incorporates multilingual OCR, semantic retrieval via vector embeddings, and domain-adapted LLMs, deployed through local servers or cloud APIs to meet distinct user needs. A web-based evaluation involving a total of 100 participants assessed the systems across six dimensions: (i) Ease of Use, (ii) Relevance, (iii) Transparency, (iv) Responsiveness, (v) Accuracy, and (vi) Likelihood of Recommendation. Based on user feedback and our development experience, we documented twelve key lessons learned, highlighting technical, operational, and ethical challenges affecting the reliability and usability of RAG systems in practice.
AB - Retrieval-Augmented Generation (RAG) systems are emerging as a key approach for grounding Large Language Models (LLMs) in external knowledge, addressing limitations in factual accuracy and contextual relevance. However, there is a lack of empirical studies that report on the development of RAG-based implementations grounded in real-world use cases, evaluated through general user involvement, and accompanied by systematic documentation of lessons learned. This paper presents five domain-specific RAG applications developed for real-world scenarios across governance, cybersecurity, agriculture, industrial research, and medical diagnostics. Each system incorporates multilingual OCR, semantic retrieval via vector embeddings, and domain-adapted LLMs, deployed through local servers or cloud APIs to meet distinct user needs. A web-based evaluation involving a total of 100 participants assessed the systems across six dimensions: (i) Ease of Use, (ii) Relevance, (iii) Transparency, (iv) Responsiveness, (v) Accuracy, and (vi) Likelihood of Recommendation. Based on user feedback and our development experience, we documented twelve key lessons learned, highlighting technical, operational, and ethical challenges affecting the reliability and usability of RAG systems in practice.
KW - AI System Lifecycle
KW - Empirical Software Engineering
KW - Generative AI
KW - Human Centred Evaluation
KW - LLMs
KW - RAG
KW - System Design
KW - System Implementation
U2 - 10.1007/978-3-032-04200-2_10
DO - 10.1007/978-3-032-04200-2_10
M3 - Conference contribution
AN - SCOPUS:105016567836
SN - 9783032041999
T3 - Lecture Notes in Computer Science
SP - 143
EP - 158
BT - Software Engineering and Advanced Applications - 51st Euromicro Conference, SEAA 2025, Proceedings
A2 - Taibi, Davide
A2 - Smite, Darja
PB - Springer
Y2 - 10 September 2025 through 12 September 2025
ER -