Why Adapt RAG for Agile? Challenges, Frameworks, and the Role of Evaluator Agent

Ayman Asad Khan; Toufique Hasan; Mika Saari; Kai Kristian Kemell; Jussi Rasku

doi:10.1007/978-3-032-05799-0_3

Why Adapt RAG for Agile? Challenges, Frameworks, and the Role of Evaluator Agent

Ayman Asad Khan^*
, Toufique Hasan
, Mika Saari
, Kai Kristian Kemell
, Jussi Rasku

^*Corresponding author for this work

Computing Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

3 Downloads (Pure)

Abstract

This paper proposes the adaptation of Retrieval Augmented Generation (RAG) based systems into Agile Software Development workflows, addressing the need for specialized evaluation methods that align with Agile’s iterative, feedback driven nature. Through a systematic mapping study of 20 papers, we identify existing evaluation frameworks for RAG systems and explore how they can be applied within Agile settings. The research highlights the role of Evaluator-Agents in aligning the responsiveness of RAG systems, with a focus on Multi-agent AI systems that are more efficient at handling complex, distributed tasks than Single-agent systems. The findings contribute to identify the potential of RAG and its evaluation in facilitating real time feedback and continuous improvement for Agile development.

Original language	English
Title of host publication	Agile Processes in Software Engineering and Extreme Programming – Workshops - XP 2025 Workshops, Revised Selected Papers
Editors	Sibylle Peter, Martin Kropp, Torgeir Dingsøyr, Clare Dillon, Philipp Diebold, Deepti Jain, Maria Ilaria Lunesu, Andrea Pinna
Publisher	Springer
Pages	22-31
Number of pages	10
ISBN (Print)	978-3-032-05798-3
DOIs	https://doi.org/10.1007/978-3-032-05799-0_3
Publication status	Published - 2025
Publication type	A4 Article in conference proceedings
Event	International Conference on Agile Software Development - Brugg-Windisch, Switzerland Duration: 2 Jun 2025 → 5 Jun 2025

Publication series

Name	Lecture Notes in Business Information Processing
Volume	561 LNBIP
ISSN (Print)	1865-1348
ISSN (Electronic)	1865-1356

Conference

Conference	International Conference on Agile Software Development
Country/Territory	Switzerland
City	Brugg-Windisch
Period	2/06/25 → 5/06/25

Keywords

Agile
AI
Evaluation
Large Language Models (LLMs)
Mapping-Study
Multi-agent
Retrieval-Augmented Generation (RAG)

Publication forum classification

Publication forum level 1

ASJC Scopus subject areas

Management Information Systems
Control and Systems Engineering
Business and International Management
Information Systems
Modelling and Simulation
Information Systems and Management

Access to Document

10.1007/978-3-032-05799-0_3Licence: CC BY

Why Adapt RAG for AgileFinal published version, 707 KBLicence: CC BY

https://urn.fi/URN:NBN:fi:tuni-202601271950Licence: CC BY

Cite this

Khan, A. A., Hasan, T., Saari, M., Kemell, K. K., & Rasku, J. (2025). Why Adapt RAG for Agile? Challenges, Frameworks, and the Role of Evaluator Agent. In S. Peter, M. Kropp, T. Dingsøyr, C. Dillon, P. Diebold, D. Jain, M. I. Lunesu, & A. Pinna (Eds.), Agile Processes in Software Engineering and Extreme Programming – Workshops - XP 2025 Workshops, Revised Selected Papers (pp. 22-31). (Lecture Notes in Business Information Processing; Vol. 561 LNBIP). Springer. https://doi.org/10.1007/978-3-032-05799-0_3

Khan, Ayman Asad ; Hasan, Toufique ; Saari, Mika et al. / Why Adapt RAG for Agile? Challenges, Frameworks, and the Role of Evaluator Agent. Agile Processes in Software Engineering and Extreme Programming – Workshops - XP 2025 Workshops, Revised Selected Papers. editor / Sibylle Peter ; Martin Kropp ; Torgeir Dingsøyr ; Clare Dillon ; Philipp Diebold ; Deepti Jain ; Maria Ilaria Lunesu ; Andrea Pinna. Springer, 2025. pp. 22-31 (Lecture Notes in Business Information Processing).

@inproceedings{2210189b22c94b0a85e7f2163adc34f2,

title = "Why Adapt RAG for Agile? Challenges, Frameworks, and the Role of Evaluator Agent",

abstract = "This paper proposes the adaptation of Retrieval Augmented Generation (RAG) based systems into Agile Software Development workflows, addressing the need for specialized evaluation methods that align with Agile{\textquoteright}s iterative, feedback driven nature. Through a systematic mapping study of 20 papers, we identify existing evaluation frameworks for RAG systems and explore how they can be applied within Agile settings. The research highlights the role of Evaluator-Agents in aligning the responsiveness of RAG systems, with a focus on Multi-agent AI systems that are more efficient at handling complex, distributed tasks than Single-agent systems. The findings contribute to identify the potential of RAG and its evaluation in facilitating real time feedback and continuous improvement for Agile development.",

keywords = "Agile, AI, Evaluation, Large Language Models (LLMs), Mapping-Study, Multi-agent, Retrieval-Augmented Generation (RAG)",

author = "Khan, \{Ayman Asad\} and Toufique Hasan and Mika Saari and Kemell, \{Kai Kristian\} and Jussi Rasku",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2026.; International Conference on Agile Software Development ; Conference date: 02-06-2025 Through 05-06-2025",

year = "2025",

doi = "10.1007/978-3-032-05799-0\_3",

language = "English",

isbn = "978-3-032-05798-3",

series = "Lecture Notes in Business Information Processing",

publisher = "Springer",

pages = "22--31",

editor = "Sibylle Peter and Martin Kropp and Torgeir Dings{\o}yr and Clare Dillon and Philipp Diebold and Deepti Jain and Lunesu, \{Maria Ilaria\} and Andrea Pinna",

booktitle = "Agile Processes in Software Engineering and Extreme Programming {\textendash} Workshops - XP 2025 Workshops, Revised Selected Papers",

}

Khan, AA , Hasan, T , Saari, M , Kemell, KK & Rasku, J 2025, Why Adapt RAG for Agile? Challenges, Frameworks, and the Role of Evaluator Agent. in S Peter, M Kropp, T Dingsøyr, C Dillon, P Diebold, D Jain, MI Lunesu & A Pinna (eds), Agile Processes in Software Engineering and Extreme Programming – Workshops - XP 2025 Workshops, Revised Selected Papers. Lecture Notes in Business Information Processing, vol. 561 LNBIP, Springer, pp. 22-31, International Conference on Agile Software Development, Brugg-Windisch, Switzerland, 2/06/25. https://doi.org/10.1007/978-3-032-05799-0_3

Why Adapt RAG for Agile? Challenges, Frameworks, and the Role of Evaluator Agent. / Khan, Ayman Asad ; Hasan, Toufique ; Saari, Mika et al.
Agile Processes in Software Engineering and Extreme Programming – Workshops - XP 2025 Workshops, Revised Selected Papers. ed. / Sibylle Peter; Martin Kropp; Torgeir Dingsøyr; Clare Dillon; Philipp Diebold; Deepti Jain; Maria Ilaria Lunesu; Andrea Pinna. Springer, 2025. p. 22-31 (Lecture Notes in Business Information Processing; Vol. 561 LNBIP).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Why Adapt RAG for Agile? Challenges, Frameworks, and the Role of Evaluator Agent

AU - Khan, Ayman Asad

AU - Hasan, Toufique

AU - Saari, Mika

AU - Kemell, Kai Kristian

AU - Rasku, Jussi

N1 - Publisher Copyright: © The Author(s) 2026.

PY - 2025

Y1 - 2025

N2 - This paper proposes the adaptation of Retrieval Augmented Generation (RAG) based systems into Agile Software Development workflows, addressing the need for specialized evaluation methods that align with Agile’s iterative, feedback driven nature. Through a systematic mapping study of 20 papers, we identify existing evaluation frameworks for RAG systems and explore how they can be applied within Agile settings. The research highlights the role of Evaluator-Agents in aligning the responsiveness of RAG systems, with a focus on Multi-agent AI systems that are more efficient at handling complex, distributed tasks than Single-agent systems. The findings contribute to identify the potential of RAG and its evaluation in facilitating real time feedback and continuous improvement for Agile development.

AB - This paper proposes the adaptation of Retrieval Augmented Generation (RAG) based systems into Agile Software Development workflows, addressing the need for specialized evaluation methods that align with Agile’s iterative, feedback driven nature. Through a systematic mapping study of 20 papers, we identify existing evaluation frameworks for RAG systems and explore how they can be applied within Agile settings. The research highlights the role of Evaluator-Agents in aligning the responsiveness of RAG systems, with a focus on Multi-agent AI systems that are more efficient at handling complex, distributed tasks than Single-agent systems. The findings contribute to identify the potential of RAG and its evaluation in facilitating real time feedback and continuous improvement for Agile development.

KW - Agile

KW - AI

KW - Evaluation

KW - Large Language Models (LLMs)

KW - Mapping-Study

KW - Multi-agent

KW - Retrieval-Augmented Generation (RAG)

UR - https://www.scopus.com/pages/publications/105021828134

U2 - 10.1007/978-3-032-05799-0_3

DO - 10.1007/978-3-032-05799-0_3

M3 - Conference contribution

AN - SCOPUS:105021828134

SN - 978-3-032-05798-3

T3 - Lecture Notes in Business Information Processing

SP - 22

EP - 31

BT - Agile Processes in Software Engineering and Extreme Programming – Workshops - XP 2025 Workshops, Revised Selected Papers

A2 - Peter, Sibylle

A2 - Kropp, Martin

A2 - Dingsøyr, Torgeir

A2 - Dillon, Clare

A2 - Diebold, Philipp

A2 - Jain, Deepti

A2 - Lunesu, Maria Ilaria

A2 - Pinna, Andrea

PB - Springer

T2 - International Conference on Agile Software Development

Y2 - 2 June 2025 through 5 June 2025

ER -

Khan AA , Hasan T , Saari M , Kemell KK , Rasku J. Why Adapt RAG for Agile? Challenges, Frameworks, and the Role of Evaluator Agent. In Peter S, Kropp M, Dingsøyr T, Dillon C, Diebold P, Jain D, Lunesu MI, Pinna A, editors, Agile Processes in Software Engineering and Extreme Programming – Workshops - XP 2025 Workshops, Revised Selected Papers. Springer. 2025. p. 22-31. (Lecture Notes in Business Information Processing). doi: 10.1007/978-3-032-05799-0_3