LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Review

This dataset is associated with the paper titled "LLM-Based Agents for Code Generation: A Multi-Vocal Literature Review." The goal of this study is to systematically investigate the use of LLM-based agentic systems in code generation by analyzing both peer-reviewed and grey literature. The study aims to identify commonly used models, benchmarks, architectural designs, challenges, proposed solutions, and future research directions in this domain.

The data is stored in a Microsoft Excel file consisting of several worksheets. Below is a simplified description of each worksheet.

Summary

This sheet provides an overall summary of both peer-reviewed and grey literature. It includes the initial results obtained from title and keyword searches, abstract screening, and full-paper review.

Research Questions

This sheet contains the research questions of the study along with the complete search strings used during the literature search process.

Grey Literature Data

This sheet contains all grey literature data retrieved from three sources: Google, Yahoo, and Bing. The initial search yielded 482 studies from these sources.

Grey Literature Process

This sheet documents the selection process for grey literature. After removing 84 duplicate studies from the initial 482 results, 398 studies remained. The sheet presents the filtering stages, including title and keyword screening and full-text review.



Grey color represents studies selected for final full-paper reading.

Green color represents studies selected after title and keyword screening but excluded during full-paper review.The sheet also includes quality assessment criteria, through which two papers were excluded.Additionally, three papers identified through the snowballing process are included at the end of this sheet.


Grey Literature Data Extraction

This sheet contains the detailed data extraction for the final 40 selected grey literature studies.

Peer-Reviewed Studies Process

This sheet describes the selection process for peer-reviewed papers collected from five digital libraries and through backward and forward snowballing.



Grey-colored papers indicate inclusion for full-paper screening.

Green-highlighted papers indicate inclusion after abstract screening but exclusion after full-paper review.

Blue represents the paper included after titel and keywords reading.


For search engines such as Google Scholar (which initially returned 918 results), only studies selected after title screening were stored, as Google Scholar does not allow full export of search results. Similarly, only relevant studies were added during backward and forward snowballing.

Peer-Reviewed Data Extraction

This sheet contains detailed data extraction for the finalized 74 peer-reviewed studies.

Demographic Data Analysis

This sheet provides demographic information about the selected studies, including authors’ affiliations, publication years, types of studies, and publication venues.

Reasons Analysis

This sheet presents the analysis results for RQ2, identifying the reasons for adopting agent-based systems in code generation.

Model Analysis

This sheet lists and analyzes all models utilized by the selected studies to implement agentic systems.

Benchmarks Analysis

This sheet provides the list and analysis of benchmarks used by the selected studies to evaluate agent-based code generation systems.

Challenges and Solutions Analysis

This sheet presents the identified challenges in agent-based code generation systems along with the proposed solutions reported in the literature.

Future Work Analysis

This sheet summarizes the future research directions identified in the selected studies.