AI on Trial: LLM-as-a-Judge for Private and Reliable Clinical Decision-Making

Artificial Intelligence (AI) is changing the way healthcare works by helping doctors make better decisions. Large Language Models (LLMs) play a key role in this shift, not only by answering medical questions, but also by actively collecting critical information through follow-up questions. However, effectively applying LLMs to complex Clinical Decision-Making (CDM) tasks remains a challenge. In this paper, we propose AI on Trial– an enhanced framework built on top of MDAgents. It assigns LLMs to work alone or in teams based on the complexity of medical queries. To improve the reliability of responses, we extend the original MDAgent system by incorporating LLM-as-a-Judge, a clinically informed evaluator that scores responses based on quality, completeness, safety and clarity, while also providing the reasoning behind its assessment. This judge also explains its reasoning using Chain-of-Thoughts (CoT’s). We also add a second judge that acts as a team of expert reviewers. It uses four LLM agents to carefully check the response and give a final score and confidence level. This two-step review process makes the system more accurate and trustworthy. Our results show that integrating the multi-agent evaluation with MDAgents improves performance, making it a strong tool for AI-based clinical support.