RAGScore

RAGScore is an open-source tool designed to generate QA datasets and robustly evaluate Retrieval-Augmented Generation (RAG) systems in just two commands. It enables users to quickly audit the performance of their RAG pipelines by auto-generating tailored QA pairs from documentation and evaluating endpoints for accuracy, completeness, relevance, and other diagnostic metrics. RAGScore supports multiple LLMs (OpenAI, Anthropic, Ollama, vLLM, and more) and provides both CLI and Python API interfaces, making it suitable for researchers, engineers, and anyone deploying RAG systems in production or private environments.