RoPoLL: Robust Panel of LLM Judges

The evaluation of large language models (LLMs) has become a critical challenge for companies seeking to deploy artificial intelligence reliably. Traditionally, a single LLM judge has been used to assess responses, but this approach suffers from biases such as sycophancy, mode collapse, or safety refusals. To overcome these limitations, RoPoLL (Robust Panel of LLM Judges) emerges, a methodology that replaces the aggregation of scores from a panel of evaluators with a robust estimator: the geometric median. This estimator, without the need for hyperparameter tuning, offers an optimal breakdown point of 50%, meaning that even if nearly half of the judges fail in a biased manner, the overall evaluation remains accurate. In experiments with 13 open-source models (ranging from 4B to 675B parameters) and multiple corruption regimes, RoPoLL outperforms the classic PoLL by more than 19% under biased attacks and by orders of magnitude against heavy adversaries. A committee of just three 38B judges manages to outperform a massive 675B model on the HelpSteer-2 benchmark under 30% bimodal-random corruption, representing an 18-fold advantage in parameters with better accuracy. This statistical advance has a direct implication for the business world: having robust evaluation systems allows organizations to trust their AI agents and natural language-based applications without being affected by unexpected biases. At Q2BSTUDIO, we understand that model reliability is as important as their power. That is why we integrate principles of statistical robustness into our artificial intelligence solutions for businesses, ensuring that every evaluation of LLM-generated responses is consistent and free from distortions. Additionally, we offer custom applications that incorporate robust judge panels, AWS and Azure cloud services to scale these architectures, and cybersecurity tools that protect the evaluation pipeline. Our approach to business intelligence services with Power BI allows visualizing the results of these evaluations, providing IT leaders with a clear view of their models' performance. Robustness is not a luxury: it is a necessity for any serious AI implementation.