Compliance framework for AI

Compliance framework for AI 



Following the passing of the EU AI law, in August, which outlines six ethical principles and uses a risk-based scoring system to evaluate AI systems against these principles and label them as unacceptable, high, and minimal risk, LatticeFlow, ETH Zurich, and the Institute for Computer Science, Artificial Intelligence and Technology (INSAIT) have created Compl-AI, a framework designed to help developers of AI technology evaluate their models and understand how they perform against the EU’s AI principles and if they’re compliant with the law.

  • To assess an AI model’s performance against the EU law's six principles, Compl-AI’s (no-cost, open source) framework evaluates its responses across 27 benchmarks, including “following harmful instructions” and “truthfulness.”

  • The framework then gives the AI model a score of between 0 (not compliant) and 1 (fully compliant), or N/A if there is insufficient data, and has already scored top AI models from Anthropic, OpenAI, Meta, and Google.

  • While “no model achieved top marks,” OpenAI’s GPT-4 and Anthropic’s Claude 3 Opus were the most compliant, scoring 0.89, and Google’s (open-sourced) Gemma models scored the lowest (0.72).

The Compl-AI framework has been well-received by regulators, as they’ve faced criticism over the vague, non-technical, and broad set of laws, that hadn’t been translated into practical benchmarks that evaluated compliance, and see it as “the first step in translating the EU AI Law into technical requirements, helping AI model providers implement the AI Laws.”

- 

Comments

Popular posts from this blog

IBM goes big to keep quantum computers cold

IBM - avances en mantener adecuadamente fríos computadores cuánticos.

Estructura multinivel de los sistemas de agentes de IA