The complex algorithms that now determine everything from car insurance premiums to life insurance eligibility are operating largely behind a digital curtain, a reality that state regulators are determined to change. As the insurance industry rapidly integrates artificial intelligence into its core operations, the National Association of Insurance Commissioners (NAIC) is moving to pull back that curtain with a detailed new proposal. This initiative, known as the “AI Systems Evaluation Tool,” aims to create a standardized framework for oversight, but its method—a comprehensive questionnaire—has ignited a debate about whether it is a crucial step toward transparency or an impractical burden on both insurers and the regulators themselves.
This push for granular data represents a critical inflection point in the regulation of technology. At stake is the fundamental balance between fostering innovation in a competitive market and upholding the long-standing mandate to protect consumers from unfair or discriminatory practices. The development of the NAIC’s tool signifies a clear regulatory desire to move beyond abstract principles and into the specific mechanics of how AI is shaping the insurance landscape, a move that the industry is watching with a mix of apprehension and cautious engagement.
When an Algorithm Is the Underwriter: The Growing Push for AI Transparency
The central question fueling the regulatory push is disarmingly simple yet profoundly complex: How can consumers and regulators be certain that the AI systems determining insurance eligibility and premiums are fair and unbiased? As insurers leverage vast datasets to refine underwriting, claims processing, and pricing, the potential for algorithms to inadvertently perpetuate or even amplify existing societal biases has become a primary concern for consumer advocates and oversight bodies. The opacity of some complex models—often referred to as “black boxes”—makes it difficult to trace exactly how a decision was reached.
This challenge highlights a growing tension between the pace of technological adoption and the more deliberate speed of regulatory development. While insurers are moving quickly to deploy AI for efficiency and competitive advantage, regulators are tasked with building frameworks that can effectively monitor these dynamic systems without stifling innovation. This gap has created an environment where rules are struggling to keep up with reality, prompting bodies like the NAIC to seek more direct methods of inquiry to ensure accountability.
Beyond the Bulletin: Why the NAIC Is Demanding More Than Policies
The foundation for the current initiative was laid with the NAIC’s 2023 Model Bulletin on AI, which established high-level governance principles. That document encouraged insurers to develop and maintain robust risk management frameworks, but it largely left the specifics of implementation to the companies themselves. The bulletin served as an important statement of intent, signaling to the industry that regulators were focused on the responsible use of artificial intelligence.
However, the new “AI Systems Evaluation Tool” marks a significant departure from these abstract guidelines. It represents a regulatory shift from accepting high-level policy statements to demanding concrete, granular data about how AI is actively being used. This move is driven by the understanding that a policy on paper provides no guarantee of fair outcomes in practice. By compelling insurers to disclose detailed information about their models, data sources, and governance checks, the NAIC aims to create a more empirical basis for oversight and consumer protection in an increasingly data-driven marketplace.
Inside the “AI Systems Evaluation Tool”: A Four Part Regulatory Deep Dive
The proposed tool is structured as a meticulous questionnaire designed to provide regulators with a comprehensive view of an insurer’s AI ecosystem. It is divided into four distinct sections, each probing a different facet of AI implementation. The first part aims to quantify the scope of AI systems in operation, essentially creating an inventory of all models used. The second section assesses the company’s overarching governance and risk management framework, examining the policies and procedures in place to control AI-related risks.
The third and fourth parts delve even deeper into the technical specifics. Part three focuses on the mechanics of high-risk models, asking for details on how they function and are tested for fairness. Part four examines the lifeblood of these systems: the data. It probes the sources, quality, and suitability of the data used to train and run the algorithms. The development of this tool by the NAIC’s Big Data and Artificial Intelligence Working Group has been a slow and deliberate process, with a recent four-hour session covering only the first of the four sections, underscoring the complexity of the task.
A Clash of Perspectives: The Regulator’s Mandate vs Industry’s Burden
The regulatory viewpoint, articulated by Iowa Insurance Commissioner Doug Ommen, frames the tool as a necessary “conversation starter.” He emphasizes that the goal is not to stifle technological progress but to understand how insurers are applying AI and ensure it aligns with consumer protection laws. Ommen has also been careful to distinguish the NAIC’s role, stating that the focus is on regulating the application of AI by insurers, not the development of the technology itself, a clarification meant to separate the NAIC’s work from broader federal tech initiatives.
This perspective is met with sharp criticism from industry experts. John Romano, a managing director at the advisory firm Baker Tilly, characterizes the proposal as more of an “inventory device” than a true evaluation tool. His primary concern is that it will force insurers to produce a massive volume of highly technical information that state regulators, who often lack deep data science expertise, may be ill-equipped to analyze. This could lead to a situation of data overload without producing meaningful regulatory insights. Similarly, Heidi Lawson, an attorney at Fenwick, acknowledges the need for a standardized approach but calls the questionnaire an “old fashioned” and “limited” method for tackling such a complex technological issue.
Beyond the Questionnaire: Charting a Modern Path for AI Oversight
The ongoing debate raises a pivotal question: Is a static, detailed Q&A the most effective way to regulate a technology as dynamic and complex as artificial intelligence? Critics argue that by the time insurers complete the questionnaire and regulators analyze the responses, the underlying models and data could have already changed significantly. This has led to calls for more modern and adaptive approaches to oversight.
Experts like Lawson suggest alternative strategies, such as leveraging specialized third-party firms to conduct technical diagnostics on AI models. These audits could actively test for accuracy, fairness, and model drift in near-real time, providing a more dynamic assessment than a written survey. This path presents strategic considerations for both sides. Insurers must begin preparing their systems and governance for a future of increased technical scrutiny, while regulators face the challenge of building the internal capacity—or external partnerships—necessary to analyze the complex data they are demanding.
The NAIC’s initiative to create a standardized AI evaluation tool marked a critical moment in the history of insurance regulation. It clearly exposed the fundamental tensions between the regulatory imperative for transparency and the industry’s deep-seated concerns about practicality, data security, and the potential for a compliance exercise that yielded little substantive value. The vigorous debate over the tool’s methodology—pitting a traditional questionnaire against calls for more dynamic, technical auditing—ultimately framed the central challenge that would define the next era of oversight: how to effectively govern a technology that was, by its very nature, constantly in motion.
