The Shadow AI Problem in Insurance: Navigating Trust, Oversight and Innovation
Key Highlights
- Agencies are using AI mainly for email drafting, transcription and administrative tasks, but treating it as a human oversight tool rather than an autonomous operator.
- Carriers are skeptical about AI's measurable impact on workflow efficiency and are concerned about accountability, auditability and regulatory risks associated with unmonitored automation.
- The industry needs a trusted middleware layer to authenticate, standardize and audit AI interactions, similar to payment platforms like PayPal, to improve visibility and control.
- Responsible AI adoption requires transparency, clear ownership, bias mitigation, regulatory compliance and role-based permissions to build trust among stakeholders.
- Process mapping and workflow analysis are recommended first steps for insurance leaders to identify where AI can support or improve operations without replacing human judgment.
As insurance agencies quietly experiment with AI tools, carriers are confronting a growing governance gap they cannot fully see, monitor or trust. The future of insurance AI may depend less on automation itself and more on the systems needed to govern, orchestrate and audit it.
Insurance leaders are hearing increasingly bold claims about artificial intelligence transforming distribution. But inside many agencies, the reality is far messier: Fragmented experimentation, inconsistent governance and the growing concern among carriers about workflows they can neither fully see nor control.
In many ways, the insurance industry may not have an AI problem as much as a trust problem.
Across agencies, AI is increasingly being used to draft emails, summarize documents, support quoting workflows, transcribe calls and reduce administrative burden. Yet while agencies often view these tools as productivity enhancers, carriers are asking a harder question: How do we govern what we cannot fully observe?
That tension sits at the center of what some industry leaders are beginning to describe as insurance’s emerging “shadow AI” problem — a growing gap between experimentation and oversight.
To better understand where AI experimentation is colliding with governance concerns, I spoke with two leaders who sit on opposite sides of the insurance ecosystem. Jason Cass, managing partner of Virtual Intelligence VI, co-founder of Agency Intelligence and owner of a multi-location independent insurance agency, helps agencies navigate operational realities and emerging technology. Delanea Davis, co-founder and managing partner of Experience Design International, brings a carrier-side lens informed by years of translating frontline agency experiences into actionable insights for insurers. While both see enormous long-term potential in AI, their perspectives reveal an industry still working through a fundamental question: How do you innovate responsibly when trust, visibility, and accountability have yet to catch up?
“I think AI in agencies is way overhyped right now, and it’s the Wild West,” said Cass. “I absolutely believe agentic bots are the future workforce of the independent insurance system. But over the next several years, this is still going to be messy and immature.”
Cass is careful to distinguish long-term optimism from present-day reality. While he believes AI will eventually reshape insurance operations, he argues that many organizations are confusing experimentation with readiness.
“What I’m seeing in agencies today is mostly very basic use: Email drafting, transcription, summarization, quoting support and administrative work,” he said. “That’s where AI is maximized right now. If you’re not using it for those productivity gains, you’re already behind.”
Where Cass becomes more cautious is when organizations begin treating AI as a replacement for human judgment.
“In my opinion, there is no task in an insurance agency today that should happen without human oversight,” he said. “Today, AI should function as a copilot. We’re not at a place where agentic systems should be operating independently in healthcare, insurance or similarly regulated industries.”
Why carriers see risk where agencies see productivity
For Delanea Davis, the issue looks different from the carrier side of the table.
“What I’m hearing from carriers is that there’s a disconnect between all the AI talk and the measurable outcomes carriers expect to see,” Davis said. When agencies claim they are “using AI,” carriers often evaluate those claims through operational metrics rather than excitement.
“Carriers immediately ask: Has submission quality improved? Are we getting fewer phone calls? Are forms being filled out more accurately? Has workflow efficiency actually improved?” she said.
In many cases, Davis said, carriers are not seeing meaningful movement in those key performance indicators yet.
“From their perspective, there’s skepticism,” she said. “Either agencies aren’t using AI the way they say they are, or they’re using it in fragmented ways that aren’t translating into measurable business outcomes.”
That skepticism becomes more pronounced when conversations shift from productivity to operational risk.
“The biggest concern I hear from carriers is accountability,” Davis said. “At the end of the day, regulators place the burden on carriers. If there are bots in the workflow, no human supervision and no proof of what happened in the process, the carrier gets in trouble.”
Davis pointed to a recent Florida example involving automated decision-making around deductible thresholds as a cautionary signal for insurers. Even in seemingly straightforward workflows, regulators stepped in because there was insufficient human oversight.
“That reinforced something I hear repeatedly: Carriers want a human in the loop,” she said.
Another growing concern is authentication and auditability.
For agencies, AI often feels like a productivity assistant. For carriers, it can feel like a governance blind spot.
“Even though there’s little evidence of it happening today, there’s fear that a bot could eventually operate using a licensed producer’s credentials,” Davis said. “From the carrier side, that uncertainty around trust, attribution and auditability is unsettling.”
Cass agrees that human oversight remains essential but believes some carrier fears may be overstated.
“I really do not believe quality agencies are allowing bots to operate end to end without licensed oversight,” he said. “We have licenses for a reason. We carry E&O risk for a reason. No agency owner I know is risking their livelihood by letting automation run unchecked.” Still, he argues carriers may be focused on the wrong threat.
“The bigger governance issue is agencies using personally identifiable information inside large language models,” Cass said. “Are there guardrails around usage? Are people putting information into systems they shouldn’t? That’s where governance matters.”
Defining the shadow AI problem
The phrase "shadow AI" can mean very different things depending on who is using it — and that distinction reveals much about the carrier-agency divide. For carriers, shadow AI often evokes an unsettling image: Invisible automation happening behind the curtain, influencing workflows without transparency or clear accountability.
“From a carrier standpoint, the concern is: Is a person doing this, or is AI doing this? How was it trained? What decisions were influenced? What role did a human actually play?” Davis said.
Cass sees the term differently.
“In my [agency] world, shadow AI literally means the AI is shadowing the agency,” he said. “It sits in the background and watches work happen. It’s observing how people operate so it can begin understanding workflows and helping build orchestration.”
The Core Pillars of AI Governance
Effective AI governance structures operations around several critical focus areas:
- Transparency: Ensuring stakeholders understand how AI models are built, trained, and make decisions.
- Accountability: Assigning clear ownership and responsibility for AI operations, risks, and outputs.
- Fairness and Bias Mitigation: Actively measuring and reducing prejudiced outputs or discriminatory behavior within AI models
- Regulatory Compliance: Adhering to evolving legal frameworks and data privacy standards.
He describes AI maturity as a progression: First it observes, then suggests, then acts and eventually becomes autonomous. “Right now, most agencies should still be in that earlier stage,” Cass said. “There’s still a progression and human involvement.”
That difference in framing matters. For agencies, AI often feels like a productivity assistant. For carriers, it can feel like a governance blind spot.
Governance is not the same as orchestration
As the discussion moves beyond experimentation, both Davis and Cass argue the industry may be missing a more fundamental piece of infrastructure. When organizations talk about responsible AI, conversations often focus on governance: Transparency, accountability, fairness, compliance and risk mitigation.
Those things matter, Cass said, but governance alone is not enough.
“When we talk about what’s missing, I think the biggest word is orchestration,” he said. “Governance prevents bad things from happening. Orchestration allows good things to happen.”
He compares today’s environment to millions of cars suddenly appearing without roads or navigation systems.
“The cars are the bots,” Cass said. “Before we start worrying about speed limits, we need roads for them to travel on.”
In practical terms, orchestration means agreed-upon workflows, permission structures, integrations and standards for how AI participates in insurance operations. Without that connective infrastructure, organizations are left trying to govern systems that lack consistent pathways for participation.
Davis believes carriers increasingly want something resembling a trusted middleware layer — an intermediary system capable of authenticating, standardizing and auditing interactions between agencies, AI systems and carrier platforms.
“What I’m hearing from carrier leaders responsible for AI strategy is that what they really want resembles what banking created with platforms like Zelle or PayPal,” Davis said.
The analogy is instructive.
“They don’t control everything happening before or after a transaction,” she said. “They own one piece of the process and authenticate the transfer.”
In insurance, such an orchestration layer could help carriers gain visibility into AI-enabled workflows without stifling innovation at the agency level.
What Would It Take for Carriers to Trust AI?
If broader AI adoption in insurance is going to scale, both experts agree that trust will require more structure than currently exists.
“The biggest thing carriers need is explainability,” Davis said. “If there’s ever an audit or issue, they need to be able to untangle exactly what happened in a workflow.”
That means understanding:
- What role AI played
- Who approved decisions
- What data was used
- How workflows unfolded
- Where humans intervened
“Carriers want confidence that AI-enabled workflows can be understood, audited and trusted,” Davis said. She also cited a growing list of requirements carriers are beginning to discuss: Authenticated AI identities, role-based permissions, approved use cases, audit trails, monitoring, exception handling and clearly defined human-in-the-loop checkpoints.
Winners route work. Stuck organizations pile work onto people.
EXECUTIVE TL;DR
TL;DR: Insurance agencies are increasingly experimenting with AI for email drafting, summarization, quoting support and workflow efficiency, but carriers remain wary of workflows they cannot fully see, govern or audit. While agencies often view AI as a productivity tool, carriers are focused on accountability, explainability, authentication and regulatory exposure.
According to agency leader Jason Cass and carrier strategist Delanea Davis, the biggest barrier to responsible AI adoption may not be the technology itself but the lack of orchestration, shared standards and trusted infrastructure that connect agencies, carriers and AI systems. Their advice: Stop chasing AI hype, map workflows first, treat AI as a copilot rather than a replacement and build the governance required to scale trust.
Cass agrees that those expectations are both realistic and necessary. “The reality is that responsible adoption takes structure,” he said. He argues carriers should move away from prohibiting AI and move toward certifying it.
“We already certify things in regulated industries all the time,” Cass said. “Instead of carriers trying to stop agencies from using these tools, carriers should say: ‘Here are the standards. If a vendor meets these requirements, we’re comfortable with it.’”
He also believes vendors themselves need greater accountability. “Vendors need skin in the game,” he said. “You can’t just push out products because they’re flashy or faster without accountability.”
The accountability question
Responsibility, however, remains unsettled. Cass holds a direct view. “Accountability sits with the licensed entity,” he said. “If something goes out under my agency’s name, I own it. Full stop.”
Davis sees a more layered reality. “Both things can be true,” she said. “Agencies may own operational accountability through licensing and workflow decisions, while carriers still carry regulatory exposure.”
That tension may ultimately define the next phase of insurance AI.
What leaders should do next
For executives navigating AI adoption today, both Davis and Cass recommend resisting the temptation to start with technology. Instead, start with workflows.
“My biggest recommendation is process mapping,” Davis said. “Document the full end-to-end workflow and ask where AI could realistically support or improve a step in the process.”
Cass offers an equally practical lens. “Winners route work. Stuck organizations pile work onto people,” he said.
Rather than viewing AI as a replacement strategy, he argues leaders should focus on routing repetitive, lower-value work away from licensed professionals so employees can focus on customer conversations, growth and expertise-driven decisions.
“This really isn’t about replacing people,” Cass said. “It’s about structuring work better and using technology correctly to help people succeed.”
For insurance leaders, the lesson may be simple: The future of AI in distribution will depend less on who adopts it fastest and more on who builds the trust, governance and orchestration required to use it responsibly.
Future Ready, The Strategic Case to Structure, Route and Scale
The operating manual for building modern, intelligent insurance agencies. A comprehensive guide to reimagining agency structure through routing, automation, and strategic metrics.
About the Author
Jess Mand
Contributor
Jess Mand is an award-winning communications strategist and founder of INDEMAND Communications, where she helps organizations translate complex ideas into clear, compelling narratives that drive connection and action. She partners with Fortune 500 companies, growth-stage firms, and mission-driven organizations to design communication strategies, content programs, and experiential campaigns that engage employees and elevate leadership messages. Known for her creative storytelling and pragmatic approach, Jess brings a rare blend of strategic insight and human-centered perspective to every project she leads.
Resources
Quiz
Your Competitive Edge, DeliveredStay ahead of the curve with weekly insights into emerging technologies, cybersecurity, and digital transformation. TechEDGE brings you expert perspectives, real-world applications, and the innovations driving tomorrow’s breakthroughs, so you’re always equipped to lead the next wave of change.
