Model Output Liability

This document does not constitute legal advice. All contract structures and liability allocation should be reviewed by qualified counsel. The following serves only as introductory understanding for product decisions and contract negotiations.

Nature of the problem

Traditional SaaS tools do not make decisions for users — Notion does not auto-write wrong contract clauses; Excel does not auto-compute wrong numbers. When a user makes a mistake with a tool, responsibility lies with the user.

Agent products change this structure. The agent can:

Autonomously decide which tool to invoke
Autonomously decide what data to input
Autonomously decide when to complete vs retry
Autonomously interact with external systems (send email, write files, call APIs)

Every step may produce irreversible external impact. If the agent sends a wrong email to a customer, writes a wrong contract clause, or operates on the wrong database record — where does responsibility lie?

There is no simple answer. Legal precedent is sparse, regulatory frameworks immature. But the vendor must specify clauses in the contract, otherwise it is reactive when disputes arise.

Three-party liability structure

Four parties are involved:

End user — the individual submitting task requests
Customer enterprise — the paying entity, the organization deploying the agent
Vendor — the agent product supplier (i.e., you)
Upstream model provider — Anthropic / OpenAI / self-hosted model source

Ideally, responsibility is allocated by these principles:

Source of error	Primary responsible party
User submitted incorrect request	User / customer enterprise
User submitted correct request, agent misunderstood	Vendor
User correct, agent understood correctly but output wrong	Vendor (with upstream model traceability)
Vendor-provided tool itself has design flaw	Vendor
Upstream model systematic bias / error	Vendor (with right to back-pursue upstream)
User bypassed HITL and approved directly	User / customer enterprise

Key principle: vendor bears primary responsibility in customer-facing contracts, but internally reserves the right to back-pursue upstream model providers.

Legal weight of HITL

HITL (Human-in-the-Loop) is not just a quality control mechanism — it is also a liability transfer mechanism.

Consider two contract clauses:

A. Fully autonomous agent:

“The agent autonomously completes all task steps; the vendor bears primary responsibility for agent decisions and outputs.”

B. HITL mode:

“High-risk actions (external email send, payment execution, data modification, etc.) require user manual confirmation. Effects produced after user confirmation are the responsibility of the user.”

Legal significance of clause B:

Final decision authority for “irreversible actions” is given to a human; technically the agent can still execute, but legally a human presses “confirm”
Even if the agent recommended incorrectly, the human pressing confirm shifts responsibility to the human
Legally analogous to “Autonomous Driving L2” vs “L4” — L2 the driver still responsible, L4 the manufacturer responsible

Vendor’s practical choice: unless requirements explicitly require otherwise and the legal team approves, default to HITL mode. Fully autonomous agents create enormous risk exposure for the vendor, and current legal frameworks provide insufficient protection.

Sample contract clauses

For enterprise customer service agreements, the following clauses should be included (sample text; counsel must review):

14. Agent Autonomous Actions and Liability

14.1 Customer acknowledges that agent product outputs are generated by
    probabilistic models and may contain errors, inaccuracies, or content
    unsuitable for specific scenarios. Customer is responsible for
    reviewing agent outputs and independently verifying their suitability.

14.2 For actions requiring user confirmation through the HITL workflow,
    all consequences arising from user confirmation are the responsibility
    of Customer. Vendor bears no responsibility for outcomes of actions
    after user confirmation.

14.3 For actions executed autonomously by the agent without HITL
    confirmation, Vendor bears responsibility within reasonable bounds,
    subject to liability caps, force majeure, and other clauses in this
    agreement.

14.4 Customer shall configure a list of actions the agent shall not execute
    autonomously ("high-risk action list"), e.g., external email send,
    payment execution, contract signing, database schema modification.
    Listed actions require HITL confirmation; unlisted actions may be
    executed autonomously by the agent.

14.5 Vendor's liability for systematic failures or output bias from upstream
    model providers (Anthropic, OpenAI, etc.) is bounded by this Service
    Agreement. Customer understands that upstream model providers' terms
    apply to their relationship with Vendor; Vendor will reasonably pass
    through available protections.

Insurance product landscape

E&O (Errors & Omissions) insurance traditionally covers software-error customer losses. Agent product insurance status:

Most E&O policies do not explicitly cover AI autonomous decisions by default — policy language typically presumes software is a “passive tool”
Some insurers (AXA, Munich Re, etc.) have begun offering AI-specific riders covering “losses from agent autonomous decisions”
Premiums are noticeably higher than traditional E&O — agent risk exposure lacks historical data, underwriters price conservatively
Vendors with robust HITL clauses typically obtain lower premiums

Practical recommendations:

Early stage (< $5M ARR): traditional E&O + explicit AI disclosure; discuss policy language with carrier
Mid stage ($5M-50M ARR): consider AI-specific riders
Before large enterprise contracts: let customers specify insurance terms in contracts (many enterprise customers require vendors to hold E&O at specific coverage)

Regulatory direction

As of this writing, key agent product regulatory developments:

EU AI Act (2024 in force, 2025-2026 phased implementation) — risk-tier classification; high-risk uses (healthcare, hiring, credit scoring) have specific obligations; systems with high agent autonomy tend toward “high-risk” classification
US White House AI Executive Order (2023-) — currently constrains foundation model providers more than application providers, with spillover risk
China’s Generative AI Service Management Interim Measures (2023) — content moderation responsibility explicitly assigned to service providers
Industry self-regulation — professional associations in finance, healthcare, legal sectors are drafting practical guides for AI autonomous decisions

Compliance strategy: do not bet on “regulation will always lag” — preserve HITL boundaries and audit trails in product architecture; when regulation lands, the product does not need rework.

Cross-section connections

HITL position in the cost-benefit model: economics/controls-and-roi
Technical implementation of audit trails: operations/overview
Upstream model provider data terms: data-feedback