Why We Don't Trust Our AI by Default

TL;DR

AI-generated legal work product will sometimes be wrong. That's true of every vendor, including us, and the useful question for an attorney evaluating any AI tool isn't "does it ever make mistakes" — it's "what catches the mistake before it reaches you." This piece lays out the standard we hold AI output to: a scorecard where a single fabricated fact or unsupported legal conclusion is an automatic failure, regardless of how well the output scores everywhere else, and a set of maintenance commitments for the parts of our product that depend on legal rules that keep changing, like California's family law form set. We also walk through a real, easy-to-make classification error in that form set as a concrete example of what unmonitored mapping looks like when it goes wrong.

Confidence Is the Real Risk, Not Ignorance

A model that doesn't know something usually hedges. A model that's confidently wrong doesn't. That distinction matters more in family law than in most AI applications, because the correct answer to "which forms does this case need" depends on facts that shift the entire packet: whether children are involved, how the other party was served, whether parentage is already established, whether the matter proceeds by default, agreement, or trial.

California's form set makes this concrete. FL-100 and FL-110 are the conventional starting point for nearly everyone. FL-105 is required only if there are minor children. The service method determines whether you need FL-115 or FL-117 — they are not interchangeable. FL-142 and FL-160 are alternative ways of presenting the same asset-and-debt information, not duplicates. And as of January 1, 2026, there's an entirely separate joint-petition pathway (FL-700/FL-710/FL-720) that predates most training data.

A model trained on "what FL-100 is" can still be completely wrong about "does this case need FL-100 or the new joint-petition track." That's not a knowledge gap you fix by making the model smarter. It's a mapping problem, and mappings decay the moment a rule changes underneath them.

A Mistake That's Easy to Make

Here's a concrete illustration of how subtle this gets. FL-145 is Form Interrogatories — Family Law. It belongs in a discovery workflow. It has nothing to do with the financial-disclosure packet (FL-140, FL-141, FL-142, FL-150) — but it sits close to those forms numerically, and a mapping built once and never re-checked against the source could easily list it alongside them. The result would look complete and confident. It would also misroute the packet for anyone relying on it.

That's the pattern worth sitting with: the error isn't a wild hallucination that's obviously wrong on its face. It's a quiet, plausible misclassification that requires someone to actually check against the primary source to catch — which is the entire argument for treating source-checking as a standing process rather than a one-time step.

What We Hold Our Own AI To

Knowing the mapping problem exists doesn't solve it — it just tells you where to build guardrails. Any AI output has to clear a weighted scorecard before it's trusted with real casework:

The weights aren't the important part. The important part is this: critical failures are scored outside the weighted index. A single invented fact or an unsupported legal conclusion is an automatic failure at the relevant gate, no matter how well the output scores on everything else. A model that's 95% excellent and occasionally fabricates a filing date doesn't average out to "pretty good" — it fails. In legal work, one confidently wrong fact can outweigh ten correct ones, and a scoring system that averages that away is measuring the wrong thing.

The scorecard is also not the whole decision. Which AI model actually powers a given workflow gets weighed alongside cost, latency, reliability, security, observability, and portability — a model that scores well on accuracy but can't meet the operational bar doesn't get used either.

Rules Aren't Static, So We Don't Treat Them Like They Are

Catching a mismatch like the FL-145 example only works as an ongoing practice if the underlying registry is treated as something that requires maintenance — not a fact sheet that was accurate on the day it was written and left alone after that.

We hold form-mapping content to four requirements:

1. An owner. A person, not "the model," is accountable for whether the mapping is current.
2. A source. Every entry traces to official California Courts material, not to what a prior output assumed.
3. A review cadence. Revision dates and effective dates get checked on a schedule, not only after something breaks.
4. Regression tests and release controls. Changes go through the same discipline as a code change — tested before release, not patched live.

If you're evaluating an AI vendor for anything touching legal documents, ask directly which of these four exists at their company. "The model is trained on current forms" answers none of them, and it's the answer most vendors give.

The Registry, as It Stands Today

For reference, here's the current form map — reviewed against official California Courts materials in July 2026. It's a product-design scope map, not legal advice or an exhaustive filing checklist; county requirements and case-specific facts can change the actual packet, and attorney review still governs any real filing.

What to Ask Any AI Vendor, Not Just Us

If you're evaluating AI for anything touching family law documents — ours or anyone else's — these are the questions that actually separate rigor from marketing:

• Who owns the mapping between case facts and which forms apply, and how often do they check it against the primary source?
• What happens when the AI is confidently wrong — is there a rule that catches it, or does it just average into an overall "quality" number?
• Is the sourcing tied to specific official pages and dates, or to a general claim of "trained on current law"?
• Do they treat their own output as something to keep checking, or as a feature they shipped once and moved on from?

We don't think any of this is a finished state, for us or anyone else in this space. The registry above will need updates as California's rules change again. The scorecard will get adjusted as we learn where it's too lenient or too strict. Treating both as things that require continuous scrutiny — rather than a one-time build — is the actual standard, and it's the one we're asking to be held to.

Sources

• California Courts — Start a divorce: required forms
• California Courts — Respond to divorce papers
• California Courts — Joint petition for divorce or legal separation
• California Courts — Petition for custody and support
• California Courts — Request custody and parenting-time orders
• California Courts — Child support forms
• California Courts — Financial disclosures
• California Courts — Finalize a divorce
• California Rule of Court 5.275 — certification standards for support-calculation software
• State Bar of California — Ethics & Technology Resources — 2026 guidance on generative and agentic AI

For more on how Aparti approaches AI in family law practice, see AI software to auto-generate FL-100, FL-140, FL-142, and FL-150 California divorce forms and the Family Law Collections Playbook.

This piece describes a product-design scope map, not legal advice or an exhaustive filing checklist.