Modeling parties in legal matters

Party extraction fails when one label carries too much meaning. We separated identity, documentary authority, legal subject, role, and changing legal state because each requires different evidence and different tests.

Our current reference experiment concentrates on subjects and roles. Entity resolution and instrument posture remain separate PartyGraph layers outside the subject catalog. Legal positions, events, and resulting states will require their own claim types.

The evaluation that forced us to rethink the method asked a narrow question: could the existing pipeline identify the principal parties in matters it had not seen during development? In this case, a principal referred to an entity with a direct stake in the matter's economics or risk allocation.

Test harness

The design tried to isolate generalization by freezing the system and scoring rules before exposing it to the test matters. We selected five previously unread matters and pinned the extractor, model, scorer, and acceptance thresholds.

To build the answer key, reviewers started with the frozen predictions, compiled evidence for every predicted entity plus 40 named-entity recognition candidates per matter, and applied more than 124 logged corrections. Five adversarial model reviewers then tried to refute the corrected labels. We scored the frozen predictions once, after the project owner—a human lawyer—manually ratified the remaining disputed calls.

What we observed

The run produced 179 predictions against 43 labeled principals. Precision was 0.851. Recall was 0.930, and F1 was 0.889. Deal-shaped matters performed better than litigation, settlement, and investigation matters. The review also found three fabricated entities. Each carried an invented or misattributed quotation.

Takeaway

The frozen comparison exposed extractor failures. Prediction-conditioned reference construction prevented an independent accuracy estimate. Reviewers had started from the pipeline's predictions, so all five adversarial passes audited the same prediction-conditioned label set. The extractor never saw the labels, so ordinary train/test leakage was absent. The dependence ran in the other direction. The extractor helped define the reference used to score it, making shared omissions difficult to detect. The 0.889 F1 is therefore a directional baseline. It cannot support a calibrated estimate of performance.

Adjustment

We made verbatim evidence resolution a deterministic admission gate. We also required prediction-blind annotators, preserved annotator outputs, claim-level evidence, model provenance, explicit contested outcomes, and a whole-matter omission audit. Those changes shifted the immediate problem from improving the extractor to constructing a reference that could support a credible measurement.

The second experiment tried to build the richer answer key that the first run lacked. It captured entities, instruments, roles, relationships, and exact evidence across one complete synthetic matter.

Test harness

We partitioned 143 documents from a test matter into 30 frozen batches. Independent model sessions proposed claims, a source-reopened adjudicator issued 1,163 decisions, an auditor reviewed a 108-decision sample, and reconciliation repaired the remaining structural conflicts. Validators checked quotation text, source hashes, duplicate claims, and dangling references.

What we observed

The reconciled graph passed its structural validators with 1,133 claims, 1,554 evidence references, and 24 contested claims. It looked rigorous. A later role-taxonomy crosswalk compared each free-text role with the available reusable role definitions. Of 751 proposed mappings, 82 preserved the intended legal meaning, 666 had no valid mapping, and 3 remained contested.

Takeaway

Structural validity and semantic usefulness are different properties. The validators established internal consistency across records, quotations, hashes, and duplicate keys. Semantic fit with the intended product domain remained untested. The source was a patent-litigation matter, while Dealteam was primarily a transactional product. The reference represented inventorship, docket events, litigation posture, and matter-specific counsel roles accurately enough to pass its checks. The transactional pipeline needed to learn a different set of relationships first.

Adjustment

To steer the model toward more useful semantics, we limited induction of test matters to contract projects. We also separated three artifacts that the litigation run had blurred together: the source-backed inventory of one matter, the reusable cross-matter catalog, and the party graph produced when that catalog is applied. A matter can support 165 local subjects without requiring 165 new catalog entries.

The third experiment returned to purely-transactional matters and treated the catalog of subject areas as the object under development. Each new fixture could confirm an existing subject area, refine its boundary, or propose a new reusable subject area.

Test harness

A high-capability model reviewed one complete matter at a time and proposed subject names, definitions, aliases, boundary rules, and suggested roles. We compared each proposal with the approved catalog. Changed and new entries returned to candidate status, and at least one human-lawyer reviewer ratified a semantic redline before the next fixture. The process produced successive approved catalog revisions through version 1.5.

What we observed

The catalog became coherent enough for useful legal review. It distinguished subjects from roles, recorded aliases, and stated when adjacent concepts should merge or remain separate. Each round preserved the catalog and discarded the source observations that induced its entries. Most provenance identified a document rather than an exact quotation and an explicit induction decision.

We nevertheless tried to test semantic retrieval against the catalog. The test expanded citations into passages of roughly 4,000 characters and assigned each passage one expected subject. A passage of that size can span several clauses and support several subjects.

Takeaway

As a result, the retrieval score was not identifiable: low top-five recall could have come from the catalog, the retrieval method, the passage boundaries, the single-label assumption, or the labels themselves. More importantly, we could not reverse the catalog-building process and ask whether a new mapper recovered the concept from the source observations that had induced it. The coherent catalog lacked a defensible expected output against which to score a mapper.

Adjustment

We archived the catalog and made the per-matter source inventory the primary asset. Every future subject had to retain exact evidence, its matter application, and the consolidation decision that produced the reusable concept. We also reserved untouched matters in advance and deferred global catalog induction until several matter references could be considered jointly.

Guide v3 was the first direct test of whether independent model sessions could construct the same subject inventory without seeing pipeline output or an existing catalog. We used an unread secured-finance matter for this run.

Test harness

Two isolated GPT-5.6 High sessions reviewed all 40 documents and accounted for all 1,470 projected source units. A source unit was a stable text block in the normalized document projection. Each session moved directly from those units to a consolidated list of subjects and roles. A source-reopened adjudicator crosswalked every proposal, searched the complete matter for omissions, and applied six gates covering subject recall, unsupported subjects, shared omissions, boundary changes, normalized role recall, and unresolved material errors.

What we observed

Adjudication accepted 52 subjects and 129 normalized subject-role pairs. Annotator A found 42 of 52 subjects, or 80.77%. Annotator B found 38, or 73.08%. Both missed the same four subjects. Nine of 45 adjudication decisions required a material merge or split. Role recall was 74.42% for A and 55.81% for B. The pilot failed five of its six gates.

Takeaway

Mechanical source-unit coverage had overstated semantic coverage. One annotator could place 96 to 119 units into a generic exclusion bucket without identifying the mechanisms inside them. The direct jump from documents to taxonomy also hid the error stage. A missing final subject might never have been discovered, or it might have been discovered and then merged away during consolidation.

The role gate measured the wrong object. Role lists were subject-scoped suggestions for later party mapping. Exact normalized role recall treated them as exhaustive answer keys, rewarded longer lists, encouraged synonymous role proliferation, and allowed role wording to influence whether two reviewers treated a subject as one legal unit or several.

Adjustment

We split discovery from consolidation. A discovery session would record source mechanisms without naming final reusable subjects. A fresh source-blind session would consolidate that frozen ledger, making discovery loss distinguishable from boundary loss. We also made exclusions granular, separated reusable subjects from their matter-specific applications, and moved role enrichment after subject qualification.

Next, we tested the revised workflow on a new unread matter. The design now exposed the stage at which a subject disappeared and gave the combined result an independent falsification test.

Test harness

Two blind discovery sessions each accounted for every source unit. Two fresh source-blind sessions consolidated the resulting ledgers. They worked only from the recorded observations and quotations. Original documents and another reviewer's proposed taxonomy remained unavailable. A source-reopened adjudicator compared those inventories, consulted the frozen documents, and swept the whole matter for shared omissions. A third source-first workflow independently read the documents without seeing either principal workflow or the adjudication. It froze its own inventory before a final auditor compared the two results and reopened source for every difference.

We fixed the subject-level gates before the run and removed role coverage from qualification. The composite fallback had its own thresholds: no more than 5% unsupported subjects, no more than 5% omissions, less than 10% material boundary revisions, and no unresolved material errors.

What we observed

The component workflow still failed. Annotator A found 53 of 56 adjudicated subjects, or 94.64%. Annotator B found 47, or 83.93%, below the 85% gate. Sixteen of 43 adjudication decisions required a material boundary change, a 37.21% failure against the 10% limit.

The third blind workflow proposed 92 subjects. The final auditor accounted for all 92 proposals and all 56 adjudicated subjects, reopened the source, affirmed the 56, and added two omissions. The corrected inventory contained 58 subjects, including two audit-only omissions, for an omission rate of 3.45%. The audit rejected no adjudicated subject, revised no adjudicated boundary, and left no material error unresolved. The composite passed its registered fallback gates.

Takeaway

The revised guide improved discovery. One blind workflow still failed to reproduce the final inventory. Boundary stability remained the largest source of disagreement. Boundary stability measures whether reviewers agree on the legal unit itself; terminology is irrelevant. A guaranty may be one subject, or defense waivers, subrogation, and contribution may require separate subjects when they change independently or assign different participant functions.

The independent audit established a specific result: the combined multi-pass reference survived a fresh source-first challenge within the registered error thresholds. The failed component metrics remain failed, and neither principal workflow proved reliable by itself.

Adjustment

We registered the composite fallback as part of the method. Failed component metrics remain attached to every passing composite. A combined candidate becomes provisional reference material only after the third workflow and final audit satisfy the fixed unsupported, omission, boundary-revision, and unresolved-error gates.

Run Five established only that the complete multi-pass process could produce one reference that passed the registered gates. Run Six tested whether that result was reproducible. We froze the guide, workflow sequence, and gates, then applied them to three additional registered transactional matters. The hypothesis was that the complete process would produce qualified references under the same thresholds without matter-specific changes, even when an individual workflow or the first composite failed.

Test harness

Each matter used the established sequence: two blind discovery and consolidation workflows, source-reopened adjudication, a third blind source-first workflow, and a final source-reopened audit. A failed composite could enter one confirmation cycle. Its failed output became the candidate for a fresh blind challenger and a new confirmer. The numerical gates did not change.

What we observed

Development matter A failed its first composite audit. The audit produced 138 corrected subjects and added 12 omissions, an 8.70% rate against the 5% gate. We classified those 138 subjects as a candidate. A fresh challenger and confirmer produced 152 subjects. Seven were absent from the candidate, or 4.61%, and six candidate boundaries required material revision, or 4.35%. The confirmation passed.

Development matter B passed on its first composite audit. The principal workflows produced 87 and 134 subjects. Both met the recall and unsupported-proposal gates. Adjudication still required 17 material boundary decisions out of 93, or 18.28%. The independent audit produced 165 corrected subjects. Eight were absent from the adjudication, or 4.85%, and three adjudicated boundaries required revision, or 1.96%. No adjudicated subject was unsupported, and no material error remained unresolved.

Development matter C also passed its first composite audit. Its component failures were more pronounced. Annotator A recovered 74 of the 90 adjudicated subjects, or 82.22%, below the 85% gate; annotator B recovered 86, or 95.56%. Neither annotator proposed an unsupported subject, and they had no shared omission. The adjudicator still needed 14 material merge or split decisions out of 79, or 17.72%, nearly twice the permitted rate.

The third blind workflow produced 93 subjects. The auditor reopened the source and produced 101 corrected subjects supported by 338 exact quotations. It rejected two unsupported adjudicated subjects, recovered four subjects absent from the adjudication, and materially revised seven adjudicated boundaries. The final omission rate was 3.96%, the boundary-revision rate was 7.78%, and no material error remained unresolved.

Takeaway

Run Six has now met its stated test on the first three registered development matters. The frozen composite procedure qualified two references on their first audit and one after the registered confirmation cycle. Across the pilot and three development matters, final omission rates were 3.45%, 4.61%, 4.85%, and 3.96%. Final boundary-revision rates were 0%, 4.35%, 1.96%, and 7.78%. No final audit left a material error unresolved.

The component workflow failed the boundary-stability gate in all four matters. It also missed the blind-recall gate in the pilot and again in development matter C. The evidence therefore supports the composite reference method. Single-annotator replacement and single-pass catalog generation remain unsupported.

Adjustment

Development matter C shows why the composite procedure is necessary. The principal path failed recall and boundary gates; the independent challenge and audit corrected it within the final gates. We will neither simplify nor alter the procedure before the final registered development matter. If that matter qualifies, we will induce one reusable catalog from all qualified references at once.

Repeated legal relationships should converge on shared subject definitions, while deal-specific facts remain attached to their local matter applications. We expect the number of new catalog subjects to fall as the corpus grows. Persistent merge and split disputes will identify catalog boundaries that still need revision.

We will then freeze the catalog candidate and test a cost-optimized mapper on untouched transactional matters. That evaluation will ask whether the production pipeline can recover the referenced entities, subject applications, useful roles, and exact evidence more reliably than document-level name matching. The composite process builds the answer key. The mapping evaluation tests the product.

The catalog is built from qualified matter references that preserve the evidence and decisions behind every reusable subject. Each subject must be reconstructible from its exact source evidence, matter-specific observations, consolidation decisions, disagreements, and audits.

Each packet pins the guide, schema, source manifest, projected source units, and model provenance. Validators reconstruct coverage, quotation resolution, hashes, cross-record references, and reported metrics. The language models still make the semantic decisions. Deterministic code establishes whether those decisions are internally consistent and traceable to source.

The catalog will be induced jointly from qualified matter references. This avoids making the first fixture the permanent conceptual anchor. It also lets a catalog entry retain all supporting matters, competing formulations, negative examples, and boundary decisions.

Each subject area can contain many roles. The catalog suggests recurring role pairs, e.g., licensor/licensee, discloser/recipient, and seller/buyer, so the model can reuse familiar labels when they fit.

The three development references show what this standard produces.

Across the three matters, reviewers examined 418 qualified subjects and admitted 475 role suggestions across 318 of them, backed by 406 exact evidence items. For the other 100 subjects, they recorded an explicit no-role finding because the matter did not support a participant function.

The results show source provenance and disciplined abstention. We deliberately keep the role catalog open-ended. A matter may require labels the catalog does not anticipate. The pipeline's task is to determine which entities occupy which roles within each subject area, reusing catalog labels when they fit.

The experiment now establishes the following claim: a controlled multi-pass process can construct source-backed provisional subject references for complex transactional matters while measuring omissions, unsupported proposals, and material boundary changes.

We call these references provisional gold to mark them as our best current answer keys. The label remains provisional because a lawyer working on a live matter is the best arbiter of whether a claim is useful and correct. The system must preserve that person's correction as a claim-level decision, then test any proposed catalog or policy change before applying it elsewhere.

One registered development fixture remains before the holdout stage. It will use the same frozen workflow. We will then induce the reusable catalog jointly from all qualified matter references. Joint induction removes fixture-order dependence.

Every catalog subject should retain its supporting matters, exact evidence, aliases, boundary tests, negative examples, and role suggestions. Genuine ambiguity should remain explicit. A term may point to two subjects when the source context decides between them.

Only after that consolidation will we run the cost-optimized mapping pipeline against untouched transactional matters. The comparison will test entity identity, subject application, role usefulness, evidence validity, and the lawyer's ability to verify or correct each claim. That comparison is the product test; catalog size alone tells us nothing about performance.