We check our reviews against human reviewers.
Valara ships with an eval harness that runs the real review workflow over a labeled set of appraisals and lines each result up against the review a human wrote for the same file. This page explains how that comparison works. It does not report scores; the harness is a tool we run, and the numbers move as the product changes.
How the comparison is built.
The value of comparing to a human reviewer is that both sides answer the same questions. The harness makes that literal: the AI review and the human review end up as the same object.
- 01
A labeled dataset
We keep a set of real appraisals, each paired with a review written by a human reviewer. The files are grouped by the grade the human assigned, from low to strong, so the set spans the range of report quality.
- 02
The same workflow you would get
Each appraisal is sent to the /api/eval route, which runs the full production review workflow synchronously and returns the same result schema a live review produces. There is no separate "eval model": we test the pipeline that runs in the product.
- 03
The human review, same shape
Each paired human review is transcribed into that identical schema. Because both the AI review and the human review are the same structured object, every field lines up for a direct comparison.
- 04
A side-by-side diff
The harness compares the two assessments field by field (scores, risk level, escalation, recommendation, and the individual findings) and writes a per-case report plus a combined comparison.
Residential and commercial, evaluated separately.
The two report types are graded differently, so they have their own eval runs, each against its own human-reviewed set.
Residential
Form-driven reports are run through the residential pipeline and diffed against the human reviewer’s checklist, grade, and accept-or-reject recommendation.
Commercial
Narrative reports are run through the commercial pipeline. Both the Valara review and the human review are expressed as the same commercial assessment object, so quality score, findings, and recommendation compare directly.
No headline number.
You will not find an agreement percentage or an accuracy score on this page. Those numbers depend on which files are in the set and which version of the workflow produced the review, and both change over time. We would rather show you the method, then run a review on a file you already know the answer to.
Related: what appraisal review means and how your files are handled.
Judge it on a file you already know.
Run a sample review, or bring an appraisal your team has already graded and compare.
5 credits included, no card required.