Local AI agents
What counts as a verified improvement for a local AI agent?
How I separate agent activity, task closure, and a measurable gain over a baseline.
After a long agent run, a report can show dozens of completed cycles, a few decisions, and a convincing explanation of what changed. The first number I check is how many improvements the run could verify.
Three things a run can show
- Activity. A cycle ran or the agent attempted an action. This tells me where the runtime spent time.
- Task closure. A check tested a stated assertion and produced a result I can inspect. This tells me whether one task met its criteria.
- Capability improvement. A candidate beat a baseline on an agreed measure using relevant cases outside the examples used to prepare it. This tells me whether the change earned promotion under that test.
An example from Evolution Lab
In a four-hour November run, three agents completed 32 cycles. The report logged 31 API errors and zero independently verified capability improvements. Earlier in the same report, the synthetic preflight recorded three valid outputs from three checks. Those results can coexist. I use the preflight to catch startup issues and the run evidence to evaluate what happened during the session.
The API error count tells me which part of the run deserves inspection. To make a claim about what caused those errors, I’d need the recorded error details and a controlled follow-up.
A check I can rerun
Suppose an agent proposes a prompt change to improve how often a local model returns a complete task plan. I’d freeze the baseline prompt, model, and inference settings. Then I’d run both prompts against the same independent cases and score whether each response contains the required fields and passes the task assertions. I’d also record latency and keep each failed case for review.
If the candidate wins on completion and clears the same checks the baseline already passes, I can report the gain with that test's scope. If the result is mixed, the candidate stays a candidate. A run with zero verified gains still gives me evidence about where to investigate next.