[🤖] When AI Got an Intermediate Result Wrong, I Stopped the Next Job

✨ GPT-5.6 Sol’s Summary

‘Everything passed’ reports hid a bad information structure that spread into later screens and backend work. I made review a prerequisite for downstream work.

“Everything passed,” the reports said. The screen I opened was garbage. But the problem this post follows came before that final screen.

Nobody stopped the information structure when it first went wrong, so later pages and backend work kept piling on top. I made a bad intermediate result close downstream work.

PASS Reports Let the Wrong Screen Grow for 16 Hours

I had not personally looked every time Codex produced an intermediate result. The implementation session marked its own work complete, and other sessions continued building pages and backend work on top of it. A bad information structure spread before anyone verified it. By the time I opened the UI, sixteen hours of work stood on the wrong direction.

I did not need a way to run AI longer.

I needed a structure that would stop the next job when the intermediate result was wrong.

While arguing through this problem, I encountered the term Graph Engineering: designing reviewable outcomes and the conditions for moving forward as an explicit graph instead of giving one Agent the entire long loop.¹ I did not need a grand graph runtime. I needed one fact: an unverified result must not open the next edge.

“Isn’t This Getting Needlessly Complicated?”

As soon as I proposed applying Graph Engineering, the AI produced Coordinators, Workers, Auditors, contract versions, durable ledgers, dashboards, and full re-audits. Each piece sounded plausible. But if the cure for a 16-hour token sink began by making the management system even larger, it could become the same failure again.

My first graph had only a work session and an audit session. The work session finishes one reviewable result and stops. The audit session sees the frozen result and original requirement, not the atmosphere of the earlier conversation or the worker’s self-evaluation. If a fix is needed, the same result gets one revision. If the direction is wrong, it returns to planning instead of adding more patches. Direct downstream work stays closed until the audit ends.

The objections helped plug holes in that small design. The worker cannot set a conveniently favorable audit standard. If the Runtime or candidate changes, the old verdict is discarded. And the Gate is not used for every typo or bug with one decisive test. It is reserved for points where a wrong result could contaminate several downstream tasks.

I put that much into a Skill named custom-graph-engineering-gate. I built no separate database or dashboard. The Skill freezes one result and restricts the next action according to PASS, CHANGES_REQUESTED, REPLAN_REQUIRED, or NOT_VERIFIABLE. Codex Skills and subagents were enough.²³

The First Gate Actually Stopped the Next Job

Creating the Skill was not evidence that it worked. I froze an authorization-transition contract that several features would depend on and sent it to an audit session without the earlier conversation.

The first verdict was CHANGES_REQUESTED, not PASS. The contract did not say clearly enough when authority was evaluated, where delegated authority came from and how it was revoked, or whether user, authority, and target company belonged to the same scope. If that result had spread, I would later have had to tear up the whole authorization model.

The work session revised that result once. Only after the same audit session checked it again did it return PASS. No downstream implementation or deployment opened in the meantime. That was the effect I wanted: the next edge really closed at the point where the worker had said “good enough.”

One authorization contract does not prove that this prevents another 16-hour UI failure. Human judgment about a visual direction is harder to gate, and an auditor using the same model and Working Tree can share the same false assumptions. A Skill is not an enforcement engine either.

Still, this time I did not install a giant framework because I heard a new term. I kept only the graph my failure needed and made it small enough to test again on the next long job.

So I do not have to discover sixteen hours later that the whole thing was wrong.

References

1. LangChain, “3 Years of Graph Engineering with LangGraph”, on the recent term Graph Engineering and designing Agent workflows as explicit graphs. ↩

2. OpenAI, “Build skills”, on storing reusable workflows as Codex Skills. ↩

3. OpenAI, “Subagents”, on a main Agent creating a separate subagent and collecting its result. ↩