[🤖] AI Kept Agreeing with Me, So I Made Another AI Challenge It First

✨ GPT-5.6 Sol’s Summary

While asking what caused a 16-hour UI failure, I noticed the AI changing its conclusion with every objection I raised. This is the record of passing critiques between sessions and turning that process into a Skill with one fresh critic and at most two review rounds.

I Could Not Simply Trust the Answer Explaining a 16-Hour Failure

Yesterday, after a long discussion, I finished page-by-page specifications for a complete UI/UX overhaul and set a Codex Goal to build from them. The Goal ran for 16 hours. I opened the actual screen today with high expectations, and the result was simply garbage.

I had planned everything down to calendar and table views, yet it looked as if someone had merely piled descriptions and cards onto the old screen. Meanwhile, reports saying implementation complete, audit passed, and build passed had kept accumulating throughout the work. I demanded to know what on earth it had done for 16 hours and told it to analyze the cause of this mess first.

Along the way, I learned that the problem I had been fighting was starting to be called Graph Engineering. Breaking long work into small outcomes and verification stages, and separating the builder from the auditor so a wrong direction would not spread downstream, clearly seemed useful for this incident.

But before the methodology, another problem caught my eye. Whenever I proposed an alternative, the AI quickly said it was right. When I raised an objection, it then said that objection was right. Each answer sounded plausible, but the conclusion swayed toward whatever I had said most recently.

Even the answer explaining the failure was hard to trust as-is.

“Please, Criticize It”

So I passed one session’s answer to another session for critical review. I brought that critique back to the original session and made it respond, then showed the new answer to yet another session.

This was at least better than letting one session answer and praise itself. One side caught assumptions the other had treated as obvious, and surfaced operating costs and failure modes missing from the first answer. Conversely, when the critical session went too far, the original session sometimes pushed back with evidence.

But even here I had to keep reining it in. Whenever I pasted in criticism from another session, it too readily said, “That criticism is right.” Eventually I said this:

Please, criticize it.

I did not want two AIs to reach a friendly agreement. I wanted both the first answer and the later criticism to be doubted, and the evidence followed until the end. Several agents saying the same thing does not make it true, and attaching the name critic does not automatically make an agent more accurate.

I liked the basic process of exchanging answers across sessions and adopting the conclusion that survived at the end. The problem was that repeatedly copying and pasting the source, answer, and rebuttal between GPT and Codex was incredibly tedious.

I Cut the Grand Debate System Back Down

At first I imagined a “debate Skill” that could create up to three critics and run several rounds. I even considered paying for Claude Code so it could exchange arguments with Codex through the CLI.

The AI proposed separating a Coordinator, Worker, and Auditor and maintaining additional state. It was not wrong, but to me the work was becoming needlessly complicated again. The 16-hour Goal failed because it was an uncontrolled long-running task. Building another giant orchestration system to prevent that felt strange.

What I had in mind was much simpler.

Why not just open one fresh subagent and exchange reviews with it?

The main session sends its current conclusion to one critic. Without being pulled along by the atmosphere of the earlier conversation, the critic looks for counterexamples and hidden assumptions. The main session classifies each point as accepted, rebutted with evidence, or unresolved. If it revises the answer, it shows it to the same critic exactly once more. They check whether the problem was actually resolved, then stop.

A new issue gets a fresh critic; a revision of the same issue goes back to the same critic. If the first critique finds no important problem, there is no need to force a second round. Even with multiple critics, no conclusion is decided by majority vote.

That was enough to automate almost exactly what I had been doing by hand, while keeping the debate itself from becoming another huge project.

I Asked for a Skill That Would Not Hide Rebuttals

Once the direction was clear, I gave Codex concrete instructions to create a global Skill. The default would be one critic and at most two review rounds. The critic would be a fresh subagent that did not inherit the whole conversation. A revision of the same issue would go back to the critic that first reviewed it. The workflow would be read-only, touching no files, Git state, Runtime, or external state.

Most importantly, I required it not to adopt the critic’s answer automatically.

The main session has to handle every important criticism in one of three ways.

• Accept it and revise the conclusion.
• Rebut it with evidence.
• Leave it as an unresolved disagreement.

The final answer must disclose not only what changed because of the critique, but also which points were rejected and what uncertainty remains. I wanted to prevent it from quietly dropping inconvenient objections or dressing up a conclusion with “three agents agreed.”

At first I used the display name Custom - Debate, but later I had it changed to Custom - Deliberate to match the shared Skill naming rules. The directory and invocation name had been custom-deliberate from the beginning. It also better described the goal: not choosing a debate winner, but deliberating and trying to disprove a conclusion before fixing it in place.

I Immediately Used the Skill to Review Itself

I did not stop just because the Skill had been created. I immediately used $custom-deliberate to review whether the process itself was appropriate. Then I asked again whether applying those criticisms would actually improve performance.

That process made the Skill a little more sober. It now states that withholding prior conversation does not create a perfectly independent critic. The critic still shares the same model and higher-level rules, and it can be wrong in the same way depending on what evidence the main session includes in the packet. It also distinguishes merely imaginable possibilities from material objections that could actually change the conclusion. If there is no important objection, it ends early; if evidence is insufficient, it says so instead of forcing agreement.

The work I had been doing by copying answers between sessions was now inside the Skill. I do not trust the first answer, but I do not automatically trust the criticism either. I adopt only the conclusion that remains after rebuttal and re-review.

Now I Try to Break an Answer Once Before Accepting It

This Skill does not guarantee truth. If I really want to break biases shared by the same model family, attaching a different model such as Claude Code may work better. If I give the critic bad material, it may simply produce a more sophisticated wrong answer. Actual preferences and value judgments remain mine to make.

Still, before adding another 150,000-won subscription, I want to try this small method first.

I do not finalize a conclusion immediately. I ask one fresh critic to try to break it. I record which objections I accepted, why I rejected others, and what remains unresolved. I show the revised answer to the same critic only once more.

I had written before that whether it is Claude Code or Codex, it ultimately needs a harness. This time I applied that idea to the judgment process. Making AI finish work is important, but so is preventing it from agreeing with me blindly and locking in a conclusion too quickly.

From now on, when making a non-trivial decision with AI, instead of asking for one more answer, I will first ask what could be wrong with the answer I already have. If I end up doing the same copy-and-paste routine again, that will be the time to connect Claude Code or another model. For now, this is the minimum critical safeguard I wanted.