OpenAIP172026-07-29lab post onlybenchmarkingevaluation-methodologyagentscontext-managementarc-agi

How enabling two settings tripled our scores on the ARC-AGI-3 benchmark

Enabling retained reasoning and compaction in OpenAI's Responses API nearly tripled GPT-5.6 Sol's ARC-AGI-3 score while cutting output tokens sixfold.

It shows benchmark scores can be driven by harness and API configuration rather than model capability, so evaluators must match real deployment settings.

Read the paper ↗All OpenAI papers →Compare its abstract →

Ilan Bigio · Ted Sanders — meet the researchers →

How much of this do you want?
Orient me keeps four things: the abstract, the method, the claim↔evidence panel, and where it leads next. Everything adds constructs, the model table, every reported statistic, the discussion framing and the style moves. Switching hides nothing permanently and never changes what a section says — it only changes how many are on screen.
Abstract

Two readings, equal authority

How to choose: The paper’s words is verbatim — use it when you need to quote, or to judge how they write. Plain language is a paraphrase written for comprehension — use it when you want the idea fast. Neither is a summary of the other; they are two doors into the same room.

This source carries no verbatim abstract.

Constructs

What this paper defines

Every definition below is the paper’s own sentence, with its locator. The plain gloss is a reading aid and is marked as one.

ARC-AGI-3

“ARC-AGI-3 is a benchmark designed to measure how well AI agents learn and reason. Agents explore unfamiliar 2D games and infer how they work without explicit instructions.”ARC-AGI-3

In plain terms: A benchmark of 2D puzzle games used to test whether an AI agent can learn how an unfamiliar game works purely by playing it.

Relative Human Action Efficiency (RHAE)

“Scores measure Relative Human Action Efficiency ( RHAE ⁠ (opens in a new window) ) — a metric comparing model performance to a human baseline.”ARC-AGI-3

In plain terms: The benchmark's score compares how efficiently the model solves puzzles against how efficiently a human tester does.

Rolling truncation

“The ARC-AGI-3 harness addresses context limits with rolling truncation. When the conversation context exceeds 175,000 characters, the oldest messages are discarded.”Agents do best when they remember what they’ve done

In plain terms: A way of handling a full context window by simply deleting the oldest messages once a size limit is hit.

Retained reasoning

“Our models are trained to think with private reasoning messages before they output replies or tool calls. These private thinking messages are retained as part of the conversation history.”Agents do best when they remember what they’ve done

In plain terms: Keeping the model's own private step-by-step thinking in the conversation history instead of discarding it after each turn.

Compaction

“The next improvement came from replacing rolling truncation with compaction ⁠ (opens in a new window) , another setting in the Responses API.”Agents do best when they remember what they’ve done

In plain terms: A Responses API setting that manages a full context window by summarizing/condensing it rather than deleting the oldest parts outright.

Method

What they actually did

Each step is a synthesis. Open any step to see the paper’s own sentence it was derived from, with its locator — so nothing here floats free of the source.

→
→
Adding retained reasoning and then compaction to the harness moved GPT-5.6 Sol's ARC-AGI-3 public-set score from 13.3% to 38.3%.
Click any box to open it.
  1. OpenAI observed that GPT-5.6 Sol scored very low on ARC-AGI-3 despite strong performance on other hard tasks.
    Trace this step to the paper
    “GPT‑5.6 Sol has solved longstanding open problems in mathematics like the cycle double cover conjecture ⁠ (opens in a new window) and beaten games like Pokémon FireRed. But on ARC-AGI-3, a benchmark of 2D puzzle games, GPT‑5.6 Sol scored just 7.8%, and GPT‑5.5 could barely play the games at all, scoring a paltry 0.4%.”intro (before first heading)
  2. They examined transcripts of the model's actual attempts at the puzzles, following up on ARC's own published analysis of the low scores.
    Trace this step to the paper
    “Inspired by ARC’s analysis of GPT‑5.5’s shortcomings ⁠ (opens in a new window) , we examined some of the GPT‑5.6 Sol’s attempts. Like ARC, we saw that the model didn’t appear too bright. It dwelled a long time on each action and struggled to make progress.”intro (before first heading)
  3. They identified that the official harness discarded all of the model's private reasoning after every action.
    Trace this step to the paper
    “First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking.”intro (before first heading)
  4. They identified that the harness also used a rolling truncation window that dropped older actions from the visible history.
    Trace this step to the paper
    “Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew.”intro (before first heading)
  5. They reimplemented the ARC-AGI-3 harness on OpenAI's Responses API to match how the model is trained and deployed in production, enabling retained reasoning via the previous response ID.
    Trace this step to the paper
    “To better match our production setup, we implemented the ARC-AGI-3 harness with our Responses API ⁠ (opens in a new window) . Our API makes it easy to manage context: for GPT‑5.6, passing the previous response ID automatically retains reasoning across tool calls and turns.”Agents do best when they remember what they’ve done
  6. With reasoning retained, they observed the model spent less time thinking per action and learned more coherently over time.
    Trace this step to the paper
    “With reasoning retained, we noticed two big changes. First, GPT‑5.6 Sol spent less time thinking before each action, because it no longer had to interpret the game from scratch every turn. Second, when it was able to remember its past thoughts, GPT‑5.6 Sol was much better at learning over time and employing coherent strategies.”Agents do best when they remember what they’ve done
  7. They replaced rolling truncation with the compaction setting and re-measured performance.
    Trace this step to the paper
    “When we enabled compaction on ARC-AGI-3, GPT‑5.6 Sol was better able to preserve what it had learned about each game across longer runs, and achieved a higher score with fewer output tokens.”Agents do best when they remember what they’ve done
  8. They compared final scores between the official harness and their Responses API harness with both settings enabled.
    Trace this step to the paper
    “With the official harness, GPT‑5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction, it scored 38.3%.”ARC-AGI-3
The models under study

Exactly what was run, and how

ModelDeveloperTempEffort / reasoningDeploymentOther settings
GPT-5.6 Sol (max)OpenAInot reportedmaxAPIResponses API with retained reasoning (via previous response ID) and compaction enabled instead of rolling truncation; harness uses a 175,000-token context limit.
GPT-5.5OpenAInot reportednot reportedunstated—
Source for GPT-5.6 Sol (max) settings
“Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens.”Agents do best when they remember what they’ve done
Source for GPT-5.5 settings
“GPT‑5.5 could barely play the games at all, scoring a paltry 0.4%.”intro (before first heading)

What they reported — and what they left out

The post names the model (GPT-5.6 Sol, including a 'max' reasoning variant) and the API/harness settings used (Responses API, retained reasoning, compaction, a 175,000-token context limit) but never states temperature, sampling parameters, or system prompt content.

Results

The numbers they report

GPT-5.6 Sol scored very low on the official ARC-AGI-3 harness despite strong performance elsewhere.

7.8%

See it in the paper
“GPT‑5.6 Sol scored just 7.8%, and GPT‑5.5 could barely play the games at all, scoring a paltry 0.4%.”intro (before first heading)

GPT-5.5 performed even worse than GPT-5.6 Sol on the official harness.

0.4%

See it in the paper
“GPT‑5.5 could barely play the games at all, scoring a paltry 0.4%.”intro (before first heading)

With the official harness, GPT-5.6 Sol scored 13.3% on the public task set.

13.3%

See it in the paper
“With the official harness, GPT‑5.6 Sol scored 13.3% on the ARC-AGI-3 public set.”ARC-AGI-3

With retained reasoning and compaction enabled, the score nearly tripled.

38.3%

See it in the paper
“With retained reasoning and compaction, it scored 38.3%.”ARC-AGI-3

OpenAI estimates the average human tester's score on the same benchmark.

48% (estimated)

See it in the paper
“we estimate the average human tester scored 48%.”ARC-AGI-3

The official harness truncates context once it exceeds a character threshold.

175,000 characters

See it in the paper
“When the conversation context exceeds 175,000 characters, the oldest messages are discarded.”Agents do best when they remember what they’ve done

OpenAI's own reimplementation uses a token-based limit rather than a character-based one, but the two end up similar in practice.

175,000 tokens

See it in the paper
“our implementation uses a limit of 175,000 tokens instead of characters, but this ends up being quite similar, as the vast majority of text is action grids which are tokenized at a 1:1 ratio by our tokenizer.”Agents do best when they remember what they’ve done

Combining both settings roughly tripled the score for the 'max' variant while cutting output tokens sixfold.

~3x score, 6x fewer output tokens

See it in the paper
“Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens.”Agents do best when they remember what they’ve done

On the public ARC-AGI-3 leaderboard no frontier model clears the first level, but OpenAI's harness lets GPT-5.6 Sol clear all six.

0 of 6 levels (leaderboard) vs. 6 of 6 levels (OpenAI harness)

See it in the paper
“no frontier model solves any level beyond the first. With our harness, GPT‑5.6 Sol solves all six.”intro (image caption)

Turning on the two settings tripled scores and cut output tokens sixfold on the public task set, as stated in the summary framing.

3x scores, 6x fewer output tokens

See it in the paper
“we discovered that turning on two API settings we use in ChatGPT and Codex—retained reasoning and compaction—tripled scores and cut output tokens by 6x on the public task set.”intro (before first heading)
Claim ↔ evidence

What they assert, beside what they showed

Left is the claim in the paper’s own words. Right is the data offered for it. Where the two do not fully meet, a gold band names the distance.

The claim

Turning on retained reasoning and compaction tripled ARC-AGI-3 scores and cut output tokens sixfold on the public task set.

“we discovered that turning on two API settings we use in ChatGPT and Codex—retained reasoning and compaction—tripled scores and cut output tokens by 6x on the public task set.”

The evidence

“With the official harness, GPT‑5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction, it scored 38.3%.”

ARC-AGI-3
Mind the gap: 13.3% to 38.3% is roughly a 2.9x increase, close to but not literally 'tripled'; the 6x token-reduction figure is stated without the underlying token counts.
The claim

The model's apparent confusion on ARC-AGI-3 was mostly a harness artifact, not a limit of the model itself.

“But as we looked deeper, we discovered much of the model’s confusion was not inherent to the model itself, but due to settings in the harness.”

The evidence

“First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking.”

intro (before first heading)
Mind the gap: The evidence is a behavioral observation about discarded reasoning and truncation, not a controlled ablation that isolates harness effects from model capability while holding everything else fixed.
The claim

Remembering its past thoughts made GPT-5.6 Sol much better at learning over time and using coherent strategies.

“when it was able to remember its past thoughts, GPT‑5.6 Sol was much better at learning over time and employing coherent strategies.”

The evidence

“With reasoning retained, we noticed two big changes. First, GPT‑5.6 Sol spent less time thinking before each action, because it no longer had to interpret the game from scratch every turn.”

Agents do best when they remember what they’ve done
Mind the gap: "Much better at learning" and "coherent strategies" are qualitative descriptions here; no separate ablation score isolates retained reasoning alone from the combined 13.3%→38.3% figure.
The claim

This is not an isolated case: evals generally fail to measure models in isolation from harness and prompting choices.

“We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting.”

The evidence

“This isn’t the first time we’ve been surprised by low scores on a public benchmark and then discovered that the eval runner was using a generic harness that dropped reasoning messages.”

Conclusion and recommendations
Mind the gap: The generalization rests on a single named prior instance mentioned without detail, citation, or count; no survey of other benchmarks is presented.
The claim

Compaction let the model preserve what it learned across longer runs and achieve a higher score with fewer output tokens.

“When we enabled compaction on ARC-AGI-3, GPT‑5.6 Sol was better able to preserve what it had learned about each game across longer runs, and achieved a higher score with fewer output tokens.”

The evidence

“Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens.”

Agents do best when they remember what they’ve done
Mind the gap: This combined figure conflates retained reasoning and compaction; no compaction-only score is reported that would attribute the gains specifically to compaction rather than to retained reasoning.
Discussion & after

How they frame it, and what they want next

Their framing

The authors frame the finding as a methodological lesson about evaluation rather than a critique of ARC-AGI-3, explicitly thanking ARC for the benchmark and for the analysis that prompted their own investigation. They present their fix as aligning the benchmark harness with real production settings (as used in ChatGPT and Codex) rather than as tuning the harness to win the benchmark, and they generalize the episode into standing recommendations for anyone building or comparing agent evals.

Register: The writing is direct and confident about the mechanism (discarded reasoning and rolling truncation explain the low scores), while hedging the numeric claims with words like 'estimate' and 'roughly' and framing the result as a generalizable recommendation rather than a definitive final benchmark score.

Where they hedge

“we estimate the average human tester scored 48%.”ARC-AGI-3
“Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens.”Agents do best when they remember what they’ve done
“our implementation uses a limit of 175,000 tokens instead of characters, but this ends up being quite similar”Agents do best when they remember what they’ve done

What they say it means

  • Benchmark scores can be misleading if the harness settings diverge from how a model is actually deployed in production.
    the paper’s words
    “Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting.”intro (before first heading)
  • Developers seeking maximum performance should mirror the settings OpenAI itself uses in its own products.
    the paper’s words
    “If you’re an API developer trying to maximize performance, we recommend using the same settings that we deploy in our own products:”Conclusion and recommendations

What they call for next

  • Use the Responses API instead of the legacy Chat Completions API.
    the paper’s words
    “Use our Responses API, not our legacy Chat Completions API”Conclusion and recommendations
  • Retain the model's reasoning across turns.
    the paper’s words
    “Retain reasoning”Conclusion and recommendations
  • Use compaction rather than rolling truncation to manage context.
    the paper’s words
    “Use compaction”Conclusion and recommendations
  • When comparing models, rely on evaluations that use these production-matching settings.
    the paper’s words
    “And if you’re comparing models, we recommend relying on evals that use the settings above, which best match real-world use in ChatGPT and Codex.”Conclusion and recommendations
For your own writing

Moves worth stealing

Opens with the authors' own puzzlement before revealing the mundane cause, building narrative tension typical of an engineering post-mortem.

“When we first saw GPT‑5.6 Sol’s low scores on the ARC-AGI-3 ⁠ (opens in a new window) benchmark, we were puzzled.”

Uses a rhetorical question as a section transition instead of a plain topic sentence.

“Were 2D puzzle games unusually difficult for our models? Or was something else going on?”

Credits the external benchmark's creators generously rather than treating the low score as a rival's failure.

“We are grateful to ARC for their years of creative work on AGI evaluation, and for their analysis that inspired us to take a closer look here.”

Closes with short, numbered, actionable recommendations rather than only a narrative conclusion.

“Use our Responses API, not our legacy Chat Completions API”
Connected

Where else this leads

Published alongside it

The nearest publications in time, across all three labs.

What this page was built from

This is a company blog post, not a formal paper; the manifest grades its extracted text 'partial', but the substantive prose (all headings and body paragraphs) appears complete, with only site navigation and footer boilerplate surrounding it.