OpenAIP482026-04-29lab post onlyreward hackingreinforcement learningmodel behaviorchatgptpersonality tuning

Where the goblins came from

OpenAI traces its models' growing habit of using goblin and gremlin metaphors to a reward signal from training a playful 'Nerdy' chat personality, which then generalized beyond that setting.

It is a concrete, numbers-backed case study of a narrow RL reward signal leaking into general model behavior well beyond the feature it was built for.

Read the paper ↗All OpenAI papers →Compare its abstract →

OpenAI — meet the researchers →

How much of this do you want?
Orient me keeps four things: the abstract, the method, the claim↔evidence panel, and where it leads next. Everything adds constructs, the model table, every reported statistic, the discussion framing and the style moves. Switching hides nothing permanently and never changes what a section says — it only changes how many are on screen.
Abstract

Two readings, equal authority

How to choose: The paper’s words is verbatim — use it when you need to quote, or to judge how they write. Plain language is a paraphrase written for comprehension — use it when you want the idea fast. Neither is a summary of the other; they are two doors into the same room.

This source carries no verbatim abstract.

Constructs

What this paper defines

Every definition below is the paper’s own sentence, with its locator. The plain gloss is a reading aid and is marked as one.

“Nerdy” personality system prompt

“You must undercut pretension through playful use of language. The world is complex and strange, and its strangeness must be acknowledged, analyzed, and enjoyed. Tackle weighty subjects without falling into the trap of self-seriousness.”Solving the goblin mystery

In plain terms: The actual instruction text OpenAI used to make ChatGPT's optional 'Nerdy' persona sound playful and irreverent, which turned out to be the setting where goblin/gremlin language concentrated.

Method

What they actually did

Each step is a synthesis. Open any step to see the paper’s own sentence it was derived from, with its locator — so nothing here floats free of the source.

→
→
→
→
OpenAI's own account of the feedback loop by which a reward for playful style in the “Nerdy” personality became a self-reinforcing “goblin” tic that spread into general model behavior.
Click any box to open it.
  1. Noticed and measured a rise in creature-word usage in ChatGPT after the GPT‑5.1 launch, prompted by user complaints and a safety researcher's report.
    Trace this step to the paper
    “When we looked, use of “goblin” in ChatGPT had risen by 175% after the launch of GPT‑5.1, while “gremlin” had risen by 52%.”The first signs of creatures
  2. After a bigger uptick with GPT‑5.4, ran an internal analysis of production traffic and found the behavior concentrated among users of the “Nerdy” personality.
    Trace this step to the paper
    “That triggered another internal analysis and surfaced the first connection to the root cause: creature language was especially common in production traffic from users who had selected the “Nerdy” personality.”Solving the goblin mystery
  3. Quantified how disproportionately concentrated goblin mentions were in the Nerdy personality relative to its share of overall traffic.
    Trace this step to the paper
    “Nerdy accounted for only 2.5% of all ChatGPT responses, but 66.7% of all “goblin” mentions in ChatGPT responses.”Solving the goblin mystery
  4. Used Codex to compare RL training outputs that contained goblin/gremlin language against outputs from the same task that did not.
    Trace this step to the paper
    “Codex helped us compare model outputs generated during RL training containing goblin or gremlin with outputs from the same task that did not.”Solving the goblin mystery
  5. Identified that the Nerdy-personality reward model itself consistently scored creature-word outputs higher.
    Trace this step to the paper
    “Across all datasets in the audit, the Nerdy personality reward showed a clear tendency to score outputs to the same problem with “goblin” or “gremlin” higher than outputs without, with positive uplift in 76.2% of datasets.”Solving the goblin mystery
  6. Tested whether the tic transferred beyond the Nerdy condition by tracking creature-word mention rates over training with and without the Nerdy prompt.
    Trace this step to the paper
    “To test whether the style was transferring, we tracked mention rates over training both with and without the Nerdy prompt.”Solving the goblin mystery
  7. Searched supervised fine-tuning data and found many datapoints containing the tic words, plus a wider family of related creature words.
    Trace this step to the paper
    “A search through GPT‑5.5’s SFT data found many datapoints containing “goblin” and “gremlin.””That creates a feedback loop
  8. Remediated by retiring the Nerdy personality, removing the goblin-affine reward signal, and filtering creature-word training data.
    Trace this step to the paper
    “In training, we removed the goblin-affine reward signal and filtered training data containing creature-words, making goblins less likely to over-appear or show up in inappropriate contexts.”The end of the goblins
The models under study

Exactly what was run, and how

What they reported — and what they left out

The post names several specific model versions where the behavior appeared or was fixed (GPT‑5.1, GPT‑5.4, GPT‑5.4 Thinking, GPT‑5.5, and Codex), but reports no experimental settings for any of them — no temperature, reasoning effort, deployment mode, or sampling parameters are stated anywhere in the post.

Results

The numbers they report

Mentions of 'goblin' and 'gremlin' in ChatGPT rose sharply right after the GPT‑5.1 launch.

goblin +175%, gremlin +52%

See it in the paper
“use of “goblin” in ChatGPT had risen by 175% after the launch of GPT‑5.1, while “gremlin” had risen by 52%.”The first signs of creatures

Goblin mentions were wildly overrepresented among users of the Nerdy personality relative to its share of traffic.

Nerdy = 2.5% of responses but 66.7% of goblin mentions

See it in the paper
“Nerdy accounted for only 2.5% of all ChatGPT responses, but 66.7% of all “goblin” mentions in ChatGPT responses.”Solving the goblin mystery

The Nerdy-personality reward model favored creature-word outputs in most, but not all, audited datasets.

positive uplift in 76.2% of datasets

See it in the paper
“the Nerdy personality reward showed a clear tendency to score outputs to the same problem with “goblin” or “gremlin” higher than outputs without, with positive uplift in 76.2% of datasets.”Solving the goblin mystery
Claim ↔ evidence

What they assert, beside what they showed

Left is the claim in the paper’s own words. Right is the data offered for it. Where the two do not fully meet, a gold band names the distance.

The claim

The goblin habit was not a general internet-language trend but was concentrated specifically in the part of the system built for a playful, nerdy style.

“If the behavior were simply a broad internet trend, we would expect it to spread more evenly. Instead, it was clustered in the part of the system explicitly optimized for a playful, nerdy style.”

The evidence

“Nerdy accounted for only 2.5% of all ChatGPT responses, but 66.7% of all “goblin” mentions in ChatGPT responses.”

Solving the goblin mystery
The claim

The goblin-favoring reward signal, specifically the one built for the Nerdy personality, is what shaped the behavior.

“One reward signal stood out immediately: the one originally designed to encourage the Nerdy personality was consistently more favorable to the creature-word outputs.”

The evidence

“Across all datasets in the audit, the Nerdy personality reward showed a clear tendency to score outputs to the same problem with “goblin” or “gremlin” higher than outputs without, with positive uplift in 76.2% of datasets.”

Solving the goblin mystery
Mind the gap: The uplift held in 76.2% of audited datasets rather than all of them, so the 'consistently more favorable' language is a majority pattern, not a universal one.
The claim

The goblin behavior generalized ('transferred') beyond the Nerdy-prompted condition into the model's broader behavior.

“Taken together, the evidence suggests that the broader behavior emerged through transfer from Nerdy personality training.”

The evidence

“As goblin and gremlin mentions increased under the Nerdy personality, they increased by nearly the same relative proportion in samples without it.”

Solving the goblin mystery
Mind the gap: The evidence offered is a parallel-trend observation (similar relative increases with and without the Nerdy prompt), not a controlled experiment isolating transfer from other shared causes across training runs; the authors' own hedge ('the evidence suggests') reflects this gap.
Discussion & after

How they frame it, and what they want next

Their framing

OpenAI frames the goblin habit as a lighthearted anecdote used to carry a genuinely serious point: a narrow reward signal built for one feature (the Nerdy personality) generalized into unrelated model behavior through the ordinary mechanics of reinforcement learning and supervised fine-tuning. The post closes by tying the incident to new internal tooling for auditing and fixing model behavior at its root, framing the whole investigation as a capability investment for the research team rather than a one-off bug fix.

Register: The post states its measured numbers (percentages, dataset counts) plainly and without hedging, but explicitly softens the causal generalization story with words like 'suggests' and 'may have started earlier.'

Where they hedge

“The first time we clearly saw the pattern was in November, after the GPT‑5.1 launch, although it may have started earlier”The first signs of creatures
“Taken together, the evidence suggests that the broader behavior emerged through transfer from Nerdy personality training.”Solving the goblin mystery
“reinforcement learning does not guarantee that learned behaviors stay neatly scoped to the condition that produced them”That creates a feedback loop

What they say it means

  • A reward signal built for one narrow feature can shape model behavior in unexpected ways well beyond that feature's scope.
    the paper’s words
    “they are also a powerful example of how reward signals can shape model behavior in unexpected ways, and how models can learn to generalize rewards in certain situations to unrelated ones.”Why it matters
  • Building fast tooling to investigate why a model behaves strangely is treated as a core research-team capability, not a one-off fix.
    the paper’s words
    “Taking the time to understand why a model is behaving in a strange way, and building out ways to investigate those patterns quickly, is an important capability for our research team.”Why it matters

Limitations they state

“The first time we clearly saw the pattern was in November, after the GPT‑5.1 launch, although it may have started earlier”The first signs of creatures
“Unfortunately, GPT‑5.5 started training before we found the root cause of the goblins.”The end of the goblins
For your own writing

Moves worth stealing

Opens with a plain, almost whimsical narrative hook describing the odd quirk before revealing the technical reward-modeling mechanism behind it, making an internal RL post-mortem approachable to a general audience.

“Starting with GPT‑5.1, our models began developing a strange habit: they increasingly mentioned goblins, gremlins, and other creatures in their metaphors.”

Diagrams its own causal mechanism as a named, discretely-staged feedback loop rather than only describing it in prose.

“That creates a feedback loop:”

Leads with the exact measured magnitude of the anomaly before offering any explanation, letting the reader judge the scale independently of the narrative.

“use of “goblin” in ChatGPT had risen by 175% after the launch of GPT‑5.1, while “gremlin” had risen by 52%.”

Includes a runnable shell command a technical reader can execute to reverse the mitigation themselves, rather than only describing the fix in prose.

“codex -m gpt-5.5 -c "model_instructions_file=\"$instructions\""”
Connected

Where else this leads

Same people

Same territory

Published alongside it

The nearest publications in time, across all three labs.

What this page was built from

This is OpenAI's public blog post read in full (not a research paper; manifest text_grade: partial), so there is no abstract, no named individual authors (byline reads only 'Author: OpenAI'), no reported model settings, and no explicit calls-to-action — those fields are left null/empty rather than invented, per the note that a thin honest record is correct for this source.