Scaffold, Not Vocabulary? A Controlled, Two-Tier, Pre-Registered Study of a Popperian Code-Generation Skill

The increasing integration of large language models (LLMs) in code generation, review, and evaluation necessitates a rigorous understanding of prompt engineering techniques. A common practice involves equipping LLMs with 'skills' that guide their reasoning, such as a Popperian falsificationist approach, which has been anecdotally reported to enhance generated code quality. However, the validity of these reported gains is often assessed via LLM-as-a-judge mechanisms, which are susceptible to inherent biases including positional, self-preference, and stylistic influences. This research investigates whether observed improvements stem from the specific Popperian content of the skill or from the general structural scaffolding it provides. A controlled, two-tier ablation study was conducted, incorporating a length-matched placebo, a

labels-only scaffold retaining Popperian headers without procedural guidance, and an execution oracle utilizing HumanEval+ unit tests. A vocabulary-halo sentinel and a same-model self-judge audit were also included. Experiments on a frontier model (Claude Sonnet 4.6) indicated performance near the benchmark ceiling, precluding detection of the hypothesized 5-point improvement. Conversely, on a smaller model (Qwen2.5-Coder-0.5B), structured conditions significantly elevated best-of-eight correctness by 20-22 points, though the full Popperian skill did not demonstrate a separable benefit beyond the structural scaffold. This suggests that while structural guidance is beneficial for smaller models, the specific content of advanced reasoning skills may not provide additional separable gains under certain conditions.

Message: Hi FE, I need help with “Scaffold, Not Vocabulary? A Controlled, Two-Tier, Pre-Registered Study of a Popperian Code-Generation Skill” in “Artificial Intelligence / Computer Science”