ARCANA: A Benchmark for Abstraction and Analogy 

Dataset generation: Target-concept-target loop

ARCANA comprises 200 tasks spanning 35 distinct concepts, with each task targeting a unique analogical mapping. ARCANA tasks are grounded in diverse real-world objects, concepts, and physical phenomena, which we refer to as targets. Each target can be abstracted into a small set of higher-level concepts that govern the grid transformation. Once such concepts are extracted, they can be slipped and re-grounded to other targets.

For example, consider ring1 above, the concepts can be summarized by three—move, shelter, and 3D—where 3D rings move and partially occlude (shelter) the column pattern. The concepts can be can be slipped and re-grounded to other targets, such as: 3D candles burning on a cake and burn a certain length (move) then expose the cake behind (shelter) them, which is shown below.

Dataset generation: Grounding targets into grids

Concept attention. Bridging targets and discrete ARC grids requires concept attention: emphasizing task-relevant concepts of a target while suppressing concepts that are secondary or irrelevant to the core rule.

For example, in the melt target, ice melts only upon contact with hot water (red cells); after melting, both the ice and the hot-water become cold water (blue cells). The final cold-water volume (the number of grid cells) equals the sum of the initial ice volume and hot-water volume. We therefore focus on the melting condition and this discrete conservation rule, while ignoring physical nuances such as density differences across phases and temperatures.

Concept Expansion. Beyond concept attention, we sometimes augment a target with additional concepts (e.g., time) to fully specify its dynamics. We refer to this as concept expansion.

For instance, in the charge target, green cells (analogous to charge or data) propagate along directed "wires'' from a source container at the red terminal to a sink container at the green terminal. At each time step, each wire transfers one cell if the sink has remaining capacity. Introducing an explicit temporal process (and treating it as effectively continuous) is necessary to correctly solve the test.

Dataset generation: generation principles

We construct the benchmark according to four principles that are intentionally simple and human-grounded:

(1) Human-familiar targets. The targets are common and recognizable to humans; given the target, most people who are familiar with it can solve the task accurately with little effort.

(2) Concept-faithful grounding. The ARC grids inherit the target's core concepts, and the grid-target analogy is intended to be explicit.

(3) Multiple targets, one rule. A task may admit multiple plausible analogical targets, but they must agree on the same underlying transformation rule, such as claw and chopsticks.

(4) Human-aligned correctness.  When multiple rules are consistent with the examples, we choose the one most natural to humans, best aligned with everyday concepts, and lowest in cognitive burden. 

Here we provide all categories and the detailed tasks in the table below. And we also provide more tasks in this section.

Experiment setup

Since solving an ARCANA task often benefits from recognizing the intended real-world target, unfamiliarity with the target can increase failure rates. To reduce such idiosyncratic variance and obtain a more reliable estimate, each task is randomly assigned to at least six participants.

We recruit undergraduate participants via an online platform. In addition to final grid answers, we ask participants to (i) provide a brief one-sentence description of the inferred target, preferably in analogical terms, and (ii) rate the task's thinking and drawing difficulty on a five-level scale (very easy to very hard), enabling finer-grained analyses of human reasoning. To ensure data validity, participants must pass a tutorial and an attention check; submissions from participants who fail the attention check are excluded. After filtering, we obtain 1659 question-level responses from 80 participants.

You may also try our online platform with our personal information masked: ARCANACogTest 

Experimental results

ARCANA tasks may admit multiple plausible targets that nonetheless instantiate the same underlying rule (Principle~3), automated evaluation could be unreliable; we therefore manually evaluate every response. All submissions are categorized into five types: CC (answer Correct, analogy Correct), CW (answer Correct, analogy Wrong), WC (answer Wrong, analogy Correct), WW (answer Wrong, analogy Wrong), and WT (answer Wrong with Typo). WT captures minor typographical mistakes: the participant expresses the intended rule but leaves one or two erroneous cells/blocks that are typically irrelevant to the core transformation (e.g., forgetting to erase the original pattern after shifting it). If the mistakes affect the rule expression, we instead label the response as WC.

Experimental results are shown below.

The Pearson coefficient on the 2*2 table after excluding typo responses is computed as 0.391, indicating a moderate positive association between producing a valid analogical interpretation and producing a correct grid output. The fact that the coefficient does not exceed 0.6 (i.e., not a strong association) is also unsurprising: even after a participant identifies a plausible analogy, successfully solving the task can still require additional concept expansion and careful, error-free operations on the grid. Overall, these results are sufficient to conclude that analogy and answers are meaningfully related, and that analogy can serve as a useful aid for solving ARCANA tasks.

ARCANA is designed around human-centered priors, therefore human performance provides a natural reference point. We next analyze task-level human accuracy and subjective difficulty to assess whether ARCANA is human-aligned. We report two accuracy variants: counting typo as wrong versus as correct.

The figure shows that ARCANA exhibits a reasonable, non-trivial difficulty profile for humans: most tasks fall into intermediate-to-high accuracy ranges, and treating typo as correct shifts the distribution upward. Together, these results support the validity of ARCANA as a human-grounded benchmark.

There are more minor analysis not included in the paper, such as the subjective difficulty distribution and the time consumption (filtered 30s-30min). For subjective difficulty, participants separately rate thinking and drawing difficulty using a five-level scale (very easy, easy, medium, hard, very hard), which we map to scores 1-5. For time consumption, we filtered all time excluded from 30s-30min for validation, and the mean time consumption is about 5.8 min.

Experiment setup

We evaluate all ARCANA tasks on four representative models: GPT 5.2, Gemini 3 Pro, Claude Sonnet 4.5, and Grok 4.1 Fast. These models are required to directly output the final answer grid, which is motivated by the fact that some ARCANA rules are difficult to faithfully express as executable code, making code-based evaluation potentially restrictive (e.g., cat1 task shown below). Under this output format, LLMs may also produce minor typographical mistakes. Accordingly, all LLM responses are evaluated manually with the same criteria as in the human study, and results are reported under two conventions: counting typos as correct (+Typo) versus as wrong (-Typo).

We employ two prompting strategies: chain-of-thought and analogical prompting. Due to the substantial stochasticity in model outputs, we follow the common practice in ARC evaluations and report pass@3 (top-3). The average cost and tokens per task for each model are also reported.

Overall, LLM performance on ARCANA is below human performance in the table: the pass@3 upper bounds of LLMs remain lower than the pass@1 human average. Nevertheless, the strongest models still solve a non-trivial subset of tasks, suggesting that current LLMs possess meaningful (albeit limited) capability for ARCANA problem solving. We emphasize that these results are influenced by evaluation budget (cost per task) and by the inherent stochasticity of LLM outputs. 

Additionally, we provide category-wise experimental results here (which is in Appendix B), including human accuracy, GPT 5.2 and Gemini 3 Pro with their answer accuracy on language, vision-language, +typo, -typo,  and their analogy accuracy.

Experiment analysis

Language vs. vision-language prompting.  Generally, vision-language (VL) prompting yields lower performance than language-only prompting for most models (with one notable exception: GPT 5.2 improves under VL when typos are counted as correct). Importantly, however, these aggregate comparisons do not imply that VL is strictly worse. As shown in table, the Total column is higher than either prompting alone, indicating that the two prompting modalities succeed on different subsets of tasks. Category-level inspection also suggests that some concepts benefit substantially from visual grounding (Gemini on electric_clock category and moon_and_tide1 task, GPT on pendulum category). Together, these observations support our viewpoint that ARCANA is inherently visual in nature, yet not merely visual problems.

Conversely, we also observe categories where performance decreases after introducing vision-language prompting. Take Gemini 3 Pro and GPT 5.2 as examples, these decreases do not follow a consistent pattern, suggesting that the impact of visual inputs is model-dependent and intertwined with output stochasticity, rather than reflecting a uniform modality effect.

Correct analogies support LLMs solving ARCANA. Under analogical prompting, we collect and manually evaluate model-generated targets, and find that correct analogies tend to co-occur with correct answers, suggesting that analogy can materially support ARCANA task solving. Focusing on the two strongest models (GPT 5.2 and Gemini 3 Pro), the pulley category is particularly diagnostic: Gemini achieves 100% analogy correctness while GPT achieves 0%; correspondingly, Gemini reaches 67% answer accuracy in pulley whereas GPT remains at 0%. Overall, these results suggest that analogical reasoning is a meaningful bottleneck for ARCANA performance.

Also we list some examples here (which is in Appendix B). For moon_and_tide1, Gemini explicitly identifies "the lunar cycle and its effect on tides,'' whereas GPT's best response remains at a surface-level grid description ("Yellow controls a 2-row tide shift of the blue sea''). For newton_cradle1 and newton_cradle2, Gemini readily recognizes the intended physical concept, while GPT tends to produce less grounded interpretations ("Pipe insertion with opposite-side ejection''). As a result, Gemini could generate the answers of moon_and_tide1, newton_cradle1, and newton_cradle2 correctly. Despite these failures, GPT can occasionally produce high-quality analogies: for candy_crush1, it generates a correct match-3 interpretation ("swap, clear lines, gravity, cascade''), which help GPT generate the correct answer. Notably, this task also exhibits substantial variation in human analogy accuracy, plausibly reflecting differences in participant background (our participants are undergraduates in Chinese universities).

Category-wise performance and human-model gaps. Overall, human accuracy is relatively stable across categories, whereas LLM performance is more uneven. Humans outperform LLMs in most categories, but there are also categories in which some models match or exceed the human average (typically those dominated by precise, fine-grained manipulations, e.g., shelter_then_move). At the same time, several categories expose clear gaps where LLMs lag far behind humans (e.g., water_deep_concept and animal), indicating that substantial differences remain between current LLMs and human analogical reasoning.

The scale and breadth of modern LLMs suggest substantial potential for approaching human-like analogy. To narrow this performance gap and probe the limits of LLM analogical reasoning, we introduce a new reasoning framework, ConceptSIR.

Inspired from cognitive theories such as Copycat theory, we propose ConceptSIR, an LLM-driven analogical reasoning framework built around Concept Slippage, Introspection, and Reflection.

Concept Proposal and Slippage

Concept Slippage, a core mechanism in Copycat theory, describes how activating one concept can trigger related concepts and higher-level relational structures, thereby expanding the search over possible interpretations. This is particularly useful for LLMs, whose outputs can be overly context-dependent; increasing the diversity of candidate targets helps avoid getting trapped in a "local minima'' of concepts. Motivated by our human study (each ARCANA task admits a valid human-grounded target) and by the breadth of modern LLM pretraining, we use a concept proposal and slippage stage to elicit and diversify candidate analogical targets. 

Concretely, we prompt proposals from four human-prior dimensions (color, shape, dynamics, and holistic recognition) and then feed these candidates into a slippage step to activate a broader concept neighborhood. From a Bayesian perspective, we  hypothesize that the neuron activations in LLMs can be viewed as a form of concept slippage—potentially a shallower analogue of the corresponding process in the human brain. Therefore, in our framework, concept slippage is implemented as an additional prompting operation that leverages the combinational creativity of LLMs to activate a more diverse set of concepts encoded in their knowledge base.

Concept Execution

To reliably apply an inferred rule underlying all given demonstrations to the question grid, we represent the execution step as code rather than natural language. Natural-language rules are often ambiguous or internally inconsistent, making it difficult to ensure faithful, deterministic application to grids. Using code provides an explicit, verifiable realization of a candidate rule and is broadly consistent with prior ARC solvers.

Concretely, for each candidate target produced by the slippage stage, we prompt the LLM to synthesize a corresponding program as one possible instantiation of the target's rule, which is then executed and checked against the examples.

Introspection

Introspection iteratively aligns the model's analogical interpretation and its execution with human cognition. Many ARC tasks admit multiple rules consistent with the demonstrations, but humans typically favor one interpretation as the intended solution. For example, in a clock task, both "rotate the pointer on the right dial by 90 degree'' and ``rotate the hour hand by 90 degree'' may fit all examples, yet the latter is more natural because "hour hand" composes more directly with the concept of a "clock". 

After each code update, we run the program on the examples and questions, using introspection to identify abnormal behaviors and producing targeted correction instructions; once all examples are matched, we perform a final introspection pass on the question output to filter invalid or human-misaligned solutions.

Reflection

Inspired by genetic algorithms, we perform two levels of probabilistic mutations in parallel: code mutation and concept mutation.

At the code level, reflection repairs implementation and logic errors by comparing program outputs on the examples against ground truth, selecting the best-performing candidate as a reference (according to the prediction errors), and probabilistically mutating other candidates to explore alternative implementations.

At the concept level, mutation includes broader slippage (to diversify targets) and finer-grained concept augmentation, which enriches a target with relevant sub-concepts needed for faithful execution (e.g., adding a "pile-up" concept on top of "gravity" for bead_chain1).

Together, these mechanisms promote both diversity and implementability of candidate solutions.

Qualitative experimental results

Due to time and cost limitations, we only evaluate ConceptSIR on a small subset of ARCANA tasks, most of which cannot be solved by all the LLMs during baseline testing. In this setting, we observe that ConceptSIR enables LLMs to solve the chosen tasks, and further analyze the resulting reasoning trajectories.

In our experiments, we observe that Gemini 3 Pro performs best in the concept proposal and slippage stages, while GPT 5.2 excels in code-centric processes. Accordingly, in our ConceptSIR, we adopt Gemini 3 Pro for concept proposal and slippage, and GPT 5.2 for subsequent code-centric stages, including execution-oriented code generation and iterative refinement.

Concept slippage contributes to program generation. We study tasks where failure is caused primarily by selecting an incorrect analogical target. Here we take two tasks as examples, which are refraction2, bead_chain1, and candle1. Refraction2 is not solved by all LLM baselines, and bead_chain1 is solved once in 12 attempts by only GPT 5.2. In ConceptSIR, refraction2 is solved in 2 reflection steps, and among the three codes generated by Concept execution stage, two are correct to solve task bead_chain1 in 3 reflection steps.

As for task refraction2 in baseline testing, GPT 5.2 generates the targets ambiguously, like "light bouncing and slow down". In ConceptSIR, Gemini 3 Pro generate a more accurate target as "Light-ray / ray-tracing in a room with different materials", which further contributes to the correct program and output from GPT 5.2.

Furthermore, we analyze the reasoning trajectory on bead_chain1. We note that during baseline testing, GPT 5.2 extensively uses the word "flower", e.g., "Flowers below an orange knot are replanted into the brown garden beds", "Transplanting plants into planter boxes". Noting that the targets cannot be viewed as incorrect ones, they are still different from the targets from Gemini 3 Pro (e.g., "The task is analogous to 'beads sliding down strings'.") that are more similar to humans. In concept proposal, Gemini 3 Pro propose four different targets from color, shape, dynamics, holistic recognition, and generate three targets through concept slippage.

Introspection and reflection contribute to program modification. In our case studies,  we find that introspection and reflection play a critical role in refining candidate programs into correct and generalizable solutions. We focus on candle1 and bead_chain1 tasks. Noting that candle1 is also not solved by all LLM baselines, whereas human participants solve 5 out of 6 instances (counting typos as correct).

In a failure code of bead_chain1, the target concept of “beads falling into a container” is under-specified; reflection introduces an implicit discrete packing constraint (e.g., “bays behave like discrete parking slots”), which captures the pile-up mechanism required to implement the rule. Moreover, in a failure code of candle1, reflection reveals that the correct operation is to “burn down” the candle by a fixed number of cells, rather than merely shifting an orange segment downward, and concept is slipped from "lighthouse" to "candle and shadow".

Additionally, in the success code of candle1, we also observe the function of introspection. First, we found in step 5, GPT 5.2 generates a code that solves all of the examples, but fails on the question. The code deletes all of the candles, which is absolutely not correct. Accordingly, during the introspection, LLMs response indicates that: "Missing Objects": In the "example output" generated by the code, the candles are completely missing; "Missing Effects (Shadows)": The code output is almost entirely yellow, meaning it considers every empty pixel to be illuminated". Introspection effectively filter the invalid output, and finally the code is refined to generate the correct answer.

Our framework is more similar to an agent, and we aim to shed light on the effectiveness of our framework based on the resulting reasoning trajectories. We also seriously state that our framework leaves a lot space for improvement, for example, concept execution form could be changed from code to other more robust forms, additional explicit memory modules could be added in order to conclude the ``experience" during concept proposal (more effective prompting aspects), more aligned rule with human (more human-aligned introspection), human feedback for better alignment, and so on. Furthermore, during testing on ConceptSIR, we notice that the analogy accuracy of Gemini drops. We assume that during baseline testing, Gemini 3 Pro analyzes the grids in detail and notices the relations (e.g., multiple relationship) on grids, and associates it to more complicated concepts. This indicates the potential increase space for concept proposal and slippage.