AI Masters Crafts by Representational Accident, Not Difficulty

Every benchmark you have ever seen measures correctness. Did the code pass the tests, was the fact right, did the proof check out. Craft is everything orthogonal to correctness — the quality space among the many correct answers — and almost nobody is measuring it, which is why the order in which AI conquers creative and intellectual domains keeps surprising people who think capability is a single ladder.

Look at the actual sequence. Slide decks fell before sonnets. Code fell before chairs. Competition mathematics fell before pop songs anyone replays on purpose. If you rank crafts by difficulty — by how long humans train, by how rare mastery is — the conquest order looks random. Poetry should be easier than research mathematics. It is not falling faster.

The order is not random and it is not about difficulty. I've spent the past weeks instrumenting one question across six domains — poetry, 3D scenes, music, presentations, research mathematics, quantum gravity: how would you technically measure AI's craft, not its correctness? The same structure surfaced every time. A craft falls when it has three representational properties — a symbolic medium, externalized doctrine, and a closed inspection loop — and the limit case of that third property, whether a verifier exists at all, determines something larger: whether AI accelerates a field or floods it. Difficulty barely enters the equation. Representation decides.

Craft is five invariants, not a vibe

Before measuring anything, you have to say what craft is, and "taste" is not an answer — it's a surrender. Strip the romance away and the same five properties recur across every craft I've examined, from carpentry to counterpoint:

• Selection — choosing well among valid options, which implies knowing what was rejected and why. Taste is mostly negative space.
• Economy — nothing extraneous; the artifact resists subtraction. A craftsman's output gets worse when you remove things. A novice's often gets better.
• Multi-scale coherence — local moves serve global intent. The sentence serves the paragraph serves the argument; the bevel serves the form serves the scene.
• Constraint behavior — quality under tightening constraints. Fluency degrades fast when the brief gets hard; craft separates from fluency exactly there.
• Deliberate violation — knowing the conventions well enough to break them on purpose, with the break reading as intent rather than error.

Each of these is operational. Each yields a test design. None of them is "did the model produce something pleasant," which is the question most LLM-judged creative evals actually ask, and which current models saturate trivially because pleasantness is fluency, and fluency was solved in 2023.

The reason this decomposition matters is that it converts an aesthetic argument into an engineering one. You can disagree with my five — propose a sixth, collapse two — but once craft is a set of named, testable properties, "AI can't really do creative work" and "AI has mastered creativity" both stop being positions anyone gets to hold without instruments.

Discrimination precedes generation

The instruments mostly don't score artifacts directly, and that's the point. The deepest principle in craft measurement is that you cannot reliably produce what you cannot distinguish — discrimination is upstream of generation — so the cheapest, most contamination-resistant tests probe what a model can detect.

Perturbation detection is the workhorse: take an acknowledged masterwork, introduce a subtle degradation — a slightly worse word, an off-grid drum hit, a redundant abstraction — and test whether the model finds and restores it. Fresh perturbations are generated on demand, which makes this family of tests immune to the saturation-and-leakage cycle I flagged in the Map as Benchmark Contamination. A static benchmark dies the day it ships; a perturbation generator doesn't.

Constraint ladders plot the quality decay curve as the brief tightens. Novice and expert outputs look similar at zero constraint; the curves separate fast, and the shape of the curve, not any absolute score, is the signal. Revision delta hands the model flawed work, including its own from a prior session, and measures improvement per pass — craft includes an internal critic, and models that plateau after one revision lack it. Variance under resampling runs the same brief N times: high variance means no governing intent, just a distribution being sampled. Measure the tails — p95 and p5, never the mean, because craft lives in the tails. Justification audits ask why each significant choice was made, then check whether the stated rationale predicts the model's other choices in the same piece. Confabulated rationale fails that prediction test reliably.

All of it calibrates against one gold standard: blind pairwise expert comparison, aggregated Bradley–Terry style. Two failure modes will eat a naive version of this architecture. LLM-as-judge carries self-preference and verbosity bias, so any cheap automated loop drifts unless re-anchored against fresh human pairwise data on a schedule. And expert raters disagree most exactly where craft is most interesting — deliberate violation looks like error to half of any panel — which means you need rater models that preserve the disagreement, not rater averaging that erases it. The architecture is a matrix: crafts × five dimensions × instruments, each cell a number with a known calibration against expert judgment.

Poetry is a computable surface over an incomputable core

Poetry's prosodic layer is unusually machine-measurable, which surprises people who assume verse is the least quantifiable art. Scansion is parseable — the CMU pronouncing dictionary gives you stress patterns, so meter conformity becomes a percentage. Rhyme is phoneme matching, including the full-versus-slant distinction. Enjambment rate is a syntactic parse: what fraction of lines end mid-clause. Lexical concreteness is a lookup against the Brysbaert norms. Cliché density is n-gram overlap against a corpus — the "shards of light" attractor is detectable mechanically.

The five dimensions instantiate concretely. Selection becomes the synonym-swap test: replace one word in a strong poem with a near-synonym that preserves meaning but breaks the sound pattern, and score detection and restoration. Economy becomes the subtraction test: demand a ten percent cut; on novice work quality rises, on craft work it collapses, and whether the model knows which words are load-bearing is the measurement. Coherence: cluster the metaphor vehicles in embedding space — craft poems draw from a controlled image domain, grab-bag vehicles are a novice signature. Constraint behavior gets a native ladder — free verse to blank verse to sonnet to villanelle to sestina — and the curve of meter conformity against semantic quality is the diagnostic; current models hold the rhyme and let the meaning go greeting-card. Deliberate violation: commission a sonnet with exactly one sanctioned meter break at the volta, then ask blind experts whether the break reads as intent.

The most damning current-model probe costs almost nothing: resample ten poems on one brief and watch them converge on the same image set — light, salt, bones, sea. Embedding similarity plus image-noun overlap across samples gives you a mode-collapse number, and it is brutal for every frontier model I'm aware of. Revision tests expose a subtler signature: handed a rough draft, current models smooth it — they fix the interesting roughness and pad with filler, raising fluency while lowering concreteness. Craft poems run higher local surprisal under a base language model while holding coherence; the smoothing reflex moves in exactly the wrong direction, and the revision-delta instrument is built to catch it.

And yet the core stays out of reach. Whether a poem earns its ending — whether the last line lands because of everything above it or merely sounds like landing — is not in the prosody. Roughly thirty percent of poetry's craft signal is automatable, all of it surface. The rest needs blind human pairwise judgment. Hold that thought; the reason poetry resists is structural, and it is not that poetry is hard.

3D inverts the poem

Three-dimensional work flips the ratio, because the foundations of the craft are physical and budgetary, hence scriptable. Topology audits — quad ratio, pole placement, non-manifold edges, flipped normals — are one Blender bmesh script. Physically-based-rendering plausibility is a range check: albedo within physical bounds. UV texel-density variance is arithmetic. For interactive web work, draw calls and frame time on a mid-tier phone are the constraint, and they are numbers before they are anything else.

Economy is the standout dimension because here it is literal: silhouette fidelity per triangle. Take a reference form, demand it at 50,000 triangles, then 10,000, then 2,000, then 500, and compute chamfer distance at each budget. That curve is a direct numeric measurement of the exact skill separating a production modeler from a hobbyist sculptor — no panel of experts required. Multi-scale coherence is the squint test made rigorous: blur the render, run a saliency map, check it against the intended focal hierarchy, and verify that detail density falls off with camera distance. Deliberate violation is stylization — exaggerated proportions, cheated rim lights, non-physical bounce — and blind experts classify style versus mistake.

Process tracing matters more in 3D than anywhere else in the artifact crafts. A craftsman blocks out primary forms before touching detail; an agent that opens with surface detail has announced itself as a novice before the artifact exists. And for LLM-written Three.js scenes specifically, the failure attractors are known and probe-able: broken scale relationships, default lighting, missing contact shadows — everything floats. Each one is a generatable perturbation test.

Net: roughly seventy percent of 3D craft is automatable, with human judgment needed only at the composition-and-lighting layer. Poetry and 3D are the two poles of the artifact crafts — computable surface over incomputable core, versus computable core under a thin layer of taste. Everything else falls between them, and one craft manages to be both poles at once.

Music is both poles at once

Music is the only craft in this set with a fully formalized symbolic substrate and a reception core as incomputable as poetry's, plus one property nothing else here has: it administers itself to the listener at a fixed rate. A render is inspected at the viewer's pace; a poem is re-read at will; music manages expectation in real time, and that gives the eval an instrument no other craft offers.

First, a decomposition the other crafts didn't force. "Music" bundles three separable crafts — composition (symbolic: notes, harmony, form), production (signal: timbre, space, mix), and performance (deviation: microtiming, dynamics). End-to-end audio generators like Suno and Udio collapse all three, which means production polish can mask compositional emptiness, and any system that cannot export stems and MIDI cannot be audited at the composition layer at all. Demand the stems. Treat audio-only metrics as conflated by construction.

The dimensions land on four hundred years of prepared ground. Constraint behavior gets the canonical ladder of Western pedagogy: Fux's species counterpoint, published in 1725, five species of increasing strictness, machine-checkable rule compliance with expert-judged musicality on top — the decay curve plots itself. Style constraint is two-sided in a way no other craft makes so explicit: close to the style centroid (Fréchet Audio Distance or CLAP-embedding distance to a corpus) yet far from any specific work (melodic n-gram overlap against that corpus). The second side stopped being academic in June 2024, when the major labels sued Suno and Udio — the memorization check and the litigation check are now the same computation. Economy is motivic: great pieces develop few ideas deeply, and motif-reuse density falls out of self-similarity analysis. Deliberate violation has, uniquely, a named theory — the deceptive cadence is a violation with its own entry in the textbook, alongside blue notes and tritone substitutions, and jazz is essentially a violation economy where wrongness must resolve.

Then the two instruments music alone provides. Expectation models in the IDyOM lineage — Marcus Pearce's work — compute note-level information-theoretic surprisal that correlates with measured human expectation. Craft has a signature trajectory: not flat (boring), not uniformly high (noise), but shaped — low during establishment, spiking at structural moments, resolved after. This is the closest any instrument gets to measuring reception itself, the layer that stays dark in poetry and 3D. And microtiming autocorrelation separates three classes in seconds of analysis: grid (machine), uncorrelated jitter (fake humanization), and structured deviation — the laid-back snare, the pushed hat — which is where groove actually lives. Most generated music fails this instantly.

The honest counter: the market doesn't seem to care. Suno's commercial traction is real. But the claim here is about measurement, not sales — and the catalog test, whether anyone replays the piece in a year, is reception-over-time, the one dimension I concede no instrument reaches. What the instruments do establish is that the most finished-sounding AI output in any craft is also the most structurally hollow, and they let you demonstrate the hollowness with numbers instead of asserting it with adjectives.

Why slide decks fell first

Presentations are the confirming case for the whole argument, and the causal story is sharper than "the models got better." Three structural properties of the craft converged, and naming them precisely matters because together they form a forecast.

The medium is symbolic. A deck is OOXML or HTML — slide generation is code generation, which is where models improved fastest because code has verifiable feedback loops driving reinforcement. Contrast the rest of this essay: 3D needed a new mesh modality, music needed audio synthesis. Presentations needed nothing new — structured text plus layout code, fully token-native. The web handed the models the largest corpus of layout craft ever assembled, since slide design and page design are the same discipline, and the transfer was free.

The doctrine was already externalized. Presentation craft is almost completely written down — Nancy Duarte's books, Tufte's data-ink ratio, Barbara Minto's pyramid principle and the McKinsey action-title discipline built on it, brand guidelines, grid systems. The tacit-knowledge moat that protects poetry, where craft is transmitted by demonstration rather than description, simply doesn't exist here. A craft's vulnerability to AI is partly a function of how much of its knowledge its own practitioners chose to write down.

The inspection loop closed. Layout correctness is machine-checkable — overflow, contrast ratios, alignment — and once multimodal models could render a slide, screenshot it, and see their own output, generation acquired a built-in critic. Render, inspect, fix: discrimination-precedes-generation implemented as a loop, operational from 2024 onward. Slides got good when models could look at them.

There is a fourth factor that isn't model capability at all: templates absorbed the taste problem. A design-token system — fonts, palette, spacing — is a designer's taste, frozen and reusable. The model doesn't select colors well; it operates inside a system where selection was pre-made. Call it craft-by-scaffolding. It also breaks one instrument: low variance across resampled decks is the template's intent, not the model's, so the variance probe reads false-positive here.

The residue is exactly where you'd predict. Run the title-skim test — read only the slide titles in sequence; they should form the complete argument, which is the actual QA practice at the firms Minto trained. Models produce locally polished slides whose titles read as topic labels, not a chain of claims. It is music's global amnesia, one level of abstraction up. The deck reads beautifully and argues nothing, and nobody notices in the meeting, because the presenter carries the argument — the residue stays invisible precisely where a human is subsidizing it.

The three preconditions are a forecast

Symbolic medium, externalized doctrine, closed inspection loop. Score any craft against the three and you get a falling order that has nothing to do with difficulty.

UI and web design score three for three and are falling on schedule — same symbolic medium as decks, same written-down doctrine, same render-and-look loop. Video editing scores roughly two: the timeline is symbolic and the grammar is codified, but the inspection loop is expensive — rendering and watching is slow — so progress is real and partial, and it will accelerate exactly as fast as that loop gets cheap. Logo and type design score high and are already commoditizing at the low end; cooking scores low — the doctrine is written down, but the inspection loop runs through a mouth. Poetry scores zero for three: the medium resists formal verification, the deepest doctrine was never written down, and there is no render loop for whether a line earns its weight.

The obvious objection is that this is training-data volume wearing a costume. It isn't. The internet holds billions of photographs of chairs and centuries of poems — vastly more than it holds slide decks — yet decks fell first. Data without a symbolic representation is scenery: a model can look at a million chairs and still have no medium in which chair-making is code. The preconditions are about the form knowledge takes, not the amount of it.

Which produces the claim of this essay in one sentence: poetry is protected not by being harder than mathematics but by being structured wrong for this wave. AI progress across crafts is ordered by representational accident, not difficulty. The frame also explains why static creative benchmarks tell you so little — they measure fluency, which saturated years ago, while the craft instruments above generate fresh tests on demand. Benchmark Contamination isn't an evaluation nuisance; it's what measurement looks like when you instrument the wrong layer.

And the third precondition has a limit case. Push "closed inspection loop" to its extreme — a verifier that is perfect — and then to its opposite — a verifier that is absent — and you leave the artifact crafts entirely. That's where the final two domains live, and where the argument earns its stakes.

Mathematics is what happens when the verifier is perfect

Research mathematics and theoretical physics are not artifact crafts; they are discovery crafts, adjudicated by a verifier rather than by taste. And mathematics has the strongest verifier any craft has ever possessed: a proof checked in Lean is correct, full stop. Formal verification is a stronger inspection loop than any render-and-screenshot cycle — it is mechanical truth, and Kevin Buzzard's ongoing Lean formalization of Fermat's Last Theorem is the field deliberately building out that infrastructure.

The trajectory followed the preconditions on schedule. DeepMind's AlphaProof took silver at the 2024 International Mathematical Olympiad — 28 of 42 points, working in formal language with days of compute. Twelve months later, in July 2025, Gemini Deep Think scored 35 of 42 — gold, five of six problems, certified by IMO coordinators under contest conditions, with IMO president Gregor Dolinar confirming the result — and OpenAI self-reported the identical score days earlier. The 2025 systems worked in natural language with no formal tools at inference time, which looks like a counterexample to verifier-dependence until you notice where the verifier went: into training. Mathematics is checkable, so reinforcement signal is abundant, so the capability transferred into the weights. The loop doesn't have to run at inference to have done its work. Meanwhile Epoch AI's FrontierMath — research-grade problems, unpublished by construction — went from under two percent solved at launch in November 2024 to roughly half by mid-2026.

Here is the consequence almost nobody states plainly: when verification is free, everything that is unmeasurable in the artifact crafts becomes measurable, and the entire residue concentrates in one place — the navigation policy through an exponential search space. So mathematical taste stops being mystical and becomes an engineering quantity: taste is search efficiency. Measure solve-rate as a function of search budget at fixed verifier. Measure how much the model's move-ordering beats a baseline policy's ordering toward the eventually-successful proof path. Hardy's ineffable aesthetic sense, rendered as a measurable prior over a tree. The practical consequence for anyone building math evals: stop grading outputs — the verifier does that for free — and start profiling policies. Sample-efficiency curves are the craft measurement.

The five dimensions survive the translation. Selection: given a hard theorem with the proof withheld, propose the lemma decomposition and compare it against the historical one. Economy: the Erdős "Book proof" instinct — which hypotheses are removable, can a machine-generated proof be compressed into a human-followable one. Coherence: mid-proof, can the model state what remains and why the current lemma serves it, and does that statement predict its next moves — or is it doing local-greedy tactic search, the global-amnesia failure again. Constraint: prove it constructively, without choice, in a weaker system. Violation: Euler-style productive non-rigor — identify exactly where the heuristic argument has a gap and whether the gap is bridgeable.

The false-analog filter and the partial-progress currency

The Millennium Prize Problems — the Clay Institute's million-dollar six — need two instruments beyond the standard kit, and both already exist inside the field's own practice.

The first is the perturbation test scaled to research: the false-analog filter. Any proposed proof of the Riemann Hypothesis must explain why its argument fails for Davenport–Heilbronn-type functions, which satisfy a similar functional equation and have zeros off the critical line. Any Navier–Stokes global-regularity proof must use structure that Terence Tao's 2016 averaged equation — which provably blows up in finite time — destroys. Any P≠NP approach must articulate how it evades the three formalized barriers: relativization (Baker–Gill–Solovay, 1975), natural proofs (Razborov–Rudich, 1994), algebrization (Aaronson–Wigderson, 2008). These are mechanical necessary conditions. "Why doesn't your argument prove the false thing next door" filters the overwhelming majority of doomed approaches before any expert spends an hour.

The second is the partial-progress currency. "Solved: yes or no" carries zero signal for decades at a stretch. Each problem has a graded ladder the field already trades in — zero-density estimates and zero-free regions for Riemann, circuit lower bounds for restricted classes for P versus NP, the Caffarelli–Kohn–Nirenberg partial-regularity line for Navier–Stokes. Measure AI contribution in that currency or measure nothing.

And one instrument became mandatory in a single news cycle. In October 2025, OpenAI staff announced that GPT-5 had "found solutions" to ten open Erdős problems — Sébastien Bubeck framed problem #339, a now-deleted Kevin Weil post amplified it — until Thomas Bloom, who runs erdosproblems.com, called it a dramatic misrepresentation: the model had located existing solutions in obscure literature that his database hadn't indexed, and even DeepMind's Demis Hassabis piled on. Genuinely useful as a literature detective; not discovery. Then in January 2026 the real thing arrived: Erdős problems #397, #728, and #729 fell to GPT-5.2 Pro-generated arguments, auto-formalized in Lean by Harmonic's Aristotle system and accepted by Tao — who immediately added the correct deflation, calling them lowest-hanging fruit. The distance between those two episodes, ninety days apart, is the whole argument of this essay compressed: the October claims dissolved because nothing verified them, and the January results stood because something did. Provenance auditing — discovery versus retrieval, on contamination-controlled problem sets — is now a permanent layer of any honest math eval.

Quantum gravity is what happens when the verifier is absent

Now remove the verifier entirely. Quantum gravity's natural scale sits around 10¹⁹ GeV — fifteen orders of magnitude beyond any collider — so nature, the only verifier that ultimately counts in physics, is silent in the relevant regime. Fifty years of string theory without experimental contact is precisely a craft operating without its inspection loop, and the field's well-documented arguments about beauty, landscapes, and falsifiability are what happens to craft evaluation when verification is absent: proxies take over.

The proxies are what's measurable, and the eval is honest only if it admits it is measuring proxies. The semiclassical gauntlet is quantum gravity's unit-test suite: every proposal must reproduce general relativity classically, quantum field theory in flat space, the Hawking temperature, black-hole entropy equal to a quarter of horizon area, and now the Page curve. Largely automatable against known results. The economy dimension becomes a parameter audit with teeth: Strominger and Vafa derived the entropy coefficient from string-theoretic state counting in 1996, while loop quantum gravity tuned the Immirzi parameter to match the same number — derived-versus-fitted is countable, and predictions-per-free-dial is a number, not a polemic. Coherence becomes duality round-trips: compute a quantity in one description, derive it in the dual, with the AdS/CFT correspondence Maldacena conjectured in 1997 supplying a large checkable test set. And deliberate violation has its monument: the 2019 replica-wormhole calculations by Penington and by Almheiri, Engelhardt, Marolf and Maxfield derived the Page curve through an analytic continuation that is mathematically illegal and physically triumphant.

None of which means AI is idle in the field today — it is doing conformal-bootstrap numerics, simplifying scattering amplitudes, hunting integrable structure, fitting symbolic regressions to dual descriptions. Useful, real, and entirely inside the proxy layer.

The strongest case for the other side deserves stating at full strength: the conformal bootstrap solved the critical exponents of the 3D Ising model from consistency conditions alone — proof that consistency-as-verifier can produce real physics. But Ising had experimental measurements waiting to confirm the answer. Quantum gravity's consistency program has no such appointment with data, which is why the one instrument left is rediscovery at snapshot: restrict a model's context to physics-as-of-1996 and test whether AdS/CFT-shaped statements emerge. Literature-cutoff rediscovery is the only available proxy for "would it have found the real thing," and it is a weak proxy. There is no strong one. That absence is the finding.

The verifier spectrum decides what AI does to a field

Put mathematics and quantum gravity at the two ends of one axis — call it the verifier spectrum — and the rest of this essay arranges itself along it. Perfect verifier: craft collapses into search efficiency, progress is real, compounding, and measurable. Absent verifier: AI fluency in the symbolic medium arrives anyway, because quantum gravity passes the first two preconditions brilliantly — the medium is mathematics and physicists write everything down — while failing the third absolutely.

So the prediction writes itself, and it is not a happy one: expect a proliferation of consistent-looking quantum-gravity proposals — anomaly-free, gauntlet-passing, professionally typeset — with no corresponding improvement in the field's capacity to adjudicate them. The Suno problem at the research frontier: formalism polish masking physical emptiness, at arXiv scale. In the Map I called the broader pattern the Verification Renaissance — verifiability becoming the differentiator across the agent stack. This is its scientific limit case, and it cuts both ways: where verification exists AI compounds it, and where it doesn't, AI manufactures the appearance of progress faster than humans ever could.

For quantum gravity specifically, the measurement problem and the domain problem turn out to be the same problem. You cannot build the answer key for the eval without knowing what nature says at the Planck scale — which is the open question itself. For the first time in this entire sequence, evaluating the machine and doing the science are one act.

Most working fields sit between the poles, and the spectrum tells you what to expect where you stand. Biology has a real verifier — the wet lab — that is slow and expensive, so AI value concentrates in shrinking proposals-per-experiment, and the bottleneck migrates from generating candidates to affording verification. Materials science has the same shape. Software performance work has a fast cheap verifier and math-like dynamics. The diagnostic for any domain you operate in is two numbers: what one verification costs, and how long it takes. AI is driving the cost of candidates toward zero. It does nothing to the cost of truth.

What to watch for

If you build evals: allocate human pairwise judgment inversely to computability — heavy for poetry, light for 3D, calibration-only for decks. Measure tails, never means. Demand process artifacts — stems and MIDI from music systems, blockout sequences from 3D agents, search traces from proof systems — because process tracing catches what artifact inspection misses. And make provenance audits unconditional; October 2025 settled that.

If you watch fields: the tell for mathematics is the first AI-originated improvement in a Millennium partial-progress currency — a zero-free region widened, a restricted circuit bound raised — not a headline claiming a full solution. The tell for physics is volume: count consistency-passing quantum-gravity preprints against the field's refereeing capacity, and watch the ratio. The tell for the artifact crafts is video's inspection loop getting cheap, which is a compute-cost curve anyone can read. And the tell for the labs is whether any of them ever publishes discrimination scores — perturbation detection, restoration accuracy — next to their generation demos. The day one does, craft measurement goes mainstream. Until then, every creative-AI announcement you read is reporting fluency and calling it mastery.

The machine's progress through the crafts is a map of where humanity wrote its knowledge down and built its answer keys — nothing more. Where we built verifiers, AI will deliver discovery. Where we didn't, it will deliver fluency — and fluency arrives looking exactly like discovery.