AuraGrade · public scorecard

Verified accuracy benchmark

No other PSA pre-grading tool publishes verified accuracy numbers — they all claim percentages with no public auditable test set. AuraGrade publishes ours.

Experimental · Heuristic v0Model: heuristic-v0.1.1·Snapshot: 2026-05-24·55 verified pairs

Headline result

Within-1 accuracy

47.3%

Top-1 accuracy

16.4%

ECE (calibration error)

0.306

Within-1 = predicted grade is within ±1 of the actual PSA grade. ECE near 0 = well-calibrated probabilities (when we say P(10)=0.7, the actual frequency in that bin is also ~0.7).

Important caveat: dataset is slab-skewed

Read this

Most pairs in the current benchmark are photos of cards already in PSA slabs. We scrape them this way because the PSA cert number is printed on the slab label, which lets us cross-verify the grade against PSA's API.

But the AuraGrade pipeline is designed for raw card photos. Slab photos introduce systematic artifacts (plastic-case glare, edge distortion, center-of-frame composition) that the model reads as defects. The Floor Rule drags slab predictions down — you can see this in the confusion matrix where almost every PSA 10 gets predicted as PSA 8 or 9.

Therefore: these numbers are the worst-case estimate of model performance. Real raw-card accuracy will be higher. We'll know how much higher when contributors via /contribute send raw photos + report their PSA results back.

All pairs

n = 55

Top-1

16.4%

Within-1

47.3%

ECE

0.306

Confusion matrix

true \ predicted	PSA 9	PSA 8
PSA 10	13	22
PSA 9	3	8
PSA 8	3	6

Slab photos only

n = 49

Top-1

14.3%

Within-1

44.9%

ECE

0.299

Confusion matrix

true \ predicted	PSA 9	PSA 8
PSA 10	12	21
PSA 9	2	7
PSA 8	2	5

Raw photos only

n = 0

No samples in this slice yet.

Methodology

· Pairs: 55 (card image, PSA-verified grade)

· Image source: public Reddit posts on r/PokemonTCG, r/PokeInvesting, r/PSAGrading

· Grade verification: Gemini Flash OCR of the PSA slab label → PSA public cert lookup API

· Model: heuristic-v0.1.1 — heuristic CV pipeline (Sobel centering + per-pillar scoring + Floor Rule aggregator + calibration shift)

· Metrics: top-1 (predicted modal == verified), within-1 (predicted within ±1 of verified), ECE (10-bin expected calibration error on P(verified band))

· Buckets: the model outputs PSA 10 / PSA 9 / PSA 8 distributions. Anything verified ≤8 is bucketed into PSA 8 for top-1 comparison.

· Not committed: we don't publish individual image URLs out of respect for the original posters. The per-row table below shows cert + grade only.

Per-pair results (heuristic)

Cert / card	Verified	Predicted	P(true)	Δ	Source
Charmander · XY	PSA 9	PSA 8	33%	-1	?
Bulbasaur · 2025 POKEMON MEG EN	PSA 10	PSA 8	23%	-2	?
Ditto · Delta Species	PSA 1	PSA 8	44%	+7	?
Lugia · Silver Tempest	PSA 10	PSA 9	27%	-1	?
Houndoom · SFA EN	PSA 9	PSA 9	44%	0	?
Charizard · Base Set	PSA 8	PSA 9	19%	+1	?
Shining Charizard · Neo Destiny	PSA 5	PSA 8	45%	+3	slab
Cynthia's Garchomp · DRI EN	PSA 10	PSA 8	20%	-2	slab
Bulbasaur	PSA 10	PSA 8	22%	-2	slab
Mega Gardevoir ex	PSA 10	PSA 9	26%	-1	slab
Leafeon ex · Prismatic Evolutions	PSA 9	PSA 8	34%	-1	slab
Gengar VMAX · Fusion Strike	PSA 10	PSA 8	22%	-2	slab
Greninja · SV5a JP	PSA 10	PSA 8	22%	-2	slab
Pikachu VMAX · Vivid Voltage	PSA 10	PSA 9	27%	-1	slab
Blaziken · World Champions Pack	PSA 10	PSA 8	23%	-2	slab
テツノカシラex · SV5M	PSA 10	PSA 9	27%	-1	slab
Feraligatr · Neo Premium File	PSA 10	PSA 9	27%	-1	slab
Salamence	PSA 8	PSA 8	43%	0	slab
Espeon-Holo · Town on No Map	PSA 10	PSA 8	21%	-2	slab
Venusaur · Bulbasaur Deck	PSA 9	PSA 8	33%	-1	slab
Team Rocket's Mewtwo ex · DRI EN	PSA 9	PSA 8	32%	-1	slab
Octillery	PSA 6	PSA 9	28%	+3	slab
Dark Tyranitar · Team Rocket Returns	PSA 10	PSA 9	28%	-1	slab
Dragonite · Game Boy Promo	PSA 9	PSA 8	34%	-1	slab
Lugia V · Silver Tempest	PSA 10	PSA 8	23%	-2	slab
Mew · Corocoro Comics Promo	PSA 5	PSA 8	46%	+3	slab
Meowth ex	PSA 10	PSA 8	23%	-2	slab
Umbreon VMAX · Brilliant Stars	PSA 10	PSA 8	23%	-2	slab
Charizard · Legendary Collection	PSA 3	PSA 8	47%	+5	slab
Umbreon ex · Prismatic Evolutions	PSA 10	PSA 9	27%	-1	slab
Dark Blastoise · Rocket	PSA 9	PSA 8	33%	-1	slab
Charizard · Japanese Basic	PSA 10	PSA 9	27%	-1	slab
Charizard · SV2a JP	PSA 10	PSA 8	22%	-2	slab
N's PP Up · ASC EN	PSA 10	PSA 8	23%	-2	slab
Shining Charizard · Neo 4	PSA 10	PSA 8	25%	-2	slab
RESHIRAM	PSA 10	PSA 8	23%	-2	slab
Charizard · Base Set	PSA 4	PSA 8	44%	+4	slab
Deoxys · Holon Phantoms	PSA 10	PSA 8	20%	-2	slab
Gyarados · Legendary Collection	PSA 10	PSA 9	25%	-1	slab
Latias & Latios · Sun & Moon Team Up	PSA 10	PSA 9	26%	-1	slab
Raichu · Burning Shadows	PSA 10	PSA 8	20%	-2	slab
Rosa's Encouragement	PSA 10	PSA 8	22%	-2	slab
Shining Mew · 2001 P.M. Japanese Promo	PSA 10	PSA 9	27%	-1	slab
Rayquaza GX · Sun & Moon	PSA 10	PSA 8	22%	-2	slab
Umbreon & Darkrai · SM Black Star	PSA 9	PSA 8	34%	-1	slab
Ancient Mew · 2000 Movie Promo	PSA 9	PSA 9	45%	0	slab
Eevee & Snorlax GX · SM Promo	PSA 9	PSA 9	44%	0	slab
Pikachu	PSA 10	PSA 9	27%	-1	slab
Greninja ex · TWM IT-Twilight Masquerade	PSA 10	PSA 8	23%	-2	slab
Espeon	PSA 9	PSA 8	33%	-1	slab
Mewtwo	PSA 10	PSA 9	26%	-1	slab
Charizard · Legendary Collection	PSA 1	PSA 9	29%	+8	slab
Vaporeon ex · PRE EN	PSA 10	PSA 8	23%	-2	slab
Diancie · Hidden Fates	PSA 10	PSA 8	26%	-2	slab
Pikachu · Ascended Heroes	PSA 10	PSA 8	22%	-2	slab

Phase 2 status: DINOv2 features unlock 3x accuracy

In Progress

You are currently viewing the Phase 2 SOTA Prototype data. We tested whether a foundation vision model (DINOv2-base, frozen) carries grading signal that hand-crafted features miss. Zero training, just k-NN classification over 768-dim CLS embeddings.

The prototype shows a massive jump in Top-1 accuracy (49% vs 16.4%). This validates that deep feature extraction is required to "see" through PSA slab cases and accurately judge raw card surfaces.

Next steps: Train pillar Dirichlet heads on top of DINOv2 features, swap into production /api/score pipeline, and redeploy.