Pattern Bake-off, Round 1
Four image-text models compared on 3,753 public texture photos, plus every check run on the bake-off workspace so far. No IKEA product data is in this page.
The short answers
The models
Each model scored every pattern term on every image, twice: once with Stephen’s wording and background crop, once with plain wording and no crop. The plain setup isolates the model from the setup.
| Model | From | Family | Why it is in | Licence |
|---|---|---|---|---|
| SigLIP so400m | SigLIP | Stephen’s model; the baseline | Apache-2.0 | |
| SigLIP 2 so400m | SigLIP | Is the newer version better? | Apache-2.0 | |
| OpenCLIP ViT-H-14 | LAION community | OpenCLIP | Different family and training photos | MIT |
| Perception Encoder PE-Core-L-14 | Meta | Perception Encoder | A third, recent family | Apache-2.0 |
How well each model ranks the right images
Mean average precision over the 4 terms with clear public labels (paisley, checkered, dot, stripe). 1.0 would be perfect. Whiskers show the 95% interval from 1,000 resamples of the images.
Perception Encoder holds its score with either setup, so its lead comes from the model. SigLIP drops without Stephen’s setup, so part of its strength comes from his wording and crop. With only 4 checkable terms, the order could change on other terms.
Stephen’s two numbers, with his setup
Over the 383 images that carry a clearly mapped label, ranked over all 44 specific pattern terms, exactly as his script counts.
| Model | First choice right | Right answer in top 3 |
|---|---|---|
| SigLIP 2 | 308/383 · 80.4% | 359/383 · 93.7% |
| Perception Encoder | 305/383 · 79.6% | 356/383 · 93.0% |
| SigLIP (Stephen’s) | 303/383 · 79.1% | 325/383 · 84.9% |
| OpenCLIP | 295/383 · 77.0% | 356/383 · 93.0% |
Stephen reported 90%+ on 100 IKEA products that already carried pattern tags, scored against those tags. These figures use texture photos and human labels, so they are not directly comparable.
Per term, with Stephen’s setup (average precision)
| Model | paisley 83 true | checkered 81 true | dot 89 true | stripe 130 true |
|---|---|---|---|---|
| Perception Encoder | 0.984 | 0.838 | 0.758 | 0.515 |
| SigLIP (Stephen’s) | 0.992 | 0.729 | 0.714 | 0.483 |
| OpenCLIP | 0.979 | 0.730 | 0.553 | 0.483 |
| SigLIP 2 | 0.980 | 0.796 | 0.309 | 0.516 |
Out of 3,753 images. Bold marks the best model per term. Stripe is the hardest for every model.
Speed and cost per million images
| Model | Model time per image | Hours per million | Rough cloud cost per million |
|---|---|---|---|
| OpenCLIP | 16 ms | 4.6 h | ≈ $3.70 |
| Perception Encoder | 21 ms | 5.8 h | ≈ $4.70 |
| SigLIP (Stephen’s) | 35 ms | 9.8 h | ≈ $7.80 |
| SigLIP 2 | 35 ms | 9.8 h | ≈ $7.90 |
Measured on an RTX 4080 SUPER shared with other programs, on small texture images, so only the model time transfers to product photos. The cost assumes $0.80 per cloud GPU-hour, a stated guess to be refined on IKEA hardware.
What was tested
Checks that run automatically on every change
- passTaxonomy check. Term lists built from the spreadsheet match a hand count; every spreadsheet error is flagged (5 mislabelled pattern rows, 15 form rows).A silently wrong term list would make every score wrong.
- passScoring arithmetic. Every formula is checked on a small made-up example with answers worked out by hand.Proves the numbers are calculated correctly, not just present.
- passDry runs with a pretend model. Every image ends up as a result or an explained failure; broken images are set aside; empty input is handled.No image silently drops out of the counts.
- passSafety. Runs refuse final-exam images, IKEA runs refuse to save outside the IKEA folder, and internet attempts are caught.The safeguards are tested, not just written down.
- passStephen-comparison rehearsal. The comparison script runs on a stand-in copy of his script and never opens the decoy files beside it.Noz’s real run should work first time.
- passOdd cases, IKEA mode and review pages, on fake data. One image, duplicates, a term with no examples, labels containing “/”, bulk accept and reject, and decision files.These are where data tools usually break.
868 checks pass on the CPU-only gate. With the real data store, 982 pass.
Real runs on the graphics card
| Check | Result |
|---|---|
| All 8 runs complete (4 models × 2 setups) | pass 3,753 of 3,753 images processed in every run, no failures |
| Internet attempts during runs | pass 0 in all 8 runs; the deliberate test connection was caught |
| Final-exam images opened | pass 0; 7 practice images identical to exam images were set aside |
| Card memory | pass 3.2–5.2 GB peak; no out-of-memory events |
Reviews and safeguards
| Review | Outcome |
|---|---|
| IKEA image guard (blocks Claude from the IKEA folder) | pass 56 of 56 simulated cases; live tests blocked, including other agents |
| Spec review by four agents before building | applied 32 findings, 4 blocking, all folded into the spec |
| Security review of the workspace code | pass no blocking issues; two library advisories noted for the next slice |
| Product acceptance review | fixed 8 findings, including one IKEA safety gap, all fixed and re-checked |
Waiting for Noz
- on returnRun the Stephen comparisonOne command; proves the rebuild matches his code.
- on returnReview the public label mappingFor example, whether “grid” counts as “windowpane”.
- on returnApprove the term guide62 unofficial descriptions, plus 11 candidate findings for the taxonomy owners.
- before IKEA runIKEA tag file names and columns, and the firewall ruleThen the IKEA run, from Noz’s own terminal.