Today's AI scorers measure technique. Judging blind, frontier AI is beginning to see art.
There is one recognised way to name the best photographs: contests with a professional jury.
Not Instagram likes. Not even "panels of professionals". A photograph earns its standing only when truly accomplished photographers judge it.
This matters because of how today's AI photo-scorers were actually built. Every one of them learned from images labelled by ordinary people — amateurs, registered users, paid annotators — never by a professional photography jury.
| System | Who judges / annotates | Expert jury? |
|---|---|---|
| Top photography contests | Selected, recognised professionals — photographers, curators, editors, art professionals | YES |
| NIMA / AVA | Registered DPChallenge users / amateur photographers; ~200 ratings per image | NO |
| LAION-Aesthetic V1 | Human raters via Simulacra, asked simply how much they liked the image | NO |
| LAION-Aesthetic V2 | Mixed human-rating datasets, including AVA / DPChallenge | NO |
| Everypixel | Trained on large stock-photo sets; algorithmic technical-aesthetic scoring | NO |
| PickScore | Real users of Pick-a-Pic | NO |
| ImageReward | Annotators trained for image-preference labelling | NO |
| HPS v2 | Human annotators ranking generated images (~798k judgements) | NO |
| VisionReward | Human annotators giving multidimensional preference labels | NO |
Almost every automatic scorer — Google's NIMA included — learned from a crowd (the AVA set), not from masters.2 A crowd feels "nice." A world jury sees what the crowd cannot. Researchers have shown models trained this way simply don't carry over to professional photography.3
"Models trained on the crowd-sourced AVA dataset behave differently on professional photographs, and do not generalize well to them."
— paraphrased from Chambe et al., Deep learning for assessing the aesthetics of professional photographs, 2022The right way is to learn from real contests, judged at the highest level.
Our work builds on FOTOAWARD — an independent international non-profit photography contest, running since 2018 and judged in two jury rounds after an initial audience stage, whose defining feature is a jury of world-class experts.
Among the jury members are the finest photographers alive — repeat winners of the most prestigious awards in the world (World Press Photo, Wildlife Photographer of the Year, Bird Photographer of the Year, Trierenberg Super Circuit, Golden Turtle, GDT European, WPPI, etc.), National Geographic photographers, Nikon ambassadors and book authors, from nine countries on four continents. See the 2025 jury ↗
The competition — independent and non-commercial since 2018 — draws winners of Hasselblad Masters, Smithsonian, HIPA and Xposure. From it comes a rare, structured record of expert taste — eight years of graded verdicts, layer by layer.
What we are looking for are the patterns in the winning work that bring these top experts to agreement. Independent agreement among judges is a measurable, scientific thing — statisticians call it inter-rater reliability.1
FOTOAWARD 2025 — winners ↗
Our own tests point clearly in one direction: today, the best available judge of artistic worth is not these scoring services, but the leading frontier AI models. This is an illustration from our daily work, not a formal study. We took ten photographs that all reached the final of our contest — every one strong — and asked three of the world's leading AI models to score each on artistic worth, 0 to 100. Then we lined those scores up against how our jury actually voted, and against the technical scorer Everypixel.
About the jury index: a raw vote count understates the signal. In the spirit of Fleiss' kappa — the standard measure of inter-rater agreement — what matters is how statistically unlikely a given level of consensus is. Of 326 works, just one drew 8 of 11 votes; three drew six. Our index combines that rarity with how close the vote came to unanimous, on a 0–100 scale. It is deliberately conservative: 8 of 11 is near-miraculous agreement, but short of a unanimous 11, so it reads 85, not 100.
That quick illustration used a loose, generic instruction, and the models squeezed almost everything into a narrow band. It raised the real question: can a machine, judging blind, see what a world-class jury sees? We then tested it properly, on the full record. See the blind study ↓
Judging blind, a frontier model predicts each juror's picks better than a fellow juror does.
We gave one of the latest top frontier models (Model 1 — Anthropic's Claude Opus 5.5) every photograph the jury saw in two full semifinals and in all six finals, 2020–2025: 9,589 different photographs. It saw only the image: random file names, no titles, no names, no votes, no ratings, in mixed batches of 20, each batch scored by a fresh instance under a short instruction fixed in advance. Then we compared its scores with every juror's real ballot.
The test is simple. Take one juror. How well do their own picks get predicted by one other juror, by the AI, and by the whole rest of the jury combined? A score of 0.5 is a coin toss; 1.0 is perfect.
| Round | Photographs | Jurors | One fellow juror | AI, blind | Rest of the jury combined |
|---|---|---|---|---|---|
| Semifinal 2020 — everything the jury saw | 3,549 | 10 | 0.53 | 0.70 | 0.70 |
| Semifinal 2022 — everything the jury saw | 4,094 | 9 | 0.57 | 0.71 | 0.74 |
| Finals 2020–2025 — all finalists | 2,686 | 62 seats | 0.54 | 0.62 | 0.67 |
What it shows. In every round the AI predicts a juror's choices far better than another juror does: in the finals, for 59 of 62 juror seats. In the 2020 semifinal it matches the rest of the jury combined; in 2022 it comes close. In the final, where every work is already strong, the jury as a whole still leads. Asked to cast a ballot of the same size as a juror, the AI agrees with each juror about as much as jurors agree with each other (Cohen's kappa 0.08 against 0.09). The pattern is consistent: a machine comes closest where the field is wide, and a world-class jury still leads where the work is already outstanding. None of this replaces expert judgement — it is measured against it.
The open question: how should a machine be asked? We have compared several families of instructions on the same blind sets, from the simplest to the expert-informed, as well as different ways of combining independent judgements. The preliminary differences are real and intriguing, and they matter most at the very top of the field, where prizes are decided. Settling them properly takes many repeated blind passes over the whole archive, with more than one model.
It is not one model's quirk. A second top frontier model from another developer (Model 2) scored 640 finalists under the identical blind protocol and performed the same, within statistical error. The Spark test was repeated with the same model on Amazon Bedrock and reproduced within statistical error: AUC 0.79 with the original instruction (0.81 with a second one) on a 100-photograph pilot, against 0.85 for the original run on the same photographs. Scores repeat well: the same work scored twice by fresh instances, intraclass correlation 0.86.
None of this means AI can't help photographers today.
Because we study how AI reads photographs every day, we have built real tools around it — free, and already in use on our contest. Load a photograph and the system can analyse it, write a critique, suggest titles, compose a poem or a short story from it, turn it into a 19th-century engraving, or check whether it was AI-generated.
FotoMatcher — the right contest for any photograph, found by reading the photograph itself.
We have built a unique, continuously updated database covering close to every photography contest in the world. But the real work is not matching by theme or contest name: the AI studies the photograph itself and weighs how well it truly fits each of the hundreds of open contests — even when genres and categories can't tell you — plus many other factors.
It is a hard, separate project — and it is close to completion. fotomatcher.com ↗
