While the current batch of frontier LLMs has many feeling the AGI, the LLM performance contour still contains glaring, unhumanlike discontinuities. LLMs can be superhuman at some complex tasks, but wildly deficient in adjacent simple tasks.
Interpreting screenshots is one of those areas.
LLMs have made huge strides translating a screenshot into a working HTML and CSS implementation. But they still make visual mistakes so simple and obvious to a human that they leave you scratching your head.
I figure the root cause is one of two things. Either:
- Screenshot-to-HTML tasks aren't represented well in training corpora, or
- The tradeoffs of training for pixel-level image understanding are too great to pursue without new model architectures
Either way, the first step to improving is quantification. So here's a small contribution: FontBench.
FontBench provides LLMs with a cropped image of text against a white background and tests whether the model can identify six attributes:
- Font family (e.g. Arial)
- Category (serif, non-serif, mono, handwriting, other)
- Weight (thin, regular, bold, black)
- Modifier (regular, italic, underline, strikethrough, small-caps)
- Letter spacing (tight, normal, loose)
- Line height (tight, normal, loose)
The weight, modifier, letter spacing, and line height are broad categories that a professional designer would be able to distinguish. The specific fonts were sampled from a list of 50 popular fonts on the web. This list is biased by Google Fonts availability because I don't own a license to all the famous fonts of design history. The full V1 release has 1,824 images across 50 fonts.
The benchmark task is to produce those six field values when given an image rendering of the font. The images look like these:
Here is the Pareto frontier of accuracy and cost across a handful of frontier models. For more, see the full interactive results.
Three things stand out:
- GPT Astra is miles ahead of its nearest competitor.
- But GPT Astra still isn't great: only 40.1% accuracy.
- Font family prediction is the hardest task. If we ignore that property and just ask about category, weight, etc, then Astra's accuracy would double to 83.2%.
The dataset is available in this repo. I'd love any fixes or improvements. And if anyone wants to pitch in funding to run this on more LLMs, I'd gladly recognize your generosity on the benchmark page!
My hope is that the availability of more design-oriented benchmarks will help drive LLMs toward better performance for design comprehension tasks.