Eye Tracking Based Cognitive Evaluation of Automatic Readability Assessment Methods
Faculty of Data and Decision Sciences, Technion – Israel Institute of Technology
Computational Linguistics, 2026 · Presented at EMNLP 2026
TL;DRReadability scores, from 1948 formulas to frontier LLMs, are poor predictors of reading ease measured with eye tracking. Psycholinguistic measures, especially surprisal, do better.
Abstract
Automatic methods for scoring text readability have been studied for over a century, and are widely used in research and in user-facing applications in many domains. Thus far, the development and evaluation of such methods have primarily relied on two types of offline human behavioral data, performance on reading comprehension tests and ratings of text readability levels. In this work, we instead focus on a fundamental and understudied aspect of readability, real-time reading ease, captured with online reading measures using eye tracking. We introduce a new cognitive evaluation framework for readability scoring methods that quantifies their ability to account for reading ease, while controlling for content variation across texts. Applying this evaluation to prominent traditional readability formulas, NLP-based methods, commercial systems used in education, and frontier LLMs suggests that they are all poor predictors of English reading ease in adults as compared to word properties commonly used in psycholinguistics for the prediction of reading times. This outcome holds across L1 and L2 speakers, different reading regimes, and textual units of different lengths. Our results reveal an important limitation of a wide range of methods for readability scoring, highlight the utility of real-time behavioral benchmarks for readability research, and call for new, cognitively driven readability scoring approaches that can better account for how humans experience texts in real time.
Why Eye Movements?
Readability methods are usually validated against comprehension tests or human difficulty labels. Eye movements give a cognitive signal of how easy a text is to read, in real time.
Content-Controlled Evaluation
Each paragraph has an original and a simplified version written by language teachers: same topic and content, only the wording changes.
Original86 words
Viennese-born Helena Hartlauer, 32, said she was not surprised at her city's top position. The municipality's social democratic government has a long tradition of investing in high-quality social housing, making Vienna almost uniquely affordable among major cities. "I live in a 100sq-meter turn-of-the-century apartment in a good area about 20 minutes' walk from the city center. But my rent is just €800 (£625) a month." An equivalent apartment in London would cost upwards of £2,000 and even more in New York, ranked 44th in the table.Simplified90 words
Helena Hartlauer, 32, is from Vienna. She said she was not surprised about her city's top position. For many years, Vienna's government has spent money on good social housing. This makes Vienna a cheap place to live compared to other big cities. "I live in a 100 square-meter apartment in a good area about 20 minutes' walk from the city center. But my rent is just €800 (£625) a month." A similar apartment in London costs over £2,000 and even more in New York, which came 44th in the study.
One reader per version, replayed at 4× speed with Scanpath Studio. Individual readers vary, so the evaluation averages about 100 readers per version.
Method
Across all text pairs: how well each method reflects reading ease. Δ = original − simplified.
Example
Illustrative numbers.
Readability Methods
20 readability methods, benchmarked against 9 psycholinguistic measures from reading research.
Traditional Formulas
- Flesch RE
- Dale-Chall
- Gunning Fog
- ARI
- Coleman-Liau
- Flesch-Kincaid
Modern NLP Methods
- CML2RI
- CAREC
- CARES
- SBERT
- PPL
- SLOR
LLMs
- Llama 3.3 70B
- GPT-4o
- GPT-5
- Gemini 2.0 Flash
- Gemini 2.5 Pro
- Claude Sonnet 4.0
Commercial Systems
- TextEvaluator
- Lexile
Psycholinguistic Measures
- Idea Density
- Integration Cost
- Embedding Depth
- PLL
- UID
- Entropy
- Word Length
- Word Frequency
- Surprisal
Results
Readability Methods
Traditional formulas, NLP models, commercial systems and LLMs, from 1948 to 2025, are weak and mostly not significant predictors of reading ease.
Psycholinguistic Measures
Psycholinguistic measures used to predict reading times do better. Surprisal is the best predictor overall.
Data
OneStop: about 640 readers (360 native and 278 non-native English speakers) reading 162 original and simplified paragraph pairs with eye tracking.
Robustness
Readers
Native & non-native English speakers
Reading Regimes
Reading for comprehension & information seeking
Text Units
Sentences & paragraphs
Measures
11 eye-tracking measures & reading speed
Takeaways
- A cognitive evaluation of readability methods.
- Content-controlled evaluation: the same text in two versions.
- Existing readability scores may not reflect reading ease.
- Future readability measures should reflect reading ease.
- Surprisal is a simple, strong baseline.
BibTeX
@article{grutekeklein2026eye,
title = {Eye Tracking Based Cognitive Evaluation of Automatic Readability Assessment Methods},
author = {Gruteke Klein, Keren and Frenkel, Shachar and Shubi, Omer and Berzak, Yevgeni},
journal = {Computational Linguistics},
year = {2026},
doi = {10.1162/COLI.a.649},
url = {https://doi.org/10.1162/COLI.a.649}
}