Independent researchers report in a new arXiv benchmark that large language models handle a single instruction well but fall apart as soon as several constraints must hold together, according to the study’s authors Large Language Models Can Follow Instructions, But Not Many at Once (arXiv). The work introduces Constraint Saturation Evaluation (CSE), a test that asks models to satisfy many explicit constraints jointly rather than one at a time.
What the benchmark measured
CSE is built to remove the usual guesswork in judging model output. It procedurally generates tasks that vary the number of simultaneous constraints, k, and scores every constraint with a deterministic, rule-based verifier — no LLM judge is involved. Across 15 models, 36 constraint types, and 369,753 individual checks spanning k=1 to 12, the setup isolates exactly how performance changes as the load grows Constraint Saturation Evaluation (doi).
Where performance falls apart
The first and most striking result is asymmetric: models pass individual constraints at a steady, predictable rate, yet the odds of satisfying all of them together collapse. A model that handles each constraint about 41% of the time at k=8 succeeds on every one of the eight just 5.7% of the time arXiv paper.
Not all constraints hurt equally. Structural requirements — those that need the model to track state across the whole answer — lose roughly twice as much baseline capability per added constraint as simple lexical checks such as banned words. Failures are also nearly independent, which is precisely why the drop is multiplicative rather than additive: one wrong sentence count can invalidate every constraint that reads it arXiv paper.
Why it matters for builders
The practical cliff is sharp. Probe-level success drops below 50% once a model faces 7 simultaneous constraints, even for the strongest model tested, and 12 of the 15 models cross that failure line at 3 constraints or fewer Constraint Saturation Evaluation (doi).
For teams shipping agents, structured outputs, or safety guardrails, the takeaway is to cap how many independent rules a single prompt must enforce and to split verification into separate passes. For deeper context on where capable models fit in production stacks, see our coverage of OpenAI says small GPT-5.6 models can run agents.
