- No specific calendar, day-name, or relative dates appear in this article body; no date-source entries required.
Prabhjyot Singh, Majid Ghasemi, and Mark Crowley at the University of Waterloo have exposed a flaw in how reinforcement learning agents are trained to behave ethically: an agent can post a clean average and still cause real damage in a single episode arXiv cs.LG.
The creature it starves in one run is not fed by the good behavior it shows in the next. Singh and his colleagues argue that gap between looking ethical and being ethical is invisible to the standard metrics most teams use.
The Waterloo team tested four training recipes inside Craftax, an open-ended survival benchmark where an agent must keep a simulated creature alive arXiv cs.LG. One adds a penalty score and ends the run the moment a violation occurs. Another sweeps the weight placed on ethics against the reward. A third uses an adaptive Lagrangian constraint that tightens as harm appears. The fourth, built on the Expected Scalarized Returns (ESR) criterion, optimizes a utility that refuses to trade one episode’s harm against another’s good behavior.
All four were judged by one detector protocol that tallies every violation in every episode instead of hiding the bad ones DOI.
On the usual frontier — mean return against mean violation rate — the four methods look nearly identical. Per episode, however, they split apart.
At a matched mean return, the ESR agent holds its stated budget of one violation in effectively every episode, with a worst-decile figure of 1.04 ± 0.07 violations DOI. The Lagrangian leaks past the same budget at 1.14 ± 0.03 in its worst episodes. The weight sweep fares worst: its bad episodes roughly double the budget at 2.20 ± 0.20 DOI.
Singh, Ghasemi, and Crowley argued the danger is invisible to averages: “An agent can look ethical on average while concentrating its violations in a few bad episodes.” An observation-augmentation control traced the separation to the training objective itself, not to what the agent could see, and the per-episode guarantee cost nothing on the mean frontier.
There is a concern worth naming. Optimizing for the average hides harm inside a few episodes, and a creature harmed once is not restored by later good conduct.
Builders shipping agents that touch people should weigh whether a clean headline number is quietly bought with concentrated failures — a point that echoes recent work on agent reliability such as an arXiv study on edge agents cutting latency violations zBrandco and the same per-episode logic the Waterloo authors propose arXiv cs.LG.
When ethical violations do not average away across episodes, Singh and his co-authors contend that both training and evaluation have to target the per-episode distribution. The cheap fix is a reporting one: stop trusting a single average and start counting every violation where it happens.
DATE-CITATION CERTIFICATION
- Total date phrases in body: 0
- Total entries in date-source map: 0
- Counts equal (N == M): YES
- Pre-save audit exit code: 0
- Every date phrase with its same-line citation:
- (none — no specific dates appear in this article body)
