The Disturbance Problem:How Predictive AI Models Reshape Preferences.

Canthal Tilt. A highly accurate machine learning model is trained to predict facial attractiveness using swipe data from a dating app. Analysis of the model reveals that one feature correlates strongly with attractiveness: positive canthal tilt.

What happens when this pattern becomes known? My conjecture is that people attend to canthal tilt more, and it comes to figure in their judgments. As a result, their preferences change. The model’s predictions become inaccurate, not because the model was poorly trained, but because interpreting a model of preferences can change those preferences. I call this problem disturbance. I argue that disturbance is a third problem for machine learning, distinct from the familiar worries that models may be inaccurate or unjust, and that this is a problem for us as well as for the model.

How does disturbance occur? Evaluative judgments rely on reasons, only some of which are explicit. We also respond to implicit reasons, which we use without being able to articulate them—or even explicitly disavow while actually responding to them, as Nomy Arpaly observes. An interpreted model changes the balance of reasons in two ways. First, it makes previously implicit reasons explicit, likely (but not always) increasing their weight. Second, it provides statistical justification for using those reasons. An art critic can do the first without the second. Consider:

Crush Notes. You enjoy jazz but know little about jazz or music theory. You overhear a knowledgeable critic praise Thelonious Monk’s use of crush notes and believe this explanation of why Monk is great. Now you like Monk’s music—and all uses of crush notes, even mediocre ones—even more.

Unlike Canthal Tilt, Crush Notes is obviously not morally icky, but it is still a deficient preference change: it leaves you with testimony rather than statistical justification. A model raises salience accompanied by the authority of statistical evidence rather than easily discounted testimony, and across a whole population. When enough people find in the model a pattern in their own evaluations, the very feature it predicts changes.

Disturbance is distinct from two familiar problems in machine learning. The problem of external validity asks whether a model trained on past data will perform well in the future. Hume’s and Nelson Goodman’s problems of induction, which are other ways of stating the problem, raise only undercutting doubt; disturbance gives rebutting doubt, evidence that models will fail in the future. On the other hand, the problem of alignment asks whether a model accurately promotes human values. But even a morally good, fair, and accurate model can disturb the feature it predicts. Nor is disturbance one of Ian Hacking’s looping effects, in which categorizing people changes their self-understanding and behavior: disturbance changes how a population evaluates other objects, not (just) themselves.

The ethical cost, which the benefits of a model may outweigh, has two parts. The first is straightforward: changing preferences changes behavior, which has moral import, good or bad. The second is that disturbance undermines the authenticity and understanding of our preferences, much as brute deference to aesthetic or moral testimony does. The trouble lies in the norm, defended by Alison Hills and Robert Hopkins, that ideal evaluative judgments must be based on reasons we competently grasp.

Competent grasp comes at two levels. Compare first-order inexplicability (“I judge that x is good because… I don’t know”) with second-order inexplicability (“I judge that x is good because x has F and G. Why is F good-making? I don’t know”). Some first-order inexplicability is common, and disturbance is possible only because it is present. But in ideal cases, whatever first-order reasons one responds to are accountable to one’s second-order beliefs about why those features are good-making. A model provides first-order beliefs that are not accountable to the second order. Through the statistical authority of the model and its effects on our attention, disturbance passes off a purely descriptive fact about correlation as a normative reason for evaluation.

Is the solution just more modeling? One might modify the original model, or add an auxiliary one, to predict and mitigate disturbance. But evaluation has a volitional element: if a model predicts how my evaluations will change, I can simply resolve to evaluate otherwise. A further model that predicts my response is open to the same reply, and there cannot be auxiliary models all the way down. In any case, a correction would restore only the model’s accuracy, and leave the ethical cost in place. The two familiar worries about predictive models are that they may be inaccurate or unjust. But even accurate and just models cause disturbance, and can undermine our responsibility for our preferences.

Written for a seminar on the history of automated decision-making (Shamik Dasgupta and Ben Recht); a condensed version was given at the 16th Braga Meetings on Ethics and Political Philosophy, July 2026.

Questions, objections, or a better example? Write to me: patrick.pan@patrickpan.com.