The Hard Problem of AI Alignment: Value Forks in Moral Judgment, Markus Kneer & Juri Viehoff, Proceedings of the 2025 Acm Conference on Fairness, Accountability, and Transparency (2025)
Abstract: Complex moral trade-offs are a basic feature of human life: for example, confronted with scarce medical resources, doctors must frequently choose who amongst equally deserving candidates receives medical treatment. But choosing what to do in moral trade-offs is no longer a ‘humans-only’ task, but often falls to AI agents. In this article, we report findings from a series of experiments (N=1029) intended to establish whether agent-type (Human vs. AI) matters for what should be done in moral trade-offs. We find that, relative to a human decision-maker, participants more often judge that AI agents should opt for fairness at the expense of maximizing utility. In our discussion, we explain how the reported differences (we call them agent-type ‘value forks’) matters for the study of moral value alignment, and we hypothesize what could explain these value forks. We close by reflecting on limits of our results and indicate avenues of further research.