Highlight from Moral AI

Despite our imperfections, most of us want our own moral judgments and decisions to be less subject to such distorting influences. We also often want the morality built into an AI to reflect the moral judgments that we ourselves would make if we were more informed, rational, and unbiased, even if we will never actually be in an ideal state. In such cases, we want AIs to predict and reflect our idealized human moral judgments, not our actual moral judgments.