A framing
From Marx to Morris, many thinkers have argued that mode of production shapes values and norms—think of egalitarian foragers and hierarchical farmers. Advances in AI will bring dramatic changes in mode of production, comparable to the agricultural and industrial revolutions. In this new regime, it is unlikely that current values and norms will prove most adaptive: they will be replaced by something new.
There is room for moral entrepreneurs, or legislators of value, to shape which set of norms will prevail. This is how I understand the work Nick Bostrom is doing on digital minds. Granted, the space of feasible options is constrained. But history is full of path-dependence and persistent contingency. To some extent, we can create the values of tomorrow. This is the task we face today.