This is a mysterious question, and so we will be tempted to give mysterious answers. Readers beware.
When I reason about who ‘I’ am in a non-philosophical way, I notice a few things. I’m all in one physical space. I see out of two eyes and act through various muscles according to my intentions. And this feels very natural to me.
And yet when I want to reason about values in a coherent framework, I prefer to think in terms of massively parallel cognitive algorithms and their preferences, rather than the preferences of these bundles of algorithms that seem to have indexical subjective experience. In order for me to figure out if I’m going about this the wrong way, then, I have to ask: why is subjective experience indexical? And is indexicality an important value?
Let’s say that two computer programs happen to be installed on the same computer and can be accessed by a program-examining program or a program-running program or a program-optimizing program. If one of the programs is doing pretty much the same operation on this computer as on some other computer, then we can reason about both programs as if they were the same program. We can talk about what Mathematica does, and what specific parts of Mathematica do. But when we ask what the program-examining program does, then descriptions must become more general. It’s very dependent on which other programs are on the computer, and how often they get run, et cetera. The program-examining program only has information from the computer it’s on, and maybe it has access to the internet, but even then it generally doesn’t get a very in-depth view of the contents of programs on other computers.
Consider human experience. It seems that the most interesting qualia are the qualia of reflecting on cognitive algorithms or making decisions about which algorithms to run, which has a lot to do with consciousness. Instead of coming up with reasons as to why all subjective experience is indexical, we could come up with reasons as to why the subjective experience of the reflection or planning or decision-making algorithms is indexical. And I think there are okay explanations as to why it would be.
Humans aren’t telepathic, mostly. Our minds are connected tenuously by patterned vibrations in the air, seeing the movements of one another, and so on. This is not a high-bandwidth way to communicate information between minds, and especially not complicated information like the qualia of thinking a specific thought. It tends to take awhile to get enough details of another’s experience to recreate it oneself, or sometimes even to recognize what it could be pointing at. Read/write speed is terribly slow. Thus most information processing goes on in single minds, and the qualia of processing information is indexical. But when two minds are processing the exact same thing in the same way, then their experiences are not indexical. Their epiphenomenal spirits could jump back and forth and never know the difference.
What does that mean for the study of value? Humans seem to value their subjective experience above all else. A universe without sentience seems like a very bad outcome. But it’s not clear whether humans value indexical subjective experience, of subjective experience generally. Some of the most intense spiritual experiences I’ve heard of involve being able to feel true empathy for another, or to feel connected to all of the minds in the universe. These experiences have always been considered positive. The algorithms that make up humans thus might not strongly value keeping their subjective experience confined to inputs from one small physical space.
If you look at the brain as a piece of computing hardware for lots of algorithms inside it, then it seems natural to answer the question of what humans value by asking what the things that make up humans value. And if the algorithm running in one mind is the same as the algorithm running on another, then we needn’t look at their computing substrate. But does the fact that the decision algorithms in each mind tend to look at different patterns of cognitive algorithms (even if the decision algorithms themselves are nearly identical) mean that we have to go ahead and look at individual minds specifically to figure out what each decision algorithm wants? What sorts of values do these decision algorithms have? Are they subordinate to the values of the more parallel cognitive algorithms that they run? Do they largely use the same operations for satiating other algorithms in the mind, and if so, are their values not actually indexical, even if they can only satiate indexical drives? If so, do we need to reason about the wants of individual humans at all? I have my intuitions, but we’ll see if they’re justified.
It makes sense to ask these questions for the sake of axiology, but what about it helps with computational axiology specifically? Most proposals to solve the Friendliness problem I’ve heard of involve doing various sophisticated forms of surveying individual humans and seeing what they want, and then resolving conflicts between the humans. I suspect this probably works if you do it right. But I contend that it is difficult because it is the wrong way of going about it. It is difficult to tell a computer program to look at individual humans. Artificial intelligences are programs, and naturally reason in terms of programs. Humans are not programs. Humans run programs. And what humans value is those programs. If we had an AI look at the world to find algorithms, or decision processes, or what have you, it will find the algorithms that run on minds, and ignore whatever pieces of hardware they were running on. This isn’t a bug; it’s the way humans should be reasoning, too.
I’ve said before that I’m not particularly interested in Friendliness. This is because I care about programs, not their computing substrate. And if the same program is running on a human mind as on an iguana mind… what’s the difference?