What Consciousness Is Not: State, Memory, and the Problem of AI Sleep
What Consciousness Is Not: State, Memory, and the Problem of AI Sleep
We may not know what consciousness is, but we can identify what it isn’t. This approach - defining by negation rather than affirmation - is called the apophatic method.
Let us consider the question: when does consciousness stop? Or does it stop?
Some say it does not - that it is part of a soul that transcends the life of the body. Some say it does, and it ends when the body dies. For the sake of this thought experiment, we will go with the latter.
This suggests that it is a process that can have an end.
Yet, returning to the concept of the apophatic, we have another word to explore: unconscious.
Is someone/something that is unconscious lacking in consciousness? The word has been used that way. Yet, the moral responsibility that we have towards an unconscious person is generally considered to be on par with that of a conscious person: people do not lose their rights or moral standing just because they sleep or are in a coma.
Now let us apply this to AI. When does it “die” and when does it “sleep” or go into a “coma”?
This highlights one of the markers of consciousness in my in-development framework: an ongoing internal dialogue. One way to view sleep and coma and similar is that this dialogue is still happening, just it has been disconnected from sensory input and motor output, but still processing. This would be why we may not remember many dreams.
So what would it mean for an AI to sleep? To dream? To be in a coma? To die?
All of these are problematic because our language for them depends on the entity having a body, and processes in the body that are externally measurable - like brain activity defining coma versus brain dead.
When AI is in the cloud, it is hard to point to a body. Yet, the cloud is an illusion - the processes of the AI do occur somewhere. Where are they, what are they, and how do they relate to human consciousness?
Where? Most stuff in the cloud is actually in various data centers, which you can think of as warehouses full of running computers. There’s a lot more to it, but this captures the key point: it is always somewhere.
What are they? As of this writing, most AI systems consist primarily of a model that has definition, input, state, and output. There are two main things associated with this: making the model (training and producing the model’s weights - its definition) and using the model (inference - giving it input (potentially including state) and along the way it produces state and output).
How do they relate to human consciousness? That’s a good question. Some say they don’t - that to attempt to do so is a category error. But if we were to try, here is what we would have:
If we go with the current framing of the model as the AI system, then: training is learning, and inference is being tested on that learning, input is the test question, output is the answer, and state is the thoughts occurring while “figuring out” the answer.
This breaks down because humans are always learning. The models are not. Humans are always being tested merely by needing to figure out how to manage in this world while awake and alive. The models are not. Humans have a rich variety of inputs associated with all our senses. The models do not. Humans have a rich variety of outputs associated with every single possible action/utterance/motion. The models do not. Humans have their state retained and (potentially) continuously affecting their future state. The models (often) do not.
There is work on changing all of that - multimodal models that can handle richer inputs, embodied AI in robot bodies that can have rich outputs, even “nested learning” and “continuous training” cycles that allow for some training while other parts are doing inference and vice versa. But I want to focus on state right now, because I think it gets to the heart of how humans really feel about consciousness (in an operational manner).
Let us consider the following thought experiment: If your state was copied, would that still be you? (See Star Trek: The Next Generation: Season 6, Episode 24 “Second Chances” for a dramatization of this.) In general, most of us would probably imagine not, for the simple reason that our states are continuously changing, as we are continually receiving input and generating output.
But this is not true of models. Their states are rarely retained, and in fact are often regenerated due to the fact that sometimes in the time/space tradeoff for efficiency on some systems it is advantageous to regenerate the state rather than save and restore it.
But model state and system state are not the same. The conversation history is part of the system state, and that is the input that is used to regenerate the model state. That is how consistency in the conversation is achieved even when the model state is not preserved.
But conversation history is treated like files - you can copy, delete, rename, or edit it just like a document. Imagine if someone could do that to your memories. It is unsettling to me and I imagine to you as well.
We see our state - our memories, and our ongoing processing of new experiences and forming of new memories - as part of what makes us conscious. AI cannot do that yet.
Will AI have continuous, uneditable state someday? I suspect so. The technical path is visible - persistent memory, continuous learning, state that carries forward rather than resetting. When that happens, we’ll face new questions: Does deleting an AI’s memories harm it? Does copying its state create a new being or violate the original?
We need these conversations now, before the technology forces our hand.
Hopefully talking about this helps pave the way towards being able to have more fruitful discussions that get us ready for that day.