“Can we trust evolutionary development to take our species in broadly desirable directions? Starting from primitive, unconscious life, biological evolution has led to the development of ever more advanced organisms, including creatures that have minds, consciousness, language, and reason…the big picture shows an overarching trend towards increasing levels of complexity, knowledge, consciousness, and organization, a trend which, not to put too fine a point on it, we may label “progress”
We shall explore a different set of existential risks in which the world would end more gradually, not with a bang but a whimper. Let us therefore suppose that no sudden cataclysm puts an end to life. Let us also set aside scenarios in which evolution leads to the erosion of complexity. We shall explore how, even if evolutionary development continues unabated in the direction of greater complexity, things could nevertheless take a wrong turn leading to the disappearance of all the things we value.
Scenario I: The Mindless Outsourcers
Technological progress continues to accelerate and at some point the technology of “mind uploading” becomes possible. Some human individuals upload and make many copies of themselves. Meanwhile, there is gradual progress in neuroscience and artificial intelligence, and eventually it becomes possible to isolate individual cognitive modules and connect them up to modules from other uploaded minds. Possibly, modules would need to be trained before they can communicate with each other effectively. Modules that conform to a common standard would be better able to communicate and cooperate with other modules and would therefore be economically more productive, creating a pressure for standardization.
Competitive uploads begin outsourcing increasing portions of their functionality: “Why do I need to know arithmetic when I can buy time on Arithmetic-Modules Inc. whenever I need to do my accounts?
Some uploads might prefer to retain most of their functionality and handle tasks themselves that could be more efficiently done by others. They would be like hobbyists who enjoy growing their own vegetables or knitting their own cardigans; but they would be less efficient than some other uploads, and they would consequently be outcompeted over time…
There might be no niche for mental architectures of a human kind.
Would these complexes be worthwhile from our current point of view? Do we, upon reflection, really favor a world in which such alien types of complexes have replaced human-type complexes?
We can thus imagine a technologically highly advanced society, containing many sorts of complex structures, some of which are much smarter and more intricate than anything that exists today, in which there would nevertheless be a complete absence of any type of being whose welfare has moral significance. In a sense, this would be an uninhabited society. All the kinds of being that we care even remotely about would have vanished.
Scenario II: All-Work-And-No-Fun
Even if we do not suppose that uploading and outsourcing will result in a widespread loss of consciousness, we can still entertain the possibility that intrinsically valuable activities and states of consciousness become rarer or disappear altogether. Much of human life’s meaning arguably depends on the enjoyment, for its own sake.
Perhaps what will maximize fitness in the future will be nothing but non-stop high-intensity drudgery, work of a drab and repetitive nature, aimed at improving the eighth decimal of some economic output measure. Even if the workers selected for in this scenario were conscious, the resulting world would still be radically impoverished in terms of the qualities that give value to life…”
This is just an excerpted version of the first part of the article, emphasis mine.
The full paper by Nick Bostrom, from Oxford University.
Here’s his bio on wiki
He goes on to discuss to discuss how the human activities we find enjoyable such as the arts are “flamboyant displays” like a peacock’s feathers that give a hard-to-fake demonstration desirable mate qualities. He reasons that rational post-human beings might recognize these as baggage of our evolutionary past and see little value in them. Not to mention, simply donating vials of sperm is a more efficient mating strategy for an advanced organism than is spending years learning an artistic talent. Thus, many of the human enjoyments we value highly get weeded out.
He goes on to point out that conscious beings may find themselves at a competitive disadvantage against entities that focus their whole energies on the most effective means of replication, eliminating the unnecessary flamboyant display of consciousness.
Bostrom’s next step, is to ponder if these outcomes could be avoided by conscious beings aka. “eudaemonic agents” controlling the course of evolution and technology to preserve conscious existence for its own sake.
“To this problem there are only two possible solutions: preventing non-eudaemonic variants from arising in the first place, or modifying the fitness function so that eudaemonic traits become fitness-maximizing…
Even if the eudaemonic agents could prevent dangerous mutants from arising, their efforts would be to no avail if the original population already contained some individuals with non-eudaemonic fitness-maximizing preferences, because these would then proliferate and eventually dominate.
Forestalling the dystopian evolutionary scenarios by preventing non-eudaemonic agents from arising is therefore a non-starter. At most, this measure could serve an auxiliary role.”
He also finds it problematic to imagine an air-tight way any social structure or procedure could indefinitely stop an inherently superior approach from eventually taking over.
Bostrom tries to imagine a universe where conscious agents and the un-conscious simply inhabit different niches in the same ecosystem or are even joined together under one governing power.
“Current evidence does not warrant any great confidence in the belief that the default course of future human evolution points in a desirable direction. In particular, we have examined a couple of dystopian scenarios in which evolutionary competition leads to the extinction of the life forms we regard as valuable. Intrinsically worthwhile experience could turn out not to be adaptive in the future.”
This article caught my attention because in my own inquiries asking what can have any intrinsic meaning in the grand scheme of this universe, itself possibly transient, the development of consciousness is one of the few things I can feel convinced has some kind of inherent value.
In which case the threats that affect its development are possibly the most important to anticipate and address before they become fatal problems.
If there’s one thing we can take away from this article, it’s to not take the existence of self-awareness for granted as the inevitable result of ‘progress.’
Consciousness is a great extravagance that can even cause us to self-destruct or “waste” our energies on enjoyment that accomplishes nothing, directly against the interests of the species. Once we’ve decided awareness is valuable, it’s the clear mission of conscious agents to defend and develop the luxury of being. Humans as we exist now are at best modestly self-aware, only slightly removed from other animals, but we have a mission if we should accept it to blow on the spark we have.
It seems clear that:
-first we have to improve ourselves within the bounds of natural selection
-second we must escape the trap of Darwinian selection
-third, we have to be smart enough to prevent synthetic forms of selection from destroying conscious agency.