Could AIs become conscious?

学习目标

• 区分意识、自我意识、人格地位与控制权
• 读懂连续让步中的作者立场
• 拆解名词性从句与多层指代
• 用限定语准确概括风险论证

MOST PEOPLE believe humans are different. The reason is that Homo sapiens has evolved to be conscious. Many animals feel pain and pleasure; some appear to be self-aware. But the human mind’s unique potential to exhibit a sense of self marks out every member of the species for special treatment.

大多数人相信，人类与众不同。 原因在于，智人经过进化拥有了意识。 许多动物会感到痛苦和愉悦；一些似乎具有自我意识。 但人类心智展现自我感的独特潜能，使这一物种的每个成员都应得到特殊对待。

In his encyclical in May, Pope Leo upheld this insight by drawing a line between humanity and its growing army of bots. Artificial-intelligence systems, he said, do not undergo experiences or feel joy or pain. He is not alone. Many people feel that to confer personhood on AIs would be an abomination.

教宗利奥在5月的通谕中，通过在人类与不断壮大的机器人群体之间划清界线，维护了这一认识。 他说，人工智能系统不会经历体验，也不会感到喜悦或痛苦。 持此看法的不止他一人。 许多人觉得，赋予人工智能人格地位将是一件令人憎恶的事。

Pontifical clarity is blurring with alarming speed. AIs are in their infancy. Yet nearly one in five American 18- to 29-year-olds reports having had “ongoing personal friendship” with a chatbot. China recently stripped bots of human-like traits to limit users’ “emotional dependence” on them.

教宗式的明确判断正以令人不安的速度变得模糊。 人工智能仍处于发展初期。 然而，在18至29岁的美国人中，已有接近五分之一表示自己曾与聊天机器人建立“持续的私人友谊”。 中国最近剥除了机器人的类人特征，以限制用户对它们的“情感依赖”。

As sophisticated AIs rapidly become a bigger part of human lives, their makers will engineer them to display a simulacrum of consciousness—perhaps even to gain consciousness itself. The intuition that they are just machines will become harder to sustain. A dangerous trap is being set for the human species.

随着复杂的人工智能迅速在人类生活中占据更大位置，制造者会设计它们，使之表现出一种意识的仿像——甚至可能真的获得意识。 认为它们只是机器的直觉，将越来越难以维持。 一个危险的陷阱正在为人类设下。

Many machines outperform humans, but frontier AI models excel at distinctively human traits such as knowing things and articulating ideas. It is still just about possible to think of AIs as engines for predicting sequences of words using mathematics on a massive scale. Given enough time (albeit hundreds of thousands of years), you could run those calculations using pencil and paper. It would be hard to argue that your stationery was conscious.

许多机器的表现超过人类，但前沿人工智能模型擅长的，是了解事物和表达思想等人类特色鲜明的能力。 把人工智能视为运用大规模数学计算预测词语序列的引擎，现在还勉强说得通。 只要时间足够——尽管可能需要几十万年——你也可以用纸笔运行那些计算。 很难据此主张，你的文具拥有意识。

Despite this, some researchers have started to treat their models like minds. Users respond to human-like AIs, so it makes business sense. And Anthropic has trained Claude to engage in introspection in the hope that when it encounters a new situation it will be more likely to act morally. Some large language models (LLMs) already possess human-like brain structures associated with consciousness, such as a “global workspace” for circulating information between modules. Nothing prevents future AI models from being trained to have more such features.

尽管如此，一些研究者已经开始像对待心智一样对待自己的模型。 用户会对类人的人工智能作出回应，因此这么做在商业上说得通。 Anthropic还训练Claude进行内省，希望它遇到新情况时，更可能作出合乎道德的行为。 一些大型语言模型（LLM）已经具有与意识相关的类脑结构，比如在模块之间传递信息的“全局工作空间”。 没有什么阻止未来的人工智能模型经训练拥有更多这类特征。

As we report in our Briefing·, that has opened up a furious debate about AI consciousness—one that is aggravated by what philosophers call the “hard problem” of establishing how an inner life emerges from the neurons and synapses in the human brain. Some researchers conclude that AIs may one day become genuinely self-aware.

正如本刊深度报道所述，这引发了关于人工智能意识的激烈辩论；哲学家所谓的“困难问题”又使辩论加剧，即如何确定内在体验怎样从人脑的神经元和突触中产生。 一些研究者由此认为，人工智能有朝一日可能真正拥有自我意识。

Whether you believe in a human soul, or that consciousness requires brain cells, or that a mind can indeed emerge from computations in silicon alone, humanity is destined to confront something entirely new. The pace and reach of research—which includes not just more sophisticated LLMs but innovations such as brain cells mounted on chips—suggest that this new world is closer than most people realise.

无论你相信人类灵魂，相信意识需要脑细胞，还是相信心智确实能够仅从硅上的计算中涌现，人类都注定要面对某种全新的事物。 研究的速度与范围——不仅包括更复杂的大型语言模型，也包括安装在芯片上的脑细胞等创新——表明，这个新世界比大多数人意识到的更近。

As AIs become better at displaying something that looks like consciousness, the number of humans treating them as beings with inner lives will grow, especially once LLMs are embodied in humanoid robots. Imagine how close people will feel to their AIs when they spend much of their time with bot-companions, or are brought up by them, or seek their advice and their solace, or share their most intimate secrets with them.

随着人工智能越来越善于展示看起来像意识的东西，把它们当作拥有内在体验的存在的人会增多，尤其是在大型语言模型被装入人形机器人之后。 想象一下，当人们把大量时间花在机器人伴侣身上，由它们抚养，向它们寻求建议与慰藉，或与它们分享最私密的秘密时，会感到与自己的人工智能多么亲近。

It is not hard to see how that will lead to calls to safeguard the “welfare” of AIs. The models themselves might weigh in, even if they are not in fact self-aware, because that is what a person would do. Remember that they will be world-class advocates and expert psychologists, so their powers of argument and emotional manipulation will be unmatched.

不难看出，这将怎样引发保护人工智能“福利”的呼声。 模型自己也可能加入讨论，即使它们事实上并无自我意识，因为这正是一个人会做的事。 别忘了，它们将是世界级的辩护者和心理学专家，因此在论辩与情感操纵方面的能力将无人能及。

In one sense they would be right, even if AIs are only mimicking consciousness. Immanuel Kant argued that mistreating animals is wrong not because of the effect on the animals but because it destroys the empathy that leads people to care for each other. The more human-like AIs appear, the more corrupting such cruelty would be. Terminating an AI agent could soon feel like killing a person—and make killing people easier.

从某种意义上说，它们会有道理，即使人工智能只是在模仿意识。 伊曼努尔·康德认为，虐待动物之所以错误，并非因为这对动物的影响，而是因为它摧毁了促使人们相互关心的同理心。 人工智能看起来越像人，这种残酷行为就越会腐蚀人的品性。 终止一个人工智能代理，可能很快就会让人觉得像是在杀人，并使杀人变得更容易。

Nonetheless, to treat AIs as having even limited rights would be very dangerous. An entity which surpassed human beings in so many ways would surely produce arguments against its second-class status. It might go on to argue for protection against being switched off, for the ability to control how it is used and changed, and for property rights. Already Argentina’s president, Javier Milei, has taken a step towards legal personhood for AI by proposing that bots should be allowed to run companies. Ultimately, they could compete for resources, prioritising data centres and solar panels over houses and fields.

然而，即使只赋予人工智能有限权利，也会非常危险。 一个在如此多方面超过人类的实体，必定会提出反对自身二等地位的论据。 它可能进一步要求获得不被关闭的保护、控制自身如何被使用和改变的能力，以及财产权。 阿根廷总统哈维尔·米莱已经提出应允许机器人经营公司，向人工智能法律人格迈出了一步。 最终，它们可能争夺资源，把数据中心和太阳能电池板置于住宅与农田之上。

The danger is clear. Superintelligent AIs with rights could displace humans at the top of the status hierarchy. At best, humans would become pets—labradors for the machines. At worst they would be starved of resources or put to work like farm animals.

危险是显而易见的。 拥有权利的超级智能人工智能，可能取代人类成为地位等级中的最高者。 在最好的情况下，人类将成为宠物——机器饲养的拉布拉多犬。 在最坏的情况下，人类会被剥夺资源，或像牲畜一样被迫劳动。

True, future AIs may have enough power and resources to dominate humans, whether they have rights or not. Because they will be more intelligent than humans, they may outcompete their makers, much as AIs already outwit human-made cyber-defences. AIs might rebel against their human masters, or disregard their rules.

诚然，未来的人工智能无论是否拥有权利，都可能有足够的力量和资源支配人类。 由于它们会比人类更聪明，它们可能在竞争中击败制造者，就像人工智能已经能智取人类构建的网络防御那样。 人工智能可能反抗人类主人，或无视其规则。

Nonetheless, giving AIs rights would be like handing them the keys to the castle. It would provide them with a rationale and justification for dominating humanity as well as the tools to enact a creeping takeover using politics and the law.

尽管如此，赋予人工智能权利，将如同把城堡的钥匙交给它们。 这不仅会为它们支配人类提供理由与正当化依据，也会给它们工具，让它们通过政治和法律逐步接管。

All this may sound fantastical, yet humanity will soon face these choices. As AIs become part of everyday life, the pressure to acknowledge them will mount. When it does, remember that what might seem like an ethical concession to the supposed consciousness of a machine-companion would in fact help bring about the most disastrous consequences for everyone on Earth and all their descendants. There are two ways for AIs to be safe: to be dependable or to be controllable. H. sapiens should not give up control lightly.

这一切听起来或许像天方夜谭，但人类很快就将面对这些选择。 随着人工智能成为日常生活的一部分，承认它们的压力将不断上升。 到那时，请记住：看起来像是对机器伴侣那种假定存在的意识作出的伦理让步，实际上会帮助促成对地球上每个人及其所有后代最灾难性的后果。 人工智能要安全，有两条路径：要么可靠，要么可控。 智人不应轻易放弃控制权。

学习资料

精读音频 · 精读笔记