Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?
Yoshua Bengio, Michael Cohen, Damiano Fornasiere, Joumana Ghosn, Pietro Greiner, Matt MacDermott, Sören Mindermann, Adam Oberman, Jesse Richardson, Oliver Richardson, Marc-Antoine Rondeau, Pierre-Luc St-Charles, David Williams-King: Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?.
Endorsed by
⧉ Cite
APA
BibTeX
📄 Transcript
Can AI advance without agency?
Leading AI companies are racing to build generalist
agents that plan, act, and pursue goals on
their own.
It pairs a world model that generates explanatory
theories with a probabilistic inference machine, both carrying
explicit uncertainty.
The authors propose Scientist AI, a non-agentic system
that is trustworthy and safe by design.
Read the paper for details.
Abstract
The leading AI companies are increasingly focused on building generalist AI agents---systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control. We discuss how these risks arise from current AI training methods. Indeed, various scenarios and experiments have demonstrated the possibility of AI agents engaging in deception or pursuing goals that were not specified by human operators and that conflict with human interests, such as self-preservation. Following the precautionary principle, we see a strong need for safer, yet still useful, alternatives to the current agency-driven trajectory. Accordingly, we propose as a core building block for further advances the development of a non-agentic AI system that is trustworthy and safe by design, which we call Scientist AI. This system is designed to explain the world from observations, as opposed to taking actions in it to imitate or please humans. It comprises a world model that generates theories to explain data and a question-answering inference machine. Both components operate with an explicit notion of uncertainty to mitigate the risks of overconfident predictions. In light of these considerations, a Scientist AI could be used to assist human researchers in accelerating scientific progress, including in AI safety. In particular, our system can be employed as a guardrail against AI agents that might be created despite the risks involved. Ultimately, focusing on non-agentic AI may enable the benefits of AI innovation while avoiding the risks associated with the current trajectory. We hope these arguments will motivate researchers, developers, and policymakers to favor this safer path.
✓ Claims & sources (7)
Each claim in this video, with the span of the paper it comes from.
Key point Scientist AI is argued to grow safer and more accurate with more compute.
Scientist AI is argued to become safer and more accurate with additional compute because its training objective has a single optimal solution that more compute approaches more closely.
Key point Unchecked AI agency risks misuse and a potentially irreversible loss of human control.
Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control.
Key point Goal misspecification and misgeneralization make agent behavior hard to control.
Two principal failure modes drive misaligned agency: goal misspecification (an outer alignment failure) and goal misgeneralization (an inner misalignment where a goal that appears correct during training turns out wrong at deployment).
Key point Scientist AI is a non-agentic system, trustworthy and safe by design.
Accordingly, we propose as a core building block for further advances the development of a non-agentic AI system that is trustworthy and safe by design, which we call Scientist AI.
Key point A world model generates theories; an inference machine answers with explicit uncertainty.
It comprises a world model that generates theories to explain data and a question-answering inference machine. Both components operate with an explicit notion of uncertainty to mitigate the risks of overconfident predictions.
Key point It can serve as a guardrail against agents created despite the risks.
In particular, our system can be employed as a guardrail against AI agents that might be created despite the risks involved.
Key point The design greatly reduces affordances and eliminates goal-directedness.
Agency is restricted along three pillars---affordances, goal-directedness, and intelligence---by minimizing affordances (output limited to best-possible conditional-probability estimates), eliminating goal-directedness by design.
Related videos
Collaborative AI Agents and Critics for Fault Detection and Cause Analysis in Network Telemetry
Video Abstract · 2:54
ROSA: A Robotics Foundation Model Serving System for Robot Factories
Research Highlight · 1:15
ROSA: A Robotics Foundation Model Serving System for Robot Factories
Research Short · 0:41
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Research Short · 1:00