Call for red lines to prevent unacceptable AI risks

We urgently call for international red lines to prevent unacceptable AI risks.

Launched during the 80th session of the United Nations General Assembly, this call has broad support from prominent leaders in policy, academia, and industry.

300+ prominent figures

10 former heads of state and ministers

90+ organizations

15 Nobel Prize and Turing Award recipients

What Signatories Say

It is in our vital common interest to prevent AI from inflicting serious and potentially irreversible damages to humanity, and we should act accordingly.

Ahmet Üzümcü
Former Director General of the Organization for the Prohibition of Chemical Weapons

For thousands of years, humans have learned—sometimes the hard way—that powerful technologies can have dangerous as well as beneficial consequences. With AI, we may not get a chance to learn from our mistakes, because AI is the first technology that can make decisions by itself, invent new ideas by itself, and escape our control. Humans must agree on clear red lines for AI before the technology reshapes society beyond our understanding and destroys the foundations of our humanity.

Yuval Noah Harari
Author of "Sapiens"

Humanity in its long history has never met intelligence higher than ours. Within a few years, we will. But we are far from being prepared for it in terms of regulations, safeguards, and governance.

Csaba Kőrösi
Former President of the UN General Assembly

The development of highly capable AI could be the most significant event in human history. It is imperative that world powers act decisively to ensure it is not the last.

Stuart Russell
Distinguished Professor of Computer Science at the University of California, Berkeley

The current race towards ever more capable and autonomous AI systems poses major risks to our societies and we urgently need international collaboration to address them. Establishing red lines is a crucial step towards preventing unacceptable AI risks.

Yoshua Bengio
2018 Turing Award Winner

This should be a major wake up call for policy makers and AI developers.

Lord Tim Clement-Jones
House of Lords' Science Innovation and Technology Spokesperson

AI Red Lines need unity of knowing and acting, both from human and from AI, and it takes the World to save ourselves and our next generations.

Yi Zeng
Dean
Beijing Institute of AI Safety and Governance

Without AI safeguards, we may soon face epistemic chaos, engineered pandemics, and systematic human rights violation. History teaches us that when confronted with irreversible, borderless threats, cooperation is the only rational way to pursue national interests.

Maria Ressa
Nobel Peace Prize Laureate

Signatories

Joseph Stiglitz

Nobel Laureate in Economics

Professor of Finance and Business
Columbia University

Juan Manuel Santos

Former President of Colombia
Nobel Peace Prize Laureate

Chair
The Elders

Maria Ressa

Nobel Peace Prize Laureate

Professor of Practice
Institute of Global Politics
Columbia university

Co-founder and CEO
Rappler

Daron Acemoğlu

Nobel Laureate in Economics

Institute Professor
MIT

Mary Robinson

First Woman President of Ireland

Former UN High Commissioner for Human Rights

Member
The Elders

Ahmet Üzümcü

Nobel Peace Prize Recipient on Behalf of the OPCW

Former Director General
Organization for the Prohibition of Chemical Weapons (OPCW)

Senior Fellow
European Leadership Network (ELN)

Geoffrey Hinton

Nobel Laureate in Physics
Turing Award Winner

Emeritus Professor of Computer Science
University of Toronto

Yoshua Bengio

Most Cited Living Scientist
Turing Award Winner

Full Professor
Université de Montréal

Chair
International Scientific Report on the Safety of Advanced AI

Co-President and Scientific Director
LawZero

Founder and Scientific Advisor
Mila – Quebec AI Institute

John Hopfield

Nobel Laureate in Physics

Emeritus Professor
Princeton University

Yuval Noah Harari

Author of 'Sapiens'

Professor of History
Hebrew University of Jerusalem

Jennifer Doudna

Nobel Laureate in Chemistry

Professor
University of California, Berkeley
Co-developer of CRISPR-Cas9, gene-editing tool

Enrico Letta

Former Prime Minister of Italy

President
Agenzia di Ricerche e Legislazione (AREL)

Csaba Kőrösi

77th President of the UN General Assembly

Strategic Director
Blue Planet Foundation

Giorgio Parisi

Nobel Laureate in Physics

Emeritus Professor
University of Rome La Sapienza

Sir Oliver Hart

Nobel Laureate in Economics

Professor
Harvard University

Ya-Qin Zhang

Chair Professor and Dean, Institute for AI Industry Research
Tsinghua University

Former President
Baidu

Stuart Russell

Professor and Smith-Zadeh Chair in Engineering
University of California, Berkeley

Founder
Center for Human-Compatible Artificial Intelligence (CHAI)

Joseph Sifakis

Turing Award Winner

Research Director Emeritus, Verimag Lab
Université Grenoble - Alpes

Kate Crawford

TIME 100 AI

Professor
University of Southern California

Senior Principal Researcher
MSR

Wojciech Zaremba

Co-founder of OpenAI

Former Research Scientist
Facebook AI Research

Former Research Scientist
Google Brain

Yanis Varoufakis

Former Minister of Finance of Greece

Professor
University of Athens

Peter Norvig

Education Fellow
Stanford, Institute for Human-Center AI (HAI)

Director of Engineering
Google

George Church

Professor
Harvard Medical School & MIT

Ian Goodfellow

Principal Scientist
Google DeepMind

Inventor of generative adversarial networks

Founder of Google Brain's machine learning security research team

Jakub Pachocki

Chief Scientist
OpenAI

Andrew Chi-Chih Yao

Turing Award Winner

Professor
Tsinghua University

Gillian Hadfield

Bloomberg Distinguished Professor of AI Alignment and Governance
Johns Hopkins University

Sir Stephen Fry

Writer, Director, Actor

Yi Zeng

TIME 100 AI

Dean
Beijing Institute of AI Safety and Governance

Gustavo Béliz

Former Governor of Argentina

Former Minister
Government of Argentina

Former Secretary of the President
Government of Argentina

Chair
Economic and Social Council of Argentina

Baroness Beeban Kidron

Crossbench Peer
UK House of Lords

Advisor to the Institute for Ethics in AI
Oxford University

Founder and Chair
5Rights Foundation

Gbegna Sesan

Executive Director
Paradigm Initiative

Anna Ascani

Vice President
Italian Chamber of Deputies

Dan Hendrycks

TIME 100 AI

Executive Director
Center for AI Safety

Advisor
xAI

Advisor
Scale AI

Dawn Song

Professor
University of California, Berkeley

Gary Marcus

Professor Emeritus
New York University

Audrey Tang

Taiwan's Cyber Ambassador and First Digital Minister
TIME 100 AI

Senior Accelerator Fellow, Institute for Ethics in AI
University of Oxford

Maria João Rodrigues

Former Portuguese Minister of Employment

Former Member of the European Parliament

President of the Foundation for European Progressive Studies (FEPS)

Rachel Adams

Founding CEO
Global Center on AI Governance

Adetola A.Salau

Special Adviser to the Honourable Minister of Education
Federal Ministry of Education Nigeria

Thibaut Bruttin

Director General
Reporters Without Borders (RSF)

Maria Chiara Carrozza

Former Italian Minister of Education, University and Research

Full Professor of Biomedical Engineering and Biorobotics
University of Milano-Bicocca

Daniel Kokotajlo

TIME 100 AI

Former researcher
OpenAI

Co-founder and Lead
AI Futures Project

Lord Tim Clement-Jones

Peer / Science Innovation and Technology Spokesperson
UK House of Lords

Xue Lan

TIME 100 AI

Dean, Schwarzman College
Tsinghua University

Nicolas Miailhe

Founder and Non-Executive Chairman
The Future Society

Co-founder and CEO
PRISM Eval

Marc Rotenberg

Executive Director and Founder
Center for AI and Digital Policy

Laurence Devillers

Knight of the Legion of Honour

Professor of AI
Sorbonne University/CNRS

President
Blaise Pascal Foundation

Jason Clinton

Chief Information Security Officer
Anthropic

Jean Jouzel

Nobel Peace Prize co-Recipient as a Vice-Chair of the IPCC

Emeritus Director of Research
French Alternative Energies and Atomic Energy Commission (CEA)

Former Vice President
Intergovernmental Panel on Climate Change (IPCC)

Riccardo Valentini

Nobel Peace Prize as a member of the IPCC board

Professor
University of Tuscia

Robert Trager

Director
Oxford Martin AI Governance Initiative, University of Oxford

Brice Lalonde

Former French Minister of the Environment

Former advisor on Sustainable Development to the UN Global Compact

Senator Scott Wiener

California State Senator
Author of California AI safety legislation

Sören Mindermann

Scientific Lead
International AI Safety Report

Brando Benifei

Member of the European Parliament

European AI Act co-Rapporteur

Miguel Luengo-Oroz

CEO
Spotlab.ai

Former first Chief Data Scientist
United Nations

Professor
Universidad Politécnica de Madrid

Sergey Lagodinsky

Member of the European Parliament

Vice-Chair
Greens/EFA Group

Shadow Rapporteur on the AI Act
Member of the AI Act Implementation Working Group in the European Parliament

Max Tegmark

TIME 100 AI

Professor
MIT

President
Future of Life Institute

Sneha Revanur

TIME 100 AI

Founder/President
Encode

Roman Yampolskiy

Professor of Computer Science
University of Louisville

Mark Nitzberg

Executive Director
CHAI

Interim Executive Director
IASEAI

Organizer

Niki Iliadis

Director, Global AI Governance
The Future Society

Lead Organizer

Charbel-Raphaël Segerie

Executive Director
French Center for AI Safety (CeSIA)

Lead Organizer

What are red lines in the context of AI?

AI red lines are specific prohibitions on AI uses or behaviors that are deemed too dangerous to permit under any circumstances. They are limits, agreed upon internationally, to prevent AI from causing universally unacceptable risks.

Why are international AI red lines important?

International AI red lines are critical because they establish clear prohibitions on the development, deployment, and use of systems that pose unacceptable risks to humanity. They are:

Urgent: Their primary purpose is to prevent the most severe and potentially irreversible harms for humanity and global stability.
Feasible: Red lines represent the lowest common denominator on which states can agree. Even governments divided by economic or geopolitical rivalries share a common interest in avoiding disasters that would transcend their borders.
Widely Supported: Major AI companies have already acknowledged the need for red lines, including at the AI Seoul Summit 2024. Top Scientists from the US and China have already asked for specific red lines, and this is the most widely supported measure by research institutes, think tanks, and independent organizations.

In short, red lines are the most practical step the global community can take now to prevent severe risks while allowing safe innovation to continue.

Can you give concrete examples of possible red lines?

The red lines could focus either on AI behaviors (i.e., what the AI systems can do) or on AI uses (i.e., how humans and organizations are allowed to use such systems). The following examples show the kind of boundaries that can command broad international consensus.

Note that the campaign does not focus on endorsing any specific red lines. Their specific definition and clarification will have to be the result of scientific-diplomatic dialogues.

Examples of red lines on AI uses:

Nuclear command and control: Prohibiting the delegation of nuclear launch authority, or critical command-and-control decisions, to AI systems (a principle already agreed upon by the US and China).
Lethal Autonomous Weapons: Prohibiting the deployment and use of weapon systems used for killing a human without meaningful human control and clear human accountability.
Mass surveillance: Prohibiting the use of AI systems for social scoring and mass surveillance (adopted by all 193 UNESCO member states).
Human impersonation: Prohibiting the use and deployment of AI systems that deceive users into believing they are interacting with a human without disclosing their AI nature.

Examples of red line on AI behaviors:

Cyber malicious use: Prohibiting the uncontrolled release of cyberoffensive agents capable of disrupting critical infrastructure.
Weapons of mass destruction: Prohibiting the deployment of AI systems that facilitate the development of weapons of mass destruction or that violate the Biological and Chemical Weapons Conventions.
Autonomous self-replication: Prohibiting the development and deployment of AI systems capable of replicating or significantly improving themselves without explicit human authorization (Consensus from high-level Chinese and US Scientists).
The termination principle: Prohibiting the development of AI systems that cannot be immediately terminated if meaningful human control over them is lost (based on the Universal Guidelines for AI).

Red lines on AI behaviors have already started being operationalized in the Safety and Security frameworks from AI companies such as Anthropic’s Responsible Scaling Policy, OpenAI’s Preparedness Framework, and DeepMind’s Frontier Safety Framework. For example, for AI models above a critical level of cyber-offense capability, OpenAI states that “Until we have specified safeguards and security controls standards that would meet a critical standard, halt further development.” Definitions of critical capabilities that require robust mitigations would need to be harmonized and strengthened between those different companies.

Are international AI red lines even possible?

Yes, history shows that international cooperation on high-stakes risks is entirely achievable. When the cost of inaction is too catastrophic, humanity has consistently come together to establish binding rules to prevent global disasters or profound harms to humanity and global stability.

The Treaty on the Non-Proliferation of Nuclear Weapons (1970) and the Biological Weapons Convention (1975) were negotiated and ratified at the height of the Cold War, proving that cooperation is possible despite mutual distrust and hostility. The Montreal Protocol (1987) averted a global environmental catastrophe by phasing out ozone-depleting substances, and the UN Declaration on Human Cloning (2005) established a crucial global norm to safeguard human dignity from the potential harms of reproductive cloning. Most recently, the High Seas Treaty (2025) provided a comprehensive set of regulations for high seas conservation and serves as a sign of optimism for international diplomacy.

In the face of global, irreversible threats that know no borders, international cooperation is the most rational form of national self-interest.

Are we starting from scratch?

No. Red lines on AI already exist and are gaining momentum. Some examples include:

Global Norms and Principles: The UNESCO Recommendation on the Ethics of AI (2021), adopted by all 193 member states, explicitly calls for prohibiting the use of AI systems for social scoring and mass surveillance.
Binding Legal Frameworks: The Council of Europe's Framework Convention on AI is the first-ever international treaty on the subject, establishing binding rules for its signatories to ensure AI systems are compatible with human rights, democracy, and the rule of law. The EU AI Act creates an “unacceptable risk” tier for applications that are strictly banned within the European Union.
National Policies: America’s AI Action Plan explicitly calls for the creation of an evaluation ecosystem, upon which we can set thresholds.
US-China Bilateral Dialogue: In 2024, both Heads of State agreed that AI should never make decisions over the use of nuclear weapons and that humans need to remain in control.
Scientific Consensus: The Universal Guidelines for AI (2018), supported by hundreds of experts and dozens of civil society organizations, establishes a clear “Termination Principle”, an obligation to shut down any AI system if meaningful human control can no longer be ensured. The IDAIS Beijing Statement on AI Safety (2024), a consensus from leading international experts, explicitly calls for several red lines, such as restricting AI systems that can autonomously replicate or evolve without human oversight.
Industry Commitments: At the AI Seoul Summit, leading AI companies made the Seoul Commitments, formally pledging to “set out thresholds at which severe risks posed by a model or system, unless adequately mitigated, would be deemed intolerable”. Several leading companies have also adopted internal governance frameworks designed to restrict AI deployment or development and implement safeguards if specific high-risk capabilities are discovered.

Our call is to build upon the principles and precedents set by these powerful efforts, forging a coherent, binding, and universally adopted international framework. A further discussion of the existing list of emerging red lines is available here.

Who would enforce these red lines?

There is no single global authority for AI, so enforcement would likely combine different levels of governance, including:

International Treaty: A binding agreement would harmonize rules across countries. This prevents a regulatory arbitrage "race to the bottom" where companies could evade regulations by moving to less strict jurisdictions.
National Governments: Nations would be tasked with translating the international agreement into domestic law. This would involve creating regulatory bodies to license advanced AI systems, conduct mandatory safety audits, and impose severe penalties for violations within their jurisdictions.
International Technical Verification Body: An impartial international body, modeled on organizations like the International Atomic Energy Agency (IAEA), could develop standardized auditing protocols and independently verify that AI systems from any company or country comply with the agreed-upon red lines. The International Network of AI Safety and Security Institutes is well-positioned to play a role in this process.

Why 2026?

Waiting longer could mean less room, both technically and politically, for effective intervention, while the likelihood of cross-border harm increases sharply. That is why 2026 must be the year the world acts.

The pace of AI development means that risks once seen as speculative are already emerging — including biological misuse risks (Williams et al, 2025), systems showing deceptive behavior, and even resistance to control (Greenblatt et al., 2024). In their own assessments of the biological misuse potential, leading AI companies place their newest frontier models on a medium (Anthropic, 2025) to high (OpenAI, 2025) risk spectrum.

AI’s coding capabilities are also improving rapidly, meaning superhuman programming ability may soon be possible, accelerating AI progress even further. According to recent safety evaluations (Google DeepMind, 2024), experts forecast that AIs could become capable of autonomously replicating and proliferating on the internet as early as late 2025, with the median forecast landing in 2027.

What should the next steps be?

Launched ahead of the 80th session of the UN General Assembly, the campaign seeks to encourage diplomatic action toward concrete pathways for international agreements on red lines for AI.

Several complementary pathways could be envisaged:

A group of pioneering countries or a “coalition of the willing,” potentially drawing on countries already engaged in the G7 Hiroshima AI process, could advance the concept of AI red lines across the G7, G20, and BRICS agendas.
The newly established UN Independent Scientific Panel on AI could publish a thematic brief articulating scientific consensus on technically clear and verifiable red lines, with technical contributions from the OECD.
Building on this groundwork, states could use the AI Impact Summit in India in February 2026 to endorse initial red lines for AI. Such red lines could build on the Seoul Commitments by translating voluntary corporate pledges into shared risk thresholds that, in turn, could be embedded in national regulation and public procurement.
The UN Global Dialogue on AI Governance could lead a global consultation with scientists, civil society, and industry to define a set of clear and verifiable red lines, to be summarized in an outcome document at the July 2026 Dialogue in Geneva.
By the end of 2026, either (i) a UN General Assembly resolution could be initiated, noting and welcoming these red lines and inviting negotiations, or (ii) a joint ministerial statement by an alliance of willing states could launch negotiations for a binding international treaty.

Any future treaty should be built on three pillars: a clear list of prohibitions; robust, auditable verification mechanisms; and the appointment of an independent body established by the Parties to oversee implementation.

What Signatories Say

Signatories

Joseph Stiglitz

Juan Manuel Santos

Maria Ressa

Daron Acemoğlu

Mary Robinson

Ahmet Üzümcü

Geoffrey Hinton

Yoshua Bengio

John Hopfield

Yuval Noah Harari

Jennifer Doudna

Enrico Letta

Csaba Kőrösi

Giorgio Parisi

Sir Oliver Hart

Ya-Qin Zhang

Stuart Russell

Joseph Sifakis

Kate Crawford

Wojciech Zaremba

Yanis Varoufakis

Peter Norvig

George Church

Ian Goodfellow

Jakub Pachocki

Andrew Chi-Chih Yao

Gillian Hadfield

Sir Stephen Fry

Yi Zeng

Gustavo Béliz

Baroness Beeban Kidron

Gbegna Sesan

Anna Ascani

Dan Hendrycks

Dawn Song

Gary Marcus

Audrey Tang

Maria João Rodrigues

Rachel Adams

Adetola A.Salau

Thibaut Bruttin

Maria Chiara Carrozza

Daniel Kokotajlo

Lord Tim Clement-Jones

Xue Lan

Nicolas Miailhe

Marc Rotenberg

Laurence Devillers

Jason Clinton

Jean Jouzel

Riccardo Valentini

Robert Trager

Brice Lalonde

Senator Scott Wiener

Sören Mindermann

Brando Benifei

Miguel Luengo-Oroz

Sergey Lagodinsky

Max Tegmark

Sneha Revanur

Roman Yampolskiy

Mark Nitzberg

Niki Iliadis

Charbel-Raphaël Segerie

Organizers

Partners

Signatory Organizations

Sign the call

Sign as an individual

Sign as an organization

Frequently Asked Questions

Media Inquiries

Media Coverage