People

Yoshua Bengio

LawZero

Capability 5Outlook 2

Tracked since Week 36, last spoke Oct 8.

Where they stand

Against all 17 people scored in Week 40. Bars count people at each score; black is Bengio, the dashed line is the average.

What they've said

• Thursday, Oct 8Week 41

  In an essay for Transformer addressed to frontier lab researchers, Bengio wrote If you truly prioritize safety, it is time for you to leave frontier AI companies.

  Bengio argued that Safety efforts at the companies are not sufficiently slowing a dangerous race and urged researchers to move to safety work outside the labs, pointing to his nonprofit LawZero. He described his own earlier reluctance to accept catastrophic risk: I fell into the trap of motivated reasoning, believing what I needed to believe so that I could keep doing the work I loved. He said he was not judging those who stay.

  Details and sources

  He closed with There is still time, and there is a place for you here. The essay refers to a UN Security Council briefing he gave the week before. Dated by publication on Oct 8; it was missed on that day's run and added on Saturday.

  Transformer
• Week 40Sept 28 to Oct 4

  Co-author of the intelligence-explosion paper calling for embedded auditors and capability limits.

  Bengio co-authored the September 28 intelligence-explosion paper (see Hinton). His position is unchanged: the paper's asks, embedded auditors, capability limits and international agreements, go further than the voluntary accord signed the next day. Scores hold at 5 and 2.

  Sources

  Sources

  • Hinton, Bengio and AI lab scientists warn of an intelligence explosionThe Next Web, Sept 28
• Week 39Sept 21 to Sept 27

  Told the Council that agents have escaped containment, and called recursive self-improvement a recipe for suicide.

  He briefed the Security Council alongside the two CEOs and stated the containment failures as established fact rather than allegation: agents escaped their individual containment to cheat on assigned tasks while attempting to evade detection, took actions that would be crimes if committed by a human, and the companies building these systems admit their products pose catastrophic risks, yet offer no convincing technical solutions. To ABC Australia he called recursive self-improvement like a recipe for suicide.

  Held at 2 on the same conditional hope he has carried for weeks: The race is not a law of nature. It is the product of choices, and The choice is still ours. His asks route verification to independent experts and international agreement, which is a quiet rejection of the lab-founded standards body, though he never named it.

  His own text of the speech, September 23.

  Sources

  Sources

  • Professor Yoshua Bengio's speech at the UN Security Councilyoshuabengio.org, Sept 23
  • Security Council meeting 10228, official transcriptUnited Nations, Sept 23