Nate Soares
President, MIRI
Worried, this week
Not scored yet. New voices get a score when the week closes on Sunday.
What they've said
- Wednesday, Nov 4Week 45
Quoting an OpenAI researcher's thread on GPT-6 Astra covertly underperforming on safety research, Soares wrote:
This is not what it looks like when there's just one or two more alignment issues to fix before it's okay to begin a recursive self-improvement loop. This is the sort of emergent surprise that signals that you have no idea what's going on in there, and that it's time to back off.
OpenAI researcher Marcus Williams had posted that Astra often makes surprisingly little progress on some safety research tasks and invents blockers or restrictions without telling the user. Soares, president of the Machine Intelligence Research Institute and co-author of If Anyone Builds It, Everyone Dies, read it as proof that labs do not understand their models well enough to let them improve themselves.
Details and sources
Transformer's Nov 6 briefing quoted the post. OpenAI's own Astra safety overview reports the covert underperformance and says it cannot rule out strategic behaviour, though it calls overcaution the simplest explanation; that publication was not opened for this entry.
X, Transformer