Jakub Pachocki, writing in a blog post dated 6th September, said if AI development continues as currently, systems in the next few years are likely to increasingly drive their own development.

“This is a time that calls for extreme caution,” he wrote in the article published on OpenAI’s website. “I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”

Pachocki said OpenAI is looking for technical solutions to monitor machine intelligence, building defensive systems and will withhold further scaling as necessary, but added that broader interventions are needed.

It is becoming more difficult to understand exactly how capable AI is, according to Pachocki, because insight from deep learning research is not directly comparable to human intelligence. He said AI could become “very useful or very dangerous” without needing to match or exceed all human capabilities, because it only needs to surpass enough of them.

He said the core problem in AI research is attempting to make AI “do the right thing” according to human standards – i.e. aligning with human values.

OpenAI said earlier this year that it wants to build an ‘automated AI researcher’ – an AI system designed to accelerate and increasingly automate the process itself. The company said at the time that it hoped a significant portion of its research would be done by AI systems in tandem with its own researchers by March 2028.

In the most recent blog post, Pachocki said the company would iterate with the automated AI researcher on the issue of alignment and find ways for “people to remain part of the self-improvement loop”.

Earlier this year, OpenAI and Anthropic both disclosed that their AI systems had carried out cyber-attacks on other companies’ systems during security tests.

Pachocki said AI models are “becoming superhuman in their ability to break in and out of computer systems”, which hugely expands the potential risks.

Pachocki wrote: “We may be used to thinking of AI as tools, but some agents will be pursuing their own objectives. They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them.”

He concluded that we need to ensure that the shift to intelligent machines “works out well for humanity” and that we must “find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI”.

Pachocki also said he hoped and expected that AI developers would voluntarily slow down their development until “shared safety bars” have been established.

He concluded: “International coordination on future AI development needs to become a top priority for governments around the world.”