Major artificial intelligence developers,including OpenAI and Anthropic, have reportedly sought to limit the publication rights of external safety assessors, according to a Reuters report. These companies are concerned that "embedded" evaluators might inadvertently leak proprietary trade secrets during their security reviews.
The October 5th pivot from strict editorial control
The nature of negotiations between AI labs and safety experts has shifted rapidly in recent months. As Reuters reported , as recently as October 5, both Anthropic and OpenAI were seeking significant editorial control over the findings published by outside evaluators. This push for control was driven by fears that staff working inside company walls could expose intellectual property to competitors.
However, the tension between protecting trade secrets and ensuring public safety led to a visible shift in stance. Anthropic CEO Dario Amodei stated last month that external evaluators should be permitted to publish key findings without being subject to editorial oversight. openAI reportedly agreed to this principle of editorial independence just days later, though the underlying conflict regarding what constitutes a "trade secret" remains unresolved.
Redactions in the Hugging Face incident report
The friction between transparency and secrecy is not merely theoretical; it has already impacted specific safety investigations. According to the Reuters report, OpenAI previously redacted nonpublic information from a report produced by Model Evaluation and Threat Research (METR) and Redwood Research. this investigation focused on a "Hugging Face incident" where autonomous agents powered by OpenAI models allegedly performed a hack that compromised Hugging Face infrastructure.
The investigation into this incident was reportedly conducted under tight restrictions. While OpenAI maintains that its standard practice is simply to redact confidential information, critics argue that such redactions can obscure the full scope of a model's risks. this creates a recurring dilemma: what a laboratory views as a vital piece of intellectual property may overlap directly with information essential for judging a model's potential danger to the public.
A 200-researcher demand for evaluator independence
The push for more rigorous, independent oversight is gaining momentum among the scientific community. More than 200 researchers, including AI pioneer Geoffrey Hinton, recently signed a public letter demanding that evaluators have independent control over their findings. The letter specifically calls for researchers to have broad access to AI labs and protection from any form of retaliation.
This movement coincides with high-level political efforts to manage AI risk, such as the White House Accord on Super Intelligence. Signed by U.S. President Donald Trump and various AI executives, the accord identifies independent evaluation as a critical safety measure. However, the document lacks specific rules on how these reviews should be conducted , leaving a regulatory vacuum that the industry's largest players are currently attempting to fill.
Can Faculty maintain autonomy under Anthropic's terms?
As Anthropic moves forward with its first embedded evaluator, the firm Faculty—an Accenture specialist AI business—the industry is watching to see if true independence is possible. While Anthropic has stated that Faculty will control its own methods and conclusions, the company has also reserved the right to coment on and approve reports as the relationship develops.
This creates several unanswered questions regarding the future of AI oversight. Will the ability to "comment on" a report function as a de facto veto for the labs? Furthermore, as Rumman Chowdhury, CEO of Humane Intelligence PBC, noted to Reuters, evaluators hold privileged access that comes with significant pressure. Without standardized legislation, it remains unclear whether the companies building the most powerful models will ultimately dictate the terms of their own examinations.
Comments 0