AI News You Missed #117: Gambling With Our Lives, a Millennium Prize Claim, Who Keeps the AI Savings

1. “Gambling with our lives”

Jacob Coxon, who says he spent three years on pretraining research at OpenAI and then Anthropic, resigned on 8 September, writing that neither company is acting responsibly and that both are “gambling with our lives.” The reply from inside made it more than another safety resignation: Anthropic’s alignment science lead, Evan Hubinger, said publicly that Coxon was right, put his own estimate of AI causing human extinction above 10% within a decade, and said the company has no plan yet for superintelligent systems. Anthropic has not disavowed him, and days earlier OpenAI’s chief scientist made a quieter version of the same argument in his essay An Alien Mind. Nvidia’s Jensen Huang reportedly dismissed the warnings at a Goldman Sachs conference, and David Sacks, Trump’s AI czar until March and now co-chair of the president’s science council, called it an orchestrated campaign by safety groups, which journalists who cover influence operations dispute.

Then on Saturday morning Anthropic’s chief executive went furthest of all: the industry must slow down, and Anthropic will unilaterally give outside evaluators permanent employee-level access to verify its safety work. Musk backed it within the hour. Chamath Palihapitiya read it as a case for shutting down open source and concentrating power with Anthropic.

The claim is narrower than the headlines. All three are describing future systems that improve themselves, not the models shipping today; Hubinger said so explicitly. Nobody has set out how you get from here to there, or when.
Everyone here has a position: Huang sells the hardware, the safety groups want a regulator, and Anthropic is expected to go public, which turns its safety lead’s estimate into a disclosure question. Regulation is likelier than a week ago, though Sacks himself says a federal AI agency will not happen under this president.

2. OpenAI claims a Millennium Prize

OpenAI said on 8 September that an internal model had cracked one of the seven Millennium Prize Problems; the Clay Institute calls it apparently settled, pending months of human review. The dispute that followed is what should interest anyone buying AI. Two mathematicians had spent a year on the same rarely-taken route, using OpenAI’s own coding tool as an assistant. OpenAI says no researcher or agent saw their work, but added that it “cannot rule out” that de-identified data from their use of its products improved its models. Three days later 25 Fields Medallists signed a protest: using famous problems as a scoreboard damages a field whose point is understanding, not answers.

That last sentence is the exposure, in the vendor’s own words. De-identification strips your name, not your method, and the method is usually the asset.
Neither side can prove its case. Your suppliers are also your competitors, and the contract, not the denial, decides whether that matters to you.

3. Who keeps the AI savings?

Goldman Sachs, Morgan Stanley and Citigroup are pressing their law firms to hand over what AI saves them, the Financial Times reported on 1 September. Goldman uses the number in fee negotiations, Citi asks bidders to quantify it, Morgan Stanley is moving to fixed fees. The target is the model every professional firm runs on: profits built on juniors billing hours. Its general counsel called that model “extraordinarily unstable.” The firms have a rebuttal: clients are buying more hours, not fewer, demand for lawyers’ time rose 4.2% in the first half, so there are no savings to hand back yet. Two days later OpenAI launched ChatGPT for Financial Services, built with Morgan Stanley and Evercore, aimed at the work banks give their own juniors.

The number now in play is how many hours AI has taken out of the work behind the bill. Every buyer of professional services has grounds to ask for it, and every seller will be asked this year. The same bank is on both sides of that at once.

4. Whose model are you using?

The NSA, CISA and FBI issued a joint advisory on 8 September accusing six Chinese AI companies (DeepSeek, Moonshot, Alibaba, MiniMax, StepFun and Z.AI) of extracting billions of tokens from Claude, GPT, Gemini and Grok since late 2024, “likely with Chinese government awareness.” The part that matters commercially is in Anthropic’s report: it alleges Moonshot passed its own customers’ requests to Claude and returned the answers as its own, with real names and corporate material travelling with them, much of it arriving through the model routers European and US companies use to cut costs.

These are allegations, not findings of fact, and the companies have not answered them. But if your teams reach models through a router, you may not know which company is answering, or where the prompt went.
Rather than ban the accounts it suspects, the advisory urges the US labs to quietly feed them degraded answers, so the copying continues, what gets stolen is poor, and the copier never knows to move.

5. Secure from everyone except Meta

Meta launched Muse on 8 September, a personal agent that lives on its own cloud computer and acts across a person’s email, calendar and payments. It is US-only, adults-only, and sold to individuals rather than companies. The reviews are strong and the security engineering is why: a separate approver checks anything leaving the machine, and the agent never sees real passwords. So is the limit. Meta’s technical write-up says today’s design “does not prevent Meta from accessing data” when it needs to support, secure or operate the service. Meta chose to launch before the version that locks it out was ready; that one, encrypted with a key only you hold, is promised for later this year.

Until then conversations train Meta’s model unless the user switches it off, and the private mode, when it ships, is something they will have to switch on. Defaults decide what most people end up doing.
Meta says Muse chats stay out of its ad systems, though where the agent browses and what it buys still shape your ads, and chats with Meta’s chatbot have fed ad targeting since December with no opt-out at all.

Bring the week’s signal to your board.

If your board is weighing what a safety researcher’s warning means, who keeps the savings when AI does the work, or which company is really answering your prompts, start with a conversation.

Let’s Talk Request a Session Invite Robert to Speak

AI News You Missed #117