Comment: Oh, the concerned “AI” bros

Contents

We have been reading a lot recently about the concerned “AI” bros ([Editor’s note:]tante puts “AI” in quotation marks because the term is misleading, is associated with vague expectations, and has nothing to do with real intelligence): Dario Amodei and Sam Altman both talk about slowing down the development of “frontier AI”, and even Elon Musk agrees:

An unholy alliance that should give pause for thought, especially since the three gentlemen involved deeply despise each other. Is the concern of the “AI” bros for the future so great that it overshadows all conflicts? Is the current case of Jacob Coxon, who quit his job at Anthropic out of fear that “‘AI’ could wipe us all out,” also an expression of this fear of real consequences? Coxon even appears on US television to emphasize that humans have few countermeasures left because “the AI” simply spreads like a virus to other machines if one wanted to shut it down. Is this a real scenario?

Of course not. And the reporting is unfortunately really annoyingly bad.

Warnings years ago

First, it is of course important to know that Anthropic split off from OpenAI because OpenAI was not cult-like enough for the splitting individuals: Although Amodei officially describes it as different ideas about the safety of “AI” and the question of how to increase the capabilities of “AI”, the real motives seem to shine through even in the statement. Anthropic was founded by people from the Effective Altruism (EA) bubble, who truly believe that LLMs are simultaneously an “existential Risk” (i.e., a risk to the survival of humanity, here was a breaking point with OpenAI’s view) and a new god that emerges if one only applies the “Scaling Laws”, i.e., building ever larger systems. This system should then largely be handed over control so that it can solve almost all of humanity’s problems.

A commentary by tante

Jürgen Geuter (better known online as “tante”) studied computer science and philosophy and works as a research director focused on the research, implementation, and testing of new technologies. As an independent researcher, author, sociotechnologist, and speaker, he explores topics at the intersection of technology, society, and politics.

This way of thinking had (and has) a certain influence at OpenAI too, but a more pragmatic perspective still prevails there: In the end, they hope to find a business before Sam Altman runs out of stories. Anthropic, on the other hand, is much more “on a mission.” So the agreement is unusual even from this perspective, requiring a closer look. Anthropic, in particular, is in a dilemma here, which has always been resolved in favor of more development.

Now we have all played the story of “our magical AI models are too dangerous/powerful to release” several times. We all remember when even GPT-2 was too powerful for a public release in 2019. A model that no one is interested in anymore today because it is too bad for any usual use case.

But the slash is important: The claim of “our model is super dangerous” is primarily the claim that a model is powerful. It is an advertising message to keep the hype around LLMs going.

Because where does the industry stand right now?

Anthropic and OpenAI want to go public soon, partly because it is becoming increasingly difficult for them to get other funding for their highly loss-making companies, and the large ETFs and pension funds would be very welcome.

Although Anthropic has claimed to be profitable for several months, no one has seen real numbers yet. None of the “AI” companies keep accounts according to accepted criteria. Instead, they have established the key figure, ARR (Annualized Recurring Revenue). This means: The companies pick out 4 good weeks, multiply them by 13, and claim that this describes the year. A creative form of economic calculation, to say the least.

In addition, Anthropic has recently received highly subsidized compute from Elon Musk so that Musk’s Colossus datacenter – that is, the one in Memphis with the illegal methane generators – is utilized a little more. It is not surprising that Anthropic can find a time window in this way in which the numbers do not look disastrous. By offering Anthropic compute resources at extremely favorable (probably not cost-covering) conditions, Elon Musk is thus taking over at least part of Anthropic’s losses.

For the IPOs, the “AI” companies currently need every positive message to overcome the general discontent about “AI” and data centers: “AI” is incredibly unpopular (in a poll in March in the US, even less popular than Donald Trump and ICE). It takes a miracle to ensure that the IPO goes as planned. Because that means it has to be the biggest IPO of all time; otherwise, it would be considered a massive weakness. SpaceX’s IPO, where the stock plummeted after a short run, has also made investors more cautious.

The “AI” companies therefore need press. Press that ensures that people believe that “AI” – i.e., the systems that, after an investment and a PR campaign the likes of which no technology in human history has ever received, still do not function reliably – represents “the future.” And in that context, it is of course good to present oneself as “cautious” and “prudent” to counteract the vibe the industry has conveyed so far.

But there is an even more direct economic argument:

The flattening performance curve

“AI” models are getting better and better, according to some benchmarks. But in practice, many of these improvements primarily manifest in higher costs: the ever-larger models, which increasingly call themselves in circles (“reasoning”), consume more and more resources for largely similar performance. Because every marginal improvement of the models comes with exponential costs, not only in inference (i.e., the use of the models), but of course also in training: training ever larger models causes development costs to rise ever higher – even if specialized chips bring certain efficiency gains.

The “AI” companies therefore have exploding costs for new versions of a product that do not feel massively better for users but are significantly pricier to operate. This is a problem in an already loss-making industry where more and more customers are increasingly restricting their “AI” spending. The market to be shared is no longer expanding as bet on but shows signs of possibly even shrinking. At the same time, the cost of one’s own product continues to increase. And none of the “AI” labs currently have truly new ideas in the pipeline that could enable real leaps in quality.

In this situation, the coordinated narrative of “we would rather not train new models for a while because they are too dangerous” is naturally useful: One can save the expensive development costs for a while to make one’s own numbers look a little better for the IPO. At the same time, one hopes that Nvidia or someone else will find some efficiency gains that might make the current plateau economically viable (the good old “thoughts and prayers” business strategy).