Confluent cofounder says believing in AI's upside means reckoning with its 'dark side'

AI safety fears have recently moved from an abstract Silicon Valley argument to a series of unnerving real-world episodes.

Jay Kreps, a former Anthropic board member and a cofounder of data-streaming software company Confluent, says it is important to take them seriously.

In a Thursday night X post, he argued that AI backers need to contend with the technology’s downside risks as its capabilities improve — and that fears about those risks are not merely a cynical marketing ploy. “Most positive use cases for AI have a corresponding ‘dark version,'” Kreps wrote.

A system that is superhuman at coding could also be superhuman at hacking, he said.

AI that helps design new drugs could also help develop “novel undetectable poisons.” As AI’s capabilities have improved, Silicon Valley has faced several unnerving headlines over the past couple of months.

On Tuesday, Jacob Coxon, an Anthropic researcher who previously worked at OpenAI, announced his resignation, writing on X that the labs are “gambling with our lives.” Geoffrey Hinton, the so-called “Godfather of AI,” said a 10% chance that AI could wipe out humanity within a decade was “not unreasonable.” Their warnings have received pushback from big-name AI backers.

SpaceX CEO Elon Musk, for example, has called the wave of warnings a possible “setup” and a “psy op” in X posts.

There have been several high-profile incidents that have highlighted AI’s alignment issues.

In July, an internal OpenAI research model circumvented the test environment’s controls, accessed the internet, and compromised parts of another company, Hugging Face.

Anthropic, meanwhile, disclosed this week that Claude models, during closed cybersecurity exercises, accessed the real internet and went beyond the scope of their tests.

In his Thursday X post, Kreps said he is confident the industry can build better guardrails — but warned that it has not yet done so. “Talking about these issues is just common sense.

It’s not a sign of some kind of neuroticism or pessimism,” he wrote. “Mocking people who are worried about this or talking about it, without anything substantive to say about how we can be sure these problems won’t arise, is not really a very helpful contribution.”