Cognition's CEO says token spend leaderboards are 'directionally correct' — but some people get carried away

Token leaderboards have a right idea, wrong execution problem, says the CEO of one red-hot AI coding startup.

On an episode of the “Founders” podcast released on Sunday, Cognition’s Scott Wu said there are better ways to incentivize employees to use AI. “It is directionally correct, but I think there are definitely some places where people have gotten carried away,” Wu said. “People are like, ‘We rank our engineers by how many tokens they’re spending.’ Well, let’s try and rank people by how much output they’re actually producing.” Cognition was founded in 2023 by Wu, Steven Hao, and Walden Yan and is best known for Devin, an autonomous AI software engineer.

The San Francisco-based company has attracted backing from investors including Founders Fund, Khosla Ventures, Elad Gil, and Pear VC.

In May, it raised more than $1 billion at a $26 billion post-money valuation, making it one of the most valuable AI coding startups globally.

On the podcast, Wu acknowledged that compute is expensive.

But he said if engineers can ship three times more than they would without AI, it is “clearly worth it.” Still, he added that companies need to ensure they tie rewards to less lofty metrics.

Specific outcomes could include how many tickets are resolved and how much cheaper and faster a project can be.

Wu is the latest tech leader to critique tokenmaxxing — using tons of AI tools like Claude, Codex, and Cursor to boost productivity and get ahead on internal AI use dashboards and reviews.

Earlier this month, the chief technology officer of legal AI startup Legora said employees should not be rewarded just for using AI. “A lot of people, say, get a leaderboard and bring up token usage at performance reviews,” said Jacob Lauritzen on a podcast. “That leads to tokenmaxxing, which is people just burn tokens just to look good.” “That’s a really stupid way to do anything,” he added.

At a Bloomberg conference this month, Andrew Feldman, the CEO of Cerebras Systems, said that the idea of giving employees unlimited tokens was “boneheaded from the get-go.” “You don’t need a Ferrari to go to the grocery store, right?

Use a lower-cost open source model,” he said about being more efficient with tokens. “What we’re learning is how to shop at Costco.”