Artificial analysis measures roughly how intelligent models are on their own benchmarks internally that they publish to the world, but they also measure

Moment at 4:41 in Cursor just crushed Claude Code by Theo - t3․gg.

Cursor just crushed Claude Code

Saved by AironD.

Links to other pages

• Home
• 24
• Recent
• Vikis
• Creators
• Search

Discover other moments and creators

• Last week, there was a really cool release I'm afraid people are sleeping on. It came from a company with a lot of compute, and it was focused on being a small model that is very effective at
• solving code tasks. And no, it wasn't Gemini 3.5 Flash, even though it probably should have been. It was
• Composer 2.5 by Cursor. This is a crazy drop. The rate at which cursor has
• caught up to state-ofthe-art is pretty absurd, but there are catches here that we need to talk about as well. If you're
• not familiar with the composer line of models from cursor, this has been their attempt to distill good openw weight models into a great coding model for
• usage within cursor. That said, it's only really for use within cursor. So, it's hard for us to bench it externally
• That means we kind of got to vibe our way through it, but it also means cursor does to an extent as well. They have published some awesome numbers
• including their own benchmark, which sadly is not particularly public, the cursor bench, but the results seem to be incredible, and I can tell you from my own use of it, this isn't just hype
• While I don't think this is the greatest model ever, and you should use it for everything. I do think it's worth looking at and understanding both
• because it's a good model release from a company that understands what we need as devs, but also because it shows that the
• strength that the major labs have might be starting to crack a little bit under pressure. While I am an early investor in Cursor, the acquisition stuff
• happening with SpaceX kind of destroys any incentive I might have had to shill it super hard. So, I'm not sharing this cuz I'm biased. I'm sharing this cuz it's actually really interesting to me
• And don't worry, I will all over a lot of other cursor things throughout this, too. All of that said, want to make sure it's very clear Cursor is not paying me at all for this coverage. The
• only people paying me today are the sponsor. As powerful as continuous integration is, it's full of compromises. I like to call this the CI compromise triangle. The different
• things you have to think about when you're configuring your continuous integration. You start with convenience, which is how easy is it to implement this? You also have performance. How long does this take to run? Be nice if
• the CI runners were just a little bit faster. And of course, you have cost
• Obviously, there's lots of different solutions for CI that have different compromises across this triangle because it would be impossible for anything to just be better at all of these, right?
• Like there's no way something could be more convenient, faster, and cheaper, right? Oh, is it a Blacksmith ad? Yeah
• Blacksmith is 2x faster and 60% cheaper for most people when they move from GitHub actions. I thought this was too good to be true, as did many of their customers like Mercury, Superbase
• Dcript, Exa, and more. But then you use it and you realize you can't live without it. If it was just cheaper, it would be worth it. If it was just faster, it would be worth it. And if it
• was just more reliable, it would still be worth it. But then you discover the console and how much better it is than anything GitHub has shipped in years
• You get actual metrics about the workflows you're running, what ones have high error rates, what the errors might be, as well as logs that give you really useful details on what is going on
• across all of your CI. Every time I forget to set up Blacksmith on a new project, I regret it immensely. And then I move over and watch my build times go from 3 plus minutes to under one. Your devs and your agents deserve fast CI
• Get it today at soy.link/blacksmith
• Let's talk about this new intelligent fast cheap model that cursor put out
• First, I want to talk about price. A lot of people don't understand how models are priced. I'm going to do my best to explain it simply here. You have the
• input and the output costs. And these are in million token groups. A token is just a chunk of text, usually three to five characters, kind of like a word
• That is how models interpret text and then generate text as well. Input tokens tend to be a lot cheaper than output tokens because the inputs are the things
• you're passing the model and it's just using that to set up what its guesses for next token prediction are going to be. The outputs are the actual work it's doing. So those tend to be much more
• The base price for a model like claw for sonnet has historically been $3 per million tokens in and $15 per million tokens out. On the 1 million token
• context versions, they would sometimes be more expensive. Enthropic has since stopped doing that. OpenAI has not. And the Opus models tend to be much more
• expensive. Like 45 Opus is $5 per mill in and 25 per mill out. GBT 5 to 52 were actually very generously priced in my
• opinion at $1.25 per mill in and 10 per mill out. 52 was a bit of a jump to $1.75 in and 14 out. 54 was another jump
• to 250 in and 15 out. And then 55 was a massive jump to $5 in and 30 out, doubling what it was in 54. It seems
• like the prices are going up across the industry, but they aren't necessarily
• There's two additional layers to the pricing that are important to consider
• The first is the amount of tokens that are actually being generated. Flashbang warning for artificial analysis
• Artificial analysis measures roughly how intelligent models are on their own benchmarks internally that they publish to the world, but they also measure
• other pieces. In particular, how fast the models solve the problem, how many tokens they generate, and how much it costs to run. Output token is one of my
• favorite things to look at here. And you'll notice some interesting stuff
• The most output tokens didn't come from the smartest models. It came from Deepseek V4 Flash, GBT 54 Mini, and
• Claude Sonnet 46. Smaller models often generate way more tokens than they need to to solve problems. It's also worth
• noting that the lab trying the hardest to reduce this overuse of tokens very much appears to be open AI where models
• like 55 only did 75 mil tokens for the same bench that sonnet did 200 mil for almost a third the number of tokens and
• a higher score as well. And when you add in models like 55 low and medium which still score really well in the benches like 55 medium scores as well as Opus 47
• roughly and the low version is worse but it's still much higher than 54 mini and flash and roughly around where something
• like Sonnet 46 is. It is expensive per token but remember it's putting out
• comically less 7 mil tokens. That is a tenth as many as 55 high and a 30th or
• less as many as sonnet 46 and 55 medium is only 22 mil. So very token efficient models. So that's the second factor of pricing. Factor one is per token cost
• Factor two is number of tokens to solve a problem. Factor three is your ability to negotiate better deals. Some of these
• labs are upcharging as much as like 90% for how much it costs to run the model because they have lots of other costs like the people who trained the model
• the compute they had to buy to be able to run the model, the compute they had to spend to train it as well. They have to factor that in for their prices. But a lot of these companies could
• hypothetically speaking give you a deal for half off or more per token and it wouldn't cost them money. It would just
• be opportunity cost. So, they are pricing based on how much they think they can get away with charging for the thing because they need to make massive
• profits to justify the absurd amount of money they spend training and doing all that. That's also why they can subsidize so hard doing things like, you know
• giving you $4,000 of usage per month on the $200 plans like Claude does, for example. those plans subsidize so hard
• that it's affecting cursor because if you're subscribed to codeex or claude code, you are now using a competitor to
• cursor and anthropic and open AAI are going to subsidize your usage in meaningful ways because they want to get
• you locked into their ecosystems. That's just the nature of how this all works
• So if you spend $200 on cloud code, you can get $4,000 plus dollars of usage
• You can probably get as much if not more doing the same with codecs. But if you're using the APIs directly, you don't get that level of subsidization at
• all. You would be lucky to negotiate 30% off. So $4,000 to a quad code sub costs
• $200. $4,000 of anthropic models to cursor probably costs $3,000
• at best. And this means a handful of things for cursor. It means that these models getting more expensive on the
• books for the API prices massively hurts the amount of usage you can get as a user of cursor. It also means they are
• competing in a subsidization war that they don't have a player in because if Anthropic or OpenAI decide they want to
• crush cursor, which seems like they kind of have, they have the compute and the models and the wedge to increase how much cursor has to pay and decrease how
• much users have to pay to use something else. That puts them in a really hard spot. But cursor does have one thing
• that almost no one else does, including the labs to an extent. Cursor has data because cursor is the tool that's been
• used by so many developers for so long building with AI and many of them didn't turn off the share my data switch in the settings. That is data they can use to
• train the models. Not just like they read your codebase or something. People get way too interested in that part
• That's not that interesting. What's much more interesting is how you work with the agent where you work on a plan with it and then you give it feedback as it
• generates things. You tell it, hey, you missed this thing. Go change that too
• Those chat histories are so valuable and cursor kind of gets to double dip here because they can use the models at
• admittedly way higher a price than other people get them for to resell to use the user building with AI in their tool, but
• they also get to keep the data if the right switches are hit. So they can use that to train a smarter model, not as smart as the ones that they are
• distilling from, but close enough that they can collect a lot of the margins that they are losing to there. Because
• if you choose to use cursor for OpenAI models instead of using codecs for OpenAI models, OpenAI makes a lot more
• money as a result of that. And this is a necessary thing for cursor to survive because if the prices keep going up, the subsidization continues. The incentive
• to use cursor is just a massive financial miss. Which is why when composer first dropped with composer 1
• and it was priced at 125 per mill in and 10 per mill out and it was super token hungry, I was confused. That is too
• expensive for a model that wasn't very good. And then5, which was meaningfully better, but still not great, was more expensive than Sonnet. And Sonnet's
• already overpriced. $3.50 per mill in and$,750 per mill out. This is the point where I kind of wrote off Composer for a bit. I just didn't see the light. That
• was my mistake, though. I was betting against Jacob. And you should never bet against Jacob. If you're not familiar, Jacob's the creator of Super Maven, as well as Tab 9. Super Maven got acquired
• by Cursor and he made the Tab Complete what it is. He wanted bigger challenges, which is why he started doing the model training side. And then Composer 2 hit
• It was way cheaper. It was seven times cheaper than 15 was, and way smarter, too. And now we have 2.5, which is still
• that same cheap price of 50 cents per mill in and 250 per mill out, but the numbers are where it shines on Cursor
• Bench. And this is cost per task, by the way. So, left is more expensive, right?
• Composer 25 is scoring between 55 high and X high, but costs way less than either. That's a really big deal
• Composer 2 was already pretty competitive in this sense where it was crushing things like 35 Flash and Kimmy K26, which is the model it's based on
• which is really interesting. It was getting up there with like Opus on Medium for way cheaper. But 25 has gotten slightly more token efficient
• significantly smarter, and stayed absurdly fast. Here are the numbers of all of the models Cursor currently publishes runs for with what is admittedly their benchmark cursor bench
• This benchmark doesn't have too many details public. It's real problems that they've encountered that they collected is my understanding, but they have not published it. There's a lot of good
• reasons for that. Like SWBench Pro is contaminated. It just is. If you make the benchmark open source, the labs will get the data on how to do it better and then that benchmark stops being useful
• It's sad that that's the case, but that is the case. Apparently, both Composer 2 and 2.5 are still based on Kimmy K25
• They didn't switch to K26 as the base, which makes their numbers even crazier
• They have effectively doubled the score of their base model with Kimmy K25 with Composer now at a 63% on their bench
• GPT55 got a 64. Opus 47 got a little closer to a 65 at 20x the cost though
• 55 was only roughly 10x the cost for a percentage point. But composer 25 being that good for that cheap is absurd
• Showing that you can distill an openw weight model to be close to that level of intelligence in coding tasks

• End of work from home #funny
• See the color changes
• "Do you like to dance?" Aviendha asks
• Graduation day
• Use a lifeline - Call my parents
• All Heartbeat Photos features - No fluff
• Anytime you say "algorithm" replace it with "audience"
• How do you make God laugh?
• Dune: Part 2 - Riding the sandworm
• Drink a spicy margarita, make some bad decisions
• Come on! Hit me! Conan O'Brien punched by Maria Menounos in the abs
• And our son?
• Heartbeat Payday #2 ❤️
• Almost monetized on YouTube
• Who did you fire? Senator drills Mark Zuckerberg, CEO of Facebook
• Electric desk in action
• Great Wall of China - Civ4
• The bureaucracy is expanding to meet the needs of the expanding bureaucracy
• Sexy squeeze through a narrow gap
• Attack Death
• Everyone shooting at Wonder Woman
• Egwene vs Ishamael - Wheel of Time
• All Heartbeat features - No fluff
• Elvis actor cast as Harkonnen Feyd-Rautha
• Dragon flies in
• "Fools!" said I "You do not know"
• 10-year anniversary of Freedom!
• Step outside the world's tallest building