Name: I don’t really like GPT-5.5…
Channel: Theo - t3․gg
Description: that one kind of cooked. Their expert SWE bench, which is an internal bench they run, got a 73% instead of the 68.5

that one kind of cooked. Their expert SWE bench, which is an internal bench they run, got a 73% instead of the 68.5

Saved by Arn D.

Discover other moments and creators

Sup nerds. You might notice a couple things different about m...
wanted to cover it and I think my coverage might surprise you...
this is a video about GPT 5.5, the latest model released by O...
I am. But honestly, overall, I'm a little bit disappointed. T...
that I want to really talk about here because as powerful and...
also comes with a pretty massive price hike at $5 per million...
higher than Opus 4.7. While it is much more token efficient t...
hopefully better than I'm about to justify it by switching to...
much. It's too much work to get everything set up, to get Xco...
fun. I've just not been able to enjoy building mobile apps. B...
ROR. I've never had this good of an experience building for i...
impossible. These guys made it literally three clicks. Once I...
Not only that, but you can run the apps on your real device,...
how crazy is it that I could vibe code with a single prompt a...
other things you need to build good iOS apps. In Swift, they'...
local device without having to set up an Apple developer acco...
this good for getting an app from Idea to reality to your fri...
results speak for themselves. If you want to build real apps...
Okay, let's dive into GPT 5.5. See what OpenAI has to say abo...
Starting with their blog post, they say it's the smartest and...
it's not the best in the world. Might be that Mythos is bette...
things. So, we'll keep scrolling to the parts that matter. We...
We evaluated the model across our full suite of safety and pr...
biology capabilities and collected feedback on real use cases...
it was secure. They also call it the fact that this model is...
to fix that. A lot of this appears to have come from a partne...
They trained and shipped this on the GB200, I believe. Yeah,...
state-of-the-art from Nvidia. That's what they shipped this a...
the workarounds in the near future. They will support it soon...
to have some early access to test things there and uh have so...
Benchmark dominance
compared to the 75.1 they had before
They've been leading there for a bit now. They don't evaluate...
very different thoughts about the new Pro model, which we'll...
that one kind of cooked. Their expert SWE bench, which is an...
I got before. If you're wondering why they didn't test this o...
you're wondering why they didn't test 3.1 Pro on it, I don't... [eo5a2]
Not my words. Okay, those weren't my words. Then we have GDP... [eo5a2]
versus 83.0. I would go as far as to say that's a little dish...
less often than 5.4 did. It just tied enough to balance out t...
GDP valve is a lame benchmark. I It's delayed more releases t...
beating out Opus 47, but still not by much, just 7%. Toolatho...
been benched on it. And in Google models, as always, kind of...
did pretty well, but it did lose to 5.5 Pro. And 5.5 Pro also...
but it is worth calling out that this is the Pro models numbe...
While I have been impressed with the browsing capabilities of...
graphical, all that stuff is still far ahead of everybody els...
this browse comp test comes down to being better at tool call...
again, Google still cooks there. I still use Google models fo...
The Pro model in particular seems to be very good at this stu...
That's a lie. I want to do one more bench. Flashbang warning...
the best benchmark, it is a pretty good cohesive cover a lot...
interesting though is when you turn on the 5.5 versions that...
even low, you'll see some very interesting numbers. 5.5 mediu...
While the price per token is higher, the token efficiency gai...
intelligence, you do have to spend more money. and the standa...
to the tokens used to run the benchmark section, you'll see s...
the highest possible run version, use 75 million tokens, whic...
Opus 47 is not quite double, but still quite a lot more token...
even medium, you have to go really far down the end here. To...
medium used only 22 million tokens. That is actually a very i...
I could do the math here, but I'm lazy
So instead, I'm going to go to the intelligence versus output...
light. Yeah, we can delete a lot of these. Now I've removed a...
has pretty tight dominance on the intelligence versus output...
The x- axis is how many tokens were used. The y-axis is the i...
well for the token cost compared to other things. It is actua... [eo5a2]
while also performing that intelligently and 5.5 low absolute...
Obviously, these tokens cost more per token and I wish I coul...
yet. OpenAI visualizes in a very appley chart that's hard to...
comparable to like Opus 47 was done in roughly half as many t...
meh it's expensive and they're trying to justify it by showin...
feel way faster because it takes half as many tokens to do a...
reasoning levels. This is the first time I've seen them recom...
instead. And now we get to the frontend capabilities. They sh...
still full of cards like everywhere. Pretty sure almost all o...
somewhere. But there is something also quite impressive about...
When I saw how well these models supposedly did 3D game stuff...
certainly loves to do that. One of the first things I started...
an idea for a game I had forever ago called Fish Slop. It's a...
Insane Aquarium. And I was mostly testing the capabilities of...
and try to just flush out an idea. I end up getting way furth...
chaos for a while now, as I'm sure you all have noticed. So,...
So I took the existing source code, pointed models at it, and...
"Modernize this, clean it up, make a good, minimal starting p...
For some reason, it did not want to touch the assets that I h...
but it worked. I will say actually the movement feels slightl...
And it's impressive that it was able to generate these assets...
For some reason, it didn't want to use Phaser, though. It act...
It also seems like it really up the timings for like when the...
Yeah, I I thought this was better than it was. Now that I'm a...
I'm realizing, oh yeah, this this kind of ass. And now I have...
I don't know what it is, but it is bad about this. It didn't...
very quickly if you don't feed them instantaneously
There's a lot of little things I don't love about this versio...
And now we have the version that we got out of 5.5. Immediate...
assets from the original for the fish and the like submarine...
Little things like the coral on the bottom, while not perfect...
going across. I like the wavy lines here. I don't love it, bu...
It's got something I would call almost taste, but also someth...

Graduation day
Come on! Hit me! Conan O'Brien punched by Maria Menounos in the abs
Everyone shooting at Wonder Woman
Who did you fire? Senator drills Mark Zuckerberg, CEO of Facebook
Almost monetized on YouTube
Use a lifeline - Call my parents
The bureaucracy is expanding to meet the needs of the expanding bureaucracy
Dune: Part 2 - Riding the sandworm
Egwene vs Ishamael - Wheel of Time
Attack Death
"Do you like to dance?" Aviendha asks
Dragon flies in
Elvis actor cast as Harkonnen Feyd-Rautha
Drink a spicy margarita, make some bad decisions
And our son?
All Heartbeat Photos features - No fluff
Great Wall of China - Civ4
See the color changes
End of work from home #funny
Step outside the world's tallest building
All Heartbeat features - No fluff
10-year anniversary of Freedom!
How do you make God laugh?
Sexy squeeze through a narrow gap
Anytime you say "algorithm" replace it with "audience"
Electric desk in action
Heartbeat Payday #2 ❤️
"Fools!" said I "You do not know"

that one kind of cooked. Their expert SWE bench, which is an...

Links to other pages

Discover other moments and creators