I've been waiting so long for something amazing to come out of the OpenAI and Cerebras collaboration.
> In our evaluations, GPT-5.6 Sol on Ultrafast mode answered all 2,500 HLE questions in 11 hours and 11 minutes. Claude Fable 5 needed 78 hours and 27 minutes, more than three days of continuous compute, to arrive at the same conclusions. In other words, Ultrafast worked through the frontier of human knowledge in a single working day, achieving comparable accuracy nearly 7× faster.
This is actually insane.
Hopefully the release ultrafast of Terra and Luna too.
I discovered yesterday that the “amazing thing that comes out of OpenAI” is Sol, due to its token efficiency.
Dollar for tokens, Sol and Fable are the same price.
However, Sol uses (literally: in testing) around 10-100x less output tokens compared to Fable for the same task.
We run our frontier models nearly 24/7, so switching to Sol will save us around $500 per day.
And, due to less guardrails, Sol also performed better, and we lost less tokens due to guardrails shutting down sessions (I feel like it’s illegal to take $50 of someone’s token money and then shut down a session with guardrails before they get an answer, and yet Anthropic do it to us constantly… either take our money and commit, or trigger the guardrails immediately)
Luna came out about a month ago, you're saying that the cost saving from switching to Luna has saved your company $20 000 000+ in 1 months spending on API usage?
OpenAI and Anthropic are believed to be making billions per month in revenue selling tokens to enterprises. It’s not that surprising for enterprise SaaS they would have individual customers making up ~1% of their total sales. Although, perhaps a bit more surprising those few customers are posting about it here.
Another interpretation would this is a counterfactual savings, like they previously paid $1M for y tokens, and now that tokens are cheaper they increased usage and paid $1M for 20*y tokens.
A second angle on the counterfactual savings would be Luna telling them not to pursue a potential session with the expected/extrapolated (from which sessions they did ignore Luna on, e.g. just to keep efficiency statistics current) sunk costs at time of getting shut down used to derive the quoted number.
A third angle is that it is not true and that's just some attempt at influencing an HN thread, which is happening quite regularly pro and against AI. We know that the pro-AI has a way bigger budget to burn with this kind of operations tho.
the easier way to do this, with any harness, is to use an expensive main agent that is told to delegate all code reading, writing, exploration, research etc to weaker subagents to conserve tokens. It's an inversion of the pattern but the resulting split is the same
I append this to many of my opus claude code prompts
`You may use a Fable subagent to answer questions, solve problems, and provide an adversarial review of your ideas and code`
You can use a similar pattern in most any harness, and you can tell them to use other harnesses. In claude you can write `Use codex cli to have Sol56 Xhigh provide an adversarial review to your plan before presenting it to me` or `Use opencode cli with GLM 5.3 to verify all code review findings before presenting` or whatever you're doing, as long as those other tools are setup and ready to be called.
IMO: This isn't useful as a token saving pattern in my experience with agentic engineering, but it is useful as a quality-enhancer.
Evidently, Claude's tokenizer vocabulary size is ~15k[1]. On one hand, it's quite mind blowing. On the other hand, Anthropic models' token (in)efficiency makes a lot of sense in that light.
Not just that, they normalize everything into lowercase and use a special character to capitalize words (what about languages with non-trivial normalization/capitalization?) and mark beginning and end of each word, all of that diluting already small vocabulary. That smells like manual tuning of what should be done statistically, I wonder what technical merit they saw in that - I know they mentioned better generalization, but this is pretty counterintuitive.
Sadly it's not too counterintuitive; remember the old "how many r's are in the word strawberry"?
Also different tokens for the same named entity/concept if they almost entirely exclusively occur in non-overlapping contexts, and are themselves rare/uncommon in the first place, will result in behavior that's similar to the speech/phrasing/vocabulary registers humans exhibit, where the aspects of the named entity/concept get largely compartmentalized.
The most severe case along these lines were the old BERT models that ran over straight UTF-8 bytes (plus a handful special tokens).
But for the modern post-GPT2 LLMs such radical simplicity seems to mostly not be considered suitable.
Note that CJK (the big one in particular, so Chinese semantic and Japanese Kanji) encodes each one into multiple UTF-8 bytes giving some automatic scaling for semantically dense languages; similar effects also apply to e.g. APL code.
What kind of bot would say `less guardrails` instead of `fewer guardrails`? I guess someone could instruct an LLM to deliberately make mistakes, but isn't that too paranoid?
15 years ago i was astounished by the intellectual deepness of this community. Now i'm aware this has always been a cult and their former cult leader Mr. Altman wants to destroy this capitalist society. He isn't even hiding motives. People just stopped listening carefully.
“include common grammatical imperfections and awkwardness common in casual message board interactions.” I’m not saying that’s what’s happening here, but a high school student told me that’s basically what they do to make papers not sound like AI.
Other languages use different characters for quotes, if anything that's an indication that's not written by a LLM because it's not favoring the standard English character.
Answering 2,500 independent questions is an embarrassingly parallel workload, all it needs is scale out. It would be more meaningful to know how much time was required for a single complete answer to a difficult HLE question.
The animation in the article shows GPT-5.6 Sol Ultrafast solving a question at 3 secs vs Fable's 27 secs. The "11 hour vs 78 hours" animation implied that they were solved in sequence and not in parallel
> Answering 2,500 independent questions is an embarrassingly parallel workload, all it needs is scale out.
I believe the assumption of the benchmark creators is that it's meant to measure the sequential speed that a single instance of the LLM and piece of hardware can get through all the tasks from start to finish, like running a race. As a rudimentary comparison, sort of like doing prime number calculations as a benchmark of the CPUs in one bare metal server. Of course you'd get a speedup in total number of primes searched if you ran the same software of GIMPS on 8 servers with the same hardware in parallel rather than 1 server.
Of course if you took all the individual questions in humanity's last exam and fed them in parallel into separate queries to Claude that land on separate hardware instances of the claude model you'd get a speed up. Because each question is independent and not related to knowledge/calculations that are performed in any other question it is indeed very open to speed up by breaking it into separately dispatched parallel tasks.
Amadahl’s law dictates it’s approximately always better (as in, more efficient computationally) to have one super fast thing than many slower things doing the same job in parallel.
The task the majority of people here care about (and which makes up a huge part of AI lab revenue these days) is collaborative coding with a single human which is not embarrassingly parallel.
I am aware. I was talking about HLE. Talking about how fast you can run through HLE in a serial manner is like talking about how how fast your non-simd single threaded cpu ray tracer is. Its simply not how any sensible person would tackle this problem.
I'm finding Luna suprisingly adequate for my work. I slept on it due to the benchmarks, but it's very fast and even on low reasoning I'm finding it more than adequate for "menial" work. (The speed is crucial for "interactive" work -- if a model is fast enough it goes from "async" to "real time", subjectively, which is a huge difference.)
In fact, I'd say it's overqualified for the kind of work I'm doing, because it spends >half the time verifying trivial changes (and the verification isn't as helpful as you'd expect, even with bigger models).
Maybe I can prompt it to be less aggressive about that (the new GPT models do it even without prompting).
Anyway, Ultrafast Luna would be amazing, though I strongly doubt they can offer Cerebras at anything approaching the current prices. Now we wait for Moore's Law? :)
Fast models is why I was hoping Taalas would get their butts into gear and eventually release a consumer priced card. I'd love to have a pcie card that screams along at 15k t/s even if on a heavily quantized 2026 level model forever.
Faster & cheaper tokens = more reasoning capability and more reasoning = better problem solving as far as I have seen.
I'm considering a home project done in Rust. I've not even touched Rust before, I'm a Java dev. It'll be something reasonably complicated, using a bit of unsafe etc. Do you know how Luna does writing Rust code?
I feel odd to use these models, because it feels like a faster model doesn't feel that much faster if it spends reading files, making edits and running checks.
Sol medium/high planner orchestrating -> Luna xhigh subagents doing implementation
...has been REALLY good for me. Even on xhigh, Luna is crazy cheap.
Subjectively I'd say it's way better than Sonnet at a fraction of the cost. Luna xhigh can do some decently challenging things on its own, but when orchestrated by a model that is actually good like Sol, I am finding it very very nice.
The cool kids have custom harnesses and workflows and stuff, yeah. I'm still using Superpowers in Codex. Planning in Sol, Luna subagents. https://github.com/obra/superpowers
I feel like I could be doing a lot better somehow. Regardless though Luna (xhigh specifically) is super good/cheap/fast for a lot of things
Feels like the 90's again where single threaded speed is improving fast. ASICs and wafer scale rather than node shrinks, but end result to me the consumer feels the same.
I wish, but I don’t think this qualifies as single thread performance. Cerebras is whole wafer computation though right? So you get the result sooner, but the same amount of computation has occurred, just with huge parallelism.
ChatJimmy is a much smaller model and, AFAIK, has no reasoning capability. Absolutely insane raw speed, like a supercar, while Sol is more like a freight truck.
> At such output speed, I wouldn’t expect reasoning.
As the sibling comment to yours mentioned, if they had a reasoning model “hardware-ified” onto a custom chip (as is their plan for IIRC this or next year, a new ASIC), it’d output fast decode speeds for the regular output as well as reasoning sections. Both would be ≈equally fast.
Yeah, I thought reasoning was literally just chain of thought in the output token stream, with the model itself adding delimiters to indicate what part of the output is internal reasoning, and what part is an answer to the user. Is that wrong?
No, as the sibling comment mentioned, your understanding was correct there.
What’s more, the only technical difference in speeds could be, and likely also is with the HC1 chip, between prefill (prompt processing) and decode (text generation) speeds. I don’t know whether it’s the case with Taalas’ chip, but in the “software-based” LLMs we typically see and use so far, those two stages hit different parts of a computer (processing/compute-bound vs. memory/bandwidth-bound).
Haha, at first I thought you meant that the knowledge of the existence of an LLM that’s so fast is terrible because it’s ruined every other LLM for you!
Apparently they baked the Llama 3.1 8B model weights [0] into silicon (the actual hardware is called Taalas HC1).
I guess for the trillion parameter models this would not scale due to cost? Imagine buying GPT 6 in the form of a PCI-E card, pulling these speeds, with up to 120 cct agent sessions. It'd be beyond wild.
[0] the weights are also using some cut down small format, but HC2 will have regular FP4 supposedly, and support for 20B params on one die
> Never heard of it before, that's fucking insane.
They've been acquired by AMD. Those saying the model sucks are completely missing the point: it was a proof-of-concept.
The question is: what happens to a model like Anthropic's Fable 5 that does, what, 70 tokens/s (and requires lots of output tokens) when the latest open-weights model is etched on silicon and does 14 000 tokens/s?
Shall the better model still have the upper hand or will the raw speed compensate?
Shall the better model still have
the upper hand or will the raw speed
compensate?
At 14,000 tokens/sec there's just so much ridiculous stuff that might be possible. Let's assume that this POC proves they can take the next step, and can eventually etch a capable ~27B model into silicon. Let's call it Fred.
Ralph loops automatically get real real interesting again. 200x the iteration speed. This is such a clear win I feel like there's hardly anything to talk about. Instead of one stubborn iterating idiot, you could have dozens of idiots competing in parallel, genetic algorithm style.
The other common orchestration pattern I see is "big model for planning, small parallel subagents implementing, big model reviewing" Today it's Sol dispatching a handful of Luna subagents. Tomorrow maybe it's Sol dispatching as many Fred subagents as it could possibly want.
But what patterns have we not even thought about yet in a world where subagents are 200x faster/cheaper?
What if instead of dispatching single Haiku/Luna/etc subagents, we dispatched "teams" of Fred agents? Maybe each team is 8 Freds. Five come up with competing ideas and the other three vote on a winner.
Or what if they were heterogenous teams? One Luna and a bunch of Freds.
What if instead of a two-tier orchestration system (Sol->Luna) it was three-tier or n-tier? (Sol->Luna->Fred->...Fred)
Those ideas overlap a bit, and crazy shit like Gas Town has already explored even wilder ideas I guess. But man, 14000 tk/sec opens up so much stuff.
At 14,000t/s that's effectively a motor cortex for an android, you no longer need to train the robot to walk, it has a general idea for how to walk (baked into the 1b model), and then just corrects based on sensor input, in real time.
I still personally think that a heavy lean into MoE will be better for that sort of thing. Our brains are subdivided into large parts but I'm sure (and I'm not a brain scientist) that those parts can be subdivided even further into systems that run at various frequencies and latencies depending on what they're used for.
I was thinking about it the other day actually. How our brains evolved structure. I imagine it was purely just down to evolution adding/clustering additional cells around the areas where additional cells were needed. And after long enough a natural brain architecture emerged.
Makes me wonder if we're on the right track with transformer architecture/attention but if it'd be more effective on a larger scale, like MoE with a billion "experts".
That seems promising to me too, although, the thing I've always read is that you can't make the "experts" too narrow. Even if you had a "coding expert" it has to know a lot more than coding - if you tell it to make an online store it needs to parse your language, understand the internet, what a "store" is in this context, etc.
I am not a primary source, probably not even a secondary or tertiary source, so take this with all the grains of salt.
Oh for sure, but I think generally multiple experts are selected in an MoE pass for a token, so presumably it'd select programming related ones as well as general knowledge/language.
Only problem would be the routing layer works on the previous token as far as I understand so it might need more informational depth than just "a token" to select experts, I suppose in the same way attention works.
I wish I had the GPUs to run those sorts of experiments ha ha.
If anything, writing "7x" is actually just a poor emulation of writing "7×".
Saying something is "done at 7x speed" should be read as "done at seven times speed" not as "done at seven x speed". So using the 'times' (multiplication) symbol is the better form in my opinion; it just happens to be significantly easier to type "x" instead, which is how we got here.
I fully agree that speech evolves, but maybe this is just in the US? I've never really heard anyone say it here in Australia, it's usually "times", although admittedly almost no one uses the phrase/format here.
> In our evaluations, GPT-5.6 Sol on Ultrafast mode answered all 2,500 HLE questions in 11 hours and 11 minutes. Claude Fable 5 needed 78 hours and 27 minutes, more than three days of continuous compute, to arrive at the same conclusions. In other words, Ultrafast worked through the frontier of human knowledge in a single working day, achieving comparable accuracy nearly 7× faster.
This is actually insane.
Hopefully the release ultrafast of Terra and Luna too.