C++ has had smart pointers for memory (and other resource) management for a long time now (see e.g. the Windows ATL classes for working with COM objects and resources).
There are a number of challenges that make browsers more challenging (even in memory safe languages like Swift and Rust).
1. Back references/pointers like `parentElement` to other objects in the graph that create dependency cycles (where traditional/simple reference counting will prevent the objects being deallocated).
2. Interacting with (and creating resources in) a garbage collector when running/evaluating JavaScript code.
3. Just-in-Time (JIT) compilation of JavaScript and other complicated interpretation-compilation pipelines that can allocate and transfer objects between the different stages.
Actually, if you implement a JavaScript runtime at all your choice is either too slow for modern web users or `unsafe` everywhere. Runtime values are often simultaneously either words that encode immediates or pointers to heap blocks and doing this the proper way blows up time and space. Doing it the fast way means you have `unsafe` everywhere and Rust's lifetime model can't help you at all. Everyone chooses the fast way.
Maybe Rust let's you encapsulate the unsafe a little better than C++ but the wins are going to be surprisingly small. If you had to build it today you'd maybe use Rust but the case for rewriting an existing C based runtime in Rust is not that strong.
Also for things like interpreter loops the absence of computed goto in Rust stable is a real performance killer. Explicit tail calls (the `become` keyword in nightly) work but maybe you don't want your big rewrite to rely on that.
A common writing tool is to use a story grid. You have chapters/similar along the Y axis and title, characters, plot elements, etc. along the X axis. That way you can keep track of what information you are revealing at each story beat/chapter, make sure that plot twists work (e.g. when outlining something like The Sixth Sense).
Depending on how big the story is and how many characters/plot threads there are, I can see the Excel files getting large.
This has happened with other accelerants in various media/fields:
1. easy access to video recording and editing equipment has made it a lot easier to produce videos on sites like YouTube;
2. easy access to audio recording and editing (Audacity, etc.) has made it a lot easier to produce podcasts and other audio content (via LibriVox, Apple Podcasts, etc.);
3. easy access to writing software and self-publishing (Amazon, fan fiction sites, blogging platforms) has made it a lot easier to produce stories and books;
4. easy access to game engines and tools (Unity, Unreal, Godot, Twine, RenPy, etc.) have made it a lot easier to produce games.
Yes, and as Howey explains in TFA those things might be good for consumers and creators who create for fun, but they are not good for people who create in order to earn a living.
you are not taking into account that creators can be funded in different ways. For example like in Patreon.
You can give part of your money to fund a creator that wants to create in order to earn a living.
I believe that is better when people create for fun. If they need to create to earn a living they are forced to create and that is not the best incentive to create. Fun is a better incentive
> I believe that is better when people create for fun. If they need to create to earn a living they are forced to create and that is not the best incentive to create. Fun is a better incentive
I'm not sure that's true (thousands of famous creative works from the Sistine Chapel to the Benin Bronzes were created on commission/for remuneration), but even if it is, how are creators gonna eat? If the answer is "get a day job", I think that excludes a lot of folks from practically being able to create things. Patreon is not a great way to pay the bills for all but the most famous and popular people on it.
With IA, creation is cheap. It's inevitable that automation would make some jobs disappear. There's no solution, less people will be able to earn a living for creating art, I think.
I think that's probably at least somewhat true, and regrettable/something to be worked on in your community, and a very different statement from
> [it] is better when people create for fun. If they need to create to earn a living they are forced to create and that is not the best incentive to create. Fun is a better incentive
There will be less creators that earn a living by creating. But I believe there will be more art.
E.g. after Michelangelo created his style, then IA can mimick it almost for free. Imagine people with great talent like him, expressing their ideas and also being able to use IA to create, plus thousands of less talented people basing their AI art on them.
IIUC, the main problem with the current Li batteries is that the two plates can over time grow material that will 1) degrade performance; and 2) make it more likely to short circuit and catch fire. Similarly with electric car batteries after accidents where the battery is damaged, short circuits, and then catches fire.
So the main risk here would be the likelyhood of short circuiting under different failure scenarios.
From the paper (page 6 with a comparison to GLU and SwiGLU) they are not using tanh directly (i.e. f(x) = tanh(x)) but:
f_gate(b,x) = b * tanh(x / b) * sigmoid(x)
f_up(b,x) = b * tanh(x / b)
Looking at the graph I wonder if this is to try and get the best of both GLU (better representation at higher values of x >~ 5) and SwiGLU (the value bump just before 0).
In the linked "Kimi-K3 Technical Report [pdf]" paper, section 2.3 (Stable LatentMoE, p6) has the table with those equations on (top of p7, using β_1 for the gate branch and β_2 for the up branch). They talk specifically about the function in section 2.3.2 (Sigmoid Tanh Unit GLU, bottom of p7).
Figure 2 (p3) has the architecture diagram for K3 (which is the same one on the Kimi K3 blog post https://www.kimi.com/blog/kimi-k3). AFAICT, that diagram along with section 2.3 should answer your question.
Note: I'm not familiar with LLM/NN architecture to answer it more precisely than that; I only have a surface level understanding from watching various YouTube channels like 3B1B and Welch Labs on the subject.
Maybe they are in the process of uploading the weights and git history and have taken down the holding page/project to not have the "coming soon" in the git history.
If it is a mixture of experts (MoE) model like the 2.x models, won't this reduce the hardware needed to run the model?
The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090. On a single B200 you can have 5-6 experts loaded into VRAM at a time. Realistically that would be 3-4 to account for the context. [!]
[!] With this and other MoE models it looks like an interesting area for research would be to detect or predict which models would be needed ahead of time. That way you could schedule the load into VRAM step before the weights are needed. That way you shouldn't lose much/any performance from offloading the weights to RAM.
You need whole weights in VRAM for optimal performance. Don't be confused by "experts" in the name -- you don't get to load static subset of experts and blast next 100 tokens with them. In typical MoE model they get switched "randomly" on every token, so all experts have to be readily available.
> With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090.
Kimi K2.6 is released as INT4 already.
So 5090 with K2.6 is just gonna sit idle 99% of the time, waiting for next slice of weights to load.
5.6 Sol calculates that single 5090 in raw compute & memory bandwidth can run K2.6 at 35 t/s (256k context depth) -- if it somehow had enough memory to hold whole model in VRAM. Man, I hope HBF succeeds and Nvidia brings it to consumer cards in 5 years..
> In typical MoE model they get switched "randomly" on every token, so all experts have to be readily available.
It's worse than that: a typical MoE model routes a separate set of experts at every layer, not just every token! But in practice, RAM offload (for systems with non-unified VRAM) and even SSD offload still work surprisingly well given some amount of caching.
You can likely recover compute intensity and throughput by batching requests together, which (in practice, depending on sparsity) will end up reusing some of the loaded experts with high probability; though the obvious tradeoff is that having to store KV caches for the wider batches may leave you with less room to cache experts across layers and tokens.
(Plus if you're batching so widely that you end up loading essentially entire model layers, MTP then becomes applicable even for a MoE model. But this typically only applies if you're doing inference on a very large scale, or if your memory bandwidth is so scarce that you have to recover compute intensity by any means feasible.)
> It's worse than that: a typical MoE model routes a separate set of experts at every layer, not just every token! But in practice, RAM offload (for systems with non-unified VRAM) and even SSD offload still work surprisingly well given some amount of caching.
Caching really has nothing to do with this. With RAM offload you can mostly benefit from:
1) Batching for prefill is a huge win, even with MoE, since the batch sizes can be so large.
2) Keeping non-expert weights in VRAM, so the percentage of weights used per token in VRAM is higher. This benefit reduces with larger models, though.
> You can likely recover compute intensity and throughput by batching requests together, which (in practice, depending on sparsity) will end up reusing some of the loaded experts with high probability;
With MoE it's low probability.
> MTP then becomes applicable even for a MoE model
For even the sparsest MoE open models, having more than a handful of inferences in the batch is enough to make it more likely than not that you'll get some MoE weight reuse within any given layer. This assumes totally random sampling, ignoring any cross-request correlation that would push that probability even higher in many practical scenarios.
> With MTP it becomes _extremely_ low probability.
This is actually right, MTP is only ever worthwhile in very special cases involving either dense models or extremely wide batching of MoE ones that somehow still leaves unused room for parallelization (which AIUI would involve an assumption of very abundant compute with very limited memory bandwidth).
> For even the sparsest MoE open models, having more than a handful of inferences in the batch is enough to make it more likely than not that you'll get some MoE weight reuse within any given layer. This assumes totally random sampling, ignoring any cross-request correlation that would push that probability even higher in many practical scenarios.
If you tell me the model and the number of parallel streams, I will do the math.
> The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090
Without leveraging system RAM and/or SSDs, I don't think you can, or how exactly are you running this, if this is something you are doing today? With CPU/expert offloading you could probably do it with a 5090 + 1TB of RAM or something like that, but absolutely not on a single 5090 entirely within VRAM.
> There are a lot of optimisations that are not in the public sphere
Sure, but if we're participating in public discussions, isn't it more fun if we talk about things people can actually read and understand, rather than secret stuff other's can say work, but no can actually validate or know how it works?
It sounds like "hybrid approaches are much better than the public is aware, because everything else is private and secret", but also: ok, so what? No one can run that anyways, (yet?), so why it matters?
Alright, I guess I misunderstood. To be fair, this part:
> The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090.
Is painting a very different perspective, even considering the latter parts it's hard to read that as "Of course offloading everything else that doesn't fit on the GPU itself". But anyways, it's been clarified now so no harm :)
I assume you mean putting only the 32B active parameters on the GPU, and the rest on a bunch of regular server DRAM like on a 768GB to 1024GB RAM server?
Because Kimi K2.6 in Q4 is about 584GB GGUF size on disk and will use slightly more than that in RAM, Q8 is 595GB.
You're talking about running this "at home" for 1 user, using a mix of VRAM and RAM (total should be ~1.5TB). That's certainly possible. It'll be slow, especially prompt processing, but doable for single users.
But my comment on running it was more towards serving this profitably at scale. You get much better throughput / unit of compute if you load everything in VRAM and serve many requests at the same time. That's how all inference providers are doing it.
I was talking about running this on a server, hence my comments re 1xB200. Obviously, the more hardware/VRAM you have the better/faster you can run these large models. But if you are a small/medium sized company you could feasibly do it on just one B200. It all depends on how much hardware you can afford to run.
It can be useful for checking input token usage before sending it to the model, e.g. preventing calls above a given token bound or grouping requests into batches.
It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.
Firefox has had profiles for a long time (via about:profiles and a command-line argument). Unfortunately, the new profiles are not compatible with the old ones and cannot be migrated.
Firefox added a layer on top to allow to show the list of profiles in the UI.
I recently started a new job and built 3 profiles on Windows, then switched to Ubuntu. The migration of my profiles was painful because of that new layer: migration of each profile was working fine, but didn't see each other.
Thankfully Claude was helpful to debug and fix the settings. I had to edit SQLite databases.
There are a number of challenges that make browsers more challenging (even in memory safe languages like Swift and Rust).
1. Back references/pointers like `parentElement` to other objects in the graph that create dependency cycles (where traditional/simple reference counting will prevent the objects being deallocated).
2. Interacting with (and creating resources in) a garbage collector when running/evaluating JavaScript code.
3. Just-in-Time (JIT) compilation of JavaScript and other complicated interpretation-compilation pipelines that can allocate and transfer objects between the different stages.
reply