I hope these get small and good enough to create “pet like” AIs for games. You know, like scream “follow me” at an NPC, STT stack translates it and feeds it to a local Jev-like model that then picks a number of things for the NPC to do.
I tried to use jev for this. I'll share what I learned for the interested.
--
The setup was a simple map with different rooms. Each room had 1-3 doors.
For the inputs: The AI had an array of "known places" empty at the start, the current position, the current doors with no information about where they lead to, and the list of past actions
The goal / task for the AI was to explore all rooms and save them to known places. The AI needed to decide if to move or save the place at every turn.
--
So I wasn't able to make the AI explore all of the rooms. The AI kinda always wanted to move to the first option when moving. Out of 6 rooms it was able to save 3. My hypothesis is that jev as it is now is really bad at making connections and understanding it's input. So for example, even if it had a list of previous actions, it wasn't able to reason about it and know where to go. For this to work I'd need to explicitly tell it where it did not go. So you could say that the model is also not good with uncertainty / ambiguous scenarios.
edit: one last thing is that i replaced jev with an standard llm and it finished the goal no problem with the same information given
edit 2: it also felt like the same tradeoffs between small model vs large model. With small models you need to be very conscious and careful with the input while large models are more forgiving. Maybe jev is a small model, and we just need a larger one.
Yeah the model has to be small, so need to do as much pre processing for it as possible I guess. It essentially doesn’t have any reasoning, so the reasoning would need to be input context (list of things that need to be done, resource shortfall or items needed or whatever) and assuming that you might run this once per second per NPC, it would need to be teeny tiny, maybe 0.3B or smaller.
Stay the hell away from spooky stuff like psychics and tarot readers - the more you don’t believe in it, the better.
But better again that you do believe, and know that these are not harmless fun, but that there are dark and hidden and evil things in this world to stay away from.
Gemma 30B with 256K context runs at 20 tok/sec on my M3 Max with 128GB RAM so I think there’s something wrong with your setup. This should run at ~30-40 toks. Maybe your inference engine is not optimised for Mac.
then used it in a very minimalistic pi with a very small system prompt.
Didn't spend much time to try to optimize it tbh, but my issue was not the speed. it just could not make a decision on how to implement the task, kept going on an on.
Yes, I use LM Studio with MLX support, which is specifically faster on M series Macs. I am not sure if llama is the same, but I guess what I’m saying is if you want the performance to be good on M series you have to use models packaged in the right format.
Maybe you have to set reasoning effort to low. 3.8 27B on x-high (default) reasons forever on anything complex. I asked it to write down the answer plan so far leaving open questions as open and it wrote the plan twice in reasoning (and more times partially) while it dilly-dallied about open questions before realizing “ok the user just asked to leave questions open.” That was a Q6o quant with unquantized KV cache.
Neat. This is pretty close to what I’d be looking for, if I could really afford to pull the trigger on something like this (realistically I can’t - iMessage on laptop is too important for work).
If it natively plugged into the Apple ecosystem, now that would be difficult to refuse.
1) If you want to make Claude usable how about addressing the fact that OpenAI 20x lasts all day on Astra, yet Claude 20x pips out in 20 mins of agentic with Fable 5.1, which is totally useless.
And unlike Sol: Opus like yoda, anything for unusable he is.
2) The first AI company to reduce their token pricing such that Astra/Fable can run for less than $150/day from credits would see a 5x usage increase and 20x spend increase from me. It’s currently around $500 for Astra and $5000 for Claude.
3) They seem to recently have addressed the fact that their safeguards were OTT so kudos to them for realising and now Fable is usable for wire and memory level programming.
I am a huge advocate of this kind of app paradigm:
It’s now really easy to spin a terminal emulator and make cross platform terminal apps that work the same on Windows, macOS and Linux regardless of users terminal. You can also make sure the term sends everything required to make the app “nice”, like mouse hover events.
I started using this paradigm for some of my own company’s apps around 3 months ago because it was neat to allow the app to spawn DWM windows for sub windows, dialogs etc. but the app can also fall back to standard TTY for using over SSH or from within a docker or any other terminal. Also, using your own terminal window gives proper taskbar integration (stacking windows) and you can set your own app name/icon.
(It’s possible to 1-shot this in Python using the prompt “use ghostling as a reference, write a terminal emulator in Python using raylib and libghostty that hosts my app” and the first pass was around half the speed of Ghostty.)
I disagree. First we got everything as HTML/Electron apps, now the trend is everything as TUIs. Cross-platform, better, but still worse somehow.
What falls to the wayside are the actual and real productivity and usability improvements afforded by "real" GUI apps over the last 40-50 years.
Don't get me wrong I'm a huge terminal user and absolutely love the clarity of a terminal and flexibility of a shell. But if I have to "learn" to use a TUI app - and they're not as simple to use as their authors might believe - I might as well learn to use the full GUI app with all the niceties available because of it.
reply