I keep being confused about how people's understanding of the models get stuck at next token prediction. Isn't this entirely neglecting the RL training? I might be misunderstanding something but to my mind it makes the issue way fuzzier than it's being painted here.
I keep being confused about how people's understanding of the models get stuck at next token prediction.
Heh. A lot of anti-ai hucksters I see posting on LinkedIn just LOVE to use the phrase "next token prediction" and the word "autoregressive". They've almost become shibboleths that identify members of that camp. That and the classic rallying cry of "Linear Algebra isn't intelligent!"
The best take I've seen on that recently, was somebody who made the point "just think of the next token prediction part as the output layer". Which makes perfect sense.. if you're replying in natural language, at some point in the flow, you have to construct a sentence and starting at the head and predicting next tokens is perfectly reasonable. I'm doing it literally as I'm typing these characters, for crying out loud!
But the mistake is to think that LLM's only "predict next tokens" with no consideration of the possibility that they are actually constructing richer representations, building concepts, making analogies, doing abduction, induction, etc. My own (admittedly anecdotal) take on working with LLM's suggests to me that they do do those things, albeit probably not the same way humans do.
I think a lot of folks are missing the point by being overly reductive when they start talking about "next token prediction" and "autoregressive". It's like, can we say "Phil (me) isn't intelligent because there's nothing going on but some electrical impulses and chemistry happening inside his brain. Everybody knows electricity and chemistry aren't intelligent!"
You can just as easily argue that markets caused all this. They don't incentivize the preservation of the planet or even life at all. There is no reason to believe that anything that comes out of markets organically is good, they don't converge towards worlds that have sound values. If they do in some instance then it's just by happenstance as in the solar example.
I'm not familiar with what you're talking about. Can you share a real world example of a case like this? You're saying a social media comment about a crime is somehow punished harshly and much harsher than the crime itself?
In my Northern European small city, just yesterday, there was some non-native shouting "WHITE NIGGA" at children in the city center. Nothing happend to him.
Are you sure that link's the one you meant to share?
> The former childminder from Northampton, who is married to a Conservative councillor, had posted an abhorrent message on X, calling for people to "set fire" to hotels housing asylum seekers following the murder of three young girls at a dance class in Southport in July 2024.
> Yet the fact both men were able to address a huge crowd in London is perhaps evidence that there is rather more leeway for free speech in this country than those likening the UK to a "tin pot dictatorship" suggest.
The right love to play-down the woman who called for terrorist attacks on British streets as "just a tweet".
Meanwhile there's been two arrests around two prominent right-wing politicians getting threats and celebration of death this week. And they're all silent again.
The link is far broader than that, but even in this case: there was no threat. No more than a university lecturer tweeting that he wanted a white genocide.
It's a common (and somewhat right-wing) trope, but it's not really true. Thousands of people aren't arrested for "mean comments" or similar. That isn't to say the UK has no problem with the balance of freedom of expression and regulation, but it is nothing like the impression you'd get from some sources. There are some high-profile cases which are commonly misrepresented to support these claims.
It is definitely true. It's "right wing" because the soft power is with the left (or was until maybe very recently) and so only principled people of the left (e.g. Bill Maher) and all the people of the right (either principled on speech or saying things that are more likely to be censored) are going to be talking about it.
I think generally the argument is that these usually aren’t credible threats, they’re hyperbole, and the police are overstepping by arresting people for saying they’ll do something when they have no evidence that that person would actually do that. A free society doesn’t arrest someone for saying “I’m going to burn down so-and-so’s house.” It waits until you actually go to so-and-so’s house with a can of petrol, and then arrests you.
It’s the same problem as amber alerts - the stories you here about are the dramatic stranger danger ones, but the vast majority of the statistics are “boring” custody disputes.
The poster up thread said people were being arrested for "mean tweets" and now you've moved it to "but those death threats weren't serious!" No, the original claim was dishonest bullshit.
I've noticed this as well and I'm rather concerned about it. Those cheap rip offs will ruin countless classics for her and it's really sad. I'm looking for the most boring stuff for her to watch just so she doesn't become fully jaded by age 18.
Side note: if you need new stories, I've found a lot of weird and fun stuff to read on Royal Road.
And it's all the same to you? You don't care which values those things have? Of course there will always be underlying values. I wouldn't go as far as calling everything political narratives.
I think it's just pretty clear that Elon's values are not what most people want the world to be shaped by.
I asked ChatGPT how many billionaires the US can have if every one of them has "just" 1 billion and if everyone else has basically nothing (i.e. zero net worth).
It said that the theoretical maximum is 174,000. In a country of 340 million.
That sounds to me like the chance to make it is a lot lower than 0.05%
Maybe we should just stop pretending that everyone can make it and that it's mostly people who already come about with vast amounts of luck (means = luck, opportunity = luck) who have even a chance to make it.
People with cool startup ideas cannot just make it. Working hard is not enough. Growth is not good enough. They will likely just be destroyed in an increasingly unfair and predatory market.
That's what I'm thinking too. There is a lot of noise and I know teams where the majority of the people writing Python just have no idea what they're doing.
I'm working with Clojure which is used mostly by senior engineers and it still blows my mind how well Claude writes software in it even though it's a fringe language. It's even able to pick up in-house DSLs written with macros.
Was looking for somebody to mention that LLMs are weirdly good at clojure! Have you tried hooking your LLM up to a repl? It’s crazy stuff. Type systems are a great feedback loop, but a LLM with a REPL is something else. Much more dangerous though!!
reply