This is cool. I'm so eager for faster innovation in the hardware space, as opposed to some people's concept of innovation being who can make the most addictive social feed.
Given the huge amount of money being spent on AI chips in the US, what prevents US AI labs from doing the same level of software optimization? It could be a solve for some of the capacity constraints.
They have already been doing it for months https://openai.com/index/openai-broadcom-jalapeno-inference-... . OpenAI on their custom chip brought up lightspeed deepseek as experiment by using AI in the exact same way as this zAI blogpost. And the kernel optimization contests/etc have all been havily done through AI based optimization loops for half a year+.
> what prevents US AI labs from doing the same level of software optimization?
Because they don’t have to. Most of the time money would buy you newest and/or more hardwares so there’s low/minimal interest to optimize the code or approach.
I would be extremely surprised if US labs weren't aggressively trying to optimise their stacks in exactly the same way. Any gain in performance or efficiency directly affects the bottom line as well as research speed.
They do, when Luna got 5x cheaper it was directly attributed to some unknown % inference optimization.
US labs are quite cut throat about dealing with stuff costing them money (inference). This sort of engineering excellence doesn't always feel that way because they are simultaneously quite lax about stuff costing other people money.
Same here. I'm still a better software architect that AI, but there is no question that my AI generated and reviewed code has fewer bugs than code I hand write. It takes some humility to acknowledge that your coding prowess is less of a useful skill than it used to be.
There's an issue where people assumed the syntactic activity of writing code was what mattered. The reality is that this was always a smaller part of the role, as opposed to thinking about observability, serviceability, and test automation. The ability to write software that is properly separated from concerns and when to enact those separations matters.
At the same time, I think we're far too far down the systems path now. We've hit a point where interviewing has become purely systems design "because the AI writes the code".
Not that I'm ever asked, but I inherently believe the act of critical thinking, communication, and expression are the key skills for those who already have the appropriate coding/engineering/cs/etc background. I now only interview for those skills - but through the lens of impossible to solve systems design conversations as opposed to problems. It tells me a lot about how people think.
So is architecting, testing, validating, and even occasionally using.
This isn't the first time I've seen this phrase recently, but I'm not sure what the thought is a cliche or what it is intended to convey (don't read my note as negative, I sincerely am unsure what connotation folks are trying to say).
The idea behind "writing is thinking" is that people often overestimate their understanding until pressed to express it in words (or code).
How many times in your career did you sit down to tackle a task thinking you knew exactly how to approach it only to realize during implementation that there were edge cases you hadn't considered, API contracts that were now broken, or that the feature was trying to solve the wrong problem.
Having to be the one at the helm during implementation made you intimately aware of not only the problem at hand, but the current state of the codebase. That's something you can't replace with automation. You can't compress all of that context into your brain in a handful of prompts with Claude.
Remember the words of your math teacher--
"Watching someone else solve the problem doesn't mean you can now solve it too."
Not sure what you mean as system design conversations because while in theory those can be good in practice the ones I have been at had been techbro wankery where the interviewer had a particular answer in mind. Like designing your own memcached clone for example is a terrible task for systems design.
What you're mentioning is 100% what's wrong with the industry. Agreed! To me a systems design conversation is a conversation - not a design goal. The idea is to determine ability and psychology:
1. When you press on someone's design respectfully, do they get defensive. Do they become argumentative.
2. When thoughtfully pointing out a concern, how does the candidate take it?
3. When you suggest a technology that makes no sense to intentionally challenge knowledge, does the candidate recognize why it makes no sense? Are they able to share what the negative of the approach is. If you indicate that you know the question is "senseless" but want their feedback, how do they communicate?
4. When you hard request a change that requires a literal rethink and rewrite do they become argumentative? Do they embrace the change?
5. When discussing testing, how do they think about it? I come down to the nitty gritty and ask about postive vs negative cases, table driven testing, what types of tests matter (for our situation) and why.
6. We discuss timeline tradeoffs, and then have the conversation about the candidate's approach given updates to see how they think.
You'll notice that I am never looking for a solution. I'm seeking communication, description, partnership while having a (relatively) thorough gasp of the subject matter.
Every single time I get a response from a candidate such as "I don't know, I'd have to learn more - or use AI to, or.. what do you think" turns out to be something I LOVE, because it creates a great fabric for the interview.
Programming languages, design languages and architecture are all inventions made to help humans write understandable source.
LLMs don’t really need to do any of that. They can store very large trees of understanding and therefore implement any application in raw binary. Why bother with abstractions at all
LLMs for sure need those things. maybe not the same abstractions as humans do but without understabdable code an LLM will just fail to accomplish the task you ask it to do.
I was thinking about that. I reckon generating massive amounts of synthetic training data for that goal should be possible - you've already got the Python, after all. It's also possible for machine code, but you'd have to target more platforms.
But ultimately, I think human readability outweighs any theoretical advantage you get from removing a step in the compilation process.
Targeting byte code or asm instead of high level would be silly for everyday tasks. You blow up the number of tokens, reduce your effective context, and there's just more places for it to make a mistake, which most likely won't be caught by the assembler (unlike compilers).
> The answer to that is to "donate" to about 270 congress critters.
Not too far off. You gotta ask why such a lucrative business has a near monopoly, and the answer is not that potential competitors don't notice their profit margin.
Yes. As a merchant it's pretty cool that the money just shows up and I don't have to chase down as much bad debt. The processing costs easily pay for themselves.
> "sweeten" the deal for the local towns where they are building
Yes. And datacenter companies have to foolish to slow-walk doing this. When you are building a 10 to 50 billion dollar construction project, it may be worth giving the local town 100 million (plus 10k directly to every resident) to do with as they please, and it would certainly grease the wheels.
They are slow walking it because you simply can't build the generation fast enough anyway. It's also why you are seeing a lot of project with their own generation plants on site, either stuff like the SpaceX/X/AI (I dunno what you call it man) datacenters - i.e very adhoc methane stuff or proper LNG generators ala Meta.
reply