Recently listened to the Machine Learning Street Talk podcast with Jeremy Howard posted a couple weeks ago. He spent quite a bit of time talking about the limitations of LLMs. Some highlights that I thought were interesting, in chronological order (warning: long!):
“[Piotr Wozniak] believes that creativity comes from having a lot of stuff remembered, which is to say putting together stuff you've remembered in interesting ways is a great way to be creative. LLMs are actually quite good at that, but there's a kind of creativity they're not at all good at, which is, you know, moving outside the distribution... you have to be so nuanced about this stuff because if you say like they're not creative, it gives you the--can give you the wrong idea because they can do very creative seeming things. But if it's like, well, can they really extrapolate outside the training distribution? The answer is no, they can't. But the training distribution is so big, and the number of ways to interpolate between them is so vast, we don't really know yet what the limitations of that is.”
“I see it every day, you know, because my work is R&D. I'm constantly on the edge of and outside the training data. I'm doing things that haven't been done before. And there's this weird thing, I don't know if you've ever seen it before, I see it but I see it multiple times every day, where the LM goes from being incredibly clever to like worse than stupid, like not understanding the most basic fundamental premises about how the world works. And it's like, oh, whoops, I fell outside the training data distribution. It's gone dumb.”
“I think Boden might be pretty shocked at how far compositional creativity can go when you can compose the entirety of the human knowledge corpus. And I think this is where people often get confused.”
“the vast majority of work in software engineering isn't typing in the code.”
“anytime I've made any attempt to getting an LLM to like design a solution to something that hasn't been designed lots of times before, it's horrible.”
“LLMs cosplay understanding things. Like, they pretend to understand things.”
“The difference between pretending to be intelligent and actually being intelligent is entirely unimportant, as long as you're in the region in which the pretense is actually effective, you know. So it's actually fine for a great many tasks that LLMs only pretend to be intelligent, because for all intents and purposes, it just doesn't matter until you get to the point where it can't pretend anymore. And then you realize, like, oh my god. This thing's so stupid.”
“And getting better at the particular prompting skills, whatever details of the current generation of AI, CLI frameworks isn't growing. You know, that's like that's as helpful as learning about the details of some AWS API when you don't actually understand how the Internet works, you know. It's not reusable knowledge. It's ephemeral knowledge. So like, if you wanted to, you can actually use it as a learning superpower. But also, it can do the opposite.”
“No one's actually creating 50 times more high quality software than they were before. So we've actually just done a study of this, and there's a tiny uptick, tiny uptick in what people are actually shipping. That's the facts.”
“all of the pieces that make gambling addictive are present in AI based coding.”
“almost everybody I know who got very enthusiastic about AI powered coding in recent months have totally changed their mind about it when they finally went back and looked at, like, how much stuff that I built during those days of great enthusiasm am I using today? Are my customers using today? Am I making money from today? Almost all the money is being made by influencers, you know, or by the companies that produce the tokens.”
“The thing about AI based coding is that it's like a slot machine, and that you you have an illusion of control, you know, you can get to craft your prompt, and your list of MCPs, and your skills, and whatever, and then in the end, you pull the lever. Right? You put in the prompt, and something comes back, and it's like cherry, cherry, it's like, oh, next time I'll change my prompt a bit, I'll add a bit more context, pull the lever again, pull the lever again. It's the stochastic thing. You get the occasional win. It's like, oh, I won. I got a feature. So it's got all these hallmarks of like, loss disguised as a win, somewhat stochastic, feeling of control, all the stuff that gaming companies try to engineer into their gaming rooms.”
“I say that empirically. They're really bad at software engineering. And then I think that's possibly always gonna be true, because, you know, we're asking them to often move outside of their training data, you know, if we're trying to build something that literally hasn't been built before and do it in a better way than has been done before, we're saying, like, don't just copy what was in the training data.”
“this is a confusing point for a lot of people, because they see AI being very good at coding. And then you think like, that's software engineering. You know, it's like, it must be good at software engineering. But it's they're different tasks. There's not a huge amount of overlap between them. And there's no current empirical data to suggest that LLMs are gaining any competency at software engineering.”
“if you want to build something that's not just a copy, then you can't outsource that to an LLM. There's no theoretical reason to believe that you'll ever be able to, And there's no empirical data to suggest that you'll ever be able to.”