The Truth Can’t Be Improved Upon
Hey there, let’s chat about something cool in the AI world - this idea that we’re hitting a data wall with large language models (LLMs). Spoiler alert: I don’t buy it.
Here’s the deal: it might look like LLMs are plateauing, but there’s actually a really interesting reason for that. A lot of the questions we throw at these models have correct answers. Think about it - if you ask an LLM what 2 + 2 is, and it says 4, how can you improve on that? You can’t, because you can’t improve upon truth. That’s a powerful principle we need to keep in mind.
So, why does it seem like we’re hitting a wall? It’s not because of a lack of data or improvements in the models. It’s because many of the questions we’re using to test these models have true answers that most state-of-the-art LLMs are already nailing. When you’re already getting the correct answer most of the time, it’s extremely hard to show visible improvements.
The Next Frontier: Action and Planning
Now, here’s where things get really interesting. I think the next big leap in LLM development is going to be in agentic and action-based improvements. We need to set up better ways to evaluate these models that go beyond just judging good or bad responses.
For example, take the LLM SIS leaderboard. It’s cool, but it’s mostly focused on how humans perceive the responses. What we’re missing is a solid evaluation of planning abilities. We need to create tests where the questions are all about completing tasks, and we judge how well the LLM can create a proper plan given its tools and resources.
This shift towards action and planning is going to be a game-changer. It’s not just about spitting out facts anymore - it’s about using knowledge to create meaningful plans and take actions. That’s where we’ll see the real improvements and breakthroughs.
The Bottom Line
So, are we hitting a data wall with LLMs? Nah, that’s not the case at all. What we’re seeing is the natural result of these models getting really good at answering factual questions. The next big leap is going to come from expanding what we ask these models to do - moving from simple Q&A to complex planning and decision-making tasks.
Remember, you can’t improve upon truth. But you can definitely improve how you use that truth to make decisions and take actions. That’s where the future of LLMs is heading, and I can’t wait to see what comes next.
- Joseph
Sign up for my email list to know when I post more content like this. I also post my thoughts on Twitter/X.
