Even GenAI’s biggest boosters are finally admitting that LLMs don’t scale.
(Image from Google Gemini. When I asked to put it in a MAGA hat it refused.)
The idea that training databases to spit out freeform answers to general questions would result in Artificial General Intelligence (AGI) or some form of super-human intelligence was daft to begin with.
Thinking isn’t just pulling data out of your ass. At least outside Silicon Valley it’s not. Pure calculation has been the hallmark of machine intelligence for generations. Creating giant unstructured databases and pulling random facts from them is more than a generation old. Getting a trained database to parse a general question, answering it in a variety of forms, is simply the next step along that road.
Creativity encompasses a host of disciplines, many of which we barely understand. We’re not fruit flies, so modeling the brain of a fruit fly only lets you understand how a fruit fly thinks. The artificial fruit fly brain can’t even fly. If it had a consciousness, it would feel it was in fruit fly hell, immortal yet unable to impact its environment, even to become part of the Administration.
Thinking About Thinking
I wish I were going to next month’s Heidelberg Laureate Forum because that’s where a lot of tomorrow’s AI ideas are certain to be presented.
So I asked Google to look at some of the newest AI methodologies and here’s what it offered:
- Torque Clustering: Discover patterns and structures within a database mimicking how we learn by observing, exploring, and interacting with the environment.
- JEST (Joint-Embedding Self-Training): From Google, build a “reference model” that can start on hard questions by first evaluating their difficulty, like a student separating tough homework problems from easy ones. This is supposed to reduce training time by up to 13 times.
- Self-Taught Evaluator (Meta): A system for evaluating AI models using just AI, the software creating its own tasks and adjusting strategies based on its evaluations.
- Neuro-Symbolic AI: A concept pushed by Gary Marcus that combines human reasoning with AI models, pre-training an AI program on real world truths so it doesn’t make stuff up.
Heidelberg is important because the models listed above are simply add-ons to existing LLMs. We’re either putting a process in front of the LLM or having the LLM create its own pre-process. The problem is that machine learning doesn’t yet model human thinking, so putting a different process on top of machine learning doesn’t guarantee anything.
Beyond Descartes
People already have a lot of meta-cognition. Thinking about thinking is as old as Descartes.
But putting another process on top of, or in front of, any LLM may not be the way toward Artificial General Intelligence at all. Maybe we need to first create a model of human thinking and apply that to the machine, instead of going the other way.
What LLMs have done is create enormous market demand for these new ideas in modeling thought. We need to get inside our own heads before we can model them in any algorithm. This creates enormous opportunities for brain scientists, for mathematicians, and for computer scientists who can model human thought to computers, rather than adapting computers to basic human thought processes. All those kids going to Heidelberg next month have the chance to become rock stars.
Since we’re going to have the capacity for this, after LLMs crash, let’s use it.
