Paper: Welcome to the Era of Experience: David Silver, Richard S. Sutton
This paper claims that the next big leap in AI won’t happen by giving models more human generated data. AI agents must learn through their own real world experiences instead of relying on human data. We already use RL (reinforcement learning). For example AlphaGo learned by playing millions of games, and coding agents can learn from whether tests pass. The idea isn’t new to reinforcement learning, but moving from AI that learns what humans know to AI that discovers what actually works. It also mentions that the big challenge is making that continuous experimentation safe in the real world.
Imagine an AI learning to grow cherries. Today, it might read thousands of gardening books and tell you how much water and sunlight cherries need. It will take the human knowledge of the physical world as the base fact. In the Era of Experience, the AI could grow cherries itself, try different conditions, observe the results, and discover better methods that nobody has written about. It’s a feedback mechanism involves real world experience.
Today’s RL often optimizes models using predefined tasks, reward models, preferences of human, or automatically verified answers. In the paper, for the proposed “era of experience”, AI learns continuously from the consequences of its own actions in an environment, potentially over months or years.
The authors defines the AI eras: 1. Era of Simulation: AI learns by trial and error in simulated environments. Examples: AlphaGo, AlphaZero, Atari. Very powerful, but mostly in narrowly defined problems with clear rewards. 2. Era of Human Data: AI learns from human-written text, code, demonstrations, and preferences. Examples: GPT-style language models. Broadly capable, but constrained by the information and assumptions available in human data. 3. Era of Experience (their prediction): AI interacts continuously with the real world, tests hypotheses, receives feedback, and learns strategies that humans may never have imagined.
A sketch chronology of dominant AI paradigms. The y-axis suggests the proportion of the field’s total effort and computation that is focused on RL.:
In the era of experience (direct quote from the paper)
- Agents will inhabit streams of experience, rather than short snippets of interaction.
- Their actions and observations will be richly grounded in the environment, rather than interacting via human dialogue alone.
- Their rewards will be grounded in their experience of the environment, rather than coming from human prejudgement.
- They will plan and/or reason about experience, rather than reasoning solely in human terms
To achieve this era the agents must interact with the physical world (direct quote):
To progress significantly further, a new source of data is required. This data must be generated in a way that continually improves as the agent becomes stronger; any static procedure for synthetically generating data will quickly become outstripped. This can be achieved by allowing agents to learn continually from their own experience, i.e., data that is generated by the agent interacting with its environment.
I saw this paper from a Youtube video by WelchLabs. There in the video, an interesting queston was asked as a thought experiment. Actually the experiment also coming from the paper itself.
This is my favorite LLM thought experiment. If you trained an LLM using a really old cutoff date for your training data, like 1000 AD, what would happen? If you asked your 1000 AD cutoff model what made metal shiny, it would probably reason in theistic terms (answer: The LORD made that way). If you went further back to some of humanity’s oldest texts, it would probably answer in terms of animism (answer: animal spirits inside metal). If your cutoff was around 1700, this LLM would likely explain metal shyness using Newtonian physics (answer: light reflecting on smooth surface), and a more modern model would answer in terms of quantum physics (answer: free electrons absorbing and re-emiting photons).