Skip to content
Novandri.

From Chatbots to Autonomous Systems: The Evolution of AI

The arc from learned control to tool use to runtime planning. Read what each step added, what it did not, and what is still missing.

Rachmadwipa Novandri9 min read

The story of AI "becoming autonomous" is usually told as a straight line with a destination: chatbots answered questions, then models could use tools, then agents could plan, and eventually systems will run themselves. Read as a sequence of papers, the line is messier and more interesting, because each step added a specific capability and left a specific gap that the next step happened to address.

Direct answer: Three documented steps take you from a model that answers to a system that acts. Learned control policies gave the ability to act on raw sensory input without a hand-written policy. Self-supervised tool use gave the ability to delegate a capability the model is bad at to a program that is good at it. Runtime planning gave the ability to revise a course when reality does not match the expectation. The gaps left over are exactly where the next step had to come from.

Each step is a change in what the system decides at runtime, not in how clever the model is.

A four-step flow: learn to act, delegate to tools, plan at runtime, then read the remaining gap.
Conceptual: capability added, gap left, next step filled it. Each step is a change in what the system decides at runtime, not in how clever the model is.

Step one: learning to act from raw input

The first step is not about language at all, and it is the foundation for everything that follows.

Mnih et al. present "the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning". The architecture is deliberately plain: "a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards". They applied it to seven Atari 2600 games "with no adjustment of the architecture or learning algorithm", and found it "outperforms all previous approaches on six of the games and surpasses a human expert on three of them".

3 of 7Atari games on which the learned policy surpassed a human expertMnih et al., 2015

Two details in that sentence set the pattern for everything after. First, the same network learned seven different games without modification — the capability came from the learning procedure, not from per-task engineering. Second, the output was a value function, not a plan: the system estimated future reward, and the behaviour followed from the estimate rather than from a written rule.

The gap this left is obvious in hindsight. A learned policy can act, but it cannot consult anything. It has no way to ask a calculator, read a document, or check a fact. Everything it knows is in its weights.

Step two: delegating what the model is bad at

That gap is what the tool-use line addresses, and the framing in Schick et al. is the sharpest statement of it. Language models, they write, "exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale. They also, paradoxically, struggle with basic functionality, such as arithmetic or factual lookup, where much simpler and smaller models excel."

The word "paradoxically" is doing real work: the same scale that makes a model good at open-ended tasks makes it bad at precise ones. Toolformer's answer is to have the model learn to call programs instead of trying to out-guess them. It is "a model trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction" — and the training is self-supervised, "requiring nothing more than a handful of demonstrations for each API".

The tool set they used is worth studying because it is deliberately unglamorous: "a calculator, a Q&A system, two different search engines, a translation system, and a calendar". Every one is a capability a language model is measurably worse at than a dedicated implementation. The reported result is "substantially improved zero-shot performance across a variety of downstream tasks, often competitive with much larger models, without sacrificing its core language modeling abilities."

What this step added: the ability to reach outside the weights. What it left open: the sequence of calls.

Step three: planning at runtime

The third step is the one usually called "agents", and its contribution is a representation of the plan itself.

Yao et al. generate "both reasoning traces and task-specific actions in an interleaved manner", and explain why the interleaving matters: "reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it to interface with external sources, such as knowledge bases or environments, to gather additional information."

The measurable difference appears where the previous steps fail hardest. On question answering and fact verification, Yao et al. report that ReAct "overcomes issues of hallucination and error propagation prevalent in chain-of-thought reasoning by interacting with a simple Wikipedia API". The mechanism is specific: a reasoning-only chain compounds its own errors, because each step is conditioned on the previous wrong step. An interleaved action returns ground truth from outside, which breaks the compounding.

So the progression is coherent. Learned policies could act but could not consult. Tool use could consult but could not revise. Interleaved reasoning and acting can both, because the reasoning trace is an inspectable representation of the plan that the model can rewrite mid-run.

What the sequence does not tell you

Three cautions belong alongside the narrative.

The steps are not a ladder of generality. The Atari work had nothing to do with language and everything to do with the idea that behaviour can be learned rather than written. The tool-use work depends on the tool set being small and well-documented. Neither is "more advanced" than the other; they solve different problems.

The evaluation results are narrow. "Surpasses a human expert on three of seven games" is a remarkable result and also a bounded one. "Substantially improved zero-shot performance across a variety of downstream tasks" is a generalisation claim about a set of tasks the authors chose. The lesson is about capability, not about reliability on your workload.

The hardest steps are the ones nobody writes papers about. The pipeline around an agent — what it is allowed to do, what it is measured against, who answers when it is wrong — is engineering, and it does not get summarised in an abstract. It is also where the value of a system is decided. See AI Governance: Managing AI Responsibly at Scale and The Limits of AI for that side.

Reading the arc as a practitioner

The useful way to use this history is as a checklist for your own system. Each step implies a question:

  1. Can the system act on what it observes, or does it only classify? If it only classifies, it is a model and it belongs behind a decision someone else makes.
  2. Can it consult a capability it is worse at? If not, it is burning accuracy on arithmetic and lookups. Integrate the tool.
  3. Can it revise its plan when reality disagrees? If not, it will act confidently on a stale assumption, and the failure compounds with every step.
  4. Is its plan observable? Reasoning traces are text, so they can be logged and reviewed. That is a debugging asset, not a presentation detail.
  5. Who answers for it? Not optional at any of these steps, and least optional at the last one.

None of this requires the newest architecture. It requires knowing which of the three capabilities your workflow actually needs — and buying the narrowest one that does the job.

A worked example

The following is hypothetical and illustrative. A support assistant is asked to answer a billing question that needs a policy lookup and a refund calculation. Check one, acting from input: like the Atari policy in Mnih et al., the assistant maps what it observes to an action rather than following a hand-written script, so the behaviour is learned rather than enumerated. Check two, consulting a better tool: following the framing in Schick et al., the assistant delegates the arithmetic and the factual lookup to a calculator and a search program instead of guessing, since the same scale that helps open-ended language hurts precise computation. Check three, revising the plan: in the manner of ReAct in Yao et al., the assistant interleaves a reasoning trace with actions, so when the search returns a policy that contradicts its first assumption it rewrites the plan and the trace can be logged and reviewed. The bounded outcome is a correct answer with an inspectable record; the system still answers for one narrow workflow, and the article's caution holds that capability is not reliability on an untested workload.

A counterargument

A reasonable critic could object that the article's three-step arc is a retrospective construction rather than a documented progression. The Atari, Toolformer, and ReAct papers were not written as chapters of one plan, and the claim that each step "left a gap that the next step happened to address" reads a direction into work the authors did not claim, even though the article itself warns the steps are not a ladder of generality. A critic could also press on the evidence: the reported gains are bounded - surpassing a human expert on three of seven games and "substantially improved" performance on author-chosen tasks - and the article admits capability is not reliability.

The sequence is a useful reading aid; calling it an evolution risks making a curated set of papers look like an inevitable trajectory.

References & Further Reading