Or: How We Had to Dig Up a 23-Year-Old Academic Paper Because the Industry Still Can't Agree on What an Agent Actually Is
Here we are in 2025, drowning in agent frameworks, agent orchestrators, and agentic workflows, yet somehow the tech industry collectively forgot to nail down what an "agent" actually means. While everyone's busy slapping the word "agent" onto their latest product launch, the most coherent definition still comes from Marco Colombetti and Pier Luca Lanzi's 2001 paper "Developing Rational Agents." Apparently, some problems require more than just throwing venture capital at them.
What Actually Is an Agent? (Since Nobody Else Seems to Know)
According to Colombetti and Lanzi, an agent is "a system that interacts with an environment continually and without human assistance in order to carry out a predefined task." This definition has three crucial components that most modern "agentic" systems conveniently ignore:
Continual interaction—not just one-shot responses
Environmental awareness—understanding and responding to changing conditions
Task autonomy—operating without constant human hand-holding
The fundamental relationship looks like this:
┌─────────┐
│ Agent │
└─────────┘
↑ │
x│ │a
│ ↓
┌─────────┐
│Environment│
└─────────┘
Where x represents the environment's state affecting the agent, and a represents actions the agent performs that affect the environment. This bidirectional flow is what separates true agents from glorified chatbots.
The Mathematical Foundation (Because Hand-Waving Doesn't Scale)
Real agent systems require mathematical rigor. The environment can be modeled as a finite-state stochastic process where the probability of transitioning from state x to state x' when action a is performed follows:
P(x(t+1) = x' | x(t) = x and a(t) = a) = T(x,a,x')
This transition function T: X × A → Π(X) maps state-action pairs to probability distributions over next states, with the constraint that probabilities sum to 1:
∑ T(x,a,x') = 1, for all x∈X and a∈A
x'∈X
The agent's perception of state x through sensors gives perception s with probability:
P(s(t) = s | x(t) = x) = σ(x,s)
This separation between objective environmental states and perceived states becomes crucial for understanding different agent architectures.
Reactive Agents: The Simplest That Could Possibly Work
Reactive agents operate on a simple principle: given current perception, execute action. They implement a direct mapping from perceptions to actions:
π: S → A
Where at every time step t, the action is computed as a = π(s).
The agent architecture looks like:
┌─────────────────────────────────┐
│ Agent │
│ ┌──────────┐ s ┌──────────┐ │
│ │Perception├────→│ Action │ │
│ └──────────┘ └──────────┘ │
└─────────────────────────────────┘
↑ │
x│ │a
│ ↓
┌─────────────────────────────┐
│ Environment │
└─────────────────────────────┘
The simplicity is both strength and weakness. Reactive agents can respond quickly but lack memory of past states or ability to plan ahead. They work well when the current perception contains sufficient information for optimal action selection.
Markov Agents: When Perfect Information Meets Perfect Memory
Markov agents are reactive agents operating in environments where perception is faithful (no noise or aliasing) and the environment follows the Markov property. This creates a special case where the agent can directly observe environmental states.
The system simplifies to:
┌─────────────────┐
│ Markov Agent │
│ ┌──────────┐ │
│ │ π │ │
│ └──────────┘ │
└─────────────────┘
↑ │
x│ │a
│ ↓
┌─────────────────┐
│ Environment │
│ ┌──────────┐ │
│ │ T │ │
│ └──────────┘ │
└─────────────────┘
For any policy π, we can compute the value function v_π,γ(x) representing expected discounted reward from state x:
v_π,γ(x) = R(x,π(x)) + γ ∑ T(x,π(x),x') · v_π,γ(x')
x'∈X
The discount factor γ ∈ [0,1] balances immediate versus future rewards. When γ = 0, the agent is completely greedy. When γ = 1, all future rewards have equal weight (though this can lead to infinite sums).
The remarkable property of Markov agents is that optimal policies always exist and can be computed using dynamic programming methods when the transition function T and reward function r are known.
Q-Learning: When Agents Must Learn the Hard Way
When transition probabilities are unknown, agents must learn optimal behavior through experience. Q-learning maintains a table of Q(x,a) values representing expected reward for taking action a in state x and then following the optimal policy.
The update rule is:
Q(x(t-1),a(t-1)) ← (1-α)Q(x(t-1),a(t-1)) + α(r(t) + γ max Q(x(t),a))
a∈A
Where α ∈ [0,1] is the learning rate controlling how much recent experience affects Q-values. The algorithm converges to optimal Q-values under assumptions that every state-action pair is visited infinitely often and the learning rate decreases appropriately.
Non-Markov Reactive Agents: When Reality Intrudes
Real environments rarely provide perfect information. Agents must deal with:
Perceptual aliasing: Different states producing identical perceptions
Perceptual noise: Same state producing different perceptions
The architecture expands to handle uncertainty:
┌─────────────────────────────────────┐
│ Non-Markov Agent │
│ ┌────┐ s ┌────┐ ┌────────┐ │
│ │ σ ├────→│ π ├────→│ │ │
│ └────┘ └────┘ └────────┘ │
└─────────────────────────────────────┘
↑ │
x│ │a
│ ↓
┌─────────────────────────────────────┐
│ Environment │
│ ┌─────┐ │
│ │ T │ │
│ └─────┘ │
└─────────────────────────────────────┘
The challenge becomes creating a Markov process on perceptions rather than states. This requires computing:
T_σ(s,a,s') = ∑ σ'(s,x) · T(x,a,x') · σ(x',s')
x,x'∈X
Where σ'(s,x) is the probability that environment is in state x given perception s. Unfortunately, this probability is generally non-stationary, making learning significantly more complex.
Cognitive Agents: Adding Brains to the Operation
When perception fails to provide sufficient information, agents need internal models of their environment. Cognitive agents maintain beliefs about environmental states rather than relying solely on current perception.
Bayesian Agents: Probability All the Way Down
Bayesian agents maintain a probability distribution over environmental states:
b ∈ Π(X)
Where b(x) represents the probability that the environment is currently in state x. This belief gets updated each cycle using Bayes' theorem:
σ(x,s) · ∑ T(x',a,x) · b_t(x')
x'∈X
b_{t+1}(x) = ────────────────────────────────────────
P(s(t+1) = s)
The architecture becomes:
┌─────────────────────────────────────────┐
│ Bayesian Agent │
│ ┌────┐ s ┌────┐ b ┌────┐ │
│ │ σ ├────→│ β ├────→│ π │ │
│ └────┘ └────┘ └────┘ │
└─────────────────────────────────────────┘
↑ │
x│ │a
│ ↓
┌─────────────────────────────────────────┐
│ Environment │
└─────────────────────────────────────────┘
Where β represents the belief update mechanism computing new beliefs from current perception, previous beliefs, and last action performed.
Agents with Memory: Finite State Solutions
For simpler cases with deterministic environments and noiseless but aliased perception, agents can use finite memory to disambiguate states. A memory register m helps distinguish otherwise identical perceptions:
x(t) = x_0 iff s(t) = s and m(t) = 0
x(t) = x_1 iff s(t) = s and m(t) = 1
The agent must learn both environmental actions and memory actions:
π_env: S × M → A
π_mem: S × M → M
This approach requires learning when and how to update memory registers, which adds significant complexity to the learning problem.
The Computational Reality Check
Each increase in agent sophistication comes with computational costs:
Reactive agents: O(|S|) space for policy storage
Markov agents: O(|X| × |A|) for Q-tables
Bayesian agents: O(|X|) per time step for belief updates
Memory agents: O(|S| × |M| × |A|) for full policy representation
As state spaces grow, these approaches become computationally intractable. Real systems require function approximation, generalization, and other techniques that introduce their own complexity and potential for failure.
Why This Still Matters (Especially Today)
While modern systems focus on large language models and neural architectures, the fundamental principles from 2001 remain relevant. Current "agentic" systems often lack:
Formal mathematical foundations for behavior analysis
Principled approaches to handling uncertainty and partial observability
Convergence guarantees for learning algorithms
Clear distinctions between different types of autonomous behavior
The authors presciently noted that "whether it is possible to do so within a unified mathematical approach is still to be proved" regarding integrating reactive, cognitive, and motivational components. Two decades later, we're still working on that integration.
The Uncomfortable Truth
Despite all the venture funding and product launches, we haven't fundamentally solved the core problems Colombetti and Lanzi identified in 2001. We've gotten better at scaling neural networks and handling natural language, but the fundamental challenges of autonomous reasoning, planning under uncertainty, and learning from experience remain as difficult as ever.
Perhaps instead of rebranding every API call as "agentic," the industry might benefit from revisiting these foundational principles. After all, you can't build robust autonomous systems without understanding what autonomy actually means mathematically. But then again, rigorous mathematical foundations don't make for great conference keynotes.
The real test of any "agent" system isn't whether it can complete a demo scenario, but whether it can maintain rational behavior across the infinite complexity of real-world environments. By that measure, most of today's agents are still very much works in progress.
This article draws from "Developing Rational Agents" by Marco Colombetti and Pier Luca Lanzi, published in Human and Machine Perception 3: Thinking, Deciding, and Acting (2001). While the paper is over two decades old, its mathematical rigor and systematic categorization of agent types remain more coherent than most contemporary discussions of "agentic" systems.