How to Build an AI Agent from Scratch

Imagine a support assistant with access to an order database, a billing API, an audit log, and a policy archive. You don't have to spell out every database query in advance. You give it a goal, it decides which tool to use next, your code executes that tool, and the result becomes context for the next decision. That's the core of an AI agent.

Frameworks hide this loop behind helpful abstractions. This article strips those abstractions away. We'll build the loop directly, wire real tool calls, hit the first failure modes, and end with a production-oriented scaffold you can compare against LangGraph, CrewAI, or an OpenAI Agents SDK project later.

Before we start, you need to be comfortable writing Python, making HTTP requests, and reading JSON. If you've ever written a script that calls an API and parses the response, you have enough background. We will use the OpenAI Python client, but the same loop works with any model that supports tool calling.

By the end, you can explain the agent loop in plain English, implement a small tool-using agent, and recognize why production agents need limits, logging, memory, and human approval around the core.

What is an AI agent?

Strip away framework names and an AI agent is small in concept. It's made of three parts:

1. An LLM (Large Language Model) that can select the next step
2. A set of tools your code can execute when the LLM requests them (search, calculate, read files, call APIs)
3. A loop that runs until the task is done or the agent gives up

That's it. The LLM reads a prompt, decides whether it needs to use a tool, calls it, reads the result, and decides again. This cycle repeats until the LLM produces a final answer.

The important boundary is this: the LLM doesn't "do" anything itself. It decides what should happen next. The tools do the work. The loop connects decisions to actions to observations to more decisions.

Tools like LangGraph, CrewAI, and OpenAI's tool-calling APIs all implement this same loop with different abstractions. Understanding the raw loop makes every framework easier to learn because you can see which part of the loop the framework is managing for you.

One caution before you reach for the loop at all. Anthropic's guidance is to find the simplest solution first and only add complexity when it earns its keep, which sometimes means not building an agent.^[1] A workflow with fixed code paths is more predictable and cheaper than a model that decides its own steps. Reach for an open-ended loop when the task genuinely needs the model to choose what to do next based on what it observes. The rest of this article builds that loop so you understand it, not so you reach for it by default.

A trace through one turn

Before we look at code, let's walk through a concrete scenario. Suppose a customer asks: "Can I still return order 12345?"

Our agent has three tools:

• get_order_status(order_id) looks up an order in the database
• read_file(path) reads the billing policy from a file
• get_current_date() returns today's date

Here is how one loop iteration plays out:

1. User message: "Can I still return order 12345?"
2. Agent thinks: "I need the delivery date for order 12345 and the billing policy to check the time window."
3. Agent acts: It calls get_order_status with {"order_id": "12345"}
4. Observation: The database returns {"total": 89.99, "items": 3, "delivered": "2025-04-10"}
5. Agent thinks: "The order was delivered on April 10. I need the policy to know how many days I have."
6. Agent acts: It calls read_file with {"path": "policy.txt"}
7. Observation: The file returns "Returns accepted within 30 days of delivery."
8. Agent acts: It calls get_current_date with {}
9. Observation: The date tool returns "2025-05-11".
10. Agent thinks: "April 10 to May 11 is 31 days. That exceeds the 30-day window."
11. Final answer: "Order 12345 was delivered on April 10, so the 30-day return window closed on May 10. This order is no longer eligible for a return."

Notice what happened. The LLM never touched the database or the file system directly. It emitted structured requests. Our Python code executed those requests, appended the results to the conversation, and called the LLM again. The LLM selected steps and wrote text. The code handled the real work.

The simplest possible agent

Let's start with the smallest thing that could possibly work. We'll build an agent that can use exactly one tool: a calculator. In our support scenario, this might compute a shipping surcharge or a prorated refund. By constraining the environment to a single tool, we can focus entirely on the core loop that drives the agent's decision-making process.

To make this work, we provide the LLM with a strict definition of what the tool is, what it does, and what inputs it expects. We do this by passing a JSON schema that describes our function. When the model determines that a calculation is necessary, it generates a structured request matching that schema rather than only writing plain text.^[2]

The code below sets up the OpenAI Python client and defines a calculator tool. The example uses Chat Completions because the raw message loop is easy to inspect directly: every assistant message, tool call, and observation is a plain dict you append to a list. OpenAI now recommends the newer Responses API for tool-calling and multi-turn projects, and it manages more of this state for you, but the control flow is identical. Define tools, let the model choose, execute the call, append the result, and continue. To run it locally, install the openai package, set OPENAI_API_KEY, and use a tool-capable model your account can access.^[2]

The exact sentence depends on the model, but the answer should include 1103.

This is already a working agent loop. The while True block repeats model calls. The if choice.finish_reason == "tool_calls" branch is where the model asks to act. Your code parses the arguments, executes the tool, appends the observation, and calls the model again.

The calculator uses an AST whitelist instead of eval(). That matters even in a tutorial. Readers copy code, and tool inputs are model-generated strings. Treat those strings as untrusted unless a sandbox, parser, or allowlist proves otherwise.

The mistake to avoid is stopping here. A raw loop without error handling, timeouts, cost tracking, loop limits, structured logging, and human approval is a demo, not a production system. We'll add those layers after the core pattern is clear.

Adding real tools

A calculator is fun, but a useful agent needs tools that interact with the world. Let's add three more tools that fit our support scenario: an order lookup, a policy reader, and a date helper. The following snippets show how to define these operations. Each tool accepts simple string arguments and returns string results, expanding what our agent can accomplish.

The @tool decorator is a pattern you'll see in many agent frameworks. It registers tools in a lookup table so the agent loop can find the right function when the model asks for it.

Now the agent loop becomes generic. We implement a dispatcher function that takes the tool's name and its arguments, and dynamically executes it. If the requested tool is missing from our registry, it gracefully returns an error string.

The ReAct pattern

What we've built so far follows the same outer loop popularized by ReAct (Reasoning + Acting)^[3]. In the original paper, the model emits explicit reasoning traces alongside actions. In production systems that reasoning may be hidden, summarized, or replaced with planner state, but the outer loop is the same:

Each iteration of the loop has three phases:

1. Think (Reason): The LLM reads the conversation history and decides what to do next
2. Act: It calls a tool with specific arguments
3. Observe: The tool result is added to the conversation, and the loop continues

Conceptually, the agent's next step might be: "I need to search for both values, then calculate the product." In production systems that reasoning may be an explicit scratchpad, a hidden reasoning trace, or a planner state. The important part is the loop: each tool result feeds back into the next decision.

For a deeper architecture comparison, the LeetLLM lesson on ReAct, Plan-and-Execute, and other agentic architectures covers when ReAct works, when it doesn't, and what alternatives exist for complex multi-step tasks.

Where it breaks

Here's where things get interesting. Once you move beyond calculator demos, five failure modes show up quickly. These aren't edge cases. They're the problems you hit first when a tool-using loop meets messy real-world tasks.

Failure 1: The infinite loop

The agent calls the same tool with the same arguments, gets the same result, and tries again without making progress.

What causes it

The model doesn't recognize that the tool result already answered the question, or it doesn't know how to recover from an error.

The fix

Add a maximum iteration count to stop the loop after a set number of steps, prevent unbounded recursion, and bound token cost. We pass the user message into our loop and enforce a strict loop boundary. If the task is incomplete after these iterations, we return a fallback string instead of waiting indefinitely.

Failure 2: Hallucinated tool calls

The model invents a tool that doesn't exist ("I'll use the query_database tool to...") and crashes when the lookup fails.

What causes it

The tool list in the system prompt doesn't match the model's expectations, or the model is trying to use tools it's seen in training data.

The fix

Validate tool names before execution, and return a helpful error back to the model so it can realize its mistake and recover. The updated lookup function below verifies the incoming tool name. When an invalid tool is requested, it provides an error message containing the valid options, enabling the agent to self-correct.

Failure 3: Context window overflow

After enough tool calls, the conversation history exceeds the model's context window. The agent either crashes or starts losing the original instructions.

What causes it

Each tool result adds tokens. A policy file might return 500 tokens. After 10 tool calls, you've consumed 5,000+ tokens of context just on tool results.

The fix

Summarize or truncate tool results before adding them to context. The following function cuts off long strings to avoid exceeding the maximum token limit. It takes the raw text output from a tool and ensures it stays below a specified character threshold, returning a safer, shortened string.

Failure 4: Argument parsing errors

The model generates malformed JSON for tool arguments. This happens often with smaller models.

What causes it

LLMs don't always produce valid JSON, particularly under complex argument schemas.

The fix

Retry with an error message so the model can attempt to fix its JSON format. Better yet, enable strict structured tool schemas when your provider supports them.^[4] By wrapping the JSON parser in a try-except block, we catch decoding failures. We then append the error as a tool response, prompting the model to re-generate valid arguments in the next iteration.

Failure 5: Wrong tool selection

The model picks the wrong tool for the job. It tries to calculate a date comparison, or read_file for a fact that's already in the conversation history.

What causes it

Bad tool descriptions. If the description is vague ("do math stuff"), the model can't decide when to use it.

The fix

Write precise tool descriptions that include when to use the tool, not just what it does. Here's a comparison between a vague description and an effective one. The updated description clarifies expected inputs and scenarios, which helps the LLM select the correct operation for its current step.

The LeetLLM lesson on Agent Failure States, Retries, and Fallback Strategies catalogs more failure modes and recovery strategies. This article focuses on the first five because they appear as soon as you move from a calculator demo to real tools.

Adding memory

Our agent has a problem: it forgets everything between runs. Each invocation starts fresh. For many use cases that's fine, but conversational and task-continuation agents usually need more than one memory layer. In practice it's useful to separate short-term context, retrieval memory, and durable application state.

Short-term memory (conversation history)

Short-term memory keeps the messages array around between calls instead of rebuilding it each time. Here's how we initialize the agent with a persistent message list. We define a class that stores the system prompt upon creation. The run method then appends the user's input and maintains the ongoing context across multiple questions.

This approach works for short interactions, but it scales linearly in cost and latency. Every new turn forces the model to re-process the entire conversation history, which eventually hits context window limits. For simple agents, however, this in-memory list is the most direct way to give the model a sense of continuity.

Retrieval memory (persistent)

For information that persists across sessions, you need a database. A simple retrieval layer stores embeddings and lets the agent fetch semantically related context when it needs it. We can expose this as a tool the agent can decide to call. The remember tool accepts a string fact and inserts its embedding, while the recall tool searches for related memories based on a text query.

By treating memory as another set of tools, the agent can request saves and recalls through the same loop, subject to your product policy. In more advanced setups, a separate process might summarize conversation history and extract these facts without requiring explicit tool calls from the primary reasoning agent. The following sketch shows the remember and recall tool interface.

One warning: a vector database is great for fuzzy recall, but it's the wrong source of truth for exact state like account balances, permission flags, or whether an order was already shipped. Keep exact business facts in a relational or key-value store and expose that state through tools.

The LeetLLM lesson on Agent Memory and Persistence Patterns covers the full spectrum, from simple conversation history to episodic memory and working memory architectures.

Making it production-ready

The 200-line agent works for demos. For production, you need several more layers. Here's a checklist:

Cost controls

LLM calls are expensive. A runaway agent can burn through hundreds of dollars in minutes. To prevent this, implement a cost tracking class that monitors token usage and halts the agent if it exceeds a specified budget. The CostTracker examines the usage metrics from each API response. Because providers expose usage fields and pricing tables a little differently, keep the per-model rates in configuration instead of hardcoding them deep in the loop.

Observability

When an agent fails, you need to know why. Log each loop step to create an audit trail of the agent's actions, arguments, and outcomes. By adding standard Python logging calls to each phase, we capture the chosen tools, their arguments, and the resulting values. This gives you a clean execution trace without pretending you can or should log the model's hidden chain-of-thought.

Timeouts

A tool might hang. A web search might time out. You need per-tool timeouts and a total execution timeout. A portable pattern is to run the tool in a worker and wait with a timeout. This helper function takes the tool's name and arguments alongside a timeout duration. It either returns the successful result or catches the timeout to yield an error string.

This pattern lets your main loop move on, but it doesn't terminate arbitrary Python code that's already stuck in a thread. For untrusted or side-effectful tools, use a separate process or sandbox you can kill cleanly.

Human-in-the-loop

For high-stakes actions (deleting files, sending emails, making payments), require human approval before execution. You can intercept specific tool calls and pause the process to request explicit confirmation. The updated function checks the requested tool against a restricted set. It pauses to prompt the human user and proceeds only if explicit permission is granted.

The LeetLLM lesson on Human-in-the-Loop Agent Architecture covers escalation policies, approval workflows, and how to design agent UX that builds user trust.

Evaluation

A final-answer check isn't enough for agents. Two very different trajectories can produce the same answer, and a lucky answer can hide terrible tool use. Evaluate both the outcome and the path: did the agent choose the right tool, stop at the right time, and recover correctly when something failed?

For general-purpose agents, benchmarks like GAIA measure multi-step reasoning and tool use.^[5] For coding agents, SWE-bench measures whether the system can resolve real repository tasks end to end.^[6] If you need scalable review, an LLM judge can score trajectories or outputs, but sample with humans too because judge models have blind spots of their own.^[7]

The complete agent

Putting it all together, here is a production-oriented scaffold. This version still fits in roughly 200 lines of actual logic, but now it wires together the guardrails from earlier sections: bounded loops, short-term memory, cost tracking, JSON validation, and timeout-aware tool execution.

By encapsulating the agent logic within a class, we can maintain state across multiple conversational turns. The class constructor initializes the agent with a system prompt, a predefined set of tools, and constraints on iterations and budget.

The run method orchestrates the lifecycle for a given user message. It takes the user's input, enters the iterative decision loop, safely executes any requested tools, appends the results to the context, and returns a final answer.

This is enough to back a real CLI or HTTP service. In a deployed system you would still add persistence, auth, retries with backoff, and structured traces around this loop.

Try it yourself

The best way to solidify this is to build something small and see where it breaks.

Exercise: Extend the order-support agent with a new tool called check_return_eligibility(order_id). This tool does four things:

1. Call get_order_status to find the delivery date.
2. Call read_file to read the billing policy from policy.txt and extract the return window (for example, 30 days).
3. Call get_current_date to check today's date.
4. Return either "Eligible for return" or "Return window expired on YYYY-MM-DD".

Solution sketch

You don't need to change the LLM loop at all. A simple approach is to implement check_return_eligibility as a regular Python function that calls the other tools directly, then register it with the @tool decorator. The agent will then be able to use this compound tool in one shot, or you can let the LLM orchestrate the three individual calls itself and observe which approach produces cleaner behavior.

If you let the LLM orchestrate the three calls, watch for these behaviors:

• Does it call get_order_status first, or does it try to read the policy before it knows the account ID?
• Does it correctly parse the ISO date strings and compare them?
• Does it stop after one eligibility check, or does it loop redundantly?

Before you hand this to the LLM, test the business rule without a model. That separates "my Python is wrong" from "the model chose the wrong tool path."

Expected output for a first attempt

A beginner agent often returns a verbose explanation instead of a concise yes/no. That's fine. The goal is to see the tool calls line up with the reasoning.

Quick self-check

Key takeaways

After building this from scratch, these are the practical lessons:

1. Tool descriptions carry routing policy

The prompt and model matter, but tool descriptions are routing policy. A bad tool description causes the model to pick the wrong tool or pass wrong arguments, and then the entire agent goes off the rails. Invest time here.

2. Start with one tool and add more gradually

Starting with 1-2 tools and adding more as needed usually produces cleaner behavior than starting with a large tool catalog. When you have too many tools, the model struggles with selection.

The agent loop is small. The hard engineering lives around it. The core loop (call the model, execute a tool, repeat) is the easy part. Cost controls, timeouts, error handling, observability, and human approval are where production reliability comes from.

3. Smaller models need more guardrails

Smaller or less tool-tuned models need stricter schemas, more examples in tool descriptions, and tighter output parsing. If you're building with open-weight models, budget extra time for tool reliability.

4. Have an escape hatch

Max iterations, cost caps, and timeouts aren't optional. An agent without limits is a liability. Set conservative limits first and relax them based on observed behavior.

Now that you understand the core agent loop, MCP and Tool Protocol Standards will make more sense because MCP standardizes how tools, resources, and prompts are exposed to agents. Structured Output and Constrained Generation is the natural next concept for making tool argument parsing more reliable.

Going further

What we built here is a single-agent ReAct loop. This is a common baseline architecture, and it works well for straightforward multi-step tasks where each observation can guide the next action. Production agent systems often use more structured orchestration patterns for complex workloads.^[8]

For example, when a task requires rigorous planning or extensive coordination, you might encounter advanced frameworks:

• Plan-and-Execute: The agent first creates a plan (a list of steps), then executes each step. Better for multi-step tasks.
• Multi-Agent Systems: Multiple specialized agents that coordinate. One researches, another writes, a third reviews.
• DAG-Based Orchestration: Complex workflows modeled as directed acyclic graphs with conditional branching.
• Tool Protocol Standards: MCP (Model Context Protocol)^[9] is an open standard for exposing tools, resources, and prompts through a common client-server interface, which makes integrations more portable.

These advanced architectures might seem intimidating at first, but they all build on the same foundation: an LLM, a set of tools, and an iterative loop. Once you understand this core loop, the rest is mostly about arranging the same building blocks into different configurations.

You now have a working mental model of that foundation. The next articles in the LeetLLM path go deeper: ReAct and Plan-and-Execute architectures shows when to switch from a simple loop to a planner, Function Calling and Tool Use dissects how providers structure tool schemas and enforce argument shapes, and Agent Failure States and Recovery catalogs the full set of production failure modes and tested recovery strategies.