What Five Weeks of AI_devs 4 Taught Me About Building With LLMs

AI slop sucks. And “AI” is not even AI. Large language models are glorified word generators. Yeah, AI is everywhere and it’s easy to hate it. To say the least: I was hesitant.

I saw how quickly trends come and go in the “AI” world. Getting specialized in something that’ll be obsolete within months is laughable. But hey, I quickly learned that there are concepts that will stay with us. Prompting is one of them, and I quickly learned that every letter of the prompt can matter.

And prompt is only the visible edge of a larger system. Behind it are decisions about data, tools, memory, permissions, observability, product design, cost, failure modes, and the boundary between deterministic software and probabilistic reasoning.

All of that was quite… exposed when tons of data was leaked from Anthropic recently.

Anyway, the course lasted five weeks and covered a wide range of topics: language model APIs, structured outputs, function calling, MCP, multimodality, context engineering, RAG, agent design, evaluation, internal company workflows, production architecture, and AI security.

But the deeper lesson was simpler:

Building useful AI systems is not only about asking models better questions. It is about designing environments in which models can perform useful work reliably.

That distinction changed how I think about AI engineering.

First Principles

Before summarizing the course, it helps to reduce the topic to first principles.

An AI system has five fundamental parts:

1. A model that can reason, generate, classify, transform, and decide.
2. Context that shapes what the model can understand in the current moment.
3. Tools that let the model affect the outside world.
4. State and memory that preserve information beyond a single interaction.
5. Control systems that make the whole process observable, secure, measurable, and economically viable.

If any of these parts is weak, the system becomes unreliable.

A strong model with poor context gives confident nonsense. A strong model with dangerous tools becomes a security risk. A strong model without evaluation becomes impossible to improve. A useful prototype without cost controls may ruin your budget. A clever agent without permissions may do the wrong thing at scale.

This is why AI engineering needs to be treated as engineering, not prompt improvisation.

The core questions are not only:

• What should I ask the model?
• Which model is best?
• How do I make the answer sound right?

The better questions are:

• What does the model need to know?
• What should the model never be allowed to decide?
• Which parts should be deterministic software?
• Which parts benefit from probabilistic reasoning?
• How do I validate the output?
• How do I observe failures?
• How do I keep data, tools, and users safe?
• How do I measure whether the system is improving or deteriorating?

AI_devs 4 was valuable because it kept returning to those questions from different angles.

The Shift: From Chat to Systems

The first stage of the course focused on programming interactions with language models. That included model APIs, structured outputs, JSON Schema, prompt organization, provider capabilities, smaller versus larger models, and the practical limits of working with token-based systems.

The important idea was that a language model should not be treated as a text box. It should be treated as a component inside an application.

That means the application needs contracts.

If the model returns data, the data should be structured. If the system expects JSON, the schema should be explicit. If the model can fail, the software should handle that failure. If the result triggers a business action, it should be validated before execution.

This is one of the clearest boundaries between experimentation and engineering. In a demo, a natural-language answer may be enough. In a product, the model has to cooperate with software.

The same principle applied to multimodality. Images, PDFs, audio, and video are not just “attachments”. They require ingestion, transformation, preprocessing, interpretation, and often multiple steps before the model can use them well.

A production AI application is rarely one model call. It often is a pipeline.

Tools Are Interfaces for Models

The next major concept was tool use: function calling, custom tools, native tools, APIs, CLI tools, and MCP.

This was one of the most practical parts of the course because it reframed APIs from a model’s perspective.

Developers usually design APIs for other developers. AI systems force a different question: can a model understand this interface well enough to use it correctly?

That changes the design criteria.

Names matter. Descriptions matter. Parameter schemas matter. Defaults matter. Error messages matter. Permission boundaries matter. The response after a successful tool call matters because it becomes part of the next step of reasoning.

A tool is not just an implementation detail. It is part of the model’s working environment.

A poorly described tool can make a capable model look weak. A focused tool with clear schema and useful feedback can make a smaller model effective. Too many tools can overwhelm the context window and make selection unreliable. That leads to patterns like dynamic tool lists and tool search, where the model first discovers the right capability before using it.

The lesson for me was direct: tool design is AI system design.

Context Is Architecture

One of the strongest themes in the course was context engineering.

At first, it is tempting to think that larger context windows solve the problem. They do not. More context can mean more noise, more cost, more latency, and more opportunities for the model to lose track of what matters.

Context is not a storage bucket. It is an active design decision.

The course covered dynamic system instructions, conversation state, external documents, masking, shared context between threads, global context, local task context, and strategies for deciding what should be placed in the prompt versus stored elsewhere.

This is where the distinction between context, memory, and knowledge became important.

Context is what the model sees now.

Knowledge is information the system can retrieve when needed.

Memory is information the system preserves across interactions and uses to adapt future behavior.

Conflating these creates weak systems. Dumping everything into the prompt is expensive and fragile. Hiding everything behind retrieval can fail when search quality is poor. Letting memory grow without structure creates noise and risk.

Good AI systems decide what belongs where.

That principle also shaped the course’s treatment of RAG. Retrieval-augmented generation is not solved by adding a vector database. It depends on source quality, document structure, metadata, chunking, query transformation, ranking, context presentation, and evaluation.

The first-order principle is simple: the model can only reason over the information it receives, and retrieval decides what information arrives.

Agents Need Boundaries

The course moved from workflows to agents, but it did not present agents as magic.

A workflow follows known steps. An agent receives a goal, context, and tools, then decides some of the steps itself. That autonomy is useful when the path is uncertain, but it also increases risk.

The mature question is not “Can I make this an agent?”

The mature question is “What level of autonomy does this task deserve?”

Some tasks should remain deterministic. Some should use a model for classification or extraction. Some should use an LLM inside a controlled workflow. Some justify an agent with tools, memory, and planning. Some require multiple specialized agents with separate responsibilities.

The course emphasized agent roles, instructions, tool assignment, context sharing, manager-worker patterns, multi-agent systems, file-system-based memory, sandboxing, and human-in-the-loop approval.

The most useful design principle was specialization.

One broad agent with many responsibilities is harder to evaluate, harder to secure, and easier to confuse. Smaller agents with narrower responsibilities can be tested, observed, and improved more effectively.

This maps well to traditional software design. Clear boundaries reduce complexity. AI does not remove that principle. It makes it more important.

Evaluation Turns Guesswork Into Engineering

A recurring production theme was evaluation.

Manual testing is not enough. A prompt that works three times in a chat window may fail with a different user, a different model version, a longer context, a malformed input, or a changed tool response.

The course covered observability, logging, replaying interaction state, versioning system instructions, building test datasets, defining success metrics, and using model-based judging carefully.

This matters because AI behavior changes.

Providers release new models. APIs evolve. Output formats shift. Smaller models may appear to work until edge cases expose them. Fallback models may respond differently to the same prompt. Long-running agents may accumulate errors across steps.

Without evaluation, teams are left with vibes.

With evaluation, AI systems become improvable.

That is one of the most important professional habits I took from the course: treat prompts, tools, model choices, and context strategies as versioned artifacts that can be tested.

Production Changes the Problem

The final stage of the course focused heavily on production realities.

This included architecture, provider abstraction, model migration, rate limits, cost monitoring, latency, fallback models, moderation, privacy, tenant isolation, logging, prompt caching, and security.

These topics are easy to ignore during a prototype. They become central as soon as users, data, and business processes are involved.

A production AI system has to answer practical questions:

• What happens when the provider is down?
• What happens when the model changes behavior?
• What happens when one user generates most of the cost?
• What happens when a prompt injection tries to manipulate a tool?
• What happens when sensitive data appears in input or output?
• What happens when logs contain private information?
• What happens when an agent has access to files, APIs, or a browser?

The security principle was especially clear:

The model is never the security boundary.

Authorization belongs in backend code. Tool permissions should be explicit. Model output should be treated as untrusted input. Prompt injection is not solved by telling the model to ignore malicious instructions. Classic application security still applies, but AI introduces new paths for old risks.

That is the difference between building something impressive and building something responsible.

AI Inside Organizations

The course also looked beyond individual applications and into company workflows.

This was important because many AI projects fail by starting with the technology instead of the process. A chatbot is not always the right interface. Sometimes the better solution is a background workflow, a document pipeline, a review assistant, a search layer, or a tool embedded into existing operations.

The course covered process mapping, internal knowledge bases, company tools, privacy constraints, human review, asynchronous AI work, and the difference between visible chat interfaces and operational automation.

The practical lesson was that AI should be introduced where it changes the work, not where it looks impressive.

That requires understanding the business process, the users, the risk of errors, and the cost of human review.

What This Says About How I Build

The biggest change in my thinking is that I now see AI features as systems with moving parts, not isolated model calls.

When I look at an AI use case, I think about:

• the task boundary
• the data sources
• the context strategy
• the model’s role
• the tools and permissions
• the deterministic parts of the workflow
• the evaluation method
• the failure modes
• the cost profile
• the security model
• the user’s actual workflow

I also think more carefully about whether an agent is necessary. Autonomy is not automatically better. The right design may be a simple structured extraction, a deterministic workflow with one model step, a retrieval system, a specialized agent, or a multi-agent process.

The goal is not to use the most advanced pattern. The goal is to build the most reliable useful system for the problem. And sometimes less is more – forcing people into using AI won’t work. On the other hand improving existing workflows might get real traction in a team.

Skills I Can Bring Into a Team

AI_devs 4 gave me a practical foundation across the full lifecycle of AI application development.

I can reason about model interaction, structured outputs, tool design, context management, RAG, memory, agents, evaluation, and production constraints as connected parts of one architecture.

The skills I strengthened include:

• designing LLM API interactions with structured outputs and validation
• building model-facing tools with clear schemas and safe execution paths
• understanding MCP as a protocol for connecting models to external capabilities
• working with multimodal inputs such as images, PDFs, audio, and video
• designing context strategies instead of relying on oversized prompts
• building retrieval and knowledge-base workflows
• distinguishing workflows, agents, and multi-agent systems
• planning observability and evaluation for AI behavior
• thinking through model migration and provider fallback
• identifying prompt injection and unsafe tool-use risks
• considering cost, latency, privacy, and rate limits before production
• mapping business processes into practical AI-assisted workflows

This is the kind of knowledge I’d love to keep exploring: building AI systems that are useful, observable, secure, and grounded in real user needs.

Closing

AI_devs 4 started with model interaction, but the real value was the architectural perspective it built over time.

The course made one thing clear: production AI is not a prompt and not a chatbot. It is a system composed of models, tools, context, memory, data, workflows, people, and safeguards.

That is where the interesting engineering happens.

And most of all: completing the course gave me a great overview of the risks and temptations. Sometimes not doing something is the best answer, right? I’m happy that it will be way easier for me to decide the right path to take with the things ahead of me.