Allocating Intelligence. Rethinking Where AI Belongs in Software

Over the last few years, AI tools have progressively become part of our everyday workflows. We use them to write and review code, analyze information, transform data, generate documentation, investigate problems, and automate tasks that previously required either manual work or a dedicated implementation.

One reason this adoption has happened so quickly is that AI dramatically reduces the effort required to turn an intention into an executable task. Instead of implementing every step upfront, we can describe what we want to achieve in natural language, provide the relevant context and constraints, and let the model figure out how to get there. We refine the instructions when necessary, add what is missing, and eventually arrive at a prompt that reliably solves our problem.

For many tasks, this is an incredibly effective approach. If I need to do something once, spending a few minutes describing it to an AI tool can be much more convenient than designing and implementing dedicated software. Even when the task is more complex, the ability to explore the solution while executing it allows us to postpone implementation decisions until we understand the problem better.

Things become more interesting, however, when the same task needs to be performed again.

The most natural reaction is to reuse the prompt. It worked yesterday, so we run it again today. If we need the same result next week, we run it again. Over time, the prompt becomes part of our workflow, and we stop questioning whether invoking an AI model is still the best way to execute that task.

This is where I think we should start making a distinction between using AI to discover a process and using AI to execute a process.

When exploration becomes a process

The first execution of a prompt is often exploratory. We know what we want to achieve, but we might not know exactly how to get there. The AI helps us fill that gap by interpreting our intent, suggesting steps, and adapting to information that we did not fully formalize upfront.

The second execution is already slightly different. We usually improve the prompt based on what we learned from the first attempt, clarify some constraints, and perhaps define the expected output more precisely. If we execute it several more times, we may eventually notice that the same sequence of operations keeps happening.

At that point, something important has changed. We are no longer discovering how to solve the problem; we have effectively described a process.

This is why I like the idea of considering repeatability as a signal. There is no magic number of executions after which a prompt should become code, but a simple mental model could be useful:

1. the first time we solve the problem
2. the second time we validate the approach
3. from the third time onward we should at least evaluate/question whether we are still solving an AI problem or whether we have discovered a software process.

If the inputs are known, the expected outputs are known, and most of the steps in between are predictable, repeatedly asking a model to interpret those instructions may no longer be necessary.

Moving AI from runtime to build time

This leads to another distinction that I find useful: AI at runtime versus AI at build time.

Imagine a prompt that asks an AI agent to collect a set of files, inspect their structure, extract some information, transform it into JSON, validate the result, and store it somewhere. During the first iterations, letting an agent perform the entire workflow can be extremely convenient because we are still discovering what the correct workflow actually is.

Once the process stabilizes, however, most of those operations might not require intelligence at all. Reading files, validating a schema, transforming data, calling an API, or writing an output file are deterministic operations that traditional software has been doing reliably for decades.

Instead of continuing to ask the AI to execute the process, we can ask it to implement the process.

The prompt that we have refined over several iterations already contains a surprisingly good specification of what the software should do. We can use that specification to generate a script, CLI command, or small application, validate the implementation, and then let the machine execute it directly from that point onward.

The architecture effectively changes from this:

Input → Prompt → AI → Interpretation → Execution → Output
Input → Prompt → AI → Interpretation → Execution → Output
Input → Prompt → AI → Interpretation → Execution → Output

to this:

Requirements → Prompt → AI → Implementation → Validation
                              ↓
                           Script
                              ↓
                    Input → Output
                    Input → Output
                    Input → Output

The AI has not disappeared from the development process. In fact, it may have played a fundamental role in understanding the problem and creating the solution. What has changed is when we pay the cost of intelligence.

Instead of paying that cost every time the process runs, we use intelligence while designing and implementing the process, and then rely on deterministic execution whenever possible.

The obvious benefits are lower token consumption and potentially lower infrastructure costs, but I don't think cost is the most interesting part. A deterministic implementation can also be faster, easier to test, easier to observe and, importantly, more predictable.

Deterministic core, intelligent edges

Of course, real workflows are rarely completely deterministic.

A process may contain ten steps where nine are perfectly predictable and one requires understanding meaning, dealing with ambiguity or making a judgment based on information that cannot easily be expressed as a traditional rule.

This is where the discussion becomes much more interesting, because the choice is no longer between AI or no AI.

Instead, we can start thinking about where intelligence belongs inside the process.

Consider a workflow that collects documents, validates their structure, extracts metadata, applies a set of known business rules and eventually needs to decide how a particular document should be classified. Everything before that decision can probably be implemented using traditional code. The classification step, however, might depend on semantics rather than an explicit rule, which makes it a good candidate for AI. Once that decision has been made, traditional code can take control again and continue processing the result.

The architecture might therefore look something like this:

Input
  │
  ├── Collect data ───────────── Code
  ├── Validate ───────────────── Code
  ├── Transform ──────────────── Code
  ├── Apply known rules ──────── Code
  │
  ├── Semantic decision ──────── AI
  │
  ├── Execute action ─────────── Code
  ├── Validate result ────────── Code
  └── Store result ───────────── Code

Rather than having an AI agent orchestrate the entire workflow, we have created a deterministic core with intelligent edges. The software controls the process and AI is invoked only when the nature of the problem actually requires intelligence.

I find this inversion particularly important. We often start from the idea of building an AI workflow and then place traditional code inside it whenever necessary. In many situations, it might make more sense to do the opposite: build a traditional software workflow and introduce intelligence only at the specific points where deterministic logic stops being sufficient.

Intelligence as an architectural resource

This way of thinking becomes even more relevant as we get access to models designed for increasingly specific purposes.

Jev is an interesting example of this direction. Rather than approaching AI primarily as a system that generates content or orchestrates an entire task, models of this kind make it possible to think about AI as something closer to a decision primitive. The application can provide a state or context and ask for a constrained decision, score, or probability, then continue executing deterministically based on that result.

What interests me about this direction is not one particular model, but the architectural implication behind it.

Once we have identified the exact points where intelligence is required, we can also start asking how much intelligence each point actually needs.

A simple classification problem might be handled by a small specialized model. A semantic decision could be delegated to a model optimized for that kind of judgment. A genuinely complex problem involving planning, multiple constraints and deeper reasoning might justify using a much more capable reasoning model.

This means that choosing a model becomes a consequence of designing the process rather than the starting point.

Instead of asking which model should execute our workflow, we can first understand the workflow, identify where deterministic execution is sufficient, locate the points where judgment is necessary, and only then decide which AI capability is appropriate for each of those points.

I like to think about this as allocating intelligence.

When designing software, we already make similar decisions about other resources. We decide what should be cached, what belongs in persistent storage, what should happen synchronously, what should be moved to a queue, and where additional computing resources are justified. There is no reason why AI reasoning should be treated differently.

Reasoning has a cost. It introduces latency, consumes computational resources, creates dependencies on models and providers, and often introduces a degree of variability that deterministic software does not have. None of those characteristics make AI undesirable; they simply mean that intelligence should be used intentionally, in the same way that we intentionally allocate every other architectural resource.

The goal therefore shouldn't be to remove AI from a workflow. It should be to concentrate AI where it creates the most value.

What about agents and tools?

At this point, there is a reasonable objection to make. What I am describing might sound very similar to the way we already build AI agents: the model reasons about the problem, decides what needs to happen, and delegates deterministic operations to tools.

And in many cases, that is exactly right.

A tool is already a way of recognizing that part of a process does not need to be interpreted by the model. Once the agent decides that a particular operation is required, the implementation of that operation can remain deterministic. Whether that implementation is a function, an API, a CLI command, a script or a more complex workflow does not fundamentally change the principle.

What changes is where we decide to draw the boundary.

Today, we often design tools around relatively small and well-defined responsibilities, leaving the agent responsible for orchestrating them. But there is no reason why the deterministic side of that boundary must always be small. If several operations always happen together, follow predictable rules and do not require judgment between each step, they might belong to a larger deterministic workflow rather than requiring the model to orchestrate every individual operation.

Consider these two approaches:

Agent
  ↓
reason
  ↓
Tool A
  ↓
reason
  ↓
Tool B
  ↓
reason
  ↓
Tool C
  ↓
reason
  ↓
Tool D

and:

Agent
  ↓
reason / decide
  ↓
Deterministic workflow
  ├── Operation A
  ├── Operation B
  ├── Operation C
  └── Operation D
  ↓
Result
  ↓
Agent

Neither architecture is inherently better. If the transitions between those operations require interpretation, adaptation or judgment, keeping the agent involved makes perfect sense. If they always follow the same predictable sequence, however, asking the model to repeatedly decide what happens next may simply move deterministic orchestration into an unnecessarily probabilistic layer.

This is why I don't see agents and tools as an alternative to the idea of allocating intelligence. They are another implementation of the same principle.

The important question remains the same: where does the process actually require intelligence, and where have we already learned enough about the process to make execution deterministic?

Once that boundary is identified, the implementation can take many forms. The deterministic part might become a tool exposed to an agent, a standalone script, a traditional service, a workflow or simply a function inside an application. The architectural form is secondary; what matters is that we have consciously separated judgment from execution.

This is also why the boundary is not necessarily permanent. During the exploration of a new process, we may deliberately leave more responsibility to the agent because we have not yet understood which parts are stable. As the process matures and patterns become clearer, some of that responsibility can gradually move from reasoning into deterministic software.

In that sense, allocating intelligence is not only about choosing where to add AI. It is also about recognizing where a deterministic approach can provide more value, whether through greater efficiency and predictability, faster execution, lower costs, or simply a system that is easier to understand, test, and maintain.

The convenience trap

There is an interesting paradox in all of this. AI has made creating automation so easy that it can sometimes discourage us from consolidating that automation into software.

Once we have a prompt that works, the development cost of running it again is practically zero. There is no strong incentive to stop and implement anything else, especially when the model can simply repeat the task for us.

Over time, however, that prompt can quietly become infrastructure.

Every execution requires the model to read the instructions again, understand the context again, reason about the steps again, and produce the result again. In some situations, we are effectively using an LLM as an interpreter for a program that we have written in natural language.

During exploration, that flexibility is enormously valuable. Once the process becomes stable, the same flexibility may simply become unnecessary overhead.

And this is perhaps the most interesting part: AI itself has dramatically reduced the cost of making the transition from prompt to software.

If we have spent time refining a prompt until it reliably describes the task, we already have much of the specification required to implement it. Instead of executing that prompt indefinitely, we can ask AI to transform the accumulated knowledge into code, generate tests around it and help us validate that the resulting implementation behaves as expected.

The prompt doesn't have to be the final automation. It can be the prototype from which the automation emerges.

From AI-first to process-first

This ultimately leads me to a slightly different way of approaching AI automation.

Rather than beginning with "How can I automate this with AI?", I think the more useful starting point is simply "What is the process?"

Once the process is understood, we can identify which parts are predictable enough to become deterministic software and which parts genuinely depend on interpretation, judgment or reasoning. Only after making that distinction do we need to decide what kind of AI capability belongs in those remaining gaps.

Sometimes the result will still be a fully agentic workflow, because the nature of the task genuinely requires continuous reasoning and adaptation. In other cases, we may end up with a traditional application containing one or two carefully placed model calls. And sometimes, after understanding the process properly, we may discover that AI is extremely useful for building the solution but completely unnecessary for running it.

All of these are valid outcomes.

As AI makes implementation increasingly accessible, perhaps one of the skills we need to develop is not simply learning where to add AI, but learning where to stop using it.

The interesting question is no longer whether a process can be executed by AI. Increasingly, almost everything can.

The better question is whether it should be.

And when the answer is only partially, the challenge becomes much more architectural: build the process deterministically, identify the moments where judgment creates real value, and allocate intelligence exactly there.

A final note

There is a certain irony in writing an article about allocating intelligence without mentioning how AI was used to create the article itself.

I did use AI, but selectively. I used it to create the images and to review my English, helping me improve grammar and make some sentences clearer. The ideas, the reasoning behind them, the way they evolved and the perspective I have tried to articulate throughout the article come from my own experience: from using these tools myself, experimenting with different approaches, and observing how people and teams have been adopting AI over the last few years.

And this is important, because what I have written here is not meant to be a prediction of how software will inevitably be built.

It is my interpretation of what I am seeing today, which means that you may have had different experiences and may reach completely different conclusions. If you do, I would genuinely like to hear them in the comments, because comparing those perspectives is probably one of the most useful things we can do at this stage.

The landscape is changing too quickly for any of us to confidently describe what software development will look like in 2027, 2028, or beyond. What we can do is observe what is happening now, critically evaluate what works and what does not, learn from our own experience and from that of others, and keep questioning the assumptions we make along the way.

Perhaps that is the mindset that matters most right now: not trying to predict the future perfectly, but developing enough critical thinking to recognize when the future has already started changing the way we work.