Move the data to the compute, or the compute to the data?

📺 Prefer to watch? 90-second YouTube Short · 💬 Telegram

Originally published on software-engineer-blog.com.

Most cost work arrives too late. A bill lands on someone's desk three months after launch, and the hunt begins: which server can we shrink? But the real number was locked in during a design review, in one sentence about where a computation happens.

• Mental model: Your bill is not a list of servers. It is a list of decisions about where bytes move.

The concrete job

You have an orders table. Twelve million rows. One question: what was total revenue this month?

That single query lives at the exact fork where system design breaks into two paths. You must choose. And the choice you make determines whether you move 2.1 GB across a network boundary or 8 bytes.

Side A: pull the data back

Fetch the rows. Add them up in your code.

rows = db.query("SELECT amount FROM orders")
total = sum(row.amount for row in rows)

This is the version most teams write first, and it deserves real defence.

It is simple. The logic lives in a language you control and test every day. You can unit-test the aggregation function in isolation. You can reason about it. No query planner surprises. No hidden index choice breaking your assumptions next Thursday.

It is portable. Swap the database, swap the schema, swap the cloud. The Python function stays the same. You own the contract between your code and the data.

It is easy to test. Mock the result. Inject a list of rows. Verify the sum. Done.

But here is the cost: on every run, this code moves approximately 2.1 GB across a boundary. The boundary is where the meter sits. Your cloud provider counts bytes crossing that line. Your application needs heap space proportional to the data size—for 12 million rows, that is memory you pay for. Wall-clock time is dominated by transfer, not arithmetic. The addition itself takes microseconds. You are paying for 2.1 GB of I/O to do 2.1 GB ÷ 8 bytes = microseconds of work.

Side B: push the compute down

SELECT sum(amount) FROM orders

The computation runs where the data already lives. No row transfer. No memory bloat in your service. One number comes back.

2.1 GB on the wire becomes 8 bytes.

Identical answer. 288 runs a day. Every run pays for 8 bytes instead of 2.1 GB. The arithmetic happens in the database engine, which is already licensed for that work, already running, already paid for.

The trade-off table

This is not a SQL trick

This decision shows up everywhere, wearing different clothes.

• A view versus a loop in Python
• An aggregate in the database versus a scan in the application
• A join in the database versus a join in your service layer
• Compute co-located with storage versus compute pulling data remotely

Every time, the trade-off is identical: simplicity and testability and portability against transfer cost and coupling.

The honest bill for side B

Pushing computation down is not free. It costs you three things:

One: coupling. You are now coupled to that engine. SQL Server has different functions than PostgreSQL. Snowflake has different semantics than DuckDB. Your query must live somewhere, version-controlled and tested, but not in your application code. If you change databases, the query changes.

Two: logic out of code you can test. When the sum is in your Python function, you own the test. When it is in a SQL query, a planner owns the execution. You cannot easily mock a query planner's choice. You cannot unit-test a query in the same way you unit-test a function.

Three: load on the box hardest to scale sideways. Databases scale vertically far more easily than horizontally. A service scales to 100 instances. A database scales to 3 replicas, maybe. If you push computation down, you are adding load to the component that is hardest to scale. If you pull computation back, you scale the service layer, which is easy—add more instances.

Taken too far, this is how a team ends up with business rules buried in a 400-line query nobody dares change, because nobody can test it in isolation, and the query planner's choices have become a black box.

Side A is not wrong

Side A is a bet. It is a bet that the data is small.

Usually, that is a good bet. When the orders table has 50,000 rows instead of 12 million, pulling the data back makes sense. The network transfer is 87 MB. The memory footprint fits comfortably. You own the logic. You control the test.

The failure is not choosing side A.

The failure is never re-checking the bet after the data grew.

You write side A when you launch. You ship it. Months pass. The dataset grows. Orders accumulate. Suddenly, 50,000 rows becomes 12 million. That query now moves 2.1 GB. Your application crashes because it ran out of memory. Your bill explodes.

And nobody remembers why side A was chosen. The reasoning lived in someone's head, and that person is on a different team now.

One step further: caches and CDNs

A cache and a CDN are this same idea pushed one level further.

If moving data to compute is one step, then moving compute to data is the step after. But there is a step before that: stop moving the bytes at all.

A cache stores the result of a computation locally, near the consumer. When the next request comes in, the bytes are already there. No network. No wait. The cheapest byte is the one you never sent.

This is why CDNs work. This is why Redis clusters live inside your application network. This is why database read replicas live on the same LAN as your services.

The principle holds at every scale: bytes crossing a boundary cost money. Bytes that never cross a boundary cost nothing.

For LLM inference: the same decision under latency

If you are serving LLM inference, data movement is also latency.

Side A—pulling data back to your inference service—means serializing embeddings or tokens, sending them over the network, loading them into GPU memory, computing a result. Time-to-first-token (TTFT) is high. Your inference service is I/O-bound, not compute-bound.

Side B—pushing the model to the data—means running inference where the embeddings already live (in a vector database, in a cache). TTFT is lower. Time-per-output-token (TPOT) is more predictable.

The trade-off is identical: which component carries the load, and is it the one hardest to scale? The principle survives the domain shift.

One decision, different clothes

This is why your bill is not a list of servers.

It is a list of decisions about where bytes move. Every line item on your invoice traces back to a moment in a design review when someone wrote:

• "Fetch the rows, sum them in the app"
• "Run the aggregate in the database"
• "Cache the result locally"
• "Replicate the data to the edge"

Each sentence moved the meter. The server choices that follow are just plumbing. A bigger instance or a smaller one is not the decision. The decision was made when you chose where the computation happens.

Verdict

Reach for side A (pull the data back) when you are confident the dataset will stay small and you need the logic in application code you can test. Reach for side B (push the compute down) when the dataset is large, the query is simple, and the database is your scaling bottleneck anyway. Revisit the bet every time the data grows.

Watch the 90-second reel for the full visual breakdown.