DuckDB 2.0 vs Polars vs Spark on One Machine: 150M Rows Under a 4.5 GiB Cap

DuckDB 2.0.0 shipped on October 21, 2026 (announcement, PyPI). The next morning I ran the same single-machine benchmark I had already run on DuckDB 1.5.6: three engines, one 8 vCPU VM, Parquet in and Parquet out, a hard memory cap, and four queries that look like real data engineering work.

The thesis I started with, and that the numbers support:

• DuckDB beats Spark-on-one-box on memory-bound DE work. It finished every run, at every size, 5 to 12 times faster than local PySpark at 150M rows, and it still finished the heavy queries with a 1 GB memory limit.
• Polars hits a wall on window dedup. At 150M rows the ROW_NUMBER() style "latest row per customer" query never finished under the cap. Not at 4.5 GiB, not with an out-of-core budget, not at 9 GiB.
• Polars is not slow. On filters, joins and a high-cardinality group by it ties DuckDB or beats it, when it fits.

Everything below is measured on that one box. Where I interpret or give an opinion, I say so.

The contenders

Polars 2.0.0 was published on PyPI on October 6, 2026, PySpark 4.2.0 on July 14, 2026. DuckDB 2.0.0 wheels landed on PyPI on October 21, the same day as the release calendar entry.

Setup

Hardware and versions

• VM: 8 vCPU Intel Xeon, 15 GiB RAM, no swap, Linux 6.12. It is a shared box, so other processes were running. Every run waited until at least 5 GiB was free and CPU busy was under 30% over a 3 second window.
• Python 3.13.5, OpenJDK 21.0.12, pyarrow 26.0.0.
• DuckDB 2.0.0, Polars 2.0.0, PySpark 4.2.0. The "before" numbers are DuckDB 1.5.6 on the same box and data, run on October 19.

Data. A seeded synthetic star schema written with pyarrow: a sales fact table and a 50,000-row products dimension. Three sizes: 10M rows (213 MB), 50M (1.1 GB) and 150M (3.2 GB of snappy Parquet, 30 files of 5M rows). About 10 orders per customer, so 150M rows means roughly 15M customers.

Queries. The same SQL for DuckDB and Spark, the same logic as Polars expressions:

1. q1_filter_agg: filter on a timestamp and status, group by store and status.
2. q2_join_agg: fact-to-dimension join, group by category and brand.
3. q3_window_latest: latest order per customer with ROW_NUMBER(). The classic dedup and SCD pattern.
4. q4_highcard_agg: one row per customer (about 15M groups at 150M).

-- q3_window_latest
SELECT customer_id, order_id, ts, qty * price AS amount
FROM (
    SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id
                                 ORDER BY ts DESC, order_id DESC) AS rn
    FROM sales
) t
WHERE rn = 1

Method.

• Every run is a fresh process: interpreter start, import, session, read Parquet, compute, write Parquet. So no warm caches inside the engine, and Spark pays its JVM start every time.
• Engines run strictly one at a time. The engine order rotates per repetition so no engine always goes first.
• 5 repetitions at 10M, 3 at 50M and 150M. I report medians.
• A watchdog samples the RSS of the whole process tree every 50 ms and kills the run if it goes above 4.5 GiB. That is the cap. It is deliberately tighter than the box, because "fits in a small container" is the question I care about.
• Wall is the whole process. Query is read, compute and write only. The difference is setup.
• Correctness: a validation script compares row counts and rounded checksums of every output. Where an engine produced output, all three engines agreed, for every query and size.

Results at 150M rows

Median wall seconds, runs that finished under the 4.5 GiB cap in brackets.

Peak RSS (median of finished runs) tells the other half:

Three things stand out.

Spark on one box pays twice. About 5 seconds of every Spark run is setup (median 5.0 to 5.8 s, JVM plus session) before it reads a byte. Then the query itself is slower: query-only time was 7.4x DuckDB's on the filter, 7.9x on the join, 5.2x on the window dedup and 4.3x on the group by. In wall time that is 5x to 12x. And spark.driver.memory = 3g did not keep the process at 3 GiB: one window dedup run reached 7.51 GiB of process-tree RSS and was killed.

Polars is fast right up to the cap. It tied DuckDB on filter and join and was the fastest engine on the high-cardinality group by (4.21 s against 7.91 s). But only 1 of 3 runs finished: that one peaked at 4.49 GiB, and the other two crossed the cap at 4.50 and 4.55 GiB. Fast and close to the edge is a fine place for a notebook and a bad place for a scheduled job.

The window dedup is where Polars stops. At 150M rows it was killed in all 3 runs, within a few seconds of starting, at 4.52 to 4.66 GiB.

Polars and the window wall

I gave Polars two more chances on the 150M window dedup:

The out-of-core budget variable is one I found in the Polars 2.0.0 source, not a documented knob, so treat that row as "I tried", not as guidance. The same variable did help the group by: with it, Polars finished 3/3 at 4.67 s and 4.18 to 4.30 GiB.

At smaller sizes the dedup works, but the shape is already visible:

At 50M Polars already used 4.46 GiB for a query DuckDB did in 2.19 GiB, and took 4.4x longer.

This is the Polars expression I used. It is the direct translation of the SQL:

import polars as pl

amount = pl.col("qty").cast(pl.Float64) * pl.col("price")
rn = pl.int_range(pl.len()).over(
    "customer_id", order_by=["ts", "order_id"], descending=True
)
(
    pl.scan_parquet("data/sales_150m/*.parquet")
    .filter(rn == 0)
    .select("customer_id", "order_id", "ts", amount.alias("amount"))
    .sink_parquet("out/polars/150m_q3_window_latest.parquet")
)

Fair caveat: I did not test other Polars formulations of the same dedup (a sort followed by unique(keep="first"), or a group_by().agg(...) with sort_by().last()). One of them may stream better. The point stands for the common case, though: the first thing a team writes is the window, and that one did not fit.

Update (Oct 22): after this went out, Victor Cid (@Vleyked) tested exactly that in engine-bench PR #1. His sort + unique(keep="first") version drops the in-memory window fallback from the plan and finished 3/3 at 150M rows under the same 4.5 GiB cap, peaking at 2.50 to 2.74 GiB, but only with POLARS_OOC_MEMORY_BUDGET_MB=1024. With default settings or a 3072 MB budget it still went over the cap. The price is speed: a 90.58 s median against 17.56 s for DuckDB on the same 4 vCPU runner, so roughly 5x slower. His output matched DuckDB row for row (14,999,347 rows). So Polars can do this dedup under the cap if you rewrite it and tune the spill budget, and with a smarter query it gets close to DuckDB (see below). His runs were on different hardware, so they are not in the tables above.

Second update: in engine-bench PR #2 he filtered each customer down to their max timestamp first and only sorted those rows to break ties. That brought the median down to 14.23 s at 150M rows with default Polars settings, peaking at 4.04 GiB under the same cap, with output matching DuckDB row for row. On his runner DuckDB's reference was a single 18.50 s run, so I'm not calling that a repeated cross-engine win. The real lesson is that the first way you write a Polars dedup can decide whether it finishes at all.

Third update (Oct 23): Ritchie Vink, the creator of Polars, pointed out that only DuckDB was told about the memory limit, so Polars did not know it had to stay under 4 GiB, and suggested setting POLARS_OOC_MEMORY_BUDGET_MB for a fairer comparison. I reran every Polars query at 150M rows on the same box under the same 4.5 GiB cap with that budget, next to a DuckDB rerun in the same session: the high-cardinality group by finished 3/3 at 6.17 s and 3.46 GiB with a 3072 MB budget, against 9.56 s for DuckDB (the box was busier than on Oct 22 and DuckDB ran slower too, so compare within this session, not with the tables above; the Oct 22 run with the same budget gave 4.67 s). The window dedup with over(order_by) still went over the cap in every run at 3072, 1024 and 512 MB, Victor's sort plus unique version fit only at 1024 MB (31.28 s, 2.29 GiB), and the max-timestamp rewrite finished 3/3 on default settings (5.61 s, 4.02 GiB). Ritchie says Polars main now spills the group by correctly and that this ships in the next release, which was not on PyPI yet when I ran this, so I will rerun when it ships. Raw results and output checks are in engine-bench PR #3 and in results/polars_ooc_budget_2026-10-23 on the polars-ooc-budget-2026-10-23 branch.

Third update (Oct 23): Ritchie Vink, the creator of Polars, pointed out that only DuckDB was told about the memory limit, so Polars did not know it had to stay under 4 GiB, and suggested setting POLARS_OOC_MEMORY_BUDGET_MB for a fairer comparison. I reran every Polars query at 150M rows on the same box under the same 4.5 GiB cap with that budget, next to a DuckDB rerun in the same session: the high-cardinality group by finished 3/3 at 6.17 s and 3.46 GiB with a 3072 MB budget, against 9.56 s for DuckDB (the box was busier than on Oct 22 and DuckDB ran slower too, so compare within this session, not with the tables above; the Oct 22 run with the same budget gave 4.67 s). The window dedup with over(order_by) still went over the cap in every run at 3072, 1024 and 512 MB, Victor's sort plus unique version fit only at 1024 MB (31.28 s, 2.29 GiB), and the max-timestamp rewrite finished 3/3 on default settings (5.61 s, 4.02 GiB). Ritchie says Polars main now spills the group by correctly and that this ships in the next release, which was not on PyPI yet when I ran this, so I will rerun when it ships. Raw results and output checks are in engine-bench PR #3 and in results/polars_ooc_budget_2026-10-23 on the polars-ooc-budget-2026-10-23 branch.

DuckDB under pressure: memory_limit = 1 GB

The interesting DuckDB question is not "how fast with room to spare" but "what happens when the box is small". Same 150M rows, memory_limit cut to 1 GB:

With a third of the memory, DuckDB spilled to disk and still beat Spark's unconstrained run by about 3x on both queries, at a quarter to a third of Spark's peak RSS. The DuckDB docs list window functions with PARTITION BY ... ORDER BY among the blocking operators it can process larger than memory (How to Tune Workloads). This run is that sentence, measured.

The setup is three lines:

import duckdb

con = duckdb.connect()
con.execute("SET memory_limit = '1GB'")
con.execute("SET threads = 8")
con.execute("SET temp_directory = '/mnt/fast-local-disk/duckdb_tmp'")  # where spills go
con.execute("""
    COPY (
        SELECT customer_id, order_id, ts, qty * price AS amount
        FROM (
            SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id
                                         ORDER BY ts DESC, order_id DESC) AS rn
            FROM read_parquet('data/sales_150m/*.parquet')
        )
        WHERE rn = 1
    ) TO 'out/latest_per_customer.parquet' (FORMAT parquet)
""")

DuckDB 1.5.6 vs 2.0.0

Same box, same data, same scripts. 1.5.6 ran on October 19, 2.0.0 on October 22. Median wall seconds at 150M:

The useful finding is the last four rows together. On 1.5.6, SET preserve_insertion_order = false was worth 16% on the dedup and 31% on the group by under a 1 GB limit. On 2.0.0 the default already lands where the tuned setting did, and the flag changes almost nothing for these two queries. If you carry that setting in job configs as a performance trick, re-measure after upgrading. It still matters for what it was designed for: the docs recommend it when large imports or exports run out of memory, and it lets DuckDB reorder rows without an ORDER BY.

Not everything improved. The 50M window dedup was slower on 2.0.0 (1.94 s to 2.33 s), and peak RSS for the light queries went up (150M filter: 0.26 to 0.77 GiB, join: 0.23 to 0.42 GiB), while the heavy queries went down (window dedup: 3.47 to 3.01 GiB). I have not investigated why. The two runs were also three days apart on a shared box, so treat differences under about 10 to 15% as noise. The 1 GB group by drop is far outside that.

The release itself is mostly a feature release: Quack and CONNECT for client-server use, asynchronous I/O, a first-class VARIANT, triggers, a PEG parser by default (announcement). It is an LTS line supported until at least October 2027. One migration note matters for pipelines: database files written by 2.0 cannot be read by versions before 2.0. My benchmark only reads and writes Parquet, so it never touched that.

Where Spark still wins

None of this says "replace Spark". It says Spark on one box is the wrong comparison for small and medium jobs. Spark still wins when:

• The data does not fit one node's disk and memory, or the deadline does not fit one node's cores. This test stops at 3.2 GB of Parquet on purpose. DuckDB spills well, but one machine is still one machine.
• The platform is built around it. On Databricks and Fabric, Spark is where Unity Catalog or OneLake governance, Delta writes with concurrent writers, lineage, job orchestration and streaming already live. A DuckDB step outside that boundary is a step you have to secure, monitor and explain.
• The cluster is already running. The 5 second JVM start I measured per run disappears in a warm interactive cluster or a long job. The query-only gap (4x to 8x here) does not.
• The team is a Spark team. One engine everyone can debug at 02:00 is worth more than a faster engine one person understands.

My opinion as a Lead DE on Azure, Databricks and Fabric: the line is not the engine, it is the size of the job. Most jobs I see are tens of millions of rows dressed up as big data and started on a cluster. For those (file-to-file transforms before landing, validation and reconciliation checks, small dimension builds, CI tests for SQL logic) a single-node engine with a memory limit is the cheaper and more predictable choice. Anything that writes governed Delta tables stays on the platform.

Failure modes I hit, so you don't have to

1. Engine memory settings are not process memory. Spark's 3g driver heap still produced a 7.51 GiB process. DuckDB's whole process stayed close to its memory_limit: about 3.0 GiB peak at a 3 GB limit, 1.1 GiB at 1 GB. Size containers on measured RSS, not on the setting.
2. "Streaming" does not mean bounded. Polars streamed the filter, join and group by fine. The window dedup with an over(order_by=...) did not stay bounded, even with 9 GiB.
3. Close to the cap is a failure that hasn't happened yet. Polars' group by finished in 1 of 3 runs, peaking at 4.49 GiB against a 4.5 GiB cap. Same query, same data, same version. Schedule that and it fails some nights.
4. Shared hosts lie. One Spark run at 10M was killed because free memory on the whole box fell under my 1 GiB safety floor while Spark itself was at 1.6 GiB. That was the neighbours, not Spark. Gate each run on free memory and CPU, and record both.
5. Validate outputs, not just timings. Every engine produced the same row counts and checksums where it finished. Without that check, a fast wrong answer looks like a win.

Reproduce it

Everything is public at firfircelik/engine-bench: scripts, raw results, pinned versions and the query plans. Here are the parts that matter. The data generator is seeded (SEED = 20261019) and writes 5M-row chunks:

rng = np.random.default_rng([SEED, chunk_index])
t = pa.table({
    "order_id": np.arange(lo, lo + n, dtype=np.int64),
    "ts": pa.array(start + rng.integers(0, 270 * 24 * 3600, n), pa.timestamp("s")),
    "customer_id": rng.integers(0, rows // 10, n, dtype=np.int32),
    "product_id": rng.integers(0, 50_000, n, dtype=np.int32),
    "store_id": rng.integers(0, 500, n, dtype=np.int16),
    "qty": rng.integers(1, 10, n, dtype=np.int16),
    "price": np.round(rng.uniform(1.0, 500.0, n), 2),
    "status": rng.choice(STATUSES, n, p=[0.55, 0.20, 0.12, 0.08, 0.05]),
})
pq.write_table(t, f"sales_{rows // 1_000_000}m/part-{chunk_index:04d}.parquet",
               row_group_size=1_000_000, compression="snappy")

Spark session, the only non-default settings:

spark = (SparkSession.builder.master("local[8]")
         .config("spark.driver.memory", "3g")
         .config("spark.local.dir", TMP)
         .config("spark.ui.enabled", "false")
         .config("spark.sql.session.timeZone", "UTC")
         .getOrCreate())

And the watchdog, which is the part I would copy into any benchmark of your own. It measures the whole process tree, because JVMs and thread pools do not show up in the parent's RSS:

import psutil, subprocess, time

GIB = 1024 ** 3

def tree_rss(p):
    total = 0
    for q in [p] + p.children(recursive=True):
        try:
            total += q.memory_info().rss
        except psutil.NoSuchProcess:
            pass
    return total

proc = subprocess.Popen(cmd)
p, peak = psutil.Process(proc.pid), 0
while proc.poll() is None:
    rss = tree_rss(p)
    peak = max(peak, rss)
    if rss > 4.5 * GIB:
        for q in p.children(recursive=True) + [p]:
            q.kill()
        break
    time.sleep(0.05)

The takeaway

On one 8 vCPU machine with a 4.5 GiB cap, DuckDB 2.0.0 was the only engine that finished every run of every query at every size, and it did the heavy ones with 1 GB. Polars matched or beat it on the light queries and the group by, and could not finish the window dedup at 150M even with 9 GiB. Local Spark finished almost everything, 5x to 12x slower, and its memory setting did not bound its memory.

If your job fits on one machine, measure it on one machine before you start a cluster for it.

Sources

• Announcing DuckDB 2.0.0 LTS "Cinnamon Teal", DuckDB team, October 21, 2026
• DuckDB release calendar (2.0.0 LTS on 2026-10-21, 1.5.6 on 2026-09-28)
• duckdb 2.0.0 on PyPI, polars 2.0.0 on PyPI, pyspark 4.2.0 on PyPI
• DuckDB: How to Tune Workloads (larger-than-memory operators, preserve_insertion_order)
• DuckDB: Order Preservation
• All timings and memory numbers: my own runs on October 19 (DuckDB 1.5.6, and the 9 GiB Polars and Spark runs) and October 22, 2026 (DuckDB 2.0.0, Polars 2.0.0, PySpark 4.2.0).