In Cassandra, a delete is a write (and reads pay for it)

In Cassandra, a delete is actually a write.

That sounds like trivia until it slows down your reads in production. When you run DELETE, Cassandra does not remove anything. It writes a marker called a tombstone, which says "this data was deleted at this time." The data is only removed later, during compaction, and only after a waiting period. Until then, the data is gone as far as your application is concerned, but it is still physically there, and every read that touches it has to go through it.

Why Cassandra needs the tombstone

Cassandra keeps copies of your data on multiple nodes (replicas). Imagine a delete simply removed the data, and one replica was down when the delete happened. When that replica comes back, it still has the row. To the rest of the cluster, it looks like the other nodes are missing data, so repair copies the deleted row back to every node.

Deleted data coming back like this is called zombie data, and tombstones exist to prevent it. The tombstone is newer than the old value, so wherever both are seen, the tombstone wins.

This is also why tombstones have to stay for gc_grace_seconds before compaction is allowed to remove them. The default is 864000 seconds, which is 10 days. Those 10 days are the time you have to run repair on a node that was down. If you reduce gc_grace_seconds without also running repair more often, you are setting yourself up for zombie data. I have written about repair at scale in a later post in this series.

How reads pay for it

When you read data, Cassandra has to merge what is in memory (the memtable) with what is on disk (SSTables). Every tombstone in the range you are reading has to be read and compared with the live data before Cassandra can give you an answer. If a partition has collected a million tombstones, Cassandra has to go through a million deleted entries just to return whatever is still alive.

Cassandra warns you when this happens, first with a warning and then with an error.

WARN  ReadCommand - Read 812 live rows and 104,832 tombstone cells for query ...
ERROR ... Scanned over 100001 tombstones ... query aborted

The warning shows up when a read goes past tombstone_warn_threshold, which is 1,000 by default. If it goes past tombstone_failure_threshold, which is 100,000 by default, Cassandra stops the read and throws a TombstoneOverwhelmingException. Cassandra fails your query on purpose, so that it does not run the node out of memory.

It can be tempting to just raise these limits, but that only hides the problem. The tombstones are still there.

Where all these tombstones come from

I have seen every one of these in production.

• Using Cassandra like a queue. Insert, process, delete, and repeat, all in the same partition. The partition slowly fills with tombstones, and every time the consumer reads the next item, it has to go past all of them. This is exactly why using Cassandra as a queue is a well known anti pattern.
• Overwriting a whole collection. Running UPDATE t SET mymap = {...} replaces the entire collection, and Cassandra writes a tombstone for the old one first. If you only want to add items, use mymap = mymap + {...} instead.
• Inserting nulls. If your application binds a null value in a prepared statement, Cassandra writes a tombstone for that column. ORMs and code that "just binds every column" can create thousands of these without anyone noticing. Leave the value unset instead of setting it to null.
• TTL on everything. When data with a TTL expires, it also turns into a tombstone. This is fine, as long as your compaction strategy is designed for it.

How to find them

To check how many tombstones an SSTable has:

sstablemetadata /var/lib/cassandra/data/ks/table-*/*-Data.db | grep -i droppable
# Estimated droppable tombstones: 0.83

0.83 means that about 83% of that SSTable is tombstones that can be removed. To see tombstones for a single query, run TRACING ON in cqlsh before the query. For a table that is live, nodetool tablestats shows the average tombstones per slice.

How to fix them

The real fixes are in how you model the data.

1. For time series data with TTL, use TimeWindowCompactionStrategy (TWCS). Whole SSTables expire together and get dropped as files, so Cassandra hardly has to deal with individual tombstones. I have covered compaction strategies in the next post.
2. Stop deleting rows one by one in busy partitions. Put time into the partition key, so that old partitions expire as a whole instead.
3. Check your code for null binding and collection overwrites. These are the tombstones nobody meant to create.

Try it yourself

You can see this on your laptop with Docker. Start Cassandra:

docker run --name cass -d cassandra:4.1

Wait about a minute for it to start, then create a table that we will use like a queue:

docker exec -it cass cqlsh -e "
CREATE KEYSPACE lab WITH replication = {'class':'SimpleStrategy','replication_factor':1};
CREATE TABLE lab.queue (pk int, ck int, PRIMARY KEY (pk, ck));"

Add 10,000 jobs to the same partition, then delete all but the last 10, the way a queue consumer would:

docker exec cass bash -c 'for i in $(seq 1 10000); do echo "INSERT INTO lab.queue (pk, ck) VALUES (1, $i);"; done > /tmp/ins.cql && cqlsh -f /tmp/ins.cql'
docker exec cass bash -c 'for i in $(seq 1 9990); do echo "DELETE FROM lab.queue WHERE pk = 1 AND ck = $i;"; done > /tmp/del.cql && cqlsh -f /tmp/del.cql'

Now read the partition with tracing on:

docker exec -it cass cqlsh

TRACING ON;
SELECT * FROM lab.queue WHERE pk = 1;

In the trace, look for the line that says how many live rows and how many tombstones were read. To return 10 rows, Cassandra had to go through all 9,990 deleted ones.

When you are done:

docker rm -f cass

In Cassandra, a delete is cheap to run but expensive to live with, because every delete stays in the way of your reads until gc_grace_seconds has passed and compaction removes it.