Compaction

Overwrite the same key a million times and the log holds a million records for one live value. Compaction rewrites only the live set.

The mechanics:

1. Track stale_bytes as you write over existing keys.
2. When it crosses a threshold, open a new log file.
3. Copy the current value of every key in the index into it.
4. Atomically swap in the new file, then delete the old one.

Rename is atomic on POSIX; a partially-written temp file is never observed under the real name. Write to log.tmp, fsync, then rename. If the process dies mid-compaction you’re left with a stale-but-valid log, which is exactly what you want.

Once this works, the interesting extensions are real: multiple log files with a generation number, bloom filters to skip lookups, and an LSM tree if you want writes to outrun the index.