Lab: build a JSON file store and break it

Before you start

You need Python 3 on macOS or Linux; the lock uses fcntl, which Windows does not have, so use WSL there. Make an empty folder and create the four files below in it. Nothing else is needed: no packages, no database.

Every step shows our numbers, from Python 3.9 on an Apple Silicon laptop. Your times will differ; the pattern should not.

The store

store.py keeps all users in one JSON file. Every operation loads the whole file, and every change writes the whole file back. It also contains the two fixes you will try later: a file lock and an atomic save.

# A user store kept in one JSON file: the simplest thing that could work.
import json, os, sys, fcntl

PATH = "users.json"

def load():
    if not os.path.exists(PATH):
        return []
    with open(PATH) as f:
        return json.load(f)

def save(users):
    with open(PATH, "w") as f:          # truncates the file, then writes
        json.dump(users, f)

def save_atomic(users):
    tmp = PATH + ".tmp"
    with open(tmp, "w") as f:
        json.dump(users, f)
        f.flush()
        os.fsync(f.fileno())            # make sure the bytes reached the disk
    os.replace(tmp, PATH)               # swap in the new file in one step

def add_user(name, atomic=False, lock=False):
    lk = open(PATH + ".lock", "w") if lock else None
    if lk: fcntl.flock(lk, fcntl.LOCK_EX)   # one writer at a time
    try:
        users = load()
        users.append({"id": len(users) + 1, "name": name})
        (save_atomic if atomic else save)(users)
    finally:
        if lk: fcntl.flock(lk, fcntl.LOCK_UN); lk.close()

def find_user(user_id):
    return next((u for u in load() if u["id"] == user_id), None)

def update_user(user_id, name):
    users = load()
    for u in users:
        if u["id"] == user_id:
            u["name"] = name
    save(users)

def delete_user(user_id):
    save([u for u in load() if u["id"] != user_id])

if __name__ == "__main__":
    cmd = sys.argv[1]
    if cmd == "add-many":               # add-many <count> <tag> [lock]
        n, tag = int(sys.argv[2]), sys.argv[3]
        lock = len(sys.argv) > 4 and sys.argv[4] == "lock"
        for i in range(n):
            add_user(f"{tag}-{i}", lock=lock)
    elif cmd == "count":
        print(len(load()))

1. Measure how it slows down

bench.py fills the file with 100, then 10,000, then 100,000 users and times one lookup and one update in the middle of each.

# Time find, update and delete as the file grows.
import time, statistics, store

def ms(fn, reps=5):
    times = []
    for _ in range(reps):
        start = time.perf_counter()
        fn()
        times.append((time.perf_counter() - start) * 1000)
    return statistics.median(times)

for n in [100, 10_000, 100_000]:
    users = [{"id": i, "name": f"user-{i}"} for i in range(1, n + 1)]
    store.save(users)
    middle = n // 2
    find = ms(lambda: store.find_user(middle))
    update = ms(lambda: store.update_user(middle, "renamed"))
    print(f"{n:>7,} users   find {find:8.2f} ms   update {update:8.2f} ms")

Run python3 bench.py.

Users	Find (ours)	Update (ours)	Find (yours)	Update (yours)
100	0.06 ms	0.34 ms
10,000	2.69 ms	22.36 ms
100,000	30.92 ms	225.27 ms

From 10,000 to 100,000 users, both operations got about 10× slower: the cost grows in step with the file. An update costs several times a lookup, because it reads everything and writes everything back.

2. Two writers at once

two_writers.sh starts two copies of the store that each add 1,000 users at the same time, then counts the users.

# Two writers, 1,000 additions each, at the same time. Pass "lock" to use the file lock.
echo "[]" > users.json
python3 store.py add-many 1000 a $1 &
python3 store.py add-many 1000 b $1 &
wait
python3 store.py count

Run sh two_writers.sh a few times. You should get 2,000 users. We never did:

Run	Ours	Yours
Without a lock	One writer crashed with a JSON parsing error in all 10 runs; 1,000–1,002 users left

Read the crash message. The writer that died was not doing anything wrong: it opened the file while the other writer had just emptied it to write a new version, and found nothing valid to parse. Its work was lost: the few additions it had saved were overwritten by the other writer, and the rest never happened.

3. Kill a writer mid-save

crash_test.py starts a writer that saves a 200,000-user file over and over, kills it at a random moment with SIGKILL, the way a crash would, and checks whether the file still loads. It does that 20 times.

# Kill a writer at a random moment, 20 times, and check whether the file still loads.
import json, random, signal, subprocess, sys, time

mode = sys.argv[1]  # "plain" or "atomic"
writer = f"""
import store
users = [{{"id": i, "name": f"user-{{i}}"}} for i in range(1, 200_001)]
save = store.save_atomic if "{mode}" == "atomic" else store.save
while True:
    save(users)
"""
readable = 0
for _ in range(20):
    p = subprocess.Popen([sys.executable, "-c", writer])
    time.sleep(random.uniform(1.0, 2.5))
    p.send_signal(signal.SIGKILL)
    p.wait()
    try:
        json.load(open("users.json"))
        readable += 1
    except (ValueError, FileNotFoundError):
        pass
print(f"{mode}: file readable after {readable} of 20 kills")

Run python3 crash_test.py plain.

Save	Ours	Yours
Plain save	Readable after 0 of 20 kills

A save of this size takes long enough that a kill almost always lands in the middle of one, leaving a truncated file behind.

4. Fix what you can

Lock the writers. Run sh two_writers.sh lock. Each writer now takes an exclusive lock on users.json.lock before its read-change-write and releases it after, so only one writer is ever inside that section.

Save atomically. Run python3 crash_test.py atomic. save_atomic writes the new version to users.json.tmp, flushes it to the drive with fsync, and then renames it over users.json. The rename is the trick: the operating system guarantees that the name users.json always points at a complete file, the old one or the new one, even if the system crashes during the rename.

Fix	Ours	Yours
Two writers with lock	2,000 users, every run
Atomic save, 20 kills	Readable after 20 of 20 kills

Both fixes worked. Notice what they cost: the lock makes writers wait for each other, and the atomic save still rewrites the whole file every time.

What you measured

• Lookups and updates slowed down in step with the size of the file.
• Two writers without coordination crashed and lost about half the work.
• A crash during a save destroyed the file every time.
• A lock and an atomic save fixed the last two, but only because every writer cooperated.

Keep these results. The next page asks what it would take to make this store as safe as a database, and the answer is most of a database.

Clean up

Delete the folder when you are done. Nothing was installed.

Check yourself

1. Why did a writer crash in the two-writer test, rather than just losing an update?

   Show answer

   Opening the file for writing truncated it to zero bytes before the new version was written. The other writer read the file in that window, found empty or partial JSON, and failed to parse it.

   I could explain this without looking

2. What does os.replace guarantee that open(PATH, "w") does not?

   Show answer

   The name users.json always refers to a complete file: the old version until the rename, the new one after. Writing in place passes through a truncated, half-written state that readers and crashes can see.

   I could explain this without looking

3. The lock fixed the lost writes. What would break it?

   Show answer

   Any process that changes the file without taking the same lock. File locks are advisory: they only coordinate processes that ask for them.

   I could explain this without looking