What data is, and why we need a database
Should this be a file or a database?
The idea in 30 seconds
Files are still the right home for some data: configuration changed through code review, append-only logs, exports, and anything with exactly one writer. Shared state that several processes change, or that you need to query, belongs in a database. The deciding questions are who writes it, how often, and how you will look things up.
On this page
The situation
Your team runs two services: the web app and a background worker. Both need feature flags, the on/off switches that turn features on for some customers. Today the flags are hard-coded. An AI assistant opened a pull request to make them changeable at runtime, from an admin page in either service.
Review taskStore feature flags in a shared JSON file
Pull request: “Runtime feature flags. Both services read and write flags.json on the shared volume, so no new infrastructure is needed.”
# flags.py, used by both the web app and the worker
import json
PATH = "/shared/flags.json"
def get_flag(name):
with open(PATH) as f:
return json.load(f).get(name, False)
def set_flag(name, value):
with open(PATH) as f:
flags = json.load(f)
flags[name] = value
with open(PATH, "w") as f:
json.dump(flags, f)Would you approve it? Write the review you would leave.
Your call
Model review
Request changes. “No new infrastructure” is a real benefit, but this design will lose flag changes and can take both services down.
1. Two admins changing flags at once will lose one change. set_flag reads the whole file, changes one key and writes the whole file back. If both services do that at the same moment, both read the same old flags, and the second write silently undoes the first. A flag will “turn itself back on”.
2. Readers can see an empty or half-written file. open(PATH, "w") empties the file before the new contents are written. Any get_flag that runs in that instant fails with a JSON error. Every request checks flags, so a parsing error in get_flag becomes an error in your web app. In our two-writer experiment, one writer crashed this way in every run.
3. A crash in the middle of a write corrupts the file for good. If the process is killed while json.dump is writing, the file is left half-written and every later read fails until someone repairs it by hand. When we killed a writer at random moments, the file was unreadable afterwards 20 times out of 20.
What I would ask for instead, in order of preference:
- Use the database the services already share. A
feature_flagstable with one row per flag. Changing a flag is oneUPDATE, which the database applies safely even when two admins click at once, and a crash cannot leave half a change behind. - If it really must be a file: write it atomically (write a temporary file, then rename it over the original, so readers only ever see the complete old or complete new version) and take an exclusive lock around every read-change-write. Both services must use the same lock, because file locks only protect processes that ask for them.
Compare it with yours. Did your review catch the main problems?
When a file is the right answer
Files are not wrong; they are a tool with a narrow sweet spot. A file is a good choice when the data is:
- Configuration changed through code review. It is read at startup, and changes arrive as commits, one at a time.
- Append-only. Logs, audit trails and event exports that are written once and never edited.
- Immutable datasets. A nightly export or a machine-learning training set that is produced once and read many times.
- Owned by exactly one writer. One process writes; any number of others only read, and the writer replaces the file atomically.
- Static content that a web server or CDN hands out as-is.
When it belongs in a database
Reach for a database as soon as any of these is true:
- More than one process changes it. Two writers are enough to need coordination.
- A crash must not lose or corrupt it. Money, orders, accounts, anything a customer would notice.
- You need to find things by more than one key. “All orders over ₹5,000 from last week” means a query, not a file read.
- Several changes must happen together. Debit one account and credit another: all or nothing.
- Rules must hold. Unique emails, no order for a user who does not exist.
Check yourself
Name three properties of data that make a plain file a good home for it.
Show answer
Any three of: changed only through code review, append-only, immutable once produced, exactly one writer, or static content served as-is.
The flags pull request uses a lock around writes but not reads. What can still go wrong?
Show answer
Readers can still open the file while a writer has emptied it and not finished writing, and fail to parse it. Atomic replacement (write a temporary file, then rename it into place) is what protects readers.
Why does a single SQL
UPDATEavoid the lost-update problem the JSON file has?Show answer
Each
UPDATEchanges only its own row, not a copy of all the flags read earlier. If two transactions update the same row at once, PostgreSQL makes the second wait until the first commits, then applies the second change on top of the result.
Finished reading?
Saved in this browser only.
Sources
Where each claim on this page comes from.
- 1.open(2) manual page, macOSPrimary· O_TRUNC·
man 2 openOpening a file for writing with truncation empties it before the new contents are written.
- 2.rename(2) manual page, macOSPrimary· DESCRIPTION·
man 2 renamerename() guarantees that an instance of the new name always exists, even if the system crashes in the middle of the operation.
- 3.flock(2) manual page, macOSPrimary· DESCRIPTION·
man 2 flockflock() locks are advisory; they only protect processes that also take the lock.
- 4.PostgreSQL documentation: 3.4 TransactionsPrimary· §3.4 Transactions
A transaction bundles several steps into one all-or-nothing operation.
- 5.PostgreSQL documentation: 13.2 Transaction IsolationPrimary· §13.2.1 Read Committed Isolation Level
When a row is already being updated by another transaction, the second updater waits for the first to commit or roll back, then applies its change to the updated row.
Go further
What to read next, and why it is worth the time.
- 1.flock(2) manual page, macOSPrimary·
man 2 flockShort, and the clearest statement of what a file lock does and does not promise.
Applies to PostgreSQL 18. Draft: not yet fact-checked end to end.