How programs store records in files

A file is bytes; a format gives them meaning

To the operating system, a file is nothing but a numbered sequence of bytes. It does not know about users or orders. A format is an agreement between the programs that write and read the file about what those bytes mean.

Format	One record looks like	Good at	Weak at
CSV	42,Asha,asha@example.com	Tables, spreadsheets, exports	Nesting, types: everything is text
JSON	{"id": 42, "name": "Asha"} inside one big array	Nested data, readable by people	The whole file is one value, so it is read and written as a whole
JSON Lines	One JSON object per line	Appending records one at a time	Changing or removing a record in the middle
Binary formats	Bytes laid out for machines (Parquet, Protocol Buffers)	Size and speed	You need a tool to read them

All four share one property that matters a lot: records have no fixed size or position. “Asha” takes fewer bytes than “Venkataraman”, so record 42 could start anywhere. There is no way to jump straight to it.

Finding one record means reading everything

If you cannot jump to a record, you have to read from the start until you find it. For a JSON file it is worse: the whole file is one array, so the program reads and parses every record into memory before it can look at any of them.

That cost grows in a straight line with the size of the file. Double the users, double the time to find one.

Changing a record means rewriting the file

Opening a file for writing normally truncates it: the old contents are thrown away and the new ones written from the beginning. To change one record in a JSON file, a program reads everything, changes the record in memory, and writes everything back.

There are two cheaper alternatives, and both lead somewhere interesting:

• Append instead of rewrite. With JSON Lines, adding a record is one write at the end of the file. Opening a file in append mode makes every write land at the end. But to change record 42 you either rewrite the file or append a new version of record 42 and let readers take the last one. Then the file keeps growing with stale versions, and something has to clean them up from time to time. That is the core idea behind log-structured storage engines.
• Give every record a fixed size and position. If each record takes exactly 100 bytes, record 42 starts at byte 4,200, and you can read or overwrite it without touching anything else. Databases build on this idea: they store data in fixed-size pages and keep indexes that say which page holds which key.

Either way, you have started building a database.

Check yourself

1. Why can’t a program jump straight to record 42 in a CSV or JSON file?

   Show answer

   Records have no fixed size, so their byte positions depend on every record before them. Without an index of positions, the only way to find one is to read from the start.

   I could explain this without looking

2. Why does changing one user’s name in a large JSON file take seconds?

   Show answer

   JSON stores the whole collection as one value. To change any part of it, the program parses everything into memory and writes the entire file back out.

   I could explain this without looking

3. JSON Lines lets you append a record with one write. What problem does that leave for updates and deletes?

   Show answer

   You must either rewrite the file, or append a new version or a deletion marker and have readers keep only the latest. The file then fills with stale versions that something must periodically clean up.

   I could explain this without looking