CodeParkFan-out on write makes reads fast but writes expensive. Fan-out on read makes writes cheap but reads slow. For a celebrity, use a hybrid: don't push their posts, and merge them in when a follower reads the feed.
Fan-out on write (push): when someone posts, write the post id into each follower's ready-made feed. This feed is usually a list with a size limit, kept in a cache. Reading a feed is then one fast lookup.
The cost is write amplification: one post creates one write for every follower. Work is also wasted on users who never open the app.
Fan-out on read (pull): store each post only with its author. When a user opens the feed, fetch recent posts from everyone they follow and merge them. Writes are cheap. But reads are slow and expensive for users who follow many accounts.
The celebrity problem: one celebrity post with fan-out on write means 50 million writes. The usual answer is a hybrid:
- Use push for ordinary accounts.
- For accounts above a follower limit, skip the fan-out.
- At read time, merge the celebrity's recent posts into the feed.
Two more details:
- Feeds store only post ids. The app then fills in the full posts from a post cache. This step is called hydration.
- The fan-out runs in the background (asynchronously) through a queue, so the person posting doesn't wait.
- Push gives fast reads but causes heavy write amplification.
- Pull gives cheap writes but slow, expensive reads.
- A hybrid uses push for normal users and pull for celebrities.
- Feeds hold only post ids, which are filled in from a cache.
- Fan-out runs in the background through a queue.