You need to serve or process a 4 GB file. Why is fs.readFile a bad choice, and what should you use?

fs.readFile loads the whole file into memory at once. Use fs.createReadStream instead, which reads the file in small chunks.

fs.readFile reads the whole file into a single Buffer before your callback runs. For a 4 GB file, that means gigabytes of memory for each request. It can also simply fail, because there is a maximum buffer size. Even with medium-sized files, a few requests at the same time can use up all the memory.

fs.createReadStream reads in chunks, so memory use stays about the same no matter how big the file is. Connect it to the destination with pipeline. The destination can be an HTTP response, or a step like zlib.createGzip() on the way. pipeline gives you two things:

• Backpressure: when the client is slow, the stream stops reading from disk until the client catches up.
• Cleanup: if an error happens, the file descriptor is closed for you.

To process the file line by line, wrap the stream in readline.createInterface. Then loop over its lines with for await.