You build a string by calling chunk.toString() on each data chunk of a UTF-8 stream. Occasionally the output contains � replacement characters. Why, and how do you fix it?

A multi-byte character is split across two chunks, and each half is decoded on its own. Each broken half becomes the replacement character �.

In UTF-8, ASCII characters take 1 byte, but other characters take 2 to 4 bytes. A stream cuts its chunks at any byte position. It does not care where a character starts or ends. So one character's bytes can land partly in one chunk and partly in the next. When you call chunk.toString() on each chunk, Node sees an incomplete character and outputs �.

The fix is to decode across chunk boundaries. You have three options:

• Call readable.setEncoding('utf8'). The stream then emits strings, and it holds back an incomplete byte sequence until the rest arrives.
• Use a StringDecoder from node:string_decoder. Its write() method keeps partial characters in a buffer until the rest arrives. Its end() method outputs whatever is left.
• Collect all the buffers in an array and call Buffer.concat(chunks).toString() once at the end. This is fine when the total size is bounded, meaning you know it stays small.