CodeParkDon't stream big files through your API. Let the client upload directly to object storage with a pre-signed URL, then process the file in the background.
Why not through the Node API?
- Each upload ties up connections and bandwidth on your app servers for minutes.
- This makes the servers hard to scale.
- If the connection drops, the upload starts over.
The better design:
1. Ask for permission. The client asks the API to start an upload. The API checks auth and quotas. It creates a metadata record with status pending.
2. Get a pre-signed URL. The API returns a pre-signed URL for object storage such as S3. This is a short-lived URL that allows exactly that one upload. This idea is called the valet key pattern: like giving a parking valet a key that only opens one car.
3. Upload directly. The client uploads straight to storage, not through your servers. For large files, use multipart upload. The file is split into parts. The parts upload in parallel, and each part can be retried on its own. So uploads can resume after a failure.
4. Process in the background. When the upload finishes, a storage event or a client callback puts a job on a queue. Workers then:
- validate the file
- scan it for viruses
- transcode it (convert the video format) or create thumbnails
- set the status to
ready
5. Serve downloads through a CDN (content delivery network: servers close to users that cache files) or through pre-signed GET URLs.
- Don't send large files through your app servers.
- A pre-signed URL lets the client upload directly to object storage for a short time.
- Multipart upload sends parts in parallel and lets uploads resume.
- A completion event puts a job on a queue for background processing.
- Track the file's status in a metadata record, and serve files through a CDN.