Node.js Streams: Processing Data You Cannot Fit in Memory

A stream processes data in chunks instead of loading it all at once. The reason to care is concrete: reading a 4 GB file into a buffer fails, and streaming it uses a few megabytes.

The problem, in two lines#

// Allocates the whole file. Fails above ~2 GB and hurts long before that.
const data = await readFile('huge.csv');
 
// Constant memory, regardless of file size.
createReadStream('huge.csv').pipe(transform).pipe(createWriteStream('out.csv'));

Under concurrency the difference is starker: ten simultaneous requests each buffering a 200 MB file is 2 GB of RAM. Streaming, it is negligible.

The four types#

• Readable — a source. fs.createReadStream, an HTTP request, process.stdin.
• Writable — a sink. fs.createWriteStream, an HTTP response, process.stdout.
• Duplex — both, independently. A TCP socket.
• Transform — a Duplex where output is derived from input. zlib.createGzip().

Always use pipeline(), never pipe()#

pipe() has a real design flaw: it does not forward errors, and it does not clean up the other streams when one fails. A failed write leaves the read stream open — a file-descriptor leak that only shows up under load.

import { pipeline } from 'node:stream/promises';
 
// Errors propagate, and every stream is destroyed on failure.
await pipeline(
  createReadStream('input.csv'),
  createGzip(),
  createWriteStream('output.csv.gz'),
);

pipeline from stream/promises is the modern form: it awaits, it rejects on error, and it guarantees cleanup. There is no reason to use raw pipe() in new code.

Back-pressure is the point#

If a source produces faster than the destination consumes, unbounded buffering follows. Streams handle this automatically: write() returns false when the destination's buffer is full, and pipe/pipeline pause the source until a drain event.

You only need to handle it manually when writing to a stream directly:

for (const row of rows) {
  if (!output.write(row)) {
    // Buffer full — wait before writing more.
    await once(output, 'drain');
  }
}

Ignoring the return value of write() is how a "streaming" implementation quietly buffers gigabytes.

Async iteration is usually the clearest form#

import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';
 
const lines = createInterface({
  input: createReadStream('access.log'),
  crlfDelay: Infinity, // treat \r\n as one break
});
 
let errors = 0;
for await (const line of lines) {
  if (line.includes(' 500 ')) errors++;
}

for await gives you back-pressure for free — the loop body cannot run ahead of what it processes — and reads like ordinary code.

Writing a Transform#

import { Transform } from 'node:stream';
 
const redact = new Transform({
  objectMode: true,
  transform(chunk, _encoding, callback) {
    // callback(error) reports failure; callback(null, value) emits.
    try {
      this.push({ ...chunk, email: '[redacted]' });
      callback();
    } catch (error) {
      callback(error);
    }
  },
});

objectMode: true is what lets a stream carry JavaScript objects rather than Buffers — essential for CSV-row or JSON-record pipelines.

Web Streams vs Node streams#

Node also implements the WHATWG Web Streams API (ReadableStream, TransformStream), which is what fetch bodies and Next.js Route Handlers use. They are not the same objects, but they convert:

import { Readable } from 'node:stream';
 
const nodeStream = Readable.fromWeb(response.body);
const webStream = Readable.toWeb(createReadStream('file.txt'));

For new code that runs in both Node and edge runtimes, prefer Web Streams — they are the portable API.

Frequently asked questions#

When is a stream not worth it?#

For data comfortably smaller than a few megabytes. readFile is simpler, and simpler wins when the constraint does not bind.

Why does my stream hang?#

Usually an unconsumed readable, or a transform that never calls its callback. A paused readable with no consumer will simply wait forever.

How do I limit concurrency across many streams?#

A worker pool or a semaphore around the pipeline calls. Streams bound memory per pipeline, not the number of pipelines.

Can I stream a JSON array?#

Not with JSON.parse, which needs the whole document. Use a streaming parser, or prefer newline-delimited JSON (.ndjson) where each line is a complete record — far easier to stream and to resume.

Related reading#

• Async/Await Explained
• CSV to JSON Conversion — the classic streaming use case

References#

• Node.js docs: Stream
• Node.js docs: stream/promises pipeline