Node.js Streams, Pipe and chaining

A stream is an abstract interface implemented by various objects in Node.js.  Due to asynchronous and event driven nature Node.js is very good at handling I/O bound tasks/streams, streams are actually unix pipes.

For example a request to an HTTP server is a stream, as is stdout. request is a readable stream and response is a writable stream, streams could be Readable, Writable, or Duplex (both readable and writable). Readable streams let you read data from a source while writable streams let you write data to a destination. A “duplex” stream is one that is both Readable and Writable, such as a TCP socket connection.

All streams are instances of EventEmitter. Take a look at following example.

var fs = require('fs');
var readStream = fs.createReadStream('file.txt');
var text = '';

readStream.on('data', function(chunk) {
text+=chunk;
});

readStream.on('end', function() {
 console.log(text);
});

The function fs.createReadStream() gives you a readable stream object. using that object you listen to data event with attach callback. As soon as chunks of data are read and passed to your callback, we appends those chunks to our text string. At then end when there is no more data to read, the stream emits an end event. In the above code, we listen to this event to using another callback and we logged the text string.

You can set encoding on the stream by calling readStream.setEncoding().As a result, the data is interpreted as utf8 and passed to your callback as string.

At this point we also have interesting function called pipe() which we can use to transfer streams flow from source to destination. like we have piped the readable stream to writable stream like bellow.

var fs = require('fs');
var readStream = fs.createReadStream('file1.txt');
var writeStream = fs.createWriteStream('file2.txt');

readStream.pipe(writeStream);

So now we have piped readable stream of file1.txt to writable stream created for file2.txt, pipe() will manages the data flow, you don’t have handle it yourself and callbacks.

Its also comes up with chaining feature, so multiple destinations can be piped safely.

var readStream = fs.createReadStream('file.txt');
var zlib = zlib.createGzip();
var writeStream = fs.createWriteStream('file.txt.gz');
readStream.pipe(zlib).pipe(writeStream);

In above example we have piped readStream to gzip. Purpose of pipe() method is to limit the buffering of data to acceptable levels, so that sources and destinations of varying speed will not overwhelm the available memory.