cloe's story

How to download files using Node.js

There are three approaches to writing a file downloader app using Node - i. HTTP.get, ii. curl, iii. wget. I have created functions for all of them. To get the examples working makes sure you have the dependencies and app variables intact. Read the comments thoroughly, you will not only learn how to download files, but will also learn more about Node's child_process, File System, Buffers, and Streams. Let's start with HTTP.get.

Downloading using HTTP.get

HTTP.get is Node's built-in mechanism for making HTTP GET requests, which can also be used for downloading files using the HTTP protocol. The advantage of using HTTP.get is that you don't rely on any external programs to download files.

// Dependencies
var fs = require('fs');
var url = require('url');
var http = require('http');
var exec = require('child_process').exec;
var spawn = require('child_process').spawn;

// App variables
var file_url = 'http://upload.wikimedia.org/wikipedia/commons/4/4f/Big%26Small_edit_1.jpg';
var DOWNLOAD_DIR = './downloads/';

// We will be downloading the files to a directory, so make sure it's there
// This step is not required if you have manually created the directory
var mkdir = 'mkdir -p ' + DOWNLOAD_DIR;
var child = exec(mkdir, function(err, stdout, stderr) {
    if (err) throw err;
    else download_file_httpget(file_url);
});

// Function to download file using HTTP.get
var download_file_httpget = function(file_url) {
var options = {
    host: url.parse(file_url).host,
    port: 80,
    path: url.parse(file_url).pathname
};

var file_name = url.parse(file_url).pathname.split('/').pop();
var file = fs.createWriteStream(DOWNLOAD_DIR + file_name);

http.get(options, function(res) {
    res.on('data', function(data) {
            file.write(data);
        }).on('end', function() {
            file.end();
            console.log(file_name + ' downloaded to ' + DOWNLOAD_DIR);
        });
    });
};

The above function is probably the best way to download files using HTTP.get in Node. Make a HTTP.get request and create a writable stream using fs.createWriteStream. Since the HTTP.get's response is a stream, it has the data event, which carries the chunks of data sent by the server. One each data event, write the data to the writeable stream. Once the server finishes sending data, close the instance of fs.createWriteStream. If you are trying to use fs.write orfs.writeFile or any of their variants, they will fail for medium to large files. Usefs.createWriteStream instead for reliable results.

Downloading using curl

To download files using curl in Node.js we will need to use Node's child_process module. We will be calling curl using child_process's spawn method. We are using spawn instead of execfor the sake of convenience - spawn returns a stream with data event and doesn't have buffer size issue unlike exec. That doesn't mean exec is inferior to spawn; in fact we will use exec to download files using wget.

// Function to download file using curl
var download_file_curl = function(file_url) {

    // extract the file name
    var file_name = url.parse(file_url).pathname.split('/').pop();
    // create an instance of writable stream
    var file = fs.createWriteStream(DOWNLOAD_DIR + file_name);
    // execute curl using child_process' spawn function
    var curl = spawn('curl', [file_url]);
    // add a 'data' event listener for the spawn instance
    curl.stdout.on('data', function(data) { file.write(data); });
    // add an 'end' event listener to close the writeable stream
    curl.stdout.on('end', function(data) {
        file.end();
        console.log(file_name + ' downloaded to ' + DOWNLOAD_DIR);
    });
    // when the spawn child process exits, check if there were any errors and close the writeable stream
    curl.on('exit', function(code) {
        if (code != 0) {
            console.log('Failed: ' + code);
        }
    });
};

The way data was written to the instance of fs.createWriteStream is similar to way we did for HTTP.get. The only difference is that the data and end events are listened on the stdout object ofspawn. Also we listen to spawn's exit event to make note of any errors.

Downloading using wget

Although it says downloading using wget, this example applies to downloading using curl with the -O option too. This method of downloading looks the most simple from coding point of view.

// Function to download file using wget
var download_file_wget = function(file_url) {

    // extract the file name
    var file_name = url.parse(file_url).pathname.split('/').pop();
    // compose the wget command
    var wget = 'wget -P ' + DOWNLOAD_DIR + ' ' + file_url;
    // excute wget using child_process' exec function

    var child = exec(wget, function(err, stdout, stderr) {
        if (err) throw err;
        else console.log(file_name + ' downloaded to ' + DOWNLOAD_DIR);
    });
};

In the above method, we used child_process's exec function to run wget. Why exec and notspawn? Because we just want wget to tell us if the work was done properly or not, we are not interested in buffers and streams. We are making wget do all the dirty work of making request, handling data, and saving the file for us. As you might have guessed, this method is the fastest among the three methods I described.

So now the question is - which method is the best? The answer - whatever suits your need. The wgetmethod is probably the best is you want to save the files to the local disk, but certainly not if you want to send those files as a response to a current client request; for something like that you would need to use a stream. All the three methods have multiple options, you choice will ultimately depend on what your needs are. Happy downloading!

Further Reading

1. Node.js HTTP
2. Node.js fs
3. Node.js Child Processes
4. Node.js Buffers
5. Node.js Streams