Async Wait for Files to Change using Node.js' FS

• Download example - 1.4 KB

Introduction

Let us assume the following scenario. We have a web site or web application where the user can trigger a specific action in the UI. The result of this action will be written to a file to disk. Now it is our job to write the end-to-end tests (a.k.a. e2e tests) for this software, eventually using a framework like Protractor [^].

Probable use cases may incldue:

• Stand alone web applications with access on the file system
• Server client applications where the server should save data to a file
• Server client applications when the client should download a file

The software under test will write given data to a file on the file system (or not). Our test must assert whether this happens as specified and in the specified max time. Therefore, we need a test that waits for the given file to appear. When the file is created, the test can proceed with the next step, e.g., asserting the file content. If the file does not appear in a required time span, the test should fail.

Concept

Sleep

The most simple approach would be just to pause the test for an acceptable time span while the application is creating the file. After this, the test can check whether the file is present and proceed or exit with red. But this approach comes with a problem. On the one hand, the time to wait should be as short as possible to make our tests fast. On the other hand, this duration should be long enough not to make the test flaky when we extend the application logic or change the test hardware.

Wait

For the discussed timing issues, we need some more advanced solution which allows us to wait for the file and react when it is created. Writing our e2e test would be much easier when we had a library with this API:

• Watch a set of files for a given period of time. Files do not mandatory have to exist in the beginning.
• Notify, if a file changes within the timeout: file created, file (content) changed or file deleted.
• Notify, if the file does not change within the given timeout.

And finally, as we want to use Protractor and Jasmine, the library API must fit into Protractor's asynchronous call pattern.

Writing the Code

This code is a thin wrapper around Node.js' fs library. The watchFile function (see documentation [^]) watches a file and notifies the caller each time the file is changed. This includes creating, modifying and deleting the file. All we will do is add a timeout and add a little interpretation for callback arguments.

Overview

The API will provide two functions: One to start waiting for a file change and one to stop. Behind the scenes, we have a private function to handle the notification callback. That's it.

Start Waiting

Let's start with the startWaiting function. This function starts looking for a file to change for a given amount of time. It can be applied on a file, a directory or any other valid target of fs.watchFile.

With fs.watchFile, we register a callback function to be called on file changes. Wrap the onChanged callback in this anonymous function to pass the client's callback as well as some local variables.
We start the timer with setTimeout. It will call stopWaiting and notify the client with a 'Timed out.' message. The setTimeout result we store in timer to allow us to cancel the timer later on.

function startWaiting(path, callback, timeout) {
    var timer = setTimeout(function () {
            stopWaiting(path);
            callback('Timed out.');
    }, timeout);

    fs.watchFile(path, options, function (curr, prev) {
        onChanged(curr, prev, path, timer, callback);
    });
}

Stop Waiting

The stopWaiting function ends watching the file by calling fs.unwatchFile while the timer is still running. It takes the path to the target file as single argument and is as simple as this:

function stopWaiting(path) {
    fs.unwatchFile(path, this);
}

React on File Changes

StartWaiting registers onChanged as callback on fs.watchFile. This function is the most complex one in this example. So let's invest some time here.

Identify the Change Event

The arguments current and previous contain the file property stats [^] from fs.watchFile. Comparing these two objects will help us in sorting out, what actually happened. The properties mode (file type and mode) or mtime (time stamp of last modification) seem to be the most suitable (see man7.org [^] for further explanations).

We distinguish the following event types:

• File created
• File modified
• File deleted
• No file

The last type may sound a little surprising because it is not actually a change event. When fs.watchFile is called for a file or directory that does not exist, the API responds with a callback call containing empty objects. The decision is up to us, whether or not we notify our client on this case.

Cleanup Callbacks

For the e2e test, we want to react on the first event on the file system only. Therefore, we cleanup the callback registrations with stopWaiting and clearTimeout (Skip this step for the 'No file' event).

Notify the Caller

The last step in this function will be to notify the original caller (a.k.a. client) about the event. It is up to our API design, which details we want to share with the client. We can either limit the API to the identified type string or pass the file stats as well.

function onChanged(current, previous, path, timer, clientCallback) {
    var type = 'File modified.';
    if (current.mode === 0 && previous.mode === 0) type = 'No file.';
    else if (current.mode > 0 && previous.mode === 0) type = 'File created.';
    else if (current.mode === 0 && previous.mode > 0) type = 'File deleted.';

    if (type !== 'No file.') {
        stopWaiting(path);
        clearTimeout(timer);
    }

    clientCallback(type, current, previous);
}

Watch Options

There is one little detail we skipped so far. The fs.watchFile function takes an options argument to define the polling behavior. For this demo, we set the interval to 500 milliseconds.

var options = {
    persistent: true,
    interval: 500
};

Points of Interest

Example Code

To see the full example, download the attachment. You can start the little Node.js script without npm install.

$ node index.js

Alternatives

fs.watch

Node.js' fs.watch function (documented here [^]) allowed us to watch for changes on a directory or file. In contrast to fs.watchFile, this function uses higher order OS functions instead of polling (e.g. inotify on Linux and ReadDirectoryChangesW on Windows), what would be definitely the more elegant solution. As this function throws an error if the target file does not exist in the beginning, it is difficult to monitor a specific file in the e2e test scenario. One option could be to monitor the parent directory. But then, we had to check for the modified file on every change event on the directory to select the relevant ones.

fs.access

Another approach could be fs.access (documentation [^]), which checks whether the file is accessible by the current application. In some use cases, these details can be very helpful. For this function, we had to implement the pull loops by hand. I would only recommend doing so, if accessibility is a vulnerable criteria.

Conclusion

Node.js' fs library does not contain an API function to get notified about the creation of a specific file. But with a little effort, we can write a thin adapter to provide this behavior. Our library waits for a file or directory to be changed (created, modified or deleted). This file must not be present at the beginning. Now this library can be consumed by our e2e test.

If you have any comments, questions or ideas for improvement, feel free to write a comment below.

History

• March 25^th, 2016: Initial version