Using system-default browser UI from node.js

Introduction

There are many cases when you have to create a desktop application with web technologies.

I remember using HTA for this many years ago. Now we have node-webkit but it has a limitation which matter sometimes: application size. Blink is an excellent engine but modern Windows and Mac OS X are shipped with built-in browsers that doesn’t require too much browser-specific work. Why not to make use of them?

This article describes creation of an open-source project which aims to help creating cross-platform lightweight javascript+html apps targeting modern operating systems.

The source code and binary releases are available to download from GitHub.

Background

Some things here are rather tricky. After completion of several projects related browser embedding, webview apps, website-to-apps, and so on, I have decided to create an open-source framework to simplify work, combine all the hacks in one place and share the code. I have found now abandoned app.js and borrowed some ideas from it, replacing WebKit (chrome embedded framework) with different browser hosts.

The project has been made open-source because I think someone can find it useful. Now there's no twitter, mailing list and the only documentation is wiki pages on github because I'm not sure whether such concept will be relevant and usable for now, so any comments and thoughts are welcome.

Usage

All the functionality is contained in ui module which is described in detail in this article. A very basic app is created in this way:

var ui = require('ui');
var app = ui.run({
  url: '/',
  server: { basePath: 'app' },
  window: { width: 600, height: 400 },
  support: { msie: '6.0.2', webkit: '533.16', webkitgtk: '2.3' }
});
app.server.backend = function (req, res) { /*serve non-static content*/ };
app.window.onMessage = function(data) { /*handle message*/ };

This app checks if minimal supported browser requirements are met (if not, downloads webkit on Windows), creates a window with browser webview, and loads index.html file from app folder.

Node.js

Browser javascript APIs doesn’t support all functions vital for desktop apps. To provide missing functionality, node.js is linked into executable.
First, the app is started as usual node.js app. Node.js allows us to override app startup script with _third_party_main.js:

(comment from node.js source)
// To allow people to extend Node in different ways, this hook allows
// one to drop a file lib/_third_party_main.js into the build
// directory which will be executed instead of Node's normal loading.

To disable terminal window on Windows, we must create Windows entry point (WinMain proc) and compile node.js with Windows subsystem (/SUBSYSTEM:Windows flag). And here we get the first trouble: node.js fails on startup. If we investigate this, we’ll notice that node.js actually fails on interaction with stdio (stdout, stderr and stdin). So, to fix this, standard streams must be replaced.
First, we detect, do we actually have usable stdio or not? If not, the property is deleted from process and replaced with empty PassThrough stream.

function fixStdio() {
    var
        tty_wrap = process.binding('tty_wrap'),
        knownHandleTypes = ['TTY', 'FILE', 'PIPE', 'TCP'];
    ['stdin', 'stdout', 'stderr'].forEach(function(name, fd) {
        var handleType = tty_wrap.guessHandleType(fd);
        if (knownHandleTypes.indexOf(handleType) < 0) {
            delete process[name];
            process[name] = new stream.PassThrough();
        }
    });
}

And that’s it – now node.js can run as a GUI app.

Adding Native Built-In Node.js Module

Window with webview (or web browser control) is implemented in C++ and are exposed to javascript from a node.js addon
To reduce loading time, the module is statically linked, as all other node natives. This is done in following way:

void UiInit(Handle<Object> exports
#ifndef BUILDING_NODE_EXTENSION
    ,Handle<Value> unused, Handle<Context> context, void* priv
#endif
        ) {
  // module initialization goes here
}
#ifdef BUILDING_NODE_EXTENSION
NODE_MODULE(ui_wnd, UiInit)
#else
NODE_MODULE_CONTEXT_AWARE_BUILTIN(ui_wnd, UiInit)
#endif

It’s possible to compile as node addon (it won’t work though but original idea was to build as addon), so initialization could be performed in two modes.
Then we load native binding in such way:

process.binding('ui_wnd');

This native binding is loaded inside module file ui.js which is included into build as other natives, and this is the module that will be returned from require('ui') call.

Packaging Apps into Executable

To reduce garbage in working directory of the redistributable app and make portable apps, javascript files can be packaged. They are actually zipped into executable in ZIP format, exactly like in SFX archive.

The executable contains its code and app payload. On startup, the engine reads archive header and when required, extracts files contents. File reading is designed in such way so it doesn't load entire archive in memory and reads files when required instead.

Virtual File System

To access files packaged into executable, node.js should have some file name which is <executable_dir>/folder/.../file.ext. There’s no possibility to set up a sort of filesystem link at certain point and serve files from memory. To tell fs module that files within this folder should be read in custom way instead of filesystem access, we replace native fs bindings and create some checks in them:

var binding = process.binding('fs');
var functions = { binding: { access: binding.access } }
binding.access = function(path, mode, req) {
    var f = getFile(path);
    if (!f)
        return functions.binding.access.apply(binding, arguments);
    // custom implementation
};

If the file was found in virtual filesystem, it will be served from it. Otherwise, the request will be redirected to file system. The exception is write requests: we cannot write to app archive, so all file opens with write access go directly to filesystem.

Using this technique, we can package application files, node_modules and content files into archive and work with them as if they were in real filesystem, transparent to the app and node.js. However if the application is aware of vfs and wants to perform some optimization, there’s a possibility to distinguish that the file has been found in vfs: fs.statSync('file').vfs.

Creating a Window with WebView

A class representing OS window is created in C++ node.js addon and derived from node::ObjectWrap:

class UiWindow : public node::ObjectWrap {
    // ...
    virtual void Show(WindowRect& rect) = 0;
    virtual void Close() = 0;
private:

    static v8::Persistent<v8::Function> _constructor;

    static void New(const v8::FunctionCallbackInfo<v8::Value>& args);
    static void Show(const v8::FunctionCallbackInfo<v8::Value>& args);
    static void Close(const v8::FunctionCallbackInfo<v8::Value>& args);
    // ...
}

To export symbols to javascript, we initialize them in this way:

void UiWindow::Init(Handle<Object> exports) {
    Isolate *isolate = Isolate::GetCurrent();
    HandleScope scope(isolate);
    Local<FunctionTemplate> tpl = FunctionTemplate::New(isolate, New);
    tpl->SetClassName(String::NewFromUtf8(isolate, "UiWindow"));

    // static methods
    NODE_SET_METHOD(tpl, "alert", Alert);
    // prototype methods
    NODE_SET_PROTOTYPE_METHOD(tpl, "show", Show);
    
    auto protoTpl = tpl->PrototypeTemplate();
    // properties
    protoTpl->SetAccessor(String::NewFromUtf8(isolate, "width"), GetWidth, SetWidth, Handle<Value>(), DEFAULT, PropertyAttribute::DontDelete);
    // constants (static fields)
    tpl->Set(isolate, "STATE_NORMAL", Int32::New(isolate, WINDOW_STATE::WINDOW_STATE_NORMAL));
    // constructor function 
    _constructor.Reset(isolate, tpl->GetFunction());
    // class export
    exports->Set(String::NewFromUtf8(isolate, "Window"), tpl->GetFunction());
}

After the addon is loaded, in javascript, we can use:

var window = new ui.Window({ /* config object */ });
window.show();
window.close();

Inside Show method window invokes os-specific implementation (WinAPI on windows, Cocoa on Mac and GTK+ on Linux), this is like any other typical app, without anything special or interesting here, so I won’t pay attention to it.

WebView integration on Mac and Linux is pretty straightforward. Internet Explorer is created as an ActiveX control; I had to add several hacks to prevent undesired keyboard events, dialogs and navigation.

Event Emission

Window can emit events: show, close, move, etc… Window UI code is executed on main thread, unlike node.js, so to interact with node thread, we need some synchronization. To reduce os-specific code, this is performed in cross-platform way with uv library built in to node.js:

First, on window creation we store node thread id and async handle:

class UiWindow {
	// ...
    uv_thread_t _threadId;
    static uv_async_t _uvAsyncHandle;
    static void AsyncCallback(uv_async_t *handle);
}
uv_async_init(uv_default_loop(), &_this->_uvAsyncHandle, &UiWindow::AsyncCallback);

When the event actually happens, os-specific implementation calls EmitEvent function:

void UiWindow::EmitEvent(WindowEventData* ev) {
    ev->Sender = this;
    // … add pending event to list
    uv_async_send(&this->_uvAsyncHandle);
}

Then, uv calls AsyncCallback in node thread:

void UiWindow::AsyncCallback(uv_async_t *handle) {
    uv_mutex_lock(&_pendingEventsLock);
    WindowEventData* ev = _pendingEvents;
    _pendingEvents = NULL;
    uv_mutex_unlock(&_pendingEventsLock);
    // handle pending events
}

Events are added to list because UV can call AsyncCallback one time for several events; there’s no guarantee that it will be called once per event. Window is inherited from EventEmitter (in ui module), in this way function emit is added to the prototype. Then we get this emit function and call it:

Local<Value> emit = _this->handle()->Get(String::NewFromUtf8(isolate, "emit"));
Local<Function> emitFn = Local<Function>::Cast(emit);
Handle<Value> argv[] = { String::NewFromUtf8(isolate, "ready") };
emitFn->Call(hndl, 1, argv);

That's it. Window can now generate events invoked from any thread, pass arguments and process output from event subscribers, e.g. window close cancel:

window.on('close', function(e) { e.cancel = true; });

Browser Interaction

Adding communication API object

To interact browsers, backend object providing messaging methods is added to window’s global context. This object is created with a script injected into webview on javascript context initialization with different methods, depending on the browser used. On IE we can make use of NavigateComplete event:

void IoUiBrowserEventHandler::NavigateComplete() {
    _host->OleWebObject->DoVerb(OLEIVERB_UIACTIVATE, NULL, _host->Site, -1, *_host->Window, &rect);
    _host->ExecScript(L"window.backend = {"
        L"postMessage: function(data, cb) { external.pm(JSON.stringify(data), cb ? function(res, err) { if (typeof cb === 'function') cb(JSON.parse(res), err); } : null); },"
        L"onMessage: null"
        L"};");
}

On CEF, there’s OnContextCreated callback:

void IoUiCefApp::OnContextCreated(CefRefPtr<CefBrowser> browser, CefRefPtr<CefFrame> frame, CefRefPtr<CefV8Context> context) {}

WebKit webview initialization on mac happens on didCommitLoadForFrame signal:

- (void)webView:(WebView *)sender didCommitLoadForFrame:(WebFrame *)frame {}

Calling C++ from javascript

To call javascript methods from window, in IE window.external object is used; it’s added as a COM object implementing IDispatch interface. Functions are called directly on that object:

class IoUiSite : public IDocHostUIHandler {
STDMETHODIMP GetExternal(IDispatch **ppDispatch) {
    *ppDispatch = _host->External;
    return S_OK;
}
}
class IoUiExternal : public IDispatch {
	// ...implement Invoke and handle calls
}

On Mac OS X, we can create just a simple object, add methods to it and allow to call them from WebView, by responding to webScriptNameForSelector and isSelectorExcludedFromWebScript:

@interface IoUiWebExternal: NSObject
- (void) pm:(NSString*)msg withCallback:(WebScriptObject*)callback;
@end

@implementation IoUiWebExternal
- (void) pm:(NSString*)msg withCallback:(WebScriptObject*)callback {
	// handle call
}
+ (NSString *) webScriptNameForSelector:(SEL)sel {
    // tell javascriptcore engine about the method
    if (sel == @selector(pm:withCallback:))
        return @"pm";
    return nil;
}
+ (BOOL) isSelectorExcludedFromWebScript:(SEL)sel { return NO; }
@end

On CEF, we can just add a native method to existing object. This is done in such way:

auto window = context->GetGlobal();
auto backend = window->GetValue("backend");
auto pmFn = window->CreateFunction("_pm", new IoUiBackendObjectPostMessageFn(browser));
backend->SetValue("_pm", pmFn, CefV8Value::PropertyAttribute::V8_PROPERTY_ATTRIBUTE_DONTDELETE);

A function object is actually a class implementing function call method:

class IoUiBackendObjectPostMessageFn : public CefV8Handler {
public:
    virtual bool Execute(const CefString& name, CefRefPtr<CefV8Value> object, const CefV8ValueList& arguments,
        CefRefPtr<CefV8Value>& retval, CefString& exception) override;
private:
    IMPLEMENT_REFCOUNTING(IoUiBackendObjectPostMessageFn);
};

Chrome Embedded Framework

Not all IE versions are practically usable nowadays, while old OS market share is now is still not zero, so I had to add support for Windows XP by embedding Chrome Embedded Framework (CEF). CEF includes complete rendering engine (Blink) and V8; its binary is a set of DLLs, resource files and locales. When the app is started, first, it checks whether CEF binary is present, and if it is, starts CEF host. To reduce startup time, CEF is launched in single-process model:

CefSettings appSettings;
appSettings.single_process = true;

Anyway, if the renderer process has crashed, there’s no need to continue app execution, so there would be no benefits of multi-process architecture, it would just slow down the app.

Downloading CEF

Chrome Embedded Framework is large in size (about 30MB compressed), that's why it is downloaded only on old systems instead of embedding into the app. The application checks browser version with user-provided requirements (from support key), and if it's lower than expected, shows progress dialog and starts downloading CEF. Once download is finished, the archive is extracted and CEF DLL is loaded into the app.

Reading ZIP files with node.js

One of the requirements for my project was serving video files from archives. I haven't found any javascript implemetation capable to stream files from ZIP archives without reading entire archive in memory, that's why I have forked adm-zip and created node-stream-zip which can stream files from huge archives and decompress them on the fly with node's built-in zlib module. First, it reads ZIP header, takes file sizes and offsets from it and, when requested, streams zipped data, passing it through zlib decompression stream and CRC checking pass-through stream. This works pretty fast, doesn't slow down app startup and doesn't consume too much memory.

Points of Interest

• MSIE: typeof window.external === 'unknown' (this is so-called host object)
• MSIE: silent mode does not turn off all supplementary user interaction dialogs on all IE control versions
• node.js is by default compiled with V8 as shared library, with all functions exported from executable; it was turned off to reduce executable size

Downloads

I have not attached downloads to the article, you can find them on GitHub:

• project page
• binary releases
• some examples

History

2015-04-19: first public import
2015-05-10: some bugfixes and website