Azure Redis Viewer - Chrome Application for Azure Developers

Introduction

In this article I want to share my experience of developing a Chrome Application called "Azure Tools". I'm going to describe general idea of the first tool - Redis Viewer, and explain the chosen solutions and design approaches. Azure Tools can be useful for Azure developers and its code can be useful for those who build Single Page Applications (SPA) in general and Chrome App in particular.

Technologies:

• Angular - javascript SPA (Single Page Application) framework
• RedisJs - open source redis client
• Browserify - provides require() style analyzer to organize code
• Net-Chromify - network communication module for Chrome App
• Karma/Jasmine - unit tests
• Metro Bootstrap - css

Source code - GitHub

Azure Tools - Google Web Store Installation

Thanks Sam Holder for review and his comments.

Choosing Chrome App as a Platform

Why is Azure Tools not a plugin for Visual Studio? It's a developer tool. - The idea was to create the application which can work on most of platforms: Windows, Unix (ideally, mobile as well). I see it as a big lack of many existing, good tools like Azure Storage Explorer which only work on Windows, but might be used by developers on Mac OS. 

Why is Azure Tools not a desktop app written on one of cross-platform technology? - I was looking for compromise between web and desktop apps and one of disadvantages of desktop apps is difficulties in delivery and updating to new versions of the software. Chrome App provides desktop app user experience and automated delivery/update process by Google Web Store.

Why it's not a web application? - Access to Azure services requires providing credentials and sensitive information by a user. I personally would think twice before sending my cloud credentials to third party web server, even if the owner swears they do not store any credentials and use them only to load information for you.  A Chrome App has more trust from this point of view, it can work with cloud service directly from user's machine without "intermediary" server.

Design Challenges

I faced with few challenges when I started thinking about Azure Tools.

First of all Azure Tools should be single-tier, entirely javascript app. If Chrome App platform does not allow communicate with Azure directly that requires implementation of server side - "intermediary" between user app and cloud services, and as I said that's what I definitely didn't want to do for security reasons. Also development of Azure Tools as a Chrome App with server side for doesn't really make sense and it probably be better to just be a web app.

Second follows from the first: Chrome App should utilize NodeJs facilities and packages and have enough permissions to do that.

Third, application should have a modular design. This doesn't mean the app should load modules dynamically, on demand, but is mostly about code organization and ability reuse common modules and simplicity when adding new modules.

Last but not least, how to organize automated testing of a Chrome App.

Redis Viewer Functionality

Redis Viewer is the first and at the moment only module of Azure Tools.

If you didn't know about Redis yet, I highly recommend you to learn more about this technology. In few words Redis is a distributed cache and in-memory key-value storage. Each value in redis represents one of the data structure: string, list, set, hash, sorted set, bitmap and other. Apart from common opertions like key expiration or create, read, update, delete value by key, Redis provides specific operations for each kind of data structure like get substring for string value and sort for lists. And it processes comands fast. Very fast. 

Microsoft provides Redis as a service of Azure cloud platform. Everybody can create their own Redis cluster under an Azure account, and can persist data and execute commands using REST interface.

The main purpose of Redis Viewer tool is the visualisation of Azure Redis content and basic operations on Redis storage data. The viewer includes following features:

• Displays a list of existing keys and their type and value
• Allows to search keys by specific pattern
• Add new key-value items
• Update existing value
• Delete by key
• Supports string, set and hash types

UI

This is basically what the UI of Redis Viewer looks like:

Redis Viewer is a Chrome App so you can install it from Web Store and the app will be added to your Chrome App Launcher panel and to your Chrome browser on any supported platform.

Running NodeJs Redis Client in Chrome

I found useful Browserify util to organize code following NodeJs module system and to have ability use NodeJs modules in Chrome App. The basic idea of Browserify is that developer can reference a javascript module using required() method and then execute Browserify preprocessing which uses code analysis to find all require() dependencies and combine them in a single file. For instance, if your file app.js requires module a.js which requires module b.js you can reference them in your code like this:

// app.js
var a = require('./a.js').A;

// a.js
var b = require('./b.js').B;
exports.A = function(){}

// b.js
exports.B = function(){}

After running Browserify preprocessing command: 

browserify app.js -o bundle.js

there will be a generated bundle.js file which contains following code:

The code is ugly but as you can see it includes both: dependencies a and b which Browserify analyzer found and included in bundle file.

Considering that, I'm able to add NodeJs Redis client via npm and preprocess it using Browserify to use in Chrome App. Just one more problem: Redis client uses net module which encapsulates functionality of network communication like read/write on sockets, data serialization etc. This module installs as part of NodeJs and not working in Chrome App, because Chrome App supposes to work with sockets via Chrome API. So the next step is that I need replace net with implementation that using Chrome API. There is a package called net-chromify which I installed via npm.

So to prepare code base which can communicate with Redis server I need to execute following command:

browserify app.js -r ./node_modules/net-chromify/index.js:net > bundle.js

that makes Browserify preprocessing of app.js file and uses net-chromify implementation in places where the net module is being used. Preprocessed code saves in bundle.js file.

Note: Actually there were still issues with Redis client after all those steps. Not many, but required making changes in Redis client to get it working in Chrome App. So if you need fixed version you can check it out from AzureTools project repo. I will take care to push the fixes into the main RedisJs repo.

And finally to get Redis client working in Chrome App I need to set permissions for tcp operations in app manifest:

"permissions": [
    {
        "socket": [
            "tcp-listen:*:*",
            "tcp-connect",
            "resolve-host"
        ]

    }
]

Redis Scanner

Redis provides couple of commands which actually should be deprecated I think. One of them is KEYS command that allows to fetch all keys from the Redis database, but it slows down Redis server and worsens its performance. That means any viewer which uses KEYS command to obtain a list of keys cann't be used against production environment.

Alternatively, set of SCAN commands can be used for the same purpose but with much less impact on performance. That approach underlies Redis Scanner module and also Redis Viewer limits the number of loaded keys from database by 100 (this limitation does not apply if user is searching keys by pattern).

Redis Scanner usage looks like this:

// keys of type set or hash set
// load value entries not as a single set but one by one
// so entries must be grouped by key
var groupByKey = function(type, key, value) {
    var existing = self.Keys.filter(function(item) { return item.Key == key; });
    if (existing !== null && existing[0] !== undefined) {
       var values = JSON.parse(existing[0].Value);
       values.push(value);
       existing[0].Value = JSON.stringify(values);
       return false;
    } else {
       self.Keys.push({ Key: key, Type: type, Value: JSON.stringify([value]) });
       return true;
    }
}

// load redis data
var maxItemsToLoad = 100;
self.loadKeys = function(pattern) {
    self.Keys.length = 0;
    var loadedCount = 0;

    var client = $redisScannerFactory({
        pattern: pattern,

        each_callback: function(type, key, subkey, p, value, cb) {
             switch(type) {
               case 'set': if(groupByKey(type, key, value)) { loadedCount++; }; break;
               case 'hash': if(groupByKey(type, key, value)) { loadedCount++ }; break;
               default: 
                  self.Keys.push({ Key: key, Type: type, Value: value}); 
                  loadedCount++;
                  break;
               }

             if ((searchViewModel.Pattern === '' || searchViewModel.Pattern === '*')
                && loadedCount >= maxItemsToLoad) {
                showInfo('First ' + maxItemsToLoad + 
                         ' loaded. Use search to find specific keys.');
                cb(true);
             } else {
                cb(false);
             }
         },

         done_callback: function(err) {
             if (err) {
                 $messageBus.publish('redis-communication-error', err);
             }

             $dataTablePresenter.showKeys(self.Keys, self.updateKey, self.removeKey);
         }
   });
};

Using Angular

Modularity

Angular in conjunction with Browserify allows to organize code in modular way in Chrome App and separate application concerns. The following code demonstrates modular composition of Azure Tools:

require('./exceptionHandling/exceptionHandlingModule.js').register(angular);
require('./common/commonModule.js').register(angular, angularRoute);
require('./common/dialogsModule.js').register(angular, angularRoute);
require('./common/actionBarModule.js').register(angular);
require('./redis/redisModule.js').register(angular, angularRoute);
require('./tiles/tilesModule.js').register(angular, angularRoute);

var app = angular.module('app', [
            'exceptionOverride',
            'common',
            'actionBar',
            'dialogs',
            'tiles',
            'tiles.redis'
        ], function() {}).controller('AppController', ['$state', function ($state) { }]);

Each module registers it's own controllers, directives, service, constants in angular. The 'app' module is application root which aggregates all application modules. Any new module or tool must be registered in application root.

Modules can not be loaded dynamically. They are all compiled by Browserify in a single bundle.js file which packs in Chrome App.

Navigation

As you could notice from Azure Tools module structure there are 'tiles' and 'tiles.redis' modules registered. 'tiles' represents a start screen and lists the available Azure tools. 'tiles.redis' is Redis Viewer which can be launched from start screen. The same way all subsequent tools can work.

But Azure Tools is a single page application and there is no url navigation like in web application. So to implement navigation and screens switching within a single html page I used angular-ui plugin.

Angular-ui allows to declare screens as a states and transitions between them. This code registers tiles and redis screens:

// tilesModule.js
tilesModule.config(function ($stateProvider) {
    $stateProvider
        .state('tiles', {
            url: "", // empty url redirects to tiles start screen
            templateUrl: "tiles/view/index.html",
            controller: 'TilesController',
            params: {
                seq: {}
            }
        });
});

// redisModule.js
redisModule
   .config(function ($stateProvider) {
       $stateProvider
           .state('redis', {
               url: "/redis",
               templateUrl: "redis/view/index.html",
               controller: 'RedisController',
               params: {
                   seq: {}
               }
           });
   });

Placeholder for current screen in HTML:

        <div id="content" ui-view>
        </div>

Navigation from tiles screen to redis screen performs following code:

// tilesViewModel.js
self.goToRedis = function () { $state.go('redis', {});} // where $state implemented by angular-ui 
                                                        // and can be injected in view model(controller)

Angular-UI is great from MVVM perspective, because developer can specify states in view model, and describe application flow in abstract way, and move details like animated transition between screens to view layer. That's great. 

Redis Viewer MVVM

Model represented by redis structures: string, set and hash and repositories for CUD (create/update/delete) operations on them. For read operation Redis Scanner is used and repositories use Redis NodeJs client to fetch entities. We talked about scanner and Redis NodeJs client in sections above. Additionally there is redis settings class which is keeping user credentials to access Azure Redis cluster and validation rules.

View Model represented by AngularJs controller (personally, I would be happy to see viewModel() function in angular, but controller is more generic name for different MVx patterns).

     // redisViewModel.js
     module
        .controller('RedisController', [
            '$scope',
            '$timeout',
            'messageBus',
            'activeDatabase',
            'redisRepositoryFactory',
            'redisScannerFactory',
            'validator',
            'redisSettings',
            'dataTablePresenter',
            'actionBarItems',
            'dialogViewModel',
            'confirmViewModel',
            'notifyViewModel',
            'busyIndicator',
            function(
                // infrastructure and utils
                $scope,
                $timeout,
                messageBus,

                // model
                activeDatabase,
                redisRepositoryFactory,
                redisScannerFactory,
                validator,
                redisSettings,
                
                // view models
                dataTablePresenter,
                actionBarItems,
                dialogViewModel,
                confirmViewModel,
                notifyViewModel,
                busyIndicator) { ... }]);

Redis view model depend on other view models like 'dialogViewModel', 'notifyViewModel', 'busyIndicator' etc.; model: 'redisRepositoryFactory', 'redisScannerFactory' etc., infrastructure and util object.

As you might notice, it's not pure MVVM implementation because of dataTablePresenter. I use it to wrap operation on jquery DataTables (grid) like drawing table rows with Redis data, expanding/collapsing details, sorting etc. DataTables are not MVVM-friendly and implementation of angular directives might be swimming againt the river. So abstract presenter works fine for me in this case: it doesn't break testability and redisViewModel doesn't work directly with DOM elements.

View is html files and a bit Metro Bootrap js code. I'm using following project structure for view layer in Azure Tools:

     -- app
     ---- content
     ------ css
     ------ img
     ------ js             // view specific Metro Bootrap code here
     ---- redis            // redis module
     ------ view
     -------- index.html
     ---- tiles            // tiles module
     ------ view
     -------- index.html

Unhandled Error Handling

Angular allows you to add a custom handler for unhandled exceptions which occurred only in angular context, handler will not be invoked for errors out of angular. Azure Tools handles unexpected errors by displaying appropriate message to the user and link to send email to developer with error details.

angular
    .module('exceptionOverride', [])
    .factory('$exceptionHandler', [function () {
        return function (exception, cause) {
            var data = {
                type: 'angular',
                localtime: Date.now()
            };

            if (cause) { data.cause = cause; }
            if (exception) {
                if (exception.message) { data.message = exception.message; }
                if (exception.name) { data.name = exception.name; }
                if (exception.stack) { data.stack = exception.stack; }
            }

            var el = document.getElementById('sendEmail');
            var alertArea = document.getElementById('alertArea');
            if (el && alertArea) {
                alertArea.style.display = "block";
                el.href = 'mailto:' + supportEmail + '?subject=' + 'Bug Report' + '&body='
                    + data.message + '|' + data.name + '|' + data.stack
                    + '|' + data.type + '|' + data.url + '|' + data.localtime;
            }

            throw exception;
        };
    }]);

Unit Testing

MVVM design of Azure Tools and Angular dependency injection allow to cover application logic with a unit tests. I use Jasmine testing framework and Karma test launcher for automation.

First, I need to mock out real Redis client with fake implementation of Redis storage. Fake implementation of Redis storage is pretty trivial and you can find it under app/redis/model/mocks folder in sources. 

Once fake Redis implementation was done I had another problem. How to inject fake implementation in redisViewModel which I want to test? My initial mistake was that I tried to reference testing redisViewModel.js file and all files with dependencies in Karma configuration. Such approach required manual injection of dependencies in unit test, but I had already configured most of dependencies in application root and just need to mock out couple of them. So I reworked tests to load browserified bundle.js files in Karma and to use angular-mocks to mock out model services.

redisViewModel test of adding new Redis key looks like this:

// redisTests.js
describe('RedisController', function () {
    beforeEach(module('app'));
    beforeEach(function () {
        // replace real Redis client
        // with mock implementation
        angular.module('tiles.redis')
            .factory('redisClientFactory', function () {
                return redisClientFactoryMock;
            })
          .factory('redisScannerFactory', [
            'redisDataAccess', 'redisScanner',
            function (redisDataAccess, redisScanner) {
                return redisScannerMock;
            }
          ]);
    });

    it('should save string value',
       inject(function ($rootScope, $controller, actionBarItems, dialogViewModel) {
           // arrange
           var scope = $rootScope.$new();
           $controller("RedisController", {
               $scope: scope,
           });
           var viewModel = scope.RedisViewModel;
           data = [
                { Key: 'key:1', Type: 'string', Value: '1' },
                { Key: 'key:2', Type: 'string', Value: '2' }
           ];

           // act
           // user clicks add key
           actionBarItems.addKey();
           // provides key and value for string data structure
           dialogViewModel.BodyViewModel.Key = 'key:3';
           dialogViewModel.BodyViewModel.Value = '3';
           // saves new key in redis storage
           dialogViewModel.save();

           // assert
           expect(viewModel.Keys.length).toBe(3); // 2 initial + 1 new added
           expect(viewModel.Keys[0].Key).toBe('key:1');
           expect(viewModel.Keys[1].Key).toBe('key:2');
           expect(viewModel.Keys[2].Key).toBe('key:3');
       }));
});

Karma configuration for tests:

//  karma.config.js
module.exports = function(config) {
    config.set({
    basePath: '',
    frameworks: ['jasmine'],
    browsers: ['Chrome'],
    files: [
         'app/node_modules/angular/angular.js',
         'app/node_modules/angular/angular-*.js',
         'tests/lib/angular/angular-mocks.js',

         'tests/*.js',

         'app/bundle.js',
         'app/redis/model/mocks/*.js',
    ]});
};

Command line to run unit tests:

karma start karma.config.js

Test results:

Demo

Conclusion

A Chrome App is good compromise between web and desktop applications. Of course, it can't suite every type of application, so your choice must be justified.

Thanks for reading. I'd very appreciative of your comments and a like on GitHub and Code Project.