VRT - Realtime HTML5 Data Visualization Application

Introduction

Hello, my name is Odd Marthon Lende. I am employed as a Technical Support Engineer in Stavanger, Norway.

I started working on this project 3-4 years ago, it started out as a hobby project at work. The mission was to create a monitoring application. An application that receives data on a web service and reflects it back to browser clients.

This project has taken too much of my time and energy, it has been abandoned several times for many months. There have been periods (months, weeks) with a lot of new ideas and writing. To complete my work project, I have written a similar application using only Microsoft technology and a few 3^rd party libraries in 3-4 months.

VRT

Is/should be a versatile visualization application in the browser using standard web technologies that enables the user/developer to create visualizations of data inside containers called widgets. These widgets have a standard interface that allows you to read data in a standard way and create your own visualization using the web technology of your choice (HTML, Canvas, WebGL, 3^rd party libraries) inside this container. These programs can then be grouped together into a group called a dashboard. The widgets can then be positioned and scaled inside the body of the browser window. A standard interface for accessing and reading data into a common format (table/matrix), so it can easily be extended with different data sources. I have been working on reading JSON, CSV and also implemented a data source using the Web Audio API.

cd vrt             # Change the directory to where you cloned the repository
npm install        # Installs the required nodejs modules
node standalone.js # This starts the nodejs process

# If the process started successfully,
# you should be able to open the browser (only Google Chrome currently) to
# http://localhost:23485

VRT is still in early development and I will not spend any more time developing it if there is no interest for it. I've released the source on github and was hoping to get some developers/designers interested.

The Struggle for Speed

D3

The first serious attempt was to create a simple table as a widget, I found a library called D3.

D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG, and CSS. D3’s emphasis on web standards gives you the full capabilities of modern browsers without tying yourself to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation.

Creating a table using HTML elements with not many columns and rows worked fine, try creating 500 000 - 1 000 000 elements and look at the browser memory / CPU and you'll know what I mean. D3 is a library that allows you to create SVG graphics, so for every data point there will be created a new DOM element or more in memory, it will bring the browser to a halt eventually.

D3 is a great library, but I was not very impressed with the speed and memory usage when loading a large amount of data.

I do not use it for any of the visualizations in the latest update, but I've used it in the UI a lot.

Canvas

I also experimented with canvas, the code has been deleted in the current revision, but it is still there in the repository. I tried optimizing, so a big difference here from D3 would be having the possiblity to not redraw all the data points when there is an update. To add a new line to the canvas when a new datapoint arrived, I would copy the image data and shift the whole image position horizontally and using the previous point from memory and then draw a line from the previous point to the new point. When pumping up the data thoughtput, it would start lagging a lot.

It's about being able to draw the data using as few programming steps as possible.

WebGL

This is the best solution to the problems. It provides the ability to harness the power of the graphics card and program with fewer JavaScript calls that will result in more speed.

Receive Data - Upload to graphics card - Render.

To use WebGL, I added an extra layer, that is a chain of programs that runs on the graphics card. It is represented as a JavaScript object that is instantiated one or multiple times and connected together to create an rendering chain, it is an abstraction of a vertex/pixel shader and has some functionality to tween/transition parameters. This is how the blurring occurs when an UI dialog is opened/closed.

Please visit the github page for more information and fork or star it. If people want to develop it, I think the next step for me would be to start writing some documentation.