Dynamic JQPlot

• Download sample code

Introduction

This article explains how to build a web application that can dynamically build a graph based on the JQPlot library.

Though the code is written in Java, it can be easily ported to C#, C++, VB or any other language.

Tools and libraries

To create this project the following was used:

• NetBeans 7.2
• JQPlot
• jQuery

Source Code

This article is based on a real application called Staff which can be viewed here: http://www.staff.oma.be and as such can partly be seen as a case study. It might be helpful to check out this site and do a "View Source". The website plots timelines of data that are related to solar physics.

However, to get you started I have stripped down the Staff application to its bare minimum.

Background

The goal was to create an application, capable of showing timeline data with four constraints:

1. It needed to handle lots of data
2. It needed to be fast
3. It needed to be accessible (for everyone)
4. it needed to be usable (ease of use)

These constraints posed an initial problem because accessibility and lots of data contradict with speed. The mission: To build a web application that could handle billions of records.

Note that, despite the timeline character of this article (1 value, 1 datetimestamp), the code can be reused for other kinds of plots as well.

Using the code

Basically there are two streams of workflow coming together. The workflow of building the JSON code itself to generate the plot and the flow getting the data out of the database and getting it into the plot.

1. Building the JSON code

This part is easy, it is nothing more then copying the properties of the JQPlot (found here) and building classes with it with the standard getters and setters.

These are the classes you can get very far with:

• GraphAxes
• GraphCanvasOverlay
• GraphCursor
• GraphGrid
• GraphHighlighter
• GraphLegend
• GraphSeries

Note: Not all properties of the JQPlot API are already in there. They are, however, easily added. Usually I did a raw HTML tryout based on the JQPlot examples and worked my way up from there.

To be on the safe side I transformed some of the properties into enumerations, but most of them are as-is. The most notable feature of each class is the override of the toString method which returns the object as JSON code. To combine all the objects I generated the Graph class also with an overridden toString method.

Using the code is limited to two steps:

1. set all the properties you want.
2. use the Graph's toString method to write the JSON to your web page.

Couldn't be more easier.

public class Graph {
    private String graphtitle = "";
    private ArrayList<GraphSeries> graphseries;
    private ArrayList<GraphAxes> graphaxes;
    private GraphLegend graphlegend;
    private GraphCanvasOverlay graphcanvasoverlay;
    private GraphHighLighter graphhiglighter;
    private GraphCursor graphcursor;
    private GraphGrid graphgrid;
 
//getters - setters here

    @Override
    /** returns the complete string javascript representation for 1 JQPlot object */
    public String toString(){
        StringBuilder builder = new StringBuilder("optionsObj = 
          $.extend(true, {}, { title: '"+ graphtitle +"',\r\n");
        
        //write the series part
        if(graphseries != null){
            builder.append("series:[");
            for(int i = 0; i < graphseries.size(); i++){
                GraphSeries gs = graphseries.get(i);
                if(gs != null){
                    builder.append(gs.toString());
                }
            }
            //remove the final ','
            builder.delete(builder.length()-1, builder.length());
            builder.append("]");
            builder.append(",\r\n");
        }
        
        
        //write the axes part
        if(graphaxes != null){
            builder.append("axes:{");
            for(int i = 0; i < graphaxes.size(); i++){
                GraphAxes ga = graphaxes.get(i);
                if(ga != null){
                    builder.append(ga.toString());
                }
            }
            //remove the final ','
            builder.delete(builder.length()-1, builder.length());
            builder.append("},\r\n");
        }
        
        //write the grid
        if(graphgrid != null){
            builder.append("grid:{");
            builder.append(graphgrid.toString());
            builder.append("},");            
        }
        
        //write the legend
        if(graphlegend != null){
            builder.append("legend:{");
            builder.append(graphlegend.toString());
            builder.append("},");
        }
        
        //write the canvasoverlay
        if(graphcanvasoverlay != null){
            builder.append("canvasOverlay:{");
            builder.append(graphcanvasoverlay.toString());
            builder.append("},\r\n");
        }
        
        //write highlighter properties
        if(graphhiglighter != null){
            builder.append("highlighter:{");
            builder.append(graphhiglighter.toString());
            builder.append("},\r\n");
        }
        
        //write cursor properties
        if(graphcursor != null){
            builder.append("cursor:{");
            builder.append(graphcursor.toString());
            builder.append("}\r\n");
        }
        builder.append("});");
        
        return builder.toString();
    }  

The Graph class contains several objects directly related to each object of the JQPlot library. Each class has its own overridden toString method to write out that part of the JSON code. The graph class just ties it all together. I refrained from using to much layout (white space characters) which saves some bytes to be sent. The datasets could become quite large.

2. Getting the data.

This will differ depending on your own needs. However I will explain how we managed to plot so much data.

The starting point of our reasoning here is that we have huge amounts of data (one of the tables holds more than 1.8x10⁹ records). If someone selects the entire timerange for this amount data, well, just forget about that.

It is also useless to even try to show so many points. My laptop screen has a resolution of 1920x1080, but many screens have only 1280x1024 or less. Simply put: getting more than, let's say 2000 points is useless anyway. One could argue that you can use a scrollbar to plot part of the graph outside the window, but this would, for me, go against constraint #4, usability.

The solution to this is to downsample the data to various levels. Most of our data is 1 datapoint / minute. (not all of them, which makes it rather complex, but for not making things too difficult, let's assume 1 datapoint / minute). That meant downsampling to various "lower resolution" timeframes like hour, day, month, year.

In our case we used standard averages on the data, but more advanced calculations (including error margins and showing those) are, of course, possible. The difficulty in downsampling is when you go outside of your normal timeframes which where 5 minute averages and Carrington Rotations.

The other difficulty was the fact that when data was uploaded, some datapoints are marked "incorrect". We imported these on the highest resolution data, but don't want these to mess up the averages. While downsampling, only points that were complete, where accepted. For example when we want to create the hour downsample table you make sure you have 60 points available on the 1 minute table to calculate that average with.

In our case, data is also imported near real-time where each source, and this is important, is restructured in more or less the same way, which makes the design of your code more straightforward. The near real-time import is a complete separate module running as a cron job on a Linux server.

Finally, it goes without say that a good database structure and indexes are important. Our database is a PostGreSql database, but MySQL, SQL-Server and Oracle will do as well.

3. Putting the data and the graph together

To put the data from the database into the plot proved also to be quite a challenge.

To determine the correct (downsample) table to select the data from, the start and end datetime was taken and the difference calculated. Through some metadata (that holds among other things the highest resolution of that timeline) the downsampling level can be found.

E.g., if your highest cadence is 1 minute data and you want to plot maximum 2000 points in one series, you should take 1 hour averages (instead of 1 minute data) if your time difference exceeds 2000 minutes or 33 hours. if you go above 2000 hours you should take the day averages etc... (notice that 2000 hours is already 120 000 records of 1 minute data.)

The second thing was that some data series needed the same axes, while others needed a different axis. Combing the correct series, with the correct axis has to be defined based on your own business rules, but was luckily feasible with the Graph classes from step 1. In my case I used a GraphBuilder object to build the Graph object based on the selected data.

Here's the code from the sample source code. This is about as simple as you can get for plotting two series on one axis. To start out working on this, you might want to try to plot these two series on two different axis instead of one and work your way up to more complex situations from there.

/** Builds the Graph object from the selected data. */
public Graph Build(ArrayList<DataPoint> seriesa, 
  ArrayList<DataPoint> seriesb, GregorianCalendar start, GregorianCalendar end){
    Graph graph = new Graph();

    ArrayList<GraphAxes> al_axes = new ArrayList<GraphAxes>();
    ArrayList<GraphSeries> al_series = new ArrayList<GraphSeries>();
    
    GraphCanvasOverlay gco    = getGraphCanvasOverlay();
    GraphCursor gc        = getGraphCursor();
    GraphHighLighter ghl    = getGraphHighLighter();
    GraphLegend gl        = getGraphLegend();
    GraphGrid gg        = getGraphGrid();
    GraphAxes xaxis        = getGraphXAxes(start, end);
    al_axes.add(xaxis);
    
    GraphSeries serie = getGraphSeries("Series A", 0);
    al_series.add(serie);
    serie = getGraphSeries("Series B", 0);
    al_series.add(serie);
    GraphAxes yaxis = getGraphYAxes(0, "Calculated Value");
    al_axes.add(yaxis);
    
    addGraphCanvasOverlayObjects(gco, 0);

    graph.setGraphTitle("Codeproject JQPlot example");
    graph.setAxes(al_axes);
    graph.setSeries(al_series);
    graph.setLegend(gl);
    graph.setCanvasOverlay(gco);
    graph.setGraphCursor(gc);
    graph.setGraphGrid(gg);
    graph.setHighlighter(ghl);

    return graph;
}

Finally you need to write out the JavaScript which not only includes the JSON part of the graph, but also some JavaScript arrays for the actual data and the HTML to put the plot area in.

This is how the basic HTML looks like:

<%
// This part loads the PageBuilder object.
PageBuilder pb = new PageBuilder();
%> 

<script><%= pb.WritePlotData() %></script>

<div id="main">
    <div class="plotcanvas">
    <div id="plotarea">
        <!-- This is the actual plotting area. -->
        <div id="chart" class="graph"></div>    
    </div>
    </div>
</div>
<script type="text/javascript" class="code">
    <%= pb.WriteGraphDefinition() %>
    $(document).ready(function(){
        CreateGraphAsync();
    });
</script>

The WritePlotData and WriteGraphDefinition functions, only generate and "spit out" HTML and JavaScript.

4. Dotting the i's

It is my experience that even with a fancy, blazingly fast architecture, an application can still remain uninteresting due to the user interface being or ugly, or difficult or both. That is why we have the usability constraint.

In order to solve this problem we considered some rules on the user interface:

• Look and feel must remain consistent throughout the page.
• It should look nice taking in mind we don't live in 256 color age, nor is anybody interested in a rainbow colored webpage (they exist, trust me, that's horrible).
• Coming on the page, the most important part should be in view, attracting the user's attention. (in this case the plot)
• User controls that are often used should be close by and ready to use, advanced, less used features can be hidden in a menu.

Conclusion

The four constraints we started out with have been met:

• lots of data

The huge still growing database contains billions of records without noticeably slowing down the application.

• speed

Despite being a web application the loading speed is within acceptable limits. This is due to the good handling of the large datasets by downsampling and to the simplicity of the design that views a webpage as a "piece of text". Furthermore, client-side scripting is used for many of the controls, limiting postbacks to the server. The only postback done in our application is when a new dataset has to be fetched.

• accessibility

The web page is available to everyone if you have internet connection.

• usability

The web page is very easy to use.

Points of Interest

The basic point of view in the way of working is that your web browser is nothing more than a text interpreter, no more different than Notepad++ or Word. This is the reason why I used the StringBuilder so much as this is very efficient.

The real application where this article comes from is called Staff and the first version can be found online here. The example code is a simplified version of the real application where the complexity is far greater, but where I also used much better and cleaner coding techniques (defensive programming, exception handling, logging, ...). I deliberately kept the code for this article at a minimum.

Note: I have more a .NET background and have not that much experience with Java. It is very well possible there are other solutions to some problems. However, the Staff application works and is very fast because of the good design which is for the complexity of the encountered problems surprisingly simple.

History

• Version 1: February 12, 2013.