Introduction
Recently, after seeing a lot of advertisement for web-based visualization dashboards, I set out to answer the following question. Is there an easy way to provide simple web-enabled 2D/3D visualization dashboards for natively built C/C++ applications for free?
Background
I have reviewed many web-based visualization solutions (dashboard software). Many of them require supporting client side technologies such as .NET, Flash, Java, etc... In our particular situation, we did not want to burden our end-users with the responsibility of maintaining client-side frameworks. Another issue was the availability of these technologies for all of the various browser implementations and their supported platforms. In addition, many of the solutions rendered data on the client side, an approach that suffers in performance for huge data sets. When I talk of huge data sets, I am talking about tens of thousands of data points. With no budget for development, the biggest factor was cost.
So, how do you provide a web-enabled 2D-3D data visualization solution for natively built applications on a zero budget? One answer, write your own embeddable web-server that can take 2D or 3D data, render it to an image, and send the rendered image to a client-browser for visualization. This might sound like a major development effort, but it's not. In fact, the solution presented here took less than 15 minutes to develop.
The Solution
The diagram above illustrates our approach. In our approach, we use three components:
- gnuplot
- an embedded web-server
- bubble (plug-in)
Let's begin with gnuplot. gnuplot is a powerful natively built plotting tool that has been around for years, runs on most platforms, and is well documented (there are books on gnuplot). It has an extensive scripting language that is conducive to dynamic plots, a requirement for even the most basic dashboard solution. In addition, it is free to distribute -- no royalties.
Since we are talking about adding the functionality to natively built applications, the simplest approach is an embedded one. We use the Snorkel SDK to develop the embedded server component. Snorkel (http://snorkelembedded.webs.com/) is an embedded/application server-SDK that contains a powerful set of APIs designed to make coding solutions like this one easy, and as with gnuplot, it is also free.
Data visualization is a useful tool that we will likely want to reuse in future projects. For the purpose of reusability, we write the renderer as a plug-in, that is, a bubble in Snorkel terminology. The bubble will act as an interface between our embedded web-server and gnuplot.
For those who are not familiar with the term bubble, bubbles are shared objects that encapsulate and associate one or more functions with file extensions (MIME types), URIs, and/or proprietary protocols. Not to be confused with CGIs, bubbles run within the web-server not as a separate process.
In general, bubbles contain a single exported function, bubble_main. As part of the load process, the Snorkel runtime calls each bubble's bubble_main routine to perform any necessary initialization and to identify the functional purpose of the bubble. It is from this routine that associations between MIMEs, URIs, and user defined protocols are established.
In this project, we associate the extension gpng with gnuplot script files. A gnuplot script is a text file that contains a set of instructions that gnuplot executes to produce an image. We register the extension-MIME type in the bubble's bubble_main routine using the API function snorkel_obj_set.
. .
51 #define GNUENV "GNUPLOT"
52 #define TERMINAL "set terminal png transparent truecolor enhanced"
53 #define GNU_ARGS " -e "
54 #define EXTENSION "gpng"
55 #define TYPE FILE_PNG
56 #define MODE "wt"
.
.
241
242
243 byte_t SNORKEL_EXPORT
244 bubble_main (snorkel_obj_t server)
245 {
246 snorkel_obj_set (server,
247 snorkel_attrib_mime,
248 EXTENSION,
249 TYPE, encodingtype_binary, gnuplot_gpng);
250 return 1;
251 }
In the call to snorkel_obj_set, we identify the extension, the file type, the encoding, and the routine responsible for rendering MIMEs of type gpng. Basically, the call instructs the server to render all files whose filenames have the extension .gpng with the function gnuplot_gpng.
The gnuplot_gpng function takes incoming requests for URIs of type gpng and sends their URLs to gnuplot for rendering/execution. gnuplot converts incoming gpng URLs to temporary PNG (Portable Network Graphics) files, and the function gnuplot_gpng streams them back to the requesting client's browser before deleting them.
51 #define GNUENV "GNUPLOT"
52 #define TERMINAL "set terminal png transparent truecolor enhanced"
53 #define GNU_ARGS " -e "
54 #define EXTENSION "gpng"
55 #define TYPE FILE_PNG
56 #define MODE "wt"
57
58 #if !defined(WIN32) && !defined(WIN64)
59 #define _popen popen
60 #define _pclose pclose
61 #define _dir_sep '/'
62 #define sprintf_s snprintf
63 #else
64 #define _dir_sep '\\'
65 #endif
.
.
.
117
118
125 call_status_t
126 gnuplot_gpng (snorkel_obj_t http,
127 snorkel_obj_t connection, char *pszurl)
128 {
129 unsigned int canary[] = _CANARY_VALUE;
130 char szwork_dir[512];
131 char sztmpname[256];
132 char szcommand[1024];
133 char *gnuplot = 0;
134 char *psz = 0;
135 FILE *fd = 0;
136 FILE *gnupipe = 0;
137 char *psztmp = 0;
138 char *pszfile = 0;
139
140
141
146 sztmpname[0] = 0;
147 szcommand[0] = 0;
148 szwork_dir[0] = 0;
149 strncat (szwork_dir, pszurl, sizeof (szwork_dir));
150 pszfile = strrchr (szwork_dir, _dir_sep);
151 if (pszfile)
152 {
153 *pszfile = 0;
154 pszfile++;
155 }
156 else
157 {
158 _stkchk;
159 return ERROR_STRING ("could not determine working directory\r\n");
160 }
161
162
163
166 gnuplot = get_pgnuplot ();
167 if (!gnuplot)
168 {
169 _stkchk;
170 return ERROR_STRING ("GNUPLOT not set\r\n");
171 }
172
173
179 tmpnam (sztmpname);
180
181 psztmp =
182 (sztmpname[0] == _dir_sep) ? &sztmpname[1] : sztmpname;
183
184
189 sprintf_s (szcommand, sizeof (szcommand),
190 "%s\"cd '%s';%s;set output '%s';load '%s'\"",
191 gnuplot, szwork_dir, TERMINAL, psztmp, pszfile);
192
193 #if defined(_DEBUG) || defined(DEBUG)
194 printf("%s\n",szcommand);
195 #endif
196
197
202 gnupipe = _popen (szcommand, MODE);
203 if (!gnupipe)
204 {
205 _stkchk;
206 return
207 ERROR_STRING
208 ("error: could not open gnuplot pipe\r\n");
209 }
210 fflush (gnupipe);
211 _pclose (gnupipe);
212
213
214
220 sprintf_s (szcommand, sizeof (szcommand), "%s%s", szwork_dir,
221 sztmpname);
222
223
224 if (snorkel_file_stream (connection,
225 szcommand,
226 0,
227 SNORKEL_BINARY) == SNORKEL_ERROR)
228 {
229 remove (szcommand);
230 _stkchk;
231 return ERROR_STRING ("could not stream data\r\n");
232 }
233
234
236 remove (szcommand);
237 _stkchk;
238
239 return HTTP_SUCCESS;
240 }
Let us take a closer look at the accompanying source. We begin with lines 146-160, where we extract the working directory and resource name from the provided URL. Next, we call the function get_gnuplot to acquire the command to execute gnuplot. On lines 173-183, we create a temporary file name that we later use for generating the PNG file.
Next, on lines 184-192, we build our command line and invoke gnuplot using an open pipe, lines 202-211. gnuplot executes the script and renders it to our temporary file as a PNG image. We then take the PNG file produced by gnuplot and, on lines 224-232, stream it back to the requesting client. Once we are done, we delete the temporary file and return success.
Since our bubble will work with any Snorkel based embedded solution, we reuse the server from one of our previous articles to test the bubble.
Running the Example
To run the example:
On Linux
- Expand the source-binary bundle dashboard.zip
- Change directories to deployment_directory/bin/Linux
- Set the library search path variable LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/deployment_directory/lib/Linux
On Windows
- Expand the source-binary bundle dashboard.zip
- Open a DOS/command prompt and change directories to deployment_directory/bin/wintel
Start the Server
- Set the environment variable GNUPLOT to point to the fully qualified path of the gnuplot executable. Note: gnuplot is not included with this article, but you can download its binaries from here (http://sourceforge.net/projects/gnuplot/files)
- From within the deployment_directory/bin/platform directory, enter the command "fsrv -p 8080" to start the server
- Launch your favorite browser and enter http://localhost:8080
The index file in deployment_directory/bin/platform references four separate gpng files located in the same directory, displaying them in a two by two table. The files were taken from gnuplot examples and modified for this project.
<html><body>
<table width="90%" cellpadding="0">
<tr><td><img src="finance.gpng" alt=""></td>
<td><img src="finance2.gpng" alt=""></td>
</tr>
<tr><td><img src="iterate.gpng" alt=""></td>
<td><img src="transparent.gpng" alt=""></td>
</tr>
</table></body></html>
You can display each plot individually by renaming the index file (for example, appending a %), or removing it. You will need to stop the server if it is running, prior to performing this action. With the index file hidden or removed and the server started, you can select each file individually from within the browser -- displaying them one at a time.
For example, selecting the file finance2.gpng from within a browser produces the following image:
The gnuscript script that produced the image above is listed below:
set label 1 "Acme Widgets" at graph 0.5, graph 0.9 center front
set label 2 "Courtesy of Bollinger Capital" at graph 0.01, 0.07
set label 3 " www.BollingerBands.com" at graph 0.01, 0.03
set logscale y
set yrange [75:105]
set ytics (105, 100, 95, 90, 85, 80)
set xrange [50:253]
set grid
set lmargin 9
set rmargin 2
set format x ""
set xtics (66, 87, 109, 130, 151, 174, 193, 215, 235)
set multiplot
set title "Change to Bollinger Boxes"
set size 1, 0.7
set origin 0, 0.3
set bmargin 0
set ylabel "price" offset 1
plot 'finance.dat' using 0:3:3:($2>$5?$2:$5):($2>$5?$2:$5) notitle with candlesticks lt 3, \
'finance.dat' using 0:($2<$5?$5:1/0):($2<$5?$5:1/0):
($2<$5?$2:1/0):($2<$5?$2:1/0) notitle with candlesticks lt 2, \
'finance.dat' using 0:($2>$5?$2:1/0):($2>$5?$2:1/0):
($2>$5?$5:1/0):($2>$5?$5:1/0) notitle with candlesticks lt 1, \
'finance.dat' using 0:($2<$5?$2:$5):($2<$5?$2:$5):4:4 notitle with candlesticks lt 3, \
'finance.dat' using 0:9 notitle with lines lt 3, \
'finance.dat' using 0:10 notitle with lines lt 1, \
'finance.dat' using 0:11 notitle with lines lt 2, \
'finance.dat' using 0:8 axes x1y2 notitle with lines lt 4
unset label 1
unset label 2
unset label 3
unset title
set bmargin
set format x
set size 1.0, 0.3
set origin 0.0, 0.0
set tmargin 0
unset logscale y
set autoscale y
set format y "%1.0f"
set ytics 500
set xtics ("6/03" 66, "7/03" 87, "8/03" 109, "9/03" 130,
"10/03" 151, "11/03" 174, "12/03" 193, "1/04" 215, "2/04" 235)
set ylabel "volume (0000)" offset 1
plot 'finance.dat' using 0:($6/10000) notitle with impulses lt 3, \
'finance.dat' using 0:($7/10000) notitle with lines lt 1
It is important to note that right clicking on a gpng file and selecting "Save As" also invokes the gnuplot_gpng function. As a result, instead of downloading the script, the PNG image is downloaded instead.
Conclusion
This project has been on the back burner for quite some time because my original estimates for developing it were large due to a misguided perception of complexity. In actuality, the project turned out to be quite easy.
gnuplot can plot just about anything, and is the ultimate tool for plotting scientific data. I look forward to using it in this form with other natively built solutions.
For those who might be concerned with using Snorkel, it has been in development and testing for the past couple of years, and is a good alternative to other application/embedded server solutions. If you have questions on using the Snorkel SDK, obtaining Snorkel SSL binaries, additional platform support, or enhancement requests, feel free to contact me at wcapers64@gmail.com. Due to unexposed elements within the Snorkel runtime, which are part of a much larger yet to be released project that is currently under development, it is a closed source solution. This is because I have yet to determine whether we will offer the solution provided by the project for free. However, as a CodeProject member, I understand the value of giving back to the community and will continue to offer the basic Snorkel binaries free of charge so that others can benefit from its ease of use and performance. Visit us at http://snorkelembedded.webs.com/ for the latest versions of the Snorkel SDK.
