Introduction
Florian Rappl, a 28-year-old PhD student at the University
of Regensburg in Germany, won the Windows* 8 & Ultrabook™ App Innovation
Contest for creating Sumerics, an app that performs a complex analysis of
Ultrabook™ device sensor data and displays the results in visually compelling
2D and 3D graphs. Built using existing frameworks and languages—specifically
Windows* Presentation Foundation (WPF) and C#—Sumerics takes full advantage of the
sensor and touch screen capabilities of Ultrabook devices running Microsoft Windows
8. While
Rappl chose the well-established WPF for rendering the user interface, he relied
on the relatively new open-source library MahApps.Metro to give Sumerics’ UI
its modern, touch-friendly look and feel.
Sumerics is available on the Intel
AppUp® center.
Figure 1. A simple 3D
plot rendered by Sumerics.
About the Sumerics App
Rappl designed Sumerics with two goals in mind: First, he
wanted to present an intuitive touch-enabled user interface for visualizing
complex data series gathered either from various static files, such as CSV, or
from real-time dynamic data from Ultrabook device sensors. Second, he wanted to
make it easy for users to directly access these data series as inputs in
computational analysis. To enable this analysis, Rappl exposed all sensors as
functions for displaying and analyzing data, a design decision that allows users
to gather information about the current states of the Ultrabook device sensors
for further exploration. Rappl said anyone who deals with data visualization might
benefit from the app. For example, Sumerics could be used in an advanced course
on acoustical physics to analyze words spoken into the Ultrabook device microphone
through the use of Sumerics’ fast Fourier transform algorithm, enabling users
to obtain a visual representation of the frequencies for certain words by
specific speakers.
Figure 2. The Sumerics "sensors"
tab, where instant plots for most sensors can be created. By default this tab
is activated and gives instant information about sensors, without the need for
users to type a command.
Using the Ultrabook device keyboard, attached mouse, or
other peripheral, a Sumerics user can input and edit data, similar to using
MATLAB*, a high-level language and interactive environment for numerical
computation, visualization, and programming. However, because everything is
designed with touch in mind, the graphs that Sumerics displays can be rotated,
translated, scaled, saved, and printed using on-screen touch commands. In this
way, Sumerics takes advantage of the power and flexibility of Ultrabook devices,
which Rappl said provides the killer combination of a tablet’s portability and
a laptop’s high performance and ability to run standard software.
Challenges
Faced in Creating Sumerics
During the process of creating Sumerics, Rappl faced four
primary challenges.
- Deciding whether to develop a Windows Store app or an app to
run in Windows 8 Desktop
- Deciding whether to build the app using HTML5 or WPF
- Properly planning for touch and sensor implementation
- Building a math parser and deciding which plotting
libraries to use
The following sections describe how Rappl addressed each of
these challenges.
Navigating Windows 8
Windows 8 presents a choice to end users. They can navigate
the Ultrabook device in Desktop mode, which is similar to navigating PCs and
laptops running earlier versions of Windows. Or they can switch to one of
several flat, laterally scrolling tiles using the modern, touch-oriented UI that
is already familiar to smartphone and tablet users. Consequently, developers must
decide how end users will eventually access their apps.
Rappl chose to build Sumerics for the Windows 8 desktop,
primarily because he already had experience building Windows desktop apps,
particularly using C# with the WPF UI framework, which has been available for several
years. Accordingly, many open-source libraries are available, including ones
that make it easy to build touch-enabled UIs.
One disadvantage Rappl found is that, as is, the app coded
in C# with WPF for the desktop UI cannot be submitted to the Windows Store.
Apps in the Windows Store are generally coded in JavaScript* for UIs built
around HTML5 and CSS3, and Rappl said he would have used this approach if he
wanted to make sure Sumerics was distributed on Windows Store. (For now it’s
available on the Intel AppUp® center). Rappl noted that it is possible to use
the WinRT component library to write a JavaScript wrapper for a C#-based app,
which can then be distributed through the Windows Store. However, at the time Sumerics
was created, Rappl said that bugs were associated with this approach, and
indeed all Windows Store apps built with C#/VB/C++ and XAML, in part because of
difficulties in taking advantage of the WPF binding capabilities.
Using existing tools and technologies, Rappl built a
touch-enabled app that takes full advantage of Ultrabook device sensors and
runs smoothly on Ultrabook devices running Windows 8, providing user access and
navigation for Sumerics in the Desktop mode. Given the long and established
legacy of the Windows desktop, Sumerics has a channel to reach a potentially
large audience.
HTML5 versus WPF
For Rappl, a key part of navigating Windows 8 was deciding
whether to build the app using HTML5 or WPF. He concluded that HTML5 undeniably
has a bright future, which he expects to eventually be the de facto standard
for building platform-independent apps. However, since he was already well-versed
in WPF he build the UI around that graphical subsystem, which can be used to
build compelling Windows 8 desktop apps that are touch- and sensor-enabled. Rappl
said WPF introduces device-independent pixels, which offer a significant
advantage compared to the alternative of the much older Windows Forms, in use
since the early days of the Microsoft .NET Framework. He lauded the data-binding
capabilities of WPF, which also fully utilize the concepts of attached
properties and dependency properties.
Figure 3: The subplot
feature, which allow the user to view multiple plots from the same data series
simultaneously.
However, WPF has its own challenges. For example, like
Windows Forms, the controls can be heavy. But because everything (other than
images) is a vector and can run over DirectX*, the UI can be accelerated, thus
minimizing this problem. The controls can also be completely customized, and
developers can make full use of the ability to differentiate between the logical
and visual tree. Plus, the 3D capabilities in WPF provide a modest productivity
boost.
As for HTML5 (used in combination with CSS3 and JavaScript),
Rappl thinks the markup language will in all likelihood be central for apps
headed for a cross-platform touch screen future, which arguably has already
arrived. But issues remain. For example, it’s frustrating, he said, that while
HTML5 and C# are both cross-platform, the combination of the two is not. Attempting
to work around this constraint using C#-to-JavaScript, source-to-source
compilers can be confusing. And any move to JavaScript, a dynamic language,
means sacrificing the speed and responsiveness associated with C#, a static
language. So for Rappl, who wanted a maximally responsive app for his
sophisticated users, C# with WPF was the way to go.
Properly planning for touch and sensor implementation
Sumerics was the first app Rappl developed for touch screens.
Rappl’s three main takeaways: First, for ease of use, make each button at least
40x40 pixels—large enough to be clearly visible and easy to push or tap.
Second, use tooltips (the popup text enabled by placing the mouse cursor over
the element) as a complementary feature; always work to ensure that using the
program is obvious without having to read the text. Third, ensure the main features
of the app are accessible even without a keyboard—a simple enough rule to
adhere to when designing for tablets and smartphones, but more difficult to
follow when building a touch-aware app, specifically one that will be accessed
on the Windows 8 desktop on an Ultrabook device.
Rappl said dealing with sensors turned out to a secondary
part of his overall development work. This was because the main focus of
Sumerics is data visualization of data series that can come from any source, be
it an imported CSV file or an onboard accelerometer. Indeed, Sumerics works
well even if it’s used just to visualize data series imported from static
files. Also, because it was relatively straightforward to expose sensor data as
functions using the Windows
Sensor and Location Platform, Rappl was able to spend most of his time
working on how to best render compelling graphs and plots.
Figure 4. The Sumerics’
"interaction" tab, where users can type commands, get help, or view variables
currently available in the workspace (shown on the right). These variables are
used to store values from previous calculations. The tab presents an overview that
gives the user information about not only which variables are available, but
also what kind of value is stored in them, such as a scalar, matrix, or plot. A
graphical representation of the value is given when the user touches the
variable.
Building a math parser and deciding which plotting libraries to
use
At the heart of Sumerics is Yet Another Math Parser (YAMP*),
which takes a string of characters and transforms them into mathematical
expressions. When Rappl saw the Windows* 8 & Ultrabook™ App Innovation
Contest advertised on CodeProject, he decided to simultaneously build both the
Sumerics app and a math parser, one that he hoped could take advantage of the
features of C#. By releasing the YAMP code to the open-source community, Rappl tapped
their expertise during the code review process central to the open-source
world.
Here’s a high-level look at the Sumerics
architecture, where green represents third-party libraries, orange is the YAMP
external library, and blue represents the new libraries associated with Sumerics:
Sumerics obviously relies heavily on YAMP, which requires a
certain format for commands. For example, if a YAMP user wants to clear certain
variables, he or she must enter the following command:
clear("a", "b", "c")
With Sumerics the command is
simpler:
clear a b c
This
works because Sumerics has a command parser as well. Rappl added the
parser to make it as easy as possible for end users to call on YAMP functions. If the command parser finds a valid command, Sumerics
executes it. Otherwise the whole expression is passed to YAMP, which then parses
and interprets the expression.
Rappl
says writing the command parser was easy and required only an abstract base
class. Below, a code snippet shows how he employed reflection to register the
methods that the Sumerics command parser can use.
public static void RegisterCommands()
{
var lib = Assembly.GetExecutingAssembly();
var types = lib.GetTypes();
foreach (var type in types)
{
if (type.IsAbstract)
continue;
if (type.IsSubclassOf(typeof(YCommand)))
{
var cmd = type.GetConstructor(Type.EmptyTypes).Invoke(null) as YCommand;
commands.Add(cmd.Name.ToLower(), cmd);
}
}
}
A small
subset of the classes that make some functions available from the commands
parser is displayed below.
The command parser works with an arbitrary number of
arguments. The basic code is similar to that described by Rappl in his article
about YAMP on CodeProject.
To write the specific sensor functions in the YAMP plug-in, Rappl
first started with an abstract base class, which he extended to the various sensors.
Here’s how Rappl implemented the acc()
function:
using System;
using Windows.Devices.Sensors;
namespace YAMP.Sensors
{
[Description("Provides access to the acceleration sensor of an Intel UltraBook™.")]
[Kind("Sensor")]
public class AccFunction : SensorFunction
{
static Accelerometer sensor;
static AccFunction()
{
try
{
sensor = Accelerometer.GetDefault();
}
catch { }
}
protected override void InstallReadingChangedHandler()
{
if(sensor != null)
sensor.ReadingChanged += OnReadingChanged;
}
protected override void UninstallReadingChangedHandler()
{
if (sensor != null)
sensor.ReadingChanged -= OnReadingChanged;
}
void OnReadingChanged(Accelerometer sender, AccelerometerReadingChangedEventArgs args)
{
RaiseReadingChanged(args.Reading);
}
[Description("Retrieves acceleration in (X, Y, Z)-direction in units of g. Hence usually (no movement) the returned vector will be (0, 0, 1).")]
[ExampleAttribute("acc()", "Returns a 3x1 matrix of accelerations in the x, y and z directions.")]
public MatrixValue Function()
{
return new MatrixValue(Acceleration);
}
public static double[] Acceleration
{
get
{
if (sensor == null)
return new double[3];
var acc = sensor.GetCurrentReading();
return new double[] { acc.AccelerationX, acc.AccelerationY, acc.AccelerationZ };
}
}
}
}
Note how Rappl provided many static functions, which allows
direct access from Sumerics (for live data of the sensors), without creating an
explicit instance or having YAMP interpret a fixed expression. The sensor
reading AccelerometerReading is included in a nested class. If there is no
sensor data, a null value is returned, though YAMP always returns data—specifically
scalars and matrices with correct dimensions but without any values (so every
value is 0).
Because Sumerics focuses on data visualization, it was
critical to choose the appropriate libraries for displaying 2D and 3D plots. In
the end the decision was between two open-source libraries: IronPlot and
OxyPlot. IronPlot, which has the ability to render 3D plots, provides a
suitable plotting package for IronPython (without using Mathplotlib, which
would have added a dependency on Python and a dependency on the communication
between C# and Python). OxyPlot is a platform-independent plotting package that
includes a WPF implementation. OxyPlot, according to Rappl, is well documented,
extensible, and full of great features, including the ability to annotate and
track plots.
For 3D plotting, Rappl created his own library, albeit one
built around OxyPlot. This library allowed for the rendering of not only rectangles,
lines, and more from a data series, but also a complete image. This rendering
led to a demonstrable increase in performance and made it possible to plot heat
maps and other complex graphics that display well on the various Ultrabook device
screens available today. (Several Ultrabook devices have screen resolutions of up
to 1600x900.)
Helpful Resources
Rappl read the Intel articles that focus on using Ultrabook
device sensors, including articles available at http://software.intel.com/en-us/articles/ultrabook-touch-and-sensor-resources.
He also reviewed specifications for touch-aware apps (not just touch-enabled
apps) and consulted external resources, including those on CodeProject. Additionally, Rappl relied on the Intel
AppUp® Developer certification tool for creating simple MSI setup files.
More detail about Rappl’s work on Sumerics is described in an article he wrote
for CodeProject: http://www.codeproject.com/Articles/472698/Sumerics.
Rappl has also posted several short videos of himself using Sumerics, including
one showing how to plot data captured from the Ultrabook device inclinometer,
gyrometer, and compass, on his YouTube* channel: http://www.youtube.com/user/FlorianRappl.
Things to Consider
Rappl considers HTML5 to be the future for touch screen UIs
and, with the experience he gained from creating Sumerics, stated that if he were
starting over he would likely use HTML5 instead of WPF, particularly if he
wanted to distribute his app on the Windows Store.
Conclusion
Sumerics takes data imported from static files or gathered
from Ultrabook’s many onboard sensors and produces compelling 2D and 3D graphs
that can be easily manipulated via the touch screen. This sort of advanced data
visualization might prove useful in various environments, from engineering test
labs at established companies to high school physics classrooms. (In a basic
course covering classical mechanics, students could set up an experiment to let
an Ultrabook fall from a reasonably tall building and then use Sumerics to
review data from the accelerometer to see if the device achieved
weightlessness.) Despite the relative newness of the Ultrabook device hardware
and Windows 8 operating system, Rappl built the award-winning app in a matter
of months, working around other commitments, mostly using well-established
languages and frameworks (C# and WPF) and freely available libraries.
Rappl concludes that Ultrabook devices running Windows 8
represent a relatively new category of devices, one marked by the portability
and ease-of-use of touch screen tablets along with the power of a full-featured
laptop. But his experience building a compelling app that evokes comparisons to
industry-leading MATLAB-based products demonstrates that developers well-versed
in established Microsoft technologies, particularly C# and WPF, can build a
compelling touch screen app that takes full advantage of the Ultrabook device’s
many features—including its various sensors—while staying in the familiar
territory of the Microsoft desktop, albeit one that’s now touch-enabled.
About Florian Rappl
Ostensibly Florian Rappl’s main focus is writing his dissertation—the
weighty working title is "Feynman Diagram Sampling for Quantum
Chromodynamics"—in the field of theoretical particle physics. However, he's
also active in computing in a way that far exceeds the work of most physics
grad students. Currently he teaches graduate level courses in C# programming
and in designing Web applications with HTML5, CSS3 and JavaScript. He's
involved in several high-performance computing projects at the University of Regensburg,
including one attempting to build a supercomputer with Intel® Many Integrated
Core Architecture chips. He's an MVP at CodeProject, one of the more thriving
developer communities on the Web. It turns out that winning the Windows* 8
& Ultrabook™ App Innovation Contest is just the latest in a long string
of accomplishments for Rappl, though in addition to the acclaim, this
accomplishment also netted him more than USD 30,000 in prize money plus a new
Ultrabook device.
Portions of
this document are used with permission and copyright 2012 by CodeProject. Intel
does not make any representations or warranties whatsoever regarding quality,
reliability, functionality, or compatibility of third-party vendors and their
devices. For optimization information, see software.Intel.com/en-us/articles/optimization-notice/. All products, dates, and plans are
based on current expectations and subject to change without notice. Intel, the
Intel logo, Intel AppUp, Intel Atom, the Intel Inside logo, and
Ultrabook, are trademarks of Intel Corporation in the U.S. and/or other
countries. *Other names and brands may be claimed as the property of
others. Copyright © 2013. Intel Corporation. All rights reserved.