Tilting tablets have replaced spinning globes as a way to learn geography. SensiGator, the Education category winner of the Intel® App Innovation Contest (AIC 2013), is designed in part to expand the imagination of students learning basic geography. Built for the Lenovo ThinkPad* Tablet 2 and Ultrabook™ 2 in 1, running Windows* 8.1, SensiGator thoroughly takes advantage of the various sensors and input modalities of the tablet platform—including the vibrant 10.1-inch HD touch-enabled display and the many options (Bluetooth*, USB, and mini-HDMI*) for connecting peripherals such as keyboards and monitors. These peripherals proved important because SensiGator app developer Bryan Brown did all his development and testing work on the tablet itself—a first for him as a programmer. Thanks to the Intel® Atom™ processor (dual core 4-thread, up to 1.80 GHZ), he reported that builds weren’t much slower than they would have been on a high-end PC.
For Brown, whose BlinkTalk app won a similar Intel App Innovation contest in 2012, SensiGator provided new programming challenges, including how to use the Windows Runtime (also called WinRT) API for a Windows 8 desktop app, how to simulate keyboard and mouse actions on the touch screen, and, with the goal of providing a streamlined user experience, how to navigate the map exclusively with the onboard sensors, specifically the inclinometer and gyrometer.
Video 1: SensiGator’s Intel AIC 2013 video submission which describes SensiGator's sensor-based navigation for Bing Maps.
Making the Most of the Tablet’s Screen and Sensors
The ThinkPad Tablet 2’s 10.1-inch HD screen renders Bing* Maps beautifully. The idea is to present an image that the user might see when peering down at Earth from a plane’s window, enhanced of course with the now-expected labeled network of freeways, thoroughfares, and towns, in this case provided by Bing. When the user tips the right side of the tablet down and the inclinometer detects the tilt, he or she appears to be flying east. The landscape scrolls by smoothly, largely because the tablet’s processing power allows for fast rendering of images. (Microsoft's Bing Maps Tile System, which pre-renders the map at many different levels of detail and then cuts each map into tiles for quick retrieval and display, is also a player in enabling this smooth responsiveness.)
Zooming in is simply a matter of quickly torqueing the tablet in a clockwise direction. Once the gyrometer registers an angular velocity of more than 150 degrees per second, what were small features on the map suddenly loom large on the screen as the image zooms in (see Figure 1). A few flicks in the opposite direction zooms the image out until the screen shows only continent-level details (see Figure 2).
Figure 1: Bing Maps zoomed in using SensiGator
Figure 2: Continent-level resolution in Bing Maps using SensiGator
Tapping an on-screen button centers the map on the user’s location on Earth; using data from the geolocator sensor (see Figure 3). SensiGator has a semi-transparent heads-up display that provides on-screen buttons for touch navigation and gives precise latitude and longitude information both for the midpoint of the on-screen map and the user’s actual location (see Figure 4). The tablet’s stylus can be used to insert virtual pushpins. And given the many ports on the Lenovo tablet, it’s possible to connect via USB or mini-HDMI to an external monitor, keyboard, and mouse. Brown thinks that—except for the fun of spinning a classroom orb blurringly fast and imagining its inhabitants holding on for dear life—in terms of user experience, SensiGator hands-down beats the old-fashioned globe, which probably dates back to the third-century B.C.
Figure 3: SensiGator screenshot of the current Los Angeles region
Figure 4: The SensiGator heads-up display
Tweaking the Default Multi-Touch Capability to Ensure a Good User Experience
Building something that’s easy enough for a child to use is of course devilishly difficult. Contest rules also proved tricky, as entrants had to create Windows 8 desktop apps, so it made more sense for Brown to use the Bing Maps Windows Presentation Foundation (WPF) SDK instead of Bing Maps for Windows Store Apps.
"This presented some challenges, mainly because the WPF SDK exhibits some quirky behavior when used in a Windows 8 desktop app. This behavior is especially evident when attempting multi-touch operations such as zoom-in and zoom-out using pinch gestures," Brown wrote in an August 2013 article for CodeProject* introducing SensiGator to contest judges.
After browsing a detailed Microsoft Developer Network (MSDN) forum discussion about the issue, Brown decided to override Bing Maps’ default behavior, inhibiting multi-touch and pinch-zoom controls using the TouchDown event. The code snippet in Snippet 1 shows how he did it:
private void Map_TouchDown(object sender, TouchEventArgs e)
{
e.Handled = true;
}
Snippet 1: Overriding Bing Maps’ default multi-touch feature
Brown built SensiGator as a C#/WPF app, using Visual Studio* 2012, the first version to support .NET 4.5. Reading up on the Windows 8 WinRT API in an article on Intel® Developer Zone, he learned that Visual Studio 2012 doesn't give access to WinRT from a regular desktop C# project by default; a developer first needs to declare that his or her project is targeting Windows 8. "Once this is done," said Brown, "you can reference the Windows.Devices.Sensors and Windows.Devices.Geolocation assemblies, which are required to access sensor data."
The SensiGator app references the SensorHub class library that Brown created to wrap the Windows.Devices.Sensors and Windows.Devices.Geolocation namespace APIs. A SensorClass object is instantiated in the main app for accessing tablet sensor data and spawns three continuous threads for data processing, user interface interactions (using Dispatcher.BeginInvoke), and acquiring Geolocation information. The data processing and UI threads update every 50 milliseconds, while the GeoThread updates every five seconds.
"I originally intended to instantiate a GeoLocator object at the main application level and subscribe to an event handler for location-change events, but I ultimately decided to use GetGeopositionAsync() and wrap it in a public method within my SensorClass," said Brown. "This is why Geolocation information is polled in a thread as opposed to being event-driven, but the overhead is minimal as the user's discernible geographic location is not changing rapidly with respect to the five-second polling rate."
Figure 5: SensiGator architectural overview
Simulating Keyboard and Mouse Actions
For the zoom and navigation buttons, Brown wanted to replicate the action of a keyboard so he used the Windows Input Simulator DLL, a C# SendInput wrapper that simulates keyboard and mouse actions.
Brown used the open source third-party InputSimulator DLL for some of the navigation routines to provide an easy way to access the Win32* SendInput method. After referencing the InputSimulator DLL and adding the WindowsInput namespace to the project, he was able to make calls to its SimulateKeyPress method that takes various virtual key codes (ADD, SUBTRACT, LEFT, RIGHT, and so on) as a parameter. In the case of map zooming, he created a ZoomIn() method that gets called when the tablet's angular velocity exceeds 150 degrees per second in the z-axis and also when the user taps the Zoom In navigation button (see Snippet 2).
private void btnZoomIn_Click(object sender, RoutedEventArgs e)
{
if (myMap.ZoomLevel < ZOOM_MAX)
{
myMap.Focus();
ZoomIn();
}
}
.
.
.
public void ZoomIn()
{
InputSimulator.SimulateKeyPress(VirtualKeyCode.ADD);
}
Snippet 2:Code that enables zooming in with the on-screen button
Ever the tinkerer, Brown had done work with Bing Maps prior to AIC 2013. His NuiMaps app was one of four of his entries in the Intel® Perceptual Computing Challenge, which wrapped up at the end of 2013. In this earlier work with the Bing Maps control, he used calls to the InputSimulator's SimulateKeyPress method to navigate east, west, north, and south (for example, "InputSimulator.SimulateKeyPress(VirtualKeyCode.RIGHT)"). The advantage of this approach is that the Bing Maps control automatically adjusts for zoom-level resolution when navigation is done through a keyboard's arrow keys. (See the MSDN article Understanding Scale and Resolution for more information on zoom-level resolution.)
"The downside to this is that motion appears choppy when you are simulating multiple, rapid key presses," said Brown. "In SensiGator, I wanted the motion to be more fluid, so instead of simulating keystrokes, I derived a non-linear equation to control the translation speed at different map Zoom Levels and navigated by re-centering the map on the screen." Snippet 3 shows how he did it.
cf = Sensefactor / Math.Pow(10, (myMap.ZoomLevel * 0.25 + 0.5));
.
.
myMap.Center = new Location(myMap.Center.Latitude + cf,
myMap.Center.Longitude);
Snippet 3:Controlling translation speed at different map Zoom Levels
The Sensor Nav panel contains a sensitivity slider that can be used to increase or decrease the speed of map travel when navigating by sensor data. The center function (cf) shown in Figure 8 uses the slider's value (that is, Sensefactor = sliderSense.Value). Brown notes that his equation was not derived from the information provided in the aforementioned MSDN article, but rather from empirical data.
Which End is Up? (Solving the Issue of Orientation)
Another challenge was detecting whether the tablet is in landscape or portrait orientation, and then handling map navigation accordingly. SensiGator is intended to operate with the tablet positioned face-up in landscape orientation, but it also works in portrait orientation. However, when the tablet's orientation is changed, the app needs to redefine how the tablet's inclinometer data is used for navigating the map. The code snippet in Snippet 4 shows how Brown’s app first detects the tablet's orientation and then redefines the x- and y-axes:
LocationRect bounds = myMap.BoundingRectangle;
MapWidth = bounds.Width;
MapHeight = bounds.Height;
.
.
.
if (MapHeight > MapWidth)
{
lblOrientation.Content = "Portrait Orientation";
XAxis = -Pitch;
YAxis = Roll;
}
else
{
lblOrientation.Content = "Landscape Orientation";
XAxis = Roll;
YAxis = Pitch;
}
Snippet 4: Redefining the X- and Y-axes if the tablet orientation changes
Brown did all the coding on the tablet itself, which made testing of the features easier; the conventional wisdom that "tablets are best for consuming rather than creating content" is not altogether accurate. He said that not having to download the app to the tablet for testing purposes probably improved his ability to finish the app in just six weeks. Not that the experience was without its drawbacks: for example, builds took longer than they would have on a developer-class machine.
Brown said that if he works on another app that is not too code-heavy, he might try developing on a tablet again, though he would look at alternative integrated-development environments that might make it possible to also work off the tablet. The key, he said, is to find a good Windows tablet emulator that accounts for all the sensors and that can run on the desktop, much like emulators that exist for those building tablet and phone Android* and iOS* apps.
Brown, a pro with years of experience, confessed that, "I'm not sure where the tools are right now when it comes to Visual Studio or whether proper emulator environments even exist." That comment alone hints at how wide open the possibilities are for development on Windows 8 tablets and touch screens.
Like all developers, especially those working on tight contest deadlines, Brown can list several features he would like to have tried implementing given more time. Currently, SensiGator is limited to Bing Maps’ aerial view; he’d like to experiment with the road and bird’s-eye views in the future. At this year’s Mobile World Congress, where Brown demoed SensiGator as a guest in Intel’s booth, one passerby suggested that the steeper the tilt of the tablet, the faster the user should appear to fly over the map.
About Bryan Brown
For more than 20 years, Bryan Brown has worked as a software engineer for various Los Angeles-area companies. He currently works at a company that makes devices used in spectroscopy and chromatography. He spends nights and weekends writing code for free, often working on software related to human wellness. This work is serious enough to warrant a nonprofit organization called Human-Machine Technologies, founded by Bryan in 2012.
A casual follower of AIC 2013 might think that Brown burst on the hobbyist coding scene, maybe getting a bit lucky with SensiGator. However, Brown is hardly a one-hit wonder. His earlier BlinkTalk app offers a way for locked-in patients suffering paralysis of voluntary muscles below the eyes to communicate verbally. Brown also won a second-place award in Phase 1 of the 2013 Intel Perceptual Computing Challenge for his NuiLogix project, an example of perceptual computing for network-based device control.
Another of his projects, the PerC Robot Arm Controller, nabbed a pioneer award in Phase 2 of the Intel Perceptual Computing Challenge. (Brown’s video shows the robot arm responding to voice commands to pick up, move, and plunk chess pieces down in a glass; robots that mix a favorite cocktail or at least tidy up after your kids can’t be far behind.) Duly impressed with Brown and his apps, Intel offered Brown an exhibitor pass to this year’s Mobile World Congress 2014 where he showed off SensiGator in Intel’s booth.
Talking with Brown brings to mind the 10,000-hour rule, which Malcolm Gladwell highlighted in his 2008 book Outliers, excerpted in The New Yorker in October 2008. The gist of the rule: Mastery in any field takes about 10,000 hours of practice. Gladwell pointed out that most geniuses who appear to burst on the scene have labored for years before being discovered.
Gladwell’s observation about the importance of logging time aside, embracing change may well be just as important to success in any endeavor, including writing code. Brown seems to say as much in his parting shot to other developers who might want to learn from his example:
"My advice is to never lock yourself into a particular language, platform, or tool and try to solve every problem with the same skill set. The most interesting part of this contest was that it mandated developing a Windows Desktop app to target a Windows 8 tablet. This likely caused some consternation for developers who would have otherwise gravitated to a native Windows 8 app development approach, but that's part of the fun—and reward—of moving outside your comfort zone."
Resources
Intel helps developers solve the types of challenges Brown faced to build innovative, flexible applications that take full advantage of the latest Windows 8 and Intel technologies across multiple devices. Intel Developer Zone, which Brown mined for information as he worked, provides world-class knowledge, tools, and support for Windows 8 apps, both cross-platform and desktop. Intel Developer Zone can help developers who, like Brown, are interested in creating a 2-in-1 application supporting multiple input types, touch/stylus as well as keyboard/mouse.
Brown also used these resources:
- Windows Input Simulator, open source C# SendInput wrapper for simulating keyboard and mouse
- "Odd jumping behavior with multitouch (WPF)," MSDN forum discussion on dealing with buggy pinch and zoom multi-touch gestures in Bing Maps for desktop
Related Articles
Touch Developer Guide for Ultra Mobile Devices: https://software.intel.com/en-us/articles/ultrabook-device-and-tablet-windows-touch-developer-guide
Ultrabook and Tablet Windows* 8 Sensors Development Guide: http://software.intel.com/en-us/articles/ultrabook-and-tablet-windows-8-sensors-development-guide
Touch and Sensors: http://software.intel.com/en-us/touch-and-sensors
Intel® Developer Zone offers tools and how-to information for cross-platform app development, platform and technology information, code samples, and peer expertise to help developers innovate and succeed. Join our communities for the Internet of Things, Android*, Intel® RealSense™ Technology and Windows* to download tools, access dev kits, share ideas with like-minded developers, and participate in hackathons, contests, roadshows, and local events.
