Introduction
With augmented reality (AR) reaching smartphones, tablets, wearables (such as Google Glass*), and other platforms, the market is ripe for an AR development explosion across every conceivable application niche. Developer Serhiy Posokhin and his wife Antonina Posokhina, a designer by trade, recognized the potential of this technology in the world of jewelry and pursued its potential through the Intel® App Innovation Contest 2013 (AIC 2013). The resulting app, AR Jewellery, a program that lets users visualize what a ring would look like on their hands in real time, won the contest’s Retail category and now points the way for other developers to explore and expand in this massive market space.
AR Jewellery: Formation and Function
Designed specifically to leverage a tablet’s integrated cameras and touch capabilities, AR Jewellery takes live camera input and superimposes a 3D model—of a selected ring, in this case—on top of a marker. The marker is a small, black-and-white glyph in the middle of a narrow strip that the user prints and cuts out. The user places the strip around his or her finger so that the glyph is in a ring’s usual position. AR Jewellery recognizes the glyph, places the virtual ring in the glyph’s position, and keeps it there, moving and rotating as needed along with the user’s movements. In effect, the user is "trying on" the ring, or a realistic digital facsimile of it.
Figure 1: AR Jewellery uses a glyph to help with 3D object tracking in real time.
The touch-based application can use either front- or rear-facing tablet cameras and displays the live stream in a window surrounded by icons for various ring designs and program functions. Dual-camera capability was a key element in Posokhin’s design concept, because he noticed that when women try on jewelry in the mall, they usually hold the ring on their hand at arm’s length and then up near their face to see it in a mirror. The dual-camera approach suits this dual-view paradigm.
When users like a ring they have virtually tried on, they can capture a snapshot and post it directly to Facebook. "I imagined how convenient it would be if women were able to try on jewelry without leaving home," said Posokhin. "They could share photos, drop hints about a desired gift, or reserve a product online."
Figure 2: Users can "try on" different rings contained in the program’s library of 3D-modeled designs.
Challenges Addressed During Development
Posokhin devoted considerable time to studying AR technology before setting to work on his application. He researched the various available libraries and frameworks and had to rethink his usual approach to pattern recognition and 3D graphics. As a result of his advance planning, Posokhin encountered few development challenges but admitted that he did encounter some cosmetic challenges.
Posokhin’s greatest challenge was purely aesthetic. The paper glyph that users place on their finger is clearly noticeable, and it distracts from enjoying a seamless AR experience. Initially, he tried to have AR Jewellery operate without a marker, but his attempts to have the software accurately recognize and track a finger failed. A glyph-based approach, on the other hand, has been extensively developed and documented for many years. (To learn more on the subject, Posokhin recommends reading Andrew Kirillov’s extensive AForge.Net article.)
Posokhin plans to eliminate the need for a tracking glyph in AR Jewellery, and he also plans to broaden its catalog. He is quite optimistic about the Intel® RealSense™ 3D camera and feels that having the camera and capabilities it provides will allow him to have the higher accuracy necessary to track fingers and dispense with using a glyph.
"At Mobile World Congress in 2014," he noted, "I talked to Intel engineers who were demonstrating a [Intel] RealSense [3D] camera. I asked whether it is possible to recognize every finger separately to calculate the coordinates of a 3D model and put the ring in the correct place of the ring finger. They explained that this should be possible in the new Intel® RealSense™ SDK!"
Another cosmetic issue centered on AR Jewellery’s user interface. For a cleaner, more engaging look, Posokhin’s wife, Antonina, wanted the live camera view to fill the tablet screen rather than sit inside of a wide, white border filled with icons. However, to make the contest deadline, Posokhin had to leave this feature for a future version.
Figure 3: AR Jewellery leverages both tablet cameras to accommodate how women generally like
to observe their prospective purchases.
Resources Used
As an AIC 2013 finalist, Posokhin was one of 300 developers to receive a Lenovo ThinkPad* Tablet 2 from Intel. This award allowed him to get a feel for how the app would handle in a realistic setting, because it’s the sort of tablet one might find among a higher-end clientele amenable to using technology to assist with their luxury shopping. The Tablet 2 offers several features in line with next-generation mobile apps, including noise-canceling array microphones, multi-touch 10-inch IPS display, the Windows* 8 OS, and a four-thread, 1.80-GHz Intel® Atom® processor Z2760 with integrated Intel® HD Graphics with an SGX545 core. The tablet’s two cameras were especially useful for AR Jewellery, with the front offering 2MP resolution and the rear providing 8MP. The higher-quality cameras help AR Jewellery to deliver superior results to image-conscious users. Additionally, the Tablet 2 provided a capable test environment for an AR mobile application.
Posokhin is a strong believer in helping the programming community. As such, he was happy to provide code snippets from AR Jewellery that focus on some of the software’s key features. Some of these snippets were rather lengthy, but two jumped out as being short, yet intriguing.
The snippet in Figure 4 describes how AR Jewellery stores product information in an SQLite* database:
Figure 4: AR Jewellery uses this code to enter new product models, integrating various metadata attributes.
The ability to capture and share photos from the UI to Facebook utilizes the code in Figure 5:
Figure 5: Posokhin knows that AR Jewellery users want to share what they see with their friends
and family. This Facebook functionality makes the task easy.
In creating AR Jewellery, Posokhin experimented with several tools, including:
In the end, though, the only tool he relied on for positional AR tracking was Glyph Recognition and Tracking Framework. "This library allowed me to implement almost everything I needed," noted Posokhin.
Posokhin also recommends reading the article "Windows 8* Store vs Desktop App Development" and watching the "HowTo [sic] Create 3D Blender Model for use in WPF" video series, both of which he found useful at different points in the AR Jewellery process.
Lessons Learned and Forward Thoughts
Posokhin suggests that, whenever possible, touch developers look for opportunities to replace screen taps with gestures—an intuitive approach. When representing a world of 3D objects, seek to let users manipulate those objects in three dimensions, not the flat plane of a touch screen. Of course, making gestures pervasive in AR apps requires many industry pieces to be in place, not the least of which is a widely adopted interface standard.
"Remember when browsers did not yet have a single standard?" asked Posokhin. "HTML pages in each browser would open in different ways. We all had lots of discomfort associated with this. Today, we have a similar situation with AR browsers. But as far as I am aware, with ARML 2.0 [Augmented Reality Markup Language], the leading developers of AR software already agree that a common set of gesture commands should be created. Once we have an agreed-upon set of UI controls, we can create new applications easier and faster."
The ARML 2.0 Standards Workgroup is governed by the Open Geospatial Consortium (OGC). The workgroup published its 2.0 specification in November 2012. Efforts to make the spec a world standard were pushed forward when AR heavyweights Layar, Metaio, and Wikitude demonstrated ARML 2.0 technology at the Mobile World Congress in Barcelona. Progress in ARML 2.0 adoption continues.
Resources
Intel developer tools and programs increasingly seek to nudge developers into such next-generation approaches to building applications and assist them in succeeding in a world brimming with sensors and free from screen-size constraints. Posokhin remains grateful for the opportunity that Intel App Innovation Contest 2013 presented. The contest is one of many avenues Intel promotes to help developers build powerful, forward-thinking applications able to take full advantage of the latest Windows 8 and Intel technologies across multiple device platforms. With the development tools he has available, Posokhin will be able to quickly optimize AR Jewellery for 2-in-1 Ultrabook™ devices, all-in-ones, and other form factors as well as tablets.
The seamless use of camera, touch, and keyboard/mouse input types was a major factor in AR Jewellery’s winning the Retail category of AIC 2013. The more apps that can embrace this sort of multi-modal paradigm, the more ready their developers will be for the coming era of perceptual computing.
About Serhiy Posokhin
Serhiy Posokhin is a technical director and co-owner at the IT services firm TEAM, Ltd. in the Ukraine, as well as the founder of ToniKa Design Studio with his wife, Antonina. Posokhin sees so much potential in the AR market that he created AR magix, a start-up company dedicated to actualizing that potential. The company’s first app, skewed toward girls who want to "try on" a "dress of beautiful summer flowers," can be tried in the Windows Store here. "Our mission is to delight people, using the magic of augmented reality," he noted.
Related Articles
Intel®Developer Zone offers tools and how-to information for cross-platform app development, platform and technology information, code samples, and peer expertise to help developers innovate and succeed. Join our communities for the Internet of Things, Android*, Intel® RealSense™ Technology and Windows* to download tools, access dev kits, share ideas with like-minded developers, and participate in hackathons, contests, roadshows, and local events.
Intel, the Intel logo, Intel Atom, Intel Core, Iris, and Ultrabook are trademarks of Intel Corporation in the U.S. and/or other countries.
*Other names and brands may be claimed as the property of others.
Copyright © 2014. Intel Corporation. All rights reserved.
