Introduction
In this article, we will show you how to build a web application that can detect various types of gestures using the Intel® RealSense™ SDK and front facing (F200) camera. Gesture recognition will give users of your application another innovative means for navigation and interface interaction. You will need basic knowledge of HTML, JavaScript*, and jQuery in order to complete this tutorial.
Hardware Requirements
- 4th generation (or later) Intel® CoreTM processor
- 150 MB free hard disk space
- 4 GB RAM
- Intel® RealSense™ camera (F200)
- Available USB3 port for the Intel RealSense camera (or dedicated connection for integrated camera)
Software Requirements
- Microsoft Windows* 8.1 (or later)
- A web browser such as Microsoft Internet Explorer*, Mozilla Firefox*, or Google Chrome*
- The Intel RealSense Depth Camera Manager (DCM) for the F200, which includes the camera driver and service, and the Intel RealSense SDK. Go here to download components.
- The Intel RealSense SDK Web Runtime. Currently, the best way to get this is to run one of the SDK’s JavaScript samples, which can be found in the SDK install directory. The default location is C:\Program Files (x86)\Intel\RSSDK\framework\JavaScript. The sample will detect that the web runtime is not installed, and prompt you to install it.
Setup
Please make sure that you complete the following steps before proceeding further.
- Plug your F200 camera into a USB3 port on your computer system.
- Install the DCM.
- Install the SDK.
- Install the Web Runtime.
- After installing the components, navigate to the location where you installed the SDK (we’ll use the default path):
C:\Program Files (x86)\Intel\RSSDK\framework\common\JavaScript
You should see a file called realsense.js. Please copy that file into a separate folder. We will be using it in this tutorial. For more information on deploying JavaScript applications using the SDK, click here.
Code Overview
For this tutorial, we will be using the sample code outlined below. This simple web application displays the names of gestures as they are detected by the camera. Please copy the entire code below into a new HTML file and save this file into the same folder as the realsense.js file. Alternatively, you can download the complete web application by clicking on the code sample link at the top of the article. We will go over the code in detail in the next section.
The Intel RealSense SDK relies heavily on the Promise object. If you are not familiar with JavaScript promises, please refer to this documentation for a quick overview and an API reference.
Refer to the Intel RealSense SDK documentation to find more detail about SDK functions referenced in this code sample. The SDK is online, as well as in the doc directory of your local SDK install.
<html>
<head>
<title>RealSense Sample Gesture Detection App</title>
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script>
<script type="text/javascript" src="https://autobahn.s3.amazonaws.com/autobahnjs/latest/autobahn.min.jgz"></script>
<script type="text/javascript" src="https://www.promisejs.org/polyfills/promise-6.1.0.js"></script>
<script type="text/javascript" src="realsense.js"></script>
<script>
var sense, hand_module, hand_config
var rs = intel.realsense
function DetectPlatform() {
rs.SenseManager.detectPlatform([ if (info.nextStep == Start()
}
else if (info.nextStep == $( $( }
else if (info.nextStep == $( }
else if (info.nextStep == $( }
}).catch(function (error) {
$( })
}
function Start() {
rs.SenseManager.createInstance().then(function (instance) {
sense = instance
return rs.hand.HandModule.activate(sense)
}).then(function (instance) {
hand_module = instance
hand_module.onFrameProcessed = onHandData
return sense.init()
}).then(function (result) {
return hand_module.createActiveConfiguration()
}).then(function (result) {
hand_config = result
hand_config.allAlerts = true
hand_config.allGestures = true
return hand_config.applyChanges()
}).then(function (result) {
return hand_config.release()
}).then(function (result) {
sense.captureManager.queryImageSize(rs.StreamType.STREAM_TYPE_DEPTH)
return sense.streamFrames()
}).catch(function (error) {
console.log(error)
})
}
function onHandData(sender, data) {
for (g = 0; g < data.firedGestureData.length; g++) {
$( }
}
$(document).ready(DetectPlatform)
</script>
</head>
<body>
<div id="info-area"></div>
<div id="gesture-area"></div>
</body>
</html>
The screenshot below is what the app looks like when you run it and present different types of gestures to the camera.
Detecting the Intel® RealSense™ Camera on the System
Before we can use the camera for gesture detection, we need to see if our system is ready for capture. We use the detectPlatform function for this purpose. The function takes two parameters: the first is an array of runtimes that the application will use and the second is an array of cameras that the application will work with. We pass in ['hand'] as the first argument since we will be working with just the hand module and ['front'] as the second argument since we will only be using the F200 camera.
The function returns an info object with a nextStep property. Depending on the value that we get, we can determine if the camera is ready for usage. If it is, we call the Start function to begin gesture detection. Otherwise, we output an appropriate message based on the string we receive back from the platform.
If there were any errors during this process, we output them to the screen.
rs.SenseManager.detectPlatform(['hand'], ['front']).then(function (info) {
if (info.nextStep == 'ready') {
Start()
}
else if (info.nextStep == 'unsupported') {
$('#info-area').append('<b> Platform is not supported for Intel(R) RealSense(TM) SDK: </b>')
$('#info-area').append('<b> either you are missing the required camera, or your OS and browser are not supported </b>')
}
else if (info.nextStep == 'driver') {
$('#info-area').append('<b> Please update your camera driver from your computer manufacturer </b>')
}
else if (info.nextStep == 'runtime') {
$('#info-area').append('<b> Please download the latest web runtime to run this app, located <a href="https://software.intel.com/en-us/realsense/webapp_setup_v6.exe">here</a> </b>')
}
}).catch(function (error) {
$('#info-area').append('Error detected: ' + JSON.stringify(error))
})
Setting Up the Camera for Gesture Detection
rs.SenseManager.createInstance().then(function (instance) {
sense = instance
return rs.hand.HandModule.activate(sense)
})
You need to follow a sequence of steps to set up the camera for gesture detection. First, create a new SenseManager instance and enable the camera to detect hand movement. The SenseManager is used to manage the camera pipeline.
To do this, we will call the createInstance function. The callback returns the instance that we just created, which we store in the sense variable for future use. We then call the activate function to enable the hand module, which we will need for gesture detection.
.then(function (instance) {
hand_module = instance
hand_module.onFrameProcessed = onHandData
return sense.init()
})
Next, we need to save the instance of the hand tracking module that was returned by the activate function into the hand_module variable. We then assign the onFrameProcessed property to our own custom callback function called onHandData whenever new frame data is available. Finally, we initialize the camera pipeline for processing by calling the Init function
.then(function (result) {
return hand_module.createActiveConfiguration()
})
To configure the hand tracking module for gesture detection, you have to create an active configuration instance. This is done by calling the createActiveConfiguration function.
.then(function (result) {
hand_config = result
hand_config.allAlerts = true
hand_config.allGestures = true
return hand_config.applyChanges()
})
The CreateActiveConfiguration function returns the instance of the configuration, which is stored in the hand_config variable. We then set the allAlerts property to true to enable all alert notifications. The alert notifications give us additional details such as the frame number, timestamp, and the hand identifier that triggered the alert. We also set the allGestures property to true, which is needed for gesture detection. Finally, we call the applyChanges function to apply all parameter changes to the hand tracking module. This makes the current configuration active.
.then(function (result) {
return hand_config.release()
})
We then call the release function to release the configuration.
.then(function (result) {
sense.captureManager.queryImageSize(rs.StreamType.STREAM_TYPE_DEPTH)
return sense.streamFrames()
}).catch(function (error) {
console.log(error)
})
Finally, the next sequence of functions sets up the camera to start streaming frames. When new frame data is available, the onHandData function will be invoked. If any errors were detected, we catch them and log all errors to the console.
The onHandData function
function onHandData(sender, data) {
for (g = 0; g < data.firedGestureData.length; g++) {
$('#gesture-area').append(data.firedGestureData[g].name + '<br />')
}
}
The onHandData callback is the main function where we check to see if a gesture has been detected. Remember this function is called whenever there is new hand data and that some of the data may or may not be gesture-related data. The function takes in two parameters, but we use only the data parameter. If gesture data is available, we iterate through the firedGestureData array and get the gesture name from the name property. Finally, we output the gesture name into the gesture-area div, which displays the gesture name on the web page.
Note that the camera remains on and continues to capture gesture data until you close the web page.
Conclusion
In this tutorial, we used the Intel RealSense SDK to create a simple web application that uses the F200 camera for gesture detection. We learned how to detect whether a camera is available on the system and how to set up the camera for gesture recognition. You could modify this example by checking for a specific gesture type (e.g., thumbsup or thumbsdown) using if statements and then writing code to handle that specific use case.
About the Author
Jimmy Wei is a software engineer and has been with Intel Corporation for over 9 years.
Related Resources
Notices
No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.
Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.
