Leaping into Motion

What's the Fuss?

There has been a real buzz about Virtual reality (VR) and augmented reality (AR) technologies over the past few years. Major VR developments by the big tech companies such as Sony's PlayStation VR and Google's Daydream, are pushing VR into the mainstream. The unrelenting fever of Pokemon Go in 2016 charted the unwavering course for the future development of AR. Even more excitingly, Microsoft's HoloLens sets to bring the mix of virtual and augmented reality (Mixed Reality) into our real world may not seem so far-fetched after all.

While waiting for the fruits to ripe, aren't you curious how it is possible to control an object in your computer screen using your hands or fingers in the air. What better way to find out the answer than trying your hand at creating one, albeit a simple one.

This article will get you started on writing code to perform simple manipulations of web elements in the browser using your finger in the air via a sensor device called Leap Motion controller by Leap Motion, Inc.

Knowing Your Tool

There is an old Chinese proverb that says "工欲善其事,必先利其器", which I interpret as "one needs to know one's tool well in order to do a good job". So, you will start by getting to know the basics of your tool — the Leap Motion controller.

What's Leap Motion Controller?

It's a sensor device that detects and captures hand and finger motions in the air as input to VR/AR applications.

System Architecture

Besides the Leap Motion controller hardware, you also need to install the Leap Motion SDK software on the computer that interfaces with the Leap Motion controller.

The Leap Motion SDK runs as a background process on your computer. It receives motion tracking data from your hands and fingers in the real world via the USB connected Leap Motion controller.

The motion tracking data are presented to your application as a series of snapshots called frames. Each frame contains the coordinates, directions, and other information about the hands and fingers detected in that frame. Each frame is represented as a Frame object in Leap Motion APIs. The Frame object is essentially the root of Leap Motion's tracking model.

Your software application can then access the Frame object via one of the two APIs provided by the Leap Motion SDK. These two types of APIs are the Native Interface and the WebSocket interface.

Native Interface

The native interface is a dynamic library that you can use to create Leap-enabled desktop applications in a variety of languages and technologies: C#, C++, Java, Python, Objective-C, Unity, and Unreal.

WebSocket Interface

The WebSocket interface, on the other hand, allows you to create Leap-enabled web applications that work in conventional web browsers out of the box. It provides motion tracking data in the form of JSON formatted messages which are consumed by a JavaScript library, which in turn makes them available to your web applications as regular JavaScript objects for further processing.

Leap Motion Coordinates

The Leap Motion Controller provides right-handed coordinates in units of real world millimeters within the Leap Motion frame of reference, the origin (0, 0, 0) of which is the centre of the Leap Motion controller device.

Understandably, if your application is using a different coordinate system, you will have to make the necessary conversion in code to map the Leap coordinates to your application coordinates.

Interaction Box

The Leap Motion controller can detects and tracks the movement of your hand or finger only if it is within its field of view, an invisible inverted pyramid sitting on the device. To take away much of the guessing work, Leap Motion further provides a virtual Interaction Box to help your hand or finger stays in the range. An Interaction Box in Leap Motion defines a box-shaped region completely within the field of view of the Leap Motion controller. It is Leap Motion's way to assure the users that their hands or fingers will be tracked as long as they stay within this box.

Instead of using the raw dimensions measured in millimeters, coordinates in the Interaction Box are first normalized to a range between 0 and 1, such that the minimum value of the Interaction Box maps to 0 and the maximum value of the Interaction Box maps to 1. In the Interaction Box, the minimum normalized coordinates (0,0,0) is set to the bottom, left, back corner of the box, while maximum normalized coordinates (1,1,1) the top, right, front corner.

The InteractionBox class of Leap Motion API provides a normalizePoint() method that converts the real world coordinates of the hands or fingers detected in the Interaction Box to their normalized coordinates in the range between 0 and 1. These normalized coordinates are then multiplied by the maximum range of each axis in your application coordinate system to arrive at the application coordinates, making any necessary adjustment if your application coordinate system differs from that of the Leap Motion. Got it?

Making Things Happen

Having learned the basic mechanism of a Leap Motion controller, you are now ready to get your hands dirty in cooking up code that makes use of the motion tracking data received from the controller. Let's do it...

Setting the Stage

First thing first, you have to set up your computer so that it can interface with the Leap Motion controller. Follow this setup guide to download and install the desktop developer SDK for your machine. As part of the installation, you may be prompted to upgrade the display driver, just ignore it as it is not required for this exploration trip.

With the SDK installed, plug the Leap Motion controller into your computer via the USB.

Next, get ready an HTML page with the following code and save it as, say, index.html.

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Leaping into Motion</title>
<script src="https://js.leapmotion.com/leap-0.6.4.min.js"></script>
</head>
<body>
</body>
</html>

Included in the index.html page is the Leap Motion JavaScript library as shown:

<script src="https://js.leapmotion.com/leap-0.6.4.min.js"></script>

This Leap Motion JavaScript library receives motion tracking data from the WebSocket interface and make the data available to index.html for consumption by JavaScript code.

Getting Connected

With the Leap Motion JavaScript library included, you now have access to the Leap Motion API via the Leap global namespace. To start tracking your hands or fingers, you will call the loop() function of the Leap namespace to mediate the connection between your web application and the WebSocket interface, and to invoke a callback function that receives a Frame object at regular interval. Typically, this interval is set at 60 Frame objects per second. Each Frame object contains motion tracking data of hands or fingers detected by the Leap Motion controller at a particular instance. The code to implement this is as follows:

Leap.loop(function(frame){
  // Add code to process the frame of tracked data 
})

You will add it to index.html as shown:

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Leaping into Motion</title>
<script src="https://js.leapmotion.com/leap-0.6.4.min.js"></script>
</head>
<body>
<script>
Leap.loop(function(frame){
  // Add code to process the frame of tracked data
})
</script>
</body>
</html>

Creating a Dashboard

In index.html, create a HTML table, furnished with some CSS, for outputting the Frame object data received from the controller.

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Leaping into Motion</title>
<style>
th, td {
  min-width: 300px;
  text-align: left;
  vertical-align: top	
}
</style>
<script src="https://js.leapmotion.com/leap-0.6.4.min.js"></script>
</head>
<body>

<table>
  <tr>
    <th>Frame Data</th><th>Hand Data</th><th>Finger Data</th>
  </tr>
  <tr>
    <td id="frameData">Frame Data</td><td 
    id="handData">Hand Data</td><td id="fingerData">Finger Data</td>
  </tr>
</table>

<script>
Leap.loop(function(frame){
  // Add code to process the frame of tracked data
})
</script>
</body>
</html>

Deciphering the Frame

Each Frame object passed to the callback function of Leap.loop() is identified by an id and may contain Hand objects and Finger objects, among other things. The highlighted code gets the id of a Frame object and the respective numbers of Hand objects and Finger objects contained in that Frame object, and displays them in the browser.

<script>
var frameData = document.getElementById('frameData');

Leap.loop(function(frame){

  // Get and show frame data
  frameData.innerHTML = "Frame ID: " + frame.id  + "<br>"
            + "No of Hands: " + frame.hands.length + "<br>"
            + "No of Fingers: " + frame.fingers.length + "";

})
</script>

Deciphering the Hands

Each Hand object contained in the Frame object possesses a set of properties, such as id, hand type (left or right hand), palm position, grab strength, among other things. The highlighted code gets the id, type, palmPosition, grabStrength, and pinchStrength of each Hand object contained in the Frame object, and displays them in the browser.

<script>
var frameData = document.getElementById('frameData');
var handData = document.getElementById('handData');

Leap.loop(function(frame){

  // Get and show frame data
  frameData.innerHTML = "Frame ID: " + frame.id  + "<br>"
            + "No of Hands: " + frame.hands.length + "<br>"
            + "No of Fingers: " + frame.fingers.length + "";

  // Get and show hand data
  handData.innerHTML = "";
  for(var i = 0; i < frame.hands.length; i++){
    var hand = frame.hands[i];
    handData.innerHTML += "Hand ID: " + hand.id + "<br>"
                + "Hand Type: " + hand.type + "<br>"
                + "Palm Position: " + hand.palmPosition + "<br>"
                + "Grab Strength: " + hand.grabStrength + "<br>"  
                + "Pinch Strength: " + hand.pinchStrength + "<br><br>";	  	  
  }
})
</script>

The hand.palmPosition returns a 3D vector (x, y, z) indicating the coordinates of the centre position of the palm in millimeters from the Leap origin.

Deciphering the Fingers

Similarly, each Finger object contained in the Frame object possesses a set of properties, such as id of the finger object, the id of the Hand object that it belongs to, finger tip position, finger type, among other things. The highlighted code below gets the id, handId, tipPosition, and type of each Finger object contained in the Frame object, and displays them in the browser.

<script>
var frameData = document.getElementById('frameData');
var handData = document.getElementById('handData');
var fingerData = document.getElementById('fingerData');

Leap.loop(function(frame){

  // Get and show frame data
  frameData.innerHTML = "Frame ID: " + frame.id  + "<br>"
            + "No of Hands: " + frame.hands.length + "<br>"
            + "No of Fingers: " + frame.fingers.length + "";

  // Get and show hand data
  handData.innerHTML = "";
  for(var i = 0; i < frame.hands.length; i++){
    var hand = frame.hands[i];
    handData.innerHTML += "Hand ID: " + hand.id + "<br>"
                + "Hand Type: " + hand.type + "<br>"
                + "Palm Position: " + hand.palmPosition + "<br>"
                + "Grab Strength: " + hand.grabStrength + "<br>"  
                + "Pinch Strength: " + hand.pinchStrength + "<br><br>";	  	  
  }
  // Get and show finger data
  fingerData.innerHTML = "";
  for(var i = 0; i < frame.fingers.length; i++){
    var finger = frame.fingers[i];
    fingerData.innerHTML += "Finger ID: " + finger.id  + "<br>"
                + "Belong to Hand ID: " + finger.handId + "<br>"
                + "Finger Tip Position: " + finger.tipPosition + "<br>"
                + "Finger Type: " + finger.type + "<br>" + "<br>";	  	  
  }
})
</script>

The finger.tipPosition returns a 3D vector (x, y, z) indicating the coordinates of the tip position of a finger in millimeters from the Leap origin.

Seeing is Believing

The code discussed above has been created in one of my pens called "Deciphering Raw Data" on CodePen at https://codepen.io/peterleow/pen/JyQmKj as shown:

Run the code! You should be able to see the constant updating of frame, hands, and fingers data in the browser as you move your hands and fingers above the Leap Motion controller. Have fun!

Animating Hands and Fingers

You have done the code to capture and display the constant update of frame, hands, and fingers data in the browser. Isn't that a piece of cake? However, trying to make sense of textual data, not to mention ones that are changing constantly, is hard. It will be helpful if the data can be animated using some graphical cues. That sounds interesting, right?

As a lead, let's create graphical cues for one hand and five finger tips and animate them in the browser based on their position data, i.e., palm position of the hand and tip positions of the respective fingers. For simplicity, each of these hand and fingers is represented graphically by a rounded HTML <div>. Are you ready?

In index.html, add six <div>'s along with the related CSS as highlighted below:

<style>
th, td {
  min-width: 300px;
  text-align: left;
  vertical-align: top	
}
div {
  background-color: red;
  border-radius: 50px;
  position: absolute;
}
div#palm {
  height: 100px;
  width: 100px;	
}
div.finger {
  height: 20px;
  width: 20px;	
}
</style>
<script src="https://js.leapmotion.com/leap-0.6.4.min.js"></script>
</head>
<body>

<table>
  <tr>
    <th>Frame Data</th><th>Hand Data</th><th>Finger Data</th>
  </tr>
  <tr>
    <td id="frameData">Frame Data</td><td 
    id="handData">Hand Data</td><td id="fingerData">Finger Data</td>
  </tr>
</table>

<div id="palm"></div>
<div class="finger"></div>
<div class="finger"></div>
<div class="finger"></div>
<div class="finger"></div>
<div class="finger"></div>

<script>
var frameData = document.getElementById('frameData');

In the <script> section, assign these <div>'s to some JavaScript variables as highlighted below:

<script>
var frameData = document.getElementById('frameData');
var handData = document.getElementById('handData');
var fingerData = document.getElementById('fingerData');

var palmDisplay = document.getElementById('palm');
var fingersDisplay = document.getElementsByClassName('finger');

You are ready to write code to animate one of your hand based on its palm position. The palm position is available in 3D vector coordinates in millimeters from the Leap origin available via the palmPosition property of the Hand object. Use the normalizePoint() method of the Leap's InteractionBox class to convert these coordinates to their normalized coordinates in the range between 0 and 1. The code to do this is as follows:

var normalizedPalmPosition = frame.interactionBox.normalizePoint(hand.palmPosition);

To convert these normalized coordinates to your application's coordinates, simply multiply the normalized coordinate of each axis by the maximum range of the corresponding axis of the browser screen, which ignoring the z axis as it is not required for this exercise.

The following code snippet converts the normalized x coordinate, i.e. normalizedPalmPosition[0], to the browser's x coordinate which becomes the x coordinate of the centre of <div id="palm"></div> by re-positioning.

var palmX = window.innerWidth * normalizedPalmPosition[0] - palmDisplay.offsetWidth / 2;
palmDisplay.style.left = palmX + "px";

Similarly, the following code snippet converts the normalized y coordinate, i.e., normalizedPalmPosition[1], to the browser's y coordinate which becomes the y coordinate of the centre of <div id="palm"></div> by re-positioning. Note the subtraction of normalized y coordinate from one which is needed to convert the upwards pointing y axis of Leap's coordinate system to its downwards pointing counterpart of browser's coordinate system.

var palmY = window.innerHeight * (1 - normalizedPalmPosition[1]) - palmDisplay.offsetHeight / 2;
palmDisplay.style.top = palmY + "px";

Where do you put these five lines of code? Add them to the for loop for the Hand object.

The code discussed so far has been created in one of my pens called "Animating Hands and Fingers" on CodePen and embedded at https://codepen.io/peterleow/pen/oeraBb as shown:

Run the code! See that the <div id="palm"></div> moves along with one of your hand within the region of the Interaction Box of the Leap Motion controller. Notice that the <div id="palm"></div> can sneak out of your screen. To confine its movement within the bound of your screen, you can either write additional code or simply add a true parameter to the normalizePoint() method as shown:

var normalizedPalmPosition = frame.interactionBox.normalizePoint(hand.palmPosition, true);

Wait! What about the code to animate the fingers? The answer is that you have just learned it for the hand, how much different can it be for the fingers? I shall leave it as your homework.

If you have done your homework well, you should be able to see the "fruit of your labour" as shown in this animated gif:

Do not stop here, fork my pen at CodePen, then enhance the code to animate two hands and ten fingers.

Going the Extra Mile

Having written the code to animate your hands and fingers movement in the browser, the next natural thing to look forwards to is to be able to use these animated hands or fingers to interact with your web applications. One of the most common interactions is clicking a web element, e.g., a button, to trigger some event using a mouse. Can this be done using a finger in the air in place of the mouse? You bet!

Clicking With Your Finger in the Air

Imagine there is a virtual vertical touch plane above the Interaction Box. The distance between your finger tip and this touch plane can be obtained via the touchDistance property of the Finger object and is available in the range between -1 and 1.

As shown in the diagram above, a value between zero and one indicates that the finger tip is in the hovering zone, a value of zero indicates that the finger tip has just touched the touch plane, and a value between zero and minus one indicates that the finger tip is in the touching zone. You can then use the value returned by the touchDistance property of the Finger object in your code to emulate the state that a finger is in, i.e., hovering or touching. However, It is entirely up to you to decide on the threshold value of the touchDistace property that demarcates the hovering state from the touching state. In other words, it isn't set in stone that the threshold value has to be always zero.

Through the touchDistance property, you can emulate a left mouse click using your finger in the air via the Leap Motion controller. Let's extend it to one of my pens called "Open Sesame" on CodePen at https://codepen.io/peterleow/pen/eEwbwP as shown:

It is a simple web application that has a door and two buttons — one to open the door to reveal a picture behind it and another to close it. Run it and try out the buttons using your mouse!

You mission is to spare the mouse and use your finger in the air instead to click the respective buttons.

The Leap Motion JavaScript library has already been added to this application.

To animate your finger tip on the screen, create a rounded HTML <div> with its related CSS as shown:

<div id="finger"></div>

div#finger {
  height: 10px;
  width: 10px;
  position: absolute;
  background-color: red;
  border-radius: 5px
}

Next, add the JavaScript code to animate one of your finger tips on the screen.

var fingerDisplay = document.getElementById("finger");

Leap.loop(function(frame) {
  if (frame.fingers.length > 0) {
    var finger = frame.fingers[0];
    var normalizedFingerPosition = frame.interactionBox.normalizePoint(finger.tipPosition);
 
    var appX = window.innerWidth * normalizedFingerPosition[0] - fingerDisplay.offsetWidth / 2;
    fingerDisplay.style.left = appX + "px";

    var appY = window.innerHeight * (1 - normalizedFingerPosition[1]) - 
                                     fingerDisplay.offsetHeight / 2;
    fingerDisplay.style.top = appY + "px";

    // add code to emulate left mouse click
  }
});

Run it and you should see a red dot moving along with one of your finger tip. You have just repeated what you have learned in the earlier section.

You are ready to add the code to emulate left mouse click. Follow me:

• Get the value of the touchDistance property of the finger:
  JavaScript

  var touchDistance = finger.touchDistance;

• Assuming an emulated left mouse click occurs if the touchDistance is less than zero:
  JavaScript

  if (touchDistance < 0) {
    // code to handle click event
  }

• When an emulated left mouse click is detected, you have to identify the HTML element (button or otherwise) beneath the red dot.
  JavaScript

  fingerDisplay.style.display = "none";
  var touchedElement = document.elementFromPoint(appX, appY);
  fingerDisplay.style.display = "block";

• If the HTML element beneath is the Open button, activate the click event for btnOpen.
  JavaScript

  if (touchedElement == btnOpen) {
    btnOpen.click();
  }

With the code that you have added, you can now emulate left mouse click on the Open button using your finger in the air. Check out the action in one of my pens called "Open Sesame 2" on CodePen at https://codepen.io/peterleow/pen/KvjJpR as shown:

The code is far from complete. The missing pieces are as follows:

• The code for activating the click event for btnClose if the HTML element beneath the red dot is the Close button.

• As it is now, there is no way of telling whether your finger enters or leaves the touching zone. The solution is to introduce different visual cue for each event, such as changing the red dot to blue when a finger enters the touching zone and vice versa when it leaves.
• Last but not least, you will soon notice that the door opens or closes excessively instead of at a fixed amount at each click, owing to the constant firing of the button event when the finger remains in the touching zone at each frame update. To overcome it manually, your finger has to enter and leave the touching zone at quick succession. This is neither user friendly nor palatable. One of the solutions I can offer is to use a flag (true or false) to prevent the firing of the same event from subsequent frames if the finger has not left the touching zone after entering it. In other words, there should be only one firing of click event for each cycle of entering and leaving the touching zone. Of course, the actual solution hinges upon your application requirements.

I shall leave them as your homework. Go for it!

Dragging With Your Finger in the Air

Having learned the code to emulate left mouse click using your finger in the air, why not extend it to emulate a mouse drag? Usually, you drag an object on the screen by moving the mouse while holding down its left button, right? So, a mouse drag is effected through a combination of click, hold, and move actions. Translating this into Leap, a drag occurs when your finger enters the touching zone, remains there, while moving. Got it!

Let's add the code to emulate a mouse drag to this partially completed pen called "Drag Me Along!" on CodePen at https://codepen.io/peterleow/pen/prXGGB as shown:

Since drag is an extension of click, start by copying the Leap.loop() part of the JavaScript from the "Open Sesame 2" pen to this pen.

Next, zoom in to this part of the code as shown:

if (touchedElement == btnOpen) {
  btnOpen.click();
}

This is the only code that you need to modify in order to implement the drag. However, you can only know how to modify it after finding out the answers to these two questions:

• What is the touchedElement this time?
• How to make this touchedElement move along with the finger?

I have already explained and demonstrated the code for similar implementations earlier, so I shall not repeat again.

Tips

Interacting with a web application via a Leap Motion controller is inherently a virtual experience. Since it is done without the feel and sensation of a real touch, users can neither control the pace of interactions nor know the state of their interactions with the web application. To alleviate these problems so as to make your web application more usable, consider incorporating the following measures in your implementation:

• Always provide feedback to the users on the status and progress of their interaction in the form of visual cues on the screen.
• Need to regulate the response rate of your code vis–à–vis the user's pace of interaction.

Crossing the Finishing Line...

In this article, you have learned the basic mechanism of a Leap Motion controller, gotten started on writing code to implement motion tracking of hands and fingers as well as clicking and dragging of web elements in the browser using your finger in the air via the Leap Motion controller, and picked up some usability tips on using the Leap Motion controller with your web application.

Give yourself a pat on the back!

Every Ending is Another Beginning

As the saying goes, 'Give a man a fish and you feed him for a day. Teach a man to fish and you feed him for a lifetime'. Now that you have been empowered with the 'fishing skill', it's up to you to use it well to catch a bigger 'fish' — rotating a wheel with your hand in the air via the Leap Motion controller as shown in this animated gif.

The end of a journey is the beginning of another. Hope you find this one a fruitful one.

The post Leap into Motion appeared first on Peter Leow's Code Blog.