Introduction
The Leap is a
3D motion sensing device that detects and tracks hands, fingers and finger-like
tools. In this article I'll describe how you can go about using the Leap,
and the Leap SDK, to create a .NET console application that enables you to move
the cursor using your finger.
Requirements
To run the sample project
you require the following,
- Leap (developer unit),
- Leap SDK,
- VS2012
If you don't have a Leap controller you can apply to join the
developer program and,
hopefully, get a developer unit for free. Joining the developer program will
also grant you access to the Leap SDK.
NB: The example project described here was developed using
dev unit v.06.5 and version 0.7.3 of the Leap SDK.
Running the Project
To run the sample project you have to add Leap.dll and
_LeapCSharp.dll to the project. (These two files are part of the Leap SDK).
Ensure that the Copy to Output Directory property of the two files is
set to Copy always.
Ensure that you also add a reference to LeapCSharp.NET4.0.dll,
which is also part of the Leap SDK, and contains classes that enable you to
access the data from the Leap controller.
Leap Coordinate System
Leap uses a right-handed Cartesian coordinate system, with the origin at the
center of the controller and the X and Z axes lying in the horizontal plane. The
Y axis is vertical.
Moving the Cursor
In order to move the cursor we'll need to call a function in the win32 API.
This is done by a method in class MouseCursor.
Public Class MouseCursor
Private Declare Function SetCursorPos Lib "user32" (x As Integer, y As Integer) As Boolean
Public Shared Sub MoveCursor(x As Integer, y As Integer)
SetCursorPos(x, y)
End Sub
End Class
using System;
using System.Runtime.InteropServices;
using System.Windows;
using System.Windows.Interop;
namespace ConsoleLeapMouse
{
class MouseCursor
{
[DllImport("user32.dll")]
private static extern bool SetCursorPos(int x, int y);
public static void MoveCursor(int x, int y)
{
SetCursorPos(x, y);
}
}
}
MoveCursor() takes two parameters that define the location on the screen
where the cursor should move. To get the values for these parameters we'll need
to process data coming from the Leap controller.
To get data from the Leap we need to create a Controller object.
The Controller object connects the console application with the Leap
software and provides hand and finger tracking data through
Frame objects.
Imports Leap
Module Module1
Sub Main()
Dim cntrl As New Controller
Dim listener As New LeapListener
cntrl.AddListener(listener)
Console.WriteLine("Press Enter to quit...")
Console.ReadLine()
cntrl.RemoveListener(listener)
cntrl.Dispose()
End Sub
End Module
using System;
using Leap;
namespace ConsoleLeapMouse
{
class Program
{
static void Main(string[] args)
{
Controller cntrl = new Controller();
LeapListener listener = new LeapListener();
cntrl.AddListener(listener);
Console.WriteLine("Press Enter to quit...");
Console.ReadLine();
cntrl.RemoveListener(listener);
cntrl.Dispose();
}
}
}
In the code snippet above an object of type LeapListener is added
to the Controller object. LeapListener is a subclass of the Listener
class which defines a set of callback methods that can be overridden to
respond to events dispatched by the Leap.
Imports Leap
Public Class LeapListener
Inherits Listener
Public Overrides Sub OnInit(cntrlr As Controller)
Console.WriteLine("Initialized")
End Sub
Public Overrides Sub OnConnect(cntrlr As Controller)
Console.WriteLine("Connected")
End Sub
Public Overrides Sub OnDisconnect(cntrlr As Controller)
Console.WriteLine("Disconnected")
End Sub
Public Overrides Sub OnExit(cntrlr As Controller)
Console.WriteLine("Exited")
End Sub
Private currentTime As Long
Private previousTime As Long
Private timeChange As Long
Public Overrides Sub OnFrame(cntrlr As Controller)
Dim currentFrame As Frame = cntrlr.Frame
currentTime = currentFrame.Timestamp
timeChange = currentTime - previousTime
If (timeChange > 10000) Then
If (Not currentFrame.Hands.Empty) Then
Dim finger = currentFrame.Fingers(0)
Dim screen = cntrlr.CalibratedScreens.ClosestScreenHit(finger)
If (screen IsNot Nothing And screen.IsValid) Then
Dim tipVelocity = finger.TipVelocity.Magnitude
If (tipVelocity > 25) Then
Dim xScreenIntersect = screen.Intersect(finger, True).x
Dim yScreenIntersect = screen.Intersect(finger, True).y
If (xScreenIntersect.ToString <> "NaN") Then
Dim x = CInt(xScreenIntersect * screen.WidthPixels)
Dim y = screen.HeightPixels - CInt(yScreenIntersect * screen.HeightPixels)
Console.WriteLine("Screen intersect X: " & xScreenIntersect.ToString)
Console.WriteLine("Screen intersect Y: " & yScreenIntersect.ToString)
Console.WriteLine("Width pixels: " & screen.WidthPixels.ToString)
Console.WriteLine("Height pixels: " & screen.HeightPixels.ToString)
Console.WriteLine(vbCrLf)
Console.WriteLine("x: " & x.ToString)
Console.WriteLine("y: " & y.ToString)
Console.WriteLine(vbCrLf)
Console.WriteLine("Tip velocity: " & tipVelocity)
MouseCursor.MoveCursor(x, y)
Console.WriteLine(vbCrLf & New String("=", 40) & vbCrLf)
End If
End If
End If
End If
previousTime = currentTime
End If
End Sub
End Class
using System;
using Leap;
namespace ConsoleLeapMouse
{
class LeapListener: Listener
{
public override void OnInit(Controller cntrlr)
{
Console.WriteLine("Initialized");
}
public override void OnConnect(Controller cntrlr)
{
Console.WriteLine("Connected");
}
public override void OnDisconnect(Controller cntrlr)
{
Console.WriteLine("Disconnected");
}
public override void OnExit(Controller cntrlr)
{
Console.WriteLine("Exited");
}
private long currentTime;
private long previousTime;
private long timeChange;
public override void OnFrame(Controller cntrlr)
{
Frame currentFrame = cntrlr.Frame();
currentTime = currentFrame.Timestamp;
timeChange = currentTime - previousTime;
if (timeChange > 10000)
{
if (!currentFrame.Hands.Empty)
{
Finger finger = currentFrame.Fingers[0];
Screen screen = cntrlr.CalibratedScreens.ClosestScreenHit(finger);
if (screen != null && screen.IsValid)
{
var tipVelocity = (int)finger.TipVelocity.Magnitude;
if (tipVelocity > 25)
{
var xScreenIntersect = screen.Intersect(finger, true).x;
var yScreenIntersect = screen.Intersect(finger, true).y;
if (xScreenIntersect.ToString() != "NaN")
{
var x = (int)(xScreenIntersect * screen.WidthPixels);
var y = (int)(screen.HeightPixels - (yScreenIntersect * screen.HeightPixels));
Console.WriteLine("Screen intersect X: " + xScreenIntersect.ToString());
Console.WriteLine("Screen intersect Y: " + yScreenIntersect.ToString());
Console.WriteLine("Width pixels: " + screen.WidthPixels.ToString());
Console.WriteLine("Height pixels: " + screen.HeightPixels.ToString());
Console.WriteLine("\n");
Console.WriteLine("x: " + x.ToString());
Console.WriteLine("y: " + y.ToString());
Console.WriteLine("\n");
Console.WriteLine("Tip velocity: " + tipVelocity.ToString());
MouseCursor.MoveCursor(x, y);
Console.WriteLine("\n" + new String('=', 40) + "\n");
}
}
}
}
previousTime = currentTime;
}
}
}
}
The most significant method in the code snippet above is the OnFrame() method. The OnFrame()
method is called by the Controller object
when a new frame of hand and finger tracking data is available. The data is
accessed by calling the Controller object's Frame() method.
Since the Leap has a really high frame rate, I guess a maximum of about 115fps,
attempting to move the cursor at every call of the OnFrame() method
will not give the desired results. For this reason I have specified an interval of >10ms
between frames for processing of tracking data.
NB: Frame.Timestamp() is the frame capture time
in microseconds elapsed since the Leap started.
To get the screen that a finger is pointing to we use the Controller.CalibratedScreens.ClosestScreenHit()
method and pass it a Finger object, which in this case is the first Finger
object in a list of Finger objects. The specified screen will be
the closest screen intercepting a ray projecting from the Finger
object. The projected ray emanates from the Finger TipPosition along its
Direction vector.
Finger TipPosition and Direction vectors
To get the screen position in pixels, where the finger is pointing at, we multiply the values returned by
Screen.WidthPixels() and Screen.HeightPixels() with the
normalized coordinates
returned by Screen.Intersect(). Normalized coordinates represent the
intersection point between the screen and the ray projecting from the Finger
object, defined as a percentage of the screen's width and height. Such
coordinates would be of the form (0.1, 0.1, 0).
...
Dim xScreenIntersect = screen.Intersect(finger, True).x
Dim yScreenIntersect = screen.Intersect(finger, True).y
If (xScreenIntersect.ToString <> "NaN") Then
Dim x = CInt(xScreenIntersect * screen.WidthPixels)
Dim y = screen.HeightPixels - CInt(yScreenIntersect * screen.HeightPixels)
...
End If
...
var xScreenIntersect = screen.Intersect(finger, true).x;
var yScreenIntersect = screen.Intersect(finger, true).y;
if (xScreenIntersect.ToString() != "NaN")
{
var x = (int)(xScreenIntersect * screen.WidthPixels);
var y = (int)(screen.HeightPixels - (yScreenIntersect * screen.HeightPixels));
...
}
In case you are wondering why I'm setting the value of the y variable by doing screen.HeightPixels - (yScreenIntersect * screen.HeightPixels),
it's because my computer's graphics coordinate system places the screen's origin
at the bottom-left corner.
NB: If the Finger object is pointing parallel to or away from the screen the components of the vector returned by Screen.Intersect() are all set to NaN (not-a-number).
Conclusion
In this article I have not explained how you can go
about implementing mouse clicks. For that you would require custom gestures,
plus some win32 api calls. Despite this I hope you found the information in this
article useful.
History
- 21st Feb 2013: Initial version
