Introduction
Since I have started to develop desktop applications using Kinect, I had to test my code by standing-up in front of the sensor. At the beginning, it was funny, but as soon as I had to do it tens of times within a day, it was starting to be annoying. So, I decided to create an application that could record and playback any moves!
Background
In this article, it is assumed that you have basic knowledge of using Kinect sensor & Kinect SDK (not OpenNI) and you are familiar with C#. The application is written for Kinect Beta 2 SDK driver version 1.0.0.45.
If your are not familiar with Kinect, take a look at a great series of tutorials!
Prerequisites
Using KinectRecorderInterface
Using KinectRecorderInterface
class is pretty much easy. First of all, let's take a look at the public
members/methods/events of the class.
Its use is intended to be as simple as possible!
Declaring creating the class as most of the classes.
KinectRecorderInterface recorder = new KinectRecorderInterface();
Initializing open the Kinect sensor through the recorder and select an appropriate ImageResolution
(DepthImage
can be up 640x480).
Notice: Only one application at a time can use the Kinect!!
Use to open only the DepthStream
:
recorder.InitializeSensor(ImageResolution.Resolution320x240);
Use to open both of DepthStream
& VideoStream
:
recorder.InitializeSensor(ImageResolution.Resolution320x240, ImageResolution.Resolution640x480);
Whole Initialization code:
private KinectRecorderInterface recorder;
public void InitializeKinectRecorder()
{
recorder = new KinectRecorderInterface();
recorder.InitializeSensor(ImageResolution.Resolution640x480, ImageResolution.Resolution640x480);
recorder.PlaybackSkeletonDataReady +=
new KinectRecorderInterface.PlaybackFrameHandler(recorder_PlaybackSkeletonDataReady);
recorder.SkeletonFrameReady +=
new KinectRecorderInterface.SkeletonFrameHandler(recorder_SkeletonFrameReady);
recorder.ImageFrameReady +=
new KinectRecorderInterface.ImageFrameHandler(recorder_ImageFrameReady);
recorder.DepthFrameReady +=
new KinectRecorderInterface.DepthFrameHandler(recorder_DepthFrameReady);
}
That's all!
Assigning event handlers: Now you can create proper event handlers to take data from Kinect. Notice: EventHandlers
and the EventArgs
are the same as using the Kinect SDK, except for the playback!!
recorder.SkeletonFrameReady +=
new KinectRecorderInterface.SkeletonFrameHandler(recorder_SkeletonFrameReady);
recorder.ImageFrameReady +=
new KinectRecorderInterface.ImageFrameHandler(recorder_ImageFrameReady);
recorder.DepthFrameReady +=
new KinectRecorderInterface.DepthFrameHandler(recorder_DepthFrameReady);
Playback EventHandler
is pretty much the same, but it differentiates its EventArgs
(see later on 'SkeletonData VS SerializableSkeletonData
').
recorder.PlaybackSkeletonDataReady +=
new KinectRecorderInterface.PlaybackFrameHandler(recorder_PlaybackSkeletonDataReady);
Now, you can retrieve Kinect's data as usually (in realtime) from SkeletonTracking
, DepthStream
and VideoStream
, but there is no recording or playback at the time.
Starting / Stop Recording Anytime you can start StartRecording()
method, in order to start capturing only available (Tracked) SkeletonData.
Use StopRecording()
method, in order to stop capturing and save any stored SkeletonData
. Saving data are stored by default in "Recordings" folder, unless you change it from RecordDirectory
.
Notice: If recording has no SkeletonData
, saving will be aborted.
recorder.RecordDirectory = "./StoreData";
recorder.StartRecording();
recorder.StopRecording();
Now, by default, a new folder "Recordings" has been created along with the project's executable; and a file called "recording_00x.dat" has been created within the folder.
# Start / Stop Playback
After you have recorded some moves, you are able to playback them any time you want to. Use StartPlayback()
to start playback of the last available recording. Also, you can use StartPlayback
(recording_xxx.dat), in order to playback a recording of your choice.
Notice: Each time the application is starting/recording/saving an updated catalog (List<string>
) of the recordings is available through the ExistingRecords
attribute, to let you choose any recording.
recorder.StartPlayback();
List<string> records = recorder.ExistingRecords;
recorder.StartPlayback(records[0]);
SkeletonData VS SerializableSkeletonData
As soon as I reached the point that I had to store the data, I realized that I couldn't serialize SkeletonData
because it was not marked as Serializable
. For that reason, I implemented Json.Net which worked great! But, unfortunately, when I tried to deserialize and cast (see below) the stored data, it failed again as SkeletonData
does not have any public
constructor (its constructor is set to internal
)!! A better solution is welcome!!!
<Code Not Working>
var json = System.IO.File.ReadAllText(filepath);
var skeletonData = Newtonsoft.Json.JsonConvert.DeserializeObject<List<SkeletonData>>(json);
</Code Not Working>
Finally, I didn't come up with any other solution rather than create a new class almost identical to SkeletonData
as shown below:
At the only point that you will get SerializableSkeletonData
, will be at the time that you will get SkeletonDataArgs
from PlaybackFrameHandler
. SkeletonDataArgs
holds SerializableSkeletonData
.
Using SerializableSkeletonData
: It's almost the same process as using SkeletonData
!
For example, let's say that we have the following code for SkeletonData
:
private void recorder_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs skeletonFrameArgs)
{
SkeletonData skeleton = skeletonFrameArgs.SkeletonFrame.Skeletons[0];
if (skeleton != null)
{
setEclipsePos(HeadEclipse, skeleton.Joints[JointID.Head]);
setEclipsePos(LeftHandEclipse, skeleton.Joints[JointID.HandLeft]);
setEclipsePos(RightHandEclipse, skeleton.Joints[JointID.HandRight]);
}
}
The same function through Playback using SerializableSkeletonData
:
private void recorder_PlaybackSkeletonDataReady(object sender, SkeletonDataArgs skeletonDataArgs)
{
SerializableSkeletonData skeleton = skeletonDataArgs.serializableSkeletonData;
if (skeleton != null)
{
setEclipsePos(HeadEclipse, skeleton.Joints[(int)JointID.Head]);
setEclipsePos(LeftHandEclipse, skeleton.Joints[(int)JointID.HandLeft]);
setEclipsePos(RightHandEclipse, skeleton.Joints[(int)JointID.HandRight]);
}
}
Extra Properties
KinectRecorderInterface
has a PlayPause
method while you are in recording or playback mode.
Use PlayPauseRecording()
or PlayPausePlayback()
each time you want to pause your recording or playback. Use the same method again to continue..!
recorder.PlayPauseRecording();
recorder.PlayPausePlayback();
Use Status
to check or see the current status of the recording interface!
public enum KinectRecorderStatus
{
Idle = 0,
Initializing = 1,
Recording = 2,
Playing = 3,
Saving = 4,
Loading = 5,
NotConnected = 6,
PlayingPaused = 7,
RecordingPaused = 8
}
statusLabel.Text = recorder.Status.ToString();
if (recorder.Status == KinectRecorderStatus.Playing)
{
}
Application already has HotKey
identification running in the background! You can set, up to 4 HotKeys
to handle:
- Start / Stop Recording:
RecordHotKey
- Start / Stop Playback:
PlaybackHotKey
- Pause / Resume Recording:
PauseRecordingHotKey
- Pause / Resume Playback:
PausePlaybackHotKey
It is not necessary to enable HotKeys
, but you have to assign them a Key
value.
recorder.RecordHotKey = System.Windows.Forms.Keys.F9;
Application's Class Diagram
Including only the public
attributes: