Controlling a Myo Armband with C#

Background

Thalmic Labs has started shipping their Myo armband that allows the wearer’s arm movements and gestures to control different pieces of integrated technology. How cool is that? My friend and I decided we wanted to give one a whirl and see what we could come up with. We’re both C# advocates, so we were a bit taken aback when we saw the only C# support in the SDK was made for Unity. We decided to take things into our own hands and open source a Myo C# library. We’re excited to introduce the first version of MyoSharp!

The underlying Myo components are written in C++, and there’s only several functions that are exposed from the library that we can access. In order to do this, we need to leverage platform invocation (PInvokes) from C# to tap into this functionality. Once you have the PInvokes set up, you can begin to play around!

The Workflow

Getting setup with the Myo is pretty straightforward, but it wasn’t obvious to us right away. We didn’t have anyone to walk us through how the different components were supposed to work together (just some good ol’ fashioned code) so we had to tinker around. Once we had everything mapped out, it was quite simple though.

1. The first step is opening a communication channel with the Bluetooth module. You don’t need to worry about the implementation here since it’s all done in C++ by the Thalmic devs. Calling the correct methods using PInvokes from C# allows us to tap into a stream of “events” that come through the Bluetooth module.
2. Now that we can intercept events, we need to be able to identify a Myo. After all, working with Myos is our main goal here! There’s a “pair” event that we can listen to from the Bluetooth module that notifies us of when a Myo has paired and provides us a handle to the device. This handle gets used for identifying events for a particular Myo or sending a particular Myo messages.
3. There’s a connect event that will fire when a Myo connects after it’s been paired with the Bluetooth module. A Myo can be paired but disconnected.
4. Now that we can uniquely identify a Myo, the only things we need to do are intercept events for a particular Myo and make sense of the data coming from the devices! Orientation change? Acceleration change? There’s a host of information that the device sends back, so we need to interpret it.
5. When a Myo disconnects, there’s an event that’s sent back for that as well.

Getting Started with MyoSharp

I’m going to start this off with some simple code that should illustrate just how easy it is to get started with MyoSharp. I’ll describe what’s going on in the code immediately after.

using System;

using MyoSharp.Device;
using MyoSharp.ConsoleSample.Internal;

namespace MyoSharp.ConsoleSample
{
    /// <summary>
    /// This example will show you the basics for setting up and working with
    /// a Myo using MyoSharp. Primary communication with the device happens
    /// over Bluetooth, but this C# wrapper hooks into the unmanaged Myo SDK to
    /// listen on their "hub". The unmanaged hub feeds us information about
    /// events, so a channel within MyoSharp is responsible for publishing
    /// these events for other C# code to consume. A device listener uses a
    /// channel to listen for pairing events. When a Myo pairs up, a device
    /// listener publishes events for others to listen to. Once we have access
    /// to a channel and a Myo handle (from something like a Pair event), we
    /// can create our own Myo object. With a Myo object, we can do things like
    /// cause it to vibrate or monitor for poses changes.
    /// </summary>
    internal class BasicSetupExample
    {
        #region Methods
        private static void Main(string[] args)
        {
            // create a hub that will manage Myo devices for us
            using (var hub = Hub.Create())
            {
                // listen for when the Myo connects
                hub.MyoConnected += (sender, e) =>
                {
                    Console.WriteLine("Myo {0} has connected!", e.Myo.Handle);
                    e.Myo.Vibrate(VibrationType.Short);
                    e.Myo.PoseChanged += Myo_PoseChanged;
                };

                // listen for when the Myo disconnects
                hub.MyoDisconnected += (sender, e) =>
                {
                    Console.WriteLine("Oh no! It looks like {0} arm Myo has disconnected!", e.Myo.Arm);
                    e.Myo.PoseChanged -= Myo_PoseChanged;
                };

                // wait on user input
                ConsoleHelper.UserInputLoop(hub);
            }
        }
        #endregion

        #region Event Handlers
        private static void Myo_PoseChanged(object sender, PoseEventArgs e)
        {
            Console.WriteLine("{0} arm Myo detected {1} pose!", e.Myo.Arm, e.Myo.Pose);
        }
        #endregion
    }
}

In this example, we create a hub instance. A hub will manage a collection of Myos that come online and go offline and notify listeners that are interested. Behind the scenes, the hub creates a channel instance and passes this into a device listener instance. The channel and device listener combination allows for being notified when devices come online and is the core of the hub implementation. You can manage Myos on your own by completely bypassing the Hub class and creating your own channel and device listener if you’d like. It’s totally up to you.

In the code above, we’ve hooked up several event handlers. There’s an event handler to listen for when Myo devices connect, and a similar one for when the devices disconnect. We’ve also hooked up to an instance of a Myo device for when it changes poses. This will simply give us a console message every time the hardware determines that the user is making a different pose.

When devices go offline, the hub actually keeps the instance of the Myo object around. This means that if you have device A and you hook up to its PoseChanged event, if it goes offline and comes back online several times, your event will still be hooked up to the object that represents device A. This makes managing Myos much easier compared to trying to re-hook event handlers every time a device goes on and offline. Of course, you’re free to make your own implementation using our building blocks, so there’s no reason to feel forced into this paradigm.

It’s worth mentioning that the UserInputLoop() method is only used to keep the program alive. The sample code on GitHub actually lets you use some debug commands to read some Myo statuses if you’re interested. Otherwise, you could just imagine this line is replaced by Console.ReadLine() to block waiting for the user to press enter.

Pose Sequences

Without even diving into the accelerometer, orientation, and gyroscope readings, we were looking for some quick wins to building up on the basic API that we created. One little improvement we wanted to make was the concept of pose sequences. The Myo will send events when a pose changes, but if you were interested in grouping some of these together there’s no way to do this out of the box. With a pose sequence, you can declare a series of poses and get an event triggered when the user has finished the sequence.

Here’s an example:

using System;

using MyoSharp.Device;
using MyoSharp.ConsoleSample.Internal;
using MyoSharp.Poses;

namespace MyoSharp.ConsoleSample
{
    /// <summary>
    /// Myo devices can notify you every time the device detects that the user 
    /// is performing a different pose. However, sometimes it's useful to know
    /// when a user has performed a series of poses. A 
    /// <see cref="PoseSequence"/> can monitor a Myo for a series of poses and
    /// notify you when that sequence has completed.
    /// </summary>
    internal class PoseSequenceExample
    {
        #region Methods
        private static void Main(string[] args)
        {
            // create a hub to manage Myos
            using (var hub = Hub.Create())
            {
                // listen for when a Myo connects
                hub.MyoConnected += (sender, e) =>
                {
                    Console.WriteLine("Myo {0} has connected!", e.Myo.Handle);

                    // for every Myo that connects, listen for special sequences
                    var sequence = PoseSequence.Create(
                        e.Myo, 
                        Pose.WaveOut, 
                        Pose.WaveIn);
                    sequence.PoseSequenceCompleted += Sequence_PoseSequenceCompleted;
                };

                ConsoleHelper.UserInputLoop(hub);
            }
        }
        #endregion

        #region Event Handlers
        private static void Sequence_PoseSequenceCompleted(object sender, PoseSequenceEventArgs e)
        {
            Console.WriteLine("{0} arm Myo has performed a pose sequence!", e.Myo.Arm);
            e.Myo.Vibrate(VibrationType.Medium);
        }
        #endregion
    }
}

The same basic setup occurs as the first example. We create a hub that listens for Myos, and when one connects, we hook a new PoseSequence instance to it. If you recall how the hub class works from the first example, this will hook up a new pose sequence each time the Myo connects (which, in this case, isn’t actually ideal). Just for demonstration purposes, we were opting for this shortcut though.

When creating a pose sequence, we only need to provide the Myo and the poses that create the sequence. In this example, a user will need to wave their hand out and then back in for the pose sequence to complete. There’s an event provided that will fire when the sequence has completed. If the user waves out and in several times, the event will fire for each time the sequence is completed. You’ll also notice in our event handler we actually send a vibrate command to the Myo! Most of the Myo interactions are reading values from Myo events, but in this case this is one of the commands we can actually send to it.

Held Poses

The event stream from the Myo device only sends events for poses when the device detects a change. When we were trying to make a test application with our initial API, we were getting frustrated with the fact that there was no way to trigger some action as long as a pose was being held. Some actions like zooming, panning, or adjusting levels for something are best suited to be linked to a pose being held by the user. Otherwise, if you wanted to make an application that would zoom in when the user makes a fist, the user would have to make a fist, relax, make a fist, relax, etc… until they zoomed in or out far enough. This obviously makes for poor usability, so we set out to make this an easy part of our API.

The code below has a similar setup to the previous examples, but introduces the HeldPose class:

using System;

using MyoSharp.Device;
using MyoSharp.ConsoleSample.Internal;
using MyoSharp.Poses;

namespace MyoSharp.ConsoleSample
{
    /// <summary>
    /// Myo devices can notify you every time the device detects that the user 
    /// is performing a different pose. However, sometimes it's useful to know
    /// when a user is still holding a pose and not just that they've 
    /// transitioned from one pose to another. The <see cref="HeldPose"/> class
    /// monitors a Myo and notifies you as long as a particular pose is held.
    /// </summary>
    internal class HeldPoseExample
    {
        #region Methods
        private static void Main(string[] args)
        {
            // create a hub to manage Myos
            using (var hub = Hub.Create())
            {
                // listen for when a Myo connects
                hub.MyoConnected += (sender, e) =>
                {
                    Console.WriteLine("Myo {0} has connected!", e.Myo.Handle);

                    // setup for the pose we want to watch for
                    var pose = HeldPose.Create(e.Myo, Pose.Fist, Pose.FingersSpread);

                    // set the interval for the event to be fired as long as 
                    // the pose is held by the user
                    pose.Interval = TimeSpan.FromSeconds(0.5);

                    pose.Start();
                    pose.Triggered += Pose_Triggered;
                };

                ConsoleHelper.UserInputLoop(hub);
            }
        }
        #endregion

        #region Event Handlers
        private static void Pose_Triggered(object sender, PoseEventArgs e)
        {
            Console.WriteLine("{0} arm Myo is holding pose {1}!", e.Myo.Arm, e.Pose);
        }
        #endregion
    }
}

When we create a HeldPose instance, we can pass in one or more poses that we want to monitor for being held. In the above example, we’re watching for when the user makes a fist or when they have their fingers spread. We can hook up to the Triggered event on the held pose instance, and the event arguments that we get in our event handler will tell us which pose the event is actually being triggered for.

If you take my zoom example that I started describing earlier, we could have a single event handler responsible for both zooming in and zooming out based on a pose being held. If we picked two poses, say fist and fingers spread, to mean zoom in and zoom out respectively, then we could check the pose on the event arguments in the event handler and adjust the zoom accordingly. Of course, you could always make two HeldPose instances (one for each pose) and hook up to the events separately if you’d like. This would end up creating two timer threads behind the scenes–one for each HeldPose instance.

The HeldPose class also has an interval setting. This allows the programmer to adjust the frequency that they want the Triggered event to fire, provided that a pose is being held by the user. For example, if the interval is set to be two seconds, as long as the pose is being held the Triggered event will fire every two seconds.

Roll, Pitch, and Yaw

The data that comes off the Myo can become overwhelming unless you’re well versed in vector math and trigonometry. Something that we’d like to build up and improve upon is the usability of data that comes off the Myo. We don’t want each programmer to have to write similar code to get the values from the Myo into a usable form for their application. Instead, if we can build that into MyoSharp, then everyone will benefit.

Roll, pitch, and yaw are values that we decided to bake into the API directly. So… what exactly are these things? Here’s a diagram to help illustrate:

Roll, pitch, and yaw describe rotation around one of three axes in 3D space.

The following code example shows hooking up to an event handler to get the roll, pitch, and yaw data:

using System;

using MyoSharp.Device;
using MyoSharp.ConsoleSample.Internal;

namespace MyoSharp.ConsoleSample
{
    /// <summary>
    /// This example will show you how to hook onto the orientation events on
    /// the Myo and pull roll, pitch and yaw values from it.
    /// </summary>
    internal class OrientationExample
    {
        #region Methods
        private static void Main(string[] args)
        {
            // create a hub that will manage Myo devices for us
            using (var hub = Hub.Create())
            {
                // listen for when the Myo connects
                hub.MyoConnected += (sender, e) =>
                {
                    Console.WriteLine("Myo {0} has connected!", e.Myo.Handle);
                    e.Myo.OrientationDataAcquired += Myo_OrientationDataAcquired;
                };

                // listen for when the Myo disconnects
                hub.MyoDisconnected += (sender, e) =>
                {
                    Console.WriteLine("Oh no! It looks like {0} arm Myo has disconnected!", e.Myo.Arm);
                    e.Myo.OrientationDataAcquired -= Myo_OrientationDataAcquired;
                };

                // wait on user input
                ConsoleHelper.UserInputLoop(hub);
            }
        }
        #endregion

        #region Event Handlers
        private static void Myo_OrientationDataAcquired(object sender, OrientationDataEventArgs e)
        {
            Console.Clear();
            Console.WriteLine(@"Roll: {0}", e.Roll);
            Console.WriteLine(@"Pitch: {0}", e.Pitch);
            Console.WriteLine(@"Yaw: {0}", e.Yaw);
        }
        #endregion
    }
}

Of course, if we know of more common use cases that people will be using the orientation data for, then we’d love to bake this kind of stuff right into MyoSharp to make it easier for everyone.

Closing Comments

That’s just a quick look at how you can leverage MyoSharp to make your own C# application to work with a Myo! As I said, MyoSharp is open source so we’d love to see contributions or ideas for suggestions. We’re aiming to provide as much base functionality as we can into our framework but designing it in a way that developers can extend upon each of the individual building blocks.

The post Controlling a Myo Armband with C# appeared first on Dev Leader.