3. HoloToolkit Unity - Voice Commands

Introduction

Your own voice is an important way to control your Hololens.

Background

If you have not already installed the development tools for Hololens on your PC, you might want to browse through the following article before you read any further:

• 1. HoloToolit Unity - Environment 

Using the Code

We will continue on from the last article: 2. HoloToolkit Unity - Gaze and Airtap and see if we can have our Cube perform some actions just by talking to it.

Build in Voice Commands

According to Microsoft Support, here are some basic commands you can use totally for free, without having to code a single statement:

• Select: Use this instead of air tap. Gaze at a hologram, then say "Select."
• Release: Releases a previously selected hologram.
• Place: Instead of air tapping to place a hologram, say "Place."
• Face me: Gaze at a hologram, then say “Face me” to turn it your way.
• Bigger/Smaller: Gaze at a hologram, then say “Bigger” or “Smaller” to resize it.

Cortana is Also a Strong Force to Reckon With...

You might try these:

Hey Cortana,

• What can I say?
• Increase the volume.
• Decrease the brightness.
• Shut down.
• Restart.
• Go to sleep.
• Mute.
• Launch <app name>.
• Move <app name> here (gaze at the spot you want the app to move to).
• Go to Start.
• Take a picture.
• Start recording. (Starts recording a video.)
• Stop recording. (Stops recording a video.)
• Call <contact>. (Requires Skype.)
• What time is it?
• Show me the latest NBA scores.
• How much battery do I have left?
• Tell me a joke.
• What is my IP address.

Making Your Own Commands

In unity, start your Scene from last time, Scene_2 and save it in your _CP\Scenes folder as Scene_3.

At this time, you might have gotten a bit tired of cubes so delete the one you got, go to menu Window\Asset Store and search for "Unity chan free". Download and Import the free package.

Beneath your Managers gameobject in the Hierarchy window, create a new Empty and call it SpeechManager. Press the Add Component button search for Speech Input Source and attach this script as a component. Press the + button and add Keyword: Next and Key Shortcut N. Press the + button again and this time, add Keyword: Back with key shortcut B.

In the UnityChan folder, open Prefab and drag the prefab unitychan into your Hierarchy window. Select unitychan and in the inspector window and chose the Animator component. Press the small round button (nipple??) to the right of the Controller field and choose UnityChanActionCheck. Disable the Face Update and Idle Changer script. In the Transform component, set Position(0, -0.2, 3) Rotation(0, 180, 0) and Scale(0.2, 0.2 ,0.2).

On the unitychan prefab, Add Component Capsule Collider and adjust it around unitychan. (This is in order to be able to select her with your Gaze. You will see the shape of your Cursor change when you hover over her.)

Now on the unitychan prefab, select Add Component and add a new C# script which you may want to call SpeechIdleChanger. 

Open the ScriptIdleChanger C# script for edit, and write in a new using statement:

using HoloToolkit.Unity.InputModule;

It will now be easier to find the interface ISpeechHandler.

Implement the ISpeechHandler and give it some code until you end up like this:

using UnityEngine;
using System.Collections;
using HoloToolkit.Unity.InputModule;
using System;

[RequireComponent(typeof(Animator))]

public class SpeechIdleChanger : MonoBehaviour, ISpeechHandler
{
    private Animator anim;                           
    private AnimatorStateInfo currentState;
    private AnimatorStateInfo previousState;

    void Start()
    {
        anim = GetComponent<Animator>();
    }

    void Update()
    {
        if (anim.GetBool("Next"))
        {
            currentState = anim.GetCurrentAnimatorStateInfo(0);
            if (previousState.fullPathHash != currentState.fullPathHash)
            {
                anim.SetBool("Next", false);
                previousState = currentState;
            }
        }

        if (anim.GetBool("Back"))
        {
            currentState = anim.GetCurrentAnimatorStateInfo(0);
            if (previousState.fullPathHash != currentState.fullPathHash)
            {
                anim.SetBool("Back", false);
                previousState = currentState;
            }
        }
    }

    #region ISpeechHandler
    public void OnSpeechKeywordRecognized(SpeechKeywordRecognizedEventData eventData)
    {
        string myWord = eventData.RecognizedText.ToLower();
        switch (myWord)
        {
            case "next":
                anim.SetBool("Next", true);
                break;
            case "back":
                anim.SetBool("Back", true);
                break;
            default:
                break;
        }
    }
    #endregion ISpeechHandler
}

We also threw in a requirement of the Animator component and a reference to it for good measure.

Using SpeechInputSource together with the InputManager and using the ISPeechHandler interface is the newer prefered way to handle voice commands in the HoloToolkit. You should also have a look at the earlier KeywordManager which still functions very well by raising events you can utilize. If exporting to the HoloLens Emulator or real thingie, don't forget to turn on your Microphone in the Project Settings\Player\Capabilities section.  And please have a look at the Scenes: KeywordManager and SpeechInputSource in the HoloToolkit\Input\Tests\Scenes folder.

Gentlemen (and women), please start your engines, and if all goes well and your mike is up for it, you will now be able to tell Unity Chan to perform her next animation.