For those of you who frequent this blog, you know a few days ago I wrote an introductory article on Kinect and XNA (link). In that article, I modified the Primitive 3D Sample from App Hub to render Joints from Kinect as Primitive Spheres. I’ve decided to build upon that sample, and leverage the Kinect’s NUI Microphone and the Microsoft Speech Recognition SDK to replace touch/keyboard input in the sample. I also refactored the previous sample a bit.

Before we get started, you need to make sure you have some pre-requisites installed:

• The previous sample
• Kinect for Windows Runtime Language Pack
• Microsoft Speech Platform – Software Development Kit
• Microsoft Speech Platform – Server Runtime

(Note: The SDK and Runtime are x86, as the Kinect Language Pack is only x86 for now)

Now let’s dive into it. First thing we need to do is add a few more using statements to the project:

using Microsoft.Research.Kinect.Audio;
using Microsoft.Speech.AudioFormat;
using Microsoft.Speech.Recognition;
using System.IO;

Next, add these variables under the variables we created last time:

KinectAudioSource kinectSource;
SpeechRecognitionEngine speechEngine;
Stream stream;
string RecognizerId = "SR_MS_en-US_Kinect_10.0";
bool speechNotRecognized;

These variables go with the rest of the fields we declared in the last tutorial. I’ll explain what each one of these does later. Next, take the Kinect code from our LoadContent function:

nui = new Runtime();
nui.Initialize(RuntimeOptions.UseSkeletalTracking);
nui.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(nui_SkeletonFrameReady);
nui.NuiCamera.ElevationAngle = 0;

And move it to a new function called InitalizeKinect. Your LoadContent function should look like this now:

protected override void LoadContent()
{
    spriteBatch = new SpriteBatch(GraphicsDevice);
    spriteFont = Content.Load<SpriteFont>("hudfont");

    primitives.Add(new CubePrimitive(GraphicsDevice));
    primitives.Add(new SpherePrimitive(GraphicsDevice));
    primitives.Add(new CylinderPrimitive(GraphicsDevice));
    primitives.Add(new TorusPrimitive(GraphicsDevice));
    primitives.Add(new TeapotPrimitive(GraphicsDevice));

    wireFrameState = new RasterizerState()
    {
        FillMode = FillMode.WireFrame,
        CullMode = CullMode.None,
    };

    InitalizeKinect();

}

Let’s dive into that new InitalizeKinect function:

private void InitalizeKinect()
{

    nui = new Runtime();
    nui.Initialize(RuntimeOptions.UseSkeletalTracking);
    nui.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(nui_SkeletonFrameReady);
    nui.NuiCamera.ElevationAngle = 0;

    kinectSource = new KinectAudioSource();

    kinectSource.FeatureMode = true;
    kinectSource.AutomaticGainControl = false;
    kinectSource.SystemMode = SystemMode.OptibeamArrayOnly;

    var rec = (from r in SpeechRecognitionEngine.InstalledRecognizers() where r.Id == RecognizerId select r).FirstOrDefault();

    speechEngine = new SpeechRecognitionEngine(rec.Id);

    var choices = new Choices();
    choices.Add("color");
    choices.Add("shape");
    choices.Add("wireframe");
    choices.Add("exit");

    GrammarBuilder gb = new GrammarBuilder();
    gb.Culture = rec.Culture;
    gb.Append(choices);

    var g = new Grammar(gb);

    speechEngine.LoadGrammar(g);
    speechEngine.SpeechHypothesized += new EventHandler<SpeechHypothesizedEventArgs>(sre_SpeechHypothesized);
    speechEngine.SpeechRecognized += new EventHandler<SpeechRecognizedEventArgs>(sre_SpeechRecognized);
    speechEngine.SpeechRecognitionRejected += new EventHandler<SpeechRecognitionRejectedEventArgs>(sre_SpeechRecognitionRejected);

    Console.WriteLine("Recognizing Speech");

    stream = kinectSource.Start();

    speechEngine.SetInputToAudioStream(stream,
                  new SpeechAudioFormatInfo(
                      EncodingFormat.Pcm, 16000, 16, 1,
                      32000, 2, null));

    speechEngine.RecognizeAsync(RecognizeMode.Multiple);

}

This is a lengthy function, but each part is important. Notice at the top, we initialize our Runtime and Skeletal Tracking features like last time.

The next thing to notice is the KinectAudioSource, kinectSource. This is how we access the four microphone array on the Kinect Sensor in code. Right here we create a new instance of KinectAudioSource. We also turn off Automatic Gain and Echo Cancellation, so the speech recognition capabilities can work properly.

Now we need to grab our Kinect Language Recognizer. We do that by checking what speech recognizers exist on the machine, and grab the Kinect Language Recognizer.

We then use the recognizer information we grabbed to create a new instance of SpeechRecognitionEngine. We then create a new Choices object, and add all of the words we want the speech recognition engine to recognize.

Next, we need to create a GrammarBuilder that will help us build the Grammar object used by the SpeechRecognitionEngine object. We set the GrammarBuilder’s Culture using the Speech Recognizer we got earlier, and add our words to the GrammarBuilder.

Next we’re going to wire up a few event handlers. I’m not going to dive into each event handler here, as the only one that’s really important is the SpeechRecognized handler. I’ll explain that in a bit.

Finally, we want to set our stream variable equal to the Kinect’s audio stream. We then tell to use the speech engine to use this stream for audio recognition. The last thing this function does is call the RecognizeAsync function, which tells the SpeechRecognitionEngine object to start looking for recognized words.

Let’s dive into our SpeechRecognized event handler:

void sre_SpeechRecognized(object sender, SpeechRecognizedEventArgs e)
{
    speechNotRecognized = false;
    if (e.Result.Text == "color")
    {
        currentColorIndex = (currentColorIndex + 1) % colors.Count;
    }
    else if (e.Result.Text == "wireframe")
    {
        isWireframe = !isWireframe;
    }
    else if (e.Result.Text == "shape")
    {
        currentPrimitiveIndex = (currentPrimitiveIndex + 1) % primitives.Count;
    }
    else if (e.Result.Text == "exit")
    {
        Exit();
    }
    Console.Write("\rSpeech Recognized: \t{0} \n", e.Result.Text);
}

This function is where the processing work of the recognized audio happens. The SpeechRecognizedEventArgs contains the result of the recognized speech. In the Result property, there is a Text property which you can use to compare against your choices. The code in here is pretty self explanatory. We simply change our rendering properties, currentPrimitiveIndex, isWireframe, and currentColorIndex, to be used later in the Draw function.

Those are most of the important changes. Take a look at the attached sample to see the rest of the minor changes to the Draw function and the other Event Handlers. You can download the entire sample here.

Let me know if you have any feedback on these samples, or if there is anything you can add. I’m always looking for advice, so I’m happy to hear from my readers.

Until next time,

Patrick Godwin