The Kinect for XBox One Workshop

Creating a Simple Kinect Application

So with little direct help available in the Microsoft documentation regarding creating Kinect for Xbox One applications using C#, and with Microsoft C# /WPF code samples that to a novice may seem overly complicated, how can we get started?

Fortunately there is a lot of information online that can be found using Google. A favorite site is CodeProject where you can find the article:

Kinect for Windows version 2: Color, depth and infrared streams

This application will display the Kinect sensor's color, depth, and infrared image streams. It is a C# WPF application. It is similar to but simpler than the Color Basics (WPF), Depth Basics (WPF) and Infrared Basics (WPF) SDK 2.0  samples.

In the following, we'll create this application using Visual Studio. The completed Visual Studio solution file is here: KinectStreams.zip.

To begin, open Visual Studio, then select File/New/Project/C#/WPF Application .Enter a Project Name, KinectStreams, then click Next and Finish.

Visual Studio creates the necessary files, and you can run the program by selecting Debug/Start Without Debugging.

A window with no content will appear. Close that window to end the program and resume working in Visual Studio.

Edit the code contained in the file MainWindow.xaml:

<Window x:Class="KinectStreams.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        xmlns:local="clr-namespace:KinectStreams"
        mc:Ignorable="d"
        Title="Kinect Streams" Height="350" Width="525">
    <Grid>
        <Image Name="camera" />
        <Grid>
            <StackPanel>
                <Button Content="Color" Click="Color_Click" />
                <Button Content="Depth" Click="Depth_Click" />
                <Button Content="Infrared" Click="Infrared_Click" />
            </StackPanel>
        </Grid>
    </Grid>
</Window>

Edit the corresponding C# code contained in MainWindow.xaml.cs:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;

// Requires adding Reference to Assemblies/Extensions/Microsoft.Kinect 2.0
using Microsoft.Kinect;

namespace KinectStreams
{
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        KinectSensor _sensor;
        MultiSourceFrameReader _reader;

        public enum Mode
        {
            Color,
            Depth,
            Infrared
        }

        Mode _mode = Mode.Color;

        public MainWindow()
        {
            InitializeComponent();

            // Obtain the sensor and start it up
            _sensor = KinectSensor.GetDefault(); // Different than article

            if (_sensor != null)
            {
                _sensor.Open();
            }

            // Specify the requires streams
            _reader = _sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color |
                                             FrameSourceTypes.Depth |
                                             FrameSourceTypes.Infrared);

            // Add an event handler
            _reader.MultiSourceFrameArrived += Reader_MultiSourceFrameArrived;
        }

        void Reader_MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
        {
            // Get a reference to the multi-frame
            var reference = e.FrameReference.AcquireFrame();

            // Open color frame
            using (var frame = reference.ColorFrameReference.AcquireFrame())
            {
                if (frame != null)
                {
                    // Do something with the frame...
                    if (_mode == Mode.Color)
                    {
                        camera.Source = ToBitmap(frame);
                    }
                }
            }

            // Open depth frame
            using (var frame = reference.DepthFrameReference.AcquireFrame())
            {
                if (frame != null)
                {
                    // Do something with the frame...
                    if (_mode == Mode.Depth)
                    {
                        camera.Source = ToBitmap(frame);
                    }
                }
            }

            // Open infrared frame
            using (var frame = reference.InfraredFrameReference.AcquireFrame())
            {
                if (frame != null)
                {
                    // Do something with the frame...
                    if (_mode == Mode.Infrared)
                    {
                        camera.Source = ToBitmap(frame);
                    }
                }
            }
        }

        private void Color_Click(object sender, RoutedEventArgs e)
        {
            _mode = Mode.Color;
        }

        private void Depth_Click(object sender, RoutedEventArgs e)
        {
            _mode = Mode.Depth;
        }

        private void Infrared_Click(object sender, RoutedEventArgs e)
        {
            _mode = Mode.Infrared;
        }

        // Convert a ColorFrame to an ImageSource
        private ImageSource ToBitmap(ColorFrame frame)
        {
            int width = frame.FrameDescription.Width;
            int height = frame.FrameDescription.Height;
            PixelFormat format = PixelFormats.Bgr32;
      
            byte[] pixels = new byte[width * height * ((format.BitsPerPixel + 7) / 8)];

            if (frame.RawColorImageFormat == ColorImageFormat.Bgra)
            {
                frame.CopyRawFrameDataToArray(pixels);
            }
            else
            {
                frame.CopyConvertedFrameDataToArray(pixels, ColorImageFormat.Bgra);
            }

            int stride = width * format.BitsPerPixel / 8;

            return BitmapSource.Create(width, height, 96, 96, format, null, pixels, stride);
        }

        // Convert a DepthFrame to an ImageSource
        private ImageSource ToBitmap(DepthFrame frame)
        {
            int width = frame.FrameDescription.Width;
            int height = frame.FrameDescription.Height;
            PixelFormat format = PixelFormats.Bgr32;

            ushort minDepth = frame.DepthMinReliableDistance;
            ushort maxDepth = frame.DepthMaxReliableDistance;

            ushort[] depthData = new ushort[width * height];
            byte[] pixelData = new byte[width * height * (format.BitsPerPixel + 7) / 8];

            frame.CopyFrameDataToArray(depthData);

            int colorIndex = 0;
            for (int depthIndex = 0; depthIndex < depthData.Length; ++depthIndex)
            {
                ushort depth = depthData[depthIndex];
                byte intensity = (byte)(depth >= minDepth && depth <= maxDepth ? depth : 0);

                pixelData[colorIndex++] = intensity; // Blue
                pixelData[colorIndex++] = intensity; // Green
                pixelData[colorIndex++] = intensity; // Red

                ++colorIndex;
            }

            int stride = width * format.BitsPerPixel / 8;

            return BitmapSource.Create(width, height, 96, 96, format, null, pixelData, stride);
        }

        // Convert an InfraredFrame to an ImageSource
        private ImageSource ToBitmap(InfraredFrame frame)
        {
            int width = frame.FrameDescription.Width;
            int height = frame.FrameDescription.Height;
            PixelFormat format = PixelFormats.Bgr32;

            ushort[] infraredData = new ushort[width * height];
            byte[] pixelData = new byte[width * height * (format.BitsPerPixel + 7) / 8];

            frame.CopyFrameDataToArray(infraredData);

            int colorIndex = 0;
            for (int infraredIndex = 0; infraredIndex < infraredData.Length; ++infraredIndex)
            {
                ushort ir = infraredData[infraredIndex];
                byte intensity = (byte)(ir >> 8);

                pixelData[colorIndex++] = intensity; // Blue
                pixelData[colorIndex++] = intensity; // Green   
                pixelData[colorIndex++] = intensity; // Red

                ++colorIndex;
            }

            int stride = width * format.BitsPerPixel / 8;

            return BitmapSource.Create(width, height, 96, 96, format, null, pixelData, stride);
        }
    }
}

In this program the MultiSourceFrameReader returns and displays color, depth and infrared frames.  Clicking one of the three buttons changes the display to the associated stream.

Each frame's pixel data is converted to a BitmapSource which is then be assigned to the Source property of an Image control.

The rate at which frames are generated is approximately 30 frames per second, or one frame every 0.033 seconds (33 milliseconds). This is close to the frame rate of a standard webcam.

 

 

 

The FrameSourceTypes Enumeration indentifies all possible Kinect sensor frame types.

In the next step we'll display the skeletal data returned by the Kinect sensor.