Saturday, March 26, 2011

Microsoft explains the tech behind Kinect

18:00 25 March 2011

Microsoft sunk a huge amount of R&D into its Kinect depth sensor, but with over 10 million sold since it launched in November the investment has definitely paid off. Now, the people behind Kinect at Microsoft Research Cambridge and the Xbox Incubation team have published a paper explaining how they did it.

Their skeletal-tracking algorithm works by assigning each pixel in an image to a particular body part, creating a fuzzy picture of a person where the depth of each point is known, thanks to Kinect's infrared sensor. The algorithm can also be adjusted depending on the application, so a game that only tracked the upper body could merge the lower body into one single part.

Each pixel is first evaluated for how well it fits certain features - for example, is the pixel at the top of the body, or the bottom? The score for each feature is then combined with a search through a "randomised decision forest" - essentially a collection of decisions that asks whether a pixel with a particular set of features is likely to fit a particular body part.

The researchers trained the system on a collection of motion capture data. They initially collected around 500,000 frames of data, cataloguing movements such as dancing, kicking and running, then narrowed this down to just 100,000 frames by discarding any that were closer than five centimetres together.

Once the body parts have been identified, the system then calculates the likely location of the joints within each one to build a 3D skeleton. The Xbox runs the algorithm 200 times per second, which is around ten times faster than previous body-recognition techniques, so players can easily be tracked fast enough for their motions to be incorporated in to games.

So what's next for the team behind Kinect? They plan to study how to potentially increase its accuracy by directly calculating joint positions, ditching the body-part detection all together. Kinect 2, anyone?

Source and/or read more:

Publisher and/or Author and/or Managing Editor:__Andres Agostini ─ @Futuretronium at Twitter! Futuretronium Book at