The past, present and future of gaming gesture recognition

Editor’s note: This piece first appeared on the iiNet Blog. Click here to see it in its original format.

Who would have ever thought that Tom Cruise would have helped define one of gaming’s newest innovations? Once labelled a fad, gesture recognition in gaming is here to stay, and it’s hard to imagine that the technology’s creators weren’t influenced by Cruise’s star performance in Minority Report.

You’ve seen that movie, right? Watching good ol’ Tom command a room full of super-cool looking displays, twisting and turning statistics, video feeds and the like simply by just his hands? Gold. I for one dreamed of the day where I could have a room in my own house, full of TV projections, where I could flick my hand to change channels, or throw my hands apart to zoom in on an image; in fact, Tom’s zoom-in, zoom-out trick later became a signature move on the smartphones we use today.

Minority Report-esque moves have definitely found their place in gaming, and today we’ll delve in to the past, present and future of the technology.

Gesture recognition’s humble gaming roots

The Nintendo Wii brought the gesture recognition to gaming’s centre stage; the combination of the Wii remote and the Wii sensor bar meant the game console could detect a person’s movements, and map them into a 3D computer generated world. Or, simply, by strapping on a Wii remote, those young and old mimicked sending a bowling ball down a lane in real life and marvelled as Wii Sports rewarded them (well, hopefully…) with a strike on the TV screen.

Sony PlayStation soon followed suit – its own Move and Eye technology are quite similar to Nintendo’s own gesture recognition work. The Move and Eye in Sports Champions let you become a virtual archer, as shown off in various episodes of The Big Bang Theory.

Not to be outdone, Microsoft’s Xbox 360 Kinect camera took gesture recognition one step futher. Its camera allows the Xbox 360 to recognise and track 20 individual joints of the human body as we groove along to songs in Dance Central. It also provides an experience quite similar to Minority Report, in that you can control the Xbox 360 menu with your hands instead of a traditional controller.

Where gesture recognition is headed

Gesture recognition is not part-and-parcel with gaming consoles – the new Wii U still uses the standard Wii remote and sensor bar, and Microsoft’s next Xbox is rumoured to introduce us to the Kinect 2 camera, a new version of the camera we use today.

The less obvious answers to where we’ll see gaming gesture recognition next are varied. The late Apple co-founder, Steve Jobs, told his biographer Walter Isaacson that he’d “finally cracked” the code to building the perfect TV set, adding that it would have “the simplest user interface you could imagine.” Before he passed, Jobs was rumoured to be heavily working with the same company that helped Microsoft launch its Kinect camera.

Now, it’s not certain that Jobs was planning an Apple TV controlled by gestures, but it’s a logical progression, according to Michel Tombroff, CEO of a gesture recognition software company called SoftKinetic (no relation to Kinect).

“The following consumer wave will likely come from the smart TV industry, who’s preparing to embed 3D cameras into the frame of the television,” Tombroff said. “This will allow gesture controlled interaction to replace or complement the remote control, but also offering games and other value added services, like video conferencing with background suppression or enhancement.”

“Minority Report is already outdated… [compared to] what our internal concept team comes up with every day,” Tombroff added.

Yeah okay… but, when do I get to be Tom Cruise?

If you’re looking to become Tom Cruise and command a sensory overload of screens, you’ll be waiting for a bit longer. Tombroff asserts that his technology will soon become “an extraordinary educational tool. For rehabilitation, industrial simulation, sport training [and] remote education,” but it seems like he’s referencing a one-camera to one-screen experience similar to the Xbox 360 Kinect.

In my own experience, I find it taxing to use Kinect’s hand gesture controls for in-game menus, instead opting for vocal commands if a controller isn’t handy. I think my thoughts are shared by those in charge at Microsoft themselves, as recent games like Kinect Star Wars and new video apps all seem to favour vocal commands like “Xbox, play video” rather than using a hand gesture. Apple’s own Siri technology is another example of vocal commands at work.

It seems for now that the problem with gesture recognition is that computers are limited in the way they can process information. “The raw data coming out from [a] camera requires some advanced filtering and algorithmic treatment to turn into usable and reliable data. This represent many years of software engineering on our end,” Tombroff said. In order words, if you wanted to use gesture recognition in your day-to-day work, you’d better be prepared for a sneeze to accidentally prompt the system you’re using to delete half off your email inbox.