Hand Tracking Interaction Research

Tools: C++, C#, JavaScript, Python, Unity, Processing, openFrameworks, depth cameras (Kinect etc)

In the Intel RealSense group, we were bringing many natural user interface technologies to the public (e.g. gestures, speech, face tracking and recognition, eye tracking and emotion recognition). Since these technologies still didn’t have well-defined interaction paradigms, we ourselves had to understand the potential and the challenges in the context of desktop and mobile PCs. This work was then used to inform and empower the community to develop better experiences based on this technology.

The following are some of the projects I worked on.

User research

User research was instrumental in figuring out many of the following questions. We must have done several hundred user studies in the first few years.

Generating, prototyping and evaluating ideas

We produced a number of ideas and prototypes, some for apps, some interactions concepts, technical evaluations and demos. Some of the prototypes failed the ideas spectacularly, forcing us to revisit our expectations (like my Windows 8 Metro hand interaction attempt). Others were tested with users in order to validate the assumptions, evaluate the appeal or improve interactions.

Outreach Events

Outreach events like hackathons and competitions were crucial in order to get public feedback, teach them about the tech and generate ideas.

Games and Apps

We developed dozens of games and apps, all of which were user tested regularly during development. In the beginning we were optimistic with the productivity apps but after realizing the difficulties of gesture interaction, we mostly ended up with entertainment and communication applications (e.g. with background segmentation).


Since no known NUI design paradigms existed at the time, we quickly realized that rapid prototyping was necessary in order to test our own ideas and assumptions. This included developing our own computer vision algorithms while the underlying SDKs were still being developed.

High-level Findings



Gratuitous gestures

Children interact differently

Realism linked to expectations

Importance of good feedback

Design Guidelines

Ultimately, we gathered the best practices and recommendations from our experiences to share with the community of developers in the human interface design guidelines document. ​