Mercedes-Benz Intelligent Interior

Tools: C++, Qt, QML, JavaScript, OpenGL, openFrameworks, machine-learning

Officially released as MBUX Interior Assistant in 2019, it is a set of experiences enabled by an interior-mounted camera that take a step towards an intelligent interior. As opposed to forcing the user to learn yet another set of commands and behaviors, the vehicle anticipates and understands natural user’s movements and accomplishes the actions for them. ​

I was leading the concept development with a small team from start to finish. We did the ux research, idea generation, prototyping them, testing the technology and communicating within the company to get the project funded.

Design goals


To test these experiences, I developed a custom machine learning-based hand tracking algorithm using a depth camera in a real vehicle. This allowed us to quickly build functional interaction prototypes which were instrumental in refining the user experience through user studies, as well as for convincing the organization to fund the project and make it a reality.


Ultimately, a small subset of all the prototyped ideas ended up in the first product as the 2019 Mercedes CLA MBUX Interior Assistant.

Turn on the reading lights by casually tapping them
Automatically illuminate unoccupied passenger seat
The vehicle understands who is interacting with the touch-screen or a touchpad and customizes the interface to them and to the input device being used. For example, pre-selecting the climate or seat controls, adapting the keyboard for text input based on device, allowing text input for passenger while disallowing for the driver etc.
Adapting the UI to the input device in use.
A shortcut to a favorite function.


Car HMI 2019 - Most Innovative HMI Feature
AutoSens 2019 - Most Innovative In-Cabin Application