EchoTrack Smart Camera

The problem we’re trying to solve is the issue of tracking the active speaker in a room full of speakers and keeping the camera pointed and centered on them. This is a close relative of the cocktail party problem, where one has to keep track of a single stimulus in a noisy environment. In much the same way, our system has to keep track of the loudest speaker in the environment, while picking this out from other potential sources of sounds, such as someone moving their chair or a whispered comment from a colleague. This presents some challenges for a simple DOA algorithm that points the camera at whatever’s noisiest at a given moment, especially if you want other features such as transcribing what’s said.

Our solution is to use edge AI to sharpen our estimation of where the speaker is in a given space, as well as distinguish speech from other background noise. We shall have an array of microphones to do a regular DOA algorithm to pick up where the loudest sound in a scene is, then augment that data with AI to make sure we’re picking up speech. We also intend to add extra features, such as automatic camera focus and AI-powered transcription of what’s being said and who’s saying it. We’re aiming to do this all on one board, the Nuvoton M55 Arm board that was provided for us as a showcase of the board’s capabilities.