Example-based cross-modal denoising (Technion)

Contemporary solutions to audio noise rely on unimodal (audio-only) input. But denoising based on audio-only data is very difficult when the noise source is non-stationary, complex (e.g., another speaker or music in the background), or strong and not individually accessible in any modality (unseen). In addition, this approach neglects the advantages of having an accompanying video from an integrated camera, which is becoming more and more widespread in modern communication devices.We have developed an algorithm that cleans an audio signal with the support of the accompanying video. Our approach uses an example-based technique: for a given corrupted segment of audio, we use several relevant clean audio segments, found by correlating the video (visual) content. A training movie having clear audio provides these cross-modal examples. We can then use these examples to clean the noisy segment by a number of different approaches. Of the numerous available options, we can clean the audio either by averaging them, replacing the corrupted piece, or designing an optimal filter based on the examples. In testing, cross-modal input segments having noisy audio rely on the examples for denoising. The video channel drives the search for relevant training examples. We demonstrate this in speech and music experiments.

Exploits the advantages of having accompanying video input
A natural and highly effective process in technologies with an integrated camera, microphone, and local and strong computing capabilities, such as smartphones

Teleconferencing
Telephone and cellphone communications
Post-processing of video content for forensic needs, and more

COM-1504

Israel