7
$\begingroup$

Some humans can lip-read fairly well: by watching someone who is speaking, they can tell what the speaker is saying (even without hearing the speech).

Has there been any work on building computer software to lip-read? In other words, given a video of someone speaking, is it possible to build software to infer what the person is saying (with access only to the video stream, without audio)? Has there been any research on this problem, or even deployed systems?

Background and motivation: In the US, certain laws may forbid recording audio without consent. However, there is generally no prohibition on recording video without consent of the people being recorded. (That's why you see surveillance cameras all over the place, and why they record only video but never audio.) I am curious whether technology has advanced enough that, solely from video, it might be possible for automated methods to tell what people are saying -- or whether that might become feasible in the near-future. And, apart from the privacy implications, such a technology might be pretty useful.

$\endgroup$

2 Answers 2

3
$\begingroup$

There appears to be some work in that area. See for example this paper and the references therein. There are also demonstrations of implemented systems on youtube, see for example this video

$\endgroup$
2
$\begingroup$

There is some recent work here:

LipNet: Sentence-Level Lipreading. Yannis M. Assael, Brendan Shillingford, Shimon Whiteson, and Nando de Freitas.

They achieve 93% accuracy on a corpus of 3-second videos with a talking head, compared to accuracy of 52% from experienced human lipreaders. They have a video demoing their results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.