1.3 Detection

Here we see two actors in front of a microphone. When the words scrolling along the rythmo band reach the sync bar, the actors read their respective lines. The red “on air” light indicates that the microphone is on and recording. The Time Code serves as an additional positional cue for the actors, which they can use to remember details about the scene. This is an idea similar to a piano roll, karaoke, or Guitar Hero.At the detection stage, an operator transcribes all of the original dialogue in perfect sync with the movement of the actor’s lips, using specialised software, onto a virtual rythmo band³. All mouth sounds are “detected”, vocal or otherwise: along with the dialogue, all openings and closings of the on-screen actor’s mouth are notated, including lip smacks, breathing, chewing, whistling, kissing, screaming, smoking, etc. If an actor in a scene is not visible in a shot but is still heard, that too is indicated on the band. Crowd walla⁴ - for example distant conversations in a bar that are not essential to the story but are still audible - is also detected and marked⁵. The detection process yields two versions of the script: an exact and frame-accurate transcription of every word of the original dialogue, and a version indicating only mouth openings and closings, breaths, and all other non-language specific sounds. Having these two versions facilitates not only the adaptor’s job of translating the text, but also of finding phrases that fit best with the on-screen actor’s mouth movements. In the past these two versions were painstakingly done by hand [8], but they are now automated. Once ready, the same rythmo detection can be used to adapt as many new languages as desired.

[EXAMPLE PICS OF THE TEXT WITH / WITHOUT MOUTH FORMATION MARKINGS]

As a starting point, the detector receives a script from the client, which is prepared according to the requirements of the dubbing software. As the detector plays back the movie, she cues the phrases from that script to the action on the screen such that they mach exactly. In addition to the text, using specialised notation the detector can indicate whether the mouth of the speaker is visible or not, whether a character is breathing in or out, or is breathing with the mouth open or closed. Such a level of detail is time consuming to prepare, but will add greater realism to the dubbed program than other foreign dialogue replacement methods. More importantly, it increases the efficiency of the performer dramatically at the recording session.