hey guys welcome to my collab tutorial for wave to lip this is a quick demonstration of how to use this collab okay before we get started with the collab running we first need to create a folder here called wave to lip and then the wave w has to be in cap and then another folder called wave to lip and then the lip and the w have to both start with cap so in this this folder it will contain the path for the pre tree train model you can download this pre-trained model here um this one you can download it by pressing this link and it will take you to this page and then you can just download it after you download it you upload and uh into this folder and this this folder will be your input file and then you can use the input files later when you are running that collab and then next you can get started with collab each cell has a play button thing so you have to just do in an order so you just press from the first one and then you can cue the next one by pressing it and then you can queue the next one cue the next one cue the next one wait but first it says here to go to this url browser so you have to sign basically sign in and allow access for um collab onto your google drive to access those files so after you allowed it it will give you an authentication code you copy this code and you paste it back here and then press enter after you press enter everything else will start running when it says mounted at a content g drive and yeah i can see everything start running let's queue more um q till here before the last let's try and here you have to um remember to type y to proceed and well here's a quick thing about um with a collapse is that it shuts down uh i forgot how long it shuts down after some time you don't use it so after it shuts down you have to redo everything here when you reconnect so it's kind of a hassle here but um i think it's it's still pretty good for a free freeze uh free platform for us to run stuff so now let me go prepare my input and input you have to name it into input audio dot wave and then for the input video which you're going to use to change the mouse shape will be named under input underscore vid mp4 it has to be wave um here uh i don't know about this actually the the main paper on the the main github has to be wav but you don't have you don't necessarily need wave and you can get wave by simply go to google and search uh mp3 to wave um tool or you can just record your voice in audacity and then export it as a wav file and upload it here into this wave to lip two two caps uh folder and as you can see um it takes quite a while to down uh to install all the environments and all the libraries for the environment and after these two finish running you can see it has this color uh this this thing where it tells you if you hover above it tells you last executed zero minutes ago it means it finished so now that everything is done you can start running and um lip syncing so all you need to do is run this one run this one and you wait and then once it's done you're ready to go it will be output under here um you will press this and then you open this folder and go to results and you to it will pop up here once it's done um lip syncing you can see that um rolling thing stopped and then back to this play button that means it's done it doesn't show up here because you need to refresh the folder so once you refresh it it will pop off here and then you can click and download and view it so you have to wait a few seconds and it'll download and you can play it it uses my voice to lip sync the face perfect and then if you want to do it again you don't have to necessarily start from the beginning if you already set up all these stuff without it having timed out or having the need to reconnect you can basically just delete all the files in here and then you move the new audio file and a new video here so let's say i want to change the name to input.audio to set it as the input audio and input underscore vid to set it as the input video and now you just have to wait a few seconds and then refresh this just in case and then you just gotta wait because sometimes it does not um sync right away so so you have to probably wait let's say uh 10 seconds one two three four five six seven eight nine ten okay let's do it well that wasn't ten seconds but uh i probably already got the new new video so this this what this command does is that it takes in the this input vid mp4 and input audio wave to sample data again so it's getting the new one to replace the old one so if this one doesn't update as fast and then it's still it will still use the old one because we just put in this just not not not a long time ago so it doesn't update as fast we might put an old one and we might train the old one which will make which will make it to be a waste of time so just wait a few extra seconds if you can and you just don't have to be surprised after everything is rendered and just your previous video and your previous audio so we're just gonna wait again it's pretty fast this time um you can see the progress bar here 30 wait where did it go it was it says 30 seconds but yeah you can see here it says approximate time and then this is elapsed time so approximate time it decreased really fast and now it's done you can see by um the thing stops spinning so you can see your new result remember to refresh refresh and then download the new result here another week um to easily play with this oh there you go there you go yeah that's it um oh i was gonna say oh i don't remember but yeah that's about it so thank you guys for watching check out my main main video on wave to lip and please check out my patreon or and discord if you have joined my discord if you have any problems i can't speak and i'll see you guys next time

[Music] [Music] me [Music] [Music] [Music] [Music] i enjoy it that's why i always have this kind of my halfway point but you know i always think of this view as the prize i get there i hope you liked the demo in that demo i actually demonstrated two very cool ai technologies the first ai technology i want to highlight and it's actually the main subject of this video is actually wave tweet wave tweet is like uh revolutionary you know we basically can take any video of someone actually speaking and you can feed it like any other you like with someone else speaking or even any sound and you can make this uh video or whoever is in that video the face uh the lips will actually sync to the sound that you give it it's not 100 perfect but it's uh way very close to that with a little bit more work uh i think this can be used in in many different scenarios like the one i've used today first thing i want you to do is to visit this website the github repository for wave to lip or you'll find all the source code and you'll find some demos there and also some information about authors of of this new uh ai model and they are like four or five uh researchers uh one of them is based in uh in cambridge i think uh or in oxford and uh the other four uh they are basically in india they are their work is amazing so that's the only thing i can say now i'm going to go into more detail of the actual demo they've created uh in a separate video uh what i'm going to do today is actually show you how i've combined this uh wavetrip ai technology with another technology a very cool ai as well which is from google and it's uh basically to generate synthetic voices because you heard me in a video i was speaking what korean spanish french and you know i could have gone on and on really i could have spoken like 33 language if i wanted to but obviously nobody would have that much patience for for me so i just did the sample right but you could see those uh voices in my opinion are very close to uh to the point where you might not even be able to recognize that these are syntactic voices and if you're not expecting it you might not even notice so uh the cool thing about wavenet is that it can generate a natural sounding speech and it can do it very fast and another very cool thing about it is that you can actually use this technology today using a google cloud api and i go into so much detail on how you can use these apis but the purpose of this video is actually just to give you a like a teaser you know of everything you can do and i'm going to be releasing which are going to go into more detail so the first video i will be releasing is the wave to whip video where i'm just going to focus on wave to it and i'm going to uh try the demo they have then i will also have a video on how i basically use the the google cloud apis for wavenet to generate synthetic voices and finally i'll show you how i put everything together where i actually i took this google collab notebook which was originally developed by the researcher team that did wavetlip and they've kindly given it you know for you to try you can actually try their website or you can also go to google app so just bear in mind if you try the demo in the website uh there is some limitations here you can't really give very large videos if you give very large videos like it would not work in fact i struggled to actually use this interactive demo website so if i were you i would go straight to the demo in google cloud so google cloud if you don't know it's actually a basically a notebook a python notebook which is hosted by google and gives you access to all their servers with a gpu or with a tpu so i highly recommend for today for this video i'm just going to show you the actual notebook i've created which is based on the notebook creator that you can find here the link is here this is the original google cloud notebook so it's a wave to it quick trial so this is the original google cloud notebook which you can try it's easy but i've actually modified this notebook and i added some additional functionality just to give you an idea you have here ability of calling uh the gcp apis which you do have to enable first and you do need to uh create like a secret key so you can actually use it in google speak that's called a service account and then you need to download the json file and then you need to upload it here anyway if you do that then you'll be able to use the google translation api the google the google text to speech api the google speech to text api and so on another cool thing i've added and i think that will be um that will be useful uh i've added actually a small script with ffmpeg ffmpeg is basically a video editing tool it's basically a script that you can use to do all kinds of stuff with video with this tool i actually can create a small video out of an image and you can see that we actually can animate like a cartoon like image and that's very cool and i'll show you how i've done that for one example so today it's all about showing the end result right but if you like this you will be able to go into more detail because i'm going to release videos on and i will show exactly how to develop this in fact i've recorded all of this it's just being edited right now and you'll be able to see all of it and i think it's quite interesting so you might learn a lot more out of that than just watching this video so check it out i'm going to be releasing it uh the week after i release this video so if you want to actually uh be up to date definitely consider subscribing because you never know when i'm going to release it it'll be uh sometime during the week so without further ado let's get started right so the first thing i do so i have this link here which i'm going to put in the description so all you have to do is basically click on the link which will be in the description of the video and you basically load this website and i'm doing it again so you can see it from scratch okay next you need to click connect yeah so it's trying to go connect so what's going to happen google is going to give you a machine that you can use with a gpu but just double check that you have a gpu there's a couple ways you can double check you have a gpu the first way you can double check is just check many sessions that's one way and you'll see here gpu okay but don't take my word for it the actual developers of the original notebook also put here a check for you so this will tell me if it's nvidia gpu because this notebook is only compatible with the nvidia gpu so if you run this ignore the warning it will tell you that this is a nvidia gpu so if this fails in any way and in shape or form then you know you are in the wrong server so you need to disconnect and and try again and the way to make sure you are on a gpu you basically just let me go here you go oh so you click change runtime type and then you can change whatever you want here you don't need the tpu you can just have a gp a tpu is basically uh like a tensor processing unit that has some bespoke basically that google design okay so we've done that another thing you will have to do you're going to have to upload a file into your google drive i'm not going to show you here how you're going to do that um because i'm going to it's already covered in the next video you'll see so i'm going to assume you have it there and i'm going to upload a service account for google and again i'm assuming you already know how to do that because i've showed i'm showing that in a separate video right so i have my json file here keep it safe i'm putting in a server because it's going to be recycled afterwards so you don't have to worry about it and when you create this service account just give you enough permissions okay so now we have the service account we just have to go through these steps here install all the dependencies all right so it has run this so you need to now click restart runtime you only need to do it once even though it asks you a couple of times and then you can go to the next step so you can skip this step here because we don't need it just yet this is for another type of video that i'm going to show you how you can create later oh i shouldn't have clicked here so here you need to make sure that your service account path is correct so based on the folder i upload uploaded here you'll be able to copy the path and then you place it here yeah then it will modify it so you do it on the right hand side you don't need to do it on the left hand side so next thing you need to do this is for you to basically specify the language that you want to output to so right now it's korean so it's if i let it be like this then it's going to uh generate the video with me speaking korean or someone else speaking korean so i'm going to leave it for now you can change with two different languages no problem i have to upload the video so i'm going to upload the video okay so i'm going to use a video clip where i speak english so it's uploading and while it's uploading i'm just going to show you the video okay so this is the video i've uploaded today i will show you how you can speak more than 33 languages with the google cloud platform so the video is already uploaded i'm just going to copy the path click play to make sure that it records all these variables so the first demo i'm going to show you will be to translate this video into caring so the next step is to basically mount your g drive basically you have to click on the link and you have to basically select the account you want to use and you have to click allow to uh basically trust all the permissions they need to give then you have to copy the code and and then you place the code here so now it's mounted i can go to the next step basically i'm calling the the wave to whip repository getting out the source code now check my g drive in my g drive i i've already created a wave to it folder with this basically this model file which has basically has already been trained so you click on this link and you download it and then you upload it to your g drive on the path that is mentioned here yeah so you need to create a folder inside your g drive i've already mounted the drive so i'm just going to copy the file okay so now i need to just run through these steps to install other the packages it will take a bit of time so i'm going to fast forward this so it has finished installing all the prerequisites and now we need to click on the next step which is basically downloads a face detection model which is used by the wave to leap algorithm to detect faces in a video just be mindful that if you actually use a video that has a section without the face most likely you'll have an error when it tries to do the lip syncing and now so i've added all these functions here these functions are used to interact with the with the apis in the google cloud okay first of all i'm actually at this function here is to extract audio from a video we have a method here to upload a file to cloud storage which we don't need actually a function to convert speech to text to translate to do a text speech which uses in this case webnet we have even a method to list all the voices available so we can pick a voice that we like okay so you need to make sure you load this so all the methods are loaded into memory so next thing we can do taking into account that original video is uh i've already shown you right uh it's i'm speaking english so we need to extract the audio from that video we need to basically transcribe it from audio to text it's done this is the translation and i'm going to probably have to modify it a little bit bear in mind that the transcription is not 100 correct so you need to make some changes yourself you can also add the delay um let me just check the delay i have here so i i've added a very simple delay of three seconds so it doesn't start straight away so it has a pause you can use ssml ssml is a way to describe like speech using text you can add pauses you can do many other things i'm just going to check what voices i have available for korean i'm just going to try one of them first so these are once i run this it's going to translate this transcript here in english to korean and i can listen to it now right so continuing uh so we can see already the sound is working but this is a female and obviously i'm not a female so let's see if i can do another voice and see if it's any better one thing that you you should bear in mind is that this text speech actually will obey any full stops or the comments that you have so you add some pauses so now that i have d i can basically go to the next step this is good so now i'm going to basically take this sound here and the original video i created and i'm going to replace the original sound with with a new sound in korean it takes a bit of time so you have to be patient in the meantime i'm going to look for a picture that i already took and that will be the next demo i'm going to do so wavenet plus wave to whip that's yeah this is good so the sections where i'm talking and the new audio is also uh talking uh then it's okay it will actually change my lips to match the the new audio but when there is any silence if the original video itself has any uh any lips movement right it will still be the original lips movement it doesn't necessarily make me quiet okay so we've done this this is the main demo i wanted to show you and now i wanted to add an extra here just for fun you can actually use this algorithm with cartoons and i've created the two cartoons of myself so one and two so here i'm going to use these two images to create two videos right where i'll be speaking something in either japanese or or english i'll just change the language for the sake of it and you'll see how it looks so without further ado let's do that so i'm going to just upload the images i've uploaded the two images so i'm just going to go back to that cell which i skipped initially the generate video from image so you just basically have to change the path just copy one of the images i'm going to copy the first path so i'm just going to create a small video you can specify how long you want the video for so i'm specifying 15 seconds so you can see it's quite it's very static there's nothing happening but you can see that this is how it looks like original let me copy the path here and i'm going to place it here to replace the other video click play so i'm going to change the language to japanese click play again to make sure it records then i'm just going to change to do to rerun this to translate again oh wait before i do that i need to make sure i get all the voices available so i get because i'm changing to japanese now i need to change it to another wavenet so i'm going to try and guess what the male voice is let me try this one it's quite quick okay so this is the sound now let's try and create a video out of this okay so it's done so let's see how it looks my first okay if i wanted to go any further i would try this one as well i know it works i've already tried it so okay so this is it this is the demo i wanted to give you i i didn't go into much detail i didn't go into any code but you know this is going to be the next episodes that you uh you can look forward to so i'll be going detail i'll basically be coding all these in real time so you might as well you'll be able to understand more or less the apis the gcp apis i used how i put everything together and i think you will get a lot out of it yeah that's it thank you so much for uh watching this video and i hope you got exactly what you wanted out of it and i promise you if you watch my next videos on this topic you'll get a lot more detail and you're going to be able to reproduce this yourself in fact you can already reproduces yourself because i'm going to give you the link to this google collab notebook and i'll put that into the description and if you already know how to work with the gcp you'll be able to do it if not then just wait for my next video i will basically go show you how to set that up alright thank you so much see you again soon happy coding