16,659 views 22 Jul 2023

In this video I will point out the key features of the Llama2 model and show you how you can run the Llama2 model on your local computer. I will show you how the Llama2 model compares to other closed-source models like GPT-4 or PaLM-2-L as well as other open-source models like Falcon or MPT. But this is just to get a better understanding about the Llama2 model. The main part of the video is about how you can run the Llama2 as well as the Llama2-Chat model on your local computer. I hope you will like how easy it actually is! Also, I’m still amazed that we can run a free alternative to ChatGPT on our local computer. Let me know in the comments what you think about the new Llama2 model. :-) My Medium Article for This Video: https://medium.com/p/41b7638033a8 Request Access To Llama2 https://ai.meta.com/resources/models-... Llama2-7B Hugging Face Repository https://huggingface.co/meta-llama/Lla... Github Repository https://github.com/thisserand/llama2_... Miniconda Installation Guide https://docs.conda.io/en/latest/minic... Hugging Face Access Token https://huggingface.co/settings/tokens Llama2 Paper https://scontent-fra3-2.xx.fbcdn.net/... 00:00:00 Intro 00:00:35 A Look at the Llama2 Paper 00:08:25 Request Access to Llama2 00:09:49 Installation 00:13:40 Usage 00:44:27 Outro Stay in Touch Medium https://medium.com/@martin-thissen LinkedIn https://linkedin.com/in/mthissen135/ YouTube Of course, feel free to subscribe to my channel! :-) Of course, financial support is completely voluntary, but I was asked for it: https://patreon.com/MartinThissen https://ko-fi.com/martinthissen

Chapters

View all

a few days ago meta AI introduced their llama to open source large language model and the great thing is that is free for research and Commercial usage so in this video I'm going to show you how you could build your own llama2 chatbot in Python using streamlit and replicate and without further Ado let's dive in so this is the app that we're building today and for a full description of this also check out the blog which also provides you the same information but then in written form so I'll provide you the link to this in the video description so let's have a quick overview of llama2 it was released on July 18th is democratizing access to the large language model it's transparent and it has responsible use of AI as outlined in the responsible use guide here are five key features of llama2 lamma 2 outperforms other open source large language models and benchmarks for reasoning coding proficiency and knowledge tests the model was also trained on almost twice the data of version 1. totaling two twillian tokens and there are over 1 million new human annotations and fine-tuning for chat completions the lava 2 model comes in three sizes seven billion parameters 13 and also 70. so all three versions are on the replicate platform and I'm going to show you that in just a moment llama2 also support a larger context length of up to 4096 tokens and as already mentioned version 2 has a more permissive license and that it could be used for research and also for commercial use so let's have a quick overview of the app that we'll be on today so what the user will provide are the API token The Prompt inputs which will be taken to the streamlined app and use as input it will make an API call to replicate and the Lama 2 model will be used for generating an lrm generated response and then this will be displayed in the streamlined app and this is a full view if you like to have a look and so let's have a look at the repo so this is the code and this is the repo so in the readme file here it provides you with the high level overview so the app comes in two flavors The Standard Version and also the pro version let's have a look at the Standard Version and then if you like to have more customization then the pro version might also work for you but the essence are the same let's take a look at the requirements.txt file and you'll see here that it will require two python libraries particularly streamlined has a low code web framework and replicate which holds the large language model let's have a look at the app file and you can see here that it is using up 68 lines of code and so let's compare it with the lamma2 chatbot deployed app here and it is accessible from lama2.streamlet.app and if you like to play around with this you could enter your replicate API token here so before going and taking a deep dive into the code let me show you how you could generate your own replica API token so you can see here that I'm already logged in but let me show you when I'm logged out so this is the main page or replicant go to sign in click on it and then you want to sign in with your GitHub account click on it because I'm already signed in to GitHub this should work automatically so in the dashboard it tells me what are the recent predictions that I've made and also the time consumed for each of them so be aware that larger model will take more seconds for the runtime and it will be more expensive so these recent calculations were made using the 13 billion parameters and if you're using 70 billion parameters it will get much more expensive so therefore I think it's worthy to note that before doing heavy coding but of course the larger model will provide you with better performance but the best way is to try them out and see which one works best but that's the topic for another video so click on the user icon here and then it'll give you a drop down go to the API token click on new token type in the name of a token name here which could be anything I'll type in key click on create token and then I'll just copy this but it's also worthy to note that when you signed up for a replica for the first time you'll be provided with a trial key whereby you don't have to pay for the usage but then after a certain amount of time you will run out of the trial key and therefore you have to link your credit card information in the billing here and now that we have the replicate API token let's head over back to the streamlined app and let's paste in the token here hit on enter and then you'll see here that the authentication mechanism for the API token was successful and now it is telling us to enter our prompt message let's go ahead and enter what is python and then you're gonna see that it is generating the response and it's printing out the response token by token let's type in another question to try out whether it is retaining the memory of what we've asked it second question is what is trimlet and let's ask it a third question what questions did I asked and there we go they are out of the questions that we've just asked the chatbot so let's have a look here also the clear chat history function if we click on it let's see what happens so the messages here the history has been deleted right and so let's take a walk through look into the code so you can see here that the code spans 68 line zip code AS already mentioned so the first three lines will import these trimlet replicate and the OS module then on line number six we're going to define the page title to be llama2 chatbot using also the llama and the chat emojis in the sidebar we're using the whip statement and then we're defining the following we're going to define the title for the app which is displayed here using St dot title and then we're using two statements here if and else in order to perform the authentication mechanism for the API token so there are two ways for you to enter the API token the first one is you could enter the API token in the back end directly on these trimlet Community cloud in the settings and then enter the API token as the secrets however that approach will require you the developer to pay for the token usage however you could also give the responsibility to the user so users will use their own API token which I've shown you in just a few moments ago in order to perform the model calculation and so the cost will be incurred by the user so the first if statement here we're going to display it to be successful if it is detected in the secrets otherwise it will have the text input box asking for the replicate API token which is this one the box here and then the message if it is successful right if not then it will have a yellow box telling the user to enter their credentials and then here we have a link back to the blog this blog and then here is the clear chat history which is provided down below I'll talk about that in just a moment and so we're going to use the OS dot Environ method together with the name of the API token here in order to save the information from the replicate AI either coming from the secrets or coming from the text box that the user has entered and we're going to save that into the environment variable so that will allow the replicate function which we'll talk about forthcoming here to use the API token in their calculation right so let's move on and lines 23 to 25 here we're creating the session state for messages so this will keep tabs on the chat history and then we're going to initialize that with the message of how may I assist you today in the role of an assistant then we're gonna use the St dot chat message together with the fc.write in order to write messages by iterating through the messages section States so this block will allow us to display all of the messages in the chat history which is saved into the messages session State and so the clear chat history button that you've seen is defined here on line number 32 33 and 34. so we're using the Callback function via the on click here in order to perform the deletion or clearing of the chat history from the section state lines 36 until 48 will be the function or custom function for generating the llama2 LOM model response and the codewords refactored from the GitHub repo of the a16z here where they also created a llama 2 chatbot and they are the same team that uploaded the Lana 2 model to replicate and the model that we're using in this particular tutorial so here the string dialog here this is the pre-prompt and it's providing an Instagram action to the large language model that it is to be a helpful assistant and that it does not respond as user or pretend to be user and it will only respond once as an assistant and depending on whether the role is as a user or assistant the if else here will handle that and then the generated response will be using the replica.run function and then here we're specifying the name of the lamba 2 model that we're using and the one that's specified here is the 13 billion parameters as noted before there are three different sizes of the model 7 13 and 7D and in the input here are the prompt and I've created the F string in order to combine the string dialog here which is the pre-prompt and also the prompt input which is the question coming from The Prompt here which I'll describe in just a moment that the user has entered in the text chat input here and then afterwards we're specifying the model parameter and the temperature here is specified as 0.1 top P parameter is specified as 0.9 Maps link here is specified as 512 and be note here that the larger the length for the generation of the response the more money that it will incur as well because it would take longer to generate and much more computational resource to generate so feel free to modify this to a lower number when you're trying out or testing this particular app and so this function will be used later on and I'll let you know in just a moment lines 50 until 54 is the user provided prompt and so it is defining the logic for the chat input box here that we're going to see at the bottom of the chatbot where the user will enter their prompt message so you'll notice here that when I hover it won't allow me to enter anything because the API token was not yet entered right and if I enter the API token then on enter it validates the API token now the chat input box will be activated and now you can enter your message a very cool mechanism in order to perform some validation of the token and so here we're going to take the input message or the prompt message from the user we're going to append it or save it to the session State messages variable and it's going to be saved in this format and then we're going to print out the message also by using St dot chat message and then we're going to specify the row to be user and the message will be printed out using sc.write and the prompt message will be printed out alright and here comes the fun part the generation of the LOM model response so lines 56 until 68 will perform the generation of the model response so here we're using an if statement to detect whether the row is an assistant or not and if the last message is not from the assistant it will trigger it to generate a model response meaning that if the last message is coming from the user then it will know that it should generate a response and then it will become the last message and if you ask it another question then it means that the user will become the last message which will trigger this again and so this will be a revolving Loop and so the chat message here will be using sd.chat message and then we're assigning the row to be assistance and when it is generating the response you're gonna see a spinner saying thinking and it will be spinning as well and for the response we're assigning it to the response variable and remember the custom function that we generated earlier on in lines 38 until 48. we're gonna use it here right and then we're going to take the user prompt provided from the chat input box here we're going to give it as an input parameter for the generate response function and when it is generating the response it will be creating a generator object which is not a string but then you need to iterate through the object as I'm showing you here you're iterating through the object and then you're creating firstly a empty string and then for each iteration you will add the text Chunk into the full response variable and after that it's finished you will print it out right but while you're generating the token on the Fly the token will be generated one by one it will be printed out to the placeholder here using the St dot empty right because here we use placeholder.markdown and then we're printing the full response as it is being generated so you're gonna see that streaming effects similar to chat TBT and then once it is being generated to the full list it will be printed out for the last time here okay I'm going to show you in one moment and then finally we're generating the structure for the message here using the full response and then we're saving looking at append again to the message session States and therefore we're able to give memory to the chatbot type in a question and the response is being generated you see the spinner working it's printing out the thinking message and then you're gonna see that the text Chunk is being printed out one by one so there you go you could build this llama 2 chat bot in 68 lines of code and if you'd like to customize it further you could try out version two and this gives you the pro version so you're going to see that it provides the select box here allowing you to choose different versions of llama2 and here we have three versions The 7 13 and 70 billing parameter version and then we have the slider to allow you to customize the model parameter so let's have a look it's llama2 Dash Pro and so this is the pro version so similarly it has to API token input box here but then you are able to select the parameter and also the llama2 model that are provided here and finally we also have the clear chat history function and as always we have the chat input box below here and so the links to all of these resources the demo app created here using streamlit and deploy it on the community Cloud the blog the link to the article for the release replicate platform will all be provided in the video description and if you like a quick video on llama 2 I'll also provide you the link to that which was created on the data Professor YouTube channel and so this provides you a quick look at how you could run llama2 using replicate in only a few lines of code as shown here so all of the links are in the video description so let me know in the comments section down below what awesome app that you're building and so until next time happy streamlining