Looking at the person who is talking helps us to hear them. We read that shape of their lips when a room is noisy and this helps our brain to pick up on what they are saying. It also tells us when they have paused and when is the best time to respond.
Just like looking at the person who is talking, we need to keep track of the next person who talking: the person who responds. They may have the same information that we can support, disagree with or extend.
When someone is talking they may not know if what they are saying is valid. Showing a response on your face gives valuable feedback.
If you are part of a group that is speaking to each other we show this with our body position. When we turn away or a distant this tells the group or the person that you don't care as much or that what they say doesn't really matter to you.
If you have something to say but someone else is talking you can show that you want to talk without interrupting them. This means that you are still listening to what they are saying.