Geoffrey Hinton says AI could already be developing emotions. Like fear, and greed and grief? Yes, and being annoyed. Suppose you have an AI and you are trying to get it to do some task, and it is repeatedly failing the task in the same way. You would like the AI to have learned after repeatedly failing the same way, to start thinking more outside the box. I have seen an AI do that in 1973 , but it was programmed to do that. Now you like it to learn that behaviour, and once it has learned that behaviour, after repeatedly failing it will be annoyed by the setup and try to change the setup. That's an emotion.
It's no different. When you take human emotions, there are two aspects to an emotion. There is the cognitive aspect and then there is the physiological aspect. When I get embarressed my face goes red. When an AI gets embarressed it's face won't go red and it won't sweat, but in terms of its cognitive behaviour it could be just like us.