Let's look at this verse in context. Prior to this verse in Romans 12:20, the Bible says:
"Never pay back evil for evil to anyone. Respect what is right in the sight of all men. If possible, so far as it depends on you, be at peace with all men. Never take your own revenge, beloved, but leave room for the wrath of God, for it is written, 'Vengeance is Mine, I will repay,' says the Lord." (Romans 12:17-19)
We are told to let God avenge us, so that we don't try to avenge ourselves. God is the fair judge. He will repay wrongdoers and take vengeance for us.
Furthermore, God knows the circumstances of a person's sin, why a person sinned, and what temptations that person faced. Only God is the fair Judge and Avenger.
God judges unbelievers according to their deeds (Revelation 20:12).
If other people wronged them, and they did bad things in return to others, their punishment would be a certain degree.
However, if they were shown love and kindness, yet they did not repent but did bad things to the person who showed them love, their punishment would be much more severe.
So this is what this verse means. If we show love to an unbeliever and continue to love him even though he wrongs us, if that unbeliever never repents, then he would deserve burning coals on his head.
We (who show love) indirectly heap the coals onto his head.
In God's justice, if a person is shown kindness and love but remains hard-hearted, that person will suffer greater punishment. We would also be avenged if he did us any wrong.
This verse refers to another verse in the Old Testament, which says:
"If your enemy is hungry, give him food to eat;
And if he is thirsty, give him water to drink;
For you will heap burning coals on his head,
And the Lord will reward you." (Proverbs 25:21-22)
The Lord will reward us for our good deeds to that person, but will punish that person more for not repenting and doing wrong to us.
Does this mean we should show kindness in hopes that the wrongdoer goes to hell and suffer more punishment? Of course not.
Jesus teaches us to always do the right thing on our part:
"But I say to you who hear, love your enemies, do good to those who hate you, bless those who curse you, pray for those who mistreat you. Whoever hits you on the cheek, offer him the other also; and whoever takes away your coat, do not withhold your shirt from him either. Give to everyone who asks of you, and whoever takes away what is yours, do not demand it back. Treat others the same way you want them to treat you. If you love those who love you, what credit is that to you? For even sinners love those who love them. If you do good to those who do good to you, what credit is that to you? For even sinners do the same. If you lend to those from whom you expect to receive, what credit is that to you? Even sinners lend to sinners in order to receive back the same amount. But love your enemies, and do good, and lend, expecting nothing in return; and your reward will be great, and you will be sons of the Most High; for He Himself is kind to ungrateful and evil men. Be merciful, just as your Father is merciful." (Luke 6:27-35)
When we do the right thing, the enemy will be more likely to repent and believe. If we do the wrong thing, they get an excuse for their unbelief.
If we do the right thing and the enemy repents, then he is forgiven and saved.
If we do the right thing and the enemy does not repent, then he will suffer punishment, and God will avenge us and repay him for what he did.