When being nice doesn't work, the alternative method of dispute resolution it to beat someone up!
However, might shouldn't make right! Before a duel, both parties agree to terms of the fight. This is intended to prevent those with the most power from going about uncontested.
Importantly, this means that creatures should be scaled appropriately for a duel. A creature should scale their HP before a fight; they agree to forfeit the match if they take a certain amount of damage, even if they would normally have plenty left over. As part of this, they might also agree to count non-magical damage as well, as Youkai of Touhou are typically immune to such attacks.
They will usually also avoid using some of their more dangerous attacks against lower level characters. It's up to the DM to decide what attacks and features would make a fight too one sided, and exclude them.
Also note that it's fine to use lethal force against characters who can't die, fairies in particular.
Of course, if they wish to make an example of a PC, they might not agree to a duel, or just blow the irritating PC away.
Also note that a game can be set before this rule is established, meaning that there are no restrictions on combat. It's advised that these rules are only used if players can be expected to follow them.