It can be configured in the 'system.properties' subsection of the server configuration file (\standalone\configuration\standalone.xml or \domain\configuration\domain.xml):
<system-properties>
<property name="file.encoding" value="UTF-8"/>
</system-properties>