In the first week of November, we took part in the first workshop on benchmarks for AI in Finance. We saw lots of brilliant talks from some truly distinguished speakers on the topic of benchmarks for AI in Finance. The overarching theme of the workshop was that it is crucial to build benchmarks so we can measure what success looks like and guide the problems in the field of financial AI. Here we briefly summarise some main takeaways.
Benchmarks for AI in Finance effort should be guided by the following principles:
Make the problem exciting
Involve the community: make the problem inclusive and flexible
Ensure a dynamic challenge to encourage innovation
Make the problem practically relevant. If the problem is not of practical relevance, is it supposed to drive development within a class of methods.
Efficiency: a good benchmark problem should not require huge amounts of compute to be able to attempt.
Challenges specific to the finance domain:
Proprietary nature: we need to discover ways to share
Datasets are typically small
Models are typically path dependent (haven’t seen variety of scenarios)
Principle of AI for Social Good
Create problems that will be beneficial for society (e.g. related to systemic risk).
Aim for impact
Formulate big goals: where do we want to be in 2050 when it comes to AI in Finance?