Generative AI PCG Graphs in Unreal Engine

This is a project that I completed during my time at Tesla working as a Digital Media Developer Intern on the Style and Design team making tools and other helpful assets for an internal tool. As our team was considering using Unreal's Procedural Content Generation tool to create procedural assets, I created framework to not only integrate procedural assets into their pipeline, but also experimented with potentially using generative AI to generate procedural assets within Unreal PCG's framework.

Before I explain the project itself, I want to state that I was given permission by my team to present my work and everything on this page.

As a quick reason as to why we decided to use PCG instead of using a more powerful procedural tool such as Houdini Engine, Unreal PCG is not only integrated within UE's editor already, saving the hassle of importing and connecting with Houdini Engine, but it also is quick to use, and its parametres can be modified in runtime.

However, there are certain limitations to PCG, some major ones being that for new users, figuring out which nodes to use and which parameters to modify may be less than intuitive, which can lead to users spending hours to learn which parameters that they want to modify for their ideal environment.

Hence, the purpose of this project is to create default PCG environment assets to use in a PCG asset library.

Goal

The goal of this project was to provide user, especially those who are not quite familiar with procedural generation, base environments to work with for custom projects, such that certain frequently used parameters are already tweaked on certain frequently used nodes. As such, my goal was to train a generative AI model on certain frequently used nodes and certain frequently used parameters to generate new assets for users that would look the most "like" a certain environment that they wanted.

Basic Architecture of Project

Extract data from PCGs
Normalize data
Train custom VAE model on normalized data
Regenerate new data and put back into UE

Template

The majority of the first part of this project was spent researching different methods to set up the datasets to train for a generative model. Ultimately, because Unreal Python unfortunately does not support mesh modification in real time from terminal, nor does PCG support adding and removing existing nodes, I had to use a template for all of my environment datasets to sample parameters from. The parameters would be categorised by type of mesh sampled (refer to image on left for the template, in which each row denotes a modification for each mesh), and within each type would be the nodes present in that category modifying that mesh.

Below is the order in which my Python scripts would be executed from creating the data to training it within a simple VAE model to finally piecing together the generated outcome and reintegrating it back into UE.

PCG Graph Parser and Data Schema

To obtain the data required, my parser script iterated through each node to extract relative values that matched the parameters that I manually set based on existing values in the template. The parser would find static meshes by keyword mapping to type category, where types of meshes corresponded to types of values, saved under a type category.

E.x.: “Trees” is a type category that has have a set of optimal values for each node in the template, as well as its own associated static meshes

The goal was to use AI to learn these optimal values associated with type.

Schema:

Types: “Grass”, “GroundItems”, “Structures”, “Trees”
Meshes: set for each of the types nodes
In the PCG graph, types are organized by row, so each row of nodes generates specific kind of mesh
Nodes for each type: “SurfaceSampler”, “SpatialNoise”, “DensityFilter”, “TransformPoints”

To prepare data to be read and trained on by model, data must be normalized, and placed into vectors based on parameters. For each graph, for each type within each graph, nodes and their parameters are turned into vectors.

VAE Model

VAE – Variational Autoencoder Model

A VAE model uses a decoder to parse data and form a latent space created from vectors, then uses an encoder to generate values from the created latent space. A simple diagram on how a VAE works is shown on the right.

I decided to use a VAE for this project because it is better for handling smaller datasets due to its ability to normalise gaps that smaller datasets may leave during training.

Regenerating Data

This step reverse engineers the output that the encoder from the trained VAE model gives. The values are given in vector format, then reconverted back to the schema format that will then be used to update the PCG template graph's values in Unreal. With a preloaded VAE model, running the decoder each time can produce unique values for default PCG graphs.

Some Samples of the Graphs that I Generated

As you can see, the results are not exactly... ideal. Hence, I introduced the concept of a discriminator to the existing model to train "undesirable" data to be avoided by the VAE model, and to make it more "optimised".

Adding a Discriminator

Adding a discriminator would require for there to be an additional section to my data: differentiating the desirable data from the undesirable data. I went back through all of my graph samples to denote ones with less than optimal and illogical placement of props as "undesirable", then I placed that group of graphs to be trained through a discriminator. The results of that discriminator model would then be utilised in the VAE model from earlier, such the discriminator can identify bad values with a high degree of accuracy and at least prevent egregious values from being generated in the resulting graph.

To the right is a simple diagram portraying a simple way on how the discriminator is set to work.

To the left is an update to the script workflow with the addition of the discriminator.

Some Samples of the Graphs that I Generated with the Discriminator

Next Steps

Although the graphs above exhibit a better overall scatter than the previous method without the discriminator, there are still some optimalities that I would like to consider when it comes to the aestheticness of the generated graph. In that regard, some of my next steps includes adding perhaps a model to train on the aestheticness of certain arrangements by associating images with certain scores. I could use a preexisting model for this task. Another way in which I would like to refine the results is to continue tweaking the values for the discriminator and the VAE model, in order to optimise the loss values. As I am still in the beginning phases of learning AI, I do not have the most robust understanding of which values are optimal for minimising the loss function of a model that trains a small dataset, like the custom one that I am working with. Finally, there is always room to refine the pipeline of data coming in from PCG, as I continue to explore Unreal's extensive Python documentation and see if there are any further ways for me to refine the data preprocessing for my model.

Page updated

Report abuse