THE FACE RIG PROBLEM

As mentioned in my Simulation Research section, my initial foray into Unreal Engine was not without difficulty. I assumed, that getting a rig into Unreal Engine would be as simple as exporting as an FBX and importing it into Unreal Engine. Unfortunately for me, my choice in Rig would make it so that things were not so simple...

diagnosis

Unreal Import with missing face geo

The first signs of difficulty came when I started looking into cloth simulations in Unreal. Upon importing my rig as an FBX, I was met with a peculiar problem. None of the facial geo was showing up. This included eyes, face, teeth, tongue and eyelashes. I ensured that I had selected the correct geometry for export and after several attempts I still could not get the face to appear in Unreal.

Confused, I decided to try exporting the 'Game Ready' version of the rig. I knew that many rigs come in two formats, the Game Ready versions being designed for their use in game engines such as unreal. However up until this point, I had assumed that the differences were surrounding poly counts in order to achieve better performance in a real time renderer.

Exporting the game ready version, did allow me to successfully import the rig into Unreal Engine however I was now faced with a difficult problem. As I had discovered in my Rig Research, the game ready version of my Sylvanas rig did not come equipped with a face rig. At the time, this didn't seem like it would be an issue, as I assumed I could just use the regular version instead. However my assumptions on what a 'Game Ready' rig was, lead me to an incorrect conclusion. I believed that since I was using Unreal Engine as a rendering tool and not for making games, the optimizations of a game ready rig would be irrelevant to me, and I could simply use the main rig. This was my thinking during the original research phase, however I now found myself in a position where this wouldn't be the case.

As it turns out, Game Ready rigs have nothing to do with polycount optimisations, and are more focused on joint hierarchy's and stripping away any Maya centric features. Unreal Engine only supports a single joint chain (which some rigs don't have) and Game Ready rigs are constructed with appropriate joint hierarchy's. The other element to Game Ready rigs, is removing features that are specific to Maya that a Game Engine cannot read. And this is where my problem lay. The facial rig on my Sylvanas character was constructed in a way that Unreal Engine could not recognise. 

And so I set out on a tireless 10 day research rabbit hole, filled with trial and error, experimentation and frustration. As I attempted to find a solution to the Face Rig Problem...

insanity

My first instinct towards solving the problem centred around baking the blend shapes. I knew that the face rig controllers deformed the face via driven blend shapes, and so I figured that it would be as simple as baking the vertex animation and calling it a day. Unfortunately for me, it wasn't so simple. As it turns out, whilst an FBX export does support blend shapes, it does not support vertex animation. Meaning, getting the baked animation into Unreal was going to be difficult.

Since my initial idea didn't seem to work, I found myself in a tough spot. Hours upon hours were spent googling and scouring forum pages for potential solutions, however since I didn't really know what the true cause of the problem was, it was hard to search for the answers.

Snippet of the rigs blend shapes

"Face geo not appearing in FBX export", "How to export face rig to Unreal", "How to convert vertex animation into Blend Shapes"

Countless stabs in the dark trying to stumble across some sort of solution. I wondered if it was possible to convert the vertex animation back into Blend Shapes, however this didn't seem viable. The only information I could find on exporting vertex animation was by baking a point cloud cache but after some testing I couldn't get this to work.

One file format that I know does support vertex animation however, is Alembic (.abc). I was hesitant to use Alembics since they've been weird for me in the past, but upon testing the export, I was able to successfully get the model and animations into Unreal Engine. Although, this presented a separate problem. Importing animated alembics into Unreal requires them to be imported as a Geometry Cache, and this would mean that I would not be able to use Unreal for my cloth simulations ( as the cloth paint feature requires the object to be a Skeletal Mesh). This posed me with a tough decision. Either return to Maya and figure out the nCloth method of simulation OR keep working to find an alternative solution for the face rig issue.

I decided to keep looking, figuring that if all else fails I could try and find some way to work with the Alembic.

Sylvanas paint skin weights

I then began looking into the possibility of rebuilding the face. By this point I had confirmed that the deformers and history on the face were what was preventing it from being exported, since deleting the objects history allowed for an export. Naturally, this breaks all the connections and skinning, but I wondered whether it was possible to transfer the skinning and rebuild the blend shape controllers.

Once again, several hours of attempts left me empty handed. I had tried duplicating the geo and transferring the skin weights, as well as importing the blend shapes over, but I was starting to realise that I was in over my head. I really had no idea how a rig like this functions, and therefore trying to replicate it was pointless. 

I was beginning to lose hope at this stage, and was contemplating contacting the original rigging artist to see if they had a solution. I decided to search for some contact details, and when googling the Sylvanas rig again I found a separate download page uploaded by the rigging artist himself (Truong CG Artist). On this page he did have some contact details, but also a link to some How-to's that had not been present on the site I initially got the rig from. Considering he had provided them, I figured it best to go through the how-to's first, prior to contacting him, in case the questions I had were already answered.

The playlist contained around 120 tutorials, many made by Truong himself that covered a range of topics. The majority seemed irrelevant to my problems however skimming through the playlist I picked out a couple that I thought could give me some insight.

The title of the first video mentioned the Sylvanas rig by name (despite showing the Irex for the majority of the video) but ultimately just ended up being about hiding the controls. The second video however, was far more interesting...

After watching the video, I had learned a couple of things. Firstly, the term Advanced Skeleton (which I had heard being thrown around throughout my search) was not just a generic term, but the name of a software (AdvancedSkeleton5). Not only that, but it was the same software used to rig the Sylvanas character. The character in the tutorial features the exact same control setup and so with talk of exporting to game engines, I felt like I had perhaps stumbled upon the beginnings of a solution. 

From the tutorial, the solution seemed fairly simple. Advanced Skeleton featured a button called 'Simplify Face Setup'. This would convert the deformers into a joint based setup at the cost of some minor features (such as sticky lips, zipper lips etc). But since I wasn't planning on doing any lip sync this seemed like a fine compromise.

When I downloaded the software however, I found the Simplify Face Setup option was absent. Going back to the video, I found other people were lost as to where this feature had gone, and the uploader pointed to a FAQ where they discuss how with newer AS rigs, the feature is no longer needed. This didn't help me however, since my rig is from 2017.

I managed to download an older version of the software, which did feature the Simplify Face Setup button. However, I unfortunately could not get it to run without flagging up errors. At this point, I returned to the tutorials since I knew I was on the right track but was just coming up short.

Searching through Truongs YouTube channel also lead me to stumble onto this video [above], which seemed promising, however its solutions essentially involved stripping the rig of its facial capabilities in order to export it (which isn't what I want).

But I did notice, in the description of the video, a small comment: "What if I still want to convert all face setup into Unreal Engine (old rigs)? "

And a link to a separate tutorial:

This hidden tutorial, absent from the original playlist, gave me the tiny nugget of information needed to kickstart the process. By selecting the root control of the rig, and opening up the extra attributes, you are able to see exactly which version of Advanced Skeleton was used to rig the character. With this knowledge, I was able to download a specific version closest to the source, and hit the Simplify Button once more.

The tutorial specified that the process takes a considerable amount of luck and can ultimately take up to 40 minutes to complete. Initially, hitting the button flagged up errors once again but after some tweaks and troubleshooting, I hit the button and it started to do something.

This was the first hopeful sign I had had in days and I was so anxious about the process failing or crashing, I simply left my computer alone and decided to be patient.

And to my surprise it worked! (Kind of) The process had successfully created joints for all the facial points and everything was still bound and animatable. 

Original Rig Joint Setup

'Simplified' Setup

Getting all the Geo to export was now possible but I still had trouble getting all the face shapes to import. This came down to an error on my part, as during the UE5 importing process, there was a tiny checkbox called 'Enable Morph Targets' that I had left unchecked. Once I figured this out however, I had facial animation in Unreal Engine. Hurray!

Still though, not everything was perfect...

face slipping:

Whilst the solution had worked in getting my animated face geo into unreal engine, it had created some new problems. For some reason, animation on the face now incorrectly deformed when turning the head. It was as if the deformations were happening in world space and turning the head caused the poses to slide off the face. This presented a problem, however from my testing it only occurred on left to right motion and only when the face was deformed using the external controller (there is both a nurbs curve facial controller and on-face nurbs surface controllers). It appeared that using the on-face controls was fine. It was an annoying issue that I wasn't able to solve, and whilst it was a shame not to be able to use the controller to its full capability, I settled with having to using the on-face controls.

Controller setup and on-face nurbs controls

There were however even further errors, which was the fact that my characters operate at different scales. To maintain accurate in engine measurements, I calculated that the Sylvanas rig needed a scale of 0.7. For some unknown reason, this caused problems with the nurbs surfaces which provided sluggish and unpredictable deformations. This made certain face shapes difficult to achieve which is why there is only very simple facial animation in the final film. Thankfully, the story didn't call for dialogue or a particularly emotive character, so I don't think I lost all too much.

Evaluation Modes:

Once I started to use the new face rig, I realised that it was incredibly slow. Moving any joint would take Maya a couple seconds to update. The rig was slow beforehand (which was what prompted me to make the proxy models) but now it was running exceptionally slow. Something was wrong.

I started researching online what can be the cause of a slow rig when I discovered Evaluation modes. Evaluation Modes are the order in which Maya evaluates animation. There are 3 main Evaluation Modes: Dependency Graph, Serial and Parallel. Serial and Parallel are very similar but the big difference is when compared to DG. The Dependency Graph was Maya's old method of Evaluation and is much slower. It works by computing evaluations in order, one by one, whereas the other methods allow for some operations to happen in unison. This is what I could gather at least, but at the very least, Parallel was faster.

Unbeknownst to me, during my attempts to fix the face rig, the Advanced Skeleton add-on had  switched my Evaluation mode to DG in order to work with the Sylvanas rig (which is from 2017).

Switching my Evaluation mode from DG to Parallel restored the rig's speed, but later presented me with more problems…

Blink Issues:

Once I had moved into Animation Production, I discovered weird evaluations when it came to the blink controls and eye direction. Keyframing a blink to 1 would close the eye, but then returning it to 0 would not re open it. Then after a while, it would suddenly update and open again. 

This stumped me as once again I thought that perhaps my face rig fix had broken things in the process. I also now found that playing through my animation would cause the eyes to bulge and spin unpredictably.

It had been a while between switching Evaluation modes and discovering this issue, but eventually I made the link and tried switching back to DG. It fixed the issue but at the cost of speed once again. After some testing, I determined that animations would persist regardless of how many times I switched evaluation modes. Which meant I could animate most of the character in Parallel, and switch to DG when I needed to do the facial animation and exports.

It was a messy and annoying solution, but it worked.

Eyelash Export:

Despite solving the majority of the facial issues, I still experienced problems with the eyelashes. Initially, the eye lashes were glitching off of the face in a weird way. I solved this by setting the eyelash cluster envelopes to 0. 

But then I discovered that the Eyelashes would still not import into Unreal with the rest of the face. I tried many different methods in an attempt to rectify this but ultimately conceded that it wasn’t too noticeable and that she may have to go without eyelashes.

LINKS I USED: