The biggest hurdle is acquiring a Quest Pro, which is a more professional grade VR headset (in terms of pricing and technology), currently running about $1,000 (which due to poor sales was cut to this figure after the original asking price of ~$1,500)

After that, it's a matter of following the guide in the official SDK page, and installing/updating the relevant Unity OpenXR and Oculus Integration packages.

Note: This is important, as I had quite a few issues early on in this project and a lot of confusion due to features not working or not appearing at all. Ultimately, I had to re-install and update all the Unity VR/XR plugins, and Oculus Integration packages, as well as migrate everything to a new Unity project to get everything working correctly.

We decided to focus on this one, and bring in some assets/ideas from a conceptual NPC project that the Avatar Recorder project was ultimately building too. To do that, we needed to take another character model we had in that project, we'll just call her Leather. Our goal was to figure out how difficult it would be to implement a custom character model, and still have the facial expressions driven by the Oculus hardware and software.

Diving into it, we can see *pic* that the character has roughly ~266 (except according to documentation, only 63 are necessary?) blend shapes for various ways the face can move. Creating all those blend shapes from scratch would be tedious at best, and I'm not sure how well transferring blend shapes to a different model with separate topology would work out. With that in mind, I simply took the body and clothing of the Leather character, and transplanted the head of Aura onto that. I found that modifying the topology in any way broke the blend shapes and caused strange deforming when running the face tracking (which seems obvious looking back), however I was able to delete the alien appendages on her head, and replaced it with the hair cards from Leather. Lastly I just projected the face textures from Leather onto Aura, as they have pretty similar UVs.

*Video/pics of above^

*final result Video*

Here you can see the newly LeatherAura model is in place, and the results are moderately decent. There is a little bit of clipping issues around the eye area, and you can also clearly see the Body Tracking module isn't yet currently working, though we've had success disregarding that and instead using our QTM systems to stream motion capture data in real-time which allows a truly operational full body tracking, at the cost of tens of thousands of dollars to setup, but that's besides the point.

• detailing getting full body tracking working with the Oculus Quest Pro, and their official MovementSDK template for face and eye tracking

  • how to get it to work

    • problems with outdated OVR plugin, unity versions

    • Permissions settings enabled on the device, should appear in Link after running for the first time correctly

  • examples

  • examining the setup of Aura character

    • blend shapes/shape keys

  • creating a new character model

    • using the leather female character

    • transfering blend shapes from Aura to Leather

      • fix more bugs (editing the geometry of the mesh breaks the shape keys)

• showing off only face and eye tracking, then custom model, then custom mode with full rig courtesy of Aspen and QTM with full body tracking (video somewhere?)

Networked MovementSDK - 2 players can emote and talk to each other over the network, remote or local

Facial Expressions should ideally just be sending values for blendshape deformations, but could be harder

Hand Tracking shouldn’t be too difficult, using Oculus or MovementSDK built in hands and tracking, then send the pos and rot data over

might be worth looking into Oculus Meta Avatars, designed specifically for networking and social purposes, yt tutorials?

MetaAvatars is a bit more simplified in terms of face tracking an avatar, and probably wont track well to implementing into the MovementSDK, furthermore, the Remote connection example they have doesnt actually have any remote sample, its just a imitation of a remote connection essentially using a mirrored avatar, but not really since the player sends to a data stream, and the remote avatar accepts it…

Problems:

Networking the MovementSDK may be complicated, the Sample Scene, and the prefab for Aura herself are large game objects with many complex scripts attached, furthermore, the First Person GameObject doesnt actually have a visible head mesh, only the mirrored one, so maybe its necessary to hide the FPS, and just network the mirrored mesh across the network?

Finding where the blendshapes are changed and stored, accessing those variables and sending the data across

how to then apply that data to a remote object

BlendShapeModifier.cs has a bunch of blendshapes that are populated, but it seems to only act as a modifier, and not directly setting it?

There really does seem to be about 66 blendshapes total, even though the manual says only 63 are truly needed for full facial representation

That said, it does seem to store the expression modifiers to a timestamped json file? then has a function to loading it? not sure why

FaceTrackingSystem.cs - this seems to be the script that changes the mesh to reflect whatever values the blendshapes are based on the tracking, I think making a copy of this script, making it a NetworkBehavior, and replacing the current values it grabs with data sent remotely will be the way to do it, just not sure how yet? some kind of RPC to send all the blendshape values in a function? 

RPCs only really work well for one-shots and instantiation by a button prompt, and not great for continuously updating variables, so a NetworkVariable is recommended.

NetworkVariables only support default Unity & C# types, like integers.

The BlendShapeToFaceExpression and other variables are custom OVR variables, and wouldnt work? But the currentWeight section also seems to pass in an int? Can that be replaced by a NetworkVariable int?

No. Those “int” are meant to be types, not variables, so replacing doesnt work… However may it’s implying that the OVRFaceExpression.whatever is a int, so maybe it’ll just work with the NetworkVariable?

Going back to MetaAvatars, it does seem to have some basic facial feature tracking, as evidenced by the “MirrorScene”, (though the lip-sync module uses audio input rather than lip tracking), and for the life of me I can’t figure out where it’s happening. If I can find it, it may prove to be helpful in networking faces, even if the two systems are built different on the top-most Unity layer (which does seem to be the case)

Either way, it seems clear that the MovementSDK samples weren’t built out with the intended use of multiplayer functionality, it doesnt seem like its built out for such considering the hodge podge of scripts and child objects attached

Do a while loop, checking if int faceIndex is lower than the max number of FaceExpressions enum, if so, get the weight of the current face expression by index number, then increase number.

If faceIndex is equal to greater than max, reset to 0 and repeat. Rinse and repeat, for each through each one for, forward.

Declare the weight of current face as a network variable, then send it to all connected clients (Host acts as server and client), in the connected client, grab the other client instance of aura, and set the face weights to equal the network variable.

How to set given face expression a new weight?

Figure out how to separate the two Aura instances

From testing, networking works fine, but getting Aura attached to the correct headset is problematic. By default, the OVR headset will control both Aura instances in the scene. I suspect this has to do with how the Aura prefab is built, and how it connects and decides to use the Oculus, this wasn’t an issue with the last project, but that was built relatively from scratch.

Aura relies specifically on scripts and connection to the Oculus Quest pro…

our last project just had the OVRCamera rig acting locally, then using the VRIK system to track based on whether server or client.

Photon PUN, a popular multiplayer Unity asset is free to download and use, and likely significantly easier than and less technical than Nunty’s built in Netcode For GameObjects.

So. The LocationBasedVR works, because the scene only has one instance of the OVRCameraRig, so each local player just uses that. The VRIK avatar works by tracking empty game objects, parented to the local OVRCameraRig, that’s why it works. Using the Networked Avatar script, it simply binds the empties and the OVRCameraRig controls based on isLocalPlayer and isServer/isClient.

Aura automatically binds to the Quest Pro, I’m just not sure how or where.

CorrectiveFaces seems to set the face geo to blendshape weight

OVRFaceExpressions initalizes the face tracking, and grabs the values for the blend shape weight

// run a for loop to go through every index position(instead of the 1), to get a value for each blendshape 

// get a value for each blendshape, put it into a public network variable, send across the network to each client 

// input the network variable for the corresponding index/blendshape value, profit? 

// public NetworkVariable<float> networkWeight = _currentFaceState.ExpressionWeights[1]; 

// public NetworkVariable<int> networkFaceIndex = 0; - updates as for loop runs through _currentFaceState? 

// find out exactly what value ExpressionWeights send, seems to be a float 

// do I need to use some sort of listen pattern, or simply constantly send the values over? I imagine they're constantly- 

// changing, maybe only network new values if a certain degree of change? like +10/-10 on the blendweight so it's not constantly changing? 

https://developers.facebook.com/blog/post/2022/06/28/building-your-multiplayer-vr-experience-making-multiplayer-vr-app/ 

https://docs-multiplayer.unity3d.com/netcode/current/advanced-topics/ways-synchronize/ 

https://docs-multiplayer.unity3d.com/netcode/current/basics/networkvariable/ 

https://medium.com/eincode/getting-started-with-multiplayer-player-movement-f9f7f6a4217