First thing's first, we had to setup our projector and projecting environment...

Our current projection mapping setup, pretty simple but as the other work-study students and I were essentially just familiarizing ourselves with the process, it seemed prudent to just use simple geometry, in this case, 2 small wooden columns painted white.

One of my last processes in After Effects after getting a feeling for how Projection Mapping works, and to help along our various work study students who would also have the chance to learn and experiment, I created a simple Grid Layout, which was essentially just a template Composition setup with some basic graphics that already were correctly aligned to the current projection setup.

Created Grid Layout template for use with other students and for myself. I then exported the GridLayout as a single frame PNG and exported that into Blender.

Using fSpy to try and align the camera as close as I could with the grid template image, little bit of finagling? to get it just right.

Final video of most recent After Effects GridLayout project here (that was played on the projector when it was in the nook) - hopefully I or one of the work-study students has this

Some final experiments that weren't finished here

further various experiments here:

lasers, silhouette, water

Projection Mapping Perspective Fix Notes - as drawn up by Alan, essentially the drawing on the right represents the final Camera render/game view, which is also roughly aligned with how the projector itself is facing the real-life objects.

This causes issues with perspective, as essentially, the best viewing angle is going to be from right next to the projector, because that's how the image is kind of built or developed, whether in 2D like After Effects, or 3D in Blender or Unity.

This can be fixed in Unity, as Alan has experienced a few times:

• One, just in general, I can use Unity to create RenderTargets/Textures to apply to the planar surfaces of the real-life objects, I can assign this to a Plane, Quad, or any object, then deform those objects to better align in real space (similar to using Corner Pin effect in After Effects to Warp an element).

• Secondly, and specifically for this above example, One Camera represents where the audience might be viewing the Projection Mapping, in this case, from the side as shown by the left example, this will allow you to get accurate perspective as one facing that viewing angle, that Camera then renders to a RenderTexture, which can then be assigned on GameObjects or projected, or whatever in 3D space aligned with the digital representations of the mapping objects, then finally the final Render cam will see those projected RenderTextures and everything should look correct.

Having finished up a number of different renders and tests (using both After Effects and then Blender), I decided to follow Alan's advice to try and work a bit more with the perspective, specifically using Unity which would again allow me to visualize the results in real-time (unlike in Blender), and also allow me to create my own "Corner Pin" tool like in After Effects.

The Corner Pin was pretty simple to re-invent as well, at first I was overthinking it and trying to create a script which would place empty GameObject handles at each vertex of a Plane or Quad, but frankly this was a little outside my ability range, and I then realized after digging up an old projection mapping project, Alan just used a plane mesh with 4 bones at each vertex, and used that to deform the mesh. This was a significantly simpler approach and works quite well.

This news Corner Pinnable quad mesh basically acts as a flat 2D texture than can be appropriated mapped onto the surface of real-life objects, and the texture applied to that mesh is a RenderTexture, which allows one to create complex scenes and setup, and then simply render everything out to a simple texture that can be easily wrapped onto a geometry.

A previous script Alan created to modify and hard set a Camera's frustum view, to allow the camera to be moved around, while still keeping the same view frustrum.

Ideally, this was to be used for Virtual Production within Unity, and it does work quite well, but I decided to bring it over to this Projection Mapping project to allow one of the cameras to move around, and adjust the perspective that's rendered.

(We are still working on getting Virtual Production up and running with our 60 panel, configurable LED wall, but we've decided to pursue Unreal Engine for that just due to the sheer amount of documentation the Epic team has set up for it.)

An example of the final layout of the scene. The currently selected camera is the Main Camera, and it sees two planes that are deformable, and have a RenderTarget as their Material, and the RenderTargets are captured to the right, where I have small "scenes" built out with everything necessary, and captured by a RenderCamera.

In the above instance, I currently just have an a default Blender CheckerUV texture to help align properply and check distortions on the deformable Planes.

Here, you can see the full "scene" of one RenderCamera, this then gets passed to a RenderTexture, on the deformable plane, and finally picked up by the Main Camera.

This allows me to make whatever changes I want to each individual "scene" or Render Camera, without affecting the rest of the projection mapping setup, and the projection mapping setup is easy enough to manipulate using the bones attached to the plane object, in a similar fashion to the Corner Pin tool in After Effects.

Video of full Unity experience (both in editor & projected version)? - nevermind, just editor, projector has officially been moved

Below is a short video of one of the classes that came in for the Projection Mapping workshop, unfortunately, I didn't capture much more and frankly the video wasn't going to come out great anyway due to my old phone camera.

And finally, this is the final demo video (with sound) created as an example/showcase of the possibilities of Projection Mapping, put together by myself, Alan, and work-study students Jillian and Nikki