Animated Texture Editor

Tool | Animated Texture Editor

Editor-integrated GUI. Converts videos into animated 'flipbook' textures. It features a lite video editor for adjusting the source video, and live-updates the new texture to a running instance of FM8. I used this to author 'hero' content for premier setpieces, including numerous jumbotrons on Hakone.

(Python ↔ Photoshop) → PowerShell → Game Client

Applied Skills: PyQT, OpenCV, NumPy | C# | JavaScript (Photoshop JS API)

Description

In FM8, many of our setpieces were designed with ray traced lighting in mind: scene-illuminating LED screens, mirror-reflective racing corridors, and so on. We used emissive materials to help demonstrate these features, and we assigned flashy, animated textures to many of these materials. It was my responsibility to convert externally-authored videos into these animated 'flipbook' textures, and I developed a pipeline for doing so.

Further into development, we had serious issues with the externally-authored videos for Hakone, our premier track. Many of these videos were out of 'spec' or simply weren't delivered, at all. It was a bit of a catastrophe; these videos take up one-third of the screen space, and we had a few days to go before 'ship-lock'. I was skeptical that the new videos would be submitted before the deadline, so I began authoring some high-quality replacements. My suspicions were right, and we ended up adding many of my animations into the official build. I played a vital part in actualizing the AD's vision for this flagship track.

XP

For FM8, the Art Directors wanted lots of new jumbotron videos for every track, and I was tasked with getting them into game. I chatted with one of the folks from the Environment Team to see how we'd done this in previous titles, and it was evident that the existing process was incredibly slow. And so, I decided to make a proper pipeline. I started making this tool well-before our GA deadline, and it proved to be an absolute ship-saver during crunch, when we were scrambling to get late-delivered content into game. However, right before ship lock, there were some serious issues at hand:

The external agency who was supposed to author the new jumbotron videos had not provided about one-third of the ones we needed, with 1.5 days left to go.
Many of the provided videos contained irreperable visual artifacts or were otherwise out-of-spec.

I took the initiative to repair any salvagable videos, and I authored entirely new substitutes for those which were outright unusable. To do so, I had to work around some significant hurdles:

Due to how we categorize assets for the different rendering passes, PVSTex caps the resolution to less than or equal to 2048x2048 when deploying to the ZIP.
Our engine expects that the texture's dimensions are both powers of 2, respectively.
The flipbook shader only had one flipbook texture slot, so each frame of the flipbook would have to fit within this canvas. This forced us to use very short flipbooks in player-proximal areas, and neither the ADs nor myself were particularly happy with that.
- I authored a new shader with additional texture slots, where it seamlessly jumped between different flipbook textures. It worked great; we were able to get much higher frame resolution and preserve the original video duration. However, due to 'perf' concerns, I had to go back to using the original shader.

With everything in one texture sheet, and with how the math works out, I basically had to pick between many frames with low frame resolution or few frames with high frame resolution. I went through all of our branding and made a variety of flipbooks in both formats. I also made some extra ones for Cinematics-focused assets, such as the pitkit computer screens. By this point, my work was done, and the actual texture-assignment was another team's responsibility. Unfortunately, the higher-ups wanted them to implement some significant changes, but this was never formalized into a PBI, and it was forgotten about. That is, until the day before ship-lock. They made it clear that someone needed to fix Hakone, and I decided to take it on. Per their feedback, I added:

"More texture variations for FM8-added brands"
- I made half of the necessary assets by monkeying around with gradients, cutting out logos from signage diffuse textures, and 'professionalizing' them with Photoshop's AI tools. I don't want to undersell the quality of what I produced, the ADs were very impressed with these. However, I knew that I still needed more fully-animated videos, and so I rummaged through every possible directory in search of something I could use. Eventually, I found a couple of older videos which were made by our Concept Team, and they were long enough that I was able to trim out different sections for different, respective texture variations. I didn't have access to a video editor, so I wrote an additional OpenCV function script to let me specify a range of frames to trim, and then within this range, specify a sub-range, throughout which any intermediate frames would be removed.
"X color palette/font/transition effect for Y brand or Z section of track"
- I used Photoshop actions to record the procedure I did for one video, and saved this as a re-usable action for the flipbook's respective brand or section of track. Then, I used Photoshop automation to perform these actions across a range of files, enable timelining, and re-export the MP4s.

And finally, now that I had the exportable videos, I was able to process them with my tool, adjust a few shader parameters, and close out the task.

Pseudo Code

Python Script

# ...

class SourceVideo(SourceMedia)

"""Contains the video's complete frame buffer."""

initialize(self, filePath)

super().initialize()

capture = cv2.VideoCapture(filePath)

while capture.isOpened():

_, frame = capture.read() # frame is a NumPy array

self.buffer.append(frame)

capture.release()

# ...

class TimelineWidgetItem(QtWidgets.QWidget)

"""Click-draggable widget with user-adjustable position and duration."""

initialize(self, sourceMedia, previousClip=None, nextClip=None)

hlayoutMain = QtWidgets.QHBoxLayout()

grabberL = QtWidgets.QButton()

grabberR = QtWidgets.QButton()

self.sourceMedia = sourceMedia

self.startPosition = 0 if previous is None else previous.endPosition

self.startFrame = 0

frameCount = self.startPosition + sourceMedia.length

self.endPosition = frameCount if next is None else next.startPosition

self.timelineFrames = []

# ...

class TimelineWidget(QtWidgets.QWidget)

"""Timeline containing TimelineWidgetItems."""

initialize(self)

self.timeline = QtWidgets.QHBoxLayout()

dropEvent(self, dropped)

"""Drop the TimelineWidgetItem according to the specified mouse position."""

for timelineItemIndex in self.timeline.count(),

timelineItem = self.timeline.ItemAt(timelineItemIndex)

if pos.x() < timelineItem.x() + timelineItem.size().width() // 2,

self.timeline.insertWidget(timelineItemIndex-1, dropped)

break

# ...

class AnimatedTextureTool()

initialize(self)

# ...

self.buildGUI()

# ...

buildGUI(self)

# Toolbar - UI

toolbarHlayout = QtWidgets.QHBoxLayout()

self.toolbarBtnImport = Forza.Widgets.FileBrowser("Import") # adds a new clip

self.toolbarNboxDuration = Forza.Widgets.NumberBox("Duration")

self.toolbarNboxFrameScalar = Forza.Widgets.NumberBox("Frame Size Scalar")

self.toolbarNboxFrameRate = Forza.Widgets.NumberBox("Frame Rate")

self.toolbarNboxFramesX = Forza.Widgets.NumberBox("Frames X")

self.toolbarNboxFramesY = Forza.Widgets.NumberBox("Frames Y")

self.toolbarChkboxSplitExported = Forza.Widgets.NumberBox("Flipbook Per Clip")

self.toolbarCboxOutputSizeX = Forza.Widgets.NumberBox("Output Size X")

self.toolbarCboxOutputSizeY = Forza.Widgets.NumberBox("Output Size Y")

self.toolbarBtnPlay = QtWidgets.QButton("Play")

self.toolbarBtnExport = QtWidgets.QButton("Export Timeline")

self.toolbarTxtboxOutputPath = QtWidgets.QTextBox()

# ...

# Player - UI(Upper RHS)

previewVlayout = QtWidgets.QVBoxLayout()

self.previewGviewVideo = QtWidgets.QGraphicsView() # displays the playhead-relative frame

# ...

# Selection - UI (Middle RHS)

hlayoutOperations = QtWidgets.QHBoxLayout()

self.operationsBtnSplit = QtWidgets.QButton("Split") # used for multi-texture flipbook shader

self.operationsBtnDelete = QtWidgets.QButton("Delete") # removes the selected clip

# ...

# Timeline - UI (Lower RHS)

timelineVlayout = QtWigets.QVBoxLayout()

self.timelineSldrPlayhead= QtWidgets.QSlider()

self.timelineWdgt = TimelineWidget()

# ...

# Toolbar - Signals

# ...

self.toolbarBtnExport.clicked.connect( # generate a flipbook from the current timeline

lambda: self.export(

[timelineWdgtItem for timelineWdgtItem in self.timelineWdgt],

self.toolbarNboxFramesX.value(),

self.toolbarNboxFramesY.value(),

self.toolbarCboxOutputSizeX.value(),

self.toolbarCboxOutputSizeY.value(),

self.toolbarTxtboxOutputPath.value()

)

# ...

export(self, clips, outputFramesX, outputFramesY, outputSizeX, outputSizeY, outputPath)

""""Converts the provided video clips into flipbook textures."""

for clip in clips,

flipbook = np.empty(outputSize, dtype=object)

j = 0 # X & Y are inverted, as expected when traversing matrices

i = 0

for frame in clip.timelineFrames,

flipbook [i, j] = frame

j += 1

if j >= outputFramesX,

j = 0

i += 1

if i >= outputFramesY,

break

np.save(outputFile, flipbook)

# ...

main()

animatedTextureTool = AnimatedTextureTool()Python Script