VRVideo
Viewing Videos:
Some of the videos can be viewed in Youtube:
Hamilton Mountain Youtube Link
Links to download video - videos that are cached on the device have better playback than streaming:
Short:
Burnside Skatepark (low res - 30.5mb)
Long:
Multon River Park (low res - 1.04gb)
Bridges of Portland (low res - 468mb)
Bridges of Portland (high res - 1.4gb)
Video Processing Workflow Overview
https://miro.com/app/board/uXjVIYeMW9M=/?share_link_id=541053729997
Programs Used:
Python / OpenCV - For finding keyframes in video
FFmpeg - Command line video editing program for syncing, stitching, and re-encoding with desired frame rate and resolution
Dealing with Stabilization Challenges in VR Filming with DJI RS
I've been using the DJI RS stabilizer for my VR footage, and it has been doing a great job keeping things smooth. However, if the cameras experience a bump, they stabilize independently, leading to differences that can be quite noticeable in VR. These stabilization mismatches can cause eye strain and headaches when viewed in VR.
My Video Capture System:
Equipment: Dual 4K 120 FPS cameras used for stereoscopic recording.
Synchronization: A clapboard is employed at the beginning of the recording for synchronization purposes.
Storage: Videos are initially stored on the camera's internal storage before transfer.
Other options for cameras:
Option 1: Dual Lens Single Camera
Advantages: Ideal for high-budget 180-degree VR productions.
Disadvantages: Limited to lower frame rates, high cost.
Option 2: iPhone
Advantages: Convenient as it handles everything, including saving spatial video.
Disadvantages: Limited to 1080p at 30 FPS.
Option 3: Two Separate Cameras
Advantages: Supports 4K at 120 FPS, capturing fast motion effectively.
Disadvantages: Requires extensive syncing and stitching in post-production, which is time-consuming.
USB Transfer to Computer
Transfer Process: Footage is transferred via USB to a designated computer directory.
Directory structure:
project_root/left/ (Left camera footage)
project_root/right/ (Right camera footage)
project_root/ffmpeg_output/ (Footage in an intermediate processing step)
project_root/clips_output (Footage of finished clips)
File Size: The raw video files range from 1GB to 2GB per minute before compression by FFmpeg.
Synchronization
Synchronization is achieved by using a Python script to find the keyframes (where the clapboard hits) in the left and right videos. The keyframes used to syncronzie the videos are saved to use in a ffmpeg command. I chose to use ffmpeg to stitch the videos because it if 4x faster (on my i7-14k) and outputs in the codec that plays the best on my Quest headset.
Alignment
The cameras are aligned well enough on the rig, eliminating the need for software-based alignment adjustments.
Stitching
FFMPEG is used to synchronize and stitch the left and right videos. This is saved in a .sh file:
LEFT_INPUT="stJohnsBridge5_left.MP4" # Update to your left video file path
RIGHT_INPUT="stJohnsBridge5_right.MP4" # Update to your right video file path
START_FRAME_LEFT=0 # Starting frame for left video
START_FRAME_RIGHT=169 # Starting frame for right video
OUTPUT_LEFT="left_output.mp4" # Output filename for left video
OUTPUT_RIGHT="right_output.mp4" # Output filename for right video
FINAL_OUTPUT="temp_side_by_side.mp4" # Final side-by-side video output filename
LEFT_AUDIO="temp_audio.aac" # Extracted left audio file name
FINAL_OUTPUT_SOUND="stJohnsBridge5.mp4" # Final output with sound
# Crop and center the left video to 3840x3840
ffmpeg -i "$LEFT_INPUT" -vf "select=gte(n\,$START_FRAME_LEFT),setpts=PTS-STARTPTS" -an "$OUTPUT_LEFT"
# Crop and center the right video to 3840x3840
ffmpeg -i "$RIGHT_INPUT" -vf "select=gte(n\,$START_FRAME_RIGHT),setpts=PTS-STARTPTS" -an "$OUTPUT_RIGHT"
# Stack the processed left and right videos side-by-side to create a 2160x1080 output
ffmpeg -i "$OUTPUT_LEFT" -i "$OUTPUT_RIGHT" -filter_complex "hstack" "$FINAL_OUTPUT"
# Extract audio from the original left video, starting from frame 0
ffmpeg -i "$RIGHT_INPUT" -vn -af "aselect=gte(n\,$START_FRAME_LEFT),asetpts=PTS-STARTPTS" "$LEFT_AUDIO"
# Combine the side-by-side video with the extracted audio
ffmpeg -i "$FINAL_OUTPUT" -i "$LEFT_AUDIO" -c:v copy -c:a aac -strict experimental "$FINAL_OUTPUT_SOUND"
# Delete temporary files
rm "$OUTPUT_LEFT" "$OUTPUT_RIGHT" "$FINAL_OUTPUT" "$LEFT_AUDIO"
The following steps are necessary to generate the compilation video:
Trimming Clips to Length
ffmpeg -i 'stJohnsBridge5.mp4' -ss 9 -to 24 -c:v libx264 -preset fast -crf 17 -c:a aac 'waterFountain.mp4'
Generate Compilation Video
ffmpeg -f concat -safe 0 -i mylistStJohns.sh -vf "scale=7680x3840" -c:v libx264 -preset fast -r 60 -crf 17 -c:a aac -b:a 192k stJohnsJuggling_r60_2to1scale.mp4
Inject the spatial media metadata
https://github.com/google/spatial-media/releases
to run, open a terminal: python3 gui.py
select both check box options
Convert the metadata
exiftool -api LargeFileSupport=1 -StereoMode="left-right" stJohnsJuggling_r60_injected.mp4
Check the metadata
ffprobe stJohnsJuggling_r60_2to1scale_injected.mp4
Look for:
Side data:
stereo3d: side by side
spherical: equirectangular (0.000000/0.000000/0.000000)