Austin's Journal
ALL OTHER WIKI CONTRIBUTIONS
CONTRIBUTION 1 [Table of Contents for Software Comparison] Software Comparison Page
CONTRIBUTION 2 [Wiki setup and Template VR Project for Unreal Engine 5] Unreal Engine 5
CONTRIBUTION 3 [Terrain Generation with Blender] Terrain Visualization In Blender
CONTRIBUTION 4 [Exporting Materials From Blender] Baking Blender
CONTRIBUTION 5 [Unreal Summary Page On General VR Development] VR Development Software
CONTRIBUTION 6 [Native Point Cloud Visualization for Unreal Engine 5] Point Cloud Visualization
CONTRIBUTION 7 [Adding VR to a Non-VR UE5 World] Adding VR to an Existing UE5 World
CONTRIBUTION 8 [Apple VR Headset Rumors] Apple VR Headset
CONTRIBUTION 9 [Adding VR teleporation to a multiplayer VR actor in UE5] Adding VR Multiplayer in UE5
HOURS SUMMARY
Total: 167 hours
HOURS journal
Course Learning Goals:
before after
---- ----
1 | 4 | Goal 1: articulate VR visualization software tool goals, requirements, and capabilities
2 | 4 | Goal 2: construct meaningful evaluation strategies for software libraries, frameworks, and applications; strategies include surveys, interviews, comparative use, case studies, and web research;
2 | 5 | Goal 3: execute tool evaluation strategies;
1 | 4 | Goal 4: build visualization software packages;
2 | 4 | Goal 5: comparatively analyze software tools based on evaluation;
1 | 5 | Goal 6: be familiar with a number of VR software tools and hardware;
3 | 5 | Goal 7: think critically about software;
2 | 4 | Goal 8: communicate ideas more clearly;
1 | 3 | Goal 9: effectively program useful VR-related features;
1/26/23 - 1.5 Hour
Began ticking off to-do's on HW1, setup page and introduction, joined slack channel.
Explored website for class context.
1/27/23 - 2.5 Hours
Continuation of HW1
Continued to explore site for class content.
I was reading through the project ideas page and Gaia sky seems really interesting to me! Finding a way to work multiplayer capabilities in sounds like an interesting project to explore. Unity seems to have a way to do multiplayer VR (see Meta example), so maybe there's a way to link that together with Gaia to create guided tours of the solar system of sorts? Unsure, requires more research.
Three changes should each require ~10 minutes to complete.
Move all tutorials relating to the Meta Quest 2 into one area, so it's easier to access.
The VR Development Software comparison page is quite long - adding a way to jump to certain sections would be helpful.
The "Oculus Website" Link under Oculus tutorials in the Unreal Engine page is broken, determine what page it was originally linked to and re-link it accordingly.
Three changes should each require ~1 hour to complete.
Add a page about the newly released mixed-reality Meta Quest Pro Headset, with general facts and information about it. Include what makes it better than other headsets.
A bunch of the "VR in..." would benefit from a short description of applications before presenting users with academic papers or projects. A more general overview would help users with limited understanding of possible uses in the field before delving into an extremely detailed example.
The Meshlab VR Modeling Software page is a bit sparse. Adding more information about it would be helpful to the user to understanding the complexity, as well as compatibility with systems.
The final three changes should each require ~10 hours to complete.
Add a page that denotes the differences between the types of Reality (Augmented, Virtual, Mixed, etc). Compare and contrast the pros and cons of each type. Include applications of each and possibly list a few different devices that use each.
There's a lot of content in the sidebar, consider restructuring and consolidating information in a more digestible format. Can take a while, as there are a lot of things to be reshuffled.
Add a FAQ's page (similar to What Is VR, but with other questions as well), with the most asked questions when it comes to the world of VR, collaboration, and data visualization.
Complete one of the 10-minute changes.
Added a table of contents to the comparison page for easy access to specific sections, if a user is looking for a specific comparison.
1/29/23 - 1 Hour
Reviewed Kenny Gruchalla's bio and reviewed some of featured work.
Added questions to class activity board here.
1/31/23 - 1.5 Hours
Setup Quest 2 Headset for use.
Setup host system (local) with SteamVR and Oculus Link,
Tested Quest 2 wired mode with Google Earth VR.
2/1/23 - 2.5 Hours
Read through previous student projects.
Pick 2-5 potential pieces of software to explore and evaluate for your research project
Fusion360 and VR support
Using this software and insight from previous projects, create 3-5 potential project ideas in your journal. Be prepared to share concisely (one sentence each) in class. We will ponder their levels of: 1) collaboration, 2) immersive VR, 3) scientific visualization.
Project idea 1: Using Gaia Sky, develop a way to integrate multiple people into one session. Allows for guided virtual tours in VR.
Project idea 2: Visualize weather data (rainfall, temperature, etc) in large cities. Maybe a cumulative way to view all over world.
Project idea 3: Visualizing machine parts and possibly learning to assemble machines in a collaborative way.
Put course learning goals at top of journal.
2/3/23 - 4 Hours, HW for 2/7
Worked through Google Earth VR tutorial and activity.
Setup DinoVR for next class.
Reviewed wiki and did some web research for my project ideas.
Explored a bit more into Photon Phun 2 for multiplayer aspects.
Software evaluation metrics that could be measured.
Ease of use - how long it takes the average user to get up and running for a given piece of software.
If a programming-based thing, then how long to reach a common "hello world" scenario.
Performance/resource consumption - how well it runs on different types of machines, or how well it runs on one machine compared to another piece of software.
Can measure in CPU/GPU/RAM/ROM utilization.
Can measure FPS on a given system for a given scenario.
Measure in a sense optimization between a few sets of programs.
Documentation - can compare how comprehensive each program's documentation is, plays into how easy it is to use.
Featureset - measure what can be done and can't be done on a given system in terms of features. More features = better?
Plugin/Add-on compatibility support.
Compatibility with formats of datasets.
2/6/23 - 3.5 Hours, Continuation of HW for 2/7
Continued research about following topics:
Understanding multiplayer VR
Explore Unity, Photon Phun 2, looked at some tutorials for general idea.
3 Project ideas (reiterated)
Project idea 1: Using Gaia Sky, develop a way to integrate multiple people into one session. Allows for guided virtual tours in VR.
What I've learned that is Gaia Sky is quite complex, and for a person who doesn't have that much experience working with graphics and VR, developing multiplayer for an already very well fleshed out application could prove to be a lot.
I think beginning out by learning Unity or related VR development software and exploring a system that allows for multi-person collaboration in general would be a good starting point.
Things I'll do during the project:
Explore the different types of systems that exist for multiplayer development, specifically in VR.
Implement an multi-user experience to test specifically collaborative aspects, how users can interact with one another, passing objects around.
Learn more about VR development in general.
One class activity for project:
Test the multiplayer system by allowing a bunch of students to join together and explore through a server.
Potential deliverables:
Wiki additions on explored software.
A framework for setting up multi-user VR experiences, with no VR-development experience at all. (still some programming experience required)
Project idea 2: Visualize weather data (rainfall, temperature, etc) in large cities. Maybe a cumulative way to view all over world?
(other ideas include: visualizing snowfall, allowing students to experience the largest snowfall amounts in comparison to their given height. Visualizing the size of the largest beings on earth, allowing students to experience sizes compared to them.)
Things I'll do during the project:
Explore the different types of engines used to develop VR applications, and slowly learn it.
Learn more about VR development in general.
Develop an application that looks a given dataset and displays it accordingly.
One class activity for project:
Allow students to try application and observe the varying weather across different regions, notice patterns.
Potential deliverables:
Wiki additions on explored software.
An application that is able to visualize weather data
Project idea 3: Visualizing machine parts and possibly learning to assemble machines in a collaborative way. (Mindesk, Fusion360, Solidworks)
Things I'll do during the project:
Explore the different types of CAD applications or CAD-based programs that support VR.
Learn more about VR development in general.
Display data about machines and mechanisms through CAD and VR.
One class activity for project:
Students can create and visualize their own creations in VR, understand limitations and benefits of visualizing machines/creations in VR.
Potential deliverables:
Wiki additions on explored software.
2/7/23 - 3 Hours, Preparation for Class 2/9
Did some more research in to specifics on multiplayer collaboration on wiki.
Finished DinoVR activity, filled out feedback form.
Finished in-class activity reviewing projects.
2/8/23 - 3 Hours, Preparation for Class 2/9
Repivoted view of project to something different, and less complicated.
General Project Idea: Visualizing Ecosystem Structure over a given timeframe.
Found NEON Dataset with LiDAR scans that allow us to visualilze a given area's ecosystem, elevation, etc.
Should allow for a general terrain visualization, and vegetation/tree cover in a given area.
Project Milestones:
2/14
Research, download, and understand datasets.
Figure out what each dataset shows, and what datasets can be used.
2/16
Download Unity, setup and build a test environment to get used to how Unity works in VR Development.
Start with a simple "hello world" program.
2/23
Begin to import LiDAR Data, test terrain visualization.
2/28
Import other ecosystem structure of LiDAR Data, overlay on top of terrain visualization.
3/02
Implement ability to change time, and in turn, visualization.
3/07
Make UI Tweaks, possibly adding time change function to a menu of sorts.
3/09
Present to Class! (May shift depending on success)
Class activity: have users explore different timeframes for a given area, and point out the more significant changes over time, similar to a "spot the difference" style experience.
2/11/23 - 2 Hours - Preparation for Class 2/14
Explored NEON Dataset, researched ways to visualize in Unity & Unreal Engine
2/12/23 - 3 Hours - Preparation for Class 2/14
Created powerpoint presentation for project 1 proposal.
2/13/23 - 2.5 Hours - Preparation for Class 2/14
Created new page for project 1 proposal, can be found at top of journal.
2/15/23 - 4 Hours - Experimented With Unreal Engine VR Development
Self evaluation of journal:
Journal activities are explicitly and clearly related to course deliverables - 4
deliverables are described and attributed in wiki - 3
report states total amount of time - 5
total time is appropriate - 4
Watched a few tutorial videos
Tutorials on General Unreal Engine 4
Downloaded Unreal Engine
Looked more into this plugin for GEOTIFF Data --> https://github.com/iwer/GeotiffLandscape
Decided to move to Unreal Engine 5 - Notable Better VR Support, offers additional wiki deliverable
Plugin support of LiDAR point clouds --> https://docs.unrealengine.com/5.0/en-US/lidar-point-cloud-plugin-for-unreal-engine/
Setup android studio & Visual Studio to work with VR
Unreal Engine 5 Videos:
Continued with experimenting with dataset and types:
Testing data out in Google Earth to better understand how it works.
.shp files contain heightshapes for areas.
.tif files contain heightmap
2/17/23 - 3 Hours - Continued experimenting with Unreal Engine 5
Watched multiple tutorials on Unreal Engine 5.
Played with lighting effects and models, explored Quixel Bridge for useful assets.
2/19/23 - 3 Hours - Revisiting LiDAR dataset and insertion into unreal engine 5
Watched some UE4 Tutorials, may be able to "recycle" for UE5
2/21/23 - 5 Hours - Experimenting with LiDAR dataset, looking into TerrainMagic
https://docs.unrealengine.com/5.0/en-US/lidar-point-cloud-plugin-for-unreal-engine/
Visual Studio Setup for Unreal Engine:
Ran into issues installing terrainmagic, spent some time reconfiguring visual studio & unreal engine
https://docs.unrealengine.com/5.0/en-US/editing-landscapes-in-unreal-engine/
Finally imported a (admittedly unpretty and issue ridden) copy of a section into Unreal as a heightmap
2/22/23 - 2 Hours - Working on cleaning data, Paraview Install for local and paperspace machines
Installed paraview and tutorial file for local and paperspace machine.
Trying to find solutions for overprint in unreal engine with .png heightmap installs.
2/24/23 - 3 Hours - Re-assembling data for visualization
Combined heightmaps together to generate larger heightmap in photoshop.
Matched up heightmaps so that covered area is same at all time points.
2/25/23 - 2 Hours - Re-importing heightmaps into Unreal Engine
Used landscaping tool to generate heightmap from assembled PNGs
Noticed issue: peaking in a lot of areas, not really helpful for data visualization.
2/27/23 - 1 Hours - Creating Progress Presentation, Preparation for Class
Created powerpoint for class.
Read through Melvin's starting activity.
3/1/23 - 2 Hours - Preparation for class
Read through Lexi's and Mohammed's documents, prepared for class.
Finished class Melvin's class activity and filled out google form.
Looked more into materials for Unreal Engine 5
3/4/23 - 4 Hours - Refactoring and Regenerating Dataset
Kept running into an issue of peaking within the dataset, too many fully white spots in images.
Regenerated images in photoshop manually, matched each year so that all datasets show the same information.
Also added 2016, 2017, 2019 datasets and compiled them accordingly.
Ran into issues with collages creating thin white lines between each tile.
2015 Data
2018 Data
2021 Data
3/5/23 - 4 Hours - Adding Material Coloring and Setup Levels with default start states.
Created custom material which colors data based on height of peaks.
Setup default start states for each level (dataset), which user will spawn into.
3/6/23 - 10 Hours - Re-doing the dataset in Blender and Importing it into Unreal Engine (again)
One of the issues I kept running into was the problem of performance. Since I was importing this massive .png file as a landscape, Unreal would import this as a huge landscape, which was unoptimal for performance, even on my personal gaming PC.
Additionally, there was no way for me to directly interact with the environment, should I want to change the way I view the dataset.
So I moved to Blender to regenerate the dataset, which was a process in itself learning how it worked. Fortunately, I was able to find a video that showed how to bring my heightmaps nicely into blender.
I exported these heightmaps as .fbx files to Unreal to see if they'd work, and they did. Using the same material system above, I added color to the datasets created using blender as meshes in Unreal, and kept going.
However, one issue I ran into was when the meshes were moved, they changed colors, since my material system was based on world position, which would mess everything up.
After my issues in Unreal, I had to create a color/material system that would translate colors into height, but keep it static. In this case, I would create the material in Blender, bake it into a .png, which could be used as a texture mapping onto the meshes in Unreal. I made a gradient where green was the shortest values and red the highest values, applied it to the 3d mesh, and then pulled the resulting mesh, "baking" it into a hardcoded file.
This involved learning about how to bake images, how to use the material system in blender, and importing it into Unreal accordingly.
Results below for this portion:
Initial Render in Blender
Material Overlaid on Map
"Baked" Image after Material Creation
3/7/23 - 8 Hours - Updating the Virtual world with the new datasets and setting up the environment
There was a lot that was done, so here we go!
Using Unreal Engine Vr Template
Created a new world with existing VR functions.
Troubleshot issue with teleporting functionality, though now it doesn't have collision mitigation, so it's possible to teleport outside of the designated area and into items (not ideal, but better than no teleportation at all).
Added lighting to better view items.
Updated the start menu to exit the game, as well as restart in the new level created.
Added in new meshes from Blender as grabbable items, so they can be manipulated and moved for comparison.
Loaded new materials onto meshes for coloring.
Added times to datasets, so that users can differentiate between them. (took longer than expected, had grabbing issues with linking the two items together).
Updated physics so that when items are released they aren't subject to gravity and don't float away when released (will remain stationary).
Troubleshooted issues regarding outputting to .APK for Quest 2.
Currently dealing with performance issues due to many polygons.
3/8/23 - 3 Hours - Preparing in-class activity.
Added page for in class activity here.
Uploaded APK to google drive for sharing.
Added google form for results compilation.
Updated APK with new compilation, fixing some height issues when sitting down.
3/13/23 - 3 Hours - Creating Unreal Engine 5 Tutorial.
Added page for Unreal Engine 5 here.
Added information and link for UE5 within the VR Development Software Page.
3/16/23 - 1 Hour - A Short Analysis of Project Survey
Results of preference to VR over 2D visualization
Tends to skew a bit towards VR, but still have some people who prefer 2D.
Why?
Against VR:
Users note jittery performance of web application.
Some dizzyness of locomotion in VR.
Some noted the color coding associated with the 2D visualization is more intuitive than in 3D VR.
For VR:
Ability to move VR object around and view it from different angles.
Being able to see 3rd Axis is helpful.
Easier to view and compare two heightmaps in VR, more readable and engaging.
Recommended Improvements:
Improved movement in world, move to smooth locomotion instead of teleportation.
Implement collision boxes and prevent clipping in obstacles.
Add depth controls for map manipulating dataset.
Improving lighting setting.
3/20/23 - 4 Hours - Preparing for Class 3/21
Updated wiki contributions with attribution and information regarding contributors.
Read and took notes on 7 scenarios paper.
Project 2 Draft Design
Exploring different ways to implement multiplayer VR experiences in Unreal Engine 5.
Consider: separate plugins, and systems for implementation.
Use an already existing world (probably the one that I created in project 1) , and look at different ways to work multiplayer.
Random links that
https://forums.unrealengine.com/t/vr-pc-multiplayer/691364/2
https://www.semantic3d.net/ (point cloud dataset)
https://www.cvlibs.net/datasets/kitti/ (more point clouds!)
https://pandaset.org/ (autonomous vehicle dataset)
Another way to visualize a LiDAR dataset (very unrefined project, but dataset mentioned by Prof. Laidlaw made me interested)
Need to get more info from Prof. Laidlaw in order to continue with this.
Autonomous vehicle datasets?
Another idea: Spatial audio in VR?
7 Scenarios Paper:
(We'll focus on developing VR multiplayer in Unreal for this one)
Understanding Environments and Work Practices (UWP)
There are existing visualizations that allow people to collaborate with one another, determining how people want to interact with one another is important. Will people be in the same room? Will they be able to communcate via voice or not? We can use this information to help 'inform the design of a future visualization tool'.
Evaluating visual data alaysis and reasoning (VDAR)
Assessment of ability to support visual analysis and reasoning about data.
We must consider a given plugin's supported actions and interactions between users, which will help visualization on both sides. While this may be a game implementation, it also can be dependent on the capabilities of the multiplayer system.
Evaluating communication though visualization (CTV)
Must consider how well content is communicated between two users via plugins, namely, movement of player and objects throughout the field.
Evaluating collaboration through visualization (CDA)
This is fully immersed in the world of collaboration, so this is probably the greatest and most important evaluation here.
User performance (UP)
Maybe evaluation of how the user does in a set of certain tasks that depend on the other user to participate would be a good metric.
User experience (UE)
Measuring user experience can be done in various ways during this project, the main method being through a series of surveys about how users felt their collaborative interaction was, how the world was manipulatable between users.
Visualization Algorithms (VA)
Can assess performance of different plugins, namely latency, and connections between users. Must determine if content is working fast enough to be viable for use.
3/22/23 - 3 Hour - Preparing Presentation for Project 2, Assessing Possible Datasets.
See Here
3/22/23 - 2 Hours - Preparing for Class 4/4, Hunting Down Datasets
3 Questions from 7 Scenarios that relate to project:
Does the tool support effective and efficient collaborative data analysis?
Does the tool satisfactorily support or stimulate group analysis or sensemaking?
How are certain system features used during collaborative work? What are patterns of system use?
4/5/23 - 3 Hours - More Dataset research.
4/7/23 - 2 Hours - Trying to resolve issues iwith compatible formats, parsing data.
4/13/23 - 1 Hour - Preparing presentation for progress report.
See here
4/15/23 - 6 Hours - Getting Dataset into Unreal
Did more research into PandaSet Dataset and manipulation
Installed pandaset-devkit
Began trying to manipulate data, so it's usuable.
Was able to export data to console, where it printed ~167k lines of data points.
Wrote python script to sanitize data and only include X, Y, and Z coordinates.
Took console output and wrote to .txt file (see right for example).
Ran into issues trying to compute all files at once (resource allocation issue), resorted to taking a few at a time.
https://scaleapi.github.io/pandaset-devkit/sensors.html#pandaset.sensors.Lidar
https://realtimevfx.com/t/tech-art-need-help-to-create-a-gradient-in-world-space/14078/4
4/18/23 - 5 Hours - Unreal Manipulation
Enabled LiDAR point cloud plugin in Unreal.
Imported one .txt file into world for testing.
4/20/23 - 3 Hours - Unreal Edits
Ran into issues with export of Unreal Project to VR, trying to resolve issues.
4/22/23 - 6 Hours - Debugging Unreal Issues with Export
Tried a few methods to get application loaded onto local headset, no success.
Lidar Point Cloud dataset not compatible with export to android. Was unable to find a way to compile for Android.
Tried a few different templates (Blueprint based, C++ based), with no avail.
Tried to export as a windows application
Lidar Point cloud not compatible with instanced stereo HD mode in UE 5.1.1 for some reason, needed for VR?
Causes unreal engine to crash and shaders to not compile.
4/24/23 - 5 Hours - Fixing export issues, researching more into UE5 and multiplayer
Not many options for UE5 that are simple plug-and-play.
Only viable server hosting option seems to be Photon Cloud
Plugin exists for Photon Cloud, but isn't free ($50), since it's 3rd party.
https://thegabmeister.com/blog/unreal-vr-multiplayer-photon/
https://www.photonengine.com/sdks#realtime-sdkrealtimeunrealengine
https://docs.unrealengine.com/5.1/en-US/setting-up-dedicated-servers-in-unreal-engine/
4/27/23 - 8 Hours - Attempting to implement UE5 Multiplayer
Purchased photon cloud plugin with hopes of getting it working
Was able to follow tutorials that got me in VR on the server.
Untested with other headsets
Issues: no way to teleport...
4/29/23 - 10 Hours - Debugging issues with Multiplayer Setup
Setting up connections between users.
Followed tutorials, implemented teleportation for moving around the world via the blueprint system and some templates.
Debugging issue with snap turning that caused player to shift a bit back and forth rather than snap turning.
Tried to stop issue with showing back of headset when snap turning occured.
5/1/23 - 8 Hours - Debugging multiplayer
Debugging:
Having issues seeing other players.
Having issues connecting to server successfully.
Having issues seeing hands.
Tried to add smooth locomotion to server, but was unable to.
5/3/23 - 2 Hours - Presentation Preparation and Wiki Additions
Prepared project presentations for class.
Began wiki additions.
5/6/23 - 1 Hour - Project 2 Exit Ticket Results and Analysis
Quick analysis of results:
Slight preference to VR over non-VR counterparts.
Those who disliked VR version experienced significant issues with crashing or performance due to paperspace servers.
The dataset/interaction is unexplained and could use some more elaboration.
Users noted the helpfulness of having coloring for distance from a given vehicle.
Multiplayer Feedback
No menus to setup connection, just immediately puts user in the world.
It's hard to see other users, since they're just headsets floating in the world.
Unable to see controllers, which is somehting that I attempted to implement.
Contininuous movement (smooth movement) is required for multiplayer. Teleportation means that other users can't see when users move, especially when they just *disappear*.
Some users experienced glitches when the application started, connected, but were unable to move or see other users.
Paperspace vs In Headset
Paperspace presents some issues regarding setup and running the application.
Paperspace/running it on a desktop provides better fidelity than running it in-headset. Can provide faster computation and better performance.
Some noted that there might better performance with in-headset computation rather than wireless connection to a virtual machine.
5/10/23 - 2 Hours - Added to wiki contribution
Finished Wiki Contribution #6, found here
5/12/23 - 2 Hours - Added to wiki contribution
Finished Wiki Contribution #7, found here
5/13/23 - 1 Hour - Added wiki contribution
Finished Wiki Contribution #8, found here
5/13/23 - 4 Hours - Added wiki contribution
Finished Wiki Contribution #9, found here
Final Poster For Class
Google Earth VR
House/Neighborhood grew up in.
A neighborhood along the east coast, in New Hampshire.
Dorm/apartment/house lived in at Brown.
Place that's significant to me but not famous.
Local market area in Victoria, Australia.
Screencap of getting from Brown's campus to Victoria, Australia.
Google Earth Web
House/Neighborhood grew up in.
A neighborhood along the east coast, in New Hampshire.
Dorm/apartment/house lived in at Brown.
Place that's significant to me but not famous.
Local market area in Victoria, Australia.
Screencap of getting from Brown's campus to Victoria, Australia, but in Google Earth Web Edition.