Technology
Streaming Virtual Worlds
The Modrokkr system does streaming in a different way than traditional video streaming platforms. A key difference is that we do NOT do video streaming, rather applications are rendered within the client browser while the application logic runs on servers in the Modrokkr cloud. This decoupling provides Modrokkr users tremendous flexibility, performance and scalability and also provides our users with implicit multiuser capability, an important aspect of creating immersive environments. This approach is also ideal for VR and AR applications as there is no issue with visual responsiveness. Rendering and tracking are done within the headset or device, so the motion-to-photon latency is kept ultra low.
Because nothing is stored on the user's device, users do not need to manage applications or maintain large amounts of local storage. New applications can be tried instantly without need for installation or persistent storage. Our approach also eliminates security issues with app IP and cheating in multi-user environments. App assets are implicitly safe.
Integration of new technologies is much easier within the Modrokkr platform. Because application logic runs on our cloud servers, new technologies can be rapidly integrated without affecting framerate or resolution as is the case with traditional applications. Perhaps most importantly, Modrokkr realizes the ideal of having a single platform support multiple devices implicitly. For example, a bowling game that is written for Modrokkr will work on a Macbook, a Kindle Fire 10, an Android Phone, a PC or any other machine that supports a Javascript-enabled HTML5 browser. Relying on Standards such as WebGL, WebXR and Websocket, Modrokkr applications make full use of modern internet technologies in ways that other streaming systems do not.
Visual Tools
To assist developers in creating content, Modrokkr has a web-based tool called "Modrokkr Creator". The Creator is an integrated development environment that allows developers to create applications, import and manage both visual and nonvisual assets such as meshes and textures as well as instantly test applications in live sessions.
Users can also create custom tools for their content within the Modrokkr system, accessible using the same Web interface.
Capabilities
The Modrokkr system is highly adaptive, decompressing and rendering on the client device (within a browser session) with resolution, frame rate and visual extras all based on the client's capabilities.
The Modrokkr streaming system decouples rendering and application logic in a way that optimizes both while maintaining a very low bandwidth.
Low-End
On the low-end, we have targeted the Amazon Kindle Fire 10 HD. The Fire 10 operates at 1920x1200 at 20fps with a single shadow casting light and 2 fill lights per object. Hard shadows are supported without dynamic glow. The bandwidth for a Modrokkr app being streamed to the Fire 10 is just under 2 mbits/second.
High-End
For high-end systems, Modrokkr also adapts fittingly. A Desktop PC with a high-end GPU capable of rendering 8k at 240fps, 3+ shadow casting lights, soft shadows, dynamic glow, 3D fog, PBR, etc also uses just 2 mbits/sec.
A New Way To Compress Visuals
Just like lossy compression formats made the Web possible, Lossy animation formats are what will make high performance streamed 3D visuals possible. The Modrokkr streaming protocol is built from years of experience in building lossy, high quality 3D compression systems for Visual Effects and Video Games. The Modrokkr Team has built a compression protocol that is both simple and generalized for high performance, low bandwidth 3D animation. Our system uses a form of compression that is based on Finite Elements (FE). The core of this process is the combination of a mutable Finite Element mesh with an immutable barycentric mesh.
When a user visits a Modrokkr application page, a small Javascript application is downloaded an a stream is started. During load time, the visual meshes and textures are sent along with the code to decompress the stream and play it back. The Javascript application also makes use of various web technologies to interface with VR headsets, establish AR sessions and perform data fusion for I/O devices such as controllers.
Once this is done, a stream is initiated to the actual application running in the Modrokkr Cloud. This application begins sending an ultracompressed data stream consisting of the FE mesh vertex positions. The client then uses those positions to deform and/or transform the display meshes.
Local transforms are performed within the browser so all local 3D transforms are very fast enabling unimpaired 3D movement for 2D as well as VR/AR displays.
The Mutable Modrokkr Finite Element Mesh
The Modrokkr Streaming Protocol changes the vertex positions of all mutable Finite Element geometry which is in turn used to barycentrically deform visual geometry. Character animations (seen in green) as well as fracture and deformation of environmental objects is easily accomplished with a single protocol.
The Rendered Mesh
The mutable deformations are performed by a special shader which decompresses the Modrokkr Stream. The Modrokkr Stream is generated in real-time entirely with a CPU on the cloud server the application runs on. Only a single stream needs to be generated for a scene and it can be shared with multiple users. GPUs are used on individual browsers to decode and apply the stream in real-time.
Perfect Synchronization
A Modrokkr Virtual World has no issues with synchronization between multiple users because nothing is synched downstream. Everyone sees the exact same visuals because they all receive the same geometry stream. This allows anything in a virtual world to be implicitly synchronized whether it is a finite element physics simulation, character movement or anything else that is changing dynamically within a virtual world. Rendering is done locally so there is no responsiveness delay in movement. You can see an example of this within this demonstration game we built.
Frame Interpolation and Lossy Compression
The FE vertex positions are normally linearly interpolated from assigned draw times. This means that animated movement within the Modrokkr system is lossy but not in a way that is noticable during regular use. Much the same way that JPG compresses images, the Modrokkr geometry stream compresses animation.
In all cases, the Modrokkr system is using the GPU to do the decompression and subframe interpolation so movement is buttery smooth.
Linear Compression
The compression/decompression that Modrokkr performs is also ideal for the internet in that it is highly efficient. 430:1 is a typical ratio though for higher frame rates 1720:1 is common.
Handling Non-Linear Movement
Sometimes objects moving within a Modrokkr scene are moving very fast in a nonlinear way. A bowling ball is an example of that. If simple linear interpolation was used, the lossy compression technique we are using would result in deformation artifacts. To resolve this, Modrokkr allows subframe interpolation to improve visual accuracy.
Platform As A Service: Cloud Render vs Client Render
To understand the fundamental difference between how Modrokkr does streaming, it is good to compare it to the "traditional" approach of doing streaming using video compression.
Rendering in the Cloud (Video Streaming)
Rendering in the cloud (video streaming) is done by having the app run in the cloud and output the visuals to special hardware to do a fast video render. This video is compressed in real-time and sent as a video stream to the user's client. Control (input from the user) is sent upstream to the app for processing and is reflected in the video stream.
Rendering on the Client (Modrokkr)
Rendering on the client (Modrokkr Streaming) is done by having the App run on the Modrokkr server with visuals encoded into a geometry stream which is then sent to the user's session on a web browser. Decompression and rendering is done using the GPU within an HTML 5 WebXR session. Control is sent upstream from the browser to the app and is reflected in the resulting geometry stream.
Efficiency
Unlike video streaming, the Modrokkr approach is highly efficient in terms of both power and bandwidth. A single cloud app can send the exact same geometry stream to many users and integrate input from all of them separately. No GPUs are required in the cloud as the encoding is CPU-centric. In fact, a single Raspberry Pi 4 can run an FPS that be played simultaneously by 10 people! The rendering load is handled by the web browser on each user's browser.
Modrokkr's power per user per year is approximately 19 cents. Compare this to a traditional interactive video streaming system which consumes approximately $190 per user per year. The main reason for this is that a single geometry stream can be generated by an app which is shared by many users, where a video stream-based approach necessitates a video stream being generated for each user (because they each have a different perspective that needs to be rendered individually). On top of that, the bandwidth for a video stream is very high because real-time encoding is not nearly as well-compressed as what is used for movie playback. For reference, movies played over the internet are heavily processed in a distinctly non-real-time way to have high fidelity lossy compression that keeps transmission bandwidth down.
Push Streaming
Modrokkr push streaming allows visuals from multiple sources, even multiple vendors, to be combined on the user display allowing seamless transition from one virtual world within the Metaverse to another. Programs representing payment systems, simulations and even other users can be "pushed" into a particular user display using this technology, creating an entirely new way to access content within the Metaverse.
Powerful Tools and APIs
Modrokkr has tools to allow users to upload content such as textures, shaders and meshes as well as tools to create levels, modify scripts, create particle systems, preview apps and publish apps for worldwide distribution within the Modrokkr system. Collaboration is fairly straightforward as well through sharing mods.
Modrokkr also offers powerful physics systems such as DMM and Bullet for use within immersive environments. Many other advanced technologies will soon be available including AI, procedural generation, liquid simulation among others. The traditional issues of porting and integration are far simplified with the Modrokkr approach. VR and AR are fully supported and are simply options for an app. Sharing is implicit and easy to add to any Modrokkr App.
The final result is a means of deploying visual software that is frictionless, low bandwidth and rich with features. The SaaS revolution has already taken over software for Office Productivity software, CAD, CRM and IT. Modrokkr is bringing it to Visual Applications and immersive environments.