Dewi ♥️ The desktop companion

An augmented reality prototype of a desk companion that provides a fun and micro engagement. The challenge is to provide an interactive experience that uses the timing of animation, audio, and micro-interactivity for a fun little distraction from the busy day of an XR designer.

GOAL

Gain hands-on experience using Adobe Aero for AR prototyping, focusing on an interactive experience using timing, animation, audio, and micro-interactivity.

HOW

Design and build a fully functional prototype that lets users interact with an AR 3D experience in an intuitive way on mobile devices using Adobe Aero.

Prototype Demo

5B1466FE-2C34-460D-B18E-07A52B48B81A.mov

Adobe Aero projects can be accessed by a QR code or shared through a link to the cloud for the Adobe Aero reader.

Project overview

This case study follows the development of an AR prototype conceived as a cheerful diversion for XR designers. It’s a tale of blending learning with doing, as I embraced the challenge of creating an engaging AR experience that marries animation, sound, and interactivity.

Design & Prototyping Process

Inspiration

3D Remix of a cute ghost made by a WOMP community member.

Bezi contributor, Elizabeth Hum's, AR widget, Desk Duck.

Sketch

Lo-Fi Hand Sketch

Sketching out main ideas for transitions and scenes.

Button & Prompt UI

HI-FI Canva

+ Adobe Photoshop

+ Adobe Aero Mobile app

The most challenging part of the project based on the limitations of the software.

3D Asset Creation

Womp to create 3D assets and plan out scenes.

Luma AI's text to 3D Genie to generate several project assets such as the headphones, the black cat and the tombstone.

Building Out Interactivy

Creating Dewi's personality by selecting and creating various accessories, engaging audio, and using character movement to convey emotion.

Testing & Troubleshooting

Prototype Testing Devices

iPhone 15 Pro
Mac Book Pro M2

How the behaviours are listed or stacked influences scene planning and interaction.

CONCEPT IDEATION

The idea for Dewi was sparked by a fusion of creative musings from the WOMP community and a Bezi AR widget by Elizabeth Hum.

These elements were the stepping stones that led to sketches and digital modelling, setting the stage for a narrative-driven AR journey.

(1177) AR Widget Desk Duck by Elizabeth Hum - YouTube.mov

TOO many ideas and limited options within Aero to create navigation. I included a welcome screen for the first layer of interaction.

TROUBLESHOOTING ADOBE AREO

TECHNICAL TREK

Balancing software demands with hardware limitations, Adobe Aero tested the mettle of my MacBook and iPhone 15 Pro with its resource-heavy needs. Multitasking with Photoshop added to the juggling act, pushing my device to its limits. Despite these hurdles, the process was a valuable lesson in optimizing workflow and managing digital tools.

Aero is a wonderful tool that I will continue to explore. But on the flip side, after being in beta for four years, Adobe Aero still has some issues that need to be addressed. These issues include problems with file uploads, overlapping audio, and a lack of response to Adobe's community boards. With Aero's immense potential, it's disheartening to see that these critical user issues are not being given priority.

Reference:

https://community.adobe.com/t5/adobe-aero-discussions/can-t-import-anything-to-adobe-aero-desktop/m-p/13315290

https://community.adobe.com/t5/adobe-aero-bugs/object-doesn-t-show/idi-p/13847792

https://community.adobe.com/t5/adobe-aero-bugs/problems-with-importing/idi-p/13347648

File errors uploading PNG files made in Photoshop

Screen Recording 2023-11-29 at 1.17.29 PM.mov

Dragging files into Aero desktop and glitches.

Tech support for Photoshop layer imports into Aero
and crashing the Aero app.

More file errors when trying to upload to Adobe Cloud

Asset artifacts lingering behind when switching scenes. This was solved by listing and then hiding each asset within each scene's behaviours.

aero_recording.MP4

GLITCHY BUTTONS

The initial vision was ambitious for this project's navigation. The concept was to have interactions similar to hotspots or tooltips by applying behaviours to elements of the navigational assets. Various different types of navigation ideas were experimented with and considered, but a simple sequential loop made the most sense.

This video shows the issues experienced the navigation not responding and "glitching".

DESIGN DYNAMICS

Initially drawn to intricate, layered interactivity, the project's scope evolved into a more streamlined approach, focusing on sequential navigation to enhance user flow. This pivot was not just a technical decision but a strategic move to keep user engagement at the forefront.

As with most design software, the project layout and size using the desktop view vary in placement when using the app preview and then publishing to the app. This occurred with the navigation panel veering off.

This video shows how the navigation went askew.

aero_recording-movedNavigation1.mov

3D OBJECT CREATION

Womp was used to create 3D assets, such as the various models of Dewie and the cat mask, premade by the Womp community to plan out the basic elements.

Luma AI's text to 3D Genie was used to generate several project assets, such as the headphones, a VR headset, the black cat and the tombstone.

Although the headset was not used in the final project, listed below my closing thoughts section, I have included an experimental video clip of Dewi dancing and wearing the VR headset.

Optimization

File size optimization was a crucial milestone, achieved with the help of RapidCompact, significantly reducing the digital footprint of the project. From 115 MB to 58 MB, this not only improves accessibility but also ensures a smoother user experience.

ENGAGEMENT AND EXPERIENCE

Each scene in Dewi's world was crafted with care, integrating behaviours and audio to flesh out a character that's more buddy than bytes. The project's soundscape, enriched by carefully chosen effects, added layers to Dewi's charm, making each interaction a memorable moment.

The entry page welcomes the user before loading the navigation menu with four scenes and a ghost button that returns the user to the welcome page.

Each scene has at least two behaviours that animate or trigger audio. Free recordings of a toddler's first words gave this character personality. Making it so cute that you forget it's holding a large knife behind his back.

IT'S FROSTY. The scene has a Christmas tree with a spinning star, and Dewi is dressed in a Santa hat and sipping on a latte, complete with slurping straw sounds.
CHILL OUT has Dewi dancing to club music with two animated disco balls.
TRICKS has the Dewi the ghost flying through the air, expressing a defiant attitude. The bat companion and a thunderous cloud also move.
THE G'D NITE scene shows Dewi sleeping/snoring/sneezing beside their black cat, with a twirling Maple leaf by its feet.

IT'S FROSTY

CHILL OUT

TRICKS

G'D NITE

REFLECTION AND NEXT STEPS

The journey with Dewi has been an enlightening foray into the world of AR. The challenges faced and the knowledge gained have paved the way for future explorations in AR development. Looking ahead, the experience has inspired me to consider other software platforms and to delve deeper into the potential of AR as a medium for interactive storytelling and user engagement.

Given more time (and budget), I would definitely explore:

To enhance this experience by auto-lip-syncing the voice to facial expressions in Blender and importing them into Unity. Create a similar voice from the project audio clips through a service such as 11 Labs or Icloned.

The next level would be to connect a rigged and AI-responsive model as an extension of a GPT as a personalized agent that communicates as an assistant, possibly integrating with the user's Notion or other accounts.

CLOSING thoughts

Through the trials and triumphs of developing Dewi the Ghost, this project stands as a personal testament to the learning curve in the fast-evolving field of XR. It's a narrative not just about a student project but about the resilience and curiosity that drive innovation in technology.

Being a part of Circuit Stream's XR Design and Prototyping program has dramatically challenged my design thinking and enhanced my software skills and processes.

Although I faced several technical challenges, it has been a fantastic experience that has broadened my horizons and exposed me to the immense potential of XR technology.

I am passionate about introducing XR to the publishing industry and discovering new ways to improve readers' experience, the author's expression and marketing exposure. I firmly believe that XR has the potential to break the boundaries within the publishing industry and generate fresh prospects for creative expression.

aero_recording 4.mov

HOW TO VIEW the ar experience

app clips

An ‘app clip’ is a lightweight version of an app that lets users experience some of the app’s functionality without needing to download and install the full app. Aero provides a lightweight viewer as an app clip.

Viewers don't need to install the Aero app to view the AR content

Viewers do not need to install an app. Viewers on recent iOS devices can view content in the Aero app clip without needing to install an app first.
Viewers who launch (or “invoke”) the app clip using one of the supported invocations are presented with an app clip card on their screen and can proceed to open the experience in the Aero app clip.

Users can launch a shared Aero experience in the Aero app clip via:

Using the device’s camera to scan a QR code.
Links in the Safari browser.
Links sent with iMessage.

The size of the project is 58MB. It should load within a few seconds with decent wifi. LINK to Dewi the Desktop Companion.

Once open, the app will instruct the viewer to move their device to pin the digital mesh onto a flat surface, such as a kitchen table or a clutter-free counter. TAP the pin to anchor the AR experience.

Move the device or step back until Dewi's welcome graphic comes into view on the device screen, then TAP the yellow circle.

From there, TAP the purple buttons to cycle through the scenes.

Please note: The software has an issue with timing and audio. For example, if jumping to another scene before the audio is finished, the audio may overlap and play on top of the next scene.

App clips are not supported on Android, but an equivalent Android instant app feature is coming soon.

ATTRIBUTIOn

Disco ball with coloured lights" (https://skfb.ly/oFoUO) by WwWuengy is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).
"Disco Ball animated" (https://skfb.ly/6SIQQ) by Susan King is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).
"Christmas Lights" (https://skfb.ly/o8MKO) by tronk is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).
"Candy cane lane /" (https://skfb.ly/oBNLA) by ElOsitoAzul is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).
"Stylized and Kawaii bat" (https://skfb.ly/oKq6u) by ElvisOrtizB is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).
“Child Audio” sound effects by SergeQuadrado Child says: OK, Hi, Good Bye, No, Yes, and Laughing https://pixabay.com/users/sergequadrado-24990007
“Christmas Ident” sound effects by SergeQuadrado https://pixabay.com/users/sergequadrado-24990007

Page updated

Google Sites

Report abuse