Final project

Welcome the Lab weeks, a two week period during which my partner and I worked on creating an interactive solution that uses AR.

Augmented Reality(AR) are digital solutions that can be applied as a filter to the environment it is being projected on which adds(augments) the experience of the environment it is displayed on. For this project, we designed an AR experience with the goal in mind to help us as humans empathize with our environments as the aim was to make us more aware of our environments and the consequences of pollution.

Step 2: Collect inspiration

As in the recent projects my partner and I had worked with Artivive an online tool that allows you to upload videos and pictures to create AR experiences by scanning a picture and I had just also created my first VR -Digital exhibition using Artsteps. Below you find first some the pixelated missing moldboard followed by some direct links which I was able to recover. Feel free to visit and get inspired as well.

Step 3: Create three small concepts we would like to work on

On the left, you see my first mood board on the right my first three concepts.

Step 4: Merge concepts

Before merging our concepts we defined the criteria, 1 it should be interactive, playful and changing in moods(atmosphere). The initial plan was to apply AR to the concept. For this, I made 3 sketches of how I envisioned the interaction and the application of AR to a public space such as a park or an exhibition, as this was the learning goal of my partner. From my side, my learning goal really focused on learning to work with materials and designing in 3D. As I had "recently-ish" created a series of photographs that inspired the alternative reality of the human skin my partner proposed that we could use this as inspiration to create AR that allowed the person interacting with the AR design to feel connected to their environments. Our proposal, to create an AR experience that connects humans to the importance of materials/materiality by showing them that our choices matter and have an impact on how we experience the world around us.

Step 7: Time to build

We left Unity and opted for A-frame, but didn't work directly with A-Frame instead we worked with 8thWall a cloud-based VR/AR editor that supports A-frame. I wanted to stay in unity, but my partner opted for working with the online editor. After one day of being annoyed as I had been avoiding coding. I turned around and we moved on with thinkering. At this point I had been able to design the 3D assets ( "Art Pieces") in Vectary please refer to the pictures below. This moment was very defining for our project as we made some major changes.

Step 8: Coding

The first fail, from using the 8th wall cloud coding platform is that it didn't allow us to co-code. The program had a glitch that wouldn't save the versions properly and it wouldn't allow the both of us to work on the code at the same time, rather that it would go from one user to the other. Another difficulty that we encountered is that the web version which we used for testing often times would get stuck and the AR on the phone had difficulties with loading time. But we had already changed platforms three times therefore we decided to simplify instead of changing again.

Our goal: have the art piece being displayed on the screen, the user touches the hotspots the material of the users' arm changes on the screen to the material they just touched and there is a 3D asset popping up in the screen for a few seconds then which the disappears again. Ideally, the background music would change and this would allow to set different moods and feeling per material.

Some of the resources for creating the code:

A-frame animations list

AR

https://aframe.io/blog/arjs/

Transitions

https://aframe.io/docs/0.8.0/core/animations.html#sidebar

https://riptutorial.com/aframe/topic/10071/animation---a-animation-

Hosting

https://aframe.io/docs/1.2.0/introduction/hosting-and-publishing.html#hosting-models

https://aframe.io/docs/0.8.0/introduction/models.html##sidebar

Javascript

https://aframe.io/docs/1.2.0/introduction/javascript-events-dom-apis.html

Top left a screenshot of our dashboard with all the versions to our code top right shows in light blue how we were calling all our assets onto the scene. In the assets we created the id's for our models and the GLB files are the 3D files we created in vectary and uploaded to our code.

Step 9: Story Telling

Once we had finished our code it was time to improve our storytelling. For this, we went back to our initial idea of interacting with exhibition pieces in person. The story is that the user visits an art gallery, where they see the photography series("New Skins") I have made during the photography minor in the background.

What was needed: 1. Bio-Materials 2. Interactive Gallery Pictures that used AR and lastly the 3.QR-code leading to our interactive AR design. 4 Setting The Scene. Making the Movie.

1. Design the 3D assets needed for designing the AR material in Vectary. For this I used the 3D asset features and the material library to add skins to the different elements.

2.Create the AR material needed to upload the videos onto Artivive App. For this, I followed a quick tutorial on how to use Powerpoint to create short 3D animations to create the video content needed for the 3D assets to "rise" from the pictures.

How I did it:

Start e new PowerPoint project and upload the 3D object onto the screen. The Background can be clear or it can contain the image you would like it to have it pop out from. In this case, I first made the 3D movement. This was achieved by creating multiple slides the first slide contains the 3D object, and for each slide, you alter it a bit more in for example size/rotation until you have enough slides to create the animation. When you have enough slides you can set the timer and transition in between slides. The last step is to save it as MP4 and you will now have an AR animation popping out of your screen. From this, I learned about animation in Powerpoint as well as about working with 3D design in PowerPoint both were new to me. Below you can see the two examples of the PowerPoints I made. Please scroll down on each example to see how the slides progress creating the illusion of change.

Above you can see the transitions I created without background and one of the examples with the background as both are needed for adding them to create the Artivive interactions.

3. My partner uploaded the pictures I made, and linked these to the video animations I edited for publishing the AR exprience for the photographs below you can see a screenshot of the dashboard followed by the three pictures we used.

4. Cooking the Bio-materials for creating the mini-exhibition stand.

What was needed, gelatine, food colouring, some flowers and plants I collected outside, leftover coffee grind, hot water a cooker and a container to let the materials set in. A fridge to put the materials in for setting.

I followed the standard bio-plastic recipe from the course I finished just before starting HCI, you can find all the recipes with measures on my page assignments Bio-fabricating I added the gelatine stirred it until dissolved and added the colour. I made several batches some with different colours and some with different ingredients such as with the flowers and the coffee grind. Then I proceeded to cook a bio-foam for the stand of the exhibit this was made using gelatine, water and soap. You need to whip the mixture until it has become thick then it is also ready to be set in this case poured into a plate.

5. We realized that it might break the immersion and confuse users if to scan the QR code and suddenly have AR in Front of them, because of this I designed two extra screens that were added to the code. Screen one introduced the concept and screen two which gave brief instructions on how to interact with the AR application.

6. Filmed the movie, needed for creating the illusion of being at the exhibition, with a guided path for the viewer and send it to my partner. My partner brought everything together into an experience simulation video.

Presentation slides by Suzane Dos Santos above below screenshots of my phone screen going through the steps of the experience.

Looking back this was a project with a lot of learning and a lot o tinkering.

Tools I worked with during this project:

Unreal, A-frame, Rhino 3D, Make Human, Vectary, Artivive, 8thWall and Powerpoint. I learned to make 3D assets from scratch but I also learned to work with libraries and documentation when it comes to coding and programming. What I liked the most about this project is that I got to experience man tools in a very short time, what I liked the least about this is that I find that less switching at the beginning could have left more room for tinkering and finetuning of or sample. The biggest fail I find is that we didn't both have equal access to resources such as the code editors and some of the final works such as the final pieces created on the Artivive. Also, 8thWall wasn't for free and it cost my partner 100 euros to have the link be accessible when she handed it in but the same was no longer available for when it was my turn to hand it in. This project was one during which I also worked on taking a bit more of a side step when it came to my own personal preferences and I worked on being more flexible in going for the idea, and platforms desired by my partner. As I just mentioned I don't think we made the best choices in that aspect, but for sure it was a learning experience. In the future, I would work with open-source tools if possible and would make sure that all data, documents are stored in a shared accessible location. I believe this to have been one of the most valuable learnings when it comes to utilizing the tools available out there, as now I know what terms to look for "interactions/transitions/assets/file types(OBJ/STL/etc). Against my own initial wishes, I also got to practice my coding, I am very grateful for this as just mentioned I gained practice in applying the tools but I also learned to better read code and adapt it. When it comes to time management I find that I should focus on working more structured in a way that indeed it is good to try and make something work as that is how we learn but I wouldn't switch so often and I would keep better track of progress to better measure when it is time to move on to other solutions. As my learning goal for this project was to create an experience in VR and to learn to work with materials I extended my input and efforts and created a material experience that would fit our concept. For this, I joined the NeWViewSchool Summer course during which I learned how to design within VR. I learned to create SkYbox in Unity, Create animations with the timeline, work with 3D assets and the asset store, I learned to make materials similar to what we learned in Unreal Engine, and I also learned to apply triggers. All together to stay within the time frame of this course I created my experience within 4 days. I am very happy to have taken this step, I even submitted my project for the Styly NewViewAwards 2021. Below you find my final project, and some of the processes in making it. In the future, I will be learning more about interactions as this part is one that I have been struggling most with both in code or through tools. This had been a great experience I hope you enjoy my bonus assignment this can be experienced below. Altogether this was a really great experience with long nights that turned into days, if I could take one thing with me would be to not forget myself along the way. I was so busy with assisting others then in the end I didnt' make the time to complete my own submission the first time around. I have just recently learned about painting in 3D with tools such as Tilt/Open Brush I think my fun is just beginning. What I loved the most about this course is that it has helped me find my direction as UX-designer. For my graduation project, I have chosen to further develop my career in the direction of XR and Meta worlds.