The journey- Chronologically 

First week back started strong with a few ideas i thought of during the winter break and was eager to try discuss with our mentors and start the execution. This week passed by smoothly with minimal contact hours on campus but a big responsibility on our backs, pitching our project to our preferred advisor. Since my project was technology based with hints of game design, my options where limited down to 2 advisors. After sending my projects and discussing them further with each, the final verdict was reached and my advisor was set :’)

Since projection mapping was the biggest aspect of the execution, it was an easy decision.

While most of the first semester was spent on research, there was still a huge lot to conquer. Since I was mainly working with AI art generators and these have been changing by each passing minute, I had to restart many of my trials. While I still used the same platform, the technology they’re working with has been adapted and altered to better suit the prompts written. As I dug deeper in researching I came across a few technologies to test, including stuff like copper wire and electric paint or even arduino touch sensors. One of the biggest updates in the progress is changing the projection mapping concept from a book to a building with windows that each carry a story that the audience could pick from to view.

The other progress was that I collected most of the stories from friends, family and strangers who had stories to share. The story collection process was quite out of my comfort zone as I approached many strangers, explained my whole project and purpose to them and then collected what they had to say. Yet the experience of sharing stories with them has been life changing, it brought me joy and clarity in ways I never imagined.

These couple of weeks have been quite entertaining as I deeply enjoy researching and expanding the realms I could explore and experiment with. 

Since our first checkpoint is approaching, I’m spending this week working on the presentation to have progress to present. I worked on editing all the audios so that I have a clear path of what I still have to do. Unfortunately out of the many audios i collected, many where corrupt as the mic I was using (Zoom) is extremely sensitive so it caught up on the background noises which made it difficult to clean up and share the story, regardless of them being some of my favorite stories :( 

After the past few weeks of research, I settled on 2 technology concepts that I want to explore to create this interactive experience, copper tape and electric paint or using motion sensors like Kinect for example. After transcribing the stories and feeding them into the AI generators, I collected them trying multiple ways of animation to be more entertaining for audience. I tested ways like cross reference research on Google, loading screen using after effects, smashing and morphing all the images generated yet unfortunately non satisfied my vision. After a long time of testing using multiple platforms I came to the conclusion that the best thing right now is to keep them as stills with just the audio in the background. I compiled all that research and the few created videos to showcase and to explain the potential this project holds. Adding to that, a simple mock-up of what the exhibition space is meant to look like was created using spline as a 3D modeling app to convey the vision of the building with windows to the panel of judges. 

After the presentation, I decided to take things slow to be able to overlook all the steps and not miss something. I continued to collect stories since many where unusable from the past weeks yet this time a few got corrupt and I could not extract them from the SD card. But I currently have enough stories to test my theory and show people their stories back to gather more input. Which is precisely what I was able to collect, people immediately recognized their AI generated images and easily empathised with stranger’s stories which confirms the whole humanisation theory, a win this week.   

I started by trying to figure out the best output to use for projection mapping to convey the building yet still only require one projector. After visiting the exhibition space created by level 5 I found it, the best thing I could currently use is the podium we use to showcase our work on top of. It works perfectly with the vision I had, so I decided that it would be best to ask for it from inventory. I had a meeting with Doaa Darwish, an interaction designer who worked with touch designer and projection mapping a lot before. She told me about the difficulties of the materials I required and how they’re almost impossible to retrieve in Egypt or the MENA region in general. After many failed attempts to find them we settled to start testing Kinect  as the sensor to activate the interaction. Although, I already had the Kinect I was in need for the adaptor which hindered the progress a bit.

Now that the adaptor has arrived and I can work with it, I started testing it. Since I already knew from before that Kinect is unavailable on OS, I downloaded Windows 11 on my Macbook (would definitely not recommend, don’t try this at home) Other than how time consuming and risky that was, it was unfortunately unsuccessful as well. Yet to figure that out was not easy, so i wasted alot of time to reach the conclusion that for touch designer to fire up it requires certain prerequisites that weren't needed when on OS. Since the main problem was that the GPU driver required for Windows was unavailable as the hardware is not even of a PC but rather a Macbook, i had to start using a proper PC. Doing that at university was unfortunetly quite difficult so i had to work on my own in that.

As my sister does own an HP laptop, i decided to download the required platforms on it to test using the Kinect (SDK Kinect 1.8 & TouchDesigner) while the Kinect did work, the GPU driver was also insufficient and so that failed too :(  

After a while i was able to retract another laptop that was finally able to open up touch designer. Now that things are back on track again i had a few tutorials with similar interactivity concepts to what i want to create. As many tutorials found online, many did not work how I expected which hindered the progress more. Whenever something worked, another part would explode, so troubleshooting was mainly what was happening. The main issue lied in the lack of similar tutorials to what i want to get done especially using Touch Designer, many where done using platforms like VS code. After spending some time researching, i soon realised that the Kinect i have does not have hand gestures as it is the older version.I asked around everyone i know if they have the newer one so i could borrow but it again that failed. Soon after, a friend of a friend found it and finally lent it to me and i finally started working using interactions and testing all the tutorials found.

The final few weeks started off pretty intense as there was still so much to figure out and very little time left. Soon after testing, i have come to realise that even though the GPU driver could open up touch designer, it still could not handle elaborate nodes and interactivity that i wanted so every addition should be counted as any wrong turn crashes the software, setting us back in the progression. I was able to create a simple prototype of what the projection on the building will look like and then added the videos as textures on top. I then created a cursor interaction, replacing the input from the mouse to the Kinect skeleton panel. This allowed me to make the interaction motion based so it could serve the purpose of this execution. While this was a success, i was not able to tailor it to my vision, every time i tried to link the interaction with the specific video something crashes. When all failed, i turned to AI tool created by Bing as ChatGPT is blocked in Egypt. It was helpful in some areas where it helped me enhance what I already have but it was not able to guide me with the initial problem. The closest thing i found was created by a creator in Portugal, so i emailed him asking him for help which was available on Patreon but again blocked in Egypt :( 

Since the exhibition is next week, i had to research alternative methods as plan B so I'm not left with nothing to show. Those options where including Arduino or even keyboard input interaction.

While working simultaneously on other alternatives, i was able to find someone on campus that could be able to help me. This is where Doaa Darwish, a Digital Media lecturer who offered to help me came into action. After days of spending hours on campus trying to fix the code and nodes we were able to figure out a simplified version of this execution, the day before the exhibition. Since choosing a certain video from many was quite challenging we settled on creating a switch that alters between the different videos as per the position of the person's right hand. The video is then played and paused via the grabbing hand gesture so that it becomes the play/pause button. This worked perfectly for the technical showcase but what was left is integrating the video and audio together to play simultaneously. I finalised that audio part in the morning making every video start off with the audio working. Once it was show time, a problem started arising as the laptop kept crashing every time the audio files are reloaded. In addition to that, every time i tried to connect an audio device it would make the laptop freeze causing it to stop working.

Due to all those technical difficulties, I decided it was best to eliminate the audio regardless of it's importance as my laptop would not be able to run anymore. This took a huge part out of the interactivity unfortunately but all those aspirations require a PC with better a GPU driver which was not easily available. Some of the problems that were not accounted for were the overheating of the technology. The projector, laptop and even Kinect shut down multiple times due to working for long hours so i had to take sudden breaks even if people where in the space. The good news was, most enjoyed the experience or where atleast intrigued by it which is all i wanted while pursuing this project, the look of wonder, the emapthy, the giggle when interacting with the kinect like little kids. That suddenly made all the effort and crying and worry that passed worth it, it was truly euphoric.

Hope you enjoyed this journey along with me, it was a great finale, ill sure miss all this :')