Identify mushrooms and play with their virtual characters

Team

Thomas Lawrence - Manager, ML/Lens Studio Development

Mengxuan Jia, Shengyi Gong, Xinyao Lin, Yijun Ma, Zhihui Sun

Concept

An augmented reality application which detects types of mushrooms and then provides a custom interactive experience.

Why an AR mushroom detection application?

Mushroom Identification

Mushroom detection applications already exist such as Picture Mushroom.
These detect which species of mushroom is within the user's camera frame.
These applications are informative and provide educational content for users.
This content is aimed at existing mushroom enthusiasts.
However, could augmented reality be used to interest a wider range of users?

Augmented reality interaction

We try to bring to life both mushroom and information related to each mushroom through augmented reality interactions.
This experience is aimed at children and young adults.
Experience is both for education and entertainment.

Mushroom Personification

Animated characters which personify each individual mushroom with sound effects.
Characters represented by 3D models.
Virtual animated characters bring to life the static real-world mushrooms.

Hand Gesture Interaction

Can playfully move virtual mushroom with your hands.
Movement to simulate picking the virtual mushroom.
Playful tactile stimulation physically engages users with the mushroom.
Can simulate picking the virtual mushroom without picking the real one, which could be poisonous.

Augmented Reality Lens Filters

We wanted to educate users on the effects of eating the individual mushroom.
For this we used lens filters which illustrate how the world around them would look if they ate the specififc mushroom.
This allows users to learn about the effects of each mushroom without eating them which could be dangerous.
Feature is both entertaining and educational and engages users with their environment.

Design

How our augmented reality experience is structured

User Journey

Step 1: Mushroom Search

On application start the user searches for mushrooms.

Step 2: Recognised Mushroom Information

When the application detects a specific mushroom species information is displayed related to the mushroom.
An animated and interactive 3D sprite associated with the mushroom appears.
This can be interacted with and talks telling the user information about the mushroom.

Step 3: Mushroom Hand Gesture Interaction

The virtual mushroom can then be "picked".
The user uses a picking hand gesture to interact with the sprite.

Step 4: Eating Mushroom Effect

A lens effect is then shown related to the specific mushroom chosen.
There are three lens effects for the types of mushroom; hallucinogenic, edible and poisonous.
The lens effects each illustrate the type of mushroom's effect on eating.

Implementation

How we built the application

Device

The application is for mobile phone augmented reality within an environment.

Framework

We used Lens Studio Augmented Reality framework to build our application. The app is then viewable as a filter on Snapchat.

Object Detection

To train the mushroom detection model we used TensorFlow and integrated with our application using SnapML.

Individual Contribution Timeline

Week 1 and 2

I led the brainstorming effort for various ideas.
I researched ways of implementing desired features especially using Snap ML for object detection.
I helped develop the initial storyboard (pictured left).
I led dividing the team into roles and responsibilities.

Week 3 and 4

We went together on a trip to Kew Gardens to research mushrooms for foraging (image left),
I trained initial prototype TensorFlowLite model.
Exported model as TFlite file and integrated with LensStudio as SnapML component.
Used a Classification template to detect a single type of mushroom (Fly Agaris).

Week 5 and 6

Trained a TensflowLite model which could detect three species of mushroom (Google Collab Script).
Created a script to respond to multiple classification classes on LensStudio (FILE HERE).
Organising and scoping of other streams of work (UX, Hand tracking, model creation) so we could complete project by the deadline.

Week 7 and 8

Refinement and improvement of the object detection model using more images.
Large amount of rescoping was needed on other streams of the project to complete by the deadline. I helped organise the other team members to ensure the whole project was completable.
I integrated all the different aspects of the project into applications (object detection, UI, hand tracking, 3D models, animations, sound effects, lens effects). Using behaviour scripts for integration logic as pictured left.
Ensuring that the end to end journey worked together. We used a separate application for each mushroom journey.
Final rounds of testing and debugging of user journeys
Organisation of final presentation

Reflection

A discussion of the outcomes of the project

We met our initial goal; we created an entertaining application which proves how Augmented Reality could be used to create mushroom picking experience. This experience goes beyond applications that already exist; it uses Augmented Reality to do more than provide factual information about the mushrooms and engages users with a playful experience.

The machine learning model for mushroom detection was an effective proof of concept of how object detection can be integrated with Augmented Reality. It was an entertaining challenge learning how to implement object detection. However the model was not on the level to be used to safely detect mushrooms during foraging. This could be improved by improving the model; refining the training algorithm, or using a larger and more varied training dataset of mushrooms. Also we ran the object detection algorithm in real time; this is performance intensive and slightly unpredictable. It would have been simpler and more effective if we asked users to take static pictures of mushroom from set angles, on which the model would be run. This would be an easy route to safer mushroom detection.

The user experience was intuitive and engaging. We stimulated the user visually (lens filter, animated models ), audially (sound effects) and tactily (hand gesture interaction). However this stimulating experience could have been improved by removing the many buttons in the interface, although they were needed to flow through the application, they could have been replaced with simple screen taps. The buttons detracted from the sense that the user was within a space with the real mushroom and the virtual sprite.

Our project was very ambitious with a large team working on multiple different features. It was challenge ensuring everyone was working toward the same goal and delivered their work in time to be integrated in the application. Leading this initiative was rewarding when the final project was complete.

Many of the application's features were performance intensive for a mobile phone; object detection, animations, handtracking etc. Integrating them into a single stable application proved difficult. Although we managed to get the indivdual mushroom user journeys into a single application which flowed well together, we decided to separate the full journey for individual mushrooms into different applications. I do think it could have been feasible to put the three mushroom journeys into a single application; however this would have taken further performance enhancement and testing. The issue could be resolved by using a more substantive Augmented Reality framework like AR Foundation.

LensStudio works very well for building smaller applications and does shows the promise to be expanded into a heavier AR framework; It is extremely effective at merging Augmented Reality with Machine Learning components with SnapML. However LensStudio limits the scope of its applications owing to its performance restraints.

This was a rewarding project where we successfully produced a proof of concept combining machine learing and Augmented Reality to create a mushroom picking application.

Page updated

Google Sites

Report abuse

Mushroom Picking AR