Project CamIoT

The project explored a wrist-mounted wearable IoT system that would recognize appliances and interact with them via finger gestures. While smart-home IoT systems are rather prevalent in modern society, the most popular systems rely on verbal commands, which may not always be feasible. For example, verbal commands require a nearly clear sound stage, meaning any background noise can interfere with the command. Therefore, we aimed to explore a new way of interacting with smart appliances via gestures. 

This was my first project as a research intern at UCLA HCI Lab in the beginning of 2020. As a sophomore in my undergraduate program, I was not acquainted with the research process, nor with programming beyond a relatively basic level. However, through this project, I learned how to program in Python, use OpenCV for image processing, and work with Raspberry Pi's. The process only became more difficult as the pandemic spiraled out of control in March of 2020. Everyone was sent away and could no longer physically contribute in the lab, crushing everyone's motivation to continue the project. However, after a month of hiatus, we got back to work remotely and finished the project. The researchers and I submitted the paper to IUI and UIST but were unfortunately rejected. Despite this, I still view the project as a success in that I was more comfortable working in a research environment, gained many new skills in the process, and found a way to finish the project despite the pandemic being at an all time high.  

Data Flow:

The system starts at an idle position by the user's side and initializes upon raising your arm. This gesture is detected and recognized by the IMU. Upon raising your arm and pointing at an appliance, the camera streams images to a server program via socket running on a computer that handles the image processing and appliance classification. Here, the direction of the finger is determined via a combination of color space filtering, canny edge detection, along with segmentation and dilation to isolate the largest pixel island. From this, the direction of the finger can be extracted and thus the appliance being pointed at in frame. This allows for multiple appliances to be in frame and the correct appliance to be recognized. The appliance being pointed at is then cropped from the photo and input through a ConvNet. Upon appliance recognition, the user is informed via an audio interface of the selected appliance and can begin interacting with the appliance by pointing in a certain direction. The user's pointing range of motion can be visualized as a semi-circle, divided into three quadrants. Each quadrant was hard-coded to perform a certain function for the chosen appliance. Therefore, once images are sent of the user's pointing in a certain direction,the same finger direction algorithm is used to now determine what function to perform on that appliance. This command would then be sent to the appliance via the smart home hub. 

I began by performing small software implementation tasks, such as getting the IR camera to work properly with the system and debugging any problems. I also acquainted myself with the basics of communication of network sockets. After the pandemic hit, I  worked on the audio interface of the system and created the database of appliances in which selected appliances would have certain commands assigned to them, i.e. a lamp would have the commands brighter, dimmer, and off. I also worked on improving the "raise arm" gesture detection by retraining the system with hundreds of trials. I wish I could have done more, but I recognize now that times were rather chaotic back then and I did the best with what I had on hand. 

I have included the demo video we submitted to UIST'20 below as well as the paper we submitted. Please take a look!

For more details and to check out the implementation, please see the repository below:

• GitHub Repository: https://github.com/a-chen711/camiot

Big thanks to my mentor and collaborator, Amirali Omidfar, PhD student Yuan Liang, and Professor Anthony "Xiang" Chen.