Source Code:

https://github.com/kucaantunes

Carlos Filipe Paiva da Cunha Macieira Antunes. Concluded a Bachelor's Degree in Management Informatics Engineering on 11/26/2008 at the Polytechnic Institute of Setúbal Escola Superior de Tecnologia de Setúbal. He completed MSc from Instituto Superior Técnico in Information and Business Systems on 11/28/2022. Completed MSc in Computer Engineering from ISCTE on 22/12/2022 Attending PhD in Science and Web Technology from UTAD and Bsc in Economics from Nova SBE. Former programmer / software engineer for Microsoft (Spain), Pfizer, BNP Paribas (Spain), ISQ, Proximus (Poland) and Airbus (Spain). Strong professional experience in programming, namely in C# .Net Core, Java Spring MVC and Boot and PHP languages.

MotionControllers

Source code:

https://drive.google.com/drive/folders/1gY2r6_ABqZbMkuwe0ia54R-PyIaWZRqm?usp=sharing

https://drive.google.com/drive/folders/1qzZbKDE_jOy_lQJlBgJAO_Epne4C3_6D?usp=sharing

Hand, body and eye movement controllers

With the objective to perform a more effective training for users of different ages and disabilities and to make the learning process easier, an online motion controller was implemented via web using several programming languages like JavaScript. Open CV is able to detect the players movement. Via python code it is possible to detect the position of the hands during web camera video streaming. This technology detects the hands movement while the user is interacting with the online platform, allowing to control virtual objects that interact with the system. Another possibility of getting hand movement instead of a camera is by the use of specific sensors like leap motion that has a library that allows to use the Unity game engine to develop applications which can be used for different areas like training and healthcare. With JavaScript it is possible to develop applications that interact with the movement of the hands in real time, facilitating the use for people that have visual limitations or that do not have a keyboard or a mouse.

Learning activities can be more interactive and accessible for users of different ages and types of disabilities. Based on the context of training and to facilitate the performance of physiotherapeutic tasks a controller that detects, hands, arms, legs, head and feet was used to answer the third research question of this investigation. Based on the creation of AI Agents, like the creation of a Computer-generated Imagery (CGI) girl, the sensor which in this case is the webcam requires to capture the all body in order to associate geometries and bones. A virtual girl replicates the user movements on a projected Liquid Crystal Display (LCD) that aims to help in physiotherapeutic terms by providing movements and exercises that can help some motor disabilities. The prototype was developed in C# using the Game Engine Unity providing a Mixed Reality (MR) experience. On a perspective of virtual storytelling, this project uses a process of providing interactivity to the user for story creation.

Via the Unity IDE it is possible to configure the camera settings and the lightning and also to implement AI mechanisms like the behavior of the virtual elements among other possible functionalities. The programming language used to develop the prototype was C# and some libraries and classes were imported in order to implement all the features of a mixed reality first person shooter for mobile phones compatible with some of the operating systems of the android family.

Preview of the prototype ARFPS.

Concerning the shooting system, a ray cast function was used taking in consideration direction, speed, position and distance and also spawner game object mechanisms in order for the bullets to appear after the verification of certain conditions like when the user presses the shooting button or when the AI agent or enemy is at a certain distance of the main player.

Placing the camera in the Unity Game Engine

The animations follow a set of conditions and sequences in order to be activated, the picture below shows a schematic of the necessary order and trigger mechanisms associated to the animation transitions process during gameplay.

Making animation transitions in Unity

The animations follow an AI mechanism in order to provide more functionalities to MR systems as demonstrated in this prototype, it is possible to notice that the virtual elements can have more functionalities and interaction with the users. In the future will be possible that applications may have more uses from the real-world data captured by a sensor, the infrared (IR) cameras for example can provide night vision. The process of treating information has been improving and the AI strategies nowadays are able to treat huger amounts of data making predictions with a high rate of accuracy. The presented developed prototype illustrates a process of responding to the second research question by demonstrating a set of processes to achieve the proposed objective, for this demonstration was used the C# programming language and the Unity game engine among other technologies.

AI

Use of AI for Vital Signs detection and image uploader

Source code:

https://drive.google.com/file/d/1bX_EIUR3g040kvwnwaz7vEXKsxPAUYo3/view?usp=sharing

Body temperature:

Breath Rate:

Blood Pressure:

Oximetry:

Heart rate:

Concerning Primary Care activities and to improve the current online consultation process, this dissertation presents a prototype based on obtaining the vital signs during consultation for the physician and also access to the patient history aiming to facilitate the process of detecting possible diseases and triggering the emergency procedure based on the country where the user is placed without physical contact avoiding also the contagion of the Covid-19 disease. Currently it is possible to obtain the exact value of the vital signs: for blood pressure through the sphygmomanometer which also records the heart rate, concerning body temperature the actual values can be obtained with the thermometer; for the oxygen in the blood the correct values can be measured by an oximeter and heart rate can be obtained by different devices like the oximeters that also read this value, as do some sport watches and sphygmomanometers (blood pressure devices); Heart rate can also be assessed on arterial pulses (example given: on wrist and neck).

Figure 61. Data for a specific patient.

Whenever the program detects values above or below those recorded as a reference, an emergency procedure should be triggered and the user can communicate the problem to the attending physician or the professional who will assist him. The patient should have information about the signs and symptoms that require urgent/emergent medical intervention, and, if possible, help from third parties, even if the parameter measurements are in the range considered normal in the program. Some examples are some types of chest pain (thoracic pain), sudden miss of air, signs of rapidly worsening breathlessness, deviation of the labial commissure, lack of strength in one of the limbs, among others (some possible emergency situations are chest pain, feeling faint, dizziness, shortness of breath and lack of strength). The vital signs allow doctors to understand the health state of a patient and determine the severity level.

4.4 Body Temperature Body temperature refers to the production of heat in the body and the mechanisms for its regulation and maintenance, essential to maintain systemic homeostasis (stability that the body needs to perform its functions properly). The usage of OpenCV and Python allows to get the temperature values via web camera. The program requires the installation of OpenCV libraries. Two applications were developed to get the temperature via web camera, one in Python and the other in Matlab.

Figure 62. Obtaining body temperature via Python and Matlab. Adapted from the code [65]. 4.5 Blood Pressure Blood pressure or blood tension is the force that the blood exerts on the walls of the arteries during its circulation, and results in two measures: • Systolic blood pressure or “maximum” blood pressure: appears first and measures the force with which the heart contracts and “expels” the blood from its interior. • Diastolic blood pressure or “minimum” blood pressure: this is the second value and concerns the measurement of pressure when the heart relaxes between each beat. The reading of blood pressure is usually measured in millimeters of mercury (mmHg). When blood pressure is normal, it allows blood to be distributed throughout the body, reaching all organs. If the blood pressure is chronically high (hypertension) or when it increases suddenly, it has negative consequences for health, being responsible, for example, for cerebrovascular accidents, cardiac infarctions (death of heart cells), among other possibilities. For cases where blood pressure is too low (hypotension) blood flow on cells can decrease, compromising the nutrition and oxygenation of the cells, including brain cells. A mobile application was developed in java to obtain this value using the camera of the phone and by detecting the face of the user.

Figure 63. Application developed in Java for mobile and a Pulse Blood Pressure Monitor.

4.6 Cardiac Frequency The cardiac frequency rate is the number of times your heart beats per minute. Changes in heart rate may indicate the existence of cardiac and non-cardiac pathologies and may compromise the nutrition and oxygenation of the cells, including the brain cells; in order to measure the heartbeats during gameplay we have developed an application that measures the cardiac frequency in real time with the use of Python that is base in computer vision. Appearing on the monitor the reading obtained. The results from the webcam detection were compared and tested with the ones provided by medical devices and the clock sensor.

Figure 64. Heartbeat detection and comparison. By analyzing the python code, we can see that the values obtained by the green channel allowed to determine the heart frequency. The information provided will facilitate a clinical analysis. During the gameplay the application will track the face of the player first and afterwards will report the heartbeat in real time. The picture below is the heartbeat measurement on a web application with the use of OpenCV.

Figure 65. Different processes for heartbeat detection and usage of a serious game to get the state of rest to perform the analysis. Pulse measurement of the fingerprints is one of the several functionalities of AI Care being a bit related with the heartbeat detection project. It is calculating also the number of heart beatings but instead of measuring above chest, it is measuring the pulse from the fingerprint. In case the readings are too different from these two places, the patient most likely is facing a situation of cardiac arrhythmia and should see his doctor. The application should have an alarm system that triggers a contact to the correspondent medical institution according with the national emergency system of the country in case. The user has to place his finger on the phone camera for a few seconds and check if the light is on. The measurement process is different from the one used in the heartbeat via webcam. After getting the reading, the user can press a button on the mobile Java application that will send the information to the AI Care web application, that will add this value to his profile, store in the database and report on the system the readings obtained every time the clinical record is checked. The technologies used for the mobile application were Java, Android Studio and APIs and concerning the web application was used PHP, MySQL and JavaScript mainly. A website was developed to obtain the information obtained from the mobile phone using the open source PubNub API, allowing the verification of the data obtained by the mobile phone. The website shown in the figure above was made with HTML, JavaScript and CSS languages, using an API that allows the passage of information between the phone and the website. The developed mobile application in java reads the heartbeat via the phone and sends the information obtained to the website.

Figure 66. Detection of heartbeats via finger (light required for reading) and the web application. 4.7 Oximetry The Oximetry is a test that lets you know how much oxygen is being carried in the blood. The oxygen level measured with an oximeter is called the oxygen saturation level (SaO2). SaO2 is the percentage of oxygen that blood carries (in red blood cells) compared to its maximum carrying capacity on arteries. The lack of oxygen in the blood reduces its supply to the cells and may be inadequate for its needs, a condition that can cause serious cell damage and even lead to death. The SPO2 percentage was obtained with MatLab. To perform this test, it is required appropriate lightning and placing the thumb close to the web cam. The formula used to calculate the SpO2 can be based in the AC and DC of the blue and green channel [61]. (10)

Figure 67. Obtaining the percentage of oxygen on the blood arteries.

4.8 Respiratory Frequency The respiratory rate is the number of breaths per minute, that is, the number of times the combination of inspiration (entry of air into the lungs) and exhalation (exit of air from the lungs) occurs in one minute. The respiratory rate can be evaluated by counting only the number of inspirations per minute, by observing the chest expansion at each inspiration, that is, by counting how many times the chest goes up. The respiratory rate can be obtained by counting the number of breaths during one minute, for a more accurate result your body should be at rest as in example, it is better measured if you are in a sitting position or laid down in bed. The breath rate can be obtained by counting the number of pikes, maximum in the heart rate, every time the heart rate gets a maximum means you have inhaled, showing the number of breads per minute. On the plot it is possible to identify low frequency waveforms mixed with high frequency sawtooth (small spikes). Each small peak corresponds to a pulse beat (heartbeat). And the great peak-valley transition corresponds to the respiratory cycles.

Figure 68. Breath rate detection and comparison The reference values considered in the Program for respiratory rate are: 12 to 18 breaths per minute. The developed prototype allows in the final presented solution to obtain this value using the Infra-Red (IR) sensor which can be a web cam and the exact value of the number of breaths in one minute is stored on the MySQL database.

Figure 69. Obtaining the breath rate and adding to the clinical record of the patient. Adjusted from the code [66].

Detecting diseases: