STMicroelectronics SensorTile Internet of Things Curriculum
Introduction
The SensorTile is a new Internet of Things (IoT) system provided by STMicroelectronics integrating state-of-the-art processor, wireless interfaces, and sensor systems. The SensorTile can form the foundation for wearable consumer devices, wearable medical devices, residential IoT systems and vehicle IoT systems.
The SensorTile system provides an exceptionally powerful and well-supported platform for introduction to IoT technology. The SensorTile is remarkably compact as shown below.
An important advance is now the development of EmbeddedML that enables the SensorTile system to directly learn to recognize events that are detected by the SensorTile sensor systems and the EmbeddedML Neural Network. This provides a SensorTile microsystem with machine learning capability operating entirely independently of other computing resources for both Neural Network training and execution.
The Tutorials below introduce SensorTile development methods, the SensorTile device and sensors, along with complete applications including a series of machine learning systems.
STMicroelectronics SensorTile Internet of Things Platform
The SensorTile system includes these components:
The SensorTile Processor System is an STM32L4 microprocessor based on the ARM Cortex M4 system. This provides introduction to the ARM processor architecture that is deployed on nearly every smartphone on earth.
The SensorTile Sensors includes:
The LSM6DSM combining state of the art MEMS microaccelerometer and microgyroscope.
The LSM303AGR combining microaccelerometer and magnetometer for compass heading
The LPS22HB barometric pressure sensor for determination of altitude and atmospheric pressure.
The MP34DT04 microphone
The SensorTile also includes a Bluetooth Low Energy (Bluetooth Smart) wireless interface the BlueNRG-MS system.
The SensorTile also includes non-volatile flash storage that stores the executable code that enables IoT system operation.
The SensorTile also includes a cradle accessory with additional features including:
SD Card Flash Storage System
STC3115 Battery Monitor providing detailed energy monitoring for the SensorTile
HTS221 Humidity and Temperature environmental sensors
This Web site includes Tutorial and Reference Designs intended to introduce the development environment for the SensorTile system and provide experience in novel application development.
Development environments are essential to development of software for IoT systems and other products. These provide support to developers for both creation of systems, testing, debugging, and installation of software systems on platforms. This development environment is referred to as an Integrated Development Environment (IDE). This includes all of the software tools required to create a software distribution for the SensorTile, compile this software system into the processor instruction set using a Build capability, execute this system using a Debug capability, and also create an “image” file that can be installed in the SensorTile non-volatile storage.
Tutorials
1) Tutorial 1: Introduction to STMicroelectronics Development Environment and DataLog Project Example.
There are two versions of this tutorial. One for Apple Mac Platforms, and one for Microsoft Windows Platforms.
This Tutorial provides an introduction to the SensorTile platform and its development tools.
The Tutorial steps include:
a) Installing an Integrated Development Environment (IDE) on a personal computer.
b) Obtaining reference design example project software. This will specifically include a sensor Data Logging system.
c) Usage of the IDE to Import, Build, Run, Debug and Flash the SensorTile board to run the example Data Logging project.
2) Tutorial 2: Sensor System Signal Acquisition, Event Detection and Configuration
This Tutorial provides experience in the development of applications that acquire IoT system sensor signals, detect events in the sensor system data stream, and also configure sensor systems.
The Tutorial steps provide:
a) An introduction to control of sensor signal acquisition and sensor system configuration. These topics are fundamental to IoT system development.
b) An introduction to sensor signal detection and notifications.
c) An experience in software system development for the SensorTile IoT system demonstrating important capabilities of the System WorkBench Integrated Development Environment in accelerating system development.
3) Tutorial 3: Accelerometer Sensor Systems and Orientation and Event Detection
This Tutorial provides additional experience in sensor system data processing including recognition of gesture motion. Experience from this Tutorial provides guidance for the development of capable IoT systems that recognize and may even guide specific motion.
The Tutorial steps provide:
a) An introduction to accelerometer sensor systems and methods for detection of sensor orientation by exploiting gravitational acceleration signals.
b) Measurement of orientation by the polar coordinate system.
c) An introduction to gesture recognition through the use of sensor information and state machine systems for characterizing specific behavior.
4) Tutorial 4: Introduction to Audio Sampling and Signal Processing
This Tutorial provides experience in sensing and generation of audio signals using the state-of-the-art microphone on the SensorTile system. This tutorial also introduces Digital Signal Processing with methods to both hear and visualize the capabilities of DSP systems.
The Tutorial steps provide
a) An introduction to audio sensor signal acquisition systems.
b) An introduction to audio signal generation with the opportunity to hear and see audio signal waveforms.
c) An introduction to DSP systems with several demonstrations that provide immediate and direct experience valuable for next steps in system development.
5) Tutorial 5: SensorTile Firmware Programming
This Tutorial provides valuable experience in the creation of firmware applications using the state-of-the-art SensorTile system and tools.
The Tutorial steps provide:
a) An introduction to integration of source code and header files into new SensorTile projects.
b) Experience in configuring the firmware build system.
c) Complete development experience valuable for next Tutorials that integrate multiple system resources and associated capabilities.
6) Tutorial 6: Introduction to Bluetooth Low Energy Wireless Interfaces
This Tutorial provides an introduction to the important principles of Bluetooth Wireless Interfaces. This includes also valuable experience in data transport between the SensorTile system and an embedded Linux platform - the BeagleBone.
a) This Tutorial will prepare students for many next steps in wireless IoT system development that will be appearing in the following Tutorials and Reference Designs.
b) This will include development of new firmware providing access to multiple sensors on the SensorTile platform.
7) Tutorial 7: Introduction to Bluetooth Low Energy Communication and the GATT Profile
This Tutorial introduced Bluetooth Wireless Communication principles and methods. This includes hands-on experience with communication of SensorTile sensor data to the BeagleBone platform.
a) This Tutorial provides experience in the communication protocol central to global IoT applications: The most important GATT profile and associated communication protocols. The BlueZ Bluetooth system interface, hosted on Linux platforms, is also included.
b) This Tutorial also included a demonstration of bidirectional communication with control of events on the SensorTile by an application on the BeagleBone.
8) Tutorial 8: Introduction to Motion Data Acquisition via Bluetooth Low Energy Communication
This Tutorial introduces new capabilities that are provided by new firmware enabling direct access to multiaxis motion sensor data.
a) This includes the installation of the FP-SNS-ALLMEMS1 firmware application provided by STMicroelectronics.
b) Through use of gatttool, this Tutorial provides the BeagleBone system with a high performance, wireless SensorTile motion sensor.
c) This addresses an important need for compact and wearable motion sensing systems for IoT systems.
9A) Tutorial 9A: Introduction to Inertial Displacement Sensing with SensorTile Accelerometer
This Tutorial provides an introduction to Inertial Sensing through a demonstration of displacement sensing applying the SensorTile accelerometer.
This includes an extension of the Datalog application with increased sampling rate and addition of offset rejection, noise rejection, and drift-reduction filters.
This also includes the SensorTile Animation System that permits visualization of SensorTile motion sensor data in real time.
This Tutorial provides users with the background required to develop new motion classification systems based on displacement of the SensorTile.
9B) Tutorial 9B: Real Time Rotation Angle Sensing with SensorTile
This Tutorial provides an introduction to Inertial Sensing through a demonstration of rotation angle sensing applying the SensorTile rate gyroscope.
This includes an extension of the Datalog application with increased sampling rate and addition of offset rejection, noise rejection, and drift-reduction filters.
This also includes the SensorTile Animation System that permits visualization of SensorTile motion sensor data in real time.
This Tutorial provides users with the background required to develop new motion classification systems based on rotation of the SensorTile.
10) Tutorial 10: Introduction to EmbeddedML Machine Learning
This Tutorial provides an introduction to a new breakthrough in Machine Learning technology. This introduces the EmbeddedML system (developed by Charles Zaloom of UCLA). This new system is a complete Neural Network system with automated training and execution hosted on the SensorTile microprocessor and integrated with SensorTile sensor inputs.
This is based on the familiar Datalog application and includes the complete EmbeddedML capability.
This illustrates Neural Network implementation and execution with a model system based on a logic gate structure.
This Tutorial prepares users to develop the next set of novel systems that integrate SensorTile sensor inputs for recognition of real-world events.
11) Tutorial 11: IoT Machine Learning for Recognition of Motion Events
This Tutorial provides the first introduction to a fully integrated system that acquires sensor data, learns to recognize events, and then executes on new sensor data to predict the occurrence of motion events.
Critical experience is provided in the development of methods for designing and computing the characteristic “features” that can be applied to train a Neural Network Machine Learning System to recognize events.
This system will takes advantage of the insights and experience from Tutorial 3 for sensing.
This also applies the concept of state machines from Tutorial 3 for development of sensor data acquisition, learning, and execution.
12) Tutorial 12: IoT Machine Learning for Recognition of Multiple Step Motion Patterns
This Tutorial extends the concepts of Tutorial 11 to develop systems that recognize not only motions, but motion patterns consisting of two orientation changes of the SensorTile.
This system provides experience and insight into systems that can grow to expand and enable learning and event recognition with data sources provided by multiple sensors with signals resolved in time and frequency characteristics.
13) Tutorial 13: IoT Machine Learning of Rotation Angle Motion Patterns by Gyroscope Sensing
This Tutorial introduces SensorTile self-learning systems based on motion sensing. This introduces the SensorTile microgyroscope sensor.
The SensorTile microgyroscope sensor measures rotation rate. Thus, this system introduces a method to compute angular orientation by the time integral of rotation rate.
14) Tutorial 14: Motion Pattern Recognition with Inertial Sensing and Machine Learning
This Tutorial introduces SensorTile self-learning systems based on inertial sensing for the recognition of motion patterns. This introduces the SensorTile inertial sensing and computation of displacement..
The SensorTile machine learning system provides capability for classification of motion motions in sequence, each forming a motion pattern.
15)Tutorial 15: Angle Resolved Motion Pattern Recognition with Inertial Sensing and Machine Learning
This Tutorial introduces SensorTile self-learning systems based on inertial sensing for the recognition of motion displacement angle for planar motions.
The SensorTile machine learning system provides capability for classification of motion motions according to a resolved angle.
16)Tutorial 16: Motion and Motion Speed Recognition by Pattern Recognition by Machine Learning
This Tutorial introduces SensorTile self-learning systems based on a combination of sensor data acquisition, event detection, and event timing measurement.
The SensorTile machine learning system provides capability for classification of motion motions according to angular orientation change combined with motion period time.
16)Tutorial 17: Motion Pattern Recognition with Signal Correlation
This Tutorial introduces SensorTile self-learning systems based on a combination of sensor data acquisition and signal cross correlation methods.
The SensorTile machine learning system provides capability for classification of complex motion patterns according to angular orientation change as measured by the SensorTile rate gyroscope.
Reference Designs
The SensorTile Reference Designs provide complete, end-to-end experience in development of a system. This experience prepares developers for innovation and implementation of new systems.
This Reference Design includes a SensorTile system that integrates development experience from the previous Tutorials. These Tutorials should all be completed prior to starting on this Reference Design. This system combines capability for motion detection with audio signal processing. It produces a Sensor Tile system that detects the users motion and orientation of the SensorTile to control audio signal processing.
Frequently Asked Question List
A Frequently Asked Questions List for the SensorTile System is constantly being expanded. As questions arrive from the SensorTile Forum, these will be added here.
Tutorials in Microsoft Word Format
The Tutorials listed above may be downloaded as PDF files at the links above.
The complete set of Tutorials are also available for download as Word source documents at this link.
SensorTile IoT Reference Design
The students and faculty at the Universidad De Guadalajara have developed a novel Pedometer based on the SensorTile system. This new system applies state-of-the-art signal processing methods and provides remarkable accuracy in an IoT platform. This new pedometer is provided with a compact LCD display as well as a wearable package design. The complete Pedometer project system, electronic and mechanical design along with source code is described here:
Implementation of a pedometer with the SensorTile
by
Luis Garcia Magallon, Miguel Angel Ramirez Hernandez, Juan Carlos Villa Negrete
and
Profesor Dr. Jose Miguel Moran Loza
Student Project Reference Designs
Many UCLA students have collaborated in the development of Internet of Things (IoT) SensorTile systems in course projects. The senior capstone design course has included teams who have developed novel systems for motion classification with SensorTile data sources and machine learning methods.
This has included systems with single and dual SensorTile devices along with signal processing, signal feature extraction, neural network design, neural network training, and finally in-field performance analysis.
Reference Designs have been developed by these student teams that include design and development documentation as well as source code for your use.
These are available for evaluation and guidance with the objective that these may inspire other future development,
If you have questions regarding the Reference Designs, please contact kaiser@ee.ucla.edu.
1) STMicroelectronics SensorTile Reference Design: Basketball Freethrow Classifier by Machine Learning
This Reference Design developed by Alexander Graening and James Xu describes the comprehensive development of a system capable of detecting characteristics of the Basketball Freethrow motion. This relies on the complete set of signal acquisition from SensorTIle systems, signal processing, and machine learning. The SensorTile systems are used in pairs with attachment at upper and lower arm.
2) STMicroelectronics SensorTile Reference Design: Basketball Hookshot Classifier by Machine Learning
This Reference Design developed by Ziyue Yang and Zhitong Qian describes the comprehensive development of a system capable of detecting the characteristic motion of arm and hand associated with optimal and suboptimal Basketball Hookshot motion. The complete description of SensorTIle systems, signal processing, and machine learning development is included The SensorTile systems are used in pairs with attachment at upper and lower arm.
3) STMicroelectronics SensorTile Reference Design: Tennis Motion Classifier by Machine Learning
This Reference Design developed by Bonnie Lam and Gheorge Schreiber describes the another comprehensive development of a system capable of detecting Tennis motion swing types and swing quality. This also relies on the complete set of signal acquisition from SensorTIle systems, signal processing, and machine learning. Here the SensorTile is attached to the tennis racket.
This Reference Design developed by Guang Liew and Zhijie Yao describes another development based on dual SensorTile motion sensors each providing data sources for feature extraction. This system classifies proper and improper resistance training motions by SensorTile data sources and machine learning classifier systems.
This Reference Design developed by Loic Maxwell and Craig Young describes the development of a unique system that classifies the complex motions occurring in climbing. This classifies proper and improper motion applying dual SensorTile systems and a series of signal processing, feature extraction, and machine learning solutions.
This Reference Design developed by Michael Qi and July Zamora also applies dual SensorTile data sources to the important problem of classifying proper and improper motion required in shoulder rehabilitation. This also applies dual SensorTile devices along with end-to-end development from signal acquisition to machine learning system design and performance analysis.