Advanced Applied Deep Learning
Practice Course
Sheng Yun Wu
Practice Course
Sheng Yun Wu
In Week 12, students will focus on building and deploying real-time object detection systems. This includes optimizing models for real-time inference, integrating object detection models into applications using frameworks such as OpenCV, TensorFlow Lite, and OpenVINO, and deploying models on edge devices like mobile phones and IoT devices. The emphasis is on making object detection systems fast, efficient, and deployable in real-world applications.
Description:
This example introduces the core concepts of real-time object detection, including the challenges and trade-offs between accuracy and speed, particularly in low-latency environments such as video streams.
No Code for this example – Theoretical Explanation
Challenges: Maintaining accuracy while reducing inference time, hardware constraints, model size optimization.
Applications: Surveillance, autonomous driving, industrial automation, and AR/VR.
Description:
This example demonstrates how to implement real-time object detection using YOLOv5 with OpenCV, processing frames from a video or webcam feed in real time.
import cv2
import torch
# Load pre-trained YOLOv5 model
model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True)
# Open webcam feed
cap = cv2.VideoCapture(0)
while cap.isOpened():
ret, frame = cap.read()
if not ret:
break
# Perform object detection
results = model(frame)
# Display the results
cv2.imshow('YOLOv5 Real-Time Detection', results.render()[0])
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
Description:
This example shows how to convert a TensorFlow object detection model to TensorFlow Lite format and run it on a mobile device for real-time object detection.
import tensorflow as tf
# Convert a TensorFlow model to TensorFlow Lite format
converter = tf.lite.TFLiteConverter.from_saved_model('saved_model_directory')
tflite_model = converter.convert()
# Save the TensorFlow Lite model
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
# Run inference on the TensorFlow Lite model (on a mobile device)
import tflite_runtime.interpreter as tflite
interpreter = tflite.Interpreter(model_path='model.tflite')
interpreter.allocate_tensors()
# Additional code for loading input data and running the interpreter on a mobile device
Description:
In this example, students learn how to deploy a trained object detection model using Intel’s OpenVINO toolkit, optimizing the model for edge devices like Intel Movidius or Raspberry Pi.
# Convert TensorFlow or PyTorch model to OpenVINO format
!mo.py --input_model saved_model_directory/model.pth --framework pytorch --output_dir openvino_model/
# Load and run the model on an edge device
from openvino.inference_engine import IECore
ie = IECore()
model = ie.read_network(model='openvino_model/model.xml')
exec_net = ie.load_network(network=model, device_name='CPU')
# Perform inference on a new image
# Additional code for loading and processing input data
Description:
This example focuses on optimizing object detection models for real-time inference by adjusting batch sizes, quantization, and pruning techniques.
import tensorflow as tf
# Optimize the model using quantization (TensorFlow Lite)
converter = tf.lite.TFLiteConverter.from_saved_model('saved_model_directory')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
quantized_model = converter.convert()
# Save the quantized model
with open('quantized_model.tflite', 'wb') as f:
f.write(quantized_model)
# Run the quantized model on edge devices
# Additional code for running the model
Description:
This example shows how to use an SSD (Single Shot Multibox Detector) model optimized for real-time detection on mobile devices using TensorFlow Lite.
# Load pre-trained SSD model for real-time detection
interpreter = tflite.Interpreter(model_path='ssd_mobilenet_v2.tflite')
interpreter.allocate_tensors()
# Real-time detection logic for mobile devices
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
# Additional code for capturing video frames and performing detection
Description:
This example demonstrates how to run real-time object detection on a Raspberry Pi using the MobileNet-SSD model with OpenCV.
import cv2
import numpy as np
# Load pre-trained MobileNet-SSD model
net = cv2.dnn.readNetFromCaffe('MobileNetSSD_deploy.prototxt', 'MobileNetSSD_deploy.caffemodel')
# Open webcam feed
cap = cv2.VideoCapture(0)
while True:
ret, frame = cap.read()
if not ret:
break
# Prepare frame for MobileNet-SSD
blob = cv2.dnn.blobFromImage(frame, 0.007843, (300, 300), 127.5)
net.setInput(blob)
detections = net.forward()
# Process detections and draw bounding boxes
for i in range(detections.shape[2]):
confidence = detections[0, 0, i, 2]
if confidence > 0.5:
idx = int(detections[0, 0, i, 1])
box = detections[0, 0, i, 3:7] * np.array([frame.shape[1], frame.shape[0], frame.shape[1], frame.shape[0]])
(startX, startY, endX, endY) = box.astype("int")
cv2.rectangle(frame, (startX, startY), (endX, endY), (0, 255, 0), 2)
# Display the frame
cv2.imshow('Real-Time Object Detection', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
Description:
This example demonstrates how to deploy YOLOv4 on edge devices using the Darknet framework and optimize the model for low-power environments.
# Compile Darknet for edge devices
!make
# Run YOLOv4 for real-time detection on a webcam or video feed
!./darknet detector demo cfg/coco.data cfg/yolov4.cfg yolov4.weights -c 0
Description:
This example combines object detection with object tracking using YOLO and OpenCV for continuous tracking of objects across video frames.
import cv2
import torch
# Load YOLOv5 model
model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True)
# Open webcam feed
cap = cv2.VideoCapture(0)
tracker = cv2.TrackerKCF_create() # Initialize object tracker
while cap.isOpened():
ret, frame = cap.read()
if not ret:
break
# Perform object detection
results = model(frame)
# Initialize tracker on first detection
if results.xyxy[0].shape[0] > 0:
bbox = results.xyxy[0][0].numpy()
tracker.init(frame, tuple(bbox[:4]))
# Update tracker on subsequent frames
success, box = tracker.update(frame)
if success:
(x, y, w, h) = [int(v) for v in box]
cv2.rectangle(frame, (x, y), (x + w, y + h), (255, 0, 0), 2)
# Display the result
cv2.imshow('Real-Time Object Detection & Tracking', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
Description:
This example demonstrates how to deploy a real-time object detection model on AWS Lambda using Amazon SageMaker to run inference on the cloud.
# Deploy a pre-trained YOLO model to SageMaker
import sagemaker
from sagemaker.pytorch import PyTorchModel
# Define model in S3
model = PyTorchModel(model_data='s3://path/to/yolo/model.tar.gz', role='SageMakerRole', entry_point='inference.py')
# Deploy the model to an endpoint
predictor = model.deploy(instance_type='ml.m5.large', initial_instance_count=1)
# Invoke the endpoint for real-time object detection
response = predictor.predict({'image_url': 's3://path/to/image.jpg'})
Objective: Build and deploy real-time object detection systems optimized for speed and efficiency.
Skills Developed:
Implement real-time object detection using frameworks like OpenCV and YOLO.
Deploy models on mobile devices and edge hardware using TensorFlow Lite and OpenVINO.
Optimize models using quantization and pruning techniques for efficient inference.
Integrate object tracking into real-time object detection systems.
Deploy object detection models on cloud services like AWS Lambda and SageMaker.
Tools: TensorFlow Lite, OpenVINO, PyTorch, YOLOv5, Darknet, OpenCV, SageMaker.
By the end of Week 12, students will be capable of deploying real-time object detection models in various environments, including mobile devices, edge hardware, and cloud services, preparing them for practical, real-world applications.