Advanced Applied Deep Learning

Practice Course

Sheng Yun Wu

Week 14: Object Detection and Tracking in Autonomous Systems

In Week 14, students will focus on the application of object detection and tracking in autonomous systems, such as autonomous vehicles, drones, and robotics. This week explores real-time perception, combining object detection and tracking with path planning and obstacle avoidance, and leveraging sensor fusion techniques (e.g., LIDAR + cameras) to enhance detection and tracking accuracy.

Example 1: Introduction to Object Detection and Tracking in Autonomous Systems

Description:
This example introduces how object detection and tracking are critical for autonomous systems, such as autonomous vehicles and drones, and discusses the integration of perception with decision-making and control systems.

No Code for this example – Theoretical Explanation

Key tasks: Pedestrian detection, vehicle detection, object tracking, path planning, obstacle avoidance.
Sensors: Cameras, LIDAR, RADAR, GPS.
Challenges: Real-time perception, sensor fusion, low latency, handling occlusions, and dynamic environments.

Example 2: Object Detection for Autonomous Vehicles (YOLO + OpenCV)

Description:
This example demonstrates how to use YOLO for object detection in autonomous vehicles, detecting pedestrians, vehicles, and obstacles in real time.

import cv2

import torch

# Load YOLOv5 model for vehicle and pedestrian detection

model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True)

# Open a video feed (autonomous vehicle camera feed simulation)

cap = cv2.VideoCapture('autonomous_vehicle_footage.mp4')

while cap.isOpened():

ret, frame = cap.read()

if not ret:

break

# Perform object detection

results = model(frame)

# Display detection results

cv2.imshow('Vehicle & Pedestrian Detection', results.render()[0])

if cv2.waitKey(1) & 0xFF == ord('q'):

break

cap.release()

cv2.destroyAllWindows()

Example 3: Real-Time Object Tracking for Autonomous Drones (KCF Tracker + YOLO)

Description:
This example shows how to implement object tracking in autonomous drones using a combination of YOLO for detection and KCF for tracking to follow moving targets like vehicles or people.

import cv2

import torch

# Load YOLOv5 model for object detection

model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True)

# Initialize KCF tracker

tracker = cv2.TrackerKCF_create()

# Open a video feed (drone camera feed simulation)

cap = cv2.VideoCapture('drone_footage.mp4')

# Initialize tracking after first detection

ret, frame = cap.read()

results = model(frame)

bbox = results.xyxy[0][0][:4].cpu().numpy()

bbox = tuple(map(int, bbox))

tracker.init(frame, bbox)

while cap.isOpened():

ret, frame = cap.read()

if not ret:

break

# Perform tracking

success, bbox = tracker.update(frame)

if success:

x, y, w, h = [int(v) for v in bbox]

cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2)

# Display the frame with tracking

cv2.imshow('Drone Object Tracking', frame)

if cv2.waitKey(1) & 0xFF == ord('q'):

break

cap.release()

cv2.destroyAllWindows()

Example 4: Sensor Fusion for Object Detection (LIDAR + Camera Integration)

Description:
This example explains how to integrate LIDAR and camera data to enhance object detection, combining depth information from LIDAR with image data for better object localization.

No Code for this example – Theoretical Explanation

LIDAR provides depth information that helps measure distances to objects, complementing camera data.
Fusion techniques: Combine LIDAR point clouds with 2D camera data to improve detection accuracy.
Commonly used in autonomous vehicles for obstacle detection and collision avoidance.

Example 5: Object Detection with LIDAR Data Using Open3D

Description:
This example demonstrates how to use Open3D to process LIDAR point clouds for object detection in autonomous systems.

import open3d as o3d

# Load LIDAR point cloud data

pcd = o3d.io.read_point_cloud('lidar_point_cloud.pcd')

# Visualize the LIDAR point cloud

o3d.visualization.draw_geometries([pcd])

# Additional code for object detection using point cloud clustering or deep learning methods

Example 6: Object Tracking with LIDAR Data (Kalman Filter)

Description:
This example demonstrates how to apply a Kalman Filter to track objects using LIDAR data, estimating the motion of objects in 3D space.

import numpy as np

# Kalman Filter for tracking 3D positions (LIDAR data)

class KalmanFilter3D:

def __init__(self):

self.dt = 1.0

self.A = np.eye(6) # State transition matrix for 3D (x, y, z, vx, vy, vz)

self.H = np.eye(3, 6) # Measurement matrix (we observe x, y, z)

self.Q = np.eye(6) # Process noise covariance

self.R = np.eye(3) # Measurement noise covariance

self.P = np.eye(6) # Initial estimation covariance

self.x = np.zeros((6, 1)) # Initial state (x, y, z, vx, vy, vz)

def predict(self):

self.x = np.dot(self.A, self.x)

self.P = np.dot(np.dot(self.A, self.P), self.A.T) + self.Q

return self.x

def update(self, z):

y = z - np.dot(self.H, self.x)

S = np.dot(np.dot(self.H, self.P), self.H.T) + self.R

K = np.dot(np.dot(self.P, self.H.T), np.linalg.inv(S))

self.x = self.x + np.dot(K, y)

self.P = self.P - np.dot(np.dot(K, self.H), self.P)

# Example LIDAR measurement (x, y, z)

lidar_measurement = np.array([[10], [5], [2]])

# Initialize tracker and update with LIDAR measurement

tracker = KalmanFilter3D()

tracker.predict()

tracker.update(lidar_measurement)

print(f"Predicted state: {tracker.x}")

Example 7: Object Detection and Collision Avoidance in Drones (YOLO + Obstacle Avoidance)

Description:
This example combines object detection with obstacle avoidance for autonomous drones, using YOLO for object detection and basic path planning to avoid collisions.

import cv2

import torch

# Load YOLOv5 model

model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True)

# Simulated drone camera feed

cap = cv2.VideoCapture('drone_footage.mp4')

while cap.isOpened():

ret, frame = cap.read()

if not ret:

break

# Perform object detection

results = model(frame)

detections = results.xyxy[0]

# Basic obstacle avoidance logic

for *box, conf, cls in detections:

if cls == 0: # Assuming class 0 is an obstacle

# Logic for avoiding the obstacle

print(f"Obstacle detected at {box}, adjusting flight path")

# Display results

cv2.imshow('Drone Object Detection & Collision Avoidance', results.render()[0])

if cv2.waitKey(1) & 0xFF == ord('q'):

break

cap.release()

cv2.destroyAllWindows()

Example 8: Lane Detection in Autonomous Vehicles (Hough Transform + OpenCV)

Description:
This example demonstrates how to implement lane detection in autonomous vehicles using the Hough Transform and OpenCV to detect road lanes for path planning.

import cv2

import numpy as np

def detect_lanes(image):

gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

edges = cv2.Canny(gray, 50, 150)

# Define region of interest (focus on the lower half of the image)

height, width = image.shape[:2]

mask = np.zeros_like(edges)

polygon = np.array([[(0, height), (width, height), (width, height // 2), (0, height // 2)]])

cv2.fillPoly(mask, polygon, 255)

cropped_edges = cv2.bitwise_and(edges, mask)

# Apply Hough Transform to detect lines

lines = cv2.HoughLinesP(cropped_edges, 1, np.pi / 180, 100, minLineLength=100, maxLineGap=50)

# Draw detected lanes on the image

if lines is not None:

for line in lines:

x1, y1, x2, y2 = line[0]

cv2.line(image, (x1, y1), (x2, y2), (0, 255, 0), 2)

return image

# Load a frame from an autonomous vehicle's camera

frame = cv2.imread('road_image.jpg')

# Detect lanes and display the result

lane_image = detect_lanes(frame)

cv2.imshow('Lane Detection', lane_image)

cv2.waitKey(0)

cv2.destroyAllWindows()

Example 9: Object Detection for Pedestrian Safety in Autonomous Vehicles (YOLOv5)

Description:
This example implements pedestrian detection in autonomous vehicles to enhance pedestrian safety, using YOLOv5 to detect pedestrians and trigger alerts for the vehicle control system.

import cv2

import torch

# Load YOLOv5 model for pedestrian detection

model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True)

# Simulated vehicle camera feed

cap = cv2.VideoCapture('pedestrian_crossing.mp4')

while cap.isOpened():

ret, frame = cap.read()

if not ret:

break

# Perform object detection

results = model(frame)

# Check if pedestrians are detected

for *box, conf, cls in results.xyxy[0]:

if cls == 0: # Assuming class 0 is a pedestrian

print("Pedestrian detected, applying brakes!")

# Display results

cv2.imshow('Pedestrian Detection', results.render()[0])

if cv2.waitKey(1) & 0xFF == ord('q'):

break

cap.release()

cv2.destroyAllWindows()

Example 10: Integrating Object Detection with Path Planning (A Algorithm)*

Description:
This example integrates object detection with path planning for autonomous vehicles, where detected objects (obstacles) are used to adjust the vehicle’s path using the A* algorithm.

import heapq

def a_star(start, goal, grid):

# A* algorithm for path planning

def heuristic(a, b):

return abs(a[0] - b[0]) + abs(a[1] - b[1])

open_list = []

heapq.heappush(open_list, (0, start))

came_from = {}

cost_so_far = {start: 0}

while open_list:

_, current = heapq.heappop(open_list)

if current == goal:

break

for next_node in neighbors(current, grid):

new_cost = cost_so_far[current] + 1

if next_node not in cost_so_far or new_cost < cost_so_far[next_node]:

cost_so_far[next_node] = new_cost

priority = new_cost + heuristic(goal, next_node)

heapq.heappush(open_list, (priority, next_node))

came_from[next_node] = current

return reconstruct_path(came_from, start, goal)

# Grid, start, and goal definitions for path planning

grid = [[0, 0, 0, 0],

[1, 1, 0, 1],

[0, 0, 0, 0],

[0, 1, 0, 0]]

start = (0, 0)

goal = (3, 3)

# Perform path planning considering detected objects as obstacles

path = a_star(start, goal, grid)

print(f"Planned path: {path}")

Week 14 Summary

Objective: Apply object detection and tracking to autonomous systems, combining perception with real-time decision-making and control.
Skills Developed:
- Implement object detection and tracking for autonomous vehicles, drones, and robotics.
- Integrate LIDAR data for improved object detection and tracking.
- Use path planning algorithms (A*) to dynamically adjust routes in response to detected objects.
- Implement obstacle detection and avoidance systems in autonomous drones.
- Apply lane detection and pedestrian safety features for autonomous driving.
Tools: OpenCV, PyTorch, YOLOv5, Kalman Filters, Open3D.

By the end of Week 14, students will be able to integrate object detection and tracking into autonomous systems, enabling real-time perception, path planning, and obstacle avoidance for autonomous vehicles, drones, and robots.