Demos and Videos

Abstract— This paper implements and demonstrates visual route mapping and localisation upon a Pixel Processor Array (PPA). The PPA sensor comprises of an array of Processing Elements (PEs), each of which can capture and process visual information directly. This provides significant parallel processing power allowing novel ways in which information can be processed on-sensor. Our method predicts the correct node within a topological map generated from an image sequence by measuring image similarities, spatial coherence, and exploiting the parallel nature of the PPA. Our implementation runs at +300Hz on large public datasets with +2K locations requiring 2.5W at 500 GOPS/W. We compare vs traditionally implemented methods demonstrating better F-1 performance even on simulation. As far as we are aware, we present the first on-sensor mapping and localisation system running entirely on sensor.

paper available: https://www.researchgate.net/publication/350187131_Weighted_Node_Mapping_and_Localisation_on_a_Pixel_Processor_Array

Abstract: This talk will discuss recent advances in the development of visual architectures and their algorithms towards a step change in the direction of agile robotics. Current visual architectures often used in Robotic systems were not designed for vision nor action. They were designed for video recording or graphical processing. This hinders systems requiring low lag, low energy consumption and importantly force visual algorithms to process the world in ways that prevent effective coding for actions. In this talk I will describe work towards new architectures, specifically pixel processor arrays such as the SCAMP, that allow massive parallelism and focal-plane processing with reduced energy consumption. Examples include how these new architectures allow to perform visual computation onboard agile vehicles for tasks that involve visual odometry, recognition and deep network inference.

Our paper "Fully Embedding Fast Convolutional Networks on Pixel Processor Arrays" has been accepted in ECCV 2020.

We demonstrate for the first time complete inference of a CNN upon the focal plane of a sensor. The key idea behind our approach is storing network weights "in-pixel", allowing the parallel analogue computation of the SCAMP PPA to be fully utilized.

This current work considers a baseline task for digit classification at over 3000+ frames per second. We are moving into more sophisticated tasks and our approach is general enough to easily transfer to future Pixel Processor Array devices.

Arxiv version: https://arxiv.org/abs/2004.12525

BMVC 2020 www.bmvc2020-conference.com/conference/papers/paper_0126.html

Performance, storage, and power consumption are three major factors that restrict the use of machine learning algorithms on embedded systems. However, new hardware architectures designed with visual computation in mind may hold the key to solving these bottlenecks. This work makes use of a novel visual device: the pixel processor array (PPA), to embed a convolutional neural network (CNN) onto the focal plane. We present a new high-speed implementation of strided convolutions using binary weights for the CNN on PPA devices, allowing all multiplications to be replaced by more efficient addition/subtraction operations. Image convolutions, ReLU activation functions, max-pooling and a fully-connected layer are all performed directly on the PPA’s imaging plane, exploiting its massive parallel computing capabilities. We demonstrate CNN inference across 4 different applications, running between 2,000 and 17,500 fps with power consumption lower than 1.5W. These tasks include identifying 8 classes of plankton, hand gesture classification and digit recognition.

Video here

Talk on Towards Autonomous drone racing using the SCAMP pixel processor array. RSS Workshop on Perception and Control for Fast and Agile Super-Vehicles

July 12th, RSS 2020

Our paper "A Camera That CNNs: Towards Embedded Neural Networks on Pixel Processor Arrays " has been accepted as oral for ICCV 2019 and as a conference demo.

This work takes a first step embeddeding Neural Networks directly onto the focal plane of a sensor.

Arxiv version: https://arxiv.org/abs/1909.05647

This work presents an agile reactive navigation strategy for driving a non-holonomic ground vehicle around a preset course of gates in a cluttered environment using a low-cost processor array sensor. This enables machine vision tasks to be performed directly upon the sensor’s image plane, rather than using a separate general-purpose computer. We demonstrate a small ground vehicle running through or avoiding multiple gates at high speed using minimal computational resources. To achieve this, target tracking algorithms are developed for the Pixel Processing Array and captured images are then processed directly on the vision sensor acquiring target information for controlling the ground vehicle. The algorithm can run at up to 2000 fps outdoors and 200fps at indoor illumination levels. Conducting image processing at the sensor level avoids the bottleneck of image transfer encountered in conventional sensors. The real-time performance of on-board image processing and robustness is validated through experiments. Experimental results demonstrate that the algorithm’s ability to enable a ground vehicle to navigate at an average speed of 2.20 m/s for passing through multiple gates and 3.88 m/s for a ‘slalom’ task in an environment featuring significant visual clutter.