Intelligent Vision Sensor with Three-Layer Neural Network for Object Detection

This sensor is an all-digital synchronous event-based vision sensor (EVS) with adaptive power-saving pixels and in-sensor object-of-interest (OOI) extraction. The 200×128 single- photon avalanche diode (SPAD) array employs three scene- adaptive power saving (PS) schemes saturation-based (SAT-PS), temporal (T-PS), and spatial (S-PS) to suppress power consumption under saturation, short-window abrupt changes, and spatial redundancy. A three-layer hybrid spiking/binary neural network generates 8×8 tile-wise OOI signals to gate informative-event-focused readout. Fabricated in 90 nm, the prototype consumes 9.85 mW at 200 lux, achieving 47.6% pixel- and 21.6% core-power savings, with up to 53.77% pixel saving and 51.4% total saving in composite scenes. The measured throughput is 5.58 kfps and 714 Meps (DAQ- limited). Due to IR-drop induced margin loss, real time OOI detection was validated on an FPGA (Dice score= 0.7452).

480x320 CMOS LiDAR Sensor with Tapering 1-Step Histogramming TDCs and Sub-Pixel Echo Resolvers

This chip is a 480×320 CMOS LiDAR sensor using tapered 1-step hTDCs and SPER for memory-efficient operation. The hTDC accumulates histograms with 1ns resolution over the full range and applies 500ps resolution only at the echoes, while SPER maps echoes to individual SPADs, enabling 4× image resolution. Implemented with 2×2 SPAD macro-pixels and shared AFEs (2.5T/SPAD) in a 90nm BSI CIS process, the sensor achieves 3.8cm precision at 100m and 4.1cm accuracy under 110klx sunlight at 25fps.

Multi-Range, Multi-Resolution LiDAR Sensor with 2,880-Channel Modular Survival hTDC and Delay Compensation Using Double Histogram Sampling

This chip a 720×192 dToF sensor with a 2,880-channel modular survival histogram TDC (shTDC) array. The shTDC supports non-destructive accumulation while concentrating depth on active bins, reducing memory by 75% with precision comparable to one-step hTDC. Modular hTDCs enable multi-range and multi-resolution imaging to 69.6m, while double histogram sampling cancels pixel-to-shTDC offsets. The prototype achieves 0.13% precision and 0.12% accuracy.

6.25-mW Retina-mimetic CMOS Image Sensor with SPAD-PPD Hybrid Pixel Array for HDR Imaging

This is the first image sensor with a SPAD-PPD hybrid pixels, mimicking the human retina, which features hybrid photoreceptor cells: rods and cones. The SPAD provides high- sensitivity photon-counting under low-light condition. The PPD serves as a linear analog counter by receiving avalanched electrons from the SPAD and transferring unit charges to the floating diffusion (FD) node. Additionally, the SPAD supports non-saturated counting under high-light conditions. Avalanched electrons in a short exposure time are directly transferred to the FD node for high-speed digital counting. The PPD captures images in mid-light conditions similarly to conventional 4-transistor pixels. As a result, images from hybrid pixels are synthesized to achieve a high dynamic range over 109 dB. The pixel has only three nMOS transistors shared between the SPAD and the PPD. The PPD-based analog counting and the 0.9-V time-domain ADCs provide lower power consumption compared to per-pixel digital counting in SPAD pixels, resulting in a total sensor power of 6.25 mW.

400x112 High-Resolution Solid-State LiDAR Sensor

An area-efficient, high-resolution CMOS light detec- tion and ranging (LiDAR) capable of generating 400 × 112 depth images is proposed. This sensor employs a rolling-scan architecture that is compatible with a row-addressable solid- state LiDAR system. To achieve high resolution, a single-photon avalanche diode (SPAD) analog front-end (AFE) circuit was implemented using only five nMOS transistors. A reconfigurable- resolution (RR) hTDC, capable of converting one-/two-channel signals with variable time resolutions of 1, 2, or 4 ns, is further proposed to implement a 400-channel histogramming time-to- digital converter (hTDC) with low memory overhead. The sensor leverages RR-hTDC to support two image resolution modes: macro and quadruple resolution (QR). In the macro mode, 200×56 depth images can be acquired with a 1-ns time resolution. In the QR mode, the sensor can achieve 400×112 high-resolution

All-Digital Event-Based Vision Sensor

This chip is an all-digital event-based vision sensor (EVS) using single-photon avalanche diode (SPAD) pixels. The proposed sensor introduces a multi-event generation method that enables the detection of light intensity changes across multiple levels, facilitating advanced event-image processing techniques. The three core processing techniques implemented are event-driven threshold control (EDTC), vibration event noise (VEN) filtering, and adaptive frame-rate control (AFRC). Using the multi-event-based processing, the sensor can dynamically adjust the event-generation threshold (TH), optimize the frame rate, and filter out noise caused by vibrations, thereby enhancing the accuracy of object detection in various challenging environ- ments. EDTC ensures a consistent event rate regardless of the scene’s contrast or lighting conditions, reducing the overall event rate by 55.66%. VEN filtering successfully eliminates 67.25% of the noise generated by vibrations when the sensor module is exposed to constant vibrations, and AFRC optimizes the frame rate based on the relative speed of moving objects, minimizing motion blur and the loss of event data.

320x240 CMOS LiDAR Sensor with 6-Transistor nMOS-Only SPAD Analog Front-End and Area-Efficient Priority Histogram Memory

This chip is a 320x240 direct time-of-flight (dTOF) sensor designed to be compatible with fully solid-state LiDAR systems. The sensor incorporates a row-addressable VCSEL array and a rolling-shutter 320x240 pixel array (320x480 SPADs) that are optically matched to efficiently receive optical power for a detection range of up to 48 meters. Our proposed sensor employs a 6-transistor (6-T) nMOS- only SPAD AFE circuit with a column-shared active recharging circuit. Additionally, we have developed a 2-step histogramming TDC (hTDC) with memory-capacity-efficiency. The proposed hTDC requires only 188 bits of histogram memory per column, thanks to our innovative priority memory structure. This approach provides a significant memory savings of up to 16 times when compared to conventional schemes utilizing 2-step hTDC.

Direct Time-of-Flight CMOS Depth Sensor with 600x102 SPAD Array for Addressable VCSEL-based Solid-State LiDAR System

This chip is for a fully solid-state LiDAR system that employs a VCSEL array as a transmitter (TX) and a CMOS depth sensor as a receiver. The row-addressable 192×100 VCSEL array and rolling- shuttered 192×100 pixel array (576×100 SPADs) are optically matched with one-to-one correspondence such that optical power is efficiently received for 128-m range detection. The proposed sensor employs a column-parallel 11-b 2-step multi-event TDCs for reducing TDC pileup distortion. The sensor also employs a 4-column-shared FIR filter that convolves with the laser profile function to provide the most correlated time codes with area efficiency. To reduce errors from multi-path of laser, reflection from vapor, and transparent objects, 4-echo TOF measurement is implemented by accumulating four fine histograms in the quartered histogram memory.

2.03-mW CMOS Image Sensor With an Integrated Four-Stacked Charge-Recycling Driver for Image Signal Transmission

This chip is a CMOS image sensor with a dedicated low-power imaging mode and low-power integrated transmitter. The proposed CMOS image sensor selectively operates in dual mode: a high-quality mode with a 1.5 V supply voltage of readout circuits to achieve a high signal-to-noise ratio (SNR), and a low-power mode with a 0.9 V supply voltage to support always-on imaging. To further reduce the power consumption in the low-power mode, a single-slope ADC embeds a power cutoff scheme in the comparator and a two-step conversion with dual reference voltages. To alleviate the SNR degradation in the low-power mode, which inherently occurs from voltage scaling, a correlated multiple sampling technique that consumes negligible power overhead is implemented using the proposed window-counting scheme. To reduce the significant power consumption that occurs during image signal transmission, an integrated transmitter with four-stacked charge-recycling drivers is used so that four symbols are simultaneously transmitted with a shared supply voltage. A prototype CMOS image sensor with 680 × 520 pixels is fabricated using 110-nm CMOS image sensor technology. The fabricated CMOS image sensor consumes only 301 μW (at 15 fps) in the sensor core and 2.03 mW including the transmitter and phase-locked loop (PLL) while generating low temporal random noise under 0.27 LSB with correlated multiple sampling.

CMOS LiDAR Sensor with Pre-Post Weighted-Histogramming for Sunlight Immunity Over 105 klx and SPAD-based Infinite Interference Canceling

This chip is a CMOS LiDAR sensor with high background noise (BGN) immunity. The sensor has on-chip pre-post weighted histogramming to detect only time-correlated time-of-flight (TOF) out of BGN from both sunlight and exponentially increased dark noise while enhancing sensitivity through higher excess voltage (Vex) of SPADs. The sensor also employs a SPAD-based random number generator (SRNG) for canceling interference (IF) from an infinite number of LiDARs. The sensor shows 8.08 cm accuracy for the range of 32 m under high BGN (105 klx sunlight and 48.72 kcps dark-count rate with increased Vex).

70mW Indirect Time-of-Flight Image Sensor with Depth Dynamic Range Enhancement and FDN Compensation

This chip is a low-power indirect time-of-flight (iTOF) image sensor with fixed depth noise compensation and dual-mode imaging for depth dynamic range (DDR) enhance- ment. To reduce the power consumption from high-frequency pixel modulation, a TX driver with a single-sided clock chain is employed in the sensor. The inherent phase delay of the clock chain and the delay of the row bus are measured using row-parallel and column-parallel time-to-digital convert- ers (TDCs) to compensate for the column and row fixed depth noise (FDN). To achieve a wide depth dynamic range (WDDR), the reconfigurable pixels and column circuits support dual-mode: short-range (SR) and long-range (LR) modes. A WDDR image is generated in a single frame through the mixed reconfiguration of the pixel array and interpolation. In addition, the temporal noise is suppressed without a significant time budget through a fast multiple sampling (FMS) scheme with 10b successive approx- imation register (SAR) analog-to-digital (ADCs). A prototype iTOF image sensor was fabricated using a 110 nm frontside illumination (FSI) CMOS image sensor (CIS) process and fully characterized. The sensor achieved a DDR of 4 m (0.7 to 4.7 m) with less than 1.7% nonlinearity and 0.9% depth noise. The FDN was suppressed to less than 2.1 cm at a low power consumption below 70 mW through the proposed compensation scheme using row and column TDCs. The temporal noise was only 0.48 mVrms owing to the FMS.

CMOS Image Sensor with Two-Step Single-Slope ADCs and a Detachable Super Capacitive DAC

We present a CMOS image sensor (CIS) with a 10b two-step single-slope (SS) analog-to-digital converter (ADC) for achieving a high conversion rate with improved linearity. Because of the two-step conversion, the A/D conversion time is decreased by a factor of 16 relative to the conventional SS ADC. The column-parallel capacitive DACs (CDACs) are connected with a detachable super CDAC to enhance linearity. These CDACs generate the ramp signal required for coarse conversion. In addition, a fine correlated multiple sampling (CMS) scheme suppresses temporal noise without a significant time budget, and an input crossing comparison scheme suppresses column fixed pattern noise (CFPN) from the various input common-mode voltages. The prototype CIS was fabricated using a 110 nm CIS process and was fully characterized. The proposed two-step SS ADC achieves an integral nonlinearity of -0.89/+1.04 LSB and a differential nonlinearity of -0.67/+0.91 LSB. In addition, the prototype CIS has a temporal noise and CFPN of 0.243 mVrms and 0.14%, respectively.

CMOS Skin Sensor for Mobile Skin Diagnosis Using an Electronic Cotton Pad

This chip is a complementary metal–oxide semiconductor (CMOS) skin sensor for detecting hydration, sebum, and ultraviolet (UV) protection. This sensor employs pixels comprising interdigitated capacitors (IDCs) for detecting hydration and a 30 × 24 photodiode (PD) array for detecting UV protection and sebum. The 4 × 8 pixels with IDCs over the PDs are used for area efficiency; they afford reliable detection regardless of the skin contact area and a high sensitivity, which is achieved via pixel merging. For the readout of both IDCs and PDs, a column-parallel multiple-sampling analog front-end and a 9b successive approximation register analog-to-digital converter are integrated. To detect UV protection under different wavelengths of UVA and UVB, we implement the spatiotemporal delta readout of the PDs. Furthermore, a fully characterized, proof-of-concept prototype chip is fabricated using a 110-nm CMOS process. Compared with conventional skin sensors, the proposed sensor exhibits higher sensitivities of 0.25%/min and 2.32%/mL in detecting dehydration rate and sebum levels, respectively. Moreover, the sensor can detect UV protection under UVA and UVB wavelengths. Owing to its core size of 2.32 × 4.65 mm2, the proposed sensor can potentially be integrated into cotton pads for mobile skin diagnosis.

Scanning CMOS LiDAR Sensor with on-chip histogramming TDC and Interference Suppression Filter

We developed a 36-channel scanning light detection and ranging (LiDAR) sensor with an on-chip single-photon avalanche diode (SPAD) array. The sensor has a 11b in-situ histogramming time-to-digital converter (hTDC) with a small area of 3000×78μm2/channel based on the mixed-signal accumulator. The sensor also employs an embedded interference (IF) filter for reliable direct time-of-flight (dTOF) measurement even if 32 different LiDARs interfere. The LiDAR system has a beam scanner that consists of dual laser diodes (LDs) for IF elimination and hybrid mirror such that high-resolution image of 2200 × 36 can be acquired with a wide field of view (FOV) of 120° × 8°.

Dynamic Pseudo 4-Tap CMOS Indirect Time-of-Flight Image Sensor

We developed a 320×240 iToF CIS with motion artifact suppression and BGL cancelling over 120klx. With an on-chip image signal processor (ISP) with an up-scaler, the iToF CIS generates 640×480 depth images. To enhance the depth accuracy of the 2T pixels in high-frequency demodulation, we implement a field-accelerated PPD with a trident-shaped implant (trident PPD). For the motion artifact suppression, we report a dynamic pseudo-4-tap (P4T) scheme that performs 4PH in a single frame by driving two 2T pixels with spatially-temporally alternated phases. With the P4T scheme, the region with motion generates a depth image in a single frame, whereas the region without motion generates depth image with higher accuracy in two frames. For cancelling BGL with high area efficiency while suppressing the fixed pattern noise (FPN), we implement a switching ΔΣ BGL cancelling (BGLC) scheme with an over-pixel MIM capacitor using a backside illumination (BSI) process.

Indirect Time-of-Flight Depth Sensor with Integrated Depth Frame Difference Detection 

This chip is a depth sensor with integrated frame difference detection. Instead of frame difference detection using light intensity, which is vulnerable to ambient light, the difference in depth between successive frames can be acquired. Because the conventional time-of-flight depth sensor requires two frames of depth-image acquisition with four-phase modulation, it has large power consumption, as well as a large area for external frame memories. Therefore, we propose a simple two-step comparison scheme for generating the depth frame difference in a single frame. With the proposed scheme, only a single frame is needed to obtain the frame difference, with less than half of the power consumption of the conventional depth sensor. Because the frame difference is simply generated by column-parallel circuits, no access of the external frame memory is involved, nor is a digital signal processor. In addition, we used an over-pixel metal–insulator–metal capacitor to store temporary signals for enhancing the area efficiency. A prototype chip was fabricated using a 90 nm backside illumination complementary metal–oxide–semiconductor (CMOS) image sensor process. We measured the depth frame difference in the range of 1–2.5 m. With a 10 MHz modulation frequency, a depth frame difference of >10 cm was successfully detected even for objects with different reflectivity. The maximum relative error from the difference of the reflectivity (white and wooden targets) was <3%. 

CMOS Light Detection And Ranging (LiDAR) Sensor with Integrated Time-to-Digital Converters

A histogram-based time-to-digital-converter (TDC) array for direct time-of-flight depth sensors was proposed. The 12-bit TDC array measures the time of flight (TOF) of a light pulse that is detected using an external single-photon avalanche diode (SPAD) array. As SPADs are noisy owing to dark electrons and scattered photons, statistical measurements based on histograms are essential, but require a large memory. In this work, the authors propose a mixed-signal TDC with an integrated histogram generation unit (HGU) that reduces memory requirement significantly as well as filters out invalid TOF readings. In addition, for application of the high- resolution SPAD array, an area-efficient TDC array with high uniform- ity was implemented by the proposed phase-dependent latching and temporal double sampling schemes. The prototype chip was fabricated using a 180 nm CMOS process, including 8-channel TDCs. The measurements show an integral non-linearity (INL)/a differential non- linearity (DNL) of 0.76/0.49 least significant bit (LSB) and high uniformity under 0.19 LSB. The HGU was designed off-chip for prior verification, and was post-simulated with the measured TOF from the fabricated chip. Using the histogram-based TDC, the authors could detect an object located at a distance of 3 m accurately while reducing memory requirement by more than 128 times. 

CMOS Light Detection And Ranging (LiDAR) Sensor with Programmable Filter 

We developed a CMOS LiDAR sensor with a multi-resolution single-photon avalanche diode (SPAD) array and area-efficient filter circuit with a programmable reference for environment-adaptive noise suppression. Four SPADs compose a macro-pixel for detecting targets at a short distance with high resolution, whereas 4 × 4 SPADs are reconfigured to a super-pixel for a long distance with enhanced signal-to-noise ratio. To provide environment-adaptive noise suppression in the miniaturised SPAD array, an area-efficient filter circuit with a programmable reference is proposed: a digital macro-filter corresponding to a macro-pixel and an analogue super filter corresponding to a super-pixel. The prototype chip was fabricated with a 110 nm CMOS image-sensor process, including 128 SPAD farm arrays and 128 analogue front-end circuits. With high background light of over 90 klx and a high dark-count rate of over 27.1 kHz, the time-of-flight could be measured by filtering out invalid pulses from noise without using multiple time-to-digital converters per pixel, which enables the implementation of miniaturised LiDAR-sensor systems on a chip.

Always-On CMOS Image Sensor for Mobile & Wearable Devices

The always-on sensor continuously captures images for smart sensing, such as face detection, eye tracking, and gesture recognition, and it provides high-resolution images for capturing pictures with a unified sensor. The sensor employs a switchable dual mode: always-on (AO) mode with low power consumption and photo-shooting (PS) mode with high signal-to-noise ratio. The fabricated 640 x 480 pixel prototype sensor operates at 45.5 uW (at 15 fps, 320 x 240) in the AO mode, which significantly extends battery life when performing always-on sensing.

Depth Sensor with Pipelined Background Suppression and In-Situ Noise Cancelling

We present a depth sensor with integrated background suppression scheme for detecting small signals out of unwanted background signals. For the background suppression, differential signals with suppressed common-mode background signals are sampled within a short sub-sensing time in order to avoid the saturation from strong background signals. Analog differential signals are digitally accumulated multiple times in one integration time for high SNR. The column-parallel background suppression circuits are pipelined in order to achieve short sub-sensing time. Moreover, additional operations for the noise cancelling are merged with the background suppression and no extra timing for the noise cancelling is required during the sub-sensing time. In order to extend a dynamic range further, sensitivity can be adjusted to be decreased using in-pixel capacitors when strong background signals are present. The prototype image sensor with 1328 × 1008 pixel array has been fabricated with a 0.11 mm 1P4M CIS process. We have successfully captured images from the fabricated sensor chip with strong background signal more than 9300 lx. The dynamic range of the sensor can be extended to 107.3 dB.

Time-of-Flight 3-D Camera with Adaptable Background Light Suppression Using Pixel-Binning and Super-Resolution

This sensor can operate in outdoors where the BGL can be higher than 100klx. This feature is implemented without increasing the pixel size, and hence achieving the smallest pixel size among all reported TOF sensors up to date. In addition, the sensor provides an adaptable BGL suppression using spatiotemporal resolution control by pixel-binning and super-resolution to guarantee the best image quality in all situations.

CMOS Imaging SoC with Embedded Feature Extraction Algorithm for Object-Detection

Distributed sensor nodes typically operate under the constraint of limited energy source, and power consumption is an important factor to extend the lifetime of sensor systems. Several low-power imagers have been reported for the application of wireless sensor network. However, the biggest power consumption comes from wireless signal transmission due to the large bandwidth of image signals. One way to reduce the bandwidth is to generate signals only when event happens by monitoring temporal changes. However, this event-based imaging has extraneous redundancy because the sensor may also respond to environmental conditions, such as change of illumination or background movement in addition to actual target objects.

Ultra-Low-Power CMOS Image Sensor for Wireless Sensor Node

For outdoor surveillance, sensitivity and dynamic range are important to deliver reliable images over widely changing illumination. However, constant monitoring with maximum awareness requires large power consumption and is not suitable for energy-limited applications such as battery-operated and/or energy-scavenging wireless sensor node. One of the ways to reduce power is voltage scaling. However, it significantly reduces the SNR and results in poor image quality. The signal can be easily corrupted from the noise in dark conditions or be saturated in bright conditions. Most imagers with high-sensitivity and wide dynamic range reported, consume large power > 50 mW, unsuitable for wireless imager node applications. Therefore, it is imperative to implement a sensor adaptable to environmental changes: e.g., the sensor keeps monitoring at extremely low power and only turns into high sensitivity or wide dynamic range operations when requested due to illumination changes or requested from the host for detailed image transmission. The sensor changes its operation back to the monitoring mode when enough operating energy is not available from the battery or energy-harvester. In this paper, we report an adaptive CMOS image sensor which employs four different modes: monitoring, normal, high-sensitivity and wide-dynamic-range (WDR) modes. This adaptable feature enables the reliable monitoring while significantly enhancing the battery lifetime for wireless image sensor nodes. A prototype chip has been fabricated using 0.18μm CIS process. We achieved a normalized power of 15.4 pW/frame•pixel (from the total power consumption) in monitoring mode.

High-Speed Spatial-Temporal Multi-Resolution Imager for Robot & Machine Vision 

A multi-resolution CMOS image sensor which simultaneously generates spatial-temporal multi-resolution images from dual channels: one for normal images ( 960 fps) with a reduced spatial resolution for moving objects in the region-of-interest (ROI). The entire image with the details in stationery objects and the suppressed motion-blur in moving objects can be acquired at low power consumption with optimal use of bandwidth.