background removal

Remove image backgrounds automatically in 5 seconds with just one click. Don't spend hours manually picking pixels. Upload your photo now & see the magic

Free Background Remover: Remove BG from Image

Our AI background remover instantly removes the background of your photos in seconds. Then you can add any new background or leave it transparent

Image Background Remover

Remove the background from your image instantly. Get images in a transparent, white or customized background in 3 seconds or less for free

Abstract

Imaging spectroscopy integrates traditional computer vision and spectroscopy into a single system and has gained widespread acceptance as a non-destructive scientific instrument for a wide range of applications. The current state of imaging spectroscopy spans diverse applications including but not limited to air-borne and ground-based computer vision systems. This paper presents the current state of research and industrial applications including precision agriculture, material classification, medical science, forensic science, face recognition and document image analysis, environment monitoring, and remote sensing, which can be aided through imaging spectroscopy. In this regard, we further discuss a comprehensive list of applications of imaging spectroscopy, pre-processing techniques, and spectral image acquisition systems. Likewise, publicly available databases and current software tools for spectral data analysis are also documented in this review. This review paper, therefore, could potentially serve as a reference and roadmap for people looking for literature, databases, applications, and tools to undertake additional research in imaging spectroscopy.

Keywords

Imaging spectroscopyHyperspectral imagingImage processingComputer visionRemote sensingDeep learning

1. Introduction

The electromagnetic spectrum encompasses a broad range of electromagnetic radiation, each characterized by distinct wavelengths and frequencies (Zwinkels, 2015). These include, among others, ultraviolet, visible, infrared, microwaves, and radio waves, each characterized by unique electromagnetic properties such as energy levels, propagation characteristics, and interactions with matter, making them essential subjects of study across various scientific disciplines (Someda, 2017, Weinstein, 1988).

Visible light is an electromagnetic radiation perceptible to the human eye and constitutes a narrow bandwidth within the electromagnetic spectrum, specifically spanning wavelengths between 380 and 780 nanometers (Reinhard et al., 2010). This specific spectral range forms the basis for numerous well-established techniques and applications in the field of traditional vision and imaging processing (Gonzalez, 2018).

Current advancements in sensor technology have enabled the acquisition of images across a wide range of electromagnetic wavelengths (ElMasry & Sun, 2010b). These include both multi-spectral and hyperspectral images, which encompass more than the traditional three spectral bands used in visible spectrum imaging. Multi-spectral imaging (MSI) and hyperspectral imaging (HSI) techniques involve capturing multiple images at narrow and contiguous spectral bands spanning a wider range of the electromagnetic spectrum, thereby providing an enhanced level of spectral detail. These advanced imaging methods are collectively referred to as imaging spectroscopy.

Objects in the natural environment exhibit distinct spectral responses when interacting with electromagnetic radiation (Heald & Marion, 2012). These responses encompass the absorption, transmission, and reflection of electromagnetic radiation, giving rise to unique spectral signatures for various elements and object materials (Khan, Thomas, Hardeberg, & Laligant, 2019). Consequently, imaging spectroscopy enables comprehensive spectral analysis for a wide range of applications in a diverse range of fields including remote sensing (Arellano, Tansey, Balzter, & Boyd, 2015), precision agriculture (Ravikanth, Jayas, White, Fields, & Sun, 2017), chemistry (Cheng, Sun, Pu, & Zhu, 2015), medicine (Lu & Fei, 2014), process monitoring (Pan, Chyngyz, Sun, Paliwal, & Pu, 2019), environmental applications (Bourguignon et al., 2010, Zhou and Camba, 2021), military (Shimoni, Haelterman, & Perneel, 2019), food industry (ElMasry et al., 2012, Lorente et al., 2012) and other commercial applications (Xing et al., 2019). It is estimated that the global HSI systems market is expected to grow from USD 15.4 billion in 2021 to USD 35.8 billion by 2026 Wood. The major market segments include; military surveillance, remote sensing, machine vision & optical sorting, life sciences & medical diagnostics, and other applications.

Fig. 1. Visualization of Hyperspectral cubes at different wavelengths (a), RGB Image of the hyperspectral cube (b), Pixel-wise classification map, Pure spectral signature of different objects (d), Score map obtained for each pure spectrum. The images are generated using the hyperspectral toolbox of MATLAB 2021 (a). The source image is Jasper Ridge, captured via the airborne visible/infrared imaging spectrometer (AVIRIS). The data set contains areas of water, land, road, and vegetation.

A spectral image is usually in the form of a cube, where the first two dimensions represent the spatial information and the third dimension represents a series of spectral images captured at different wavelengths. For example, a hyperspectral image contains abundant spectral data captured at hundreds of distinct wavelengths across the electromagnetic spectrum, while a multispectral image comprises images captured at several tens of wavelengths. The number of wavelengths/spectral channels in a spectral image usually depends upon the application and the type of instrument. To process this high-dimensional data effectively, many image processing and machine learning pipelines have been developed in the literature, ranging from pre-processing (Vidal & Amigo, 2012), calibration (Behmann et al., 2015) noise modeling (Acito, Diani, & Corsini, 2011), dimensionality reduction (Huang, Shi, He, Duan, & Luo, 2019), anomaly detection (Zhang, Wen, & Dai, 2016), clustering (Zeng, Cai, Liu, Cai, & Li, 2019), spectral unmixing (Bendoumi, He, & Mei, 2014), feature extraction (Kumar, Dikshit, Gupta, & Singh, 2020), representation learning (Sellami & Tabbone, 2022), classification (Yusuf & Alawneh, 2018) and regression, tailored specifically to imaging spectroscopy (Minasny & McBratney, 2008).

In imaging spectroscopy, especially in HSI, the spectrum for each pixel is measured at different wavelengths as shown in Fig. 1. In order to provide more information on what is imaged, the radiation striking each pixel is broken down into many different spectral bands (Armin Schneider, 2017). Fig. 1(a) presents a hyperspectral cube and each slice of the cube represents images captured at different wavelengths, Fig. 1(b) presents a color image from the hyperspectral cube, and part (c) presents a pixel-wise classification map. The original image is from the hyperspectral dataset Jasper Ridge, captured using AVIRIS (Green et al., 1998) air-borne spectrometer (Kruse et al., 1993). Part (d) and (e) of Fig. 1 show the unique spectral signature and spectrum score of each entity respectively. The spectrum score is computed by measuring the degree of similarities between spectra using the Spectral Angle Mapper (SAM) classification algorithm (Kruse et al., 1993). Imaging spectrometers typically operate in the 0.4 to 2.5

μm

wavelength range, capturing the visible and solar-reflected infrared spectrum (i.e., near-infrared or NIR, and short-wavelength infrared or SWIR) from the observed materials (Paoletti, Haut, Plaza, & Plaza, 2019). While hyperspectral sensors have been predominantly utilized in satellite applications, narrow-band hyperspectral sensors have also explored for ground-based computer vision systems, such as face recognition (Qureshi et al., 2020, Uzair et al., 2015), document image analysis (Qureshi, Uzair, Khurshid, & Yan, 2019), ink-mismatch detection (Khan, Shafait, & Mian, 2013), forgery detection (Khan, Yousaf, Abbas, & Khurshid, 2018), and non-destructive forensic analysis of classified documents (Khan, Khan, Yousaf, Khurshid, & Khan et al.Abbas, 2018).

Given the emergence of this field in the past decade, many researchers have provided reviews on imaging spectroscopy. For example, the use of HSI imaging in forensic science is presented in Melit Devassy and George (2021), document image analysis in Qureshi et al. (2019), skin diseases in Chen et al. (2020), cancer cell segmentation in Aloupogianni et al. (2022), deep learning methods for agriculture in Fadhlallah Guerri, Distante, Spagnolo, Bougourzi, and Taleb-Ahmed (2023), spectral reconstruction methods from RGB image in Zhang et al. (2022), and a survey on applications of HSI is presented in Khan, Khan, et al. (2018). The medical applications of HSI are discussed in Fei, 2019, Karim et al., 2023, Lu and Fei, 2014, and food and safety applications of HSI are discussed in Feng and Sun (2012). The use of machine learning and deep learning for hyperspectral image classification is presented in Datta et al. (2022).

While earlier reviews have provided valuable insights, the field of imaging spectroscopy has experienced significant advancements in recent years. For example, there has been substantial and rapid progress in HSI imaging technology. Similarly, the availability of new HSI datasets, and their usage in new applications, as shown in Fig. 2, Fig. 3. Fig. 2 illustrates that the research landscape and trends in the field of HSI during 2015–2022 have rapidly evolved. In terms of source comparison (i.e., platforms publishing HSI-related research), IEEE and ScienceDirect are leading the HSI-related research production with larger impact and visibility (Fig. 3).

Since the field is continuously evolving at a faster pace, it is important to review the recent advancements, developments, and challenges to reflect the state of the field. Thus, a comprehensive review of the recent advancements in terms of applications, databases, software, and future prospects could progressively benefit the community. In connection with this, we present an updated comprehensive review of the state of HSI and its state-of-the-art applications across various scientific and professional fields. Such a review could potentially act as a roadmap and would progressively provide useful references to research and the professional community in terms of literature, databases, tools, and applications of imaging spectroscopy.

The rest of the paper is organized as follows. We first discuss imaging spectroscopy acquisition systems with camera models and lens specifications. Within the spectroscopy, our focus is mainly on HSI. This is followed by the discussion on pre-processing challenges and methods for processing hyperspectral images, including spikes removal, dead pixels, compression, and spectral processing in Section 3. Applications of HSI in modern-day societies, such as medicine, precision agriculture, remote sensing, food quality control, material classification, document image analysis, and face recognition are discussed in Section 4. Next, we present open-source hyper-spectral databases and software libraries for processing hyperspectral images in Section 5. Finally, Section 6 summarizes the key insights from this study. A list of acronyms used in the paper is given in Table 1.

Fig. 2. Terms co-occurrence in Hyperspectral publications during 2015–2022. During 2015–2022, several distinct clusters could be seen including the largest occurrence of reflectance, spectroscopy, classification, quality, algorithm, and feature extraction (listed chronologically).

Table 1. List of acronyms used in the paper.

Acronyms

Definition

CCD

Charge-coupled device

CMOS

Complementary metal oxide semiconductor

HSI

Hyperspectral imaging

MSI

Multispectral imaging

LAI

Leaf area index

LCTF

Liquid crystal tuneable filter

UAV

Unmanned aerial vehicle

PCA

Principal component analysis

ICA

Independent component analysis

PSLR

Partial least square regression

SAM

Spectral angle mapper

MSC

Multiplicative scatter correction

CNN

Convolutional neural network

GNN

Graph neural network

NASA

National Aeronautics and Space Administration

Fig. 3. Publications related to imaging spectroscopy across various peer-reviewed journals. IEEE Transactions on Geosciences and Remote Sensing from the IEEE platform is the most prominent source with the largest G-Index and H index followed by the Journal of Food Engineering from the ScienceDirect platform. Considering the specialized nature of these journals, geosciences and food engineering fields are the ones where the most prominent applications of HSI might be seen.

2. Acquisition of hyperspectral images

Spectral sensors are used to collect data in the form of images, where each image represents a specific wavelength range of the electromagnetic spectrum known as a spectral band. These images are combined to create a three-dimensional hyperspectral data cube for further processing and analysis. The cube consists of two spatial dimensions (x and y) and a spectral dimension

comprising a range of wavelengths (Khan, Thomas, Hardeberg, & Laligant, 2017) .

There are different modes of operation for acquiring spectral data, each with its own advantages and disadvantages. One such technique is the Whiskbroom method, which involves mounting a linear array of detectors on a moving platform, such as an aircraft or satellite, and pointing it towards the ground. As the platform moves forward, the detectors collect data from a narrow strip of the ground, known as a swath. The collected data is used to create an image of the scene, with each pixel containing spectral information.

Another method is the pushbroom method, which involves line scanning across a single axis, and an image is created by either moving the camera or the objects of interest. The movement should be consistent to avoid spatial distortions in the acquired data. Plane scanning is another method for acquiring spectral data, where the entire region is scanned at different wavelength intervals. This can be done using a liquid crystal tuneable filter (LCTF) that allows a particular wavelength bandwidth at a time. The resultant images are embedded on top of each other to acquire spectral data.

A recent development in imaging spectroscopy is the integration of thin-film spectral filters on top of the image sensor. This eliminates the need for complex optical systems and moving hardware systems, allowing for rapid data acquisition and use in video mode without requiring a stationary scene.

During data acquisition, it is crucial to ensure the presence of a consistent light source and calibration tiles. Any change in illumination can affect the acquired data, which can have an impact on the further processing of the imaging spectroscopy data (Khan, Thomas, Hardeberg, & Laligant, 2018). Methods have been developed for illuminant invariant representation of images in uncontrolled imaging conditions (Khan, 2018, Khan, Thomas, and Hardeberg, 2017, Khan, Thomas, and Hardeberg, 2018, Khan, Thomas, Hardeberg, and Laligant, 2017).

Overall, the acquisition of spectral data involves various techniques and considerations, and selecting the appropriate method depends on the specific requirements of the application.

2.1. Air-borne hyperspectral sensors

Fig. 4 shows a hyperspectral image captured by an air-borne satellite sensor. Existing HSI devices can acquire 3D xy

volumes using 2D sensors ij by transforming the spectral dimension in time or arranging it in space. The precision of these sensors is often assessed in terms of spectral resolution, which is the breadth of each collected band of the spectrum. It is possible to identify objects even if they are only captured in a handful of pixels, provided the scanner identifies a large number of reasonably small frequency bands.

Fig. 4. An example of HSI imaging in mapping soil, vegetation, and water.

The figure is modified from Khan, Khan, et al. (2018).

2.1.1. Spatial scanning

Spatial resolution can be defined as the smallest detail in an image, which determines the clarity of the image (Gonzalez, 2009). In spatial scanning, each two-dimensional (2-D) sensor output represents the entire slit spectrum. Slit spectra are obtained by projecting a strip of the scene onto a slit and dispersing the slit image with a prism or grating in spatial scanning HSI systems. Generally speaking, spatial resolution shows the size of the pixel, whereas spectral resolution shows the content of the pixel in an image.

2.1.2. Spectral scanning

A single pixel’s spectrum in a hyperspectral image can reveal significantly more information about the material’s surface than in a regular image. Another acquisition approach that requires the incoming images to be filtered to produce a

xyk

image at time

is spectral acquisition in time. Each 2-D sensor output in spectral scanning represents a monochromatic (’single-colored), spatial (x, y) map of the scene. HSI spectrum scanning devices are typically based on optical band-pass filters (either tuneable or fixed). The scene is spectrally scanned by switching between filters while the platform remains steady.

Fig. 5. Schematic diagrams of an RGB camera (a) and a typical hyperspectral imager (b). Each pixel in an RGB image combines three distinct color values that are integrated from the R, G, and B spectra. Each pixel in a hyperspectral image is a continuous spectral curve formed from a number of tiny spectral bands. FO front objective, CL collimating lens group, and FL focusing lens group. The images are from the KAUST-HS open-source dataset (Li, Fu, & Heidrich, 2021).

Figure adopted from Zhang et al. (2022).

2.1.3. Temporal scanning

Temporal resolution is defined as the amount of time needed to revisit and acquire data for the exact same location by the hyperspectral sensor (Ma et al., 2021). The fundamental trade-off here is between spectral and temporal resolution, with spectral filtering performed through mechanical filter wheels (usually restricted to MSI) or acusto-optical or liquid-crystal tuneable filters (enabling HSI at a higher cost). The ability to obtain a spectrum image by simply taking a snapshot is very appealing for time-constrained applications, and this has sparked a lot of study (Hagen & Kudenov, 2013).

2.1.4. Spatio-spectral scanning

In spatio-spectral scanning, each 2-D sensor output represents a wavelength-coded (’rainbow-colored,’ = (y)), spatial (x, y) representation of the scene. A camera at some non-zero distance behind a simple slit spectroscope (slit + dispersive element) is used as a prototype for this technology, which was introduced in 2014 (Grusche, 2014).

2.2. Ground-based hyperspectral systems

Imaging spectroscopy is a versatile technology that finds applications in various fields such as medical, forensic image analysis, agriculture, and material classification. In ground-based applications, a hyperspectral imaging system typically comprises a light source, a charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) camera, and a spectrograph.

Fig. 5 shows a typical ground-based hyperspectral imaging system. The system collects data over a range of wavelengths, typically spanning from the visible to the near-infrared region of the electromagnetic spectrum. The resulting hyperspectral data cube can be analyzed using various techniques, such as spectral unmixing, classification, and feature extraction, to extract valuable information about the scene or object being imaged.

Recently, there has been growing interest in using deep learning algorithms to build hyperspectral images from RGB photos or other sparse spectral representations (Xiong et al., 2017). This approach has the potential to reduce the cost and complexity of HSI systems and make them more accessible to a wider range of applications.

3. Pre-processing of hyperspectral images and data analysis methods

Imaging spectroscopy often results in a multi-dimensional and massive amount of data, which requires extensive pre-processing before useful information extraction and analysis. In this section, we discuss some pre-processing challenges and data analysis algorithms for HS images. Hyperspectral data analysis typically involves several preprocessing steps to correct for various sources of noise and artifacts, such as radiometric calibration, atmospheric correction, and noise reduction.

Page updated

Google Sites

Report abuse