Semantic segmentation for robots is a way that we teach a robot to understand the images. To do that, the robots will use a set of algorithms to process the information to recognize, understand what is in the image in a pixel wise level. Here are some examples of semantic segmented images:

Classical Methods

Before the Era of Deep Learning, a number of robust image processing techniques were designed to do semantic segmentation, segmenting the images into semantic areas of interest. We name a few here that are still wildly used in our current world:

• Gray Level Segmentation
  The simplest method of semantic segmentation involves the assignment of hard-coded rules or properties that an area must fulfill for a specific label to be applied to it. The rules can be framed in terms of the properties of the pixel, such as its strength at the gray level.

see here: Split and Merge Algorithm

• CRFs (Conditional Random Fields)
  CRFs are a group of methods of statistical modeling used for organized prediction. CRFs should regard "neighboring background" as the relationship between pixels before making predictions, unlike discrete classifiers. This makes it an excellent candidate for the segmentation of semantics. The use of CRFs for semantic segmentation is discussed in this section. A finite set of possible states is associated with each pixel in the image. The sum of the unit and pair cost of all pixels is referred to as the CRF 's energy (or cost / loss). To obtain a good segmentation output, this value can be minimized.

see here: CRFs Algorithms

• Watershed
  Watershed segmentation is a region-based approach that uses the morphology of images. It needs at least one marker ("seed" point) to be selected within each image object, including the context as a separate object. Markers are selected by an operator or are given by an automated procedure which takes account of the objects' application-specific knowledge. They can be grown using a morphological watershed transformation once the objects are labelled.

see here: Watershed Algorithm

• GrabCut
  The Grab Cut method addresses, given certain limitations, the challenge of separating objects from backgrounds in a colored picture. The user is asked to label a single rectangle around the object, to identify the outer part of the rectangle as a definite background, and the inner part of the rectangle as an unknown combination of the object (front) and a certain background.

see here: GrabCut Algorithm

ResNet

The way in which ReNet replaces convolutional layers with multi-directional recurrent neural networks (RNN) is highlighted by ReNet, which provides an alternative way of constructing the architecture of the network. The ReNet-based representative semantic segmentation technique is ReSeg. The deep residual network (ResNet) makes a much deeper network possible and achieves better performance in different vision tasks. Its key contribution is to model the residual representation into the structure of the CNN network, which solves the difficulty of training a very deep structure of the network. This is revolutionary in a sense that it gives the robotics another architecture other than the FCNN.

DenseNet

DenseNet links every layer to each other, unlike the conventional approach that allows a network deeper or wider. Its benefit lies in the following aspects : 1) fewer parameters, 2) more function reuse, and 3) a better training mechanism that relieves the problem of degeneration of the vanishing gradient and model. The DenseNet-based representative semantic segmentation methods include DenseASPP, FC-DenseNet, and SDNet. This may be applied in a remote high accuracy high latency robotics system.

MobileNet

In order to balance accuracy and computing costs, it is necessary to design networks. Various lightweight networks have been developed in this sense. MobileNetV1 implements convolution in detail, which achieves a great efficiency increase. With 4.2 M parameters, it achieves 70.6 percent precision in the ImageNet classification task. MobileNetV2 is based on an inverted residual structure, addressing the shortcomings of MobileNetV1. With even fewer parameters, MobileNetV3 achieves better performance through the incorporation of the attention framework. For real-time applications such as robotics, MobileNetV1 and MobileNetV2 are very helpful.

ResNeXt

ResNeXt is illustrated in its homogeneous, multi-branch architecture, which has just a few hyperparameters to set, in order to boost the network efficiency while retaining the network complexity. DShortcut uses ResNeXt as its backbone for semantic segmentation methods.

Real-time semantic segmentation

A large number of methods have been proposed that target real-time semantic segmentation. While excellent success in terms of both accuracy and effectiveness has been achieved, there is still a wide area for improvement. Taking the state-of-the-art DFANet as an example, in terms of MIoU, DFANet is still 10 percent lower than Cityscapes' PSPNet. The multiobjective increase in precision and speed is therefore required in this direction of study.

Occluded objects segmentation

Regarding human vision, the occluded sections can be easily identified and retrieved by a person. However, existing computer vision technologies do not replicate this process well. One significant explanation is that, without the ability to transfer information, most existing segmentation algorithms target at a hard partition. Researchers suggested that this task be realized on the basis of depth knowledge and, for this task, they developed data such as PASCAL 3D+.

Weakly/Unsupervised segmentation

Classical methods such as watershed and GrabCut algorithms are in the definition of weakly or unsupervised segmentation. However, in modern machine learning era, most of the successful semantic segmentation are based off fully supervised learning on large annotated datasets. Researchers suggests that the weakly/unsupervised segmentation may be achieved by giving the robots a basis of knowledge and learn from classical methods such as clustering, subspace learning, and other refinements.

Semantic segmentation in Videos

Most of what we presented and learned about semantic segmentation is in the form of 2D images. The main focus of semantic segmentation also is heavily on the single-image level. However, real world applications for robotics requires visual recognition and understanding of videos footage, where each frame is highly correlated with the next one. Current methods uses oriented gradients and interframe correlation to improve the efficiency and coverage. However, when compared to image semantic segmentation, segmentation in videos still has a lot of potential to make significant practical impacts for robotics.