Version 4.0.0

There are several folders which can be categorized into 2 groups: code folder and output folder.

code folders:

output folder:

How to run it?

  1. Go to toolbox_ITSBN and open the file fn_ITSBNIMageSegm2.m. There you can change the features in the section marked official feature experiment.
  2. Go to the folder main_code and open and run main_ITSBNImageSegm.m. This code is a main code that calls all the functions necessary for image segmentation. The code can process all images in a folder at once.
  3. The segmentation results are stored in the folder Results. Other outputs are also found in the output folder table. The details of the segmentation step by step are stored in each folder separately in the folder main_code.

main_ITSBNImageSegm.m

This is the main function to run ITSBN. The code calls other functions necessary for image segmentation. Normally there are 2 modes to run the code:

  1. one-by-one mode: Each image in the database is processed and gives the segmentation one by one from first to last. This mode is suitable when having small number of images in the database.
  2. experiment mode: Each process is applied to all the images in the database at once, then move on to the next process. For instance, superpixelize all the images in the database at once, and all the superpixel output are collected in the folder superpixels. An advantage is that we can see the behavior of the parameters on all the images before moving on to the next step. The user can comment all except the process you want to run one at a time.

All the processed data (e.g., superpixels, feature vectors, ITSBN tree) can be stored for future experiments in case that the environment of superpixel, tree is needed to be controlled. The code does the following jobs:

  1. addpath to necessary folders and install toolbox
  2. Retrieve the image file name in the image database folder determined in the previous process
  3. Select a superpixel algorithm (quickshift or Mori's), then determine the size of superpixel in each level.
  4. Plot the ground truth segmentation overlaid on the original image.
  5. Construct ITSBN tree from the superpixels
  6. Extract the features from each superpixel
  7. Image segmentation using GMM and ITSBN

There are some interesting variables in this code.

  • C : A (L x 2) cell array representing # of class labels in a particular level is shown in the table below. Immediately we will know that the root level to the label level, each has 3, 6, 6, 7 and 7 class labels respectively. We put 0 in the level H^0 as it is not used in this version but possibly in the future application. Note that the high-resolution segmentation result is derived from the label (H^1) level.
  • gfd

ITSBN Toolbox

The functions in the folder toolbox_ITSBN will be explained here.

fn_SuperpixelMori2.m

function [] = fn_SuperpixelMori2(imagename, imageext, N_l4, N_l3, N_l2, N_l1)

This is the function to make superpixels in each level. The level here is fixed to 4. The output of the file is cell array img_label_hier whose format are

each column:

  • column#1: the desired number of superpixel input by the user
  • column#2: the output number of superpixel determined by the superpixel algorithm. Note that this number might not be the same as the desired number. #L
    • column#3: the superpixel indices assigned for each pixel. For instance, all pixels in the superpixel#7 region, will be assigned with 7. The index of each level always runs from 1 to #superpixel in the level. Each level look like the figure below

each row:

  • The first row: is for the label level (level H^1) N_l1
  • The second row: for level H^2, N_l2
  • The last row is the second to top level H^(L-2). Note that we know already that the root is in the level L-1 and has only 1 superpixel on the level, so we don't store the root information here. N_l4

For instance, the matrix looks like this.

The cell array img_label_hier is stored in the file imagename_img_label_hier.mat

fn_makeTreeFromLabel.m

[] = fn_makeTreeFromLabel(imagename)

The code take as an input the matrix img_label_hier, then output the followings:

  • Z : the structure (adjacency) matrix whose row and column represent child and parent respectively. The size of the matrix is (#nodes x #nodes +1), for instance, in this case we will have (306 x 307) structure matrix Z. The reason behind #nodes +1 is the phantom node, but this information in node used yet in this version of code.
  • n_level : # of level from H^(L-2) to H^(1), in this case n_level = 4.
  • H : (L+1 x 2) cell array. In this case, L = 6 (i.e., Evid, H^1, ...,H^5 ), however, we include the phantom node (in level H^6), marked by the biggest node index in the tree, in the level H{6+1,:} as well. The first column contains the level# H{level+1,1} = level; % level H[level]. That is, the information of the level H^l can be found from H{l+1,:}. The second column contains the node indices of ITSBN in the level. The cell array H is for instance shown as the table below. Note the first row of H was supposed to contain the information of the evidence level (H^0), however, I change my mind because of the idea of having evidence in every level.
  • node_label_hier : A (L-2 x 3) cell array, very similar structure to img_label_hier except that the third column of node_label_hier stores the node indices as oppose to the superpixel indices in img_label_hier.Note that the matrix

All the outputs from the code are stored in imagename_package.mat.

fn_imageFeatureExtraction3.m

[] = fn_imageFeatureExtraction3(imagename, imageext)

All the feature vectors are calculated from this code. The code take the imagename_package.mat as input and store feature vectors in the file imagename_feature.mat. The changes from the previous version (fn_imageFeatureExtraction2.m) are that the output matrices Y and Y_index store the features from all the evidence in every scale (H^1 to H^(L-1)) as oppose to only H^1 in the previous version.

  • Y : A (#superpixel x D_feature) matrix. Each row represents the extracted feature vector from a superpixel from level H^1 to H^(L-1). Each column represents the features determined in the code.
  • Y_index : A (#superpixel x 1) column vector containing the node#ID associating with the feature vector in Y. The index is in descending order from root to leaf. The table below depicts the matrix Y_index.

This code works as follows:

  1. Apply the filters (sRGB, sLuv, DWT, Texton) on the raw image first to get the pixel level features
  2. Average the feature within a superpixel
  3. Put the averaged vector in each row of Y
  4. Do step 1-3 for every level H^1 to H^(L-1)

Note that the features like DWT and texton has multi-scale property that we can benefit by having multi-scale evidence. For instance, the high-frequency coefficients are input to the bottom level, and the low-frequency coefficients are connected to the top level. However, in this version of the code we don't have time to do that.

fn_imgSegmentationGMM.m

function [I_label, I_post] = fn_imgSegmentationGMM(imagename, imageext, C, W)

The algorithm uses GMM to segment the raw data in pixel level, and the features are Lab and location x, y of the image.

fn_ITSBNImageSegm3.m

[] = fn_ITSBNImageSegm2(filename_input,...

imagename, imageext,...

C,...

iteration_max,...

exp_number,...

cpt_spread_factor,...

added_diag_cpt,...

epsilon_F)

This code, regarded as the core of the ITSBN, takes as input the image and all the features extracted from the previous process, then output the segmentation results in imagename_ITSBN_segm_result.mat. The results are composed of

  • segs : A (L-2 x 1) cell array containing pixel-level segmentation results for each level.
    • posterior_marg : A (L x 4) cell array containing the maximum posterior marginal information in all the hidden level (H^1 to H^(L-1)).

Since we are using the BNET MATLAB toolbox, we have some tricks to combine the toolbox with our method seamlessly. The code is explained in detail as follows:

  1. Select the features to use in the experiment by going to the section saying "Pick FEATURES to use in the segmentation", then you can pick the columns of Y corresponding to the features you want.
  2. In the section saying "Prepare the number of levels and points in the dataset" we prepare some information about the number of nodes and levels necessary for the remaining code.
    • N : The number of superpixel in the level H^1, i.e., in this case N = 200
    • L : The total number of level from H^0 to H^(L-1), i.e., in this case L = 6
    • K : The total number of hidden nodes in the ITSBN from H^1 to H^(L-1) excluding the phantom node, i.e., in this case K = 1 + 5 + 20 + 80 + 200 = 306
  3. Fit the GMM to the feature vectors Y, here we will get the mu_c, Sigma_c for each class label c.
  4. Initialize conditional probability table (CPT) for the ITSBN. The (L x 2) cell array CPT is composed of
    • column#1: store the number indicating the level l. You may figure out that CPT{l+1,1} = l, which makes the notion of col#1 seem redundant, but we still store l there for our convenient.
    • column#2: store the CPT matrix of that level. That is, CPT{l+1,2} is the CPT of the level l as the CPT is shared among all nodes inthe same level. Note that there is no restrictions of the dimension of the CPT, so it means we can make any arbitrary dimension as we wish. For example, we can CPT for level l and l+1 to be r x s and s x t respectively.
  5. In this step, we use BNT to do the inference for the posterior marginal distribution at each hidden node. Here we use Bayes-Net Toolbox (BNT) to infer the hidden nodes. However, BNET does not fully support our parameter learning for ITSBN, so we use BNT just to calculate the expectation at each node, then use those expectations to update parameters using EM-ITSBN outside the BNT.
    • X_index : The row vector containing the index of hidden nodes, i.e., X_index = 1:K.
    • The GMM segmentation results in
      • gmm_posterior (#node_in_H^1 x #class_in_H^1): The marginal posterior distribution
      • class_result (#node_in_H^1 x 1):The mpm class label
      • max_postr (#node_in_H^1 x 1): The value of mpm
    • The posterior gmm_posterior is inserted as "soft evidence" to the hidden nodes in level H^1. Note that the evidence Y is not explicitly connected to the nodes in H^1, instead, is converted into soft evidence for nodes in H^1 by GMM posterior. We can use this same trick for all nodes in every level of the ITSBN, which will save us a lot of memory. That is why we don't have any nodes for Y in the ITSBN structure when implemented using BNT.
  6. Calculate the cost function F
  7. Learning ITSBN parameter using EM algorithm
  8. When the algorithm converges or terminates, calculate the posterior marginal distribution of each hidden node, which is contained in the cell array posterior_marg.

Parameter tying:

In this version of code, we also add the capability of parameter tying in any level.

    • tie_level_table : A (L-1 x 2) matrix describing the association between the level H^l (column#1) and the tied parameterID (column#2). In the example, it means that the level H^1 and H^2 has its own parameter to use whereas level H^3, H^4 and H^5 share the same set of Gaussian parameters.
    • tie_parameter_table : A (#tied_paramID x 3). Col#1: unique tied parameterID, col#2 the number of class for each tie parameterID, col#3 type of covariance matrix for each parameter; 1 for 'full' and 2 for 'diagonal'.
  • C_tie_list : The number of class label for each tie parameter, for instance, C_tie_list=[7 5 3]
  • Mu_tie : A (#tied_paramID x 2) cell array. Column#1 is the tie parameter ID, column#2 is the mean vector for each class, i.e., mu is (C_t x D) where C_t is the number of class labels in that tie level t.
  • Lambda_tie :A (#tied_paramID x 2) cell array. Column#1 is the tie parameter ID, column#2 is the precision matrix for each class, i.e., Lambda is (D x D x C_t) where C_t is the number of class labels in that tie level t.

% ===== parameter tie table: =======

% col#1: level H^l

% col#2: tie parameterID

% for example, 3 3 ==>the level H^3 use the parameter#3

tie_level_table = [1 1

2 2

3 3

4 3

5 3];

% col#1: unique tie parameterID

% col#2: number of class in that

% col#3: type of covariance matrix: 1=full, 2= diagonal

% That means the tie parameter#3 has 3 classes, and they are diagonal

% covariance

tie_parameter_table = [1 7 1

2 5 1

3 3 2];

C_tie_list = tie_parameter_table(:,2)';

fn_pseudoDiag.m

a = fn_pseudoDiag(r,c)

Visualization Toolbox

The functions in the folder toolbox_visualization will be explained here.

fn_segment2boundary.m

detected_contour = fn_segment2boundary(segm_output, figure_toggle)

This code convert a segmentation output into a boundary image by differentiating the segments. The function will show the image when figure_toggle = 'on', and otherwise when 'off'.

fn_overlayGTORGBND.m

Iorg_gt_boundary = fn_overlayGTORGBND(Igray, Igt, Ibnd);

Overlay the original image, the groundtruth and the boundary on the same image. The original image can be either rgb or gray scale.

fn_visualizeITSBNgraph.m

function [] = fn_visualizeITSBNgraph(imagename, imageext)

The code calculate the centroid of each superpixel and show the boundaries of superpixels 2 levels at a time. There is no limitations in the number of the level L in this code.

fn_stackSuperpixelFeatureImage.m

[I_show] = fn_stackSuperpixelFeatureImage(index_field, feature_index, feature, display_feature)

Rearrange the feature vector matrix in cube

% #########################################################################

%

% OUTPUT:

% This code will create a stack of feature image Nrow x Ncol x feat_D. And

% if display_feature is determined, the functin will show figure too.

%

% INPUT:

% index_field = Nrow x Ncol matrix, each entry contains superpixel index in which the pixel resides.

% Also, each entry represents a pixel on the actual image.

% feature_index = a column vector containing superpixel indices in

% ascending order.

% feature = each row of this matrix means the feature vector for that

% particular superpixel.

% display_option = The feature that you want to plot, so the maximum length

% of this row vector is 3 since we have only R G B. if this is not determined,

% the function will not plot anything.

% ############################ EXAMPLE ####################################

% [I_show] = fn_stackSuperpixelFeatureImage(evidence_label, Y_index, Y);

% This option will just produce the feature stack cube, but not display the

% figure.

% [I_show] = fn_stackSuperpixelFeatureImage(evidence_label, Y_index, Y, [1 3 5]);

% This option also display the feature image too.

% #########################################################################

Texton Toolbox

The functions in the folder toolbox_ITSBN will be explained here.

Color Space Toolbox

The functions in the folder toolbox_ITSBN will be explained here.