Version 3.0.1

There are several folders which can be categorized into 2 groups: code folder and output folder.

code folders:

output folder:

How to run it?

Go to toolbox_ITSBN and open the file fn_ITSBNIMageSegm2.m. There you can change the features in the section marked official feature experiment.
Go to the folder main_code and open and run main_ITSBNImageSegm.m. This code is a main code that calls all the functions necessary for image segmentation. The code can process all images in a folder at once.
The segmentation results are stored in the folder Results. Other outputs are also found in the output folder table. The details of the segmentation step by step are stored in each folder separately in the folder main_code.

main_ITSBNImageSegm.m

This is the main function to run ITSBN. The code calls other functions necessary for image segmentation. Normally there are 2 modes to run the code:

one-by-one mode: Each image in the database is processed and gives the segmentation one by one from first to last. This mode is suitable when having small number of images in the database.
experiment mode: Each process is applied to all the images in the database at once, then move on to the next process. For instance, superpixelize all the images in the database at once, and all the superpixel output are collected in the folder superpixels. An advantage is that we can see the behavior of the parameters on all the images before moving on to the next step. The user can comment all except the process you want to run one at a time.

All the processed data (e.g., superpixels, feature vectors, ITSBN tree) can be stored for future experiments in case that the environment of superpixel, tree is needed to be controlled. The code does the following jobs:

addpath to necessary folders and install toolbox
Retrieve the image file name in the image database folder determined in the previous process
Select a superpixel algorithm (quickshift or Mori's), then determine the size of superpixel in each level.
Plot the ground truth segmentation overlaid on the original image.
Construct ITSBN tree from the superpixels
Extract the features from each superpixel
Image segmentation using GMM and ITSBN

ITSBN Toolbox

The functions in the folder toolbox_ITSBN will be explained here.

fn_SuperpixelMori2.m

function [] = fn_SuperpixelMori2(imagename, imageext, N_l4, N_l3, N_l2, N_l1)

This is the function to make superpixels in each level. The level here is fixed to 4. The output of the file is cell array img_label_hier whose format are

each column:

column#1: the desired number of superpixel input by the user
column#2: the output number of superpixel determined by the superpixel algorithm. Note that this number might not be the same as the desired number. #L
- column#3: the superpixel indices assigned for each pixel. For instance, all pixels in the superpixel#7 region, will be assigned with 7. The index of each level always runs from 1 to #superpixel in the level. Each level look like the figure below

each row:

The first row: is for the label level (level H^1) N_l1
The second row: for level H^2, N_l2
The last row is the second to top level H^(L-2). Note that we know already that the root is in the level L-1 and has only 1 superpixel on the level, so we don't store the root information here. N_l4

For instance, the matrix looks like this.

The cell array img_label_hier is stored in the file imagename_img_label_hier.mat

fn_makeTreeFromLabel.m

[] = fn_makeTreeFromLabel(imagename)

The code take as an input the matrix img_label_hier, then output the followings:

Z : the structure (adjacency) matrix whose row and column represent child and parent respectively. The size of the matrix is (#nodes x #nodes +1), for instance, in this case we will have (306 x 307) structure matrix Z. The reason behind #nodes +1 is the phantom node, but this information in node used yet in this version of code.
n_level : # of level from H^(L-2) to H^(1), in this case n_level = 4.
H : (L+1 x 2) cell array. In this case, L = 6 (i.e., Evid, H^1, ...,H^5 ), however, we include the phantom node (in level H^6), marked by the biggest node index in the tree, in the level H{6+1,:} as well. The first column contains the level# H{level+1,1} = level; % level H[level]. That is, the information of the level H^l can be found from H{l+1,:}. The second column contains the node indices of ITSBN in the level. The cell array H is for instance shown as the table below. Note the first row of H was supposed to contain the information of the evidence level (H^0), however, I change my mind because of the idea of having evidence in every level.

node_label_hier : A (L-2 x 3) cell array, very similar structure to img_label_hier except that the third column of node_label_hier stores the node indices as oppose to the superpixel indices in img_label_hier.Note that the matrix

All the outputs from the code are stored in imagename_package.mat.

fn_imageFeatureExtraction2.m

[] = fn_imageFeatureExtraction2(imagename, imageext)

All the feature vectors are calculated from this code. The code take the imagename_package.mat as input and store feature vectors in the file imagename_feature.mat. The output matrices are

Y : A (#superpixel x D_feature) matrix. Each row represents the extracted feature vector from a superpixel in level H^1. Each column represents the features determined in the code.

Y_index : A (#superpixel x 1) column vector containing the node#ID associating with the feature vector in Y. The table below depicts the matrix Y_index.

This code works as follows:

Apply the filters (sRGB, sLuv, DWT, Texton) on the raw image first to get the pixel level features
Average the feature within a superpixel
Put the averaged vector in each row of Y

Note that we have the evidence only in the level H^0 only. In the next version, we will make multiscale evidence.

fn_imgSegmentationGMM.m

function [I_label, I_post] = fn_imgSegmentationGMM(imagename, imageext, C, W)

The algorithm uses GMM to segment the raw data in pixel level, and the features are Lab and location x, y of the image.

fn_ITSBNImageSegm2.m

[] = fn_ITSBNImageSegm2(filename_input,...

imagename, imageext,...

C,...

iteration_max,...

exp_number,...

cpt_spread_factor,...

added_diag_cpt,...

epsilon_F)

This code, regarded as the core of the ITSBN, takes as input the image and all the features extracted from the previous process, then output the segmentation results in imagename_ITSBN_segm_result.mat. The results are composed of

segs : A (L-2 x 1) cell array containing pixel-level segmentation results for each level.

posterior_marg : A (L x 4) cell array containing the maximum posterior marginal information in all the hidden level (H^1 to H^(L-1)).

Since we are using the BNET MATLAB toolbox, we have some tricks to combine the toolbox with our method seamlessly. The code is explained in detail as follows:

Select the features to use in the experiment by going to the section saying "Pick FEATURES to use in the segmentation", then you can pick the columns of Y corresponding to the features you want.
In the section saying "Prepare the number of levels and points in the dataset" we prepare some information about the number of nodes and levels necessary for the remaining code.
- N : The number of superpixel in the level H^1, i.e., in this case N = 200
- L : The total number of level from H^0 to H^(L-1), i.e., in this case L = 6
- K : The total number of hidden nodes in the ITSBN from H^1 to H^(L-1) excluding the phantom node, i.e., in this case K = 1 + 5 + 20 + 80 + 200 = 306
Fit the GMM to the feature vectors Y, here we will get the mu_c, Sigma_c for each class label c.
Initialize conditional probability table (CPT) for the ITSBN. The (L x 2) cell array CPT is composed of
- column#1: store the number indicating the level l. You may figure out that CPT{l+1,1} = l, which makes the notion of col#1 seem redundant, but we still store l there for our convenient.
- column#2: store the CPT matrix of that level. That is, CPT{l+1,2} is the CPT of the level l as the CPT is shared among all nodes inthe same level. Note that there is no restrictions of the dimension of the CPT, so it means we can make any arbitrary dimension as we wish. For example, we can CPT for level l and l+1 to be r x s and s x t respectively.
In this step, we use BNT to do the inference for the posterior marginal distribution at each hidden node. Here we use Bayes-Net Toolbox (BNT) to infer the hidden nodes. However, BNET does not fully support our parameter learning for ITSBN, so we use BNT just to calculate the expectation at each node, then use those expectations to update parameters using EM-ITSBN outside the BNT.
- X_index : The row vector containing the index of hidden nodes, i.e., X_index = 1:K.
- The GMM segmentation results in
  - gmm_posterior (#node_in_H^1 x #class_in_H^1): The marginal posterior distribution
  - class_result (#node_in_H^1 x 1):The mpm class label
  - max_postr (#node_in_H^1 x 1): The value of mpm
- The posterior gmm_posterior is inserted as "soft evidence" to the hidden nodes in level H^1. Note that the evidence Y is not explicitly connected to the nodes in H^1, instead, is converted into soft evidence for nodes in H^1 by GMM posterior. We can use this same trick for all nodes in every level of the ITSBN, which will save us a lot of memory. That is why we don't have any nodes for Y in the ITSBN structure when implemented using BNT.
Calculate the cost function F
Learning ITSBN parameter using EM algorithm
When the algorithm converges or terminates, calculate the posterior marginal distribution of each hidden node, which is contained in the cell array posterior_marg.

fn_pseudoDiag.m

a = fn_pseudoDiag(r,c)

Visualization Toolbox

The functions in the folder toolbox_visualization will be explained here.

fn_segment2boundary.m

detected_contour = fn_segment2boundary(segm_output, figure_toggle)

This code convert a segmentation output into a boundary image by differentiating the segments. The function will show the image when figure_toggle = 'on', and otherwise when 'off'.

fn_overlayGTORGBND.m

Iorg_gt_boundary = fn_overlayGTORGBND(Igray, Igt, Ibnd);

Overlay the original image, the groundtruth and the boundary on the same image. The original image can be either rgb or gray scale.

fn_visualizeITSBNgraph.m

function [] = fn_visualizeITSBNgraph(imagename, imageext)

The code calculate the centroid of each superpixel and show the boundaries of superpixels 2 levels at a time. There is no limitations in the number of the level L in this code.

fn_stackSuperpixelFeatureImage.m

[I_show] = fn_stackSuperpixelFeatureImage(index_field, feature_index, feature, display_feature)

Rearrange the feature vector matrix in cube

% #########################################################################

% OUTPUT:

% This code will create a stack of feature image Nrow x Ncol x feat_D. And

% if display_feature is determined, the functin will show figure too.

% INPUT:

% index_field = Nrow x Ncol matrix, each entry contains superpixel index in which the pixel resides.

% Also, each entry represents a pixel on the actual image.

% feature_index = a column vector containing superpixel indices in

% ascending order.

% feature = each row of this matrix means the feature vector for that

% particular superpixel.

% display_option = The feature that you want to plot, so the maximum length

% of this row vector is 3 since we have only R G B. if this is not determined,

% the function will not plot anything.

% ############################ EXAMPLE ####################################

% [I_show] = fn_stackSuperpixelFeatureImage(evidence_label, Y_index, Y);

% This option will just produce the feature stack cube, but not display the

% figure.

% [I_show] = fn_stackSuperpixelFeatureImage(evidence_label, Y_index, Y, [1 3 5]);

% This option also display the feature image too.

% #########################################################################

Texton Toolbox

The functions in the folder toolbox_ITSBN will be explained here.

Color Space Toolbox

The functions in the folder toolbox_ITSBN will be explained here.

Google Sites

Report abuse