Research Consistency Validation and Final Results

EXPERIMENT using APD:

We have taken the best result obtained for APD. Since our RMSE already showed vast improvement, we prioritized best-NDCG improvement test case to run this experiment.  We ran the experiment with the condition : W1+W2=1.0 and iterating W1: [0.0,1.0](with step size 0.1)

Best_Case: NDCG

(

Social Relationship Descriptor= All;

Group Decision Strategies= Max Satisfaction, Avg Satisfaction, Minimum Misery;

Expertise Descriptor=yes;

Dissimilarity Descriptor= APD; 

(W1, W2)=(0.8, 0.2);

(MF Type)=Default;

LR Schedule=Exponential Decay with Step Size

)

EXPERIMENT using VD:

We have taken the best result obtained for APD. Since our NDCG did not show any improvement, we prioritized best-RMSE improvement test case to run this experiment.  We ran the experiment with the condition : W1+W2=1.0 and iterating W1: [0.0,1.0](with step size 0.1)

Best_Case: RMSE

(

Social Relationship Descriptor= All;

Group Decision Strategies= Max Satisfaction, Avg Satisfaction, Minimum Misery;

Expertise Descriptor=yes;

Dissimilarity Descriptor= VD; 

(W1, W2)=(0.8, 0.2);

(MF Type)=Default;

LR Schedule=Exponential Decay 

)

OBSERVATIONS:

Optimal Values Obtained for (w1,w2) for each of the following cases:

1. APD: (0.9, 0.1)

2. VD: (0.9, 0.1) 

This is an improved parameter than baseline algorithms where the authors have used (0.8, 0.2)

Once we have found our optimal (W1, W2) pair, our next step is to ensure consistency in the improvement of results. The baseline algorithm was deterministic in nature because it had no randomness in it. Using Matrix Factorization, we have introduced randomness in our proposed method in the following 2 ways:

1.  We initialize the Latent Factors(P, Q) using randomness.  

2. We also shuffle our data to start Matrix Factorization Training using randomness. 

 Therefore, it is important to validate whether our result is consistent throughout or is it just a special case for a particular seed coincidentally.

For this, we have run the best case test cases with optimal (w1,w2)  on 50 seeds from [0,50).  Finally we calculated the mean of these 50 iterations to compare with baseline error values.

APD Results:

For the APD NDCG error, we can see that the predicted mean (over all seeds) is higher than the baseline algorithm's NDCG

For the APD RMSE error, we can see that the predicted mean (over all seeds) is lower than the baseline algorithm's RMSE.

VD Results:

For the VD NDCG error, we can see that the predicted mean (over all seeds) is  lower than the baseline algorithm's NDCG. Here we found no improvement.

For the VD RMSE error, we can see that the predicted mean (over all seeds) is lower than the baseline algorithm's RMSE

We have generated valid combinations and compared them against each other in the Datasets and Results Section

We have concluded the following optimal Pipelines:

1. Combined NDCG + RMSE for APD: Baseline Algorithm(APD) + Default Matrix Factorization + Learning Rate Scheduler using exponential decay with step size + (w1,w2)=(0.9,0.1)

2. Best Case RMSE for VD : Baseline Algorithm(VD) + Default Matrix Factorization + Learning Rate Scheduler using exponential decay + (w1,w2)=(0.9,0.1)

Factors contributing to optimality:

1. Combination of  Social Relationship + Group Decision Strategies + Weight Descriptor + Dissimilarity + Matrix Factorization

2. When using APD, we use Matrix Factorization with Learning Rate Scheduler and step size

3. When using VD, we use Matrix Factorization with Learning Rate Scheduler 

Factors hindering optimality:

1. Any combination except taking into account all factors (above instance)

2. Baseline-Included Matrix Factorization

3. When using VD, we use Matrix Factorization with Learning Rate Scheduler with step size