Validation

Summary: Expectation Shock methodology has been applied to a large number of project challenges including business and social issues, local and global community, anti-social behaviour and anti-bullying - engaging both adults and children. It has been externally validated by University of Essex (ethics, questionnaire, statistics), University College London (overall process and validity of results) and by doctoral thesis. In essence:



NB: The following statistical analysis was undertaken to validate the well-foundedness of this simple, 60-second tool. (This means we do NOT have to undertake the statistical analysis or indeed read the stats below ☹) 

------------------------------

Expectation Shock! is based on the doctoral thesis: Expectation Shock! 2005 The copyright of this thesis rests with the author and no quotation from it or information derived from it may be published without the prior written consent of the author. The thesis can be viewed at University College London.

In this activity, students were asked to explore social networking technology such as Facebook. After a ‘hands-on’ experience using the technology they were asked for their views on Reliability and Usability – and also Social Interaction and Fun. The measured Perception shift was obtained as follows. There were four questions of the following format (taking the example of question 1 on Reliability):

Q1 Reliability: including ability to use the service dependably, securely and accurately.

How important is it? (on a scale of 1: low 5: high)

What did you expect before using the technology? (on a scale of 1: low 5: high)

What did you find after using the technology? (on a scale of 1: low 5: high)

The mean values for the responses to each question and its three component parts were calculated, together with the Perception shift (P-E). The data for the 18 students are presented in columns 2,3,4 below in the following table. This is followed in column 5 by the Perception shift, i.e. the difference between the student Expectation before using the technology and the student Perception after using the technology (P-E):

 

Importance

Expectation 

Perception

Perception Shift (P-E) 0.7

Reliability

4.9

3.7

4.6

0.9

Usability

4.2

3.7

4.4

0.7

Social

4.2

3.8

4.3

0.5

Fun

4.3

3.8

4.6

0.8

Importance, Expectation and Perception: 18 students

The activity gave an opportunity to undertake further statistical analysis which supported the above findings. 

Significant Differences between Perception and Expectation Means for All Variables

One-sample t tests were undertaken which suggest that the differences between the Perception and Expectation means for all variables are significant (< 0.01).

Standard Deviations of Means & Dispersion 

The standard deviations of the mean values gave a perspective on dispersion, for example: The individual values for Reliability Importance are tightly grouped around the mean (Mean 4.9 SD 0.3). The individual values for Reliability Expectation are more dispersed (Mean 3.7 SD 0.9). The individual values for Reliability Perception are less dispersed (Mean 4.56 SD 0.598). However, this pattern is not repeated for the other variables. Examination of the table below shows that Expectation means are all lower and are more dispersed than Importance and Perception means. This might suggest the possibility that subjects are more diverse in their Expectation than their Importance or Perception.  

 

Minimum

Maximum

Mean

Std. Deviation

Reliability I

4

5

4.89

0.314

Rel E

2

5

3.72

0.931

Rel P

3

5

4.56

0.598

Usability I

2

5

4.22

0.711

Us E

2

5

3.72

0.870

Us P

2

5

4.44

0.831

Social I

4

5

4.22

0.416

Soc E

2

5

3.83

0.764

Soc P

2

5

4.33

0.745

Fun I

3

5

4.33

0.667

Fun E

2

5

3.83

0.764

Fun P

3

5

4.61

0.591

Standard Deviations of Mean

Correlation Coefficients

The following table shows the Pearson correlation coefficients for the dimensions of Importance, Expectation and Perception values of the 4 variables. The largest coefficients with high significance (less than 0.01) are highlighted. Subsets of this data are then used to examine the interesting features in a hopefully clear way



Rel I

Rel E

Rel P

Us I

Us E

Us P

Soc I

Soc E

Soc P

Fun I

Fun E

Fun P

Rel I

P Corr

1

-0.105

0.033

-0.138

0.294

-0.024

0.189

-0.309

-0.079

-0.088

0.386

0.067


Sig. 2-tail

.

0.667

0.894

0.573

0.222

0.924

0.438

0.199

0.748

0.719

0.103

0.787

Rel E

P Corr

-0.105

1

0.377

0.261

0.042

0.662

0.016

0.716

0.454

0.507

0.482

0.511


Sig. 2-tail

0.667

.

0.112

0.281

0.865

0.002

0.948

0.001

0.051

0.027

0.037

0.025

Rel P

P Corr

0.033

0.377

1

0.102

0.190

0.732

0.174

0.324

0.208

0.371

0.081

0.454


Sig. 2-tail

0.894

0.112

.

0.679

0.436

0.000

0.477

0.176

0.394

0.117

0.742

0.051

Us I

P Corr

-0.138

0.261

0.102

1

-0.170

0.115

-0.167

0.477

0.594

0.195

0.375

0.470


Sig. 2-tail

0.573

0.281

0.679

.

0.488

0.640

0.495

0.039

0.007

0.423

0.114

0.042

Us E

P Corr

0.294

0.042

0.190

-0.170

1

0.094

0.017

-0.153

-0.200

0.256

0.349

0.114


Sig. 2-tail

0.222

0.865

0.436

0.488

.

0.702

0.945

0.531

0.412

0.291

0.144

0.642

Us P

P Corr

-0.024

0.662

0.732

0.115

0.094

1

0.036

0.554

0.478

0.234

0.204

0.578


Sig. 2-tail

0.924

0.002

0.000

0.640

0.702

.

0.885

0.014

0.038

0.335

0.402

0.010

Soc I

P Corr

0.189

0.016

0.174

-0.167

0.017

0.036

1

-0.233

-0.239

0.134

-0.408

-0.101


Sig. 2-tail

0.438

0.948

0.477

0.495

0.945

0.885

.

0.336

0.324

0.585

0.083

0.682

Soc E

P Corr

-0.309

0.716

0.324

0.477

-0.153

0.554

-0.233

1

0.488

0.327

0.238

0.349


Sig. 2-tail

0.199

0.001

0.176

0.039

0.531

0.014

0.336

.

0.034

0.171

0.326

0.143

Soc P

P Corr

-0.079

0.454

0.208

0.594

-0.200

0.478

-0.239

0.488

1

-0.224

0.293

0.547


Sig. 2-tail

0.748

0.051

0.394

0.007

0.412

0.038

0.324

0.034

.

0.357

0.224

0.015

Fun I

P Corr

-0.088

0.507

0.371

0.195

0.256

0.234

0.134

0.327

-0.224

1

0.436

0.329


Sig. 2-tail

0.719

0.027

0.117

0.423

0.291

0.335

0.585

0.171

0.357

.

0.062

0.169

Fun E

P Corr

0.386

0.482

0.081

0.375

0.349

0.204

-0.408

0.238

0.293

0.436

1

0.595


Sig. 2-tail

0.103

0.037

0.742

0.114

0.144

0.402

0.083

0.326

0.224

0.062

.

0.007

Fun P

P Corr

0.067

0.511

0.454

0.470

0.114

0.578

-0.101

0.349

0.547

0.329

0.595

1


Sig. 2-tail

0.787

0.025

0.051

0.042

0.642

0.010

0.682

0.143

0.015

0.169

0.007

.


N

19

19

19

19

19

19

19

19

19

19

19

19

Pearson Correlation Coefficients for Importance, Expectation and Perception

Reducing the table to those rows and columns containing high correlation values with high significance (< 0.01) produces the following:



Us P

Soc E

Soc P

Fun E

Reliability E

Pearson Correlation

0.662

0.716

0.454

0.482


Sig. (2-tailed)

0.002

0.001

0.051

0.037

Reliability P

Pearson Correlation

0.732

0.324

0.208

0.081


Sig. (2-tailed)

0.000

0.176

0.394

0.742

Usability I

Pearson Correlation

0.115

0.477

0.594

0.375


Sig. (2-tailed)

0.640

0.039

0.007

0.114

Fun P

Pearson Correlation

0.578

0.349

0.547

0.595


Sig. (2-tailed)

0.010

0.143

0.015

0.007

High Correlations with High Significance

From the above table, it can be seen that the highest correlation is between the Reliability Perception and the Usability Perception  (0.732). Perhaps this might be due to an underlying variable relating the two or it might be a general Perception shift. This possibility will be returned to.

The next highest correlation is between the Reliability Expectation and the Social Interaction Expectation (0.716). Perhaps this might be due to an underlying variable relating to the Security and Privacy aspects of both Reliability and Social Interaction. However a similarly high correlation is not found in  the Perceptions of Reliability and Social Interaction: so this is inconclusive.

There were no high significant correlations in Importance between the variables: suggesting perhaps that students see these four areas as different with no large underlying variable.

Similarly, from the Expectation Correlation Coefficients table below, there were no high, significant correlations between Expectations, with the exception of the Reliability Expectation and the Social Interaction Expectation (0.716).

 


Rel E

Us E

Soc E

Fun E

Reliability E

Pearson Correlation

1

0.042

0.716

0.482

Usability E

Pearson Correlation

0.042

1

-0.153

0.349

Social E

Pearson Correlation

0.716

-0.153

1

0.238

Fun E

Pearson Correlation

0.482

0.349

0.238

1

Expectation Correlation Coefficients

In contrast, in the Perception Correlation Coefficients table below, there were more, higher significant correlations between Perceptions: thus lending support to the conclusion in the body of the report suggesting a general Perception shift.

 


Rel P

Us P

Soc P

Fun P

Reliability P

Pearson Correlation

1

0.732

0.208

0.454

Usability P

Pearson Correlation

0.732

1

0.478

0.578

Social P

Pearson Correlation

0.208

0.478

1

0.547

Fun P

Pearson Correlation

0.454

0.578

0.547

1

Perception Correlation Coefficients