Results and recommendations

A systematic survey of publicly funded projects in the SE domain was undertaken using OAM.

Notice of restricted circulation

The data sets  linked below are being circulated only to grant holders in relation to the projects audited and are not yet in the public domain. During this consultation period with the target community, please do not share them with anyone yet, but if you wish please provide feedback using the link at the bottom of this pageThey will be made publicly available after the study is completed. Thank you for the consideration.

About OAM

OAM takes pragmatic approach to systematically review open access practices 'from the bottom up', 

read more about the OAM  Process and templates

The Audited Portfolio

A total of 100 EPSRC funded projects which ended in 2009 and 2010 has been audited and 'scored' using the OAM heuristic indicators,

Search key terms:

Auditors were asked to search using standard search engines, typically google/yahoo

In the first round of audits, auditors were asked to search using any combination of key words they wanted regarding the funded project 

(grant number, project title, other project related keywords, PI name)

In the second and subsequent rounds, to achieve higher 'similarity' between auditors, search was standardized in 3 steps: search for grant numb er, project title, PI name, then the three combined in a string

A copy of a public version of our initial data sheet is available to grant holders (please do not redistribute now, this research is still in progress)

 is available  here.

The outcomes are summarized in the visualization below: 

Outcome of the first round of audits

For validation purposes, where the outcome was not validated by the grant holder via email (step 3 of the OAM Process) the portfolio was reaudited independently

(87 out of 100 projects were reaudited)

As the outcomes of the audits were validated, we have also improved the auditing process and templates, so the

second data sheet is a bit more detailed (includes URIs of the audited resources where avaialable)

See the dataset here

The outcomes are summarized in the visualization below:

.

OTHER INTERESTING OBSERVATIONS: 

 The difference between the first set of audits and the second set of audits has been analyzed qualitative, revealing that

- some of the differences were caused by genuine human error (this is almost certain  for example when projects had obtained a high score in the first round, and a very low score in the second round, approx 6 out of 100)

- some of the other differences could have been caused by 

a) error as above

b) differences in search criteria

c) differences in perception or interpretation of the scoring heuristics between auditors (where slightly two different values were assigned by different auditors)

d) the researcher, after receiving our first email informing them that we were about to audit their projects for open access resources, have actually published a bit more stuff, but have not responded to our emails to inform us 

Hypothesis b) has been confirmed by further tests (search criteria were made consistent and the audits score had higher similarity, with a max difference between

auditors of <3)

The hypotheses c) and d) will be tested in future work, with different datasets.

PRELIMINARY CONCLUSIONS 

From the Audit results:

Additional datasets with some variance in the inclusion criteria are being gathered to permit further analysis, however from the current findings, the following conclusions can be drawn:

1) The majority of projects (just above 50%)) audited did not have any, or very few resources that could be located, retrieved and accessed openly on the web. (score <3)

2)  The third largest segment of the audited population (approx 11%) adheres to all good practices and knowledge sharing conventions  and scores very high (score >14)

3) The minority of projects audited that adopt standard knowledge sharing practices (the notable exceptions) do so consistently and in compliance with good practices. Currently they are being used as a 'model of good practice', and studied more closely, to gather additional insights into outstanding Knowledge Sharing behavior.

From  surveying the knowledge sharing policies landscape

Work in progress, Summarised here

RECOMMENDATIONS

ADD YOUR COMMENTS AND ADDITIONAL RECOMMENDATIONS USING THE FEEDBACK FORM PROVIDED, THANKS