Provide an overview of the session and the learning objectives

This Unit is all about how to identify credible information and evidence. 

By the end of this week they should be able to understand:

• What constitutes credible information

• How to identify credible information

• The peer-review process

• Whether a source of information is appropriate  

To provide some context for this session you can note that throughout their time at university they are going to be required to read and assess large quantities of information (which they should have started doing) and assess whether or not they are credible. This will be particularly important for their assessments.

Once you have heard their thoughts you can then discuss what separates these sources in terms of quality (anecdotal and empirical evidence) and focus on some key factors they will want to consider when assessing the credibility of information which moves us onto our next point.  

Regardless of whether you have done the above exercise it is worth discussing with them the following general principles they should consider when assessing the validity of information sources. 

Anecdotal or empirical?

You can note the key difference between anecdotal and empirical evidence to get the importance of this point across (if you have done the above exercise it should follow naturally).

Anecdotal information: refers to individual stories or personal experiences that are not supported by research or statistical evidence (e.g., a friend retelling a ghost story).

Empirical information: refers to evidence collected through scientific experimentation and observation and is thus considered acceptable scientific information to use (such as journal articles).

This does not mean anecdotal information is never true and or perhaps useful information. However, you can note this type of information or evidence would not be sufficient to build an academic argument on. To get this point across you could get them to complete the following exercise.

Suggested exercise: The curious case of water dowsing (takes approx. 10 mins)

What to do? Ask them to watch the following video: https://www.youtube.com/watch?v=_VAasVXtCOI 

Once they watched the video you may wish to ask them:

(1) why anecdotal sources are not appropriate evidence to use in science?

• Do no hold up to scientific scrutiny (as demonstrated in clip above)

• Survivorship bias – people focus on the successes/interesting cases  

(2) why anecdotal sources might be dangerous?

• Unfounded treatments and interventions may be used on minimal evidence which cause harm and waste resource

Another point they will want them to consider is whether a source is primary or secondary information. They will typically have experience of using textbooks if they have come from A-level/college so this is a habit we will be wanting to get them out of.  

You can note the distinction between the sources by giving some examples. For instance, you could ask them if they are aware of any famous psychology studies such as the Little Albert experiment, Zimbardo etc. and ask where they read about them. If they were from anything other than the published paper written by the author, then it is not a primary source. If, however, they have read the original papers (which typically few have) this would be a primary source.

Following on from this, you can also note why it is important they use primary sources where possible. You can give an example of this and tie it in with key textbooks recommended/included on the course.

A good textbook will contain lots of primary sources in the form of journal articles, which will help give them a good understanding of the topic and material. However, to get further detail about some of the studies which are cited they would need to refer to the studies themselves. It may be that they want to find out how big the sample was or what the limitations were, and this wasn’t necessarily covered in the textbook as there was no room for further information to be covered.

They likely won’t know what the peer-review process is or not understand it fully so it is worth talking them through this and why it is so important to academia.

There is a nice video includefrom NC State University which explains the process: https://www.youtube.com/watch?v=rOCQZ7QnoN0 

That is, that any novel science which is communicated to the scientific community of interest (by being published in a relevant science journal), is assessed by several experts in the field who consider whether the work is scientifically robust enough to be published and meaningfully contributes to the field of knowledge which has accumulated in the respective area.

You could also briefly discuss the problems with the system and how it can be susceptible to issues - best not to rip it completely apart though as they are just learning about it and the goal is to get across that peer-review helps ensure information is credible (…now might not be the time to mention that Reviewer 2 was six months late and didn’t read the paper).

Suggested discussion point:

If you are feeling brave and have time you could briefly go over the process of a review you have written (why you were invited, what you were asked to do, what the editor decided etc.) or received (try not to get angry!) to give them a sense of the process as it will likely seem quite abstract to them.

To help explain this to them, you can talk through the different types of information they may come across by using the hierarchy of credibility and flesh out each stage with an example.

For instance, you could talk about:

• word of mouth being most dangerous hence being placed at the bottom (you could make a link to “fake news”);

• unreviewed sources such as websites like Wikipedia being next in line (as they can be produced by anyone);

• editorially reviewed sources (those looked over by editor to see whether it suits their content and style) from respected outlets (e.g., the financial times) being more credible

• then peer-reviewed sources such as academic journals articles being top of the pile as they are reviewed by experts (most of the time!).

You may have other ways you wish to do this such as just listing specific types of sources and ranking them (e.g., websites down the bottom, books next and papers at the top). The difficulty with this is that websites and other sources can be such a grey area and often students have never even seen a physical journal and just think of it as another website.

Suggested (discussion point: Grey literature

Talking of grey areas, you could go over with them the idea of grey literature (literature/sources which were not created for the purpose of being published but end up on the web or elsewhere). You can note some examples, emails, newsletters, government reports, student dissertations or theses, conference papers etc. They are likely to come across grey literature if using Google/Google Scholar, less so if using university library database.

Worth noting that these can vary drastically in quality. It could be that the reader gets an insight into upcoming research before it is published or that they see erroneous information in a government letter which was subsequently updated. As such, it is best to approach with caution. We will look at some general principles they can use to consider if it is appropriate a little later.

Given the hierarchy of credibility discussed above you can note that this will primarily leave them spending a lot of time looking at peer-reviewed journal articles throughout their degree. This is also the type of evidence they should be looking to include in assessments.

It is worth noting that journal articles themselves can vary though. Some may be case studies focusing on individuals which are useful for looking at anomalies (they will probably have heard about Phineas Gage or Little Albert if they have studied psych before), some will consist of one or several experiments which can vary from observational studies to RCTs, whilst other will evaluate lots of scientific articles on a specific topic such as meta-analyses and systematic reviews.

You can note that these reviews of large bodies of literature can be incredibly useful for them when looking for literature and/or if they have been asked to evaluate the effectiveness of something in an essay.

Suggested discussion point: 

If you want to turn this into a discussion point you could get them to consider why systematic reviews and meta-analyses would be particularly useful and important to scientists?

Key points being: they provide an opportunity to consider lots of different studies and samples, consider various factors which influence an effect (i.e., moderating variables), they provide an estimate of how much of an effect something has (i.e., effect size), how reliable the effect is and (hopefully) they provide an up to date view on the state of the science on a particular topic.

Whilst journal articles will be considered the most appropriate scientific form of evidence, you can note that they are of course not infallible and there are factors that they will want to consider when looking at articles (or indeed any sources). These include, whether the sources is current, relevant to the argument being made, written by an authority, accurate with its information, and what its purpose is.

In short, to use a handy acronym, we want to consider whether the source is CRAAP!* You can run through this acronym with them to help them consider these factors when searching for any type of evidence

• Currency – how current is the source?

• Relevance – how relevant is the source to what they are searching for?

• Authority – is the source qualified to talk on the subject?

• Accurate – is the information supported by reliable evidence? Does it cite other appropriate sources?

• Purpose – what is the reason the information exists, trying to sell something or funded by private organisation

You can give some examples to flesh it out if you wish and note that this is just a heuristic tool to pose questions worth considering and not a formula which will reveal the perfect source. For example, with seminal studies they will be unlikely to be current. However, they can certainly check on the current state of the science within the field to see whether the findings of the seminal source are still valid.

The currency point sometimes produces questions as to what timeframe a source should fall within to be considered current. Worth stressing that it is all relative to field but a general rule of thumb they could use is within the last 5 years. This doesn’t mean all their sources have to have been within the last 5 years but that they should strive to find provide more up to date sources in their work.

There is a short video which looks at the CRAAP method from Western Universities which you can get them to watch:

https://www.youtube.com/watch?v=EyMT08mD7Ds&t=3s 

*  I don't know who came up with this mnemonic if you are the person or know who is please let me know

It is also worth taking some time to focus specifically on websites as this seems to be an area students struggle with (e.g., relying on simplypsychology, tutor2u.net and Wikipedia). They should not be citing these sources in their writing.  

To demonstrate why these sources may be problematic you could turn this into a discussion point and get them to consider why Wikipedia would fall short using the CRAAP method (primarily because of authority and accuracy, anyone can modify the entries on there).

Additionally, it is also worth noting that they can get some handy insights about a site based on its URL by focusing on the domain name. Some official domain names they will encounter include:

·       .com – which represents a commercial company

·       .co.uk – a commercial UK company​​

·       .ac.uk – a UK academic institution

·       .edu – a US academic institution

·       .nhs.uk - the NHS

·       .org – a non-profit organisation (e.g., a charity)

·       .gov or .gov.uk – a US or UK government site

Just because information may come from one of these official domains, this does not mean the source is reliable though. Students should take particular care when finding information on advertising links, political pieces, personal opinions pieces (e.g., blogs) and personal websites.

By the end of unit they should now know:

• What constitutes credible information

• How to identify credible information

• The peer-review process

• Whether a source of information is appropriate  

• For further reading about the problems with anecdotal evidence, in the context of medicine, have a read here: https://www.ncbi.nlm.nih.gov/books/NBK63643/

• Further information about the steps of the peer-review process: https://authorservices.wiley.com/Reviewers/journal-reviewers/what-is-peer-review/the-peer-    review-process.html

• Further information about the CRAAP test: https://learn.library.ryerson.ca/Research/evaluate