2.3.1 Uncertainty in experimental evidence

Systematic errors versus random errors

A figure showing the difference between random and systematic errors: https://www.physics.umd.edu/courses/Phys276/Hill/Information/Notes/ErrorAnalysis.html

Systematic errors

Two types:

• Offset uncertainty - all measurements are larger or smaller than the "true" value by a constant amount.
  • For example: a thermometer is not in sufficiently good thermal contact with a hot object - the readings are all lower than the "true" temperature of the object. See this paper for an example of a systematic offset: https://fathomingphysics.nsw.edu.au/wp-content/uploads/2017/07/TEHumphrey_VCalisa_Phys_Teach_vol_52_iss_3_142_2014.pdf
• Gain uncertainty - measurements are larger or smaller than the "true" value by a fixed percentage.
  • For example: A long measuring tape is stretched so that all 1cm markings on the ruler are now actually separated by 1.01cm. Each reading will give a value that is 1% higher than the "true" value.

Random errors

• Random errors correspond to the "scatter" in experimental data, and will result in readings that are scattered around the "true" value, usually with a normal distribution. The data in the paper we looked at earlier (https://fathomingphysics.nsw.edu.au/wp-content/uploads/2017/07/TEHumphrey_VCalisa_Phys_Teach_vol_52_iss_3_142_2014.pdf) also shows significant random error, with data points distributed above and below the line of best fit.

Resources:

Books

I would recommend Experimental Methods by Les Kirkup - however it is currently out of print (a new edition is due out in 2019).

Another option is "An introduction to Error Analysis (2nd Ed.)" by J.R. Taylor (University Science books, 1997).

Resources on the web:

Here's a short into: http://phys.columbia.edu/~tutorial/index.html

A bit more detail: https://courses.washington.edu/phys431/uncertainty_notes.pdf

https://www2.southeastern.edu/Academics/Faculty/rallain/plab194/error.html

2.3.2 Assess and evaluate the use of errors in:

Mathematical calculations involving degrees of uncertainty

Method 1. Using upper and lower bounds (the "straightforward" approach)

• If you need to find the uncertainty in a quantity that is a function (say sin(x)) of a variable with a known uncertainty, then you can simply calculate the upper and lower limits to the function, based on the upper and lower limits of the variable.

Method 2. Using partial differentiation (see Pg. 72 of Kirkup or http://www.webassign.net/question_assets/unccolphysmechl1/measurements/manual.html )

Standard error in the mean

Activity to help understand the standard error in the mean: https://docs.google.com/document/d/1LAkR_R84wS5AkXokgxY7QxbwwG0a__7OuudsEdy15cI/edit?usp=sharing

Graphical representations from curves of best fit

We will do this activity: https://docs.google.com/document/d/1o--pnwKryfJN81ebdgv9Hz8mmzQf7UmIF3NdOvXx-A4/edit?usp=sharing

2.3.3 Compare quantitative and qualitative research methods

Richard Feynman: “The first principle is that you must not fool yourself – and you are the easiest person to fool.”

Definitions of quantitative and qualitative research methods:

Qualitative research: This can be described as research that cannot easily be communicated or understood in numerical terms. It usually involves open ended questions and responses (such as interview questions or case studies).

Quantitative research: Is an an approach for testing objective theories by examining the relationship among variables. The variables can be measured and analysed numerically.

Mixed methods research: Contains elements of both types of research.

Some common elements in scientific investigations:

Different types of scientific inquiry share some common elements:

• a commitment to deductive testing (i.e. the idea that experimental/observational evidence determines if a theory can be accepted)
• an experimental design that protects against bias
• a consideration of alternative explanations of the results
• the interpretation of data (either qualitative or quantitative) to produce results that are reproducible and generalisable
• a discussion of how the research relates to other work done in that field

Resources:

• An overview of the value of quantitative and qualitative research in the context of the epidemiology of influenza: https://theconversation.com/when-the-numbers-arent-enough-how-different-data-work-together-in-research-103907 and associated papers: https://www.researchgate.net/publication/323155120_Much_ado_about_flu_A_mixed_methods_study_of_parental_perceptions_trust_and_information_seeking_in_a_pandemic and https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5304570/pdf/IRV-11-102.pdf
• Regression to the mean - important to take into account in designing methodology. Veritasium video: https://www.youtube.com/watch?v=1tSqSMOyNFE&pbjreload=10

2.3.4 Investigate the various methods that can be used to obtain large data sets

Remote Sensing

Remote sensing is obtaining information about an area or phenomenon through a device that does not touch the area or phenomenon under study.

Passive remote sensors detects natural energy that is reflected or emitted from an observed object or scene (most commonly, reflected sunlight). For example, a camera or a spectrometer (or your eyes!).

Active remote sensors provide their own energy (electromagnetic radiation) to illuminate the object or scene they are observing, and then detect the radiation that is reflected or backscattered from that object. For example, Radar (Radio detection and ranging) or Lidar (Light detection and ranging) instruments.

Many remote sensing devices are on-board satellites that monitor the Earth from space. Here is a detailed list of current NASA remote sensing instruments: https://earthdata.nasa.gov/user-resources/remote-sensors.

Streamed Data

Streamed data: There are many devices now used in research which can operate in an autonomous or semi autonomous mode in which data is recorded continuously and then "streamed" out of the sensor for further processing and analysis (http://streamingsystems.org/finalreport.html )

While this technology has opened new opportunities in research, it has also brought challenges. With advances in information technology it has become possible to record very large amounts of data in a short time. In some cases, e.g. the SKA discussed below, it is necessary to process the data in real time to reduce the amount of data that is placed in long term storage. In other systems the constraints may be on the communication link from the instrument. For example the Kepler telescope was situated in a sun centered orbit and had a limited capacity (once a month only) radio link back to earth (see: https://www.nasa.gov/mission_pages/kepler/spacecraft/index.html). Similar considerations apply in sensor networks where the communication back to base is limited by the amount of power required.

Other examples, as identified in http://streamingsystems.org/finalreport.html, include: Internet of People (consisting of wearable devices), Social media, financial transactions, Industrial Internet of Things, (requiring real-time responses) Cyberphysical Systems, Satellite and airborne monitors, National Security, Astronomy, Light Sources, and Instruments like the LHC, Sequencers (all involving large volumes of data), Data Assimilation (where there is sensitivity to latency), Analysis of Simulation Results, Steering and Control.

Case study - SKA: The planned square kilometer array telescope will use over 100,000 antennas, each producing tens of megabytes of data per second. Ideally the raw data would be stored so that it can be analysed later in different ways, but the cost of storing this data is prohibitive. As a result methods of real time analysis have been developed to cut the data rate to a much lower rate. See: https://www.skatelescope.org/newsandmedia/outreachandeducation/skawow/big-data/ and https://www.skatelescope.org/signal-processing/ , https://www.skatelescope.org/sdp/

Case studies in scientific methodology

We will first examine the historical development and characteristics of one particular type of research, the "large scale double blind randomised placebo controlled trial" via this document: https://docs.google.com/document/d/1X3oNAGdprMZ6pDG_Xc10RCOhO5fsyZDkKO_c4MmYOG0/edit?usp=sharing as it is such a well-known approach to research in fields such as medicine and psychology. We will then look at a range of other types of research through a number of "case studies" of particular scientific papers. We ask many of the following questions to help us examine the methodology followed in different fields of science:

• What has been investigated and why is this interesting and/or significant?
• Does this research fit easily into the framework of types of scientific research we have explored? I.e. qualitative vs quantitative vs mixed methods research? Is it useful to think of it an example of fair-testing, pattern-seeking, exploring, classifying and identifying, making things or developing systems or investigating models? (the categorisation provided in https://14254.stem.org.uk/Beyond_Fair_Testing.pdf)
• What methodology has been used by the authors to obtain valid and reliable data? (This is our inquiry question for this section!). Specifically:
  • how has bias been addressed and minimised?
  • what process has been followed to gather data? (have the techniques of remote sensing or streaming of data been used?)
  • what techniques have been used to minimise and measure uncertainty in the data?
  • how has the data been analysed?
  • have alternative explanations of the results been considered?
  • do the results have some level of general applicability?
  • has the research been reported in a way that it allow other scientists to replicate the results?

Some journal articles that we will look at for this section:

A mixed methods study (educational psychology):

• Adrian F. Ward, Kristen Duke, Ayelet Gneezy, and Maarten W. Bos, "Brain Drain: The Mere Presence of One’s Own Smartphone Reduces Available Cognitive Capacity," Journal of the Association for Consumer Research 2, no. 2 (April 2017): 140-154. https://doi.org/10.1086/691462. Background information that will help make sense of this paper is provided by this veritasium video "The Science of Thinking", : https://www.youtube.com/watch?v=UBVV8pch1dM

A longitudinal study (teaspoonology...):

• The case of the disappearing teaspoons: longitudinal cohort study of the displacement of teaspoons in an Australian research institute BMJ 2005; 331 doi: https://doi.org/10.1136/bmj.331.7531.1498 (Published 22 December 2005)

A review article (educational psychology - what is the most effective way to study?):

• Improving Students’ Learning With Effective Learning Techniques: Promising Directions From Cognitive and Educational Psychology http://www.indiana.edu/~pcl/rgoldsto/courses/dunloskyimprovinglearning.pdf

Nature (neural networks in medical diagnosis)

• "Dermatologist-level classification of skin cancer with deep neural networks" Andre Esteva et al. Nature volume 542, pages115–118 (02 February 2017) https://www.nature.com/articles/nature21056 (fulltext: https://cs.stanford.edu/people/esteva/home/assets/nature_skincancer.pdf).

Nature (Remote sensing/physics/astronomy)

• "Seven temperate terrestrial planets around the nearby ultracool dwarf star TRAPPIST-1" Michaël Gillon et al. Nature volume542, pages456–460 (23 February 2017) https://www.nature.com/articles/nature21360 (fulltext: https://arxiv.org/pdf/1703.01424.pdf)

Nature methods (Storing and accessing "big data" in neurology)

• A community-developed open-source computational ecosystem for big neuro data Joshua T. Vogelstein, Eric Perlman, Randal Burns Nature Methods, volume 15, pg. 846–847 (2018). https://www.nature.com/articles/s41592-018-0181-1 (fulltext: http://arxiv.org/pdf/1804.02835)

Other resources that might be useful:

What is scientific research? What makes good scientific research

A useful series of articles from The Conversation's Australian Science & Technology Editor, Tim Dean, on how they make editorial decisions about the science they report

• https://theconversation.com/how-we-edit-science-part-1-the-scientific-method-74521
• https://theconversation.com/how-we-edit-science-part-2-significance-testing-p-hacking-and-peer-review-74547
• https://theconversation.com/how-we-edit-science-part-3-impact-curiosity-and-red-flags-74548 (mostly useful for your own assessments of reliability)
• https://theconversation.com/how-we-edit-science-part-5-so-what-is-science-74550

‘Beyond Fair Testing’ is a resource for teaching different types of scientific enquiry, based on the work of the SKEES (SEP-King’s Enhancing Enquiries in Schools) project. It identifies several different types of scientific inquiries: https://14254.stem.org.uk/Beyond_Fair_Testing.pdf.

A manifesto for repeatable science (applicable to medical trials or other studies with many confounding variables): https://www.nature.com/articles/s41562-016-0021#t1

Tool for assisting in evaluating the design of studies involving the use of animals: https://eda.nc3rs.org.uk/

Charles Babbage (1830) "REFLECTIONS ON THE DECLINE OF SCIENCE IN ENGLAND, AND ON SOME OF ITS CAUSES"

Includes section on types of scientific misconduct (fraud, hoaxing, cooking, trimming...): http://www.gutenberg.org/files/1216/1216-h/1216-h.htm#link2H_SECT17