Outliers - the problem children of data.

“Treat outliers like children … correct them, when necessary, but never throw them out.”

- Ed Gilroy, former Statistician at USGS

What is an outlier?

An outlier is a data point that is not like the others. Do you remember the Sesame Street song “One of these things is not like the others”?  They are outside of what we expect.  They may be much higher or lower than the other data or indicate a sudden change in a trend.  They stand out when compared to historical results or a group of results that should be from the same population.

What causes outliers?

There can be several causes for outliers but I have found the following four are the most common sources.

• Errors in measurements or recording of measurements.  For example, recording my height as 9’ 5” instead of 5’ 9”.  Being Robert Wadlow was the tallest man in recorded history at 8’ 11” the measurement of me at 9’ 5” is an obvious error.  My experience has been this is the most common source of outliers.

• Unusual conditions. For example, sampling surface water during a once-in-100-year flood event or measuring your body weight after a stomach virus.

• An extreme part of the population.  Say my rockhounding/caving buddy Linda is in your sample group of American women, and you are measuring height.… She is 6’ 2”.  Less than 0.000027% (1 in 37,000) of the population of women are 6’ 2” or taller.  She would appear as an outlier, but she is part of the population just at the far end of the upper side of the curve.  Note…don’t fall into the trap of thinking extreme values will not show up in your samples.  There should be approximately 17 women in the Memphis Metro area that are taller than 6’ 2” and approximately 4,000 women in the USA.

Are there ways to test for outliers?

Yes and No.  

Box plots provide a semi-formal method to evaluate outliers based on the interquartile range by using Tukey Fences to identify outliers.  There are also Rosner’s test, Dixon’s test, and evaluation of normal probability plots.  To read more on these methods see the ITRC Groundwater Statistics for Monitoring and Compliance Section 5.10 Identification of Outliers.  Time Series plots are very useful if you have historical data.  Not only are extreme values easily identified, but so are sudden change in trends.  In the example below of groundwater elevations, we see a sudden drop of approximately 18 feet.

Note, these methods of identifing outliers assume a normally distributed population (a.k.a. the bell shape curve).  If the observations are not from a normally distributed population, the results of the test may be inaccurate.  For example, samples from a log normal population often appear to have several outliers.  An example of a log normal population is annual household income in the USA.  The median household income is around $81,000.  Some make a little more and some a little less.  The vast majority (99%) of household incomes are less than $550,000, but there are some that are much larger.  About 1 in 10,000 households have incomes over $30 million, but that would still be approximately 13,000 households in the USA for any given year.

“There are a lot of statistical methods looking at whether an outlier should be deleted ….. I don't endorse any of them.” 

-Barry Nussbaum, formerly Chief Statistician of USEPA

How should outliers be addressed?

 Some practical solutions are:

• Check for transcription errors.

• If the observation is a field parameter and there is data before the observation, review it.  For example, we collect field parameters on the groundwater as we purge the well before collecting a sample.  If the pH during the well purge was 6.5 to 6.7 and the end reported result was 7.6.  Maybe the result was transcribed as 7.6 but the observation was 6.7.

• Check for errors in units.  This is normally expressed as a result one thousand times greater or less than it should be.

• If you are evaluating laboratory results, have the chemist review the data package.

• Do the results seem to be shifted or inverted?  For example, an area that should be low is high and an area that should be high is low, or are clean wells showing impact now and an impacted well below normal range?  This could be a sample ID error, or the laboratory equipment carousel is off by one sample.  This happened once on a project when a storm caused a power surge at the laboratory.  I have also seen this happen when the sample was one-off because of human error loading the instrument.

• Are several of the results from the same event outliers?  Have the chemist review the data package.  There could be calibration or equipment issues.  Did the method change or the laboratory change?

• Unusually high results, especially metals, for samples that were not filtered could indicate that the sample was turbid (had elevated levels of suspended solids), and the solids increased the concentration.  This would be an unusual condition.

Possible ways to address outliers. 

If the result is a laboratory analysis, see if they can reanalyze the sample.  The result may be out of hold time, but a flagged result is better than a wrong result.  If reanalysis is not possible, resample if possible.  Note: a resample may confirm an outlier but it cannot completely disprove an outlier as the suspected outlier may still be an extreme result in the population, or conditions may have been different during the original sampling event.

Should I remove outliers?

Outliers should only be removed if there is evidence that they resulted from an error and cannot be corrected.  For example, we once received the following data from a laboratory for a water sample:

• Total Chromium dissolved (i.e., all the valences) – 8 mg/L.

• Hexavalent Chromium dissolved – 127 mg/L.

Being the total chromium includes trivalent (Cr3+) and hexavalent (Cr6+) it must be equal to or greater than the hexavalent chromium (give or take a little for acceptable method error).  So, this result could not be possible.  No error was identified in the analysis.  So, the observation was removed.  

Other examples of outliers that should be removed are:

• Depth to groundwater that is greater than the total depth of the well. 

• pH greater than 14.  Note you can have negative pH; they are rare but possible.

• Some negative values such as a negative dissolved oxygen or specific conductance.

• Values greater than saturation limit for water.  Be careful here, you can have super saturated dissolved oxygen, and you can have free product in your sample.

There is a judgement call for observations that appear be the result of unusual conditions such as sampling during a 100 year flood event or another population being included.  The project team and stakeholders need to discuss and come to a decision based on the purpose and goals of the work.

Ways to avoid / prevent outliers.

Since outliers often arise from errors in the data, avoiding or at least detecting and correcting errors is the best approach. 

For field data:

• If there is historical data, provide field personnel historical ranges to spot outliers immediately so errors can be addressed immediately.  If they know the suspected range for specific conductance is 450 to 600 µS/cm but the meter is reading 1,200 µS/cm, they can check the meter calibration or check the sample with another meter.  The result may be 1,200 µS/cm but this way it will have been checked and confirmed.

• Mid-day and end of day calibration checks or checks before each sample point.  It only takes 10 minutes but can save a lot of money and headaches over the long run.

Review data as soon as possible for possible outliers!!  This will allow you the opportunity to analyze samples in the laboratory or possibly resample as needed.  Don’t wait four months to look at the data.  As noted above, a resample may confirm an outlier but it cannot completely disprove an outlier.  The outlier still may be an extreme result in the population or conditions may have been different during the original sampling event.

If the observation is a critical value, for example, if the result of this sample is above X it will cost the client $100,000, consider submitting a second or third sample to be held by the laboratory if hold times allow.  If the result is above X the second and/or third samples can be analyzed to confirm the result.  If hold times do not allow for this, consider having second and/or third samples analyzed with the primary sample.  A few hundred dollars on the front end may save you several thousand dollars on the back end, and the client will be impressed by your preparedness. 

In summary, outliers are just something we must deal with.  Often, they are an error and can be corrected.  Sometimes they are an extreme part of the population.  Sometimes they are the result of unusual conditions.  And sometimes we just have no idea why they occurred.

Additional resources:

• Statistical Methods in Water Resources by the USGS 

• ITRC Groundwater Statistics for Monitoring and Compliance Section 5.10 Identification of Outliers.

• Statistical Analysis of Groundwater Monitoring Data at RCRA Facilities — Unified Guidance, EPA 530/R-09-007. United States Environmental Protection Agency 2009.