Identifying outliers in non-normally distributed data

Here is a simple function I use.  It uses Median +/- 5.2 * MAD.  If I
recall, this flags about 1/2000 of values from a true Normal distribution.

is.outlier = function (x) {
    # See: Davies, P.L. and Gather, U. (1993).
    # "The identification of multiple outliers" (with discussion)
    # J. Amer. Statist. Assoc., 88, 782-801.

    x <- na.omit(x)
    lims <- median(x) + c(-1, 1) * 5.2 * mad(x, constant = 1)
    x < lims[1] | x > lims[2]
}

Maybe the function should be called "is.patentable".  I definitely agree
with Bert's comments.

Kevin Wright



On Wed, Dec 30, 2009 at 11:47 AM, Jerry Floren
<jerry.floren at state.mn.us>wrote:

Greetings:

I could also use guidance on this topic. I provide manure sample
proficiency
sets to agricultural labs in the United States and Canada. There are
about
65 labs in the program.

My data sets are much smaller and typically non-symmetrical with obvious
outliers. Usually, there are 30 to 60 sets of data, each with triple
replicates (90 to 180 observations).

There are definitely outliers caused by the following: reporting in the
wrong units, sending in the wrong spreadsheet, entering data in the wrong
row, misplacing decimal points, calculation errors, etc. For each
analysis,
it is common that two to three labs make these types of errors.

Since there are replicates, errors like misplaced decimal points are more
obvious. However, most of the outlier errors are repeated for all three
replicates.

I use the median and Median Absolute Deviation (MAD, constant = 1) to
flag
labs for accuracy. Labs where the average of their three reps deviates
more
than 2.5 MAD values from the median are flagged for accuracy. With this
method, it is not necessary to identify the outliers.

A collegue suggested running the data twice. On the first run, outliers
more
than 4.0 MAD units from the median are removed. On the second run, values
exceeding 2.9 times the MAD are flagged for accuracy. I tried this in R
with
a normally distributed data set of 100,000, and the 4.0 MAD values were
nearly identical to the outliers identified with boxplot.

With my data set, the flags do not change very much if the data is run
one
time with the flags set at 2.5 MAD units compared to running the data
twice
and removing the 4.0 MAD outliers and flagging the second set at 2.9 MAD
units. Using either one of these methods might work for you, but I am not
sure of the statistical value of these methods.

Yours,

Jerry Floren



Brian G. Peterson wrote:

John wrote:

Hello,

I've been searching for a method for identify outliers for quite some
time now. The complication is that I cannot assume that my data is
normally distributed nor symmetrical (i.e. some distributions might
have one longer tail) so I have not been able to find any good tests.
The Walsh's Test (http://www.statistics4u.info/
fundsta...liertest.html#), as I understand assumes that the data is
symmetrical for example.

Also, while I've found some interesting articles:
http://tinyurl.com/yc7w4oq ("Missing Values, Outliers, Robust
Statistics & Non-parametric Methods")
I don't really know what to use.

Any ideas? Any R packages available for this? Thanks!

PS. My data has 1000's of observations..

Take a look at package 'robustbase', it provides most of the standard
robust
measures and calculations.

While you didn't say what kind of data you're trying to identify

outliers

in,
if it is time series data the function Return.clean in
PerformanceAnalytics may
be useful.

Regards,

   - Brian


--
Brian G. Peterson
http://braverock.com/brian/
Ph: 773-459-4973
IM: bgpbraverock

--
View this message in context:
http://n4.nabble.com/Identifying-outliers-in-non-normally-distributed-data-tp987921p991062.html
Sent from the R help mailing list archive at Nabble.com.

-- 
Kevin Wright

	[[alternative HTML version deleted]]