Theoretical Background
"Outliers" are defined as numeric values in any random data set, which have an unusually high deviation from either the statistical mean (average) or the median value. In other words, these numbers are either relatively very small or too big. Detecting the outliers in a data set represents a complex statistical problem, with a corresponding variety of different methodologies and computational techniques as described, for example, in the NIST publication [1]. In general, finding the "Outliers" in a data set could be done by calculating the deviation for each number, expressed as either a "Z-score" or "modified Z-score" and testing it against certain predefined threshold. Z-score typically refers to number of standard deviation relative to the statistical average (in other words, it's measured in "Sigmas"). Modified Z-score applies the median computation technique to measure the deviation and in many cases provides more robust statistical detection of outliers. Mathematically, the Modified Z-score could be written (as suggested by Iglewicz and Hoaglin [1]) as:
Mi=0.6745 * (Xi -Median(Xi)) / MAD,
where MAD stands for Median Absolute Deviation. Any number in a data set with the absolute value of modified Z-score exceeding 3.5 is considered an "Outlier". Modified Z-score could be used to detect outliers in Microsoft Excel worksheet as described below.
Sample Computation of Outliers in Excel Worksheet Using Media/MAD
Step 1. Open Microsoft Excel worksheet and enter a sample set of 10 randomly selected numbers in column A, starting with the first row: 3, 1, -23, 7, 0, 12, -2, 7, 2, 1 (Note: don’t enter commas)
Step 2. In the first row of column C (in other words, C1), enter the formula:
=MEDIAN(A1:A10)
The value in this cell corresponds to the median calculated on a data set entered at step 1.
Step 3. In the second row of column C (in other words, C2), enter the array formula:
{=MEDIAN(ABS(MEDIAN(A1:A10)-A1:A10))}
The value in this cell (C2) corresponds to MAD [1].
Step 4. Enter the formula:
=IF(0.6745*(ABS(C$1-A1)>3.5*C$2), "OUTLIER", "NORMAL")
in the first row of column B and extend it down to the 10th row. Final result of “outlier’s detection” should appear in column B, indicating two "outliers" numbers (-23 and 12) as shown below:
3 NORMAL
1 NORMAL
-23 OUTLIER
7 NORMAL
0 NORMAL
12 OUTLIER
-2 NORMAL
7 NORMAL
2 NORMAL
1 NORMAL
Sample Computation of Outliers Using STD/MEAN Values
The first method described above in based on Modified Z-score and underlying Median/MAD computation. In many cases (particularly for small sample sizes), it provides more robust statistical outliers detection than a "traditional" z-score, which implies the detection based on Standard Deviation and Mean (average). Just for reference purposes, the sample computation below describes the outliers detection based on "2-Sigma" z-score; in other words, any numeric value in a data set considered an "outlier" if it's absolute value of deviation from the statistical average exceeds 2*STD. Note: As mentioned above, the first method is typically more robust and recommended for small sample sizes:
Step 1. Use the same Excel Worksheet with sample set of 10 sample numbers in column A as described in the previous example.
Step 2. Enter the formula: =AVERAGE(A1:A10) into the cell F1. This corresponds to the statistical mean calculated on a data set in column A.
Step 3. Enter the formula: =STDEV(A1:A10) into the cell F2. This corresponds to the Standard Deviation.
Step 4. Enter the formula: =IF(ABS(A1-F$1)>2*F$2,"OUTLIER", "NORMAL") in the first row of column E and extend it down to the 10th row. Final result of the “outlier’s detection” using statistical STD/AVERAGE on z-score level of 2 should appear in column E, indicating only one "outlier" number: -23 (see below); this method is missing a second outliers, which is 12, as found in the previous method.
NORMAL
NORMAL
OUTLIER
NORMAL
NORMAL
NORMAL
NORMAL
NORMAL
NORMAL
NORMAL
Notes
Both methods of statistical detection of Outliers discussed above could be easily implemented within .NET Framework. Suggested MS Excel implementation based upon worksheet functions was chosen mostly to serve the didactic/demo purposes, providing the maximum level of simplicity and clarity. For more details on statistical outliers detection methodology and computation technique, you could refer to the reference [1].
References
- Detection of Outliers
