Outlier Detection Using Owl

Brian Mearns
CollibraDQ
Published in
2 min readSep 5, 2019

Numerical Outliers

Kodak Coin! in 2018 Kodak announced themselves as Kodak coin and witnessed a steep change in their stock price. Owl automatically captured this event and provided the ability to drill into the item.

Complex outliers made Simple

Even though Owl uses complex formulas to identify the correct outliers in a dataset, it uses simple terms when displaying them. If you notice below the change happened gradually, therefore if you only compared avgs or previous values you would not understand the full impact of this price change. 0% changed from yesterday and its moving/trailing avg would have caught up.

Categorical Outliers

Categorical Outliers are much different than numerical outliers and require separate techniques to automatically capture meaningful anomalies. The details regarding Owl’s methodology and testing can be found below, 3 minute read on the topic.

Categorical Outliers Don’t Exist

At least not without context

medium.com

Owl will automatically learn the normal behavior of your Strings and Categorical attributes such as STOCK,OPTION,FUTURE or state codes such as . MD,NC,D.C. when a strange pattern occurs such as NYC instead of NY Owl will show this as a categorical outlier.

Spark DataFrame Example

For more information please visit:

https://docs.owl-analytics.com/dq-visuals/outliers

--

--

Brian Mearns
CollibraDQ

Co-Founder and Engineer. Interested in Solving Problems to Save Time and Money. www.linkedin.com/company/owl-analytics