(In English: Measuring the measurer: What’s the value of experimentation & measurement?

過去十年,實驗與量度(Experimentation & Measurement)於科網界漸趨普及。其領域包含A/B測試、計量經濟學、因果推論等系統及技術。借助近年大數據機遇,大型企業如谷歌、臉書、微軟、亞馬遜等每年均進行上萬實驗,以了解用户需求及量度各產品、服務及概念的影響力(參見:《哈佛商業評論》專欄報導)。

儘管我們經常使用各種實驗與量度技術來測量各項目的價值,卻不甚了解技術本身的價值。「量量者,難也」可能是對此現象最好的概述。因此,各組織謀劃時容易因系統及技術價值不明而將其怱略。企業若要有效投資實驗與量度系統及技術,必先要了解其本身價值。

量量者,難也。

三個觀點看價值

實驗與量度系統及技術的價值可從以下三個範疇去推算:

其一、產品價值認知:透過不同實驗,各組織可認證概念價值,並盡早隔離及移除無效產品或服務以保障整體成果。

其二、完善產品質素:組織可並行試驗大量不同產品微調,並以汰弱留強方式從多達二十九類變數中選取最為用家歡迎組合(參見:布朗與鍾斯有關各種變數白皮書)。

其三、支援優先排序:實驗與量度技術可令組織作更準確項目價值估算,消除不穩定性,從而提高決策素質。

於上述三個範疇當中,又以推算支援優先排序的價值最具挑戰性。其它範疇因篇幅有限,故略而不談。

推算優先排序價值

要了解高素質優先排序的價值,我們必先要了解何謂高素質優先排序。

用一初創企業為例,假設有四個項目,姑且名為蘋果、柳橙、香蕉及葡萄。團隊能從四個項目當中任選兩項作業。決策人理應按項目價值將其排序,並優先處理首兩個項目。


(中文版: 量量者:實驗與量度技術的價值)

Experimentation & Measurement (E&M) has become increasingly popular in the past decade. Major tech organisations — Facebook, Google, Microsoft, Amazon, and more — all reported running tens of thousands of experiments every year to test their products and measure their impact quantitatively (Microsoft has a very good write-up on what they learnt during the process).

E&M capabilities come in many forms: an online controlled experiment framework (e.g. one that can run A/B tests), a team of econometrics analysts, a system capable of performing machine learning-aided causal inference, and so on.

While the existence of E&M capabilities enables one…

Bryan Liu

Machine Learning Scientist at ASOS.com and PhD student at Imperial College London.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store