Insights from Implementing Click DCG in Adobe Analytics

Abstract

Discounted Cumulative Gain (DCG) is a commonly used metric for evaluating the precision and ranking quality of search engines results. It provides a common baseline to compare and improve search algorithms in web applications, for example through A/B testing. While the metric is proven to be useful for product improvement of products with search functionality, it has not been implemented in common Web Analytics products, such as Adobe Analytics or Google Analytics.

In this work, we have implemented DCG in Adobe Analytics. This implementation allows computation of DCG@3, DCG@5, and DCG@25 using the standard log2 model. The solution uses the click-through rates for search results, hence we coin this version of DCG: Click DCG (cDCG). We show which events need to be triggered on the website and how an Adobe Analytics Calculated Metric can be built to compute cDCG.

Further to implementation, we demonstrate the value and rich insights we can gained from analyzing over 50 search pages at Elsevier, by breaking cDCG down by over 100 dimensions available in Adobe Analytics. For example, we show how cDCG can be used in query analysis to identify query terms that perform below average. And, we present how the users’ origin (i.e. country) affects cDCG. Lastly, we demonstrate how cDCG can be used in A/B experiments using Adobe Target, and how it was used in an actual A/B experiment to validate that cDCG values remained at high levels while migrating the search backend to a new technology.

Keywords: Web Analytics, Adobe Analytics, Search Optimization, DCG