Here’s a list of analytics problems I’ve devised for a class I was teaching a digital analytics course (Web & Mobile Analytics, Information Technology Program) at Aalto University in Helsinki. Some solutions to them are also considered.
- Last click fallacy = taking only the last interaction into account when analayzing channel or campaign performance (a common problem for standard Google Analytics reports)
- Analysis paralysis = the inability to know which data to analyze or where to start the analysis process from (a common problem when first facing a new analytics tool 🙂 )
- Vanity metrics = reporting ”show off” metrics as oppose to ones that are relevant and important for business objectives (a related phenomenon is what I call “metrics fallback” in which marketers use less relevant metrics basically because they look better than the primary metrics)
- Aggregation problem = seeing the general trend, but not understanding why it took place (this is a problem of “averages”)
- Multichannel problem = losing track of users when they move between online and offline (in cross-channel environment, i.e. between digital channels one can track users more easily, but the multichannel problem is a major hurdle for companies interested in knowing the total impact of their campaigns in a given channel)
- Churn problem = a special case of the aggregation problem; the aggregate numbers show growth whereas in reality we are losing customers
- Data discrepancy problem = getting different numbers from different platforms (e.g., standard Facebook conversion configuration shows almost always different numbers than GA conversion tracking)
- Optimization goal dilemma = optimizing for platform-specific metrics leads to suboptimal business results, and vice versa. It’s because platform metrics, such as Quality Score, are meant to optimize competitiveness within the platform, not outside it.
- Last click fallacy → attribution modeling, i.e. accounting for all or select interactions and dividing conversion value between them
- Analysis paralysis → choosing actionable metrics, grounded in business goals and objectives; this makes it easier to focus instead of just looking at all of the overwhelming data
- Vanity metrics → choosing the right KPIs (see previous) and sticking to them
- Aggregation problem → segmenting data (e.g. channel, campaign, geography, time)
- Multichannel problem → universal analytics (and the associated use of either client ID or customer ID, i.e. a universal connector)
- Churn problem → cohort analysis (i.e. segment users based on the timepoint of their enrollment)
- Data discrepancy problem → understanding definitions & limitations of measurement in different ad platforms (e.g., difference between lookback windows in FB and Google), using UTM parameters to track individual campaigns
- Optimization goal dilemma → making a judgment call, right? Sometimes you need to compromise; not all goals can be reached simultaneously. Ultimately you want business results, but as far as platform-specific optimization helps you getting to them, there’s no problem.
Want to add something to this list? Please write in the comments!
[edit: I’m compiling a larger list of analytics problems. Will update this post once it’s ready.]
- Zellner, A. (1969). On the Aggregation Problem: A New Approach to a Troublesome Problem. In K. A. Fox, J. K. Sengupta, & G. V. L. Narasimham (Eds.), Economic Models, Estimation and Risk Programming: Essays in Honor of Gerhard Tintner (pp. 365–374). Springer Berlin Heidelberg.
- 4 Steps to Setting Better Conversion Optimization Goals: http://searchengineland.com/4-steps-to-setting-better-conversion-optimization-goals-32964
- Analysis Paralysis: What It Is and How to Avoid It: https://community.uservoice.com/blog/analysis-paralysis-what-it-is-and-how-to-avoid-it/
- Don’t Be Fooled By Vanity Metrics: https://techcrunch.com/2011/07/30/vanity-metrics/
- Why Last-Click Doesn’t Tell the Whole Story: http://digiday.com/marketing/why-last-click-analytics-dont-tell-the-whole-story/
- Startup Killer: the Cost of Customer Acquisition: http://www.forentrepreneurs.com/startup-killer/
- Problems of standard attribution modelling: http://jonisalminen.com/2017/03/problems-of-standard-attribution-modeling/
I’m into digital marketing, startups, platforms. Download my dissertation on startup dilemmas: http://goo.gl/QRc11f