Skip to content

Tag: analytics

Automatic Analytics: Considerations for Building User-Oriented Data Analytics Systems

This is an unpublished exploratory study we wrote with Professor Jim Jansen for Machine Learning and Data Analytics Symposium (MLDAS2018), held in Doha, Qatar.

Change of landscape: For a long time, automation has been invading the field of marketing. Examples of marketing automation include the various scripts, rules, and software solutions that optimize pay-per-click spending, machine learning techniques utilized in targeting of display advertising (Google, 2017), automated tools that generate ad copy variations, and Web analytics platforms that automatically monitor the health of marketing performance, alerting the end users automatically in case of anomalies.

In particular, several steps of progress towards automating analytics insights are currently being made in the industry and research fronts of data analytics.

For example, there are several tools providing automated reporting functions (e.g., Google Analytics, TenScores, Quill Engage, etc.). While some of these tools require pre-configuration such as creating report templates, it is becoming more common that the tool itself chooses the relevant insights it wants to portray, and then delivers these insights to the decision makers, typically pinging via email. An example of such an approach is provided in Figure 1 that shows Quill Engage, a tool that automatically creates fluent text reports from Google Analytics data.

Figure 1: Quill Engage. The Tool Automatically Generates Fluent Reports From Google Analytics Data, And Provides Numerical Comparisons Based on Outliers And Trends.

As can be seen from Figure 1, the automatic analytics tool quickly displays key information and then aims to provide context to explain the trends in the key performance indicators.

Benefits: The benefits of automatic analytics are obvious. First of all, automation spares decision makers’ time, as they are not forced to log into systems, but receive the insights conveniently to their email inboxes and can rapidly take action. Since cognitive limitations (Tversky & Kahneman, 1974) are imposing serious constraints for decision makers dealing with ever-increasing amounts of “big data,” the need for smart tools that pre-process and mine the data at the user’s convenience are highly beneficial.

The core issue that automatic analytics is solving is complexity.

As a marketing manager, one has many platforms to manage and many campaigns to run within each platform. Multiple data sources, platforms, and metrics quickly introduce a degree of complexity that hinders effective processing of information by human beings, constrained by limitations of cognitive capacity.

In general, there are two primary use cases for business analytics: (1) deep analyses that provide strategic insights, and (2) day-to-day analyses that provide operational or tactical support. While one periodically needs to perform deep analyses on strategic matters, such as updating online marketing strategy, creating a new website structure, etc., the daily decisions cannot afford a thorough use of tens of reports and hundreds of potential metrics. That is why many reports and metrics are not used by decision makers in the industry at all.

The solution to this condition has to be automation: the systems have to direct human users’ attention toward noteworthy things. This means detecting anomalies on marketing performance, predicting their impact and presenting them in actionable format to decision makers, preferably by pinging them via email or other channels, such as SMS. The systems could even directly create tasks and push them to project management applications like Trello. A requisite to automatic analytics should therefore be the well-known SMART formula, meaning that Specific, Measurable, Appropriate, Realistic and Timely goals (Shahin & Mahbod, 2007). Through this principle, decision makers are able to rapidly turn insights into action.

Interfaces for automatic analytics: To accomplish the goal of automatic analytics, one trending area of is natural language systems, where users find the information by asking the system questions in free format. For example, previously, Google Analytics had a feature called Intelligence Events, which detected anomalies. Currently, Google provides automatic insights via a mobile app, in which the user can ask the system in natural language to provide information. An example of this is provided in Figure 2.

Figure 2: Screenshot from Google Analytics Android App, Showing the Functionality of Asking Questions From the Analytics System.

However, even asking the system requires effort and prior knowledge. For example, what if the question is not relevant or misses an important trend in the data? For such cases, the system must anticipate, and in fact analyze the data beforehand. This form of “intelligent anticipation” is a central feature in automatic analytics systems.

Examples: In the following, we provide some examples of current state-of-the-art tools of automatic analytics. We then generalize some principles and guidelines based on an overview of these tools.

First, in Figure 3, we present a screenshot from email sent by TenScores, a tool that automatically scans Quality Scores for Google AdWords campaigns.

Figure 3: TenScores, the Automatic Quality Score Monitoring Tool.

In search-engine advertising, Quality Scores are important because they influence the click prices paid by the advertisers (Jansen & Schuster, 2011; Salminen, 2009). In this particular case, the tool informs when there is a change in the average Quality Score of the account.

From a user experience perspective, the threshold to alerting the user is set to very low change, resulting in many emails sent to the users. This highlights the risk of automation becoming “spammy,” leading into losing user interest. The correct threshold should be set experimentally, e.g., according to open rates by experimenting with different increments of messaging frequency and impact thresholds.

In Figure 4, we can see a popular Finnish online marketplace, Tori.fi. Tori sends automatic emails to its corporate clients, showing how their listings have performed compared to previous period, and enabling the corporate clients to take direct action from within the email.

From example, one can click the blue button and the particularly listing which is not performing well, is boosted. In addition, there is a separate section (not visible from the screenshot) showing the best performing listings.

Figure 4: Tori’s Marketplace Insights Automatically Delivered to Inbox.

Risks: There are also risks associated with automatic analytics. For example, In search-engine advertising, brands are bidding against one another (Jansen & Schuster, 2011). Thus, an obvious step to further optimize their revenue by providing transparent auction information is Google sending automatic emails when the relative position (i.e., competitiveness) of a brand decreases, prompting advertisers to take action.

This potential scenario also raises questions about morality and ethics of automated analytics, especially in click auctions where the platform owners have an incentive to recommend actions that inflate click prices (Salminen, 2009). For example, in another online advertising platform, Bing Ads, the “Opportunities” feature gives suggestions marketers can implement in a click of a button. However, many of these suggestions relate to increasing the bid prices (see Figure 5).

Figure 5: An example of Bing Ads Recommending to Increase Keyword Bid.

If the default recommendation is always to raise bids, the feature does not add value to end user but might in fact destroy it. From an end user point of view, therefore, managers are encouraged to take recommendations with a grain of salt in such cases. From a research point of view, it is an interesting question to find out how much the automatic recommendations drive user actions.

Discussion: The current situation is that marketing optimization consist of various micro-tasks that are inter-connected and require analytics skills and creativity to be solved in an optimal way. The role of automated analytics, at least with the current maturity of technology, is pre-filtering this space of potential tasks into a number that is manageable to human decision makers, and, potentially, assigning the tasks priority according to their predicted performance impact.

In this scenario, humans are still needed to make the final decisions. The human decides which suggestions or insights to act upon. Nevertheless, the prospect of automatic filtering and sorting is highly beneficial in maneuvering the fragmented channel and campaign landscape taking place in practical online marketing work.

Practical guidelines:

  • As each vertical has its own KPIs, metrics and questions, there is a requirement of using many tools. For example, search-engine optimizers require drastically different information than display advertisers, and therefore it makes no sense to create a single solution. Instead, an organization should derive the tools from its business objectives and based on the specific information needed to achieve them.
  • An example of fine-grained automatic analytics is TenScores that only specializes on monitoring one metric in one channel (Quality Score in Google AdWords). Their approach makes sense because Quality Score is such an important metric for keyword advertisers and its optimization involves a complexity, enabling TenScores to provide in-depth recommendations that are valuable to end users.

However, even though the tools may be channel-specific, their operating principles can be similar. For example, stream filtering and anomaly detection algorithms are generalizable to many types of data, and thus have wide applicability. Moreover,

  • setting the frequency threshold to pinging decision makers is a key issue that should be experimented with when designing automatic analytics systems.

Even if there is automation, it is too early to speak of real artificial intelligence. The current systems always have manually set parameters and thresholds, and miss important things that are clear for individuals. For example, the previously shown Quill Engage cannot provide an explanation why the sales dropped when going from December to January — yet, this is apparent to any individual working in the gift business: Christmas season was the reason.

Implications for developers of automatic analytics systems: Developers of various analytics systems should no longer expect that their users log in to the system to browse reports. Instead, the critical information needs to be automatically mined and sent to decision makers in an actionable format (cf. SMART principle). There is already a considerable shift in the industry to this direction which will only be emphasized as customers realize the benefits of automatic analytics. Thus, we believe the future of analytics is more about detecting anomalies and opportunities, and giving decision makers easy choices to act upon. Of course, there are also new concerns in this environment, such as biased recommendations by online ad platforms – is the system advising you to increase bids because it maximizes your profit or because it increases the owner’s revenue?

Conclusion: Analytics software providers are planning to move toward the direction of providing automated insights, and researchers should follow suite. Open questions are many, especially relating to users’ interaction with automatic analytics insights: how responsive are users to the provided recommendation? What information do the users require? What actions do users take based on the information? We expect interesting studies in this field in the near future.

References

Google. (2017). Introducing Smart display campaigns. Retrieved February 12, 2018, from https://adwords.googleblog.com/2017/04/introducing-smart-display-campaigns.html

Jansen, B. J., & Schuster, S. (2011). Bidding on the buying funnel for sponsored search and keyword advertising. Journal of Electronic Commerce Research, 12(1), 1–18.

Lee, K.-C., Jalali, A., & Dasdan, A. (2013). Real Time Bid Optimization with Smooth Budget Delivery in Online Advertising. In Proceedings of the Seventh International Workshop on Data Mining for Online Advertising (p. 1:1–1:9). New York, NY, USA: ACM. https://doi.org/10.1145/2501040.2501979

Salminen, J. (2009). Power of Google: A study on online advertising exchange. Turku: Master’s thesis. Turku School of Economics.

Shahin, A., & Mahbod, M. A. (2007). Prioritization of key performance indicators: An integration of analytical hierarchy process and goal setting. International Journal of Productivity and Performance Management, 56(3), 226–240.

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131.

Use Cases for Personas

This is a joint piece by Dr. Joni Salminen and Professor Jim Jansen. The authors are working on a system for automatic persona generation at the Qatar Computing Research Institute. The system is available online at https://persona.qcri.org.

Introduction

Personas are fictive characterizations of the core audience or customers of a company, introduced into software development and marketing in the 1990s (see Jenkinson, 1994; Cooper, 1999). Personas capture and summarize key elements of key customer segments so that decision makers could better understand their audience or customers, not just by using numbers but also referring to qualitative attributes, such as key pain points and desires, needs and wants. We refer to persona creation as “giving faces to data,” as personas are ideally based on real data on customer behavior. Figure 1 shows an example of a data-driven persona in which the attributes are inferred automatically from social media data.

Figure 1: Data-driven persona.

While personas have been argued to have many benefits in the academic literature (see e.g., Nielsen, 2004; Pruitt & Grudin, 2003; Salminen et al., 2017), we are constantly facing the same questions from new client organizations wishing to use our system for automatic persona generation (APG) (An et al., 2017). Namely, they want to know how to use personas in practice. While we often make the analogy that personas are like any other analytics system, meaning that the use cases depend on the client’s information needs (i.e., what they want to know about the customers), this answer is still a bit puzzling to them.

For that reason, we decided to write this piece outlining some key use cases for personas. These are meant as examples, as the full range of use cases is much wider. We will first explore some general use cases, and then proceed to elaborate on more specific persona use cases by different organizational units.

General Use Cases of Personas

In general, there are three main purposes personas serve:

1) Customer Insights. This deals with getting to know your core audience, users or customers better. For example, APG enables an organization to understand its audience’s topics of interest and preferred social media content. Who uses? Everyone in the organization.

2) Creation Activities. Using persona information to create better products, content, marketing communication, or other outputs. Who uses? Everyone in the organization dealing with customer-facing outputs.

3) Communication. Using personas for communication across departments. While it is difficult to discuss a spreadsheet, it is much easier to communicate about a person. Sharing the persona work across divisions thus increases the chance for realization of benefits. Personas make data communicable and keep team members focused on the customer needs. Who uses? Everyone in the organization.

Specific Use Cases of Personas

In addition to shared use cases of personas, there are more specific use cases. For example, product managers can use the information to design a product that meets the needs or desires of core customers, and marketing can use personas to craft messages that resonate. Here, we are outlining specific examples of use cases within organizational units. More specifically, we allocate these use cases under four sections.

1) Customer Insights and Reporting

Journey Mapping: Plot the stages and paths of the persona lifecycle, documenting each persona’s unique state of mind, needs and concerns at each stage. Understand your website visitors’ customer journey.

Persona Discovery: Document the individuals involved in the purchase process in a way that allows decision makers to empathize with them in a consistent way.

Brand Discovery: Uncover how your core customers feel about your product or service and how they rationalize their purchase decisions.

Reporting and Feedback: Report and review data and insights to drive strategic decisions, as well as provide information to the organization as a whole.

2) Creation Activities

Planning Product Offerings: With the help of personas, organizations can more easily build the features that suit their customers’ needs. Consider the goals, desires, and limitations of core customers to guide feature, interface, and design choices.

Role Playing: Personas help product developers “get into character” and understand the circumstances of their users. They facilitate genuine understanding of the thoughts, feelings, and behaviors of core customers. Individuals have a natural tendency to relate to other humans, and it’s important to tap into this trait when making design and product development choices.

Content Creation: Content creators can leverage personas for delivery of content that will be most relevant and useful to their audience. When planning for content, we might ask “Would Jamal understand this?” or “Would Jamal be attracted by this?” Personas help one determine what kind of content is needed to resonate with core customers and in which tone or style to deliver the content. Naturally, customer analytics can and should be used to verify the results.

3) Persona Experimentation

Channel and Offering Alignment: Align every piece of offerings and marketing activity to a persona and purchase stage, identifying new channels and needs where opportunities exist.

Prediction of Popularity: Predict how a given persona will react to content, marketing messages, or products. This is a particular advantage of data-driven personas that enable using the underlying topical interests of the persona to model the likely match between personas and a given content unit.

Experimentation and Optimization: Carry out well-thought experiments with personas to produce statistically valid business insights and apply the results to optimize performance. For example, you could run Facebook Ads campaigns targeting segments corresponding to the core personas and analyze whether the campaigns perform better than broader or other customer segments.

4) Strategic Decision Making

Strategic Marketing: When you understand where your core customers spend their time online, you are able to focus your marketing spend on these channels. For example, if the data shows that your core customers prefer YouTube over Facebook, you can increase your marketing spend in the former. Think how you might describe your product for this particular type of person. For example, would Bridget better understand your offering as a “social media service” or as an “enterprise customer management tool”? Depending on the answer, the communicative strategy would be different.

Sales Strategies: Targeted offerings can help organizations convert more potential customers to subscribers, followers and customers. You can also use personas to tailor lead generation strategies which is likely to improve your lead quality and performance. By approaching your messages from a human perspective, you can create sales and marketing communication that is tailored to your core customers and, therefore, is likely to perform better.

Executives: Key decision makers can keep personas in mind while making strategic decisions. In fact, a persona can become a “silent member in the boardroom,” evoked to question the customer impact of the considered decisions.

Examples for the APG system

In the following, we will include some use case examples from the APG system that generates personas automatically from online analytics and social media data. The system is currently fully functional, and we are accepting a limited number of new clients with free of charge research licenses. See the end of this post for more details.

Figure 2: This functionality enables the client to generate personas from his chosen data source (currently, following platforms are supported: YouTube, Facebook, Google Analytics). The client can choose between 5 and 15 personas.

Figure 3: The persona profile shows detailed information about the persona. It enables human-oriented customer insights.

Figure 4: This feature enables an easy comparison of the personas across their key attributes. Improves understanding of the core customer segments.

Figure 5: This feature shows which personas most often react with which individual content.

Figure 6: This feature shows how the interests and other information of the personas change over time. Currently, APG generates new personas on a monthly basis.

Figure 7: This feature enables a gap analysis of the current audience and potential audience. The statistics are retrieved from actual audience data of the organization and the corresponding Facebook audience (via Facebook Marketing API).

Conclusion

Forrester Research (2010) reports a 20% productivity improvement with teams that use personas. Yet, using personas is not always straight-forward. Ultimately, the exact use cases depend on the client’s information needs. These needs can best be found by collaborating with persona creators to provide tailored personas that are useful specifically for a given organization in their practical decision making.

Through means of “co-creation,” clients and persona creators can figure out together how the personas could be useful for real usage scenarios. According to our experience, useful questions for defining the client’s information needs include:

  • What are your objectives for content creation / marketing?
  • What kind of customer-related decisions you make?
  • What kind of customer information you need?
  • What analytics information are you currently using?
  • What kind of customer-related questions you don’t currently get good answers to?
  • How would you use personas in your own work?
  • What information you find useful in the persona mockup?
  • What information is missing from the persona mockup?

If you are interested in the possibilities of automatic persona generation for your organization, don’t hesitate to contact us! Professor Jim Jansen will gladly provide more information: [email protected]. However, please note that for automatic persona generation to be useful for your organization, you need to have at least hundreds (preferably thousands) of content pieces published online with a wide audience viewing them. APG is great at summarizing complex audiences, but if you don’t have enough data, persona generation is better done via manual methods.

References

An, J., Haewoon, K., & Jansen, B. J. (2017). Personas for Content Creators via Decomposed Aggregate Audience Statistics. In Proceedings of Advances in Social Network Analysis and Mining (ASONAM 2017). http://www.bernardjjansen.com/uploads/2/4/1/8/24188166/jansen_personas_asonam2017.pdf

Cooper, A. (1999). The Inmates Are Running the Asylum: Why High Tech Products Drive Us Crazy and How to Restore the Sanity (1 edition). Indianapolis, IN: Sams – Pearson Education.

Forrester Research. (2010). The ROI Of Personas. Retrieved from https://www.forrester.com/report/The+ROI+Of+Personas/-/E-RES55359

Jenkinson, A. (1994). Beyond segmentation. Journal of Targeting, Measurement and Analysis for Marketing, 3(1), 60–72.

Nielsen, L. (2004). Engaging personas and narrative scenarios (Vol. 17). Samfundslitteratur. Retrieved from http://personas.dk/wp-content/samlet-udgave-til-load.pdf

Pruitt, J., & Grudin, J. (2003). Personas: Practice and Theory. In Proceedings of the 2003 Conference on Designing for User Experiences (pp. 1–15). New York, NY, USA: ACM.

Salminen, J., Sercan, Ş., Haewoon, K., Jansen, B. J., An, J., Jung, S., Vieweg, S., and Harrell, F. (2017). Generating Cultural Personas from Social Data: A Perspective of Middle Eastern Users. In Proceedings of The Fourth International Symposium on Social Networks Analysis, Management and Security (SNAMS-2017). Prague, Czech Republic. Available at http://www.bernardjjansen.com/uploads/2/4/1/8/24188166/jansen_mena_personas2017.pdf

Tips on Data Imputation From Machine Learning Experts

Missing values are a critical issue in statistics and machine learning (which is “advanced statistics”). Data imputation deals with ways to fill those missing values.

Andriy Burkov made this statement a few days ago [1]:

“The best way to fill a missing value of an attribute is to build a classifier (if the attribute is binary) or a regressor (if the attribute is real-valued) using other attributes as “X” and the attribute you want to fix as “y”.”

However, the issue is not that simple. As noted by one participant:

From Franco Costa, Developer: Java, Cloud, Machine Learning:

What if is totally independent from the other features? Nothing to learn

The discussion then quickly expanded and many machine learning experts offered their own experiences and tips for solving this problem. At the time of writing (March 8, 2018), there are 69 answers.

Here are, in my opinion, the most useful ones.

1) REMOVE MISSING VALUES

From Blaine Bateman, EAF LLC, Founder and Chief Data Engineer at EAF LLC:

Or just drop it from the predictors

From Swapnil Gaikwad, Software Engineer Cognitive Computing (Chatbots) at Light Information Systems Pvt. Ltd.:

Also I got an advice from one of my mentor is whenever we have more than 50% of the missing values in a column, we can simply omit that column (if we can), if we have enough other features to build a model.

2) ASK WHY

From Kevin Gray, Reality Science:

It’s of fundamental importance to do our best to understand why missing data are missing. Two excellent sources for an in-depth look at this topic are Applied Missing Data Analysis (Enders) and Handbook of Missing Data Methodology (Molenberghs et al.). Outlier Analysis (Aggarwal) is also relevant. FIML and MI are very commonly used by statisticians, among other approaches.

From Julio Bonis Sanz, Medical Doctor + MBA + Epidemiologist + Software Developer = Health Data Scientist:

In some analysis I have done in the past, including “missing” as a value for prediction itself have got some interesting results. The fact that for a given observation that value is missing is sometimes associated with the outcome you want to predict.

From Tero Keski-Valkama, A Hacker and a Machine Learning Generalist:

Also, you can try to check if the value being missing encodes some real phenomenon (like the responder chooses to skip the question about gender, or a machine dropping temperature values above a certain threshold) by trying to train a classifier to predict whether a value would be missing or not. It’s not always the case that values being missing are just independent random noise.

From Vishnu Sai, Decision Scientist at Mu Sigma Inc.:

In my experience, I’ve found that the technique for filling up missing values depends on the business scenario.

From David T. Kearns, Co-founder, Sustainable Data and Principal Consultant, Sustainable Services:

I think it’s important to understand the underlying cause of the missing values. If your data was gathered by survey, for example, some people will realise their views are socially unpopular and will keep them to themselves. You can’t just average out that bias – you need to take steps to reduce it during measurement. For example, design your survey process to eliminate social pressure on the respondent.

For non-human measurements, sometimes instruments can be biased or faulty. We need to understand if those biases/faults are themselves a function of the underlying measurements – do we lose data just as our values become high or low for example? This is where domain knowledge is useful – making intelligence decisions of what to do, not blind assumptions.

If you’ve done all that and still have some missing values, then you’ll be in a far stronger position to answer your question intelligently.

3) USE MISSING VALUES AS A FEATURE

From Julio Bonis Sanz, Medical Doctor + MBA + Epidemiologist + Software Developer = Health Data Scientist:

One of my cases was a predictive model of use of antibiotics by patients with chronic bronchitis. One of the variables was smoking with about 20% of missing values. It turned out that having no information in the clinical record about smoking status was itself a strong predictor of use of antibiotics because a patient missing this data were receiving worse healthcare in general. By using imputation methods you someway lose that information.

From Kirstin Juhl, Full Stack Software Developer/Manager at UnitedHealth Group:

Julio Bonis Sanz Interesting- something that I wouldn’t have thought of – missing values as a feature itself.

From Peter Fennell, Postdoctoral Researcher in Statistics and A.I. @ USC:

Thats fine if have one attribute with missing values. Or two. But what if many of your features have missing values? Do recursive filling, but that can lead to error propagation? like to think that there is value in missing value, and so giving them their own distinct label (which, eg, a tree based classifier can isolate) can be an effective option

4) USE TESTED PACKAGES SUCH AS MICE OR RANDOM FOREST

From Jehan Gonsal, Senior Insights Analyst at AIMIA:

MCMC methods seem like the best way to go. I’ve used the MICE package before and found it to be very easy to audit and theoretically defensible.

From Swapnil Gaikwad, Software Engineer Cognitive Computing (Chatbots) at Light Information Systems Pvt. Ltd.:

This is a great advice! In one of my projects, I have used the R package called MICE which does the regression to find out the missing values. It works much better than the mean method.

From Nihit Save, Data Analyst at CMS Computers Limited (INDIA):

Multivariate Imputation using Chained Equation (MICE) is an excellent algorithm which tries to achieve the same. https://www.r-bloggers.com/imputing-missing-data-with-r-mice-package/

From ROHIT MAHAJAN, Research Scholar – Data Science and Machine Learning at Aegis School of Data Science:

In R there are many packages like MICE, Amelia and most Important “missForest” which will do this for you. But it takes too much time if data is more than 500 Mb. I always follow this regressor/classifier approach for most important attributes.

From Knut Jägersberg, Data Analyst:

Another way to deal with missing values in a model based manner is by using random forests, which work for both categorical and continuous variables: https://github.com/stekhoven/missForest . This algorithm can be easily be reimplemented with i.e. a faster than in R implemented RF algorithm such as ranger (https://github.com/imbs-hl/ranger) and then scales well to larger datasets.

5) USE INTERPOLATION

From Sekhar Maddula, Actively looking for Data-science roles:

Partly agree Andriy Burkov. But at the same time there are few methods specific to the technique/algo. e.g. For Time-series data, you may think of considering interpolation methods available with the R package “imputeTS”. I also hope that there are many Interpolation methods in the field of Mathematics. We may need to try an appropriate one.

6) ANALYZE DISTRIBUTIONS

From Gurtej Khanooja, Software Engineering/Data Science Co-op at Weather Source| Striving for Excellence:

One more way of dealing with the missing values is to identify the distribution using remaining values and fill the missing values by randomly filling the values from the distribution. Works fine in a lot of cases.

From Tero Keski-Valkama, A Hacker and a Machine Learning Generalist:

If you are going to use a classifier or a regressor to fill the missing values, you should sample from the predicted distribution rather than just picking the value with the largest probability.

SUMMARY

This was the best summary comment I found:

“I have used MICE package in R to deal with imputing and luckily it produced better results. But in general we should take care of the following:

  1. Why the data is missing? Is is for some meaningful reason or what?
  2. How much data is missing?
  3. Fit a model with non missing values.
  4. Now apply the imputing technique and fit the model and compare with the earlier one.”

-Dr. Bharathula Bhargavarama Sarma, PhD in Statistics, Enthusiastic data science practitioner

FOOTNOTES

[1] https://www.linkedin.com/feed/update/urn:li:activity:6375893686550614016/

Google Analytics: 21 Questions to Get Started

I was teaching a course called “Web & Mobile Analytics” at Aalto University back in 2015.

As a part of that course, the students conducted an analytics audit for their chosen website. I’m sharing the list of questions I made for that audit, as it’s a useful list for getting to know Google Analytics.

The questions

Choose a period to look at (e.g., the past month, three months, last year, this year… generally, the longer the better because it gives you more data). Answer the questions. The questions are organized by sections of Google Analytics.

a. Audience

  • How has the website traffic evolved during the period your inspecting? How does the traffic from that period compare to earlier periods?
  • What are the 10 most popular landing pages?
  • What are the 10 pages with the highest bounce rate AND at least 100 visits in the last month? (Hint: advanced filter)

b. Behavior

  • How does the user behavior differ based on the device they’re using? (Desktop/laptop, mobile, tablet)
  • Where do people most often navigate to from the homepage?
  • How do new and old visitors differ by behavior?
  • What is the general bounce rate of the website? Which channel has the highest bounce rate?
  • How well do the users engage with the website? (Hint: Define the metrics you used to evaluate engagement.)
  • Is there a difference in engagement between men and women?

c. Acquisition

  • How is the traffic distributed among the major sources?
  • Can you find performance differences between paid and organic channels?
  • Compare the goal conversion rate of different marketing channels to the site average. What can you discover?

d. Conversion

  • What is the most profitable source of traffic?
  • What is the best sales (or conversion, based on the number of conversions) month of the year? How would you use this information in marketing planning?
  • Which channels or sources seem most promising in terms of sales potential? (Hint: look at the channels with high CVR and low traffic)
  • Analyze conversion peaks. Are there peaks? Can you find explanation to such peaks?
  • Can you find sources that generated assisted conversions? Which sources are they? Is the overall volume of assisted conversions significant?
  • Does applying another attribution model besides the last click model alter your view on the performance of marketing channels? If so, how?

e. Recommendations

  • Based on your audit, how could the case company develop its digital marketing?
  • How could the case company’s use of analytics be developed? (E.g., what data is not available?)
  • What other interesting discoveries can you make? (Mention 2–5 interesting points.)

Answering the above questions provides a basic understanding of a goal-oriented website’s situation. In the domain of analytics, asking the right questions is often the most important (and difficult) thing.

The dashboard

In addition, the students built a dashboard for the class. Again, the instructions illustrate some useful basic functions of Google Analytics.

Build a dashboard showing the following information. Include a screenshot of your dashboard in the audit report.

Where is the traffic coming from?

  • breakdown of traffic by channel

What are the major referral sources?

  • 10 biggest referral sites

How are conversions distributed geographically?

  • 5 biggest cities by conversions

How is Facebook bringing conversions?

  • Product revenue from Facebook as a function of time

Are new visitors from different channels men or women?

  • % new sessions by channels and gender

What keywords bring in the most visitors and money?

  • revenue and sessions by keyword

If you see fit, include other widgets in the dashboard based on the key performance metrics of your company.

Conclusion

Reports and dashboards are basic functions of Google Analytics. More advanced uses include custom reports and metrics, alerts, and data importing.

Simple methods for anomaly detection in e-commerce

Anomaly is a deviation from the expected value. The main challenges are: (a) how much the deviation should be to be classified as an anomaly, and (b) what time frame or subset of data should we examine.

The simplest way to answer those questions is to use your marketer’s intuition. As an e-commerce manager, you have an idea of how big of an impact constitutes an anomaly for your business. For example, if sales change by 5% in a daily year-on-year comparison, that would not typically be an anomaly in e-commerce, because the purchase patterns naturally deviate this much or even more. However, if your business has e.g. a much higher growth going on and you suddenly drop from 20% y-o-y growth to 5%, then you could consider such a shift as an anomaly.

So, the first step should be to define which metrics are most central for tracking. Then, you would define threshold values and the time period. In e-commerce, we could e.g. define the following metrics and values:

  • Bounce Rate – 50% Increase
  • Branded (Non-Paid) Search Visits – 25% Decrease
  • CPC Bounce – 50% Increase
  • CPC Visit – 33% Decrease
  • Direct Visits – 25% Decrease
  • Direct Visits – 25% Increase
  • Ecommerce Revenue – 25% Decrease
  • Ecommerce Transactions – 33% Decrease
  • Internal Search – 33% Decrease
  • Internal Search – 50% Increase
  • Non-Branded (Non-Paid) Search Visits – 25% Decrease
  • Non-Paid Bounces – 50% Increase
  • Non-Paid Visits – 25% Decrease
  • Pageviews – 25% Decrease
  • Referral Visits – 25% Decrease
  • Visits – 33% Decrease

As you can see, this is rule-based detection of anomalies: once the observed value exceeds the threshold value in a given time period (say, daily or weekly tracking), the system alerts to e-commerce manager.

The difficulty, of course, lies in defining the threshold values. Due to changing baseline values, they need to be constantly updated. Thus, there should be better ways to detect anomalies.

Another simple method is to use a simple sliding window algorithm. This algorithm can (a) update the baseline value automatically based on data, and (b) identify anomalies based on a statistical property rather than the marketer’s intuition. The parameters for such an algorithm are:

  • frequency: how often the algorithm runs, e.g. daily, weekly, or monthly. Even intra-day runs are possible, but in most e-commerce cases not necessary (exception could be technical metrics such as server response time).
  • window size: this is the period for updating. For example, if the window size is 7 days and the algorithm is run daily, it computes that data always from the past seven days, each day adding +1 to start and end date.
  • statistical threshold: this is the logic of detecting anomalies. A typical approach is to (a) compute the mean for each metric during window size, and (b) compare the new values to mean, so that a difference of more than 2 or 3 standard deviations from the mean indicates an anomaly.

Thus, the threshold values automatically adjust to the moving baseline because the mean value is re-calculated at each window size.

How to interpret anomalies?

Note that an anomaly is not necessarily a bad thing. Positive anomalies occur e.g. when a new campaign kicks off, or the company achieves some form of viral marketing. Anomalies can also arise when a season breaks in. To mitigate such effects from showing, one can configure the baseline to represent year-on-year data instead of historical data from the current year. Regardless of whether the direction of the change is positive or negative, it is useful for a marketer to know there is a change of momentum. This helps restructure campaigns, allocate resources properly, and become aware of the external effects on key performance indicators.

Platform metrics: Some ideas

I was chatting with Lauri [1] about platform research. I claimed that the research has not that many implications for real-world companies apart from the basic constructs of network effects, two-sidedness, tipping, marquee users, strategies such as envelopment, and of course many challenges, including chicken-and-egg problems, monetization dilemma, and remora’s curse (see my dissertation on startup dilemmas for more insights on those…).

But then I got to think that metrics are kind of overlooked. Most of the platform literature comes from economics and is very theoretical and math-oriented. Yet, it’s somehow apart from practical Web analytics and digital metrics. Those kind of metrics, however, are very important for platform entrepreneurs and startup founders.

On the face of it, it seems that the only difference that platforms have compared to “normal” businesses is their two-sidedness. If we have supply side (Side A) and demand side (Side B), then the metrics could be just identical for each and the main thing is to keep track of the metrics for both sides.

However, there are some dynamics at play. The company has one goal, one budget, one strategy, at least typically. That means those metrics, even though can be computed separately, are interconnected.

Here are some examples of platform metrics:

  • Number of Users/Customers (Side A, Side B)
  • Revenue (Side A, Side B)
  • Growth of Revenue (Side A, Side B)
  • Cross-correlation of Number of Users and Revenue (e.g., Side A users => Side B revenue)
  • Cost per User/Customer Acquisition (Side A, Side B)
  • Support cost (Side A, Side B)
  • Average User/Customer Lifetime (Side A, Side B)
  • Average Transaction Value (Side A, Side B)
  • Engagement Volume (Side A, Side B)
  • Profitability distribution (Side A, Side B)

Note the cross-correlation example. Basically, all the metrics can be cross-correlated to analyze how different metrics of each side affect each other. Moreover, this can be done in different time periods to increase robustness of the findings. Such correlations can reveal important information about the dynamics of network effects and tell, for example, whether to focus on adding Side A or Side B at a given point in time. A typical example is solving the cold start problem by hitting critical mass, i.e., the minimum number of users required for network effects to take place (essentially, the number of users needed for the platform to be useful). Before this point is reached, all other metrics can look grim; however, after reaching that point, the line charts of other metrics should turn from flat line to linear or exponential growth, and the platform should ideally become self-sustainable.

Basic metrics can also be used to calculate profitability, e.g.

Average Transaction Value x Average Customer Lifetime > Cost per Customer Acquisition

Business models of many startup platforms are geared towards “nickel economics,” meaning that the average transaction values are very low. In such situations, the customer acquisition cost has to be low as well, or the frequency of transactions extremely high. When these rules are violated, the whole business model does not make sense. This is partly because of competitive nature of the market, requiring sizable budgets for actual user/customer acquisition. For platforms, the situation is even more serious than for other businesses because network effects require the existence of critical mass that costs money to achieve.

In real world, customer acquisition cost (CPA) cannot usually be ignored, apart from few outliers. The CPA structure might also differ between the platform sides, and it is not self-evident what type of customer acquisition strategies yield the lowest CPAs. In fact, it is an empirical question. A highly skilled sales force can bring in new suppliers at a lower CPA than a digital marketer that lacks the skills or organic demand. Then again, under lucrative conditions, the CPA of digital advertising can be minuscule compared to sales people due to its property of scaling.

However, as apparent from the previous list, relationship durations also matter. For example, many consumers can be fickle but supplier relationships can last for years. This means that suppliers can generate revenue over a longer period of time than consumers, possibly turning a higher acquisition cost into more a more profitable investment. Therefore, churn of each side should be considered. Moreover, there are costs associated with providing customer support for each side. As Lauri noted based on his first-hand experience working for a platform company, the frequency and cost per customer encounter differ vastly by side, and require different kind of expertise from the company.

In cases where the platform has an indirect business model, also called subvention because one side is subventing the cost of the other, the set of metrics should be different. For example, if only Side B (paid users) is paying the platform but is doing so because there is Side A (free users), Side B could be monitored with financial metrics and Side A with engagement metrics.

Finally, profitability distribution refers to uneven distribution of profitable players in each market side. This structure is important to be aware of. For example, in e-commerce it is typical that there are a few “killer products” that account for a relatively large share of sales value, but the majority of the total sales value is generated by hundreds or thousands of products with small individual sales. Understanding this dynamics of adding both killer products and average products (or complements, in using platform terms) is crucial for managing the platform growth.

Footnotes:

[1] Lauri Pitkänen, my best friend.

Digital analytics maturity model

Digital analytics maturity model:

  1. Concepts — here, focus on is on buzzwords and realization that “we should do something”.
  2. Tools — here, focus is on tools, i.e. “Let’s use this shiny new technology and it will solve all our problems.”
  3. Value — here, we finally focus on what matters: how will the tools and technologies serve and integrate with our core competitive advantage, i.e. “Guys, what’s the point?”.

Applies to almost any booming technology.

Argument: Personas lose to ‘audience of one’

Introduction. In this post, I’m exploring the usefulness of personas in digital analytics. At Qatar Computing Research Institute (QCRI), we have developed a system for automatic persona generation (APG) – see the demo. Under the leadership of Professor Jim Jansen, we’re constantly working to position this system toward the intersection of customer profiles, personas, and analytics.

Three levels of data. Imagine three levels of data:

  •  customer profiles (individual)
  • personas (aggregated individual)
  • statistics (aggregated numbers: tables and charts)

Which one is the best? The answer: it depends.

Case of advertising. For advertising, usually the more individual the data, the better. The worst case is the mass advertising, where there is one message for everyone: it fails to capture the variation of preferences and tastes of the underlying audience, and is therefore inefficient and expensive. Group-based targeting, i.e. market segmentation (“women 25-34”) performs better because it is aligning the product features with the audience features. Here, the communalities of the target group allow marketers to create more tailored and effective messages, which results in less wasted ad impressions.

Case of design and development. In a similar vein, design is moving towards experimentation. You have certain conventions, first of all, that are adopted industry-wide in the long run (e.g., Amazon adopts a practice and small e-commerce sites follow suit). Many prefer being followers, and it works for the most part. But multivariate testing etc. can reveal optimal designs better than “imagining a user” or simply following conventions. Of course, personas, just like any other immersive technique, can be used as a source of inspiration and ideas. But they are just one technique, not the technique.

For example, in the case of mobile startups I would recommend experimentation over personas. A classic example is Instagram that found from data that filters were a killer feature. For such applications, it makes sense to define an experimental feature set, and adjust if based on behavioral feedback from the users.

Unfortunately, startup founders often ignore systematic testing because they have a pre-defined idea of the user (à la persona) and are not ready to get their ideas challenged. The more work is done to satisfy the imaginary user, the more harder it becomes to make out-of-the-box design choices. Yet, those kind of changes are required to improve not by small margins but by orders of magnitude. Eric Ries call this ‘sunk code fallacy’.

In my opinion, two symptoms predating such a condition can be seen when the features are not

  1. connected to analytics, so that tracking of contribution of each is possible (in isolation & to the whole)
  2. iteratively analyzed with goal metrics, so that there is an ‘action->response->action’ feedback loop.

In contrast, iterative (=repetitive) analysis of performance of each feature is a modern way to design mobile apps and websites. Avoiding the two symptoms is required for systematic optimization. Moreover, testing the features does not need to take place in parallel, but it can be one by one as sequential testing. This can in fact be preferable to avoid ‘feature creep’ (clutter) that hinders the user experience. However, for sequential testing it is preferable to create a testing roadmap with a clear schedule – otherwise, it is too easy to forget about testing.

Strategic use cases show promise. So, what is left for personas? In the end, I would say strategic decision making is very promising. Tactical and operational tasks are often better achieved by using either completely individual or completely aggregated data. But individual data is practically useless at strategic decision making. Aggregated data is useful, e.g. sales by region or customer segment, and it is hard to see anything replace that. However, personas are in between the two – they can provide more understanding on the needs and wants of the market, and act as anchor points for decision making.

Strategic decision aid is also a lucrative space; companies care less about the cost, because the decisions they make are of great importance. To correctly steer the ship, executives need need accurate information about customer preferences and have clear anchor points to align their strategic decision with (see the HubSpot case study).

In addition, aggregated analytics systems have one key weakness. They cannot describe the users very well. Numbers do not include information such as psychographics or needs, because they need to be interpreted from the data. Customer profiles are a different thing — in CRM systems, enrichment might be available but again the number of individual profiles is prohibitive for efficient decision making.

Conclusion. The more we are moving towards real-time optimization, the less useful a priori conceptualizations like target groups and personas become for marketing and design. However, they are likely to remain useful for strategic decision making and as “aggregated people analytics” that combine the coverage of numbers and the details of customer profiles. The question is: can we build personas that include the information of customer profiles, while retaining the efficiency of using large numbers? At QCRI, we’re working everyday toward that goal.

Agile methods for predicting contest outcomes by social media analysis

People think, or seem to assume, that there is some magical machine that spits out accurate predictions of future events from social media data. There is not, and that’s why each credible analysis takes human time and effort. But therein also lies the challenge: when fast decisions are needed, time-taking analyses reduce agility. Real-time events would require real-time analysis, whereas data analysis is often cumbersome and time-taking effort, including data collection, cleaning, machine training, etc.

It’s a project for weeks or days, not for hours. All the practical issues of the analysis workflow make it difficult to provide accurate predictions at a fast pace (although there are other challenges as well).

An example is Underhood.co – they predicted Saara Aalto to win X-Factor UK based on social media sentiment, but ended up being wrong. While there are many potential reasons for this, my conclusion is that their indicators lack sufficient predictive power. They are too reliant on aggregates (in this case country-level data), and had a problematic approach to begin with – just like with any prediction, the odds change on the go as new information becomes available, so you should never predict the winner weeks ahead. Of course, theirs was just a publicity stunt where they hoped being right would prove the value of their service. Another example, of course, is the US election where prediction markets were completely wrong of the outcome. That was, according to my theory, because of wrong predictors – polls ask what is your preference or what you would do, whereas social media engagement shows what people do (in social media), and as such are closer to real behavior, hence better predictors.

Even if I do think human analysts are still needed in the near future, more solutions for quick collection and analysis of social media data are needed, especially to combine the human and machine work in the best possible way. Some of these approaches can be based on automation, but others can be methodological, such as quick definition of relevant social media outlets for sampling.

Here are some ideas I have been thinking of:

I. Data collection

  1. Quick definition of choice space (e.g., candidates in a political election, X-Factor contestants)
  2. Identification of related media social media outlets (i.e., communities, topic hashtags)
  3. Collecting sample (API, scraping, or copy-paste (crowdsourcing))

Each part is case-dependent and idiosyncratic – for whatever event, I’m thinking competitions here, you have to this work from scratch. Ultimately, you cannot get the whole Internet as your data, but you want the sample to be as representative as possible. For example, it was obvious that Twitter users showed much more negative sentiment towards Trump than Facebook users, and in both platforms you had supporter groups/topic concentrations that should first be identified before any data collection. Then, the actual data collection is tricky. People again seem to assume all data is easily accessible. It’s not – while Twitter and Facebook have an API, Youtube and Reddit don’t, for example. This means the comments that you use for predicting the outcome (by analyzing their relative share of the total as well as the strength of the sentiment beyond pos/neg) need to be fetched either by webscraping or manually copying them to a spreadsheet. Due to large volumes of data, crowdsourcing could be useful — e.g., setting up a Google Sheet where crowdworkers each paste the text material in clean format. The raw text content, e.g. tweets, Facebook comments, Reddit comments, is put in separate sheets for each candidate.

II. Data analysis

  1. Cluster visualization (defining clusters, visualizing their respective sizes (plot # of voters), breakdown by source platform and potential other factors)
  2. Manual training (classifying the sentiment, or “likelihood to vote”)
  3. Machine classification (calculating the number of likely voters)

In every statistical analysis, the starting point should be visualizing the data. This shows an aggregate “helicopter view” of the situation. Such a snapshot is useful also for demonstrating the results for the end-user, to let the data speak for itself. Candidates are bubbles in the chart, their sizes in respect to the number of calculated likely voters. The data could be broken down according to source platforms, or other factors, by using the candidate as a point of gravity for the cluster.

Likelihood to vote could be classified as a scale, not binary. That is, instead of saying “sentiment is positive: YES/NO”, we could say “How likely is the person to vote?” which is the same as asking how enthusiastic or engaged he or she is. Therefore, a scale is better, e.g. ranging from -5 (definitely not voting for this candidate) to +5 (definitely voting for this candidate). The manual training, which also could be done with the help of crowd, helps the machine classifier to improve its accuracy on the go. Based on training data, it would generalize classification to all material. Now, the material is bucketed so that each candidate is evaluated separately and the number of likely voters can be calculated. It is possible that the machine classifier could benefit from training input from both candidates, inasmuch the language showing positive and negative engagement is not significantly different.

It is important to note that negative sentiment does not really matter. What we are interested in is the number of likely voters. This is because of the election dynamics – it does not matter how poor a candidates aggregate sentiment is, i.e. the ratio between haters and sympathizers, as long as his or her number of likely voters is higher than that of the competition. This effect was evident in the recent US presidential election.

The crucial thing is keep the process alive during the whole election/competition period. There is no such point where it becomes certain that one loses and the other remains, although the divide can become substantial and therefore increase the accuracy of the prediction.

III. Presentation of results

  • constantly updating feed (à la Facebook video stream)
  • cluster visualization
  • search trend widget (source: Google Trends)
  • live updating predictions (manual training –> machine model)

The results could be shown as a form of dashboard to the end user. Search trend graph and the above mentioned cluster visualization could be viable alternatives. In addition, it would be interesting to see the count of voters evolving in time – in such a way that it, along with the visualization, could be “played back” to examine the development in time. In other words, interactive visualization. As noted, the prediction, or the count of likely votes, should update real-time as a result of combined human-machine work.

Conclusion and discussion

The idea behind development of more agile methods to use social media data to predict content outcomes is that the accuracy of the prediction is based on the choice of indicators rather than the finesse of the method. For example, complex Bayesian models falsely predicted Hillary Clinton would win the election. It’s not that the models were poorly built; they just used the wrong indicators, namely polling data. This is the usual case of ‘garbage in, garbage out’, and it shows that the choice of indicators is more important than technical features of the predictive model.

The choice of indicators should be done based on their predictive power and although I don’t have strict evidence on it, it intuitively makes sense that social media engagement is a stronger indicator in many instances than survey data, because it’s based on actual preferences instead of stated preferences. Social scientists know from long tradition of survey research that there are myriad of social effects reducing the reliability of data (e.g., social desirability bias). Those, I would argue, are much smaller issue in social media engagement data.

However, to be fair, there can be issues of bias in the social media engagement data. The major concern is low participation rate: a common heuristic is that 1/10 of participants actually contribute in writing, while the other 9/10 are readers whose real thoughts remain unknown. It’s then a question of how well does the vocal minority reflect the opinion of the silent majority. Or, in some cases, this is irrelevant for competitions if the overall voting share remains low. For example, if it’s 60% it is relative much more important to mobilize the active base than if voting was close to 100% where one would need a universal acceptance.

Another issue is the non-representative sampling. This is a concern when the voting takes place offline, and online data does not accurately reflect the voting of those who do not express themselves online. However, as social media participation is constantly increasing, this becomes less of a problem. In addition, compared to other methods of data collection – apart from stratified polling, perhaps – social media is likely to give a good result on competitive predictions because of their political nature. People who strongly support a candidate are more likely to be vocal about it, and the channel for voicing their opinion is the social media.

It is evident that the value of social media engagement as a predictor is currently underestimated, as proven by the large emphasis put on political polls and virtually zero discussion on social media data. As a direct consequence of this, those who are able to leverage the social media data in the proper way will gain competitive advantage, be it betting market, or any other purpose where prediction accuracy plays a key role. The prediction work will remain a hybrid effort by man and machine.

Problems of standard attribution modelling

Attribution modelling is like digital magic.

Introduction

Wow, so I’m reading a great piece by Funk and Nabout (2015) [1]. They outline the main problems of attribution modelling. By “standard”, I refer to the commonly used method of attribution modelling, most commonly known from Google Analytics.

Previously, I’ve addressed this issue in my digital marketing class by saying that the choice of an attribution model is arbitrary, i.e. marketers can freely decide whether it’s better to use e.g. last-click model or first-click model. But now I realized this is obviously a wrong approach — given that the impact of each touch-point can be estimated. There is much more depth to attribution modelling than the standard model leads you to believe.

Five problems of standard attribution modelling

So, here are the five problems by Funk and Nabout (2015).

1. Giving touch-points accurate credit

This is the main problem to me. The impact of touch-points on conversion value needs to be weighed but it is seemingly an arbitrary rather than a statistically valid choice (that is, until we consider advanced methods!). Therefore, there is no objective rank or “betterness” between different attribution models.

2. Disregard for time

The standard attribution model does not consider the time interval between touch-points – it can range anywhere from 30 minutes to 90 days, restricted only by cookie duration. Why does this matter? Because time generally matters in consumer behavior. For example, if there is a long interval between contacts A_t and A_t+1, it may be that the effect of the first contact was not very powerful to incite a return visit. Of course, one could also argue there is a reason not to consider time, because any differences arise due to discrepancy of the natural decision-making process of the consumers which results in unknown intervals. Ignoring time would then standardize the intervals. However, if we assume patterns in consumers’ decision-making process, as it is usually done by stating that “in our product category, the purchase process is short, usually under 30 days”, then addressing time differences could yield a better forecast, say we should expect a second contact to take place at a certain point in time given our model of consumer behavior.

3. Ignoring interaction types

The nature of the touch or interaction should be considered when modeling customer journey. The standard attribution model assigns conversion value for different channels based on clicks, but the type of interaction in channels might be mixed. For example, for one conversion you might get a view in Facebook and click in AdWords whereas another conversion might have the reverse. But are views and clicks equally valuable? Most marketers would not say so. However, they would also assign some credit to views – at least according to classic advertising theory, visibility has an impact on advertising performance. Therefore, the attribution model should also consider several interaction types and the impact each type has on conversion propensity.

4. Survivorship bias

As Funk and Nabout (2015) note, “the analysis does not compare successful and unsuccessful customer journeys, [but] only looks at the former.” This is essentially a case of survivor bias – we are unable to compare those touch-points that lead to a conversion to those that did not. By doing so, we could observe that a certain channel has a higher likelihood to be included in a conversion path [2] than another channel, i.e. its weight should be higher and proportional to its ability to produce lift in the conversion rate. Excluding information on unsuccessful interaction, we risk getting Type I and Type II errors – that is, false negatives and positives.

5. Exclusion of offline data

The standard attribution model does not consider offline interactions. But research shows multi-channel consumer behavior is highly prevalent. The lack of data on these interactions is the major reason behind exclusion, but the the same it restricts the usefulness of attribution modelling to ecommerce context. Most companies, therefore, are not getting accurate information with attribution modelling beyond the online environment. And, as I’ve argued in my class, word-of-mouth is not included in the standard model either, and that is a major issue for accuracy, especially considering social media. Even if we want to measure the performance of advertising channel, social media ads have a distinct social component – they are shared and commented on, which results in additional interactions that should be considered when modeling customer journey.

Solutions

I’m still finishing reading the original article, but had to write these few lines because the points I encountered were poignant. Next I’m sure they will propose solutions, and I may update this article afterwards. At this point, I can only state two solutions that readily come to mind: 1) the use of conversion rate (CVR) as an attribution parameter — it’s a global metric and thus escapes survivorship bias; and 2) Universal Analytics, i.e. using methods such as Google’s Measurement Protocol to capture offline interactions. As someone smart said, solution to a problem leads to a new problem and that’s the case here as well — there needs to a universal identifier (“User ID” in Google’s terms) to associate online and offline interactions. In practice, this requires registration.

Conclusion

The criticism applies to standard attribution modeling, e.g. to how it is done in Google Analytics. There might be additional issues not included in the paper, such as aggregate data — to perform any type of statistical analysis, click-stream data is a must have. Also, a relevant question is: How do touch-points influence one another? And how to model that influence? Beyond technicalities, it is important for managers to understand the general limitations of current methods of attribution modelling and seek solutions in their own organizations to overcome them.

References

[1] Funk, B., & Abou Nabout, N. (2016). Cross-Channel Real-Time Response Analysis. in O. Busch (Hrsg.), Programmatic Advertising: The Successful Transformation to Automated, Data-Driven Marketing in Real-Time. (S. 141-151). Springer-Verlag.

[2] Conversion path and customer journey are essentially referring to the same thing; perhaps with the distinction that conversion path is typically considered to be digital while customer journey has a multichannel meaning.