Here’s a small case study.
We observed irrational behavior from Facebook ads. We have two ad versions running; but the one with lower CTR gets a better relevance score and lower CPC.
This seems like an irrational outcome, because in my understanding, CTR as a measure of relevance should be largest impact factor to CPC and Relevance Score.
Figure 1 Aggregate data
So, we dug a little bit futher and did a breakdown of the data. It turns out, the ad version with lower aggregate CTR performs better on mobile. Apparently this adds emphasis to the algorithm’s calculation.
Figure 2 Breakdown data
Lesson learned: Always dig in deeper to understand aggregate numbers. (If you’re interested in learning more about aggregate data problems, do a lookup on “Simpson’s paradox”.)