It’s almost impossible to beat Facebook’s or Google’s algorithms in ad optimization, because they have access to individual-level data whereas the advertiser only gets aggregates, and even their supply is limited. But, there are two opportunities I see which Google and Facebook don’t handle:
1. Use of CRM data
Especially purchase history (=lifetime value), product margins (=profitability), and other customer information that can be used for user modelling or machine learning as features. But, don’t use Google Analytics for linking this data to website analytics — Google Analytics sucks, because Google keeps individual-level information (=click-stream data) for itself and only shares, again, aggregates. Use Piwik instead.
2. Use of cross-platform data
Google doesn’t have access to Facebook’s data or vice versa, but the advertiser has. Thus, you can create more comprehensive optimization models for bidding and budgeting.