Few months ago we planned to run a lift Test in RTB to understand the incremental value of Display and have a scientific approach to the Post-impression value. The testing framework was pretty simple: Trigger a Split Cookie Pool Test in GTM.
Cookie A <- Test Group RTB
Cookie B <- Control Group No RTB
The Cookie A was added in a custom dimension in DBM. We started collecting data and segmenting our Display campaigns only for the Cookie A users.
(A cookie is mainly a device a cookie is not and ID)
What we found? 25% of users Cookie B have seen RTB display banners.
After asking to Google and troubleshooting the cookie trigger code we were to able to determine that there was absolutely no issue in the test framework.
We understood that the 25% mismatch was coming from the cross-device. Altough we were aware that they will be some failure in the multi touch path we didn’t excepted a 25%.
The typlical failure path scenario:
Cookie B No RTB banner desktop -> Cookie A RTB Banner mobile.
In conclusion if your running a cookie splitting test, make sure you can understand your A/B discrepancy.