A/B testing is the foundation of data-driven decision making. In today's world, when advertising becomes crucial to a website's revenue it is even more important to measure the effects of changes correctly. We will show how to correctly design and implement an advertisement A/B testing. We will walk through pitfalls, potential biases related to advertisement metrics and possible mitigations.