There’s an inherent problem that occurs when A/B testing the results of a new page design compared to your old one. Keep in mind, this doesn’t apply when A/B testing two designs that launch at the same time.
It also doesn’t apply if 100% of your current traffic hasn’t seen your old site.
So here is the scenario:
Let’s call this the car vs bicycle scenario. If a person is used to riding a bicycle and has never seen a car before and you place the two side by side, the person may be able to get from point a to point b faster using the bicycle. This person may also be more likely to choose the bicycle than the car. But once they learn how to use the car, they may choose that vehicle and it may be faster for them.
So if you have your old page that people are used to, even if it’s counter intuitive to unfamiliar visitors, your current audience may skew the results of a conversion test that is run over a one week period because they haven’t yet educated themselves to your new design.
The second problem that occurs is that you’ve probably worked out all of the bugs and kinks in your old design, especially if it’s been up for a long time. But you probably haven’t on your new design. So if one design has a few bugs or kinks in certain browsers during your test, but those have been worked out in your new design, your results may be skewed.