Research Is Good. And Research Is Bad.

I've read a lot of marketing articles that praise the use of A/B testing, and other research methods to raise conversion levels on ads and emails. This is a great way to evaluate design effectiveness, and objectively design marketing pieces that produce sales. The downside is that the data is often interpreted incorrectly. There's a logical and statistical fallacy that occurs many times in the marketing world. It's called post hoc ergo propter hoc.

This just means that we assume the results after an action are caused by that action. For example, someone may observe a reaction to a color or placement which is better than the reaction to a different color or placement. That person can conclude that the reason for this is the color and placement itself caused the reaction. This is a mistake. True, we can probably get great initial results according to the research testing — and most conversion experts agree that it doesn't always matter why. However, it does matter why if we really want an accurate understanding of our audiences' reactions to a design approach.

For example, someone may conclude that red buttons are just better for conversion rates. And making the button big, placed "above the fold", and adding a bit of animation will get more clicks than a light blue button. But what isn't said is that there are other factors influencing the results, namely context. Often the colors, text, content, and design surrounding the button has a bigger influence than the button itself. And consider the context of the messaging and calls-to-action. Hard sell pitches can be perceived more negatively for some non-profits than an optimistic message and design. Of course this assumption should be tested, but now we are testing the right things — rather than disembodied design elements.