As an undergraduate, I took both a literature and biology degree and then, for a couple years after college, I worked in a lab. The head of the lab told me a story that has stuck with me and, oddly, has been relevant for marketing.

All cells have a molecule called ATP. The cell cannot function without ATP. It is a way for the cell to store energy.

Some scientists were running experiments in which they were adding a solution of ATP to the cells. The solution was purchased from a company and was nearly perfectly pure. The scientists were seeing extremely interesting results. They constructed elegant theories as to how the ATP was interacting with the cells and producing these interesting results.

Then the company came out with a 100% pure solution of ATP. Everyone was very excited and rushed to try this 100% pure ATP solution.

The experiments stopped working.

After, no doubt, much cursing and some additional experiments, the scientists realized the results that they were seeing weren't caused by the ATP but by an impurity in the original solution.

The moral, as a marketer, I derive from this is that the data can be real, but what causes that data is often an extrapolation. In the same way that impurities in the solution tripped up the scientists, something unknown about the audience (or how the data is gathered) can trip up marketers.

(By the way, if anyone thinks this is a back-handed complaint against science or the scientific method, it's not. This story is an example of exactly why science works. The scientists did not rest on their laurels with the almost-pure solution; one set of results did not prove it was "true." They kept testing and ended up with a new insight they would have missed if they had not.)

P.S. Looking over some previously unopened emails from Marketing Experiments Journal, I came across this case study: Optimization Testing Tested. It's a review of test validity for optimizing webpages and looks at some of the ways data can be misleading. Interesting stuff especially as they break down where potential errors could occur.