A new rule under consideration by the FTC (see also here) will require that ads with customer testimonials show typical results, not just best-case outcomes.

Of course, following best practices in data visualization would mean you should show the central tendency and the variability (in all directions). I’m not holding my breath for density plots on the nightly news, though. A single, typical exemplar would still be an improvement over a single, cherrypicked extreme.

However… Maybe I’m too jaded, but I wonder about unintended consequences. For example, will there be a flood of crappy research after this rule? If companies are required to depict “typical” results, they may churn out poorly designed studies to get the numbers they want, hoping to lend more credibility to bogus products. And if these studies are marketed as “scientific” and then easily (and publicly) disputed, that could feed into kneejerk cynicism in the public about science more broadly.

Consider that FDA clinical trials are one of the most highly regulated forms of research around, with numerous checks and balances designed to ensure integrity.  The system mostly works, but there are still serious concerns about conflicts of interest. How well is the FTC going to ensure the quality of research on consumer products, herbal supplements, diet plans, and the like? Will there be independent investigators, peer review, mandatory publication of negative results, etc.?