What do “proven” treatment results mean?

I’ve actually been receiving Botox for Migraine since 12/14, and for me it works very well. The purpose of this article is that study design can be shady and/or flawed regardless of whether or not the treatment is effective.

With migraine studies types of sufferers are rarely separated into groups, which is why I feel most study results are lackluster. There are definitely different groupings of migraine sufferers in terms of their trigger types, and migraine types. If these started being classified or tracked even on an experimental basis I think we’d start seeing some big advances in migraine treatment.

There have been some great blog articles recently directed towards finding good medical information on the internet. Marla Jan at Luck Fupus did an article1 on MedNexus2, a search engine of vetted medical information. The Counting My Spoons did a great article3 on DISCERN4, a tool that can help a reader evaluate the accuracy of information. So I figured I’d join the pack in the effort to help you improve your relationship with Dr. Google.

This is no surprise, but I am an unapologetic research nerd. I read studies for fun, and rely on them when making treatment decisions for my clients. I’ve done multiple college level, major specific, and graduate level statistics and experimental design courses. Did I mention I was a Serials librarian all through college? I have worshipped, sniffed, I mean admired 100 year old issues of Lancet, and schooled PhD candidates on how to do literature reviews.

I care A LOT about where my information comes from, and even more about yours! So I figured I’d take a stab at putting together a series on health information. A few things I find particularly important are how to find and identify trustworthy sources, how to read a study, and what makes a good study. But we’ll start with:

What does it mean when a treatment has “Proven Results”?

Don’t we all want to know? From the pharmaceutical ads on the TV, to that ancient blood pressure drug that your doctor swears has been proven to improve migraines, everyone is peddling that they get results. Sure, these results come from studies and there are all these fancy claims and numbers. Since I always find it easier to understand things with examples, I’ve decided to go with Allergan’s claims that Botox has “proven results” with chronic migraine. Here is how they got to make that claim, and what it actually means. I promise there are some twists!


This is a screen cap from Allergan’s website5, notice the smaller the print gets the less impressive all those numbers are… we’ll get to that.

OK! now onto study design 101:

Most medical studies on a prospective treatment test the difference between a group that received the prospective treatment, and a group that received a placebo (inert) treatment. Researchers define the study by identifying who the treatment is for in a way that is measurable. So in the case of Botox for migraine the treatment is defined as being for people who have 15 or more migraines a month. A group of chronic migraine sufferers is recruited and then randomly assigned to the Botox group, or the placebo group. Both groups receive injections but only the treatment group gets Botox, the placebo group instead gets an inert substance.

A 24 week trial is conducted and the number of migraines a month is tracked. In the end there is the average reduction of migraines in the treatment group at the end of 24 weeks which was 9 migraines, the average reduction of migraines in the non treatment group was 7 migraines. This means two things

  1. The Botox group only had clinical significance of a reduction of two migraine days a month.
  2. The placebo group had a reduction of 7 migraines, meaning that there is a notable placebo effect when it comes to treatment with Botox. BUT up to 7 of those migraines were prevented just because you believe *PRAISE BOTOX*!

To calculate statistical significance researchers then use a fancy set of complicated equations to see where the difference between the control group and the treatment group fall on a bell curve. The bell curve represents the normal distribution of data, which is essentially the chance that there is a difference between the two groups. The center and highest part of the curve represents is the point where there would be no difference in the mean of the two groups. The tail of the curve is where the difference between the two means would be the greatest.

The difference between the mean data of control group and the treatment group is calculated as a point on the curve into what is known as a p value, the further away that value is from the high point of curve represents the probability of that result occurring by chance. Most studies require a p value between .01 and .05, meaning that there is a 1% to 5% probability that the difference between the two groups occurred by chance. In regular english statistical significance means the results of the study weren’t dumb luck.

However, statistical significance is different than the size of the effect. In the example of Botox it’s statistically significant, however, that is at average difference of 2 headaches per month between the group that got Botox and the group that didn’t. So the takeaway is that it is likely that a significant number of migraine sufferers will have A benefit from Botox. However, since the average of migraines prevented was only 2, a far fewer number will have a large reduction in migraines. This doesn’t mean a treatment option isn’t worth trying it is likely that some people in the study had a huge result, but the percentage of people who had 15 migraines a month prevented by Botox is far lower than 95-99%.

The FDA concurs with this explanation of the study results6:

Overall, this study suggests that an improvement does occur with Botox treatment, but that it is a rather modest amount of improvement. Substantially more of the disease symptoms are left unrelieved by Botox treatment than are relieved. The tertiary endpoint of the Activities of Daily Living items further underscores the clinical value of the observed effects. No significant effects were observed in the ADL questions.

Botox did not woo the FDA, in fact the FDA had to reject the majority of the studies Allergan had done on Botox due to bad design. The FDA report reads like a juicy season of The Bachelor, well for me at least. We’ll get to that next time when I nerdily ramble on about what nasty things study design can tell us.

Article Links and References:

Share Your Thoughts

To Comment: click the lock, and when it turns green press the Submit Button.