# "Extraodinary Claims"

Written by Paul Sutter on Tuesday, 06 March 2018. Posted in From The Desk of...The Chief Scientist

You've probably encountered the phrase "extraordinary claims require extraordinary evidence." Well, it's not just some pithy saying that relieves you of the burden of having to believe every random statement you might encounter in your life. It's actually a handy summary of a very powerful approach to statistics.

The heart here is something called Bayes' theorem, named after the Reverend Thomas Bayes, who was kicking around cool ideas in the first half of the eighteenth century. It relates some statement you're trying to test ("giant stars die as supernovas", "jelly beans cause cancer", "aliens visit Earth because their star exploded and they're going to steal our jelly beans", etc.) to the individual probabilities of each component (the number of stars you observe dying, the number of people eating jelly beans, and so on).

Of course I'm radically simplifying this, but the key takeaway is that Bayesian statistics folds in your existing knowledge of the problem directly into the math, and provides a way to create outcomes that are an updated view of the world based on any new information or experiments. Thus if you don't have a lot of info already, a quick experiment can start to point you in the right direction. But if there is already a huge amount of knowledge about a relationship, it will take a tremendous amount of good evidence to change that perspective.

Unfortunately while simple to state the application of Bayesian statistics gets really complicated really fast, and it's only recently that we've had the computational power to crunch through the intense calculations involved. Which is good. The more we move away from simple-to-apply but simple-to-abuse methods (*cough* p-values *cough*), the better.

Written by Paul Sutter on Monday, 26 February 2018. Posted in From The Desk of...The Chief Scientist

Recently I was invited to appear on Midnight in the Desert, a late-night radio show hosted by Heather Wade and usually featuring topics such as UFOs, time travel, energy beings, bigfoots (bigfeet?), and more. Honestly, I was torn. One on hand, I tend to distance myself from those subjects - I'm here to communicate what we know about the universe through science, after all. And those topics are...well, less than rigorous.

On the other hand - and the reason I ultimately accepted the offer - this is a new audience. I could stay in my comfort zone and talk to the same people who accept the same worldview that I espouse, or I could take a chance and speak to people who might be unaccepting of, and perhaps even hostile to, my values.

It was a risk - personally, professionally - so I took an approach of radical empathy. At the top of the interview I said I respected the host and the listeners, and I wasn't there to tell people what to believe. The folks I spoke to had deep, sincere, heartfelt, emotional experiences. Who was I to stomp all over that?

Instead I explained why I, as a scientist, must maintain a high skeptical bar for my own beliefs, and how we could use personal experiences to explore all sorts of cool physics about the world around us and test the limits of our knowledge. Swapping stories, if you will.

Now I'm not trying to hold myself up as some paragon of science communication virtue, but I took a risk and it paid off.

The host and callers were a blast to chat with, and the show was a wonderful opportunity to share some science. At the end of the broadcast, Heather told me that I was the first scientist to come on the program and not act smug, contemptuous, and condescending towards the topics and listeners. I was disappointed to hear that, but not altogether surprised. I happily accepted her invitation for another visit in the future.

Food for thought.

# "Statistics Brought to You by the Letter P"

Written by Paul Sutter on Sunday, 18 February 2018. Posted in From The Desk of...The Chief Scientist

Last week I mentioned an odd term, p-value, which is commonly used in deciding whether your results are worth mentioning to your colleagues and the public. Of course it has a strict and narrow meaning, and of course that meaning is abused and misinterpreted in discussions about science.

Let's say you're performing an experiment: a pregnancy test. The box claims 95% accuracy, and if you read the fine print it's referring to a p-value of 5%. You take the test and...it's positive! So are you really pregnant, or not?

Unfortunately your urgent question hasn't yet been answered. A p-value compares the hypothesis you're testing ("I'm pregnant") to what's called a null hypothesis (in this case, "no baby"), and a p-value of 5% says that if you were *not* pregnant, there would only be a 5% chance that the test would return a positive result.

You might be tempted to flip this around and state that there's a 95% chance you're actually pregnant, but you would be committing an egregious statistical sin - and this is the same sin committed wittingly or unwittingly by science communicators and sometimes scientists themselves.

Here's the problem: what if you're male? The test can still say you are, because it's not answering the question "am I pregnant?" but rather "if I'm not pregnant, what are the chances of the test returning a positive result?" It's a low number - 5% - but not zero. Thus males can still get a positive result despite never being pregnant.

The p-value by itself was only ever intended to be a "let's keep digging" guidepost, not a threshold for believability. To answer the question you actually want answered, you have to fold in prior knowledge. Combined with a low p-value, a healthy female of reproductive age can begin to conclude that there might be a baby on the way. A male...not so much. In either case, the p-value alone wasn't enough, and announcements based solely on that number need to be viewed suspiciously.