When Perception and Reality Clash in Data Analysis

We have all heard the saying "perception is reality." At first, it sounds really deep and even a bit magical, as if our personal view of the world can shape the truth itself. But if we stop for a moment and think, we will quickly realise that our perceptions are influenced by our experiences, biases, and even our mood on a dull day. Reality, however, is an absolute. It simply exists regardless of how we choose to see it. The challenge arises when we begin to mix up our own interpretation of reality with what is actually true. Just because something feels real doesn’t mean to say it is.

Over the next few weeks, I will be writing a number of blogs around the idea that “Perception is Reality.” We will see how this approach can cause problems in a number of different ways. But for now, our focus is on data analysis and statistics. How does our mismatch between perception and reality affect the way we look at our data?

You see, numbers and figures are supposed to tell us the truth. Yet, the way we often interpret these numbers can lead to a very different story from what is actually going on. And this disconnect can have serious implications. Throughout this blog, we’ll explore the concepts of perception and reality and how they affect our understanding of data. We’ll look at p values, confidence intervals, power, sensitivity, specificity, and the inevitable Type 1 and Type 2 errors. We'll use health-related examples to illustrate these ideas and, finally, reflect on how perception and reality compare.

The Perception of "Significant Results"

Imagine you have conducted a study to test whether a new drug can lower cholesterol. After many long hours of gathering data and crunching numbers, you calculate a p value of 0.03. At first, you might be tempted to think, "This result is statistically significant, and therefore the drug works!" In many scientific communities, a p value of less than 0.05 is taken as a green light that the result is not due to random chance.

However, this perception can be misleading. A statistically significant result simply means that there is a low probability that the observed effect happened by chance. It doesn’t necessarily mean that the drug has a large or even meaningful effect on cholesterol levels. For instance, the drug might lower cholesterol by only a very small amount - just enough to move the needle, but not enough to improve a patient’s overall health. So, while our perception based on a p value of 0.03 suggests something important has been discovered, the reality may be far less exciting.

Working at 95 Percent Confidence

Let’s look a little deeper into p values and confidence intervals - two staples of statistical analysis. A p value tells us the probability that the data we see would occur if there were no real effect. When we use a p value threshold of 0.05, we’re essentially saying we’re 95 percent confident in our result. This means that even if our p value appears convincing, there is always a 5 percent chance that our result is a statistical fluke rather than a reflection of reality.

Confidence intervals give us more context by providing a range in which we expect the true effect to lie. Imagine that in our cholesterol study, the confidence interval for the drug’s effect ranges from 5 percent to 15 percent. We say we are 95 percent confident that the true effect falls within this range. However, just as with p values, there is still a 5 percent chance that the true effect lies outside this interval. Both p values and confidence intervals remind us that our conclusions come with uncertainty. Our perception of certainty is, therefore, not absolute; reality remains more unpredictable than the numbers might suggest.

The Role of Power, Sensitivity, and Specificity

Looking beyond p values and confidence intervals, we also need to consider the concepts of power, sensitivity, and specificity when interpreting data. These help us gauge how reliable our tests are, and how closely they reflect what’s really going on.

Power is a study’s ability to detect a real effect when it exists. Think of it as the study’s capacity to spot something if it’s truly there. A study with high power is likely to uncover true effects, while one with low power might miss them entirely. For example, if a new diagnostic tool for breast cancer has low power, it might fail to detect many true cases - leading us to believe the tool is ineffective, when in fact it’s just not sensitive enough.

Sensitivity refers to a test’s ability to correctly identify people who have a condition (true positives). Specificity measures how well a test correctly identifies people who do not have the condition (true negatives). Suppose a new blood test is introduced for a rare disease. If the test has low sensitivity, many patients with the disease might be missed. If it has low specificity, healthy people might be wrongly diagnosed. These issues shape our perception of a test’s accuracy, even when the real-world performance tells a more complicated story.

Type 1 and Type 2 Errors

Even with all these statistical tools, there's always room for error.

A Type 1 error happens when we conclude that an effect exists when it doesn't. It’s like thinking you’ve recognised someone across the street, only to realise you’re looking at your own reflection in a shop window.

A Type 2 error occurs when we fail to detect an effect that’s actually there. This is like missing someone waving at you in a crowd because you weren’t paying attention.

These errors highlight that statistical tests always involve some uncertainty. Our perception of a result - whether we think it’s meaningful or not - might be off, simply because we’re working in a world where mistakes are built into the process.

When Numbers Matter

To bring this into focus, imagine a clinical trial testing a new treatment for depression. Suppose the study reports a statistically significant improvement with a p value of 0.04, and the confidence interval for the improvement in scores ranges from 1 to 8 points on a 30-point scale. On paper, these numbers suggest that the treatment works.

But a one-point improvement might not be enough to lift a patient out of depression in a meaningful way. Our perception, driven by the statistical significance, might lead us to overestimate the treatment's value, while in practice, the benefit could be too small to matter.

Now consider a diagnostic test for a certain type of cancer. Suppose the test has 80 percent sensitivity and 90 percent specificity. In a large population, these figures mean that some patients with cancer won’t be detected, and some healthy people might be misdiagnosed. Our perception may be that the test is reliable because of the high percentages, yet in real terms, the number of errors could still be significant - and carry real consequences for patient care.

Moving Beyond Blind Faith in the Numbers

So, how do we deal with the mismatch between perception and reality when it comes to data?

First, we have to acknowledge that tools like p values, confidence intervals, power, sensitivity, and specificity all come with limitations. They aren’t truths - they’re indicators. They give us clues, not certainties.

That means we can’t rely on them blindly. We need to look at the context of the study. Was the effect large enough to matter? Was the design solid? Do the findings translate to meaningful outcomes in practice?

In research, we shouldn’t rush to celebrate every statistically significant result. We need to ask: does this result actually make a difference? Was the study capable of detecting that difference in the first place? And are our interpretations based on the numbers themselves - or just on what we expected them to say?

Reflecting on Reality versus Perception

The idea that "perception is reality" is a tempting one. It’s neat, reassuring, and sounds like it should be true. But in the world of data analysis and statistics, it rarely holds up.

Our perceptions of results - whether we’re talking about p values, confidence intervals, power, or error - can feel like solid conclusions. But those numbers always carry a degree of uncertainty. A 95 percent level of confidence means there’s still a chance we’re wrong.

As we’ve seen through health-related examples, the gap between perception and reality can have practical consequences. While statistical tools can help guide our thinking, they’re not the final word. The truth behind the numbers is often messier, and more nuanced, than our instincts would have us believe.

In the end, being honest about that uncertainty is a strength, not a weakness. If we approach our analysis with curiosity and a healthy dose of scepticism, we stand a better chance of getting closer to the truth - even if it’s not quite what we expected to find.