Practical significance is the North Star

Why experiment?

At the end of the day we want to know if treatment is better than control and whether changing the status quo is worth it. That is why we do experimentation or derive inference from observational data. Being better is not enough to take a practical decision. Is it worth changing the status quo? Both go in conjunction. That's where practical significance comes in.

Scenarios

An algorithmic change in google search backend improved the average load time by 0.01ms. The experiment had sample size of 1M. The effect size is too small and the sample size too large. Who cares! The change will make millions for Google as billion searches happen everyday.

The new marketing campaign has doubled the sign up rates from 10% to 20%. The experiment has decent sample size but P value of 0.07. Again, the risk of changing status quo is probably worth it. Go ahead. Do it.

Our new drug improved the recovery rate by 0.05%. Trial is statistically significant. We should roll out the new drug. May be not. The effect size is too small to justify the process of approval, changing manufacturing pipeline and the whole shebang. It may not practically significant.

Conclusion

We are too caught up on P Value cutoffs and discrediting the effects because of high sample sizes. But the decision should be made based on practical significance and whether changing the status quo is worth it.