Technical

Central limit theorem states that sample means follow normal distribution. People often confuse this with and question the validity of uniform distribution of p-values under nul hypothesis. Should we not be observing extreme p-values much less ofter than usual pvalues like 0.2 or 0.3? Imagine if null hypothesis is true. This means there is no difference betweem treatment and control. import numpy as np import pandas as pd import statsmodels.api as sm from scipy import stats import matplotlib.pyplot as plt pval_lis = [] t_stat_lis = [] samp_mean_lis = [] for i in range(100000): sample = np.random.normal(0, 1, 10) samp_mean = sample.mean() samp_mean_lis.append(samp_mean) t_stat = sample.mean()/(sample.std()/np.sqrt(sample.size)) t_stat_lis.append(t_stat) pval = stats.t.sf(abs(t_stat), df=sample.size)*2 pval_lis.append(pval) plt.hist(np.array(samp_mean_lis)) Distribution of sample means ...

Practical significance is the North Star. 🌟 Why experiment? At the end of the day we want to know if treatment is better than control and whether changing the status quo is worth it. That is why we do experimentation or derive inference from observational data. Being better is not enough to take a practical decision. Is it worth changing the status quo? Both go in conjunction. That’s where practical significance comes in. ...

The first step is to nail their intuition for randomness. Here’s how I will do it. I will ask them if I flip a coin once, will it be heads or tails. The answer would be It will be heads! or It can be either! (smart kid). If it is former, I can prove them wrong in 1-3 tries and lay the path for explaining randomness. If it is later, they already have some intuition for randomness. ...

💡 This is a humor piece. I am using the scenario to communicate the use of MAB’s. Please take it with a pinch of salt. A manager at a tech company that sells Alphonso mangoes online is stuck with one of the most difficult decisions. What should the color of the “Buy Now” button be? She recently came to know that data-driven decision making is the new fad and wants to follow suit. She has a meeting with her team of data scientists and marketing folks. ...

Thompson Sampling is one of the most popular Multi-Armed bandit (MAB) algorithms - the main reason being its explainability (imagine explaining upper confidence bound to your manager) and decent performance in practice [1]. Many blog posts on the Internet show how to implement Thompson sampling (here, here, here and here). Almost all of them consider Bernoulli outcome distribution (e.g. click or no click, purchase, or no purchase) and use the Beta-Bernoulli Bayesian update procedure for simulations and usually compare the performance (Regret) to other MAB algorithms like UCB or sometimes to A/B testing. However, none of them consider Gaussian outcome distribution and especially the case when both mean and variance of the distributions are unknown (which is usually the case when you are conducting an experiment where the outcome is continuous, ex: dollars spent, time spent, etc.). ...

You probably know the COVID-19 pandemic and the race between all the pharma companies to create a vaccine. There was a lot of talk about different phases of clinical trials and the effectiveness of various vaccines. In clinical trials, companies are trying to estimate the causal effect of giving a drug on patient’s health. Causal effect is the difference between what happened if the treatment was given and what would have happened if the treatment was not given ...

Let’s take a look at this chart. There is a perfect association between ice cream sales and shark attacks at the beach. When ice cream sales go up, shark attacks go up and when ice cream sales go down, shark attacks go down. If we were to take this association seriously, in order to lessen shark attacks for public safety, we would cut down on the number of ice cream shops at the beach. Of course, this would not work. The hidden story is that more people go to the beach in the summer and this increases both the probaility of a shark attack and ice cream sales. In order to attribute the effect of ice cream sales on shark attacks, we need to establish that the effect is because of the ice cream sales, not just associated with ice cream sales. ...

Going back to the red car example, how would we establish the causal effect of red colored car on accidents. In an ideal scenario, we select a random set of drivers and give them a red car to drive for x days and record the total number of accidents. We then go back in the time machine and give the exact set of drivers a grey car to driver and see if the accidents are less. This would give us a causal effect. But unfortunately, this is not possible. ...

Over the years, I have come across multiple resources I learned things from with ease. This is a page to track those resources. Please reach out to me if you think something belongs to this list. I will add it. General Julia Evans has online zines for topics ranging from SQL, Shell scripts to Linux. These are hand-drawn explanations of stuff related to a particular topic. I stumbled upon here twitter post about how SQL actually executes a query and have been a fan ever since. ...