Technical

The fundamental problem of causal inference You probably know the COVID-19 pandemic and the race between all the pharma companies to create a vaccine. There was a lot of talk about different phases of clinical trials and the effectiveness of various vaccines. In clinical trials, companies are trying to estimate the causal effect of giving a drug on patient’s health. Causal effect is the difference between what happened if the treatment was given and what would have happened if the treatment was not given ...

Central limit theorem states that sample means follow normal distribution. People often confuse this with and question the validity of uniform distribution of p-values under nul hypothesis. Should we not be observing extreme p-values much less ofter than usual pvalues like 0.2 or 0.3? Imagine if null hypothesis is true. This means there is no difference betweem treatment and control. import numpy as np import pandas as pd import statsmodels.api as sm from scipy import stats import matplotlib.pyplot as plt pval_lis = [] t_stat_lis = [] samp_mean_lis = [] for i in range(100000): sample = np.random.normal(0, 1, 10) samp_mean = sample.mean() samp_mean_lis.append(samp_mean) t_stat = sample.mean()/(sample.std()/np.sqrt(sample.size)) t_stat_lis.append(t_stat) pval = stats.t.sf(abs(t_stat), df=sample.size)*2 pval_lis.append(pval) plt.hist(np.array(samp_mean_lis)) Distribution of sample means ...

Practical significance is the North Star. 🌟 Why experiment? At the end of the day we want to know if treatment is better than control and whether changing the status quo is worth it. That is why we do experimentation or derive inference from observational data. Being better is not enough to take a practical decision. Is it worth changing the status quo? Both go in conjunction. That’s where practical significance comes in. ...

The first step is to nail their intuition for randomness. Here’s how I will do it. I will ask them if I flip a coin once, will it be heads or tails. The answer would be It will be heads! or It can be either! (smart kid). If it is former, I can prove them wrong in 1-3 tries and lay the path for explaining randomness. If it is later, they already have some intuition for randomness. ...

💡 This is a humor piece. I am using the scenario to communicate the use of MAB’s. Please take it with a pinch of salt. A manager at a tech company that sells Alphonso mangoes online is stuck with one of the most difficult decisions. What should the color of the “Buy Now” button be? She recently came to know that data-driven decision making is the new fad and wants to follow suit. She has a meeting with her team of data scientists and marketing folks. ...

Thompson Sampling is one of the most popular Multi-Armed bandit (MAB) algorithms - the main reason being its explainability (imagine explaining upper confidence bound to your manager) and decent performance in practice [1]. Many blog posts on the Internet show how to implement Thompson sampling (here, here, here and here). Almost all of them consider Bernoulli outcome distribution (e.g. click or no click, purchase, or no purchase) and use the Beta-Bernoulli Bayesian update procedure for simulations and usually compare the performance (Regret) to other MAB algorithms like UCB or sometimes to A/B testing. However, none of them consider Gaussian outcome distribution and especially the case when both mean and variance of the distributions are unknown (which is usually the case when you are conducting an experiment where the outcome is continuous, ex: dollars spent, time spent, etc.). ...

Over the years, I have come across multiple resources I learned things from with ease. This is a page to track those resources. Please reach out to me if you think something belongs to this list. I will add it. General Julia Evans has online zines for topics ranging from SQL, Shell scripts to Linux. These are hand-drawn explanations of stuff related to a particular topic. I stumbled upon here twitter post about how SQL actually executes a query and have been a fan ever since. ...