TwoSample2

Summary

In this engaging tutorial by Erin Heerey, we delve into the mechanics of the two-sample t-test, used to determine if the differences between two independent sample means are statistically significant. The video explores the concept of treating mean differences as signals and sampling errors as noise, and how these principles are applied to understand the t-statistic formula. The process includes discussing the creation of the sampling distribution of differences between sample means and calculating the theoretical variance. Additionally, Erin explains how to handle varying sample sizes and the importance of examining data visually to detect anomalies that could affect outcomes. The tutorial is rounded off with best practices for hypothesis testing, ensuring viewers have a comprehensive understanding of the t-test.

Highlights

• Erin Heerey breaks down the two-sample t-test into understandable components. 🎓
• Learn the importance of the sampling distribution of differences and how it influences variance calculations. 📈
• Understanding t-statistic involves comparing mean differences to expected noise, factoring in sample size and variance. ⚖️

Key Takeaways

• The two-sample t-test checks if sample mean differences are statistically significant by comparing signal and noise. 🧠
• Sampling distribution for differences between means is key in understanding variance and calculating t-statistic. 📊
• Consider the homogeneity of variance and examine data visually to identify outliers before statistical testing. 👀

Overview

In this tutorial, Erin Heerey guides us through the intricacies of the two-sample t-test, a statistical method used to determine whether the difference between the means of two independent samples is meaningful. The test serves as a 'signal-to-noise' ratio, where the difference between means is the signal, and sampling error is considered noise.

The video emphasizes the role of the sampling distribution of differences between sample means and variance calculations. This distribution helps in assessing whether observed mean differences are a result of random variation or genuine effect. Understanding this concept is crucial for anyone performing a t-test.

Moreover, Erin stresses the importance of verifying assumptions such as homogeneity of variance and checking for outliers by plotting data. By following these best practices, researchers can avoid misinterpretations and ensure the robustness of their hypothesis testing efforts.

Chapters

• 00:00 - 00:30: Introduction to T-Test The chapter "Introduction to T-Test" introduces the concept of a t-test, explaining that it provides a method to assess whether the difference in the means of two independent samples is significant enough to rule out sampling error as the cause. It highlights that the t-test evaluates if the difference observed is unlikely due to random sample variation. The t-test operates similarly to a single sample t-test, functioning essentially as a signal to noise ratio.
• 00:30 - 01:00: Signal and Noise Quantification The chapter 'Signal and Noise Quantification' explains the need to measure two key components: the 'signal' and the 'noise.' The 'signal' is defined as the difference between two means, while the 'noise' refers to the extent of sampling error present in the estimates. A critical point raised is that the noise sampling distribution is complex and not merely an average distribution of all data scores, but something more intricate.
• 01:00 - 01:30: Sampling Distribution of Differences The chapter discusses the concept of the sampling distribution of differences between sample means. It begins by addressing the null hypothesis as a critical element in understanding this statistical measure.
• 01:30 - 02:00: Understanding the Null Hypothesis The chapter 'Understanding the Null Hypothesis' explains that when the null hypothesis is true, it indicates that the mean of Group 1 equals the mean of Group 2. Alternatively, this can be expressed as the difference between the means of the two groups being zero. The chapter reinforces this concept by referencing prior lessons on the T distribution, different samples, and their degrees of freedom, highlighting the observation that the mean difference of these groups is zero when the null hypothesis holds.
• 02:00 - 02:30: Sampling Variability and Mean Differences The chapter discusses the concept of sampling variability and its impact on mean differences. It explains that in a scenario where there is no difference between the sample means, the distribution should ideally be zero, but random sampling introduces variability. To understand how close the means are to zero, a sampling distribution of mean differences is created, highlighting the variability inherent in statistical sampling processes.
• 02:30 - 03:00: Creating Sampling Distribution of Differences The chapter discusses the process of creating a sampling distribution of differences between sample means. It involves taking samples from two populations, calculating the mean for each, determining the difference between these means, and recording the value. This process is repeated multiple times to construct the distribution. The chapter also hints at the need to determine the variance of this distribution.
• 03:00 - 03:30: Variance and Distribution of Differences The chapter discusses a comparison of two sample means, one equal to 10 and the other equal to 12, highlighting the difference between the two as 2. It explains that both sample means are derived from Monte Carlo samples with a defined population mean. Despite the difference in means, both samples have the same standard deviation, with the variance described as random variance.
• 03:30 - 04:00: T-Statistic Formula While averaging might seem intuitive, it's essential to compute the difference between means. The variance of the distribution of differences between means is the sum of the variances of the individual distributions. This consideration is because the focus is on a distribution of differences.
• 04:00 - 04:30: Standard Error of the Difference The chapter discusses the concept of 'Standard Error of the Difference' by explaining the distribution of differences between two groups. It highlights that the control group has a mean value that is 2 units below the test group, resulting in a mean difference of -2. The distribution of these differences is wider compared to the distribution of individual groups. This broader distribution is because of the changes made in the comparison or experimental setup.
• 04:30 - 05:00: Hypothesis Testing Steps The chapter discusses the steps involved in hypothesis testing using two population samples. It explains the process of calculating means for each sample, determining the difference, and noting the difference to form a sampling distribution of differences. The example shows the mean centered around minus two because of the subtraction order (Group 1 minus Group 2), but it could be positive two if reversed. The chapter highlights that the variance of this distribution of differences is significantly larger than the variance of individual distributions.
• 05:00 - 05:30: Degrees of Freedom and Test Statistic This chapter discusses the T distribution or T statistic, which involves comparing the means of two groups. The T statistic is calculated by taking the difference between the mean of Group 1 and the mean of Group 2, and then dividing it by the square root of the variance of Group 1 divided by the number of scores in Group 1 plus the variance of Group 2 divided by the number of scores in Group 2. This essentially measures how much the two group means deviate from each other, taking into account the variability and sample size within each group.
• 05:30 - 06:00: Interpreting T-Test Results This chapter covers the process of interpreting T-test results, emphasizing the calculations needed to compare two groups. It outlines how to compute the means and variances of each group and then describes the mathematical operations necessary to compare these values. The chapter explains the formula used to calculate the T-test statistic, involving the means of both groups, their variances, and participant numbers. The transcript provides an example of these calculations and highlights the need for additional math to arrive at the final result.
• 06:00 - 06:30: Importance of Data Visualization The chapter titled 'Importance of Data Visualization' discusses the role of data visualization in understanding and analyzing data, with a specific focus on statistical tests like t-tests. It explains how data visualization can help in interpreting statistical results, specifically when dealing with noise metrics and variations in sample sizes due to technical errors or other factors. Thus, even with unequal sample sizes in study groups, data visualization remains a crucial tool in conveying accurate information.

TwoSample2 Transcription

• 00:00 - 00:30 so let's move on to the actual t-test so the t-test provides a formalized way to determine whether the difference in the means of two independent samples is large enough that sampling error cannot explain it so this is the difference large enough that it's unlikely to have been caused by random differences in sampling so the t-test just like we saw with single sample t-tests is essentially a signal to noise ratio and this means
• 00:30 - 01:00 that we need to quantify two things we need to quantify our signal which is the difference between the two means we also need to quantify our noise which is how much sampling error there is in the estimates but and here's where the tricky part comes in sampling distribution for the for the noise estimate that we need to think about is not a simple for example average distribution of all of the scores in the data instead the sampling distribution is the
• 01:00 - 01:30 sampling distribution of differences between sample means that's a lot to think about so the sampling distribution of the difference between the sample means is quite interesting let's start with the null hypothesis when we think about understanding this so we know that if the null hypothesis
• 01:30 - 02:00 is true the mean of Group 1 equals the mean of group 2. now we can say that another way we can say the mean of Group 1 minus the amino group two equals zero right if there's no difference between those means on average the mean of Group 1 minus the mean of group 2 equals zero now remember you saw in a previous lecture when you were looking at the T distribution from a sample from from different samples with different degrees of freedoms what you noticed was the mean of that
• 02:00 - 02:30 distribution was Zero because if there is no difference between the sample means on average that distance will be zero but we also know that random sampling will introduce variability into the means so the mean of group one minus the meaning group two probably won't equal exactly zero but it should be relatively close so to figure out how close we need to create a sampling distribution of mean differences that shows this
• 02:30 - 03:00 so what we do how to think about this is think about taking a sample from population one and a sample from population two calculating the difference between calculating their means first and then calculating the difference between those means and recording that value and then we repeat this lots and lots of times just the same way we built other sampling distributions and the distribution we end up with is called the sampling distribution of differences between sample means quite a mouthful then we have to figure out the variance
• 03:00 - 03:30 now let's say we have a sample mean where it's equal to 10 and a sample mean that's equal to 12. so we know that the difference between these sample means is two right because we've defined them that way these are both Monte Carlo samples so we've defined our population mean as a mean of 10 here and a mean of 12 here by the way these have the same standard deviation the variance you see is just random variance and so
• 03:30 - 04:00 it turns out that we can't just sort of average these what we have to do is we actually need to take the difference between the means turns out that the variance of the distribution of differences between the means equals the variance of the distribution of mean one plus the variance of the distribution of mean two and that's because this is a distribution of differences and what you'll see is that the
• 04:00 - 04:30 distribution of differences here there is on average our control group is minus 2 below our test group so if Group 1 minus group 2 that gives us a mean of minus two and so those the difference between those means is going to be distributed around that but look at how much bigger the distribution is than either of these two and that's because what we've done
• 04:30 - 05:00 here is we've taken a sample from this population and a sample from this population calculated the means calculated the difference and noted that difference down so this is a sampling distribution of the differences its mean is now centered on minus two mostly because we did at Group 1 minus group two and group two is bigger it could also have been positive two if we'd done the subtraction in the other way but you can see that the variance of this distribution is substantially bigger than the variance
• 05:00 - 05:30 of the other two distributions which are much more tightly clustered around their means so let's look at the formula for t so the T distribution or the T statistic rather is the mean of Group 1 minus the mean of group two divided by the square root of the variance of Group 1 divided by the number of scores in group one plus the variance of group two divided by the
• 05:30 - 06:00 number of scores or number of people in group two so it's not dissimilar to what you saw before we just have to do a little bit more math to get there so the mean of group one and the mean of group two immunogroup one and the mean of group two divided by the variance of group one divided by its number of participants plus the variance of group two divided by its number of participants this quantity we take then the um we take then the square root of and that
• 06:00 - 06:30 gives us the the standard deviation or our noise metric for our t-test um and this is a Formula that you use if you had different numbers of participants in each group which is not really that uncommon you might recruit the same number of people for both of your groups and then you might lose one because of technical errors or because that participant doesn't meet qualifications or some other thing so it's often the case that we have unequal sample sizes
• 06:30 - 07:00 um just by pure random chance so that's what our T formula looks like now the standard error of the difference which is what we call this thing in the bottom here the standard error of the difference is in practice calculated from sample data we don't usually do this from population data of course because we don't have that so what we do is we calculate the variance for each group separately we normalize that variance by the number of participants
• 07:00 - 07:30 within that group and then it's added up because remember we talked about this counter-intuitive fact that the variance of the different scores is the variance of group one plus the variance of group two but they have to be normalized first by the number of people in each of those groups and then to get the standard error there is um we derive that number by taking the square root of the of the two normalized variance estimates
• 07:30 - 08:00 and this metric allows us to quantify the sampling error that we would expect by chance so if we're going to do hypothesis testing with t with t-test we of course have our pretty standard steps after we've designed our hypo our research study then we State what our null and research hypotheses are we have directional they can be directional or non-directional where we're just simply suggesting that there might be a difference between the means the means
• 08:00 - 08:30 are not equal versus the means are equal or we can have directional hypotheses for example the mean of group one is the same as or lower than the mean of group two and in the directional the the positive direction the mean of group one might be greater than the mean of group two and it doesn't matter here for this example please know that you can calculate greater or lower you can speculate either side of that equation
• 08:30 - 09:00 and then of course we State all of the important characteristics the inclusion and exclusion criteria the p-value at which we're going to reject the null hypothesis we figure out our rejection regions and all the ways in which we're calculating our our statistics we calculate our degrees of freedom so the degrees of freedom for the t-test is calculated as the number of people in group 1 minus one plus the number of people in group two minus one now the t-test is easy
• 09:00 - 09:30 because we only have two groups so we can simplify that to the total number of participants across both groups minus two because we have two groups here we then computer test statistic we decide whether or not to reject the null hypothesis based on whether the computed value of our statistic is more extreme than our critical value so our t-test is the ratio of the difference between the sample means
• 09:30 - 10:00 divided by the difference we might expect by chance if the null hypothesis was true if the numerator is significantly bigger than the denominator then we reject the null hypothesis and conclude that there's a difference between the population means in an observational design or that the treatment had an effect on the sample in an experimental design so that's what we're doing when we're testing data using the t-test now it's important and always a good idea we've talked about this many times to look at your data before you do any
• 10:00 - 10:30 statistical testing always a good idea to plot your data before you interpret your test because it helps you see whether your interpretation is correct and it helps you spot outliers that might influence your outcomes so if we have a sample one and a sample two oh you can see there are three outliers in Sample one it's pulling our mean um for the sample down a bit and that might actually affect the outcome of our
• 10:30 - 11:00 t-test so we need to be really careful there and that brings us to the idea of homogeneity of variance I'll talk about that in the next video