CorrelatedDesigns1

Summary

In this lecture, Erin Heerey delves into the concept of correlated samples, particularly focusing on dependent data as compared to independent data. The lecture covers how to analyze data with dependencies, including using paired samples, pre-treatment and post-treatment studies, and the statistical advantages this design can offer. A key takeaway is that dependent designs can lead to increased statistical power even with smaller sample sizes, as they account for individual differences between participants. The discussion includes a thorough explanation of calculations involving paired samples, advantages such as reduced noise, and considerations such as attrition and carryover effects. Real-world examples are also used to illustrate these concepts, emphasizing the importance of understanding the independence of data within any given study design.

Highlights

• Dependent designs can increase statistical power with smaller samples. 🔍
• Real-world studies like pre-treatment and post-treatment illustrate dependent data usage. 🧑‍🔬
• Accounting for individual variability is key to successful dependent designs.🤓

Key Takeaways

• Correlated samples involve relationships between individual data points, unlike independent samples. 🌟
• Dependent data allows for greater statistical power by accounting for relationships between data points. 📊
• Understanding and managing independence in study designs is crucial for accurate outcomes. 🎯

Overview

The lecture introduces the concept of correlated samples, particularly emphasizing the differences between dependent and independent data. Erin Heerey takes us through various scenarios where data might be paired or dependent, such as in pre-treatment and post-treatment studies, or experiments using different measurement devices.

We learn about the advantages of these dependent designs, mainly their capacity to enhance statistical power even in small sample sizes. Heerey discusses the calculation adjustments that such designs involve, with practical examples illustrating how variability is handled to improve study outcomes.

Finally, the session underscores the significance of comprehending data independence in study design. It covers potential drawbacks like attrition and carryover effects, while also providing strategies to mitigate these issues, ensuring a comprehensive understanding of this pivotal topic.

Chapters

• 00:00 - 00:30: Introduction to Correlated Samples The chapter introduces the concept of correlated samples, emphasizing the presence of relationships between individual data points. It distinguishes between dependent and independent data, highlighting that unlike independent data, where the only relationship is based on the condition of derivation, correlated samples have inherent inter-dependencies. The chapter aims to explore these dependencies and the methods to test data when such dependencies exist.
• 00:30 - 01:00: Independent vs. Dependent Data In this chapter, we explore the distinctions between independent and dependent data. Specifically, when participants are randomly assigned to different conditions such as A or B, each participant becomes an independent data point. They only contribute a single score relevant to the condition they are placed in, ensuring no correlation between the results from condition A and condition B. This setup highlights how independent data consists of distinct, singular contributions from each participant, in stark contrast to dependent data where multiple related measurements might come from the same participant.
• 01:00 - 01:30: Paired or Dependent Data Examples This chapter discusses the concept of paired or dependent data in statistical analyses. While much of the class has focused on independent data, this segment explores scenarios where data from participants is measured multiple times. The chapter references previous discussions on correlation, where an X and a Y variable are derived from the same participant. This examination of paired data helps in understanding how to handle cases where there are multiple scores per participant in an analysis.
• 01:30 - 02:00: Dependent Designs Overview This chapter discusses dependent designs, focusing on pre-treatment and post-treatment studies, and experiments involving two different measurement devices. It includes an example from a lab using remote photoplasmography to measure heart rate using video analysis. The technique involves capturing videos that are analyzed to estimate the heart rate, allowing comparison of a dependent measure before and after treatment.
• 02:00 - 02:30: Statistical Power in Dependent Designs The chapter discusses the concept of statistical power in dependent designs, focusing on scenarios where the same individual is measured in different conditions. One example provided is comparing heart rate measured by an individual using two different devices or methods. This highlights the nature of dependent data, where the same subjects are evaluated under multiple conditions or situations to test differences, such as comparing an individual's response in two separate test conditions.
• 02:30 - 03:00: Independent vs. Dependent Designs Example This chapter explains the concept of independent versus dependent designs in research. In independent designs, each participant's data is unrelated to others. In contrast, dependent designs involve data that are related, such as when the same person contributes multiple scores. The chapter emphasizes that for dependent design outcomes to be accurate, participants themselves must be independent from one another, although this isn't always the case in literature.
• 03:00 - 03:30: T-test in Independent and Dependent Designs The chapter explores the concept of T-test in the context of independent and dependent designs, particularly focusing on relationship studies. It highlights how in studies involving dyadic social interactions, like those between spouses or between parents and children, even though each dyad operates independently, the data collected within each group maintains a level of independence from other groups. The chapter emphasizes the importance of understanding these relationships to correctly apply T-tests in analyzing such data.
• 03:30 - 04:00: Calculation Differences in Design Types The chapter discusses the concept of data dependency in research designs, emphasizing how data from individuals, such as a husband and wife in a study, are interrelated. The interdependence arises from the relationship, shared experiences, and participation in the same study. It explains that while dyads (pairs of participants) are treated as independent units in analyses, individual participants within those dyads are interdependent or 'nested' within their respective dyads.
• 04:00 - 04:30: Advantages of Dependent Designs The chapter discusses the concept of dependent designs in research studies, particularly in diet studies. It emphasizes the importance of understanding the independence of data within a study, noting that data within subjects are not independent, while data across different participants should be. Various terms are used to describe these designs such as paired, correlated, or within-subjects designs.
• 04:30 - 05:00: Disadvantages and Attrition in Designs The chapter discusses the topic of dependent samples in experimental designs, specifically focusing on how two dependent measures (Measure A and Measure B) can be evaluated differently. It highlights the use of hypothesis tests, such as the t-test, to identify differences between these measures. An example is given where an independent samples t-test is used with participants randomly assigned to two groups for comparison.
• 05:00 - 05:30: Carryover and Order Effects This chapter discusses the concept of carryover and order effects in experimental research designs, particularly focusing on dependent designs. It explains how the means of group one and group two are compared by calculating the difference in their scores. The chapter highlights that, although the t-test for independent samples is used to test hypotheses about the means, the dependent design requires a slightly different approach when testing if the difference between groups is significantly different from zero. The discussion emphasizes the potential influence of existing relationships between variables in a dependent design.
• 05:30 - 06:00: Assumptions in Dependent Designs In the chapter titled 'Assumptions in Dependent Designs,' it discusses conditions A and B where both scores in the experiment are contributed by a single person. This setup can be advantageous as it allows the use of the correlation between the two scores to gain a statistical advantage. Unlike previous methods, the chapter explores a different way of testing for differences by leveraging the dependency of scores within subjects.
• 06:00 - 06:30: Randomization Tests for Dependent Samples This chapter discusses randomization tests for dependent samples, emphasizing the ability to detect significant effects even in small sample sizes. It explains that tests for correlated or related samples (Condition A and Condition B) can be powerful. An example is given where a sample with only five data points shows a statistically significant correlation of 0.7, highlighting the efficiency of such tests.

CorrelatedDesigns1 Transcription

• 00:00 - 00:30 in this lecture we're going to talk about correlated samples so these are samples where there are relationships between the individual data points we'll start by talking about what dependent data are and how they're different from independent data and then we'll talk about the actual ways of testing data using those dependencies when data are independent the only relationship between the scores is the condition that they were derived in for
• 00:30 - 01:00 example if we have a sample of participants and we randomly assign them to condition a or condition B each participant is independently sampled from every other participant each participant contributes only one value to the data set so they can contribute their summary score on your interesting measure and they only contribute one they contribute to whichever condition they happen to be in there's no relationship between the measurements in condition a and condition B
• 01:00 - 01:30 so that's what we get when data are independent and so far most of what we've talked about in the class with the exception of correlation have required data where there really is only one score per participant in a given analysis we can also study data that are paired or dependent so sometimes participants are measured more than once so we talked about this in correlation where we have an X variable and a y variable from the same person examples of this are things like
• 01:30 - 02:00 pre-treatment and post-treatment studies where we might compare a dependent measure before versus after treatment experiments with two different measurement devices so for example you might have the same individual measure twice once with once with each device for example my lab sometimes uses a technique called remote photoplasmography which is a way of gaining heart rate using video actually so we take the videos and they're analyzed in a certain way that makes us allows us to estimate
• 02:00 - 02:30 your heart rate but we could compare that to an actual heart rate monitor so we might have the same individual measured twice with the same with with two different devices or in two different manners so these are that's another example of dependent data and sometimes we want to be able to compare the same individual across multiple conditions or situations so it's the same individuals are compared under two different test conditions condition one versus condition two so those are examples of paired data
• 02:30 - 03:00 now it makes sense for most of us the data are not independent they're related to one another when the same person has contributed multiple scores however for this design to work for these designs to be accurate for their outcomes to be accurate the participants themselves must be independent of one another so that's not always true in the literature so some of you might be
• 03:00 - 03:30 interested in relationships so there are lots and lots of studies in the literature of dyadic social interaction between people who are already in relationships for example husbands and wives or parents and children and the dyads themselves are independent so if I go to a study with my husband our data will be independent of you know if my colleague goes and does a study with their spouse my data and their data or our data and their data will be independent but my
• 03:30 - 04:00 husband's and my data will be related to one another partly because we're in the same relationship and we have a very long-standing relationship with one another and partly because we're in the same conversation um so if you talk to someone in in a study in my lab your data will no longer be independent of the of that person's data for that for that study session so the idea there is that the dyads are independent but the participants are nested within diet so the participants
• 04:00 - 04:30 within a diet are not independent so you need to be really careful when you're reading the literature how you conceptualize this idea of Independence but in many of these we call them paired designs correlated designs or between uh within subjects designs um the data themselves are not independent within the person but across participants the data need to be independent yeah so when a study design Compares multiple measurements for each participant these
• 04:30 - 05:00 data are dependent samples to explore this idea we're going to consider a design that has two dependent measures measure a and measure B and we're going to examine that in a slightly different way so we know that hypothesis tests for two measures like the t-test often test for differences between the measures so for example if we had to have an independent samples t-test where we have participants randomly assigned a group one in group two we can compare the
• 05:00 - 05:30 means of group one and group two by taking a different score between the means now in a dependent design we can do that but we take that we do the difference a little bit in a slightly different way so like the independent t-test if we're doing a hypothesis test for two means we test whether the difference between a and b is different from zero but the problem is in a dependent design if there's a relationship between the
• 05:30 - 06:00 conditions A and B because both scores are contributed by a single person and so we can actually use the fact that both scores are contributed by a single person to our advantage we can gain a statistical Advantage by accounting for the relationship the correlation between the two scores so we can turn this to our advantage and again we will test for differences but we're going to do it in a slightly different way than we've done it before
• 06:00 - 06:30 so when samples are correlated or related tests for dependent designs can provide enough power to detect significant effects even if the sample sizes are really small so we have if we have condition a and condition B here this is a statistically significant correlation even though there are only five data points in it right and you can see that one two three four five data points in this sample the correlation is 0.7 and it's statistically significant
• 06:30 - 07:00 because what we're doing is we're accounting for the even though there's a fair bit of variability in this sample um we if we can account for the relationship between the two conditions we can actually find statistically significant effects and that gives us more what we call statistical power this is a concept we've talked about before so let's take a really silly toy example and I'm doing this here because I'm going to illustrate what it is that we're doing how we're accounting for the fact that data come
• 07:00 - 07:30 from the same person so in this example I'm going to show you what an independent design looks like so each of these participants you'll notice they're colored the color dots are all a slightly different color hopefully you're not colorblind and you can actually see all these different colors but all of these dots they're filled with a different dot color now if I calculate the mean of this
• 07:30 - 08:00 first of sort of group one here his mean is four if I calculate the mean of group two this mean is Let's Pretend 4.5 the difference between the means is 0.5 that's not a really big difference but let's look at what happens in the dependent design so now I want you to look at the colors here and what you can see is that the the colors of the dots have the same
• 08:00 - 08:30 interior colors so we know that participant one here contributed this score in condition a and this score in condition b or condition one and two and so you can see what these look like now the means are exactly the same I haven't changed the positions of any of these dots I've just changed their colors to reflect the idea that they come from the same person so the mean of group one is four and the mean of group 2 is 4.5 that does not change
• 08:30 - 09:00 but now we can see how different people have responded to the to these two conditions so we have three we have four people out of this total of six people whose scores have increased slightly from condition one to condition two we have two people who scores have either stayed the same or slightly declined so we can now look at the difference scores rather than the individual scores so again just like down here the average
• 09:00 - 09:30 difference is 0.5 but on this side we have a minimum let's pretend they score is one and a maximum Let's Pretend This score is 5.5 so again here's our 1 and our 5.5 but what we're doing here is we're calculating the slopes how different these scores are from one another so what we're doing is we're taking condition 2 and subtracting off condition one so this might be a maximum of two this might be our biggest
• 09:30 - 10:00 increase and this is our smallest change from the across the two conditions which is minus two now if you look at the difference between let's look at let's think about this in terms of the range of the data right so if the minimum in this data set is one and the maximum is 5.5 our range is 4.5 right because we know how to calculate the range is the maximum minus the minimum and for this we have the maximum minus the
• 10:00 - 10:30 minimum we don't actually have a substantially smaller range on this data than we do here and so we're going to use that so our our metric of variability has now gone down so our sampling error our differences between scores has reduced so our signal to noise ratio will be changed so because our signal to noise ratio is
• 10:30 - 11:00 you know sort of this in the independent design it's kind of this within group's variance and here it's the variances across the in changes across the condition which is substantially smaller based on this range than this one over here right here it's 2 2.2 and there it's 4.5 so our our range has gone down our variance and therefore our sampling error has gone down even though the difference
• 11:00 - 11:30 between the means is exactly held constant here so if we look at that numerically I'm going to show you the same numbers in two different designs so let's say we have an independent condition where we have 10 participants who have contributed data they've been randomly assigned either to group a or group b so this is a sample of 10 participants each of whom has given us a single score one in condition in condition a or
• 11:30 - 12:00 condition B depending on the group to which they were assigned so the mean of group a is five the mean of Group B is three the variance is um I didn't plan it this way but they work out exactly the same we have perfect homogeneity variance 1.58 is the or this is the standard deviation actually so the standard deviation of these two groups is 1.58 um so our critical value of T based on our degrees of freedom of 8 here because remember that in a t-test our degrees of freedom is the number of scores the
• 12:00 - 12:30 total number of scores we have minus two so our critical value of T is 2.306 for this independent design so if we compute the denominator of our t-test so we know our numerator is going to be 5 minus 3 which is two so our standard error of the difference between sample means the denominator of our t-test is the standard deviation of the two groups added together divided by the square root of the number of subjects that gives us a total value of one
• 12:30 - 13:00 so our t-test is going to be 2 divided by 1 or 2. now that's not larger than our critical value there's no difference between these groups they look like they could be different but there's no difference between them it's not statistically significant so now let's look at the same numbers this time in a dependent design so this is a different study where we've sampled only five participants this time and let's pretend they've given us exactly the same values of data and by the way
• 13:00 - 13:30 we cannot just simply take an independent design and analyze it as if it's a dependent design that will not actually improve your outcomes nor is it appropriate to do that I've had people ask me this question in the past you really can't it has to be a different kind of study but I'm doing this with the same numbers the exact same numbers that are in this table and the reason I'm doing that is because I want to show you what the difference is between these between what we're analyzing here and what we're analyzing here so again you can look and
• 13:30 - 14:00 decide that these numbers are exactly the same as what happens here and here so these are five participants this time instead of 10 they've now given us a score and condition a and a score in condition B so when we analyze this design we're not comparing condition a to condition B the way we did here as independent conditions instead what we're doing is we're taking the difference in scores so we're taking 7 minus 5 that gives us two six minus 2 gives us 4 4 minus 3 gives
• 14:00 - 14:30 us 1 and so on so what we're actually analyzing here is the difference between the means based on this subtraction not on the mean here minus the mean here right so we're doing the subtractions individually and we're taking the mean of those subtractions so the mean of the different scores here is two now that's exactly this we have the same number so it comes out exactly the same to this this value of 2 5 minus three is two so we have no different no change there
• 14:30 - 15:00 but what we do change is when we calculate our standard deviation on this set of numbers here we get 1.22 remember it was 1.58 over here it's 1.22 now because now we're not calculating the standard deviation of these scores we're calculating the standard deviation only of these scores here in the difference column so now when we calculate the standard error of the difference our denominator of our t-test we only have we have one
• 15:00 - 15:30 sample here right we have the the different scores here 1.22 divided by the square root of 5. that gives us 1.55 as our as our score so now when we conduct our t-test it's 2 divided by 0.55 and that gives us a substantially higher T value than 2 because what we've done here is reduced the noise estimate we have not
• 15:30 - 16:00 changed the signal but we have reduced the noise estimate now our T critical is different the bar goes up here because we have fewer degrees of freedom in this design we have fewer participants so here in this design because it's one sample of participants um are our different score our degrees of freedom rather is five minus one or four so our T critical is 2.776 these are T criticals by the way that are theoretical they were looked up in the back of a book
• 16:00 - 16:30 um or rather in a t table on the internet which is where I got them from um but our calculated our observed value of T now because we've reduced the noise estimate has gone up to 3.64 it's statistically significant and now we can see that even though these can these conditions look like there should be different and the in fact are different so that gives us a numeric idea of what we're looking at here so the important thing to remember is
• 16:30 - 17:00 we have decreased the signal to noise ratio so in the independent design the signal to noise ratio is is um we've decreased the noise estimate and therefore we have an increased signal to noise ratio so in the paired samples T calculation the signal is the same but by decreasing the noise estimate the ratio of signal to noise gets bigger and that means the confidence intervals will get narrower if you were to take a confidence interval on your mean and
• 17:00 - 17:30 estimator um and that makes this design better at detecting differences between between memes between effects it allows us to to more statistical power because our signal to noise ratio is more sensitive so it accounts for the individual differences between participants rather than including those individual differences in the noise or in the sampling error estimate that works because sometimes differences
• 17:30 - 18:00 between participants can be larger than the effect itself so if we have if you and I are taking a reaction time test that's let's say we have five blocks of Trials let's say in Block one your reaction time is 300 milliseconds because you're significantly younger than me and we all know reaction time gets slower as you as you get older so your reaction time at time one might be 300 milliseconds and my reaction time after block one is 500
• 18:00 - 18:30 milliseconds now both of us might improve with practice right and so let's say after five blocks of practice trials your reaction time has dropped to 200 and mine has dropped to 350. we've both improved but notice that my ending time here is still slower than your starting time was so sometimes when you have measures where there's a lot of variability between participants these dependent
• 18:30 - 19:00 designs can really help you account for that because if we can look at the difference between 300 and 200 so your change score is 100 my change score is 150. um that's much more comparable it has much lower variability in it than trying to compare you know 200 to 500 which is a much greater range so this reduces the noise estimate um and thereby increases our signal to
• 19:00 - 19:30 noise ratio some advantages to this kind of a design well first the first Advantage is that you need a smaller sample size to get a statistically significant result because this type of hypothesis test is statistically more powerful because we account for the individual differences between people and what we're then doing is we're looking at the treatment effect so we have a greater
• 19:30 - 20:00 chance of rejecting the null hypothesis with a smaller sample size and it also allows us to do something else it'll when when participants contribute multiple scores to your data set what you then know is that the people who receive treatment a are exactly the same as people who receive treatment B and so we know that there are no individual differences between those people the people who got both
• 20:00 - 20:30 treatments are exactly the same people so even though you might be introverted and I might be extroverted or vice versa it doesn't matter how that relates to Performance because I'm giving you my score in both conditions and you're giving a score in both conditions so if participants all get the same study treatments if every participant gets every study treatment then you can be certain that individuals within the treatments don't differ so this allows
• 20:30 - 21:00 each individual participant to serve as their very own control and that's a really big advantage of these designs now of course there are disadvantages as well one of these is a thing we call attrition so because every person appears in every treatment if someone drops out of the study it reduces your sample size in both groups because you can't use that participant's data at all anymore their data just is no longer usable we can also get a thing we call
• 21:00 - 21:30 carryover effects or order effects so the differences between conditions might be due to the order in which they were presented so it might be the case that if you were presented with condition one first condition one actually shapes your performance on condition two and that's known as a practice effect it could be that you're exposed to a particular type of stimulus in condition one that then in condition two changes the way you respond to the to the stimulus that's being presented in
• 21:30 - 22:00 condition two now there are many ways to get around these cons we can control carryover Effects by changing how the design is administered so if we had a design where all of the trials that were of a particular type were came in a single condition and all of the trials in another type came in a different condition we could do all the trials of condition one and then all the trials of condition two for participant for half the participants randomly assigned and vice versa we could sort of reorder them for the rest
• 22:00 - 22:30 of the participants and as long as we assign participants to trial to to condition orders randomly then we can account we can control for those carryover effects we can also randomize or counterbalance the the order in which the Trials come so the trials from both conditions could be randomly intermixed or counterbalanced in some way so that you can essentially make sure that your that the practice effects jointly affect both conditions in the same way so we
• 22:30 - 23:00 know that people get faster over time as they start to as they learn how to do a task we know that their performance becomes more stable over time as they learn how to do a tasks we also know they get bored and fatigued and that can also that can reduce performance but if the trials are randomly intermixed and if that's possible with your design then it's much easier to determine um because because that practice effect or like fatigue effect or whatever you
• 23:00 - 23:30 want to think about it as those effects will affect both studies increasingly across all of the trials so that's one Technique we use for controlling carryover effects and finally if we think about our assumptions for this test the participants themselves need to be independent and where we're looking we still have our Assumption of normality what we're looking for there is we're looking for the different scores so because what we're analyzing are the different scores what we would expect is
• 23:30 - 24:00 that the difference between the conditions should be roughly normally distributed again we would want to test for that and we would want to make sure there were no extreme outliers in the set of different scores again the participants themselves can be very different to one another but what we want is to minimize outliers in the set of different scores so even though some participants might be very different to one another just on average their different scores should not have extreme outliers
• 24:00 - 24:30 and we'll look at this this week with a randomization test in the lab where we are doing something really similar to the process of what we did before when we did randomization where we have um so when we do the paired samples t-test when we violate our assumptions instead of randomly shuffling one column we leave all the paired scores together because those paired scores are related to one another
• 24:30 - 25:00 um and then we change the way we think about the null hypothesis a little bit so in a dependent samples t-test our paired samples t-test if the null hypothesis is true and the condition or treatment had no effect then the score at time one and the score at time two should be interchangeable and so what we do when we do the randomization test there is we do the randomization test on the difference scores and what we randomly do is we change the
• 25:00 - 25:30 order in which we're subtracting the two variables from one another just randomly so instead of shuffling we make a vector of scores that are sampled from a distribution containing only the values one and minus one and then we can multiply the ones and minus ones with our actual scores and compute the t-test on the new Vector so that leaves the values exactly the same but it just simply changes the signs randomly between minus and Paul N plus and so what we do then is we calculate our t-test and we you know make a new
• 25:30 - 26:00 vector and multiply the scores the original scores against that new vector and we do that a whole bunch of times and that's the comparison distribution that we use for our t-test when we do this randomization test so it makes the randomization test actually a little bit easier than the ones we've been doing um but it's a slightly different way of thinking about what the null hypothesis means here and we'll next talk about an analysis of variance that's designed for for correlated samples we'll do that in the next video