From a sample to a claim: standard error, confidence intervals and p-values

Lecturer: Alright, let's make a start. Can everyone see the screen? Okay. One piece of housekeeping before we begin. Quiz two has moved. It is now on Wednesday, not Monday, same room, and you will want a calculator with a square root button, because, as you are about to find out, we take a lot of square roots in this half of the course.

Lecturer: So. Up to now everything we have done has been description. You have a pile of numbers, you draw a histogram, you work out a mean and a standard deviation, and you describe what you have got. Today we cross a line. Today we start making claims about things we have not measured. That is what inference is.

Lecturer: We measure a few, and we say something about all of them, and the whole lecture is really about one question: how wrong could we be?

Lecturer: Let me set up the example we are going to use for the whole hour. Suppose we want to know how long students at this university sleep on a weeknight. All of them. That group, everybody we care about, is the population, and the true average for that whole population gets a Greek letter, mu. Mu is called a parameter. It is a fixed number.

Lecturer: It exists. We just do not know it, and we are never going to know it exactly, because we are not going to follow twenty thousand people home. What we do instead is take a sample. So say we survey sixty-four students, chosen at random, and we get a sample mean of six point eight hours, with a standard deviation of one point six hours. The sample mean gets the symbol x-bar, and x-bar is called a statistic. So, parameter for the population, statistic for the sample.

Lecturer: P goes with P, S goes with S, if that helps you. Now here is the uncomfortable thing. If I had surveyed a different sixty-four students, I would not have got six point eight. I might have got six point six, or seven point one. The parameter does not move, but the statistic moves every time you take a new sample. So when I report six point eight, the honest question is, how much would that number have bounced around?

Student: Does it matter how big the university is? Like, sixty-four out of twenty thousand seems tiny.

Lecturer: Good question, and the answer surprises people. As long as the population is much bigger than the sample, the size of the population barely matters. What matters is the size of the sample. Sixty-four random students tell you about as much about a university of twenty thousand as about a country of twenty million. The word doing the heavy lifting there is random. A big sample that is not random is worse than a small one that is.

Lecturer: So let's do a thought experiment. Imagine we did not run the survey once, but thousands of times. Each time, a fresh sixty-four students, each time we write down the mean. Six point eight, six point six, seven point one, six point nine, and so on. And then we draw a histogram, not of students, but of those means. That histogram has a name.

Lecturer: It does not matter. Once you average thirty or more of them, the averages pile up in a bell shape. The usual rule of thumb is a sample size of about thirty is enough, a bit more if the population is very skewed.

Lecturer: Now, that spread, sigma over root n, is so useful that it has its own name.

Lecturer: It is the standard error of the mean. And since we do not know sigma, the population standard deviation, we use the sample standard deviation, s, in its place. So the standard error is s over the square root of n. Let's do it for our survey. The standard deviation was one point six hours. The sample size was sixty-four, and the square root of sixty-four is eight. So one point six divided by eight is zero point two.

Lecturer: The standard error is zero point two hours, which is twelve minutes. I want you to be very clear on the difference between those two numbers, because it is the single most common mistake on the exam. The standard deviation, one point six hours, tells you how much individual students differ from each other. The standard error, zero point two hours, tells you how much the sample mean would differ from survey to survey. Standard deviation is about people. Standard error is about the precision of your estimate.

Lecturer: If you collect more data, the standard deviation does not shrink, students are as varied as they ever were. The standard error does shrink. And how fast does it shrink? Look at the formula. It is the square root of n on the bottom. So to halve the standard error, you need four times the data.

Lecturer: If we went from sixty-four students to two hundred and fifty-six, the square root is sixteen, and one point six over sixteen is zero point one. Four times the work for twice the precision. That is the law of diminishing returns in statistics, and it is why studies do not just keep getting bigger.

Lecturer: Okay. So now we can build the thing you will see in every paper you ever read, the confidence interval. The logic goes like this. The sampling distribution is normal. In a normal distribution, ninety-five percent of the values fall within one point nine six standard deviations of the centre. So ninety-five percent of the time, our sample mean will land within one point nine six standard errors of the true mean.

Lecturer: And if x-bar is within that distance of mu, then mu is within that distance of x-bar. Same distance, just seen from the other end. So the recipe for a ninety-five percent confidence interval is, the sample mean, plus or minus one point nine six times the standard error. For us, one point nine six times zero point two is zero point three nine two, call it zero point three nine.

Lecturer: So six point eight minus zero point three nine is six point four one, and six point eight plus zero point three nine is seven point one nine. Our ninety-five percent confidence interval for the mean sleep of all students is six point four one to seven point one nine hours. That plus or minus part, the zero point three nine, is called the margin of error.

Lecturer: What if you want to be more confident? You can have a ninety-nine percent interval. The multiplier goes up from one point nine six to two point five seven six. Two point five seven six times zero point two is about zero point five two, so the interval becomes six point two eight to seven point three two. Notice what happened. More confidence, wider interval.

Lecturer: You never get something for nothing. If you want an interval that is both confident and narrow, the only honest way is a bigger sample.

Lecturer: Now I have to be a little bit annoying, because there is a right way and a wrong way to say what ninety-five percent means, and nearly everyone says it the wrong way. The wrong way is, there is a ninety-five percent chance that the true mean is between six point four one and seven point one nine. It sounds perfectly reasonable. It is not what the method gives you.

Lecturer: Here is why. Mu is a fixed number. It is not jumping around. It is either inside this particular interval or it is not. The thing that is random is the interval, because the interval came from a random sample. So the ninety-five percent describes the procedure.

Lecturer: If we repeated the survey a hundred times and built a hundred intervals, about ninety-five of them would contain the true mean, and about five would miss it. That is the picture on the slide, a hundred horizontal bars, and a handful, drawn in red, that do not cross the vertical line. And the awkward part is, you never know whether yours is one of the red ones. You ran one survey, you have one interval. So the careful sentence is, we are ninety-five percent confident, meaning we used a method that works ninety-five percent of the time. I am not going to take marks off if you slip in conversation.

Lecturer: I will take marks off if you write the wrong version in the exam.

Student: So is it wrong to say there's a five percent chance we're wrong?

Lecturer: It is closer. I would say, the method is wrong five percent of the time. That keeps the probability attached to the method, where it belongs.

Lecturer: Right, a complication. Everything so far quietly assumed we had a decent sized sample. Remember we swapped sigma, which we do not know, for s, which we estimated. With sixty-four people, s is a pretty good estimate. With a small sample it is not, it is itself quite wobbly, and if we ignore that, our intervals come out too narrow. We would be claiming more precision than we have.

Lecturer: The fix was worked out by a man called William Gosset, who was a brewer at Guinness in Dublin, and who published under the name Student because the brewery did not want competitors to know they employed statisticians. So it is called Student's t distribution. It looks like the normal curve, but with fatter tails, and exactly how fat depends on the degrees of freedom, which for one sample is n minus one.

Lecturer: Let's see what difference it makes. Suppose we had only surveyed sixteen students and happened to get the same mean and standard deviation. The standard error is now one point six over the square root of sixteen, so one point six over four, which is zero point four. Degrees of freedom, sixteen minus one, fifteen. And instead of one point nine six, the table gives us two point one three one. So the margin of error is two point one three one times zero point four, about zero point eight five.

Lecturer: The interval runs from five point nine five to seven point six five. Much wider than before, partly because the sample is smaller, and partly because of that bigger multiplier. As the sample grows, t settles down towards the normal value. At thirty degrees of freedom the multiplier is two point zero four two. At a hundred it is one point nine eight four. So in practice, sorry, in practice people just use t all the time when sigma is unknown, which is always, and for large samples it makes no real difference.

Lecturer: Okay, second half of the story. So far we have estimated. Now we test. Suppose the university wellbeing leaflet says, our students get a healthy seven and a half hours of sleep a

Lecturer: Night. And you look at your six point eight and think. A hypothesis test is the formal version of.

Lecturer: Minus zero point seven divided by zero point two is minus three point five. Our sample mean is three and a half standard errors below what the leaflet says. That number is the test statistic, here a z score.

Lecturer: Our p-value is far below that, so we reject the null hypothesis. The data are not compatible with seven and a half hours. Now try a different claim. Suppose the leaflet had said seven hours. Then the difference is six point eight minus seven, minus zero point two, divided by the standard error of zero point two, which is minus one. One standard error away.

Lecturer: The p-value for that is about zero point three two. That would happen about a third of the time just by the luck of the draw. So we do not reject. And notice the wording. We do not say we have proved that the mean is seven. We say we fail to reject.

Lecturer: Absence of evidence against is not evidence for. And here is something neat that ties the two halves together. Seven point five was outside our ninety-five percent confidence interval, and we rejected it. Seven was inside the interval, and we did not. That is not a coincidence. The ninety-five percent confidence interval is exactly the set of values that a test at the five percent level would not reject.

Lecturer: So the interval tells you everything the test does, and more, because it also tells you how big the effect might be. I want to spend five minutes on what a p-value is not, because it is the most misunderstood number in science. The definition, once more. The p-value is the probability of getting data at least as extreme as yours, if the null hypothesis is true. It starts from the assumption that the null is true.

Lecturer: So, misreading number one. The p-value is not the probability that the null hypothesis is true. A p-value of zero point zero three does not mean there is a three percent chance the leaflet is right. It is the probability of the data given the hypothesis, not the probability of the hypothesis given the data, and those are different things. The probability that you are in this lecture hall, given that you are a statistics student, is high. The probability that you are a statistics student,

Lecturer: Given that you are in this lecture hall, well, some of you are here for the heating. Misreading number two. A small p-value does not mean a big or important effect. It is partly about the effect and partly about the sample size. Let me show you. Suppose we survey ten thousand students, and the mean comes out just zero point zero five hours below the claim.

Lecturer: Zero point zero five hours is three minutes. Nobody cares about three minutes of sleep. But the standard error with ten thousand people is one point six over a hundred, zero point zero one six. So the test statistic is zero point zero five over zero point zero one six, which is about three point one, and the p-value is about zero point zero zero two. Highly significant, and completely unimportant. So we separate statistical significance, which means unlikely to be chance, from practical significance, which means big enough to matter.

Lecturer: Always report the size of the effect and its confidence interval, not just the p-value. And misreading number three, a p-value above zero point zero five does not show there is no effect. It may just mean your sample was too small to see it.

Lecturer: Which brings us to the two ways a test can go wrong. A type one error is rejecting a null hypothesis that is actually true. A false alarm, a false positive. And the lovely thing is that you choose how often that happens. It is alpha. Set alpha at zero point zero five, and when the null is true you will raise a false alarm five percent of the time.

Lecturer: Now, one consequence of that five percent that catches out professionals. Suppose you do not run one test, you run twenty. Twenty different outcomes, sleep and grades, sleep and mood, sleep and coffee, and suppose in truth nothing is going on in any of them. Each test has a five percent chance of a false alarm. Twenty times zero point zero five is one. So you expect one significant result from pure noise.

Lecturer: And the chance of getting at least one is one minus zero point nine five to the power of twenty, which is about sixty-four percent. Nearly two in three. So if somebody tests twenty things and shows you the one that worked, you should not be impressed.

Lecturer: The simplest protection is called the Bonferroni correction. You divide alpha by the number of tests. Twenty tests, zero point zero five over twenty, so each one has to beat zero point zero zero two five instead. It is strict, some would say too strict, but it is honest, and it is the one I expect you to know.

Lecturer: Next time we take all of this and compare two groups, students with morning lectures against students without, which is the two-sample t test. Read chapter nine before then. And remember, quiz on Wednesday, bring the calculator. Thanks, everyone.