Terry, I agree in principle that there are "better" approaches than ANOVA etc. for many kinds of data. But in practice, I think we greatly overestimate how often this will make any difference. Yes, ANOVA makes lots of assumptions, and we love to wring our hands about that. But it turns out to be extraordinarily robust to most violations of those assumptions. So given that, there's a lot to be said for approaches that are simple, well understood, and can be applied broadly. I don't think the situation is as dire as you suggest!
Something that's sometimes frustrating to me as a statistics educator is that the stats ed community has made at least some meaningful progress on this front -- simulation-based inference is pretty core to how a lot of us teach algebra-based intro-level statistics now, even if approaches of more full-scale change like Danny Kaplan's book (https://dtkaplan.github.io/Lessons-in-statistical-thinking/) haven't caught on -- and yet we seem to have had little success in communicating that approach & set of tools to folks teaching stats in cognate disciplines.
So I guess what I mean here is: I think what we really need is better ways for these conversations to happen across fields, so that it's not happening separately in stats, bio, psych, etc. at all different paces, starting from zero every time.
Thanks much for sharing your expertise! That's really encouraging, here's hoping these conversations will somehow penetrate into undergraduate biology curricula. I think if we had some kind of real accreditation at the department level (like in some other fields) maybe this wouldn't be where we are now.
As someone never formally trained in stats, I read this with curiosity. And as someone whose formal degrees all come from humanities/arts or social sciences, I'm usually 100% on-board with a statement like your last sentence. I spend a lot of my time building bridges between science and these other fields/domains. But, I have to confess, I don't actually get the connection in your last sentence. If you have the bandwidth, I'd love to hear more about how you see the Humanities fixing this biostatistics dilemma.
Understanding what p-values mean in a statistical test is more of a philosophical challenge than a mathematical issue. You could get a room full of statisticans who would argue about the "meaning" of a p-value in any particular context. It all depends on how the data were collected, the nature of the question that is being asked, and what meaning you might choose to apply to the statistical result. It's really a philosophy of science issue. What does it mean for something to be true, or supported by evidence, and how can that evidence be interpreted? [would love for an actual philosopher of science to jump in here...]
I think we'd be doing a hell of a lot better if we taught stats with any consistency at all! My only "formal" education in stats is the equivalent of an undergrad level stats 101 course, taken over a decade ago. So, basically useless for the work I currently do. As someone that has a PhD, that is pretty embarrassing to admit (and frustrating), to say the least. However, I didn't have any good biostat course options available to me in grad school, so my advisor suggested I didn't waste my time with one. I learned most of what I know about stats directly from her, plus lots of trial and error and deep dives into (probably out of date) textbooks and internet searches. Most of my peers have expressed that they're in the same boat.
With the above in mind, take the following with a grain of salt, but most of what I've learned about stats suggests that parametric tests like ANOVA are surprisingly robust to many of the violations that we fret over (as another commenter already noted) and are often a solid choice for many simple datasets (and non-parametric equivalents are often available and just as simple to carry out using a statistical software like R). The most common advice I got about experimental design in grad school was "simple experimental design = simple stats = simple (and meaningful) interpretation," and I strive to follow that advice, especially with my aforementioned lack of statistical knowledge.
Another thing to think about - even if we know what we're doing with stats, reviewers often don't and will often question or reject valid statistical analyses due to their own lack of knowledge.
There is no denying that data are shoehorned in to ANOVAs and T-Tests inappropriately. However, in disciplines where controlled studies are the norm (agriculture for example), they are functioning as designed will probably continue to endure. I haven't had many opportunities to use either since grad school (nearly everthing I do ends up going into a GLM or GLMM), but in the few instances where I have done a controlled experiment, I was super pumped to break out the old ANOVA. So easy to convey what is going on to the reader.
" But since we all have access to computers, shouldn’t we all be adopting conceptually more robust to asking and answering questions, and making avail of the computational power readily available to us to take us out of the early 19th century?"
[note - early 20th century no?]
I guess it depends who the students are -
I'm teaching theoretical ecology concepts in a graduate level interdisciplinary program. I've found that even if all the students come with some kind of computer/tablet, most don't have skills in using them for calculations of any type. I've been surprised to realise that in some groups more than half the students might not be comfortable with entering a formula or making a scatter plot. Their backgrounds are very diverse ; some students have never even made a graph.
It's not me to judge the students' expertise and skills, I've got to adapt to them and figure out a way to fill in the gaps before I can move forward. So, I'd argue that we have to start somewhere, and teaching some analytical tools (models, stats) that can be calculated even by hand, still has its uses. Besides, all these tests are still used and importantly *were* used in the past to build the body of scientific knowledge we use today.
Terry, I agree in principle that there are "better" approaches than ANOVA etc. for many kinds of data. But in practice, I think we greatly overestimate how often this will make any difference. Yes, ANOVA makes lots of assumptions, and we love to wring our hands about that. But it turns out to be extraordinarily robust to most violations of those assumptions. So given that, there's a lot to be said for approaches that are simple, well understood, and can be applied broadly. I don't think the situation is as dire as you suggest!
Something that's sometimes frustrating to me as a statistics educator is that the stats ed community has made at least some meaningful progress on this front -- simulation-based inference is pretty core to how a lot of us teach algebra-based intro-level statistics now, even if approaches of more full-scale change like Danny Kaplan's book (https://dtkaplan.github.io/Lessons-in-statistical-thinking/) haven't caught on -- and yet we seem to have had little success in communicating that approach & set of tools to folks teaching stats in cognate disciplines.
So I guess what I mean here is: I think what we really need is better ways for these conversations to happen across fields, so that it's not happening separately in stats, bio, psych, etc. at all different paces, starting from zero every time.
Thanks much for sharing your expertise! That's really encouraging, here's hoping these conversations will somehow penetrate into undergraduate biology curricula. I think if we had some kind of real accreditation at the department level (like in some other fields) maybe this wouldn't be where we are now.
As someone never formally trained in stats, I read this with curiosity. And as someone whose formal degrees all come from humanities/arts or social sciences, I'm usually 100% on-board with a statement like your last sentence. I spend a lot of my time building bridges between science and these other fields/domains. But, I have to confess, I don't actually get the connection in your last sentence. If you have the bandwidth, I'd love to hear more about how you see the Humanities fixing this biostatistics dilemma.
Understanding what p-values mean in a statistical test is more of a philosophical challenge than a mathematical issue. You could get a room full of statisticans who would argue about the "meaning" of a p-value in any particular context. It all depends on how the data were collected, the nature of the question that is being asked, and what meaning you might choose to apply to the statistical result. It's really a philosophy of science issue. What does it mean for something to be true, or supported by evidence, and how can that evidence be interpreted? [would love for an actual philosopher of science to jump in here...]
Ok, got it, thanks!! (I'd love to hear from a philosopher on this, too!)
I think we'd be doing a hell of a lot better if we taught stats with any consistency at all! My only "formal" education in stats is the equivalent of an undergrad level stats 101 course, taken over a decade ago. So, basically useless for the work I currently do. As someone that has a PhD, that is pretty embarrassing to admit (and frustrating), to say the least. However, I didn't have any good biostat course options available to me in grad school, so my advisor suggested I didn't waste my time with one. I learned most of what I know about stats directly from her, plus lots of trial and error and deep dives into (probably out of date) textbooks and internet searches. Most of my peers have expressed that they're in the same boat.
With the above in mind, take the following with a grain of salt, but most of what I've learned about stats suggests that parametric tests like ANOVA are surprisingly robust to many of the violations that we fret over (as another commenter already noted) and are often a solid choice for many simple datasets (and non-parametric equivalents are often available and just as simple to carry out using a statistical software like R). The most common advice I got about experimental design in grad school was "simple experimental design = simple stats = simple (and meaningful) interpretation," and I strive to follow that advice, especially with my aforementioned lack of statistical knowledge.
Another thing to think about - even if we know what we're doing with stats, reviewers often don't and will often question or reject valid statistical analyses due to their own lack of knowledge.
There is no denying that data are shoehorned in to ANOVAs and T-Tests inappropriately. However, in disciplines where controlled studies are the norm (agriculture for example), they are functioning as designed will probably continue to endure. I haven't had many opportunities to use either since grad school (nearly everthing I do ends up going into a GLM or GLMM), but in the few instances where I have done a controlled experiment, I was super pumped to break out the old ANOVA. So easy to convey what is going on to the reader.
" But since we all have access to computers, shouldn’t we all be adopting conceptually more robust to asking and answering questions, and making avail of the computational power readily available to us to take us out of the early 19th century?"
[note - early 20th century no?]
I guess it depends who the students are -
I'm teaching theoretical ecology concepts in a graduate level interdisciplinary program. I've found that even if all the students come with some kind of computer/tablet, most don't have skills in using them for calculations of any type. I've been surprised to realise that in some groups more than half the students might not be comfortable with entering a formula or making a scatter plot. Their backgrounds are very diverse ; some students have never even made a graph.
It's not me to judge the students' expertise and skills, I've got to adapt to them and figure out a way to fill in the gaps before I can move forward. So, I'd argue that we have to start somewhere, and teaching some analytical tools (models, stats) that can be calculated even by hand, still has its uses. Besides, all these tests are still used and importantly *were* used in the past to build the body of scientific knowledge we use today.