Science as Amateur Software Development (2023 edition)

Summary

Highlights

• Richard McElreath compares science to amateur software development and highlights the need for better professional standards 🧑‍💼.
• He mentions the chaos in scientific research due to lack of systematic processes 🔄.
• McElreath draws analogies with cooking, emphasizing organized methodologies in professional kitchens 🍳.
• Discussion on the importance of version control and continuous integration in improving scientific research 🔧.
• Errors in scientific research are often due to lack of professionalism rather than intentional misconduct 🤷‍♂️.
• Cites examples from economics and genomics where lack of careful practices caused significant issues 📉.
• McElreath advocates for adopting practices from other fields to enhance the integrity of science 🔁.

Key Takeaways

• Science can learn a lot from software development and cooking regarding professional standards 🧑‍💻👨‍🍳.
• Many scientific errors are due to sloppiness, not malice 😅.
• Continuous integration and version control are key tools that can improve scientific practices 📈.
• Science could benefit from more professional training and standards 🌟.
• Current issues in scientific reproducibility highlight the need for better professional practices 🔍.

Overview

Chapters

• 00:00 - 00:30: Introduction and Speaker Introduction The chapter serves as an introduction and begins with a welcome greeting to the audience. It then proceeds to introduce Professor Richard Michelrth, who is a Professor of Anthropology and Director of the Department of Human Behavior Ecology and Culture at the Max Planck Institute for Evolutionary Anthropology. Additionally, he is an author of a popular Bayesian statistics textbook. His research interests focus on integrating theory with practice.
• 00:30 - 02:30: Research Standards in Anthropology and Software Development The chapter discusses the parallels between research standards in anthropology and software development. It emphasizes that many researchers may approach research as a hobby rather than with professional rigor. The discussion suggests that there are valuable lessons to be learned from how software engineers operate, implying that adopting some of their professional standards could enhance the research field. The chapter features a presentation by Richard, who is highly anticipated for his engaging talks.
• 02:30 - 04:00: Challenges in Scientific Research The chapter "Challenges in Scientific Research" begins with the speaker intending to use analogies to explain concepts related to scientific research. They plan to compare scientific research to other professions, particularly software development and cooking. The speaker acknowledges a time constraint and hopes to address questions at the end. The analogy between these professions aims to illuminate the challenges and processes involved in scientific research.
• 04:00 - 12:00: Analogies with Other Professions The chapter titled 'Analogies with Other Professions' features a discussion from an anthropologist concerning the profound question of why people exist. The anthropologist suggests that this remains a deep and unexplained puzzle. They describe the nature of this inquiry as a significant scientific problem, emphasizing the challenging empirical aspects that make it particularly difficult to study. The chapter also briefly mentions the measurement of life on Earth in terms of gigatons of carbon, suggesting a scientific perspective underpinning the discussion.
• 12:00 - 27:00: Importance of Reproducibility in Research The chapter discusses the concept of evaluating the essence of life through weight, specifically focusing on the biomass distribution on Earth. It highlights that carbon is a fundamental component of life and emphasizes that plants constitute the majority of life on Earth as depicted in a 2018 paper on biomass distribution. The chapter also points out that animals, represented by a small gray triangle in the biomass diagram, account for a minimal amount of carbon (just two gigatons) compared to plants. Furthermore, the chapter seems to delve into more details about the proportion and significance of animals when examining them closely. This sets the stage for discussing the broader context and implications of reproducibility in research.

Science as Amateur Software Development (2023 edition) Transcription

• 00:00 - 00:30 hello everyone welcome back um so it's my pleasure today to introduce Professor Richard Michel rth um he's a professor uh of anthropology and the director of the department of human behavior ecology and culture at the max Blan Institute for evolutionary anthropology in lies he also wrote a very popular bi Asian statistics textbook uh his research interest lies in integrating Theory with
• 00:30 - 01:00 data analysis and study design he thinks that too many researchers uh treat research like a hobby and that actual Professional Standards are needed um for instance today we'll argue how we as scientists can learn from other professions like software Engineers so Richard thank you for accepting our invitation I know many attendees are looking forward to your ever entertaining uh presentations all right thank you um and thank you all for coming to hear me say
• 01:00 - 01:30 some uh um hopefully interesting analogies and I'm going to try to get through this at a good pace so maybe we have a little bit of time for questions at the end I know we have time constraint uh so yeah I want to draw some analogies between science as a profession and other professions specifically software development um but also cooking uh so let me get to that too um so the
• 01:30 - 02:00 what I study yeah I'm an anthropologist what I study is why people exist yeah which I think is a deep puzzle that still is unexplained um and the context of this problem as the scientific problem goes is really terrible it's it's kind of the worst possible scientific problem you can study because it's it's empirically really intractable thing so here's the big overview uh of why my occupation is is hard so um this is life on Earth me measured in gigatons of carbon
• 02:00 - 02:30 which is not how you usually evaluate life but this is how much it weighs right and car carbon is the main thing that we are right Stardust and uh most uh life on Earth is plants now that's what you see in this diagram from this 2018 paper on biomass distribution and then the little gray triangle in the corner is animals of which there's just two gigatons of carbon in total and then if you zoom in on animals now on the right of this slide um you'll see
• 02:30 - 03:00 that humans aren't even that much of all the animals right insects and fish Dominate and but here's the amazing thing about people um humans and our livestock are essentially all of the mammals on Earth essentially all of the mammals on Earth are humans and our livestock uh we're just completely weird dominant sort of species uh uh you expect any kind of species to dominate Earth you expect a plant or an insect
• 03:00 - 03:30 right this is a weird thing for us to do um and trying to figure out what produced this this weird kind of ape that gives online lectures right we we as a profession we have this very integrative and difficult empirical problem of figuring out the history of uh where we came from and our ancestors and then how we we um people the world and so on and this is a big interdisciplinary project that involves all kinds of messy evidence and modeling
• 03:30 - 04:00 right because um we can't do experiments uh we need Dynamic models to make any sense of the data at all and this is I love this it's absolutely wonderful sort of problem but it's very very difficult and uh why am I telling you this this is not an anthropology lecture well I I like to remind people that anthropology exists uh and maybe you should do some but um also that this is a kind of scientific problem that exemplifies I uh
• 04:00 - 04:30 it's a large scale version of what all scientific problems have is that in science we're trying to generate some knowledge about the world um explain why it is the way it is and maybe just for the sake of explanation but potentially to do interventions in the future and to do this right we need to somehow integrate our work with the work of other people into some common body of knowledge uh that we can agree on um can audit and
• 04:30 - 05:00 justify this is this process of continuous integration of updating of scientific beliefs um and the way this is done in science is usually almost entirely chaotic right this is the thing that philosophers of science will tell you is that there is no scientific method it's sort of like chaos Reigns and nevertheless it's amazing that we learn things right um but there are professions who have an analogous problem uh and
• 05:00 - 05:30 they have professionalized this process of continuous integration in a way that science has not and I'm not going to argue that we should um have some strict formalized version of how we do this what I am going to argue is that we could do a whole lot better uh by looking at other uh occupations so the the primary analogy I'm going to look at is software development um be and one reason to focus on this is because a lot of contemporary science involve software development
• 05:30 - 06:00 this is not a this is a thing that I think shocks a lot of people when you get into science you're fascinated by some scientific context and then you learn you have to code right it's terrible it's like a bait and switch right to get interested in Psychology uh now you have to learn R yeah and sorry but this is just the way it's going to be from now on and so this software development is now a routine part um of being a professional scientist and even if you don't code yourself you're using software and you have a responsibility
• 06:00 - 06:30 to understand how it works and um uh software development is a profession that has lots of professional tools about how to do continuous integration of code and work in teams uh so uh I'm only going to be very light on these things because you can read about these things on your own later but um teams of programmers uh have a very professionalized culture about how they work together and when you train as a as a software developer you train to learn
• 06:30 - 07:00 a common stack of tools so that wherever you end up working on whatever project you can work with people and you can work with people in a way that does quality control and this this whole uh ecology is something called continuous integration um and it involves uh code development and testing and uh lots of detailed things like separation of different kinds of of edits things called branches um and but it's very professionalized now and it's actually also quite new I would say really the
• 07:00 - 07:30 last 20 years have been a period of very rapid professionalization and Tool building in software development software engineering and research software engineering is the side of this that's just getting started where we take some of these tools not necessarily the whole package that I have on the screen here we're not interested in copying them but we're interested in borrowing the things that we need so Version Control and testing uh are things that are becoming more common in the Sciences but are incredibly rare um
• 07:30 - 08:00 I'll try to convince you of that as we go maybe you don't need convincing now give you a specific example here's a project that I've invested a lot in myself this is the the Stan math library and we use this to do statistics arbitrary models and this is a project where the people at the core of it are professional software engineers and they keep it all integrated together there are dozens of programmers making contributions to Stan in any particular time and at different
• 08:00 - 08:30 levels um and this all has to work in the end and the thing I want to emphasize to you that is a really stark contrast to how um science is done and think about the code for a second but I'll just the whole Ecology of data and hypothesis testing and the relationships between hypotheses and statistical models the really stark contrast here is that most of the code in the stand project is code that exists only to test it and guarantee that it works as
• 08:30 - 09:00 planned so that's what I show you here this is this is old if you went and looked it there'd be even more code now but there's 3.6 megabytes of Library code that's the code that you'd actually run if you if you use Stan to do a project um and there's 7.6 megabytes of testing code which is the code that is there for quality assurance and this is quite typical of professional software uh projects especially in the open source Community um is that you'll have twice the three times as much testing
• 09:00 - 09:30 code AS actual um deployed code and that's necessary though because this is a profession and they want to know that it works um let me do another analogy and then I'm going to run through some examples that are closer to science and really about science if you're not so interested in software engineering so there's this uh great old article kind of lost article from Cosmopolitan Magazine from 1967 called the computer girls and I only bring this up but it's a weird historical case but there's this fantastic quote from a very famous
• 09:30 - 10:00 computer scientist named Grace Hopper um I can zoom in on it here Grace Hopper is a real legend in computer programming she developed the first Linker which is like a compiler it's not exactly like a compiler but she was really a big deal early on in making the modern world as we know it in Computing and um she was an admiral in the Navy in the United States Navy and so there's a great quote they got her the quote for this article in Cosmo which is a real really ass something says um they're talking about
• 10:00 - 10:30 programming and what it's like and um back then lots of women worked in in software development still do but the proportion was higher back then and she says oh it's just like planning a dinner explains Dr Grace Hopper now a star scientist in assistance programming for univac um you have to plan ahead and schedule everything so it's ready when you need it programming requires patience and the ability to handle detail women are Naturals at computer programming uh so this analogy to cooking is a really nice one because programming is a lot like cooking if
• 10:30 - 11:00 those of you who like to do cooking and and Dr Hopper's analogy is very apt I think um but it it's also apt in the same professionalism sense right that um cooking as a profession is also very organized and streamlined and has really strict regimens for how people work together in kitchens uh if you just cook at home and you're a bit sloppy with it and you know you're just using feeling right as Uncle Roger would say those of you who know Uncle Roger right just use feeling um this is not this is not how
• 11:00 - 11:30 professional kitchens actually work and professional kitchens are professional there are a bunch of rules about how you stand at stations and how you move through the kitchen and and how everything is processed um where you leave your knives anybody who's worked in a kitchen you know this stuff but all of this is necessary for making the kitchen run smoothly and and things can fall apart this isn't exactly the same as continuous integration and knowledge but it's continuous integration of the meal right and there's an assembly line
• 11:30 - 12:00 here that has to produce a lot of food at exactly the right time and they start in the morning and they make it work yeah okay so just say there's nothing special about software development is the analogy is just that there are professions and where there are lots of culturally evolved rules for making the profession work better and unfortunately I think science is not one of them I am a Scientist and I love science but come on we're it's it's kind of a mess in here isn't it uh here's my favorite quote about this this this is from the previous editorinchief of the
• 12:00 - 12:30 Lancet which is the most prestigious medical journal in the world I think or at least it used to be until this quote came out no so Horton was at a at a closed door meeting of heads of granting agencies and journals in the UK uh back in in 2015 and after that meeting he couldn't share any exact quotes because it was closed door but he wrote an editorial uh in the Lancet about this meeting and lots of shocking things that had been
• 12:30 - 13:00 reported on um so this is the quote you if you'll indulge me I'll read it um the case against science is straightforward much of the scientific literature perhaps half May Simply Be untrue afflicted by studies with small sample sizes tiny effects invalid exploratory analyses and flagrant conflicts of interest together with an obsession for pursuing fashionable trends of dubious importance science has taken a turn towards Darkness yeah um the only thing I disagree here was the end I think we've been in darkness for a long time uh I'm going to make that argument later
• 13:00 - 13:30 this is not a new set of problems even though the details are perhaps different because the incentives are constantly changing in science it's never at equilibrium um but these sort of dark problems have always been around uh and we still learn things so this isn't an argument that science doesn't work it's an argument that we could do a whole lot better and that we are currently violating the Public's trust and I think this is a deep ethical problem the lack of professionalization in our field um okay
• 13:30 - 14:00 so people who are enrolled in this summer school uh this stuff is not going to shock you right you there are still scientists who've never heard that science has problems but not not the people who are enrolled in this yeah U you're here because you're interested in in trying to do better packing is this great uh term it's a terrible practice but it's a great term that that has done a lot I think to draw attention to some of these problems and it's a a fairly famous case now they're even online guides to to show you how it works and
• 14:00 - 14:30 this has gotten a lot of attention but my opinion is that there's a bunch of stuff in the Stream of integration of how we do science that is quite different than packing in the sense that it has more to do with unintentional error than intentional error and the unintentional kind of Errors can be much more devastating uh in principle and I want to show you some examples of those things today and talk about how analogies to software engineering or if prefer working in a professional kitchen
• 14:30 - 15:00 uh can help us um do better over time so let me try to back this up so here's this article from nature in 2016 where they're talking about the replication crisis and such and they asked a bunch of scientists um about various you know vaguely unethical practices or kinds of mistakes they make and at the top here we've got the things that I would classify you know and like using Dante's Inferno kind of categories as greed we've got selective reporting pressure to publish low discal power poor
• 15:00 - 15:30 analysis so are things where scientists are are conscious of these things at least in the in the periphery of their Consciousness they know it's not quite right to do these things but they do it because of of professional incentives and then there's this middle category which I call sloth and this is what I like to focus on it's the sloth problem right the lack of professionalization um it's people don't even remember how to replicate their own work um yeah this often happens in in Labs uh and they didn't take detailed
• 15:30 - 16:00 enough notes and no one can get the thing to work again um or you don't have your code right your code doesn't run uh insufficient oversight uh methods code unavailable for experimental designs um not having the data and so on and lots of problems like this where people aren't trying to be bad right it's just that there aren't there isn't this kind of professional training and a set of norms for doing better um and and that's what I want to talk about let me give you a few specific examples you'll indulge me here so and these are famous
• 16:00 - 16:30 examples apologies if you've heard them before um reach that phase of my life where I'm telling the same story over and over again I need to do better about that but uh so here's this example from economics uh two um I think Harvard economists Reinhardt and rogoff in 2010 published this this paper it was a pre-print at the time growth in a time of debt and this came out um remember there was this recession in 2008 right and there were a bunch of international
• 16:30 - 17:00 debates about public spending in order to stimulate the economy and so on and uh this paper made the argument based upon this single graph that um public debt is bad for growth that there's a negative correlation between the the the GD the debt to GDP ratio which is on the horizontal axis and the percent growth in GDP and um uh so this is the whole paper basically is this graph and this paper
• 17:00 - 17:30 actually had a big impact in parliaments and in the US Congress in the US Congress it was actually waved on the floor of the US Congress in a debate about public spending um anyway uh it turns out the the data were wrong the analysis was just wrong it was just an Excel error and um here's uh Thomas hearnen pointing at the error in the Excel spreadsheet these are the the unsung heroes of science the people who actually look at the the the detailed analysis and figure
• 17:30 - 18:00 out um how the results work and in this case um Thomas saw that there was just a formula error and they had failed to include some countries that went against that Trend and once you actually drug the formula down all the way in the Excel sheet the result vanished uh this is the actual spreadsheet you can see that if you know how Excel Works uh uh first of all apologies I'm sorry that life has done that to you that you have to use Excel it's it's it's a terrible tool but um that blue bounding box that
• 18:00 - 18:30 you see is where the formula uh input range is and you see that they just excluded some countries down below and those go against the trend uh so it's an error yeah and um but lots of policy was made on the basis of this paper before the error was discovered this is sloth right this is not this was not an intentional deception either because the authors provided the spreadsheet they weren't trying to cover anything up they wanted to get this right right this is the kind
• 18:30 - 19:00 of thing that arises from a sloppy way of working and using a tool like Excel which is not designed for scientific analysis and provides no or I say very few tools for quality control and inspection yeah it's designed to be convenient and visible and that is exactly what you don't want uh in in a profession like this okay ex I'm going to pick on Excel sorry but uh Excel does lots of other terrible things because it's not a tool designed for science it's a tool designed for business people to make tables right
• 19:00 - 19:30 that's what it's for and um it it does a bunch of stuff without your consent it'll just convert data everybody who's used this knows it it it it likes to find dates in things right Excel is is is like that annoying guy who thinks everything's a date right and uh it converts all kinds of stuff to dates and so Gene names this is the famous case they various um Gene names and these are abbreviations and they get converted so like sept in one keeps getting converted to September 1st yeah and uh this is
• 19:30 - 20:00 terrible because a huge proportion of published genomics papers have these errors that are introduced by Excel and they this is a significant problem in the genomics literature um and so for a long time biologists have tried to get Microsoft to do something about this Behavior but Microsoft just doesn't care about the scientific Community right again their marketplaces business people making tables and that's what Excel is for um So eventually um the biologist decided they would just change the
• 20:00 - 20:30 abbreviations of the genes so this creates another problem is you've got old papers that use the old system and have these errors and you have new ones and it's a never- ending cycle it's a snake that eats its own tail U but the problem is the tool and the fact that there's no professional standard for what tool we're supposed to be using yeah this is the using Excel is like the equivalent of just uh you know leaving your knife out in a kitchen or something like that yeah these are things you're not supposed to do um the consequences of things like this are are really
• 20:30 - 21:00 serious and and again a lot of you know about this but these audits of reproducibility research findings are quite depressing right so here's here's just one from 2012 that helped snowball some of this concern in biology uh uh buyer tried to replicate a bunch of published results and um only about 25% of them could be validated uh this is a serious problem this is a a waste of Public Funding when the public is paying for it yeah if only 25% of of supposed
• 21:00 - 21:30 discoveries can even be validated in any sensible way to be pursued further um and uh these are serious issues again this is probably not news to you this is this is kind of old news at this point but these things are arising in teams where no one's trying to get away with anything or at least I don't think most of this is fraud most of this is sloth you know and and professionalization help okay very quickly I said something earlier that I don't think the darkness is new this is a fantastic history of
• 21:30 - 22:00 science book called The Lost elements and I just want to advertise it to those of you who like History of Science um it's a it's a history of all the elements in the periodic table and there was a lot of false elements there this book catalogues more false discoveries of elements than real elements that currently exist uh so it isn't like I mean now when we teach the periodic table we teach it like this Immaculate thing that sprung forth from the forehead of Athena or something right and we just got it right but that's not what it was it was a mess out there because it science wasn't
• 22:00 - 22:30 professionalized then either uh it's more professionalized now than it was then so maybe we're doing better I don't know but I want to say this is not like it's some new problem that's caused by you know um late 20th century capitalism or something like that okay um I've complained about things like this for some time and and I've unfortunately developing a reputation for this so this is this natural selection of bad science paper comes up again and I only mention it say that uh this the same point is
• 22:30 - 23:00 that I think it's easy to focus on deliberate attempts to get away with things like like um thinking about people doing fraud and such and those things matter and and there's probably more fraud than we we uh know um but I think that the Norms in science have sort of evolved to satisfy particular careerist ends rather then reliability ends and that's the basic problem is this cultural evolutionary process where we're incentivized to get promotions um but it's very hard
• 23:00 - 23:30 to know if you're right in science and most of us will probably die before we know if we were right about what we study right um and I'm starting to get this feeling but I'm going to keep doing it I start to get this feeling that summarized by this slide that I'm standing around discussing science reform quite calmly all the time and meanwhile science is just like flipping out in the background um but this is just what it's like and we have to nothing's in equilibrium and the idea is for communities like the ones that I'm a member of and many of you are too we
• 23:30 - 24:00 need to keep discussing these things even though it's not clear how we're going to fix the whole system right now because we need to be ready with serious policy recommendations when the opportunity arises and it will arise because governments are getting really interested in that flipping out car in the background and when I talk to people in high level leadership uh they've all heard about this people in government uh heads of granting agencies uh the moment is coming when communities that are interested in open science and science reform are going to be able to have a big influence especially in Germany um
• 24:00 - 24:30 which is where I know people I talked to about this but also in the UK which has gotten really serious about it uh this is the hopeful message right you focus on the car but but there will come a time when we will have safety standards in place um okay so this is the argument about about the natural selection of bad science is science as a profession does a really good job of making successful scientists right and sort of how it is professors make professors right and so
• 24:30 - 25:00 um the professors who train their students to be good at being a professor those are the sorts of traditions that will propagate and so these are skills like how to get funding how to get published how to get cited how to give credit which is also a thing that helps you professionally um and and research skills themselves the professional aspects The Continuous integration the testing uh the programming skills if you will the kitchen etiquette where you leave your knife um how you avoid cutting your thumb off yeah things like
• 25:00 - 25:30 that these things are in most contexts informally transmitted people learn them at the bench with their colleagues in the lab Sciences they learn them from published papers in Behavioral Sciences uh there isn't a lot of professionalization about how these skills are taught people just kind of pick up coding right from their peers or from online tutorials um uh it we could do a lot better uh the things that are rarely taught but our essential professional
• 25:30 - 26:00 skills uh for the Sciences broadly defined are things like how we organize our data how we curate it how we do testing of our procedures which I think is almost never done people hardly ever test their code at all um how we manage distributed contributions this is the continuous integration analogy um and then this point at the end that I'm going to focus on a bit uh strongly for the remainder of my time here is The Logical connections between
• 26:00 - 26:30 hypothesis and data analysis and I think this is a real dark hole in the Sciences broadly defined I don't think there's a single science currently uh doing a good job at this even though in the stats Community there's been a lot of work on this over the last say hundred years um it's not the sort of thing that is really focused on instead we sort of focus on null hypothesis testing right and um uh but the logical connection between scientific hypothesis not null
• 26:30 - 27:00 hypothesis but the actual hypothesis and data analysis is something we could do a lot about desperate need to professionalize this more um so the couple things about sloth right so you all know that there are these audits of the replicability uh of studies and this is just to talk about the curation side of this before I pivot to the logic um so here's something that um a student in my department did a few years ago Rihanna mener did this big audit of the
• 27:00 - 27:30 social learning literature in humans and animals and just tried to figure out in What proportion of papers um the result could even be reproduced this isn't about whether it's right or not but can we reproduce it at all so this was a a huge task of emailing a bunch of people and trying to get raw data and then trying to actually take that data when it was available and reproduce a result um this is a basic quality control thing right uh and at the end here there's a bunch of details here um but at the end
• 27:30 - 28:00 uh the combined the reproducibility combined from being able to get the data and then being able to understand the data processing procedures as documented uh is only about 24% in this literature uh now I ask you if any any other profession you heard a number like this what would you think right there's a bunch of products being pushed out and in only about 20% of the cases um can we even understand and how they were produced and know that they're correct
• 28:00 - 28:30 yeah this is not acceptable yeah and the social learning literature is not special in this regard here's a great paper from anola and colleagues in 2020 doing something very similar in ecology trying to see just the availability of code uh 346 articles um randomly sampled and only about 20% is even potentially reproducible we've got to do better this is the curation and professionalization of data management code management right this should be easy uh we just need the
• 28:30 - 29:00 will and the standards and say that it's not okay that you don't have code um so on uh there's other detailed procedures that are closer more closely related to software engineering that we need training for and the the training is a serious thing that people like me have to commit to provide so at at my Institute every year we have a weeklong intensive course on these skills uh to help out students um but I know in a lot of programs there isn't an annoying person like me who's going to force that to happen but this is a
• 29:00 - 29:30 professionalization thing a campaign that I'm committed to things like Version Control um uh being being very uh serious about controlling our documents and understanding the edits this is like track changes in Microsoft Word but for code uh a very important aspect of of being professional about code development and then testing uh you know your code you want your code to be able to do something you should write code to test it and that's part of basic quality control as well and as anybody who's coded knows no matter how long you
• 29:30 - 30:00 do it you will make mistakes and so the testing is not optional and I uh any of you who watched my stats lectures know this um I test the data analysis by making synthetic simulations of the scientific hypothesis and that's also a kind of testing that's unique to science that you wouldn't see in software engineering but is also necessary is to say Can my data analysis pipeline even in principle get the analysis is right and we should prove that before we ever
• 30:00 - 30:30 introduce real data to our pipeline I think now of course I'm I'm a strict and annoying person I know I recognize that but I think that's a basic professional standard that I would like to to to lobby for documentation can we understand what people did um I've talked about continuous integration I think this is a positive message because there's this occupation called software engineering which has lots of training materials and has been professionalized and we can borrow a lot them we're going to have to adapt it to the way we work I'm not saying we become
• 30:30 - 31:00 software Engineers God forbid um I like being a scientist I don't want to be a software engineer but uh there's a lot we can benefit from here we don't have to develop it ourselves like they did yeah they had a really hard job doing this stuff okay um so yeah okay so this is about I should have put this up when I talked about Version Control yeah copying documents and uh renaming them is chos because future you is going to be very confused um by your folders right
• 31:00 - 31:30 version control um solves this quite a lot uh and how do you learn these things so if I mentioned at my Institute we teach a workshop on this every year and say you're interested in doing this yourself you don't have to invent it yourself there's this great organization um called software carpentry and they have a branch of their materials called Data carpentry which is really focused on scientists and essential skills for organizing data and and having Pipelines and managing projects with Version Control and we use these materials in my
• 31:30 - 32:00 Institute we modify them a little bit um we tune things but this is excellent It's really professionalized and they'll train instructors too you can send a member of your of your team to their uh instructor training we've done that here um it's really great I I hugely endorse this you can all the materials are free online you can go through it on your own at home with a bottle of wine if You' like it it's it's really great stuff so we're living in the future um we have benefited a huge amount from all the
• 32:00 - 32:30 professionalization in software engineering and the culture of things like this of data carpentry and software carpentry and we can uh use these things to to lift ourselves up now um yeah so say uh I work a lot with ecologists and so there's a whole module for ecologists actually and there's a module for psychologists so it's even getting customized now to the different kinds of of data needs and and cleaning problems that you have uh so I hugely endorse this and I think this is the kind of
• 32:30 - 33:00 thing this kind of basic data carpentry certification is kind of thing I'd like to see is a professionalization of being a scientist who works with data uh which is basically all of us right is to be able to say yes I've done this it should be so on our CVS it's not that you know oh I I can use Microsoft Word well of course you can but no you've gone through data carpentry and you've got the certificate that would be a minimum kind of professional standard I think um Version Control for big projects requires bigger tools and I just put
• 33:00 - 33:30 this slide up uh to advertise that those tools exist too but you need other Specialists to help that work so in my department we have this kind of continuous integration of distributed databases with different people contributing data from different aspects of the project and this has to be managed as well and we do all this with database Bas with version control on a database and it's very essential to do this kind of stuff too this would be uh impossible for an individual to manage without specialized training so this is the we need teams where we have support
• 33:30 - 34:00 staff who are focused on these sorts of things and at MOX pl we can afford this and I appreciate that in other places you don't but this is again just part of the professionalization and um research institutes that want to be taken seriously in the future I think will have to provide services like this um okay let me pivot to the logic and then I'll stop uh and and take some questions in the last few minutes um I'm very big on the idea that uh we we need
• 34:00 - 34:30 to prove logically that that our data analysis could work in principle uh and even in principle that is given some stated set of assumptions about our scientific hypothesis what we think is going on um would it be possible even in principle for our data analysis pipeline to work to show whether the hypothesis was true or not or to get the estimate uh that we want that's usually how I think of it um and I think the standard way that data analysis is done in the
• 34:30 - 35:00 Sciences does not address this at all uh people have quite vague heuristics about um how they develop an estimator for things uh and there's no testing of any kind anyway and so I I put up this this screenshot of UD Pearl's book because this is really the core of it is about causal inference which is most of scientific research is about inferring causes um and uh maybe you want to argue with me about that that later on but I defend that point quite strongly and
• 35:00 - 35:30 there is a science of this about being able to logically connect um a generative model of a of a natural phenomenon to an estimator that is a statistical procedure for studying it and that's what udip Pearl udip Pearl's probably made the largest contribution to this of any single person but there's hundreds of of mathematical uh uh uh statisticians computer scientists who work exactly on this issue and um we're living in the future again and we just need to to disseminate these insights so
• 35:30 - 36:00 a couple quick examples and then I'm going to stop um so here's here's a version of this that I was involved in personally with my colleagues where um people are are doing data analyses which are wholly unjustified there's no logic that connects their statistical procedure to the supposed scientific problem they're trying to solve and so that they're just publishing IR relevant estimates um and and in cases where it's provable that what they've done doesn't
• 36:00 - 36:30 make sense so there was this uh period years ago where I got involved in this policing biased policing debate about um whether um police mainly in North America uh tend to shoot uh minority individuals more and um and there were a bunch of papers on this trying to analyze administrative data and so on and there's a bunch of statistical problems with these sorts of papers which is how I got pulled into the Vortex um but also it's a very important civil rights issue right and and when you have
• 36:30 - 37:00 statistical knowledge I think you have an obligation to to work on these things when the opportunity arises but U so here's a paper that came out from Cesario and colleagues who are really trying to get things right these are serious people who really care about the Civil Rights issues and they're doing the best they can uh but no one trained them in how to do this logical connection between the scientific hypothesis and and the estimator so they do this adjustment they they recognize that there's this problem with the administrator records and they do a statistical adjustment right and um and
• 37:00 - 37:30 then they they report on what happened with it but their adjustment there adjustment and statistics could mean anything right it just means including some variable and um the particular adjustment they did just does not work and so my colleagues and I uh Cody Ross and Bruce winterhalder had this follow-up paper where we are we nerd out basically we like to do algebras kind of kind of people we are and we Prov that the estimator that SAR and I use all
• 37:30 - 38:00 used just doesn't work it doesn't solve the problem at all in fact it can actually um give you exactly the wrong answer in these cases now I don't mean to pick on Cesario this is just the case that I was involved in right but this kind of thing goes on a lot so let me give you one more example so uh there's this is one maybe a lot of you have heard of the hot hand effect so there's this weird North American sport called basketball and back in the 80s um uh some psychologist gilovich balone
• 38:00 - 38:30 and tersi published this paper arguing that this thing called a hotand doesn't really exist but basketball players believe it does the hot hand is his belief that players get on streaks and when they're on a streak a coach is supposed to recognize this and direct other players to pass the ball to them and that will improve your your odds of winning the game so they did a data analysis of some data from professional games and and said that the hot hand is a myth players and coaches believe in it but no statistical evidence of it this paper was hugely influential and it was
• 38:30 - 39:00 part of a big industry of arguing that people are irrational and and so on okay the paper's completely wrong totally wrong it's garbage never teach it okay here I am teaching it only teach it as a lesson of like how not to do data analysis so the problem with the the tersi paper is that their data analysis procedure was completely at hoc they didn't uh have any kind of generative representation of the hotand phenomenon and then derive an estimator or anything like that they just said Hey what if we just looked at sequences of three shots
• 39:00 - 39:30 and looked at how many streaks there are in there and um it turns out that this is a procedure that gives you deterministically the wrong answer and there's this great paper I show you on the screen from Miller and sanjuro I think this came out in 2018 or 2017 where they studi this estimator that tersi had all used and it is always bad um it's and it's especially bad in the way that derski and his colleagues used it um and when and Miller and sanjuro
• 39:30 - 40:00 then went on to derive an actual good estimator unbiased estimator and they find that the hot hand does exist and the coaches and players were correct yeah I love this story um so uh there's real value in in professionalizing this ability to connect scientific models to data analysis as well and it's obligation of people like me uh to produce the materials and the training materials to help you do that because this is not you're not on your own here we're all All in This Together We either we we all syn or float together right
• 40:00 - 40:30 and I think the simple version of this uh this is a schematic we have this four-step plan to success we want to express a theory as a probabilistic program um and uh uh in code and then prove that a planed analysis could work using that probabilistic program then we can step three we test it on synthetic data so we have the quality assurance and our colleagues can believe that the Mater at least will work in principle this would have saved to verki right um
• 40:30 - 41:00 and then we can run the thing on empirical data there are additional problems after that of interpretation and model rejection and maybe we go back to step one and sure everything is flow and I don't think this is a linear sequence but getting the quality control in here is really important and the tools exist we know how to do this in in as a community we just need to disseminate it and democratize it um there's a bunch of detail in work real workflows which is like chaos now I'm not going to work you through this but there's this great paper called basian workflow by Andrew Gilman and his
• 41:00 - 41:30 colleagues where they they problematize all the little subflows that are in real scientific data analysis um problems and there's a bunch to do here they they identify a bunch of areas for future research uh to say this is a really active area of thinking about scientific workflow in a detailed way and good things are going to come is what I want to say okay I'm going to stop there uh sorry if I run right to the end I guess um science is about continuous
• 41:30 - 42:00 integration and we should be professionals about this we should not be ashamed uh to show the public how we produced our results um or our work folders and I guess that's my only message thanks for your Indulgence thank you so much you're getting a lot of applauses here and online apparently oh yeah there's things floating on my screen that's kind of Haunting um okay i' I've I've heard a version of that presentation a while back but uh I I got
• 42:00 - 42:30 again very motivated so I hope the new listener got inspired as well I see there are several questions already in um in the Q&A box but we will also take um did I stop the recording