Blood, Sweat, and Tears, Meet Your Best Friend: AI / Seminar Day, Session II

Estimated read time: 1:20

Summary

This insightful video seminar, hosted by Caltech, explores the impactful role of AI in various industries, specifically focusing on healthcare and medical research. Moderated by Satoshi Otake from Pfizer, the panel includes AI and medical experts like Dr. Nima Anand Kumar, Christie Canaria, and David Van Valen. They discuss how AI is transforming medical imaging, drug development, and surgeries, while also managing large biological data sets. Challenges such as data integrity, privacy, and ethical considerations are addressed alongside innovations in AI technologies that mimic human intelligence. The conversation reflects on AI's potential, the importance of interdisciplinary research, and Caltech's pioneering role in AI advancement.

Highlights

AI's transformative impact on healthcare, especially in medical imaging and drug development, cannot be overstated. 🚀
Panelists emphasized the importance of interdisciplinary collaboration in advancing AI applications. 🤝
Federated learning is a promising solution to privacy concerns, allowing data to remain private yet useful. 🔍
Challenges in AI include ethical considerations and the need for large, high-quality datasets. 📊
Future AI innovations may lead to real-time feedback systems in surgeries, enhancing precision and outcomes. ⚕️

Key Takeaways

AI and healthcare have a symbiotic relationship that promises revolutionary changes in diagnostics and treatment. 🏥
Data is the new oil, driving AI advancements, but ensuring its integrity and privacy is crucial. 🔐
Caltech stands at the forefront of integrating AI into science, fostering cross-disciplinary collaborations. 🔬
The future of AI in healthcare involves creating systems that mimic human intelligence more closely. 🤖
Public sector plays a vital role in supporting and regulating AI advancements for societal benefit. 🌐

Overview

In this seminar hosted by Caltech, experts gathered to discuss the transformative power of AI in healthcare and research. Moderated by Satoshi Otake, the discussion featured prominent figures like Dr. Nima Anand Kumar, Christie Canaria, and David Van Valen, each bringing a unique perspective on the intersection of AI, medical research, and data science.

A significant theme was the integration of AI in healthcare, emphasizing advancements in medical imaging and personalized medicine. Panelists highlighted how AI tools are revolutionizing drug discovery, improving patient outcomes, and aiding medical professionals in decision-making processes.

The conversation also addressed hurdles such as data management, privacy concerns, and ethical dilemmas. With Caltech's leadership in pioneering AI research and fostering interdisciplinary collaboration, the panelists remain optimistic about overcoming these challenges and unleashing AI's full potential in creating a healthier, more innovative future.

Chapters

00:00 - 01:30: Introduction by Satoshi Otake The chapter titled 'Introduction by Satoshi Otake' is presented during the second session, where Satoshi Otake, the senior director of pharmaceutical research and development at Pfizer and chair of the Caltech Alumni Association Board of Directors, moderates discussions. Drawing inspiration from Winston Churchill and John McCarthy, the session is characterized by a theme of 'blood, sweat, and tears' as participants meet and explore AI technologies, labeled as their 'best friend'. Satoshi Otake is noted for holding over 10 patents, highlighting his expertise in the field.
01:30 - 05:00: Panel Introductions The chapter, 'Panel Introductions', begins with an acknowledgment and thanks to the panel moderator, Pat. The moderator then highlights the rapid advancements in pharmaceutical and medical technologies witnessed over the past year. In parallel, there have been significant developments in smart technology, affecting our daily lives through smartphones, smartwatches, and smart homes, setting the stage for the panel discussion.
05:00 - 08:30: Impact of AI in Medical Field This chapter discusses the integration of artificial intelligence in the medical field, focusing on the personalization of data according to user preferences. It explores the challenges and opportunities that arise in this domain and suggests strategies for optimizing the use of AI. It also highlights the interdisciplinary nature of AI's contribution to healthcare and introduces Dr. Nima Anand Kumar as a notable expert in the discussion.
08:30 - 12:00: Role of AI in Medical Procedures and Surgeries This chapter explores the contributions of a notable scientist with dual roles at Caltech and NVIDIA. Her work focuses on the intersection of AI and medical technology, evidenced by her involvement in multiple projects aimed at advancing medical procedures and surgeries. Notable projects include the development of a deep neural network for facilitating smoother and more energy-efficient drone landings, as well as the creation of quantum chemistry tools that leverage machine learning to significantly speed up calculations. Her achievements in the field have been recognized, as she was named an IEEE Fellow in 2020.
12:00 - 16:00: Trends in AI and Healthcare The chapter discusses notable figures in AI and healthcare, highlighting their achievements and contributions to the field. A notable recognition in 2018 was the New York Times' Good Tech Award received by a prominent individual for leveraging technology to create tangible benefits. This chapter introduces two panelists, including one who has an academic background from the Indian Institute of Technology and Cornell University. Another panelist, Christie Canaria, operates at the nexus of science and industry, aiding entrepreneurs and small biotech businesses, although she is participating here in a personal capacity as a member of the national panel.
16:00 - 19:00: What is AI: Definitions and Explanation The chapter titled 'What is AI: Definitions and Explanation' delves into various facets of artificial intelligence, providing definitions and explanations to elucidate the concept. It briefly mentions the role of small business innovation research (SBIR) programs geared towards supporting small businesses, particularly within the realms of biomedical imaging and next-generation sequencing. Additionally, there's a reference to machine learning as a crucial component within this sphere. The chapter also highlights Christie's background in science policy, which she has been involved with since 2013, particularly in Washington DC.
19:00 - 23:00: Research at Caltech: Anima Anandkumar's Work This chapter discusses the career and achievements of Anima Anandkumar at Caltech. Anima was awarded the AAS Science and Technology Policy Fellowship. Before that, she managed microscopy facilities at the Department of Energy's Lawrence Berkeley National Laboratory and the Beckman Institute of Biological Imaging Facility at Caltech. Her academic journey includes earning a BS from the University of California, San Diego, and a PhD from Caltech. Additionally, the chapter introduces David van Valen, who studies how living systems process information.
23:00 - 29:00: Research at Caltech: David van Valen's Work The chapter discusses the research work of David van Valen at Caltech, focusing on understanding how living systems store and transfer information, particularly in the context of diseases such as viral infections and cancer. The complexity of the task is highlighted, considering the vast number of genes even in the simplest organisms and the dynamic nature of living systems that evolve over time and space. The chapter mentions the use of genomics techniques to overcome these challenges.
29:00 - 35:00: Role of Public Sector in AI Advancement The chapter discusses the significant contributions of the public sector in advancing artificial intelligence (AI) technologies, focusing on research and development. It highlights notable figures and programs that have played pivotal roles in these advancements.
35:00 - 41:00: Challenges in Data Quality and Management The chapter titled 'Challenges in Data Quality and Management' discusses the involvement of MBA qualifications within the joint MD-PhD program at the David Geffen School of Medicine at UCLA. It introduces the panelists who will discuss the current state and future vision, particularly focusing on the integration of AI.
41:00 - 47:00: Overcoming Privacy Concerns in AI The chapter discusses the increasing visibility and interfaces of AI in daily life, particularly focusing on the impact in the medical field through tools like smart watches. It starts with a discussion led by David on AI's role in drug screening and enhancing drug development.
47:00 - 53:00: Future of AI Research and Caltech's Role The chapter explores the current methodologies in drug development, emphasizing the role of biological networks and gene targeting. It discusses the initial steps of creating disease models, applying perturbations to understand the underlying networks, and using this knowledge to identify target genes. This foundation leads to the development of small molecules or biologics, which are then tested for efficacy. The chapter outlines this process as part of understanding AI's potential in revolutionizing drug development.
53:00 - 64:00: Q&A Session: Audience Questions The chapter titled 'Q&A Session: Audience Questions' explores the transformative impact of artificial intelligence in the realm of disease mitigation and clinical trials. It discusses the role of AI in improving disease model profiling through advancements in imaging. The integration of AI with imaging technology provides deep insights into how different perturbations affect living systems. Additionally, the text touches upon the implications of AI in the development of small molecules and biologics. Overall, the chapter highlights AI's growing influence in enhancing the understanding and treatment of diseases.

Blood, Sweat, and Tears, Meet Your Best Friend: AI / Seminar Day, Session II Transcription

00:00 - 00:30 and now let's get to the business of the second session it sounds a little bit like winston churchill a lot of john mccarthy we call it blood sweat and tears meet your best friend ai satoshi otake is the moderator he's the senior director of pharmaceutical research and development at pfizer he holds something more than 10 patents and he chairs the caltech alumni association board of directors so thank you very
00:30 - 01:00 much for being with us for moderating this panel and i will turn control over to you great thank you pat now with advances in pharmaceutical medical technologies we've all witnessed certainly i have firsthand in the past year and then parallel advances in smart technology we've observed these in their daily lives smartphones smart watches smart homes now with that comes
01:00 - 01:30 personalization user preferences how are data generated what are some of the challenges and opportunities and what's the best way to utilize them and how do these disciplines interweave to create a better future i'm very excited to be hosting the session with diverse and talented panelists whom i'll introduced first is dr nima anand kumar anema works across academia industry
01:30 - 02:00 with dual positions at caltech and the gpu computing firm nvidia previously she was a principal scientist at amazon web services she has teamed up with caltech colleagues on a range of projects including a deep neural network to help drones land land more smoothly energy efficiently and quantum chemistry tools that use machine learning to perform calculations a thousand times faster than previously possible she was named an ie fellow in 2020
02:00 - 02:30 and in 2018 the new york times recognized her with its good tech award for using technology to help others in real tangible ways she received her bs from the indian institute of technology and her phd from cornell university the second panelist is christie canaria she works at the intersection of science and industry to support entrepreneurs and small businesses in biotech she appears today in her personal capacity but as a member of the national
02:30 - 03:00 cancer institute small business innovation research or sbir development center in which she provides programmatic support to small businesses applying to the sbir and the small business technology transfer research programs our portfolio includes technologies and biomedical imaging next generation sequencing and machine learning christie began her science policy work in washington dc in 2013
03:00 - 03:30 when she was awarded aas science and technology policy fellowship and previously she managed microscopy facilities at the department of energy lawrence berkeley national laboratory and the beckman institute of biological imaging facility at caltech she earned her bs from the university of california san diego and phd from caltech and the third panelist is david van valen he studies how living systems process
03:30 - 04:00 store and transfer information and aims to unravel how these processes are perturbed in disease states from viral infections to cancer the task is challenging on several fronts the sheer number of genes even in the simplest organisms and the fact that living systems evolve over time in space and the inability to average data across different types of cells and to overcome these challenges they would use this genomics techniques
04:00 - 04:30 to study the diverse genes in individual cells emission technologies to record cellular movement and behavior and david joined caltech as a visiting association 2017 was appointed associate assistant professor in 2018 he was named to the 2020 class of rita allen foundation scholars for research that holds exceptional promise for revealing new pathways to advance human health he holds bachelor's degree mathematics and physics from mit
04:30 - 05:00 and earn an mba as part of the joint md phd program and the david geffen school medicine at ucla i now welcome the panelists to join me on the screen thank you so much for the kind introduction thanks satoshi it's a pleasure to be here and welcome to all the alumni very good so let's start first talking about the current state and the vision for foreseeable future i'm particularly interested in integration of ai
05:00 - 05:30 in our daily lives um they're becoming more visible the interfaces are increasing as i mentioned the smart watches different ways in which you can monitor your health um what can you tell us the impact of ai has had in the medical field and why don't we start first with david what can you tell us about whether it's drug screening or ways in which we can enhance drug development that's a very uh a very good question um
05:30 - 06:00 so from the point of view of drug development you know you can sort of start with you know how you know drugs are sort of developed now right and so one might start with a a model a particular disease it could be some cell line model you apply some perturbations to map out the biological networks that are giving rise to that disease state once you know the networks you have a sense of what genes should be targeted and then you might develop some small molecules or some biologics to target to target them then you go back assess efficacy and if you have something that looks like it's efficacious at
06:00 - 06:30 mitigating disease you might pursue into into clinical trials and at each stage along the way um i think ai is starting to have a really transformative impact on how these different processes are done um so in the you know from the profiling of the disease models um we're seeing advances in imaging and what ai combined with imaging can do and what it can tell us about how perturbations affect living systems when you think about the uh the small molecules or the biologics
06:30 - 07:00 you know there are ai methods for mining you know large libraries of natural products there are ai methods for optimizing the performance of different receptors and different antibodies and then when you're assessing like the impact of of these therapeutics on uh or these potential therapeutics on on living systems their you know image-based profiling is having a pretty big uh impact and their ai is uh is sort of making it possible to interpret these images with an accuracy that was previously
07:00 - 07:30 impossible one thing that i would like to just throw into the conversation i'm not sure if it was uh you know mentioned before is that these ai methods are driven by data um it's the data sets um and so the the raw data combined with labels um so either machine generated labels human generator labels or whatnot um it's these two things in combination that let these methods do the you know the sometimes magical things that they do and so every time we talk about ai there's an algorithm component but
07:30 - 08:00 then there's also this data component um as well but yeah i think we're seeing some pretty some pretty transformative uh impacts um across the whole spectrum of drug development excellent thank you david and we'll definitely delve further um over this session here next um anemic what can you tell us about role ai has had uh in medical procedures surgeries and obviously you work at the interface between academia industries so can you share us more about what you've
08:00 - 08:30 observed in terms of both of these fields yeah certainly satoshi i think uh it's very exciting to be working at the intersection of ai and healthcare in fact healthcare has been one of the trending applications for ai in 2020 i think healthcare saw the most amount of investments in terms of startups so that tells you the appetite for using ai in healthcare as well as the immense advances that can come from this
08:30 - 09:00 an area where healthcare of healthcare where ai has seen the most application perhaps is medical imaging right we have large-scale data sets of all kinds of modalities of screening and uh being able to learn good models that can assess you know whether there are tumors you know what is the state of health here can make a big difference because radiologists today are in short supply and uh there's also
09:00 - 09:30 the time component if you could do this quickly maybe you can come up with more real time um you know insights into what's happening and so that's where at nvidia we've been building a lot of good infrastructure to enable hospitals to do this seamlessly and media clara is a platform that has a lot of pre-trained medical imaging models that you can easily start deploying and also you know then fine-tune that right because the idea is you could
09:30 - 10:00 start with pre-trained models but there's a lot of private data available how do you then improve the models very seamlessly and easily and as david was saying one important component is also labeling these large-scale data sets so ai assisting the labeling process could vastly speed up getting human generated labels so assisting the human in this label generation so i think that's one piece of the puzzle is getting good infrastructure
10:00 - 10:30 so that healthcare workers can quickly get started with ai models and further contribute their insights to it uh the other aspect looking forward is you know where do we go from images to let's say videos right like you mentioned surgery uh one of the projects uh i've been collaborating is with andrew hung at usc who's a medical doctor who specializes in robotic surgery so having these valuable data sets and insights of
10:30 - 11:00 how surgical procedures are connected today and what are the surgical outcomes after you know like the gestures of surgeons if we can assess and we can then predict uh how good the outcome will be and hopefully in future give real-time feedback to the surgeons right and maybe surgeons can plan out how they want to go about the surgery in very constrained spaces because every patient is different uh there could be lots of individualized um
11:00 - 11:30 planning of the surgery uh and i think that opens up a lot of uh really great ways to have a.i ate the surgeon i don't think we are anywhere near ai completely replacing the surgeon that would not be uh anything safe to do in the nearer term but ai can aid the surgeon in lots of interesting ways great thank you for sharing that anema now christine i see your role as an enabler in terms of some of the
11:30 - 12:00 exciting research that david and you may have just mentioned what are the trends that you're seeing in terms of ai healthcare space yeah thanks satoshi so i would say that in one word one of the things we're seeing is is growth and last night i went on to the public nih database to see what kinds of projects we were we were funding over over the years and there's been exponential growth in the kinds of projects that are being funded by the nih right now i think in 2015
12:00 - 12:30 a word search for ai machine learning and deep learning came up with 12 projects in 2015 and and now we're in the thousands as of 2020. so the field absolutely has been growing and i think that the trends that i'm seeing are the same sorts of things that anemia and david have also been mentioning i would say that a huge part of of applications for ai are in developing diagnostic imaging technologies and as someone who
12:30 - 13:00 as a grad student spend hours in the basement of the beckman institute at the biological imaging center i have firsthand knowledge uh and memories of manually segmenting images and so seeing how far technology has come and and the knowledge that researchers today have these new tools i'm quite jealous but also really pleased to see that the field can move that much faster with these resources for researchers i think the other area that is really growing right now is
13:00 - 13:30 natural language processing with the with the collection of so many electronic health and medical records we have a wealth of information there and new new technologies and methods to to sort of parse out and pull out key insights from clinicians notes and for medical records is another big area that we're looking at at it's seeing more and more of uh the other areas where i see a lot of growth are sort of the ones that david was referring to um designing therapeutics we have some really
13:30 - 14:00 interesting technologies trying to address radio pharmaceutical treatment therapy how do you dose patients properly or anticipate what kind of needs they're going to have how do you predict patient response or tumor response in risk in uh after receiving medication or care and how do you mine for the next wave of potential biomarkers for research tools those are the kinds of things we're seeing right now and every week i feel like i'm seeing new press releases coming out demonstrating collaborative efforts
14:00 - 14:30 between small businesses and larger businesses and academia so it's a really exciting time right now great thank you so much for sharing that insight christy so before we go any further maybe we could take a little bit of a pause here and and ask what what is ai because i think we all have very different perspectives our audience may have very different understanding of it or different perceptions on it so david you touched upon this briefly earlier so can you talk to us about what
14:30 - 15:00 is ai how does it work and what are the building blocks yeah that's a another good question um so the definition of ai um also you also deep learning is another term i use for the same for a similar collection of tools that i like to use in my class is a collection of methods that are able to derive insight from large scale data sets you know broadly speaking you have two categories um so you'll have categories that learn from data without labels and
15:00 - 15:30 then you'll have categories that learn from data with labels um category without labels usually called unsupervised um ai or unsupervised deep learning um the category with labels is called supervised uh ai or supervised um deep learning um and i think those are the you know those i guess that's how i would define it and how i would categorize it i'd say like there are you know nuances as far as like the you know the mathematical like um details of what these uh what these models actually look like
15:30 - 16:00 um but broadly speaking that's that's that's what i um that's what i like to use i'd say one theme one theme at least in the in the vision space um that these models have is that they extract features across different length skills and so you'll have a hierarchy of features to describe at different length scales that describes a particular image and then you can use those features to go and do do certain things if you're trying to classify an image if you're trying to identify relevant objects in an image um if you're doing segmentation um you
16:00 - 16:30 know those are those are different tasks that you can do with those features um yeah i guess that would be my definition um a collection of tools that can derive quote-unquote insights from from large-scale data sets thank you david uh anima christie anything else to add to this more of a definition for whether you're an expert or someone trying to get into this field sure um i you know david uh grave a great uh explanation uh i think i would go uh you know into
16:30 - 17:00 the fundamental question of what is intelligence right and there are many definitions to it but one definition i like is the ability to acquire and then now use those different skill sets in possibly changing environments so the acquisition is important right that's the learning aspect that we are able to learn and then we are able to use those skills and so if you use that notion of uh you know intelligence
17:00 - 17:30 into ai it means you have to learn from data and then you have to use those insights onto possibly you know new unseen scenarios and be able to generalize well and the other thing i would add that i think david has also really emphasized is the importance of data for these modern methods right so if you go back to classical ai uh that wasn't as heavily reliant on data especially you know when people were thinking about
17:30 - 18:00 probabilistic models there was more emphasis on humans being able to put in the right constraints engineer the right features so that could work with lower amounts of data but with the modern deep learning methods the whole emphasis is to learn the features from data itself because all models are wrong and maybe we are pretty bad at trying to figure out what's a good model for how to recognize if there's a dog in an image right because there's so many
18:00 - 18:30 variations of what a dog looks like so letting the model itself learn the features is what makes it flexible and what makes it possible to get the accuracy but that's only possible with large amounts of data and so just to trace back the history over the last decade of what made this deep learning revolution happen is what i call a trinity of data algorithms and compute coming together the imagenet challenge really
18:30 - 19:00 helped us scale up the amount of data available to researchers for the first time we had a million plus labeled images and then the gpu infrastructure having that parallelism of being able to do linear algebra very fast and neural networks are nothing but matrix multiplications with non-linearities that combination is what created this revolution so it's data neural networks and compute is the trinity of deep learning we see
19:00 - 19:30 today i might just add um not so much what is ai but when i look at what a lot of innovators in the space are doing they they talk about how they are approaching ai and it's something anemia pointed to which is um with specialized data sets and so when i see an application or a small company working in it i always sort of do a little bit of digging and look to see what they're working on and frequently they aren't developing new methods which
19:30 - 20:00 i know folks at caltech and elsewhere are working on um but many times they're bringing new data sets into systems and trying to apply these tools for specialized biomedical questions it's just an interesting way to think about what does ai mean to different populations great thank you so that was a very good introduction so let's go now to a little bit more depth so um let's first first start with anima and david um tell us more about the research that you're doing what is what is caltech involved in what is your area of expertise
20:00 - 20:30 uh tell us more about what you're working on certainly satoshi you know the beauty of celtic is there are just so many smart people around with diverse interests and so that continues to inform my research in interesting ways uh you know i'm focused both on building foundations of ai but also all these really exciting interdisciplinary applications which further give me the feedback on what kind of foundations do we need
20:30 - 21:00 for ai to work in such challenging applications i'll give you one example of this you know when i came to celtic the neuroscience program right is one of the best on the planet and in fact we celebrated 30-year anniversary for the computational neuroscience program a few years ago and i was one of the first to like realize this intersection between computation and neuroscience and so working with doriso who's neuroscientist and my colleague at
21:00 - 21:30 celtic the question we asked was what's missing currently in current ai methods compared to our human brain i mean there are such vast differences between how biologically we are processing information compared to the current artificial neural networks but is there some inspiration we can derive to improve these artificial networks and one key property we realized was the presence of feedback that we have in our own brains you know when you are seeing me on the
21:30 - 22:00 screen it's not just a feed-forward mechanism of information going from the retina to the visual cortex on top of it you also have the feedback from it cortex that's you know finally coming up with a perception of what you're seeing on the screen right and so what we have is this internal model that can fill in the gaps that can help us dream that can help us create that can help us hallucinate so that property is currently missing in the
22:00 - 22:30 current artificial neural networks because they just go through layers of processing and so what we found was adding feedback mechanisms that have a generative component you know that have an internal model so they do they don't just give an answer saying what's in an image but they have a feedback process to figure out is this really in the image and with what uncertainty can i say that it's in the image so that vastly improved the robustness of the
22:30 - 23:00 models i think that was earlier mentioned as well that these current neural networks tend to be fragile meaning you slightly add imperceptible noise into the image you know it can completely throw off its decisions right which is very dangerous if you think of applications like autonomous driving you know you can't trust these systems and what we found was adding this feedback made it a lot more robust even when it hasn't seen noisy samples during its training
23:00 - 23:30 which is what we like we want this ability to generalize to unseen scenarios much more seamlessly and i think that's an example of the kind of foundations we are building you know getting inspiration from neuroscience uh get looking at applications across campus to see you know what are what makes it hard for ai to be applied in these applications is their domain knowledge and constraints that can be integrated with flexible neural network models you know
23:30 - 24:00 this hybrid notion of add some domain knowledge at the rest of it to be flexible with neural networks what is that integration um so that's uh a quick overview of my research at celtic thank you anema david tell us more about your breakthrough research yes so i would say a large part of what we've been focusing on for the last three years is scalable scalable labeling of large biological imaging data sets
24:00 - 24:30 um so i think this is actually the real challenge um that the life sciences face if they want to unlock what what ai can do um for this field and so well for even for specifically for imaging data um for sequence data and structural data um obviously there are other there are other things that um that can be done um and so you know when you think about it it's actually a fairly uh difficult problem because images come in all different shapes forms and sizes with different different different biological systems
24:30 - 25:00 uh but broadly speaking like the labeling task is fairly conserved across all of them so generally speaking what people want to be able to capture are two different types of annotations so instance types of annotation so capturing different instances of different objects so for instance you might be looking at an image of a patient's tumor and you want to identify all the cells that are present in that in that tumor image but then there are semantic labels as well so labels that capture biological meaning and so for that same example you might also want to be able to say like okay
25:00 - 25:30 not only do i know that that's a cell but that's a t cell but that's a b cell that's a tumor cell that's a endothelial cell um for instance and so this this these two different types of labeling um sort of informs the types of software that need to be developed um to um to label these uh these image collections um and then there's also just the scale um so rather than labeling a few thousands of objects which you know most biological data sets um contain you know what we're trying to do is to have systems that can label milli
25:30 - 26:00 uh produce accurate labels for millions of objects um and so what we've been doing is just building um software tools for uh labeling um as anima mentioned uh in the nvidia clara incorporating ai elements so that the ai's actually help people label so rather than producing labels from scratch you're correcting um you know the mistakes of different um different ai models um and then doing this in this human loop uh approach where you're sort of iteratively building data sets and iteratively building uh models that are increasingly more
26:00 - 26:30 accurate and so i'd say like this is actually like fairly um fairly hard to do um in practice like we're a small lab we don't have the resources of giant companies uh empowering us but we have we have been like fairly uh fairly successful um so we've recently uh published a preprint on one data set called tissue net um where we annotated about 1.3 million cells across a variety of images uh of hum of tissues so i want to say from nine different organs on pancreas immune
26:30 - 27:00 tissues lung colon different disease states cancers not cancerous inflamed not inflamed um and then also across different imaging platforms that people use um so both imaging with light and then also imaging with mass mass cytometry and from this large image collection we've been able to train ai models that can you know perform uh cell segmentation um effectively with human level accuracy um so that's judged by comparing inner annotator agreement so how how
27:00 - 27:30 much will two humans disagree and then comparing that with human model um disagreement how much will a human and a model disagree and those two are actually indistinguishable from the models we've been able to generate um and then we've also asked uh you know a panel of board certified pathologists you know in a blinded fashion you know here's what humans would produce here's what uh this ai produces which do you prefer um and we've been able to show that they don't really display a preference um which is which is nice um so these sort of models are you know they have clinical applications um they have applications for um you know assisting like various
27:30 - 28:00 elements of drug drug discovery um they're going to be essential for projects that seek to you know map out all the different um cell types and their spatial spatial locations um across the human body uh it's just broadly useful and i think people for you know the last few years have known like we just this field has needed a tool like that um the challenge uh wasn't the wasn't building like the ai the ai methods the challenge was actually building these data sets um and so building tools to build these data sets in a scalable way
28:00 - 28:30 um and reduce the marginal cost of annotation um that's sort of what we've been spending most of our time doing great thank you for the explanation there david um christy can you tell us more about your role how did you get into your role um and you've alluded to this a little earlier how what do you see as the uh the role of the public sector um in advancing or easing the adoption of these ai into medical fields or let's say in our daily lives thanks satish you know that's a two-part
28:30 - 29:00 question so i'll quickly go over the fast forward through my time after caltech i was a research scientist at lawrence berkeley national lab and for those who don't know berkeley lab is a government lab managed for the department of energy by uc berkeley this is similar to how jpl is managed managed for nasa by caltech and so at my time at lawrence berkeley lab i had a really great window into how the government oversees
29:00 - 29:30 influences and supports um r d activities through programs and funding and the berkeley community itself has a has a strong and strong long history of civic engagement and so being a scientist there i had a lot of wonderfully positive opportunities to talk about the science happening at birthday lab to the the community and to local governments and so through that time there i really found my sweet spot in in being able to speak science to a different set of stakeholders um and helping to translate the talk of
29:30 - 30:00 technology to others um and so after after berkeley lab i took on the aaas science and tech policy fellowship which is a program that takes stem professionals and puts them into government federal uh and policy-making offices in the washington dc area so i i moved over here to the dc area in 2013 and i've been working in government since i would also just plug that anyone who is interested in applying their skill sets as stem
30:00 - 30:30 knowledgeable professionals into policy making and helping to build a world where we use data to drive our decision making i encourage you to look at these kinds of programs because you can have a really positive influence on our on our society and the way we work but when we think about policy and the role of large institutions like the federal government in supporting ai r d i think that the role there is really in building and supporting the infrastructure for force for ai innovation
30:30 - 31:00 and i think that is demonstrated with resources and funding for ai r d grants and contracts resources that way i think that's a huge part of the equation i also think that federal agencies can play a role in increasing access to others researchers in accessing and using federal data sets and models and of course to also train the next generation of an ai ready workforce i think those are some of the ways that public
31:00 - 31:30 resources like the federal government can help to support the ai infrastructure all right thank you chrissy let's now talk about challenges and opportunities um [Music] david anima you talked about it during your discussion about your research about the importance of data so how do we ensure the integrity of quality data how should they be managed um because right now compared to say 10 years ago even 20 years ago
31:30 - 32:00 there's just so much data so how do you know that the data that we're using is good that fits into the model and what is the future of data management and so i know that's a fully loaded question um anima why don't we start with uh with you uh see if you can share your thoughts on the quality data yeah certainly satoshi i mean data is the new oil right and that's what david has also been emphasizing and focusing so much on getting this high quality data and so
32:00 - 32:30 that also as christie mentioned for small startups that also becomes a differentiating aspect if they can get to good data and in the healthcare space the major challenge is the fragmentation we see in the u.s and of course the hinbar compliance and all the privacy constraints right so how do we have innovation going while respecting privacy while keeping the data integrity i think that's where techniques like federated learning uh can make a
32:30 - 33:00 big difference as an example at nvidia we trained a lung model early in the pandemic just as you know covet was spreading across the world where there were more than i think 13 hospitals that used their data sets but without having to share all that together into a central server so they kept the privacy of their data only share the gradients and could train a joint model across these hospitals and so
33:00 - 33:30 formalizing techniques like this and asking you know how much is privacy compromised when you're sharing gradients and not all of the data right can you invert it to the individual data points and and even there you know is it the privacy of the individual specific aspects uh demographic aspects may be uh sensitive like which features to protect and which are okay for not to protect right that's the other aspect we want to look into so i think this privacy is one
33:30 - 34:00 big aspect for healthcare and more broadly trustworthiness you know do you trust what the ai model is saying here you know if the ai model is saying there's no tumor with 90 certainty is that something that you can trust without having a human always having to check in detail or is the human also going to be biased because the ai is already saying there's no tumor right these are tricky questions uh of how
34:00 - 34:30 uh you know we make our downstream decision making based on what the ai model is saying and that's where good calibration methods meaning can we also get models where we can trust their uncertainties currently deep learning the standard models have no calibration so meaning they tend to be over confident when they're wrong which is terrible for the healthcare field so coming up with techniques on how to improve that
34:30 - 35:00 and that's where we are building new foundations based on what we call distributional robustness meaning even to different changes in the underlying data distribution can you tolerate uh those perturbations and how can you guarantee that you tolerate that well i think those algorithmic techniques will be integrated with human studies in terms of can these uh models be ultimately tested i think those are some interesting challenges
35:00 - 35:30 and i would add the other aspect is also these interdisciplinary collaborations right i mentioned my collaboration with the usc doctors as one example another has been with across campus at caltech we have ai for science initiative that i co-founded with the song you who is my colleague also in annenberg and our goal has been to really enhance these kind of interactions
35:30 - 36:00 you know going from organic peer to peer to an organization where people are coming together we have machine learning students volunteering their time to understand what are challenges faced by other domain scientists and how we can you know go from directly applying existing tools in ai to actually inventing new tools for uh addressing the challenges in these applications so ai for science broadly is something
36:00 - 36:30 that i have high hopes for what it can do at caltech and beyond yeah that was a very comprehensive answer and i'm struggling to see like what i can add to it um but i think the one thing i i will say is you know in the healthcare space privacy um is paramount but in the life science space i i do think there needs to be um an openness to some of the data sets because people are just aren't going to trust um the models that are being produced
36:30 - 37:00 uh without being able to see like what data is actually um actually backing in and i think as part of building those large public label data sets i do think there needs to be a change in the incentive structure of how science operates because right now if you're a life scientist operating you're not really incentivized to share like that data um so you know anime said data is the new oil um i agree one thousand percent uh the way it works if you're a
37:00 - 37:30 traditional life science researcher you collect a data set um you basically strip mine it for every single useful insight that you can get and then you go off and you publish your paper and then after you get your quote unquote academic credit then you might make the data publicly available some people deposit it in public databases others will say okay well you have to email me to get the data and only then i'll release it if um i think like you're not going to be actively competing with me um and you
37:30 - 38:00 know right now like that's just how our system um is built because that's that's what's incentivized if you get the shiny on the shiny paper and you get it first then you get rewarded um in the form of you know accolades and grant dollars and whatnot um and i think there needs to be some shift in how in this incentive structure to incentivize people sharing data sets making them more open making them public making them available earlier and in more i'd say usable and
38:00 - 38:30 curated formats until we get that change of incentive structure um you know i have a hard time seeing you know big changes thank you david uh we've touched upon the privacy concerns there um so christy over to you in terms of thinking about advances in technology um current accepted practices whether they're regulatory public acceptance how do we overcome some of these concerns in terms of you know the artificial intelligence
38:30 - 39:00 proposals may be coming but if they're based on data but there may be some concerns what are the things that we can help to to raise some of those fears yeah that's a great question it's and my fellow panelists have sort of pulled out some of the key things for for david it's it's about um the the data and incentivizing how data is collected and shared so that we have something to analyze and how anemo is referencing um trust also in the data i think the opportunity and
39:00 - 39:30 challenges are so the opportunity is that you know ai is one of the most impactful technologies of the 21st century it's across it's affecting every sector of the economy and the society and our researchers are are taking advantage of these tools that have been developed for other sectors um but many of those tools were not developed with biomedical data in mind and so if we're going to have ai um algorithms platforms methods that we can trust with our health um
39:30 - 40:00 it's going to require that we start asking we start building these platforms with the biological questions in mind as opposed to taking algorithms that work in the finance sector and trying to fit them around our you know mri data or our digital health uh data sets and so i think that what's going to the opportunity there then is building and building systems around the questions and having the questions start
40:00 - 40:30 from the bio life health care space and so it's going to require that these algorithms accept and handle different kinds of data longitudinal data multimodal data noisy data incomplete imperfect data such as the incomplete and imperfect people that they come from and then you know building into that of course the patient data and privacy issues addressing all of those once we can start getting there i think we will start having tools that
40:30 - 41:00 we can trust what's coming out of the algorithms out of the ai what it's trying to tell us and and i think it starts with those with those considerations in mind so i would say the opportunity absolutely is there to start building these methods that begin with asking the biological questions and interp interpret um our ability to interpret the ai models will be important for building trust for what comes out of those um algorithms thank you christy now over to you david
41:00 - 41:30 and anima now you've shared earlier about what type of research that you're focusing on now now obviously there's a lot more that's headed our way i'm sure christy and her colleagues are going to be seeing the next wave and then of course downstream the industry to be able to adopt and implement so where do you foresee your research evolving over time and also where do you see caltech playing a role in shaping the future of this field so
41:30 - 42:00 why don't we start first with david uh that is a good question um so i would say for my own research um they're sort of two um diver well i'd say divert diverging but also related paths um the first is actually like deploying these systems uh within our lab to accelerate the life science projects that we have ongoing um we spent a substantial amount of time and effort building systems that can accurately do cell segmentation cell tracking
42:00 - 42:30 interpreting both uh multiplexed images um of tissues but also um dynamic lifestyle imaging uh movies of cells and cell culture and then using these using these methods to really accelerate um like our internal biological discovery i think that's something that is uh that's that we're going to pursue uh there are challenges beyond just the algorithm to get the algorithms to be useful um so their software engineering challenges deployment challenges ux challenges and whatnot and sort of sorting those things out as
42:30 - 43:00 we start deploying them in the um in the laboratory i think is going to be um you know one one area of research um the second is uh actually going beyond just you know using these ai methods to automate analyses that people used to do manually um but really figuring out how can we integrate ai into the actual design of experiments and into the into the instruments that are performing the measurements um you know so you could imagine having you know microscopes that accounted for what ai methods can do with respect to
43:00 - 43:30 you know segmentation processing denoising and whatnot and you know use you know intelligently you know use that knowledge to perform better on better measurements so i'd say research of that of that variety where you're actually accounting for what ai can do and using that to build better more scalable measurements um that i think is you know what we're likely to be up to for i'd say the next you know three or five years yeah uh i think there's such an exciting
43:30 - 44:00 time to be working in ai because you know we suddenly are kind of right opening up to a host of challenges but also principled ways to move forward as i mentioned you know looking at the shortcomings of the current methods which mostly required label data or what we call supervised learning can we go beyond to do unsupervised learning and in fact this is also biologically inspired because infants only do unsupervised learning
44:00 - 44:30 right in the first years of their lives they're just observing the world around them and language only comes later so they're only getting supervision later but that's not how we are training most of our ai models so is it possible to look at patterns and data structure in data itself and clean insights and this in fact was my research even before deep learning came along right and the principle i had you know i
44:30 - 45:00 introduced was using tensors so how you can use higher order statistics in data higher order relationships to glean more insights and do that in a scalable way and now we are seeing the renaissance of tensors even further into deep learning as i mentioned earlier these neural network layers tend to be matrix multiplications but there's no reason to use them as matrices except that you know we had all the great linear algebra tools
45:00 - 45:30 right if you use them as tensor operations with higher order interactions you can capture the data structure much better so think about data like videos or 3d point cloud all these multi-dimensional data sets how we can use the power of tensors to design architectures uh that can work better on them that can generalize better because of the inductive bias so that's one area of like you know going beyond supervision
45:30 - 46:00 uh to look at unsupervised learning methods and also self-supervised learning methods you know how can we create our own supervision if we like transform the data can we still uh have invariance of what the what it what there is for instance in the image right for instance rotating the image we still know what is in the image so can we teach our ai models to also have that kind of invariance uh so that's i think uh really at the
46:00 - 46:30 core of it is can we reduce our dependence on label data um another important aspect is embodied ai right like bringing the mind and body together right now if you look at most of robotics uh there is zero intelligence you know you may have seen all the boston dynamics atlas robot doing very cool backflips doing a dance but it's all pre-programmed and so that means there is no intelligence when it's all pre-programmed so can we
46:30 - 47:00 have uh the uh you know embodied ai be able to interact with the environment learn new skills adapt and how do we build good benchmarks for that and that's where the role of simulation is really important so we are investigating how to build physically valid uh realistic simulations but still account for the distributional shift you know you cannot overfit to simulations because ultimately it has to work in the real world so generalizable ai that can work in a
47:00 - 47:30 variety of domains uh you know becomes the foundational questions there and more broadly you know what i mentioned for celtic the ai for science initiative is something we're very excited about and you know this has enabled me to work on such diversity of topics right everything from seismology to quantum chemistry fluid turbulence i know i just would have never dreamed of being able to
47:30 - 48:00 impact multiple areas in this way and that comes from having colleagues who are extremely open-minded who want to investigate how ai can be employed and working in this closed fashion so we have now have methods that can speed up navier-stokes equation which is perhaps the most challenging pde for modeling turbulent flows we can speed it up thousands of times compared to traditional numerical methods um same
48:00 - 48:30 with quantum chemistry uh that i think satoshi you mentioned in the introduction so having the ability to impact multiple areas all at once comes by thinking of initiatives like this that you know impact all areas of the campus that looks at computation at ai as the centerpiece and how we can train our also future generation of alumni our current students
48:30 - 49:00 in a way to have this common language of ai right from the beginning so you know they're always thinking in a computational way they're always thinking how they can apply ai in lots of creative ways yeah uh if i could just add um a little bit uh on top of what on anema said i think one of the things that makes caltech like very special is the core curriculum um so the fact that you can just rely on every student here knowing calculus knowing physics knowing chemistry and whatnot and the fact that you can rely on people
49:00 - 49:30 having you know the substrate for uh understanding like quantitative uh you know science means that you have a you have the potential for a workforce that you can just train to adapt on these ai methods uh my lab hosts a large number of undergraduates every year um as as surf fellows uh at different stages of training um freshmen sophomores juniors and also seniors and i'd say it would be hard
49:30 - 50:00 to do that um and to have these students um engage in the with these ai methods uh at any other place um that's just we can do it here because like again like our students have this uh this core um this core quantitative quantitative background uh i would say on um on top of you know offering that um as something that can like sort of move this whole space forward um i'd say caltech does tend to be more forward-looking and is willing to let researchers
50:00 - 50:30 uh take risks and take chances to find new things that will work um in ways that you know at other places like it would be it'd be fairly hard to do um and i think like that um that spirit i think makes it easy to um to sort of operate in these spaces um here at caltech yeah so i just add those two things very good thank you i know we're we're at time here i know we have a couple more topics we could easily cover so
50:30 - 51:00 hopefully they'll show up in q a so i'd like to invite pat back on here to be able to take some of the um q a from our questions from our audience and um we will do our best to answer them thank you satoshi some will be directed to specific panelists and those who that aren't satoshi you'll know the best place for those questions to land so so let's hear from lee who graduated in 63 wondering how will ai intersect with the
51:00 - 51:30 fabulous recent success of mrna vaccine technology which is another new and cutting-edge technology david you want to take that one first and then i can add uh that is a good question um you know i'd say like i'm not entirely certain i'd say some of the things that i can um that i can think of is predicting uh you know which viral variants are likely to be problematic and and
51:30 - 52:00 exhibit immune escape and so if you can understand that a priori you can sort of build them into the mrna vaccines um actually i'd say like that that probably like my most likely answer i need to think about a little bit more to get um to get some more ideas but yeah yeah i i agree with david on that um with many of the variants coming which do only require a couple sequence mutations this is where ai can help develop
52:00 - 52:30 the various variants which then will translate to the actual product and so rather than doing things manually and this is where uh the data that's been generated getting the feedback uh from the outside into what we do it will only accelerate so that we're not developing hundreds or thousands of these mrna therapeutics for only one of them to work out or rather it'll be a higher proportion of it satoshi do you know whether ai was involved in creating any of these new vaccines
52:30 - 53:00 um i i would say yes um i can't give you the exact thing and i can't speak obviously for uh for for the pfizer beyond tech um development but there are a lot more involvement of ai when it comes to target sequencing whether it's an actual development of the the drug itself or the biotherapeutic to increase potency to increase binding but the reality is those are really very small types of studies that you do
53:00 - 53:30 but in the end when it comes into the clinical trials then it's the cascading impact which is really difficult to predict on a very small in vitro type of study so this is where having the the deep learning um but looking at the other effects would really help make sure that the the therapeutic that's being developed is intended to be the way it has in terms of efficacy so um i would say yes and also i would say yes there's going to be more coming uh john who has graduated in 71 wants to know is ai democratic for example can
53:30 - 54:00 ordinary people use ai for their own health uh so i would say it's democratic in the sense that a lot of the libraries that are used to develop ai methods are open source and as long as you have some familiarity with programming calculus and linear algebra then you can engage with these methods and i think that's one of the remarkable things that the technology companies have done over the last you know five five or so years is the software the investments they've made and building those at those uh ecosystems
54:00 - 54:30 um the ones i'm specifically speaking of uh pi torch um so loosely associated with facebook um and tensorflow slash keras uh loosely associated with google um so i would say like they're democratic in the sense that you know anybody who wants to engage with these uh methods in these libraries can that being said you need to have the both the you know requisite uh expertise to know what you're doing so that requires like a time investment and then if you're you need to have data uh
54:30 - 55:00 which you know actually contains the information of whatever it is they want to do and you know if your data sets are beyond a certain size which almost all of them are then you need to have like the appropriate computational infrastructure to be able to train models on them and you know 9.5 times out of 10 that requires having gpu acceleration um but if you have those things then yes it's quote unquote democratic here's a question for anema from tyler who was graduated in 86 a.i he says always seems to rely on pre-assumptions and prejudices for example hearing a noise in the dark one person
55:00 - 55:30 thinks dear and another person thinks madman with a knife so how do you train ai to read between the lines yeah i think that's a great question and an unsolved one right because uh you know like we've emphasized during this whole panel a lot of deep learning relies on label data so you know it depends on what the human labeling is being fed into ai so it's garbage in garbage out so if you have labels that do not convey the right
55:30 - 56:00 information that are extremely noisy then the model cannot learn a good one and so there's a question i mean can we reduce that reliance and find other ways to directly you know look for structures and data which is unsupervised learning you know for instance causal inference we humans have an intuitive ability to judge cause and effect you know for instance this mug if i drop i know you know kind of like which angle it's likely to drop
56:00 - 56:30 and it's gonna spill the liquid so those are questions that are still hard for current ai because you know we wanna like have it be able to do this on let's say directly by looking at videos right which are high dimensional and be able to extrapolate to all kinds of new scenarios uh that's uh you know difficult for current just supervised learning methods um and so i think going beyond the dependence of uh
56:30 - 57:00 human labels is a great way to start yeah i sorry i would i would added a an alternative view which is that there's sort of this uh underlying assumption that you have this data set you train the model and like that's it and the data set itself never changes uh if you operate in a world where you're always improving your data sets where you're always adding additional labels you have an opportunity to improve model performance on these cases that where you identify that your model is like underperforming um so i think that's that's a
57:00 - 57:30 alternative view it's more capital and more labor intensive uh than if you had you know uh self-supervisor unsupervised approaches that don't rely on human labels but if you need tools now um it's a viable um it's a viable approach ari has a question that arises about every new iteration of technology which is the ethical one how do you determine what values we impart to ai who do who determines those values and who enforces them and how and that's a
57:30 - 58:00 very difficult question right because uh we as a society have to do this and we can only do this if everybody is empowered and enabled to understand what ai is doing uh you know one difficulty is these are black box models you know you don't have any kind of human interpretable explainability of why the model is making certain decisions and so that's where there is a lot of importance to educate the public of what are the current capabilities of ai right i mean there's a lot of hype
58:00 - 58:30 around ai uh you know like people saying self-driving cars would have been already here yesterday but that's not the truth right there is a lot of inability for ai to work in real world scenarios that are safety critical uh that you know cannot handle the long tail of real world possible scenarios so i think it starts to me with awareness and education but yeah it's a very difficult question in the long run of how do we go
58:30 - 59:00 through the ethical questions and frame policies uh so i would say the current scenario is not what is sustainable right it can't be a few big tech companies dictating this i think that certainly has to change i want to add from the policy perspective and pull on some questions that others had asked about democratization and our ability as individuals to influence what's important for us and set our own standards earlier this month there was an um there was a website launched
59:00 - 59:30 ai.gov ai.gov and the national ai initiative was launched publicly and so there are communities of practice where the public and the federal workforce can join and engage in these exact kinds of conversations i would say that um when you if you read the national ai initiative there are these same questions posed what are the pillars that we're going to stand up to build an infrastructure to support air and d moving forward and one of those is this idea of trustworthiness in ai and
59:30 - 60:00 and so it's a very um important question to be asking and one that we are interested in hearing from from people about how they feel about it so if you ever look for these um requests for information rfis uh we frequently put these out from the federal government side and and solicit input from from individuals for how they think things should be done there are a lot of questions along those ethical lines but a lot about the programming practicalities and here's scott who got his phd in 78
60:00 - 60:30 saying look a two-year-old child can identify a car by seeing a relatively small number of examples by contrast deep learning requires hundreds of thousands of example images does this suggest that a different ai approach is needed in terms of of training or directing the ai yes certainly i think you know that's something i've tried to emphasize right like how we can go beyond supervised learning to self-supervised and unsupervised learning and so what scott is referring to it's
60:30 - 61:00 called few short learning you know the or even zero shot like a child that has never seen a dolphin will still understand it's a living thing it's able to swim in water you know so that ability is something that's missing in most of the current algorithms because it's more focused on pattern recognition as opposed to understanding the concept and reasoning over that so if you're interested we have a recent paper called bongard logo which goes back to the bongar challenge in the 60s
61:00 - 61:30 that was more used for testing human cognition abilities you know they're simple analogy making puzzles that humans can do very easily they can easily tell what is the concept in these shapes but that is still super challenging even for the most advanced deep learning methods so having a benchmark like that and having these kind of challenges will help us evaluate what's missing and what needs to be built and interestingly what we found was neurosymbolic approaches you know these
61:30 - 62:00 are hybrid approaches that have a neural component which is right looking at continuous representations and a symbolic component that looks at the program that generated these shapes you know can it infer what was the you know program that kind of generated the shape that we are seeing and using that kind of symbolic grounding greatly improves few short learning you know there's a lot of evidence in our brain that we have we ground symbols that's how we remember concepts
62:00 - 62:30 so building that into ai can improve these few short learning capabilities we have time for one more question and only one answer and that would be i'm sorry uh i would add um you know what if making those hundred thousand labels were cheap um would you care that's the only question i would ask um no fine canon was graduated and 60 says when i studied digital imaging processing in neural networks decades ago performance was greatly limited by the available hardware so how was that changed oh absolutely that's uh what has been
62:30 - 63:00 the reason behind the previous ai winter right both compute and data were there now we have web scale data and we have gpu computing so the availability of parallel computing has fundamentally transformed what we can do with neural networks now we are in the mode of trillion parameter language models and it's mind-boggling you know the amount of compute power that is available uh but i would add that
63:00 - 63:30 i don't think this itself will get us all the way to agi right as i mentioned so many aspects of our human abilities are still missing with the foundations of our current ai algorithms so we need the synergy of better foundations with also better hardware in the coming years so thank you to satoshi and thank you to the entire panel there's so many questions we couldn't get to so i'm sure you'll be contacted or at least find people who are looking at your papers and looking at your work online to find answers to
63:30 - 64:00 questions to this engrossing cutting edge part of the 21st century one of the tools that we'll be embracing in our human future thanks we'll see you then you