Exploring the Future of AI in Finance

2025 03 27 AI Roundtable Panel 04

Estimated read time: 1:20

Summary

The U.S. Securities and Exchange Commission hosted a roundtable discussion on the future trends and impact of AI in the financial sector. The panel featured experts like Marco Enriquez, Hardep Walia, Tyler Durr, Peter Slatterie, and Sarah Hammer, who delved into topics including the role of generative AI, agentic workflows, and the ongoing challenges of automation. They explored the potential for AI to transform financial operations, discussed the associated risks such as biases, and touched on how companies can prepare for these changes. Key issues such as cost, access, and the need for collaborative governance were also discussed, alongside the necessity for continuous learning and adaptation in employing AI technologies.

Highlights

Marco Enriquez kicks off a lively discussion on the future of AI in finance, highlighting generative AI's role ✨
Hardep Walia shares insights on AI transformation at Charles Schwab and the significance of personalized AI solutions 💡
Peter Slatterie from MIT Future Tech discusses algorithmic trends and economic implications 🔍
Concerns around AI-induced biases and fairness in financial systems are addressed 🧐
A lively debate ensues on how cost and digital divide impact AI's adoption across industries 💸

Key Takeaways

Generative AI is rapidly reshaping the financial industry but still faces significant challenges and risks 🤖
Partnerships among industry, regulators, and academia are vital for navigating AI's impact 📚
Automation will likely result in partial, not full, disruption of existing job roles in the near future 👷‍♀️
Cost and access disparities could widen with AI advancements unless addressed ⚖️
Effective risk management of AI requires dynamic and ongoing evaluation of tools and methodologies 🔍

Overview

At the SEC's AI Roundtable, experts tackled the future of AI in finance, focusing heavily on generative AI and its potential disruptions. The panel was led by Marco Enriquez, and included diverse voices from tech and academia, such as Hardep Walia, Tyler Durr, Peter Slatterie, and Sarah Hammer. Discussions orbited around how generative AI could streamline financial operations and the looming complexities that accompany deploying such technologies.

A significant portion of the discussion ventured into the hurdles of adopting AI technologies within financial institutions. The experts mentioned barriers like the high costs, legal liabilities, technical debt, and talent shortages, which could thwart seamless AI integration. They highlighted that although AI can greatly optimize operations, it is crucial to implement it judiciously to avoid dependencies or unforeseen biases.

The panel concluded on a collaborative note, emphasizing the need for joint efforts among industries, regulators, and academia to ensure responsible AI usage. They advocated for frameworks that prioritize governance and risk management while also supporting innovation. As industries gear up for AI-powered transformations, continuous dialogue and adaptation remain paramount to harness AI's benefits effectively.

Chapters

00:00 - 04:00: Introduction and Panelist Introductions The chapter begins with the introduction of the final panel at an AI round table hosted by the SEC, focused on discussing future trends and the next steps for the financial industry. Marco Enriquez, an applied mathematician and leader of the Office of Data Science in the Division of Economic and Risk Analysis, welcomes the audience and introduces the panelists.
04:00 - 25:00: Generative AI in Finance In this chapter titled 'Generative AI in Finance', Hardep Walia introduces himself as part of Charles Schwab, where he leads AI and personalization efforts. He discusses his role in driving AI transformation and overseeing the development of AI-enabled products. Hardep joined Schwab through the acquisition of a fintech company he founded called Motif, which was pioneering in using Natural Language Processing (NLP) on novel data sets to create investment products.
25:00 - 41:00: Risk Management and AI Governance The chapter titled 'Risk Management and AI Governance' discusses the role of Tyler Durr, the Chief Technology Officer at a fintech company called Broadridge. He leads various departments such as technology engineering, infrastructure, cybersecurity, risk management, and Artificial Intelligence. The chapter explains how they have established a team, similar to agile squads, to assess risk, legal compliance, and product levels. The focus is on both retail investors and institutions, ensuring comprehensive risk management and governance of AI.
41:00 - 55:00: Pricing and Talent in AI Industry The chapter discusses the importance of cross-functional teams and leadership roles in the rapidly evolving AI industry. It highlights the need for robust architecture planning by platform teams. Peter Slatterie introduces himself, mentioning his non-finance background at a lab.
55:00 - 76:00: Transformative Impact of AI The chapter titled 'Transformative Impact of AI' focuses on the factors driving progress in artificial intelligence and computation. It discusses trends in algorithmic improvements, data, and hardware. Furthermore, it explores the economic implications of these advancements. A significant part of the discussion is about how organizations are responding to the risks posed by artificial intelligence, aiming to provide high-level insights into these responses.
76:00 - 77:00: Regulatory Collaboration and Conclusion The chapter begins with an individual expressing a willingness to learn and contribute despite not being as involved in the details as some colleagues. Sarah Hammer, the Executive Director at the Wharton School, then speaks, expressing gratitude for the invitation to the event and the opportunity it presents. She leads strategic initiatives in finance at Wharton and emphasizes her leadership role both within the school and at the Wharton Finance research centers. The chapter highlights the importance of regulatory collaboration and wraps up with concluding remarks.

2025 03 27 AI Roundtable Panel 04 Transcription

00:00 - 00:30 all right uh good afternoon welcome to the final panel of uh right uh this the final panel in this uh AI round table that the SEC is hosting uh in this panel we'll discuss future trends and what's next for the financial industry my name is Marco Enriquez i'm an applied mathematician and I lead the office of data science in the division of economic and risk analysis i'll invite our uh esteemed panelists to introduce
00:30 - 01:00 themselves starting with Hardep uh good afternoon um my name is Hardep Walia i'm with Charles Schwab uh where I head up AI and personalization uh my responsibilities are driving AI transformation and overseeing the develop of development of AI enabled products i came to Schwab through the acquisition of a fintech company that I founded called Motif motif was a pioneer in the application of NLP uh to novel data sets to create investment products
01:00 - 01:30 for both retail investors and institutions good afternoon i'm Tyler Durr i'm the chief technology officer at fintech company called Broadriidge and I lead uh our technology of engineering our infrastructure our cyber security or risk management and then also AI it's an area that we've established much like we do in agile squads a team that really assesses uh you know risk legal compliance in addition to product level
01:30 - 02:00 and engineering staff around the topic and much like we build out squads to make sure that we've got you know cross functional teams that are really assessing it because it's moving quickly and making sure that you've got the right uh leadership around the table in addition to that um I've got a platform teams built out that you know the architecture around this is also critically important and we'll talk about that again today but uh great to be here hello everyone my name is Peter Slatterie uh I'm maybe a bit of an outlier here and that I don't have a finance background so uh I work at a lab
02:00 - 02:30 called MIT Future Techch they try to understand what drives progress in artificial intelligence and computation so that involves things like trying to understand trends in algorithmic improvement in data in hardware um also trying to understand the economic implications of those improvements and changes so uh the piece that I work on is trying to understand how organizations are responding to risks from artificial intelligence so I guess I'm going to hopefully provide uh you know some high level uh thoughts on that
02:30 - 03:00 you know some sort of outside perspectives um I'm probably not as much in the details as a lot of my colleagues here but yeah excited to learn and contribute today thank you good afternoon everyone i'm Sarah Hammer and I'd like to start by thanking uh Marco and Val for inviting me here today and for the commission uh to for hosting this incredible event uh I'm executive director at the Wharton School i lead strategic initiatives for Wharton Finance working across our research centers uh I'm also the CEO of Wharton
03:00 - 03:30 Cipher Accelerator which is our emerging technology accelerator focused on AI and blockchainbased startups uh I'm academic director of the AI industry and law program at the University of Pennsylvania Law School where I where I sit on the adjunct faculty uh I co-chair the international expert consortium on AI with a number of colleagues uh spread around the world and uh I'm co-editor of the forthcoming Oxford Handbook on Gen AI i had to put a plug in for that uh
03:30 - 04:00 it'll be out digitally very soon and in fact the first chapter is published all right thank you all just excited to have such a diverse uh uh p panel uh panelists uh population here so uh before uh we begin I just wanted to make a note that this panel will focus primarily on generative AI right so when we say AI in this panel we'll be speaking about generative AI uh regular AI really encompasses a lot of
04:00 - 04:30 technologies and uh frankly speaking we would not be able to do it justice in the short time that we have here i'm beginning to think after today's discussions that even restricting the topic to generative AI I'm not sure we'll be able to do it justice but we have an amazing panel uh here so I think we're up to the task let's start with a big topic uh that is uh full automation so about a year ago uh Devon AI made waves by demonstrating that um that they can uh create an AI software engineer
04:30 - 05:00 this AI software engineer utilized a Python IDE and other tools to build and deploy an app end to end of course since then we've seen a lot of advances and similar capabilities and now as a question to the panel do you see this as a future based on current trends with the Gent workflows and if so which areas will benefit the most i'll be happy to jump in first Marco uh I am excited about Agentic AI and
05:00 - 05:30 obviously there are risk management uh and regulatory and policy considerations but uh the ability to use generative AI in an autonomous fashion to give the technology a goal and then to see it work towards that goal uh improving process efficiency uh operational uh systems and to reduce cost I think is very promising so as far as the financial sector goes I think of really two parts one is the enterprise use of
05:30 - 06:00 generative AI that would be operations compliance human resources for example uh where generative AI and agentic AI can have significant efficiency and costcutting benefits and the other is customerf facing which would be wealth management uh retail investors uh even specific uh tailored uh uh investment management programs for high netw worth
06:00 - 06:30 individuals and institutions so I do think there's a lot of opportunity there um I think there are obviously concerns about how it would be worked and uh whether some of our existing responsible AI practices governance and including a human in the loop uh how that will play into the future of the technology but I think it's very promising yeah so I'll just comment a little bit on uh some research from future tech which sort of relates to this so based on my experience based on my knowledge
06:30 - 07:00 from that research I think that it's very unlikely that we will get full automation anytime soon i think it's much more likely we'll get a lot of partial automation uh MIT Future Tech did a study where they examined I know this isn't generative AI but I think the the insights still apply they examined tasks that were exposed to vision models so that you know vision models could do at a level of precision that they could replace a human they found that only about 23% of those were actually costefficient uh at the moment that you would want to substitute an AI for a human there's a lot of last mile
07:00 - 07:30 problems there there's a lot of you know tasks like being a baker in a small town that we're a long way away from automating i imagine there's going to be similar sort of uh things going on in finance so I imagine we're a long long way away long long way away from uh full automation and probably we end up with a lot of partial automation instead marco to build on what Peter was saying I think it's helpful to frame where we are um we have a lot of companies in the financial services adopting technology at a much slower rate as the innovators
07:30 - 08:00 uh developing and selling technology and by the way the innovators are selling technology are developing at a much slower rate than the way they're marketing uh to all of their clients so I think it's important to frame where we are and I think it's fair to say as an industry all of us are in those early innings we're all experimenting we're all doing evaluations we're all trying to calculate the ROI so I think it's always dangerous to talk about the future when the present is still very
08:00 - 08:30 very early uh and so um Agentic every panel's referenced it holds a lot of promise um but I think I agree with Peter given where we are right now on some of the fundamental aspects of AI trying to get accuracy trying to get the ROA cases built uh companies will continue to experiment but I think right now most use cases that you've heard from earlier panels has a human in the loop and I think near-term uh and we can
08:30 - 09:00 always define what near-term means in the context of AI it could be a six months it could be a year um it's going to take us time to get to true automation as you defined and uh you know I think that the you know level of automation that you could talk about using with an engineering staff I was going to talk really more possibly about like client use cases and things that are you know impacting you know capital markets like clearing and settling transactions you know we see you know not necessarily an autonomous process
09:00 - 09:30 but a willing to use AI to do predictive analytics on trade fails and to look at risk management in capital markets of clearing and settling trades with counterparties that it would enhance if you will the ability to not only mitigate risk there's some cost savings reductions but that's not the primary driver it's really more about risk management and capital usage for firms and uh we see that as basically I would call it bolt-ons to a human uh providing operational support and really how can
09:30 - 10:00 we leverage AI to do more risk management um inside of operational jobs especially in the capital markets area marco could I just add on to what Tyler mentioned and touch on Hardep's point uh I am uh very optimistic as well about the use of generative AI in processes like clearing and settlement and uh as everyone here today probably knows that's one of our most uh high-risk and inefficient processes in the financial system everything from our payment
10:00 - 10:30 system to the way that we optimize the use of collateral to the operations around making that clearing and settlement take place and I think uh I agree that the technology when incrementally implemented around parts of that uh and to inform us whether working directly with humans alongside or analyzing data for us is very important to incorporate and so for financial infrastructure generally I think uh generative AI can be very helpful and I know we're going to get to
10:30 - 11:00 this but I just wanted to uh agree with a point that hard deep made about how we are in the early stages and we'll talk about this more I'm sure but companies are still thinking about value and one of the things that I have seen uh through my work at Wharton and with executives is that a lot of them struggle to understand how to measure return on investment because this is an incredibly expensive technology and the other thing is a lot of folks don't actually have access to this technology
11:00 - 11:30 whether it's because of connectivity issues or for financial reasons and so I'm sure the panel will get to that further but I think it's important for us to recognize that good thank you for that and so uh it's been mentioned that we're still in early days right so with regards to agentic workflows and agents in general so what what's kind of on your wish wish list to to really have that technology in your view like kind of get there to to the actual destination where it's like fully utilizable or like as helpful as can be
11:30 - 12:00 i'm I'm happy to take the first step um I think there's two things if you look at where Gen AI proliferation is right now in the enterprise it's it's internal facing it's really about driving employee productivity those were the first use cases um everything from customer service which is a wellstudied phenomena to developers to financial analysts um and you've started to see
12:00 - 12:30 early thinking around client-f facing scenarios and so that's just the normal evolution of Gen AI it is likely uh for Agentic to follow a similar path so I I I foresee a lot of scenarios internally um internal processes u that's where you're going to see the first adoption of it i think uh then eventually and we'll debate how far away that might be we'll start to see agents agentic
12:30 - 13:00 revolutions actually touching clients themselves you know on on the you know once again augmenting and getting better utilization and adoption uh a use case that we're seeing is really around data mapping and you know our business is really onboarding clients onto new regulatory solutions to give them the ability to digitize the experience for customers and in doing that the cool part about it is really building a better user experience so you've got transparency along with you
13:00 - 13:30 know,formational sharing but you've got the ability to get the right content in front of a retail investor so that they can make the right decisions about their holdings uh but if you look at the work that it takes to get there a lot of it from an engineering practice standpoint is really data mapping and you think about leveraging this type of tool um to really augment a human to be able to do that more quickly it's less about the speed and it's more about the innovation and the pace um and the accuracy is we talked about it you know today um
13:30 - 14:00 listening to many of the other panels you think about uh having humans in the loop you know there's some degree of flaw or error in that today and getting comfortable with the ability to measure the success and the output the productivity the air rate uh with AI is going to continue to be a challenge for us but it's it's one that we have today um and then I think of the prior panel when Morgan Stanley mentioned that you really got to get an air rate on what you're doing today you uh managing your operations or managing your workflow or
14:00 - 14:30 whatever it may be of getting a level set of what kind of an ROI business case are you building off of is incredibly important uh but putting the right governance structures in place I think things like that allow you to leverage and get utilization but I think it's all about uh getting good governance practices establishing a really good ROI so that you can measure the performance and then once you can see real wins you get people comfortable um and and I think that's important you've also got to spend as much time talking about things that didn't go so well uh this is
14:30 - 15:00 all about fast failing and making sure that you understand upfront when things aren't working as expected and you drop it and move on or you pivot um and I think those are elements that um this is going to continue to move quickly the more that we can do that as a process around governance and uh it's going to help us I think uh continue to get a utilization rate that's high but also understand the performance of it yeah so I think I see really two barriers to uh utilization so the first
15:00 - 15:30 one is sort of relates to the the previous point um around sort of last mile issues you know so it's maybe quite easy to get an LLM out of the box or gender of AI out of the box to you 90% performance 90% accuracy whatever but you know then you have some exponential increase to maybe get it up to the level of quality where you can really use it to you know substitute for a human and you need maybe the right level of scale for that to make sense uh so you know future tech have some work on this by um Martin uh Fleming and Neil Thompson in
15:30 - 16:00 in Brookings Institute if you want to read more about that but yeah it's sort of a widespread issue I think and the second one then I think is like sociotechnical barriers so we have legacy systems we have complex social systems legal structures all of these are filled and operated by humans who need to have shared understanding about like what the new technology is where it would be implemented um and that's you know just a really really big barrier to to utilization that's you know fundamentally hard to overcome uh and then in terms of solutions to those things I think one good solution is to
16:00 - 16:30 have more sandboxes and other is to have like entrepreneurship or entrepreneurship trying to like set up situations where AI first startups and organizations could kind of explore how do we do this if we're building from the ground up with uh with generative AI and other types of AI or whatever technology is relevant um and and then you know treating those as experiments and then uh once the experiment seems to you know once you're confident enough in the results maybe then you you bring it into the the main operation so I uh agree with many of the great
16:30 - 17:00 points that my colleagues have made and going back to something that Tyler mentioned about data mapping I think that's a very high potential use for generative AI not just in the financial sector uh but in other industries as well so uh by working with our accelerator at the Wharton School I've had the opportunity to meet a number of founders who are using Genai and in some cases Agentic AI to build businesses around the technology that do things like for example measuring our energy
17:00 - 17:30 usage uh using a combination of technologies our energy consumption data is protected by privacy laws and so by digitizing and credentializing our data and then using a generative AI model we can actually optimize in real time our use of energy there are other companies that are using such data mapping for things like weather prediction and I do think that's a high potential use for the technology um and a lot of things
17:30 - 18:00 developing uh in the financial sector as well uh I fully agree with what was said about responsible AI and governance uh this is something that is a an important point that uh has been raised repeatedly especially in our changing geopolitical environment uh we have seen of course the enactment of the EU AI act uh with its different tiered system for measuring risk within use cases of AI systems uh and other frameworks developing around the world and then
18:00 - 18:30 here in the US obviously uh we have a number of laws that are relevant already and we're in this assessment period where we think about the technology uh but I have not seen companies pulling back on responsible AI initiatives so that comprises everything from developing their principles around the use of AI for the whole institution and possibly hundreds of potential use cases putting the governance policies in place using a governance committee for example
18:30 - 19:00 that integrates all of the arms of the firm uh and then using things like risk management and auditing to continuously implement and improve that governance policy i think one of the challenges going forward is if there's non-compliance with a governance policy by a firm how do we address that because there are so many ways and incentives in fact that those governance policies can be circumvented uh but I'm optimistic that with a continued emphasis on that
19:00 - 19:30 uh that it will help us move forward in a positive way with the technology good thank you um you actually touched upon something I wanted to ask this panel about right this agentic workflow is that really do we need to reshape our thinking around risk management frameworks when it comes to these technologies right so what's working currently and what do we need to consider reconsider basically as a part of this like kind of like evolution in technologies you know on uh on risk you know the discussion that
19:30 - 20:00 we've had a lot in use case development is not just having the use case in terms of its performance return on investment looking at uh cost benefits but also the risk posture that we have in an organization like ours we're a service provider and so we we've got client data that we need to be careful with in addition to that um working with our clients you know and and their risk policies which may be different than our
20:00 - 20:30 own you know um and ensuring that you've got good risk tolerance is in place uh but I think having those discussions every use case I think has a different risk posture i think establishing what risk frameworks you have today I heard it earlier and augmenting those it's something that's going to need to continue to change and that's the only thing I'd encourage is that it's not a static process that you can lay it down and then not look at it again you're going to have to continue to evaluate it as new use cases come up uh you're comfort level with doing it and in our
20:30 - 21:00 case uh also as a service bureau ensuring that our clients are comfortable with it um and that's going to continue to be an area that is going to be a focus uh not only for our organization but from regulatory standpoint and that ones that you can demonstrate how you're doing it it's one thing to say it but then you've got to demonstrate it and I think that's much like any other FFIC or audit or regulatory body that we interface with you know you've got to have evidence
21:00 - 21:30 that you've got a riskmanagement posture in place and how are you how are you evolving it um and I think that's going to continue to be a key area but you know once again I think building on what has already been established but evolving it is incred critically important yeah so I mean in response to that question around whether we need to rethink uh you know risk management for agents I mean I think we fundamentally do so at MIT I lead the uh MIT AI air risk repository which is the the most comprehensive database of risks from
21:30 - 22:00 artificial intelligence we update it about every 3 months um and in the most recent update we've added several hundred risks uh related to to agents and added a whole category of of multi-agent risks and I think it's just you know we we've covered some of these things earlier like the providence if we have agents who have a kind of inherited agency you know who's responsible for the things that they do so there's like liability issues there's providence issues there there's like collusion issues when the agents work together if my agent works with your agent there's something that neither of us want are we both responsible which agent was
22:00 - 22:30 responsible there's also in some meta sense like this higher level complexity challenge of like we struggle now to understand what's going on these blackbox models and if we have like millions of instances of those models collaborating across different tech stacks you know the problem is at some point you know people will be saying things are going well I'm just going to stop paying attention to this and then if things go badly people will have no idea maybe why they went badly so I yes I think there's fundamentally new risks there and a lot
22:30 - 23:00 we need to think about i agree with Peter there are fundamentally different risks associated with generative AI and with agentic AI and uh I think going back to some of the things that Peter mentioned uh and have been discussed in earlier panels uh these are points of issue for our policy discussions around the world are things like bias so we have bias in our current financial system around certain client-f facing applications or processes uh and
23:00 - 23:30 the issue of fairness is one that we still need to unpack and think about when we implement AI systems and certainly there are you know uh stunning examples of where generative AI has been implemented and uh put out output that is clearly biased on his face but because of the way either the data was entered into the model or was used for training data or because of the instructions that the model was given there's this perverse result and
23:30 - 24:00 deciding how we address that whether through governance or through regulation I think is still a very important issue that becomes compounded when you're dealing with either a generative AI model or an certainly with an agentic model uh Peter mentioned liability and that is certainly an issue that uh needs to be unpacked and in fact some of these issues that we're dealing with with generative AI in the financial sector are not necessarily the same regulatory
24:00 - 24:30 issues that we would have dealt with with traditional financial applications so for liability for example the question is when you have a generative model when you have downstream use of that model for example who becomes liable if something goes wrong and you know at a simple uh level one could say well is it the builder of the model is it the provider of the data is it a third party service provider who is involved maybe it's the implementation of the model or maybe it's even the
24:30 - 25:00 client and our existing frameworks around liability things like strict liability or negligence may not be relevant uh to generative AI so that's a key issue um and uh you may have seen that recently the EU uh rescended its proposed liability directive uh for a number of different reasons and there's certainly a political discourse around that but here in the United States the ALI and others are working to develop civil liability principles for AI that I think
25:00 - 25:30 will help clarify things for the industry as we move forward thank you for those responses and that wonderful discussion um just trying to kind of maybe close um kind of like the the topic here of like agentic workflows etc um in your opinion um and so what do you think are limiting factors really um that that uh is hindering uh increased business use again like where do you think we need to like uh kind of improve the technology
25:30 - 26:00 or is it really just like an educational um uh kind of uh problem like Peter had mentioned where we just kind of need to educate folks in the enterprise about what what these things can do and you know what capabilities are enabled as a result of using these technologies The uh the thing I I' I'd indicate on that is you know having a well-defined data structure you know is incredibly important to feeding the prompts in any
26:00 - 26:30 kind of you know use of this type of technology and you know we've spent time looking at the architecture to ensure that you've got a well- definfined data ontology you're feeding into this you know that's one aspect that really needs to be thought through when you're applying this type of technology even before you talk about making it autonomous with agents um because that'll set success up for the users of this technology whether they're autonomous or not you know the data hygiene and data cleanliness is a critically important factor and so I
26:30 - 27:00 think that's one aspect um you know the other is you know getting really comfortable with once again you know you've got to try fail fast but I think it's been stated also that you've got ongoing monitoring of any kind of uh processes that you put in place that are running through AI that you've got the monitoring on a daily basis that can screen the models you've got to have the ability to run through what was let's say a production quality model that you said okay to the underlying u AI could
27:00 - 27:30 have changed and you've got to have the ability to do ongoing monitoring and I think that that's one that's going to help I think get confidence around the continued use of the models but you've got to have that guard rail on the end of it that you're ongoing monitoring is in place i I think um I think people don't really understand what people in their companies do and I think what you have to do is start looking at a role
27:30 - 28:00 break up the task and you're going to find they fall into three categories categories that are automatable by agents that's the so-called grunt work today there are things where you can augment the role um and then there are things that are just all about being human and I think as you look at the enterprise what people are surprised to find out Jeffrey Hinton the the godfather of AI made a famous prediction five years ago that there would be no radiologists left because machines can
28:00 - 28:30 do that uh sorry to report five years later there are more radiologists than there were five years ago but their roles have drastically changed and he made the same mistake thinking that all radiologists do is read MRIs and I think what we're learning about the first wave of AI and when you go to agents you've got to understand where we are today and what you'll see is AI productivity vastly differs in the enterprise by role
28:30 - 29:00 by seniority of the role so for example when you look at customer service there's a lot of research out there that says the impact of Genai is in less senior roles they call it experience compression and when you actually give Gen AI to more senior more seasoned customer service reps you actually see a dip in productivity why because they they know the answers they don't need a Genai to tell them what to do now what's fascinating is if you go look into another domain like developers
29:00 - 29:30 developers junior developers actually don't see that uptake in productivity because when you're using a code editor and you're generating code you don't have enough experience to know what is good or bad you go very senior developers they're now architects they've stopped coding again you don't see as much but that sweet spot for Gen AI is in the middle people who've seen a lot enough code so it's it's fascinating right when you start to break down what Gen AI is really doing right and I go
29:30 - 30:00 back to the MRI the Jeffrey Hinton example is we're gonna get it all wrong and it starts when you look at what your people do in the enterprise you have to go to the task level it's not enough to say oh customer service they talk to people on the phones they do a lot of rich things some automatable some augmentable and some things they're just agreement yeah so I would just like to agree with a lot of what Hardep said i mean one uh
30:00 - 30:30 interesting follow- on point to that a colleague of mine at MIT I think Aiden Toner Rogers he recently did some research looking at the impact of I think it was gender of AI on science and what he found in his case was that actually it was the more advanced researchers who got the most benefits because they had the really good ideas and were able to use AI and the junior researchers lacked that sort of knowledge so it is very very context specific another thing that came to mind here you know putting my sort of behavioral science hat on which was my
30:30 - 31:00 my background um I don't know if there's been enough work really to diagnose what the barriers are to adoption so to actually speak to people survey them and see you know is it that they're not aware of the potential benefits is it a capability issue is it a motivational issue and then you know to sort of tailor the the relative interventions to address those particular problems that you're having in your particular workforce yeah thank you i um I agree fully that the human element is critical and there are absolutely functions in the
31:00 - 31:30 financial sector and elsewhere that cannot be replaced by AI uh in fact we were working on an analysis where we looked at how can Genai be used in traditional jobs like an investment analyst for example or a VC analyst uh and that's a key question certainly for our students as they graduate Wharton and are thinking about the skills that are needed to succeed in the industry in those types of roles so then the
31:30 - 32:00 question becomes how do we break down those roles into different increments and identify whether or not generative AI could be useful um certainly for some things like due diligence document processing uh creating an Excel spreadsheet even writing code generative AI can increase efficiencies and then the question becomes you know how do humans play into that and how will investment analysts and others be needed uh and I think the answer really in my
32:00 - 32:30 opinion is that genai uh calls upon us to do the higher level analytical thinking around those issues and problems that are before us and there are certain things that genai can't do genai cannot look at a person across the room and try to identify how that person is feeling and then negotiate with that person geni cannot build a human relationship which is really essential in the financial sector we are certainly
32:30 - 33:00 an industry built on data uh that's highly regulated but at the end of the day how we do business is relationshipbased so um I think that's a really important point to keep in mind uh the other thing is I think there are limiting factors that are broader that we should think about one of them which I know we'll discuss is cost uh and again I think you know certainly there are institutions and individuals and countries that don't have access whether it's because of the need to invest in
33:00 - 33:30 the technology or they simply have connectivity issues uh and that is a limiting factor uh I think another thing which we may not have time to get into detail about is open source technology and there are certainly pros and cons to that it's a controversial issue uh but we've seen great advancements in open source and In fact there are some countries that have said this is the only way that we will be able to compete in the world of generative AI because they don't have the money to invest in a
33:30 - 34:00 foundation model um that some of us in the United States might be able to do so I think there are regulatory issues around that uh and there's certainly in some spots a belief that open source might even become the norm uh but I think we should keep that in mind because as Harde mentioned we're in the first inning of this game yeah if if I may yeah like request to put a pin on that uh but you know these are very good considerations for sure to discuss um I just wanted to keep the discussion moving because we were just talking
34:00 - 34:30 about hinder hindrances excuse me and adoptions uh barriers to adoption which uh kind of segus into really this notion of hallucination i actually haven't heard that word too much uh today act and and and really in general these days um and the question is really this so from the prior uh panels only I I think I've only heard it from the first panel um and uh it seems that hallucinations are here to stay um the first question
34:30 - 35:00 is to you all is do you believe that that's the case uh and a follow on question is if if you do believe that we just have to live with this right um how will we mitigate it uh in the future right so is it going to be really just us rethinking how we interact with the technology right and creating workflows to to basically uh try to circumvent or like work knowing that these will eventually happen or do you think that there will be some technological breakthrough that would help address that
35:00 - 35:30 so maybe Marco it's the uh ex entrepreneur in me i'm just too optimistic um technology will solve the problems that technology creates so this notion of we will never solve uh hallucinations that's a that's a difficult statement to get my arms around i think a couple of things to keep in mind uh the numbers on hallucination I know Peter uh threw out the 90% but you know out of the box you you get to 65 70 after a lot of work you
35:30 - 36:00 can get hallucination rates to 90 but that data and then it is exponentially difficult to go from 90 we're so used we live in a world of multiple nines we we can't even get two nines today and I think the thing to keep in mind um is those numbers are very general once you start to constrain the domain you start to limit the use cases you can get very very good uh results uh clearly not ready enough for consumerf facing I would argue uh but we're on that path
36:00 - 36:30 and again not to keep beating the same point we're early on i mean chat GPT was two and a half years ago and financial services companies typically were not leading the charge we've got a lot of thoughts to put in around risk management going in that kind of slows down adoption so it's it's tough to say we're not going to solve it when we're just getting started uh and I think we should be optimistic i think these are solvable problems and to tie it back to
36:30 - 37:00 your earlier comments around agents agent is kind of an exponential of hallucination if each step has a underlying assumption and now you're taking action for agents to take off you have to solve some of these fundamental problems because it really compounds the hallucination and when you start putting an action at the end of it so if you believe in agents being a viable thing in the near future you're going to have to tackle some of these issues and you're going to have to bring those at least in the 29s before it's ready to
37:00 - 37:30 scale it's funny because I feel like I've seen in the literature where people were trying to use agents as a solution to hallucination right by basically like saying that you know what's the majority like view on the answer to this question if you're the odd one out you're hallucinating right um and so that's a fascinating point and the other thing I'd like to say is that you know your point about it not being ready for prime time is very well taken um actually earlier this month the Colombia Journalism Review had just noted for AI search tools that uh I quote "Chat bots
37:30 - 38:00 are generally bad at declining to answer questions that they couldn't answer accurately offering incorrect or speculative answers instead." Uh and sometimes they even fabricated links uh when they're trying to site sources um other panelists uh views on on this issue of hallucinations and where it's going yeah so I mean I I agree with Hardep i think we shouldn't you know rule out that they will be solved or be solved in some meaningful sense i think that you know one of the big issues here
38:00 - 38:30 and elsewhere is we need to think about like marginal marginal impacts marginal risk you know we have hallucinations in humans how do they compare to humans is I think a really critical consideration here um you know another point is like I mean the hallucinations are tied to the current architecture of LLM models and you know the long context window pro window problem you know the fact that we use transformer architecture and so on but those were innovations that will be new innovations you know uh there's no reason to assume that we won't in the
38:30 - 39:00 future especially as we start using LLMs to and and AI to train AI develop better uh situations or solutions there um and then I wanted to say like I mean I think that what's a critical solution to this at the moment is a more well-rounded and nuanced evaluation regime for LLMs you know at the moment we get something really simple like oh performance very well on this benchmark but the benchmark is very selective you know if we think of psychometrics for humans we develop these these really well established systems for testing human intellectual
39:00 - 39:30 performance human job performance we should be doing more work to subject you know our our generative AI our LLMs to these sort of regimes um and I know that like uh meter who who hopefully some of you know of have been doing some good work here to try and understand what uh agents can do in real world scenarios in terms of coding tasks epoch have developed uh some new benchmarks so I think you know I'd really like to see more work there so we can understand uh you know the extent to which for example models can retain information and do
39:30 - 40:00 longerterm tasks and so on uh over time uh thank you uh I'll start with an example which several of you in this room may know which is the example of the gentleman who used Chad GPT to write his legal brief and uh this is fairly well known at this point but essentially he wrote his entire uh submission to the court using chat GBT and upon analysis
40:00 - 40:30 of that document it became clear that every case that was cited in the document did not exist so uh certainly there was a penalty for that ultimately but uh it brings to bear the point that if you are using generative AI for certain high-risk uh applications like submitting information to the courts uh you certainly need to be aware of the issue of hallucinations and uh I think
40:30 - 41:00 it it brings in mind two things so one is the training of how to use generative AI and we do still have hallucinations uh even for those who use deepseek and are big fans I will say it does hallucinate uh in my opinion the use of generative AI for legal purposes uh you will certainly find inaccuracies and hallucinations uh and there's a multi-step process involved with legal analysis that I think sometimes the
41:00 - 41:30 models can't necessarily comprehend at this point so training people to understand the technology and to recognize the hallucination and to learn how to deal with it I think is extremely important and then on the other side I think there is simply being an expert in your field so if you were a novice lawyer and you went to deep research and you used it to help you um identify information that could be useful for
41:30 - 42:00 oral argument in court you may not recognize that some of what it is giving you is a hallucination or is not accurate i think the other thing that can happen is it's very easy for the models at this point to see some information and interpret it incorrectly uh in a way that an industry professional might interpret it so uh I'm hopeful that technology will meet that issue but it remains an issue and I think it's one to to consider uh
42:00 - 42:30 focusing on you know my only other add to that is uh I believe much like Hardep indicated that technology is going to end up you know complementing and solving this i view it much like you would a product development life cycle or software development like life cycle you know building test harnesses for this type of new technology is going to continue to emerge it's going to need to be something that I think suppliers and new entrance uh from software capabilities that are startups are going to continue to innovate i think the AI models
42:30 - 43:00 themselves could help and could augment but thinking about this like a test harness we're going to need to do that as an industry one to get comfortable but then you've got to have a demonstrated process that you can evidence like any other deployment of software and so I agree with you it's going to be something that technology is going to solve it's just this is moving very quickly wonderful um I'd like to move on um to to something that was actually
43:00 - 43:30 previewed by Sarah um so we were just talking about technical obstacles let's let's move to something less technical uh and discuss costs associated with these services um so we all know I think as previewed by the earlier panels that training a model from scratch is is prohibitively expensive it's funny because I've been in this field long enough that um the the eyepopping uh sound bite before was that training a model has the same uh carbon footprint as a transatlantic flight and now that's like so obsolete right it's many times
43:30 - 44:00 that um and it consumes so many more resources than that um and so that's that's really like you know out of consideration for for I would say a majority of of folks um AI via cloud providers right um so surprisingly um so real scaled out use cases right using cloud providers could actually get pretty pricey as well um for realistic problems at scale right so consider like creating a rag system where you have to
44:00 - 44:30 then uh create like a vector database of some corpus or corpora that you have inhouse that cost money um and then right the the rag system which would make multiple calls to like an AI like an LLM them that also costs money per call and so now you have something that's actually quite expensive and not to mention right the cost to store the data right in the cloud as well um and it's funny because in the first panel Hillary Allen mentioned that the prices might actually increase like you know like after some time um and the last thing I wanted to uh mention about kind
44:30 - 45:00 of pricing uh is that uh open AAI uh had been rumored to be planning several uh specialized AI agent products right uh so now they would have something that's like a PhD HD level AI agent that would cost something like $20,000 a month a highinccome knowledge worker would cost you about $2,000 a month and a software developer agent would cost you about $10,000 a month so my question uh given all of this information is how will this pricing stratification play out in
45:00 - 45:30 industry right how does this change the ROI calculus that you're all thinking about and would this really exacerbate the differences between the halves and have nots um I I I think the calculus on cost um is a great question it's changing quite a bit i I don't think I don't see the problem as um I think pre January 20th we would all have probably similar views i think post January 20th which was when
45:30 - 46:00 DeepSc R1 launched I think the open-source movement is putting a a throttle on some of these cost issues um I don't see it as a have versus have nots i see it more uh as a can versus cannot right i think it's going to boil down to skill and even the open AI examples that you cite I mean I think what they're counting on is a provable ROI so it's still cheaper than an incremental human person so from a value
46:00 - 46:30 equation if they're right and it is autonomous it still might be cheap and I know we don't we hear those numbers and we go "Oh my god it's so expensive." Um so I think I think it's it's all going to be relative and I think the uh uh these costs to date have been coming down now I think the question earlier this morning is is it sustainable uh but you go look at the last three years everything's going down right now and I think companies especially companies
46:30 - 47:00 that don't have um other businesses like a Google can cross-ell uh uh uh their search to to invest in in some of these foundation models i think the other thing Marco to point out most people in this room are not going to be developing foundation models that are you know hundred million to a billion dollars i think it's all going to be about inference and how people use them and develop them so I I think the juryy's still out i think uh uh I I do love a
47:00 - 47:30 lot of the open-source movement the the Deep Seeks of the world and and you know their companies in Europe as well uh because I think that puts a natural pressure on what some of these commercial models are able to charge yeah the thing that I I'd say just a bit on maybe a different topic but related to cost is really the ability to attract talent that can help build these kind of capabilities and I think that's what's going to be getting incredibly competitive over the next two to three
47:30 - 48:00 years and that you're going to see most of the cost increases are going to be the talent that you need to acquire to compete actively in this type of a market if your business requires it um I think that's one you know the the other is ensuring that you can retain that talent you know over time which is going to be a challenge for any institution that has brought in um you know newer talent to the organization and then the last point that I'd make is if you think about uh 5 years from now you know uh
48:00 - 48:30 any college graduate that comes out is going to want to use and leverage this type of technology if they're in the tech space you know they're going to expect working for organizations that have access to these types of capabilities and I I think that that's going to be one that you know attracting training having educational programs for top talent coming out is uh those are the factors that I'm probably more concerned about than the cost of the models themselves i agree with Hardep i think you're going to see the the floor drop out of these over time it's going
48:30 - 49:00 to have to anything in our in our technology space if you look at the last decade compute storage the the floor drops out with capabilities uh being added and I think that's going to continue and to Sarah's point on open source being I think a real interesting viable threat to some of these commercial models that's only going to help accelerate that uh but I'm really more concerned I think as an industry about attracting talent and Tyler just u just to add to your last point I mean uh having an older child going to college
49:00 - 49:30 this fall there is a lot of anxiety around this because um it used to be go get a computer science major from Stanford and you're all set i think in this new world I think talent at the the very entry level are rethinking careers rethinking how they are uh because of the commoditization potentially of certain roles uh but there are also new I mean I mean it's it's there's a renewed push towards humanities computational linguistics to to
49:30 - 50:00 understand so this I I completely agree with your point about talent and it isn't just talent cost it's talent mobility and you're seeing it now in treelevel jobs where look at most institutions we we we were talking about Michigan I mean I think there there there's so many roles and majors that are going to change at entry level and that's going to cascade itself across Yeah um if I just may add though I I think what you know I very much appreciate you know the the enthusiasm around open source right but both of you
50:00 - 50:30 were I I feel like kind of speaking to to different components because in order to leverage that technology you need technical talent to know how to use it and install it right and so without that you're kind of stuck um right and given the scarcity of the talent and also you need the hardware to to run the models on right for inference right and those don't come cheap or you would have to turn to a cloud provider too and so I I guess there's just like you know speaking of ROI and the calculus right there's just a lot of these considerations because you're managing
50:30 - 51:00 all of these different resources and trying to figure out the the best outcome given what is available to you yeah so I mean I I agree with a lot of what was said i mean I do think prices will come down i want to kind of hone in on a particular aspect here which is the sort of risk of centralization of power and increased inequality so you know I imagine most people here know about the Medallion Fund so an eyepopping statistic about the Medallion Fund which has dramatically outperformed the market for many many years if you invested $100
51:00 - 51:30 in 1988 it would be worth 400 million in 2018 that's an incredible amount of capital they managed to generate now the thing is how does the Medallion Fund function you know it finds really talented people there's a whole bunch of frictions sociotechnical issues and training up those people getting them up to speed that was the old world in the new world maybe the Medallion Fund you know I'm speculating a little bit here but if they're able to develop agents that can execute the kind of strategies that they do if I knew more about what they do I'd be a lot wealthier you know if they can figure that out they can
51:30 - 52:00 have those agents doing those things executing the things they can just scale out as they achieve more capital they can keep on scaling so I think the returns to capital are going to get fundamentally different if agents take off and the agents will never die they're infinitely replicable if you have um you know the capital available so I mean I do think that you know that there will be potentially uh massive issues in terms of concentration of power price strat stratification access stratification and uh and maybe inequality as a result
52:00 - 52:30 so I'm very glad we brought up the issue of talent because we do have a lot of talent at the Wharton School uh and you know certainly this is one of the questions of our time is what uh should we be giving our young people to prepare them for the future world of digitization uh I think that uh I'll start with the digital divide which Peter talked about um certainly there is a digital divide and whether it's AI or
52:30 - 53:00 whether it's connectivity uh through broadband or access to satellite or mere resources that a country may or may not have to access this type of technology it is incumbent upon us to be cognizant of that divide i think um from a policy perspective uh and from an economic perspective as well um I do think that open source uh and I know we're putting a pin in open source but I think uh that is certainly an issue that has the potential to bring down the cost of AI
53:00 - 53:30 uh but there are obviously policy considerations around it uh the you know certainly the ability to jailbreak or access to information that may be available on open source um and I think the other thing about cost is thinking about how organizations will change over time generative AI is different from other technologies i believe it's a transformative technology and so today we are talking about implementation based on the way that our financial
53:30 - 54:00 institutions look today uh but as I mentioned earlier one of the things that we face when we work with executives is this question of how do we build an AI governance riskmanagement auditing policy uh that works for the organization and the individuals within that organization such as they are incentivized to follow those policies and part of the big question is is our organization designed in a way to
54:00 - 54:30 effectively implement AI or should it be designed in a way where AI is really at the center of the organization and that's a cost question uh it's a structural question it's a societal question uh and I think it's one for us uh really to think uh but we certainly have you know time as we're evolving along this chain of generative AI cool wonderful so moving on a little bit so now we're talking about how transformative uh
54:30 - 55:00 generative AI is so as a sidebar Sarah what I appreciate about your comments is that you're really helping me with my segways here so we're talking about transformative technologies uh and that is generative AI right and so what I appreciate about it personally is that it it reduces the barrier to entry right to access AI insights right so before to get sentiment of a span of text right typically you needed to train a model which required knowledge right uh from computational linguists machine learning
55:00 - 55:30 experts etc they would train this model and right then you would use it to try to predict the sentiment of a text now you just go to chat GPT and say hey does this sound happy or sad to you here's the text um but this uh this reduced barrier to entry I think can act as a double-edged sword Right and so I wanted to invite you all to uh help examine the consequences of this reduced barrier to entry across three dimensions u the first dimension
55:30 - 56:00 and and we could talk about the others later is really uh around staff and talent which we' actually kind of uh kind of spoken about earlier right so what does this mean for um advanced engineers low-level investment professionals and you know like academics and PhDs now looking at a you know a job market which may not need them anymore right do you have thoughts on that um so I I I think we um touched on
56:00 - 56:30 this earlier i think there are definitely roles um that are inherently very human and when we talk about commoditization of certain aspects of roles um and you know if you are a junior level analyst and looking at sentiment going in I think there's an opportunity you now to contribute in a different way to society and any innovation we can go back to to the industrial revolution requires um and
56:30 - 57:00 Sarah touched on this a lot of retraining and rethinking but I personally can't help but appreciate that this technology is actually good for society i think it's going to force some short-term pain as we rethink uh thankfully it's happening a lot slower than the that some of the vendors might convince you of uh but it is a serious question um and then you know all of these roles you could take every
57:00 - 57:30 function in your enterprise today there are still things so far that require a human aspect and if it really is commoditization there are better things for a human to do and I would argue we have to be thoughtful we have to be careful on the transition these transitions are never easy um but I I can't say that opening up a lot of this technology to give insights is really about redistribution of your talent and where you want them to work in the near term and then the long term well we've
57:30 - 58:00 got time to go figure that out i'll uh I'll play off of that that uh I think it's also around you know taking your roles and the role profiles you have and let's just use an engineer as an example at least we've seen in our business that the people that really deeply understand the business the subject matter experts that really understand the process they understand how the business operates they're the most valuable to the organization today and they have been these additive tools
58:00 - 58:30 are allowing them to spend more time on that you know in our case you know spending more time with clients because they're able to leverage some of the work they did that's more task oriented to your point um and that they can do the higher value activities but you know this is going to be continue to be a journey i think it's about trying to ensure at least in in our area that we've seen um it's going to require I think more focus of training our engineering staff on the business and having them become more experts on the business because that'll allow them to really leverage the tool capabilities of
58:30 - 59:00 things like AI so that they become better engineers um so I think we view that as a a real pro i think if you look at the threats and Hardy mentioned it you know for new grads coming out they're rethinking whether or not they want to be in computer science and you know for us that's a systemic threat I think to the industry is that we need to continue to graduate people that can come in and fuel the growth that we need to continue to you know lead in technology in the in the fintech markets yeah so I mean I think a lot I agree
59:00 - 59:30 with a lot of what has been said so far i think that probably some of the key things we need to do are we need to have you know there's probably a whole class similar to how you know we had web designers and and bank tellers now do different things there's going to be all of these different roles these sort of meta level oversight roles or roles we haven't even considered we can try to get foresight for those roles we can start to see you know where are we starting to see more substitution uh of human labor for AI where are we starting
59:30 - 60:00 to see sort of the emergence of new types of roles that uh we didn't previously sort of consider as roles we didn't advertise as roles we didn't have you know education for um so I think that's something we can work on i think we need to watch out for things like you know some of the again putting the the risk hat on which I'm wearing a lot of the time you know we can start to kind of lose our ability there was a study that came out recently about um I think two studies and I don't remember the exact details but the broad summary was
60:00 - 60:30 that when uh when you use AI um I think it was people who use AI more demonstrated weaker crit critical thinking skills and there's poor retention of information if you've used AI to generate it so if we start relying more and more of these things perhaps over future generations we have less and less expertise and we have less and less ability to do things without the eye and we become more and more sort of dependent on it and and disempowered now that's maybe a long-term creeping risk
60:30 - 61:00 but but I do think it is something to to keep in mind so I um I support the comments that have been made here today and I mentioned earlier uh some work that uh we have done to look at investment analyst roles uh and in fact I wrote about this in market watch in my opinion series and I think there are tasks that generative AI can certainly automate uh some tasks like processing documents working with
61:00 - 61:30 due diligence even creating pitch decks for example those are known known to be very long timeconsuming arduous tasks and then there are certain things that are uniquely human like building and maintaining relationships and the art of negotiation for example and there are technologists who would argue that generative AI or AGI can certainly fill those tasks uh but I'm optimistic that humans will continue to do the higher
61:30 - 62:00 level thinking and with that industry expertise that they obtain over time continue to be invaluable i think that AI also offers the opportunity to provide the kinds of education that we have talked about here today uh and we haven't gone into too deeply on it but certainly generative AI has the ability and is being used to personalize education in a way that we have not been able to do in the past so for example if
62:00 - 62:30 you're taking a class and you're using generative AI or a generative AI provider as you proceed with that the model can say this person actually already knows a lot about this uh I'm going to make the challenges harder for this person i'll give more quizzes i'll move us up to a different level and I know that's a very broad statement but I do think it's a potentially very important use of the technology and I guess what I would also say is that I don't think technology will wait for us on this question i think that the
62:30 - 63:00 builders will continue to build and I think it's incumbent on us to be very proactive and unafraid to think about these issues and to think about how we can provide productive solutions that will help keep us at the forefront of what needs to happen so that we're competitive in this environment thank you um I I wanted to circle back on something that Peter had really mentioned this idea of over reliance and really acrewing technical debt so the
63:00 - 63:30 other panelists that's actually one of the other dimensions I wanted to explore as a result of this reduced barrier to entry so Hardep or Tyler do you have points of views about you know this potential for right the over reliance on these technologies and how it might play out in organizations i'll uh I'll touch on the technical debt side of it you know the thing that we've seen with things like using code assistance to take you know I would call it older languages and modernize them
63:30 - 64:00 that we've seen you know real benefits for being able to eliminate what you would classify as longtale technical debt and so I think it's it's got real benefit there um that being said you know I can't stress enough that you know thinking about our engineering staff and the work that we do having them really understand the business is critically important and the reliance upon capabilities like this um you know allow people to probably take what you would classify as shortcuts you know they're able to get a lot more up to speed a lot
64:00 - 64:30 more quickly um the thing that I'd say is that measuring the value and measuring u efficiency out of that is I think critically important so that you can ensure that you're getting the right level of productivity you know you're getting the right level of preciseness and answers um that doesn't change whether it was a human involved or AI involved and I think you know making sure you've got the right metrics around how you're managing both human and non-human agents in in this regard I think is going to be important for our industry in addition
64:30 - 65:00 to that um making sure that ongoing I would call it test out development so that you've got people that can quiz out what do they understand what do they know is going to be uh an important element of this especially for the key areas you want them to continue to learn ai is just to me an enabler and um you know it's really around how you can ensure that people continue to understand the business that you're in they understand the critical points of your risk posture and I think risk posture training is going to become a lot more important you know in addition to technical debt
65:00 - 65:30 yeah and and to add to Tyler's thing I mean I think the another piece that we haven't covered so far is the data related to these functions so it's not just the tech debt are you data ready for AI and I think there's been a lot of modernization a lot of improvements on where do we get our data how do we track data so I think all of these uh and we have the similar discussion when we're talking about you know the development of cloud right 10 years ago um so I
65:30 - 66:00 think these are opportunities obviously it leads to in the enterprise a lot of rep prioritization of where your spend goes um but I think this relates to the earlier point you made Marco around what is the ROI because a lot of these it's we're we're connecting the dots where you choose to invest is a function of where you're getting the ROI and and these are dynamic systems especially with all the changes that we're seeing and I think the the the only thing that's consistent is um things are
66:00 - 66:30 moving at at a pace we've never seen before and so even when you're talking risk they've got to be dynamic at some level and the enterprise has to really develop processes and and for scale that's hard to do and so I think it's uh forcing some good conversation some good discussions around what do we need to do to be AI ready uh and that's a that's a hard thing to answer all I say again is how much I appreciate um you all helping
66:30 - 67:00 me kind of segue to the other dimension really that I was mentioning and you both addressed it already is like risk management right i mean again the um reduced barrier to entry also could carry some risks and so we also have to be uh cognizant of that right so imagine folks not understanding what the codegenerating LLM might do and just like blindly copying and pasting things right uh it works as intended it passed the unit test I don't understand what it did right um and so I appreciate both of
67:00 - 67:30 you addressing that I wanted to ask maybe Peter or Sarah if they had comments along those lines sorry we address uh trying to address risk management as a function of really easier access to generative AI outputs um and so like do we need to change our like posture around like risk management as a result of access to these technologies yeah I mean I think so um I
67:30 - 68:00 think that probably some of the things that come to mind to me there are we need more um this idea of like uh and this kind of loops back to some of the earlier things we discussion discussed in the panel like creating a sort of shared semantic tree sort of sort of shared understanding is a big theme in a lot of my thought recently so like how can we get everyone on the same page about what AI is about what specific strands of AI are what specific use cases of AI are and what the particular risks are um is
68:00 - 68:30 important I think then a thing that's come up uh in some of my conversations elsewhere is like that we have probably a need for a lot more oversight of AI a lot of people to be involved maybe in red teaming in you know trying to test and break these systems so they can understand you know how they can go wrong also uh so there's an organization called civai which does demos of um of of how you know AI could be misused we probably need to help inoculate people against some of these risks you know it's probably not crazy
68:30 - 69:00 to have some people come into your org and say I'm going to try and you know get some people to do some bad stuff with AI so that like they can kind of we can see you know how sort of future proof your org is and and let people sort of understand like here's what you know fishing looks like here's like what you know somebody trying to jailbreak the model looks like or a prompt injection attack looks like so I think there's a lot of work to be done there probably and and yeah like a change from a lot of the business as usual processes that have existed okay uh I'm going to rely on on Peter to segue to the last
69:00 - 69:30 question but Sarah I'll give you first crack at it um simply because it you you kind of hinted at it right so what can or should we uh regulators industry and academia like collaborate on and so this is a different spin on what people have been asking because I'm thinking about an active collaboration not you tell us what you think we we need to be doing but like how can we actually work together on some of these issues yeah thank you for that question i think um there are so many things that we need
69:30 - 70:00 to work together on and certainly in our current geopolitical environment there are some challenges around that uh but I'm optimistic that a lot of the work that we have done on uh AI standard setting uh and governance and regulation globally uh will continue to move forward in a productive way that we are in increasing innovation and supporting it um one of the key issues I mentioned earlier is liability and that is the subject of a lot of conversations
70:00 - 70:30 including those that will take place in Geneva at AI for good in July i think the larger issue again is responsible AI and governance in this environment and it goes to what Peter said about risk management um risk management being really a continuous process and a piece of that responsible AI puzzle so having uh your principles established what's important to you things related to bias and privacy and data architecture which Hardeep mentioned which is absolutely
70:30 - 71:00 essential from a riskmanagement perspective and then implementing not only in your governance policy but in your ongoing risk management practices throughout the organization and then having some sort of an auditing program in place I think is essential um I think there will continue to be great work on this we do already have standards from the OECD for certain uh entities and countries uh there's been work done certainly at AI for good the ITU works with ISO for standard setting and
71:00 - 71:30 technical standard setting the work done at NIST for example and then our legal organizations like the American Law Institute or specifically for the financial sector uh the Breton Woods committee I co-chair their working group on AI so I think we will continue to work together both on the technical side and on the policy principle and regulation side uh and I think we have a great opportunity but it certainly will take work and time
71:30 - 72:00 um I think uh I remember doing a panel like this uh over a decade ago and I think the topic was social media and we were we were asked by what should the regulators do uh and at the time uh no none of the regulators were using social media so it was very hard to have the discussion I I I think this time u as we were talking about what's great is both the SEC and FINRA are using these technologies and I think that's a step first step is let's make sure we're
72:00 - 72:30 using these technologies together you have productivity issues yourself I'm sure that you want to go and invest in so I think there's an opportunity for us to deploy couldn't do that with social media but we can work together on the implications of generative AI on corporate processes talent all of the things we we share a lot in common and then I think Marco especially and I want to thank you and the SEC for creating forums like this
72:30 - 73:00 and and I think it's this notion of going through this transformative change together having a dialogue making sure we're listening to each other as practitioners not just as a regulator but as co-practitioners of a lot of this technology and I think it just gives us a better sense of what's happening in the industry uh you have a great purview to lots of different firms as you see new and novel use cases i think there's a collective that we're going to need to
73:00 - 73:30 start thinking about and I think it's it's already starting to happen with forums like this so thank you again for for including us wonderful thank you Tyler peter will give you the last word uh well Marco I once again I can only say thanks for getting this form together i I think one of the things I'd indicate is you know much like cyber security this is a team sport i think us working all together is going to increase um our chances of success because we all learn from one another
73:30 - 74:00 this is emerging we've got to understand best practices and learning from one another really helps uh the last thing that I would uh indicate is that product innovation in the financial markets is critically important you know for our long-term staying power as a leader in in this in the United States that you know our continued innovation here needs to be fueled by technology like this and so it's great that we're having these kind of discussions so we can make sure that we're all on the same page we're collectively working together but once again I can say that we're learning
74:00 - 74:30 together so thank you yeah thanks i'd like to follow up on that say yeah I do also really appreciate you know this forum uh being organized the opportunity to speak here i think this is a really good example of the kind of collaboration we need um in general I'm really positive about the idea of collaboration between industry regulators and academia academia has certain incentives around doing things quite slowly quite carefully from quite a high level um but it often lacks maybe access to the data and the sort of understanding of what's really needed
74:30 - 75:00 which can you know be provided by industry and regulators and regulators can you know sort of bring um bring value from the kind of insights from academia so in terms of some of the things that I think could be valuable here I think building out knowledge infrastructure and foresight which is something we're trying to do with the MIT AI risk uh repository and the index which is sort of the follow-up project i think that's really valuable i think that potentially also um meta knowledge which is you know this we've got a lot into like the definitional issues and realistically the definitional issues I
75:00 - 75:30 think are most likely to be solved or at least explored in detail by academics you know who can sort of say well here are all the different definitions here are all the different frameworks here are like the ways that they've broken things down so I think I'm excited about that um and then yeah the big data stuff like some of the work now that future tech is starting to explore is you know future of work projecting as I alluded to earlier you know which sort of roles are more likely to be substituted or not which sort of jobs are coming online and that's really like a high level kind of initiative which you know we can pull from a lot of different data sources but
75:30 - 76:00 you know groups like us are sort of uniquely uh placed to lead and then it's of high value to to all of the sort of individuals involved great um please join me in giving a round of applause for a wonderful panel uh and I'm going to turn it back uh now to the acting director of uh the division of economic and risk analysis Rob Fischer for the closing remarks we'll make this very quick but I just
76:00 - 76:30 first want to thank our amazing panelists for the powerful insights for me this was a transformative day so it's really special thank you um thank you to everyone who joined us today and for everybody listening on the webcast uh I want to remind everybody that if you'd like to submit comments you can do so on our events page on the SEC website sec.gov um I have a list of divisions I want to thank here real quickly but
76:30 - 77:00 first the organizing energy behind this was Jill Henderson and Hayne Kim and they're sitting right back there so if you give a big round of applause for them they were awesome in addition I want to thank the division of economic and risk analysis staff the staff of the division of examinations division of trading and markets strategic hub for innovation and financial technology uh office of public affairs office of the secretary office of the chief operating officer and office of the general counsel this was a
77:00 - 77:30 team effort and it really turned out in my view very spectacular and thanks to the panelists once again thank you all [Applause]