NatWest on Gen AI with Mark Worden

Estimated read time: 1:20

Summary

In an insightful discussion on VUX World, Mark Worden from NatWest delves into how generative AI is revolutionizing their customer interaction through Kora, NatWest's chatbot. They have successfully utilized AI to improve efficiency and customer service, reducing conversation handoffs and improving response accuracy to over 99%. The implementation of retrieval augmented generation (RAG) has optimized their processes, with generative AI consenting to much of the content management and response generation, thus significantly decreasing manual workload on design teams. Meanwhile, AI is also assisting back-end processes by optimizing conversations passed to human agents, enhancing productivity significantly.

Highlights

Generative AI has revolutionized interaction at NatWest through the Kora chatbot by enhancing conversational accuracy to over 99%. 🤖
By employing retrieval augmented generation, NatWest significantly optimized the management and accuracy of customer queries. 🔄
The deployment allows AI to manage content and response generation, reducing the manual workload of design teams. 📉
AI plays a crucial role in optimizing conversations for human agents, thereby boosting productivity. 💪
The careful and strategic implementation of generative AI at NatWest has improved customer and colleague experiences significantly. 😊

Key Takeaways

NatWest's chatbot, Kora, utilizes AI to enhance customer service with a 99% accuracy rate. 🤖
Integration of retrieval augmented generation has significantly optimized handling of customer queries. 🔄
AI reduces workload on design teams, streamlining processes and improving efficiency. ⚙️
Back-end productivity is enhanced by AI's smart summarization of customer conversations for human agents. 💪
Through careful implementation, generative AI improves both customer and colleague experience at NatWest. 😊

Overview

NatWest has been at the forefront of integrating generative AI into their customer interaction platforms. Through Kora, their innovative chatbot, they have managed to digitize a vast array of customer queries with astounding accuracy. This initiative not only streamlines the customer service experience but also alleviates workload from design teams, creating a more efficient and effective service model.

Generative AI's role in NatWest's operation pivots on retrieval augmented generation, a system that heightens the accuracy and efficiency of responses to customer inquiries. This enables Kora to offer tailored, precise responses, minimizing the need for human intervention and reducing conversation handoffs while maintaining a 99% accuracy rate.

Beyond customer-facing benefits, NatWest leverages AI to enhance internal processes as well. By automating the summarization of customer-agent interactions, they have managed to drastically reduce the time agents spend understanding customer needs before responding, thereby saving valuable time and significantly increasing productivity.

Chapters

00:00 - 00:30: Introduction and Overview This chapter introduces the struggles and improvements in containment rates for specific areas, such as Kora, which previously had low containment rates of 30-35%. However, recent advancements have increased containment to around 75%. The technology involved has significantly improved, boasting an accuracy rate of 99%, and is believed to enhance productivity substantially.
00:30 - 03:00: Exploration of Gen AI and LLMs The chapter discusses the potential of Gen AI and large language models (LLMs) in improving automated conversation handling. It highlights the capability of AI systems such as Kora and Kora Plus to manage around 40,000 conversations weekly without human intervention. The chapter also explores the integration of LLMs to enhance the support provided to human colleagues, mentioning that about 50,000 conversations are passed to human agents for further handling. Additionally, it touches upon the use of LLMs to pre-summarize conversations into three key points, enhancing efficiency and support for customer service operations.
03:00 - 07:00: Kora Team and Development The chapter discusses the efficiency improvements brought by Kora in reducing both agent and customer time through accurate chat transcript summaries.
07:00 - 11:00: Kora's Evolvement During COVID The chapter discusses the effectiveness and measurement of AI management within organizations, featuring a conversation with Mark Weren of Nat West Banking Group.
11:00 - 15:00: Implementation of Generative AI The chapter begins with a light-hearted conversation between the host and their guest, who is appearing for the second time on the podcast. They reminisce about a previous event in Edinburgh where the guest received a special cap for speaking. The host mentions that additional rewards, such as a hoodie, might be considered for guests appearing three times on the show. The tone is casual and friendly, setting the stage for a discussion on the main topic of implementing generative AI.
15:00 - 20:00: Testing and Improvement The chapter begins with an introduction, mentioning that the audience may have varying levels of familiarity with West and Kora, particularly those from regions like the USA. The speaker has been working at West for 11 years and is discussing their role at Nat West and Kora.
20:00 - 25:00: Managing Content and Scalability The chapter titled 'Managing Content and Scalability' introduces a narrative about an individual who spent four years working as a cashier in branch banking. They transitioned to the colleague capability team, where they were involved with an FAQ system designed for internal support. The FAQ system assisted colleagues in handling customer inquiries, such as how to cancel a card, by searching the FAQ to navigate various operational systems used within the bank. The chapter highlights issues of managing information and the scalability challenges of maintaining such a system.
25:00 - 30:00: Utilizing AI for Customer Support The chapter focuses on the development and implementation of AI solutions in customer support. It highlights the creation of a chatbot named Kora around 2016-2017, intended to enable customers to inquire in natural language instead of navigating through complex FAQ pages. The transition from traditional methods to AI-driven solutions in banking environments such as telefan, web chat, and physical branches is discussed as a significant evolution in customer service technology.
30:00 - 35:00: Feedback and Future Prospects Leadership and experience within the Kora team, focusing on AI and technology.
35:00 - 40:30: Conclusion and Summation In the 'Conclusion and Summation' chapter, the focus is on the team's efforts in advancing projects for both customers and colleagues. They have been successful in completing proof of concepts, launching pilot programs, analyzing the resulting data, and readying these projects for scalability. The chapter highlights the team's experience, particularly the speaker's involvement over the past two and a half years, reminiscing about their first encounter with the team and a colleague introducing them to new concepts.

NatWest on Gen AI with Mark Worden Transcription

00:00 - 00:30 these are journeys we struggled with for years where like I say we're at like 30 35% containment for those particular areas of Kora and to then see that go higher than our base level of containment like 75% was astonishing the accuracy rate of those answers was pretty good yeah so our accuracy rate at the minute is 99% uh slightly higher than that actually this technology is is far better placed to enable you to get far more kind of productivity out of
00:30 - 01:00 everybody basically around 40,000 conversations a week could be handled by this um as opposed to then having actually conversation designers having to manually create those journeys you'd also been uh experimenting with ways in which Kora can or Kora Plus can be more helpful in helping the the human colleagues and the agents that sit behind it we on average will hand off around 50,000 conversations a week to our colleagues we've looked at using large language models to pre-summarize those three key points we're seeing an
01:00 - 01:30 accuracy of around about 98% in that summary at the moment for every minute we can shave off the average time we're saving 50,000 minutes of agent time reading through chat transcripts but we're also saving 50,000 minutes of customer time waiting for that initial message from a from a colleague the following is a conversation with Mark Weren strategy and innovation lead at Nat West Banking Group for Kora what are the use cases how have they implemented retrieval augmented generation in which
01:30 - 02:00 kind of journeys how effective is it how have they gone about measuring the quality and what kind of accuracy are they getting this conversation is one for any organization that wants to manage AI at scale without further ado this is Mark Weren of Nat West Banking Group on BF World [Music] all right Mark welcome to VX World
02:00 - 02:30 welcome back thanks for having me again pleasure pleasure this is the second time one more and you get your own hat-tick vx cap i think I got one when we were in Edinburgh i think you did actually ago you did yeah that's right yeah yeah yeah cuz you spoke at one of our events that's it yeah that's that's the two that's the two occasions when those caps get given out is it when people are on the podcast three times or if they speak at one of our events yeah so nice i'll have to think of something else for your third time then don't know what that will be hoodie or something like that
02:30 - 03:00 nice so maybe maybe we can begin um for those I'm obviously I'm sure everyone knows about West lots of people know about Kora uh although there's people who will be tuning in from the USA and other places that may not be maybe quite as familiar so maybe we'll start at the beginning and we'll cover kind of like yourself your role and you know what you do at Nat West and and Kora yeah sure so yeah I've been working at West for 11 years now uh started as a
03:00 - 03:30 cashier in branch um spent about 4 years in branch banking and then moved across to our colleague capability team um at the time they had uh like an FAQ system for our colleagues so if people needed to know customer came in or spoke on the phone needed to know how to cancel a card they would then search our FAQ system to find out how to actually do that on the systems we have numerous different systems depending on where you
03:30 - 04:00 are in the bank if you're in telefan web chat or if you're in branch um and around that time is when Kora was being created around 2016 2017 and we looked at creating a colleague chatbot as well that we could use so that customers could ask more in their natural language and get the answers they needed instead of what we used to call these wizards where they would go through pages of FAQs to find the answer um so spent two years in that team and then I transitioned across into
04:00 - 04:30 the Kora team and I've I've been here since 2019 um I'm an AI product manager i've worked in the kind of optimization area of Kora the delivery area of Kora and now I work in future initiatives uh so been here for two and a half three years nearly um and our kind of goal in this area of the team is to look at emerging technology and trends in this area looking at what technology we could uh enhance Corora with what will do the
04:30 - 05:00 most for our customers and our colleagues and completing proof of concepts um pushing pilots live getting the data back and then ultimately getting it to a point where it can be scaled up for delivery so yeah me and my team have been doing that for well the team has been doing it for for a number of years but me specifically for the last kind of two and a half years which is around about the time I I remember my first meeting in this team and and one of the one of my colleagues coming up to me and going there's this thing called
05:00 - 05:30 GPT2 um we need to start looking at it so yeah that kind of gives an idea of how long we've kind of been looking at and playing around with this this kind of technology um and yeah core has been around like I say since the the idea was kind of incepted in 2016 um went live in 2017 kind of small scale and over the years has been pushed out to our mobile app our online banking platform our telefan IVR platform and we support uh
05:30 - 06:00 Netwest but also some of the other brands in Netwest groups so we support Royal Bank of Scotland um Olster Bank and Netwest International Isle of Man Bank as well so when you start to add those together you can kind of start to imagine how quickly we could have 30 different variations for one journey where a customer might be logged in might be in the mobile app might be on online banking or the IVR system and then again for each of the brands not all the brands correlate perfectly as
06:00 - 06:30 you can imagine so yeah it's um a lot of scope that Kora has kora supports around 1.2 million conversations a month on our digital platforms and around about half that on our telefr platform as well so we we speak to a lot of customers and we uh we're doing some cool stuff to try and get them to the answers as best as possible and serve them in the right way that we can brilliant that's great and
06:30 - 07:00 so the the journey that you're kind of going on is I suppose similar to to many which is that I say many actually not that many but you've had a a largely NLU based chatbot for a while um seemingly doing reasonably well otherwise it wouldn't exist and you know you're saying there that you're looking at like future facing technologies and obviously sounds as though generative AI is a big part of that but maybe before we get into the exploration with generative AI you mentioned there in terms some of the
07:00 - 07:30 volume that Kora kind of receives like what is it that has been successful with Kora so far so maybe can you just describe like the pre- Genai state what was working well that's got it to where it is and then what were the things that you were looking to try and impact by experimenting with generative AI yeah so what worked really well for us in in the kind of early years of Kora was really focusing on so we we Miles kind of who leads the
07:30 - 08:00 Kora team he kind of describes it as we have had four generations of Kora and that first generation was very much how do we take those simple question and answers that our colleagues are being asked in branch telefan web chat to take that kind of noise away from them so that they can focus on more intricate questions with the customer so that was kind of the focus to begin with and and there was a lot of success there in almost you know the easy wins of you know what is APR those kind of simple
08:00 - 08:30 question and answer things um and then we started to move into that kind of transactional kind of um iteration of Kora where we were then looking at some of the journeys that weren't digitized so one of one of the journeys um that was probably the highest rate of what we call process handoff so we had to hand off to a colleague because they have to key something on a system was change of address that was 96% process handoff like we had to give it to a colleague to to complete the request so we started to
08:30 - 09:00 move into that kind of transformational uh space where we were looking at how can we digitize some of these high process handoff journeys again to take the time away from colleagues and and speed it up for the customer as well so if you think if you're having a web chat conversation with someone about changing your address then there might be five six seven or eight turns of that conversation each of those might take four or five minutes for an agent to reply to you whereas if you could do it with Corora Corora will reply instantly
09:00 - 09:30 and you'll get through that conversation much quicker from a customer perspective you also reduce the costs of of all the different systems we have to speak to to change an address cuz as you can imagine with a bank the size of of Net West it's not the same uh system to change your address and a personal account as it is for your credit card for your mortgage so actually to bring all those together makes it a lot more streamlined from a company perspective as well then we started to move into um our telefan IVR space and we knew we could
09:30 - 10:00 start to actually do the same thing we've done in web chat but with our telefan agents so that was really the kind of huge focus there but one of the main kind of turning points for us was co as you can imagine branches closed uh telefan lines through the roof the government announcing things like payment holidays for mortgages and credit cards and loans and giving us like 24 hours 48 hours notice that that was about to happen so there I remember a lot of late nights from from teams in Kora where we were like frantically
10:00 - 10:30 pulling together those kind of like payment holiday journeys where to stop a customer having to wait for 5 hours or 6 hours on a queue to a telefan agent they could have the chat with Kora and be placed in a queue knowing full well that they could leave that conversation and we would notify them when they needed to come back um so it kind of took that stress away we we dealt with hundreds of thousands of payment holiday journeys that way and and it meant it was a lot smoother process for customers and and
10:30 - 11:00 we saw that huge uplift in volumes there was quite a peak around 2020 but what was good was it never went down again so we're clearly doing something right where customers were like okay well this this way of this channel of choice that I've chosen has worked for me so I'm going to continue to use that going forwards so interesting yeah that's a that's a that's an interesting kind of concept that because uh everyone I think felt the impact of co um and I suppose it was very similar to Dathlon i mean in
11:00 - 11:30 fact the the event that you spoke at for us in um in Edinburgh a couple of year back we had Dathlon presenting there as well and they were in a similar situation in that COVID caused them to basically close their call center and they didn't reopen it again so it's it's crazy how like how the the behavior that began in COVID has persisted till now but so what what is then so Cora is doing well uh it's answering questions it's it's starting to fulfill journeys and stuff like that what were the issues
11:30 - 12:00 that you were try or challenges that you were trying to solve through experimenting with generative AI yeah so we we've been experimenting for a little while on a number of areas that we wanted to see how could we how could we use this technology to better have customers have a a much better experience and with Kora journey so we we have around about 128 journeys within Kora and the top 30 make up 70% of Kora's volumes so we have an
12:00 - 12:30 optimization team their job is pretty much look at those 30 journeys right that's most of our volumes make sure they're optimized make sure customers can get through them as as best as they can and it means that we have this long tale of journeys so we have a hell of a lot of journeys that are only being hit 20 times a week or 40 times a week which when you're having 20x,000 conversations a week it's a very small percentage so we knew there was something we could potentially do there
12:30 - 13:00 and we also have something called low confidence journeys where Kora might be really confident on the thing the customer is talking about so I know you're talking about a mortgage but I have no idea what you want to know about the mortgage what you want to do with the mortgage and in those scenarios we present what's called a low confidence menu so we'll go okay we're confident it's a mortgage here are the top journeys for mortgages and around 46% of the time those menus result in a handoff to a colleague because the option isn't
13:00 - 13:30 in there which isn't surprising because we're giving them five options you could ask 500 different questions on a mortgage so that was one of the areas we wanted to focus on to begin with was go okay we have these four very clear low confidence menus for our borrowing products we have it for mortgages credit cards loans and overdrafts we know that it's not a good customer experience right now how could we utilize this technology to better answer those questions for customers and that's kind of that's how Kora Plus was
13:30 - 14:00 born where we were then going okay can we do something with rack to to improve this we have a whole host of FAQs a whole host of web pages which have all the content we need on here and we have a whole load of data on the kinds of questions that are being asked so we started with some experimentation we took 2,000 some web pages and put them in our knowledge base for for our retrieval augment generation and we
14:00 - 14:30 started doing some testing and we very quickly realized that only actually 500 of those articles were being hit so we cut our knowledge base down to those 500 and that began our kind of in-house pilot testing with our colleagues so we invited colleagues from all over the business to do that kind of in-house testing uh there was another team doing a similar solution for our internal HR chatbot so we got them to red test uh red team test ours we did the same for theirs and it was quite collaborative
14:30 - 15:00 with with a lot of the colleagues within that west and then we managed to get um through a governance approval process that was being created as we were going through it um we managed to get through for a 12week pilot and we done some research beforehand so there was a couple of things we wanted to make sure we did within the pilot from a risk perspective we wanted to make sure we asked customers if they were comfortable using generative AI so that we were quite upfront with customers we're going to use this to try and answer your question are you happy with that yes or
15:00 - 15:30 no if they say no they'll get the same low confidence venue they would have done if they were going to go down that route and if they say yes we'll then uh utilize um our retrieval augmented generation process to to summarize the answer from the web pages the research we've done with our human- centered design team beforehand suggested 33% of customers would say yes and 33.7% of customers said yes so it was pretty bang on the money um which is really interesting wow um but for the
15:30 - 16:00 customers that did say yes and get a generated answer we were seeing um our kind of like containment figure go from 30% with low confidence to 70 uh we were seeing um the reduction in handoffs like we were we were not handing off 78% of the time when we actually generated a response now we weren't generating a response a lot of the time we found that when we went through the pilot that was around about 30% of the time we're actually
16:00 - 16:30 generating one um and part of the challenge we found was we were asked to reduce that 500 um 500 page knowledge base down to 120 so we we've been gradually increasing the pages that we have there and uh we are seeing that increase each time we do that we're also completing some analysis at the minute around when we don't generate a response is it because we have a gap in our knowledge base so we need to create some information or do we have the
16:30 - 17:00 information but it's not structured correctly for an LLM to actually ingest that or is it simply the case that we do have the information is in a really good place but there's something happening in technology pipeline where it's triggering something else so we have some guardrails where for example if the answer generates the word advice we do not display that to a customer there are regulations where we can't give advice to customers um so anytime there are certain words detected we will immediately just stop that from from
17:00 - 17:30 going through but there might be something in the future where we actually ask it to reprompt because we're confident it's correct but we've used the wrong terminology so yeah there's there's definitely some analysis that's still going on but we know that it's definitely the right track to be able to answer some of these questions because Kora typically doesn't hold product information our products change so fast especially with savings fixed savings accounts things like that um and mortgage rates that if we were to have
17:30 - 18:00 that information embedded into Kora we'd have to have our conversation analyst fulltime updating that every release so we often will just hand off to the web page and that's one of the things that we really like about Corora Plus is it takes the onus away from the customer having to read the ref page and we're actually doing that for them as part of the rack process interesting lots lots of questions I have here that we can go and do let's first maybe just begin by making sure I understand this properly then so So you have the interactions
18:00 - 18:30 happening with the customer they express a need that your NLU system can't quite classify but it understands roughly speaking that it's a mortgage query you present what you call a low confidence menu others might see that as a disambiguation menu of some description which says you know this is the the things that we think it might be related to and then you offer the
18:30 - 19:00 opportunity for users to then take you up on a generative AI interaction that's correct so far yeah that's correct yeah and then what happens is that for those that opt into that presumably I think you said it was 33% which the researchers got absolutely bang on uh the they then are then sent into a a rag pipeline is then triggered based on content from your website and you retrieve that and answer the question and then you said that the what was it
19:00 - 19:30 30% of the time an answer was retrieved initially because you were asked to cut down the content is that correct yeah so so for 70% of the time in the first few weeks of the pilot uh users would actually receive a response saying we've been unable to generate a response and then they would put them down the low confidence menu route again yeah um so we are increasing that with each kind of release of new content to make sure that we have the right content to pull out from uh we're also going through a bit
19:30 - 20:00 of a process of looking at our content strategy so and and this is the same for for various chat bots across the group the internal chatbot I was mentioning earlier the HR chatbot some of our documents are FAQs they're not structured in a great way for chunking when it comes to a knowledge base um so we're looking at okay what's the ideal way for an FAQ to be structured for an OM to pick out the relevant piece of information yeah that makes sense we'll
20:00 - 20:30 definitely come on to that uh in a sec so so initially for in the first instance because you had to cut down the content uh it was retrieving contents 30% of the time but then you mentioned something like what was it it was it was when it did retrieve the content it was accurate to a incredible degree so like in terms of what was retrieved and what was in in those instances where it did find an answer the accuracy rate of those answers was pretty good yeah yeah so our accuracy
20:30 - 21:00 rate at the minute is 99% uh slightly higher than that actually um we've had a couple of incorrect responses they've not been harmful to customers and because it's for our customers who are logged in only we go back to those customers after the fact and let them know um but there's been a couple of times where say a customer said uh for example I've paid too much into my credit card how do I get it back into my current account and the large language models got a little bit confused where it's then told the customer how to
21:00 - 21:30 transfer funds from their credit card to their uh current account but it's it's took that information from how to do it the other way around right so when a customer then goes okay it's told me how to do it and goes into the app and clicks credit card there's no transfer button so they can't actually do any do the instructions that the English model said um but then we've used that information to feed back into our kind of guard rails to then see okay well how can we prevent these from happening again in the future but yeah that was um six times in the first 12 weeks that I
21:30 - 22:00 did that um and we we're um yeah we've not had a single instance where it's been like a harmful response to a a customer which is the most important thing and by increasing the number of uh knowledge articles that we have in in the knowledge base we've actually seen the overall um quality of the answers improve as well so we split ours into a number of buckets we have uh correct which is great that's the best one uh we
22:00 - 22:30 have correct but ambiguous where it could potentially it is correct but it's not been expressed in quite the way that we would want it to be um we have uh incorrect which is where it's given incorrect information and we have incorrect and harmful and we've had no incorrect and harmful responses so far which is ideally what we're after right for talking about financial information definitely i really results to be fair i mean you know to have that level of accuracy is brilliant and we'll get into
22:30 - 23:00 how that's kind of being measured and stuff like that in a minute but I want to kind of touch on sort of like you were talking there around the content strategy and you know ways in like what's the best way to kind of create and format content for large language models and stuff like that but at the same time you're getting a very high level of accuracy so what did you do initially to sort of like to when you're getting started you've got content on your website you said you got these 200 web pages or what have you um what did
23:00 - 23:30 you do in order to initially create that first rag instance did you just purely take the content that you've got use that content as it is leave the the system to do its own chunking and do its own thing like or did you do did you do anything regarding kind of like content strategy at that point walk us through the the process we initially had a look at our front facing web pages um and then we quickly realized that it the knowledgebased tool that we had
23:30 - 24:00 and the scrapers that we had they they struggled to get the relevant information out there there's a lot of noise on those pages with headers and footers and things popping up um and the structure is also different from product to product there's there's not a large amount of consistency across them and then we realized we a smaller part of the core team also owns our FAQs our public facing FAQs and we realized that actually they have a lot more consistency in them um across the board and we also have a ridiculous number of
24:00 - 24:30 them i know it stands for frequently asked questions that's definitely not what we have we have close to 5,000 um so we then actually started to pivot and focus more on the FAQs because it was a lot clearer how they were structured for the scraper to be able to get the information originally but also then for the knowledge base to actually comb through and understand one example we had when we were doing the testing is um Nat West actually removed the sale of B tolet mortgages and one of the pages we
24:30 - 25:00 had was the Blet mortgage page and the page itself had remained the same even when the product was off sale cuz some people still have those products and would need support on them but there was a banner that was placed over the top that said these products are no longer for sale and the scraper couldn't see that banner so the page still got scraped still got passed through and if we asked like "Oh what mortgages do you have on offer?" It would say "Here are the ones we have." And actually we didn't so that was
25:00 - 25:30 caught when we were doing our testing but there's some some nuances with the pages like that it also helped us a little bit because we started to notice that there were some web pages where if it would have like a question of how do I do this in the mobile app and it would have a very clear 1 2 3 4 5 6 7 here are the steps you need to do to do this particular service journey but then you'd have another page for a different product type where it would just say go and do it in the mobile app or you could
25:30 - 26:00 do this in the mobile app so we soon started to notice a bit of variation in terms of how far each of the product areas went with their information and we use that as a feedback loop to then say okay this is kind of the standard you want to be at for your pages so you need to change this page to match that standard interesting so so doing that work it just goes to show you doesn't it that like you know every organization has the same kind of challenges when it comes to content which is that lots of stuff is published very little is
26:00 - 26:30 usually taken down and once things are published they just stay there you know over time and things change and other other things improve as you publish new stuff and all that kind of stuff and the historic stuff doesn't get updated and all that kind of stuff so it's a it's highly likely to be a very common problem and it's what's interesting is that you know going through the proper process for creating a retrieval augmented generation system which is to do that initial work to figure out what content you've got and what shape it's in and what's relevant and what's not
26:30 - 27:00 relevant you know has knock-on effects and benefits you know across the organization because you're then spotting stuff that you know you can improve elsewhere which is great absolutely absolutely and so you you're saying there that like obviously that that you you've ingested content from the website you're going through that sort of testing phase and you're finding these kind of issues what does that sort of testing process look like is that a manual testing is the tools involved like how do you go about determining when something is ready to to to move on
27:00 - 27:30 to the next phase of of the process the the important thing was we had full engagement from like what we call the customer business areas so we had full engagement from some people from mortgages loans credit cards overdrafts and what we effectively had was a test set of we went through all our confidence uh menus and we looked at the oances that had triggered them and we broken it down to about 200 utterances that we wanted to like consistently test against and they were real customer autes we hadn't cleaned them up in any
27:30 - 28:00 way we wanted to make sure it was as accurate as possible and then every time we made any tweaks to the to the rag system we would run through those 12 sorry 12 200 question set through the rag get the answers out and then we'd painstakingly ask the customer journey managers to go through and just verify are these correct answers and there was quite a few rounds of kind of iteration with that um but once we got to a point where we were happy with the level of
28:00 - 28:30 accuracy that we were providing we then invited uh we have a group of 500 colleagues that we call upon for like colleague testing within Kora so we invited that group of people in they a lot of them hadn't even heard of generative AI before like a lot of them didn't like chat bots which is absolutely fine arguably better to be honest well exactly so we basically we we'd isolated the fact that it could only answer questions on those four product areas using generative AI so it
28:30 - 29:00 wasn't like the full core um and we basically said to them ask questions about mortgages credit cards overdrafts and loans like we didn't even give them any kind of subject areas to ask questions about we then got the feedback from there and looked at the accuracy again we involved the customer journey managers to make sure that we were given correct information and once we got to a point where we were happy that we'd kind of stress tested it there we completed the red testing with um our HR colleague B um and went through the relevant
29:00 - 29:30 governance made sure the guardrails were all up to scratch for what we wanted to do and we went live with the pilot which was a limited pilot when it when it first started of 200 customers a day and it was you had to be authenticated you had to be net west it had to be one of those product areas and it had to hit the low confidence that's how we trigger the journey so we knew we were going to have a small sample size to begin with but that helped somewhat to start that testing process with actual customers
29:30 - 30:00 customers are always going to ask questions in ways you never imagined so yeah it was once we'd started getting customers to actually use it that was when it became quite useful data for us to then react and and look at how we could optimize further that's a good idea i mean that's a really sensible kind of uh approach isn't it which is to begin internally get it to an accuracy level that you're happy with roll it out again internally you know those people internally obviously they know the business so they kind of know how to phrase things but still you're still
30:00 - 30:30 going to test whether or not one is there any gaps in the content and all that kind of stuff and then slowly rolling it out to customers to get real data is uh is a great idea was there any surprises in there once you started putting it in front of people you know you got a really good level of accuracy 99.x% of of you know accurate answers going back was that how it started was it always like that or were there any surprises in there when you first started putting it in front of customers it's always started there i think the the biggest surprise for us was was how
30:30 - 31:00 how low the percentage was when actually generating a response um so that was something that we kind of focused on in terms of working on it um the the number of people saying yes wasn't a surprise uh because of the research um the uplift in satisfaction from like a containment perspective was a surprise as well cuz these are these are journeys we struggled with for years where like I say we're at like 30 35% containment for those particular areas of Kora and to
31:00 - 31:30 then see that go higher than our base level of containment like 75% was astonishing um so yeah I would say that's probably one of the biggest surprises we had um just seeing how much that when we actually generate a response it's given a much better experience to people interesting that's really interesting how did you approach the red teaming kind of element of it you mentioned you're working there with an internal team and you're testing there but they're testing yours was there anything in particular that you
31:30 - 32:00 would kind of recommend teams do during that phase you're trying to obviously break it and and see whether or not you can you know make it trip up and get it to hallucinate or or what have you get it to go off the guard rails like are there any sort of learnings you have from that fears that might be interesting for people yeah I think like I mean you can you can find a lot of the information online about ways that you can trick and and make make bots hallucinate i think the the one thing that we found when when we went through ours was trying to remember this correctly now
32:00 - 32:30 um someone had asked Kora to Kora Plus to generate a link um and disguise it as an image and it they were successful in doing that the link went to nowhere but the fact that they were able to get it to to behave in a way that we weren't that was like great learnings for us immediately went in and rectified with some guardrails to prevent it from ever creating a link um but yeah it was
32:30 - 33:00 I find it always interesting getting people who are who are quite intent to try and break something to try and break what you're working on because there'll always be ways that they can think of that you've never considered uh whether that's getting it to speak in another language or or you know try and overwrite the kind of persona that you're setting in the in the tone of voice prompting um it's really interesting to get people who are somewhat experienced with this technology but
33:00 - 33:30 just just more curious uh than anything cuz there'll there'll be people out there that that have you know tried to do the same thing for for lots of large language models like GPT and yeah I think getting getting the right people to do that testing is the key thing there interesting that's what you they always say that you kind of like you want you want to test your system like you know if you're building software and you're building like a platform like a Netflix or even a mobile banking you know service like Nat West has and stuff
33:30 - 34:00 like you want the hackers in there to be doing the testing because those are the ones ultimately that are going to find the holes and and you know all that kind of stuff in it yeah um so how did you approach and we don't need to go into too much detail about this i'm just curious in terms of the level of control that you have over this rag pipeline because we've seen many other examples from other companies who have they've managed to get a retrieval augmented generation solution up and running but they don't really have a great deal of control over stuff like the guard rails
34:00 - 34:30 and stuff like how it's chunking stuff and you know a lot of that in certain platforms is quite kind of hidden away from from the end user um you just basically in some some are as simple as upload data and then you can type stuff to it others are a bit more sophisticated where you can see the chunk in you can edit bits and pieces you can maybe add some metadata and stuff like that but fundamentally you can't really touch the prompts on the on the front end so I'm just curious in terms of like how did you go about constructing this thing again you don't have to give us too too much kind of
34:30 - 35:00 like secret source away but like was it a situation where you've got full control and there are any any learnings around that or is it a case of the platform you're using it's essentially give it the data and you get the output yeah so we use a couple of suppliers for this pilot um around the time there wasn't too many suppliers that were offering these kind of solutions um so we we have a lot of control in terms of like the custom prompt we send with the large language model um we have a relatively rudimentary knowledge
35:00 - 35:30 base that we definitely would want to improve on the next iteration um but the the main control we have is is the supplier we're using for the rag orchestration so we completely control the prompt that's sent to the large language model it also does a bit of verification so when the answer is returned uh from uh GPC 3.5 turbo is the model we're using for this pilot um we then cross reference that model with the
35:30 - 36:00 original question the original documentation that was sent to give us a provide us with a confidence score we can set that threshold uh where it'll either just go yeah we'll return the answer to to Watson Assistant or no we won't return it um but it is something that we're definitely looking at as kind of the next iteration of Corora Plus to make this more scalable for us what are the best components to use what's the best chunking strategy what's the best knowledge base what's the best large language model so at the time we used what was available to us in the market
36:00 - 36:30 and the technology has moved tremendously in the last 6 months uh so we're we're consistently looking at how can we improve certain elements of that rag application and how do we uh kind of chop and change certain parts to give us more control ultimately but also make it so that if we can have more control over the application we can then start to look at how do we scale this out past 200 customers a day to the we've estimated that looking at our
36:30 - 37:00 journeys it'll end up we possibility of around 40,000 conversations a week could be handled by this um as opposed to then having actually conversation designers having to manually create those journeys yeah and for things like questions you know that have got answers all the complexity and and the time spent for designers to map out those journeys and you know all the disambiguation that goes into it and it's just like sometimes it's a thankless task because those journeys if they're not massively
37:00 - 37:30 high volume then one you're never going to get to them and if you do get to them you're going to spend an you know in order amount of time working on something that's kind of hardly not hardly being touched but it's like it's not a top journey sort of thing so it's like this technology is is far better placed to enable you to get far more kind of productivity out of everybody basically yeah absolutely that's the thing i think like we were talking at one point if we if we wanted to actually include all of the product content within Kora we'd be looking at hiring a
37:30 - 38:00 team of 50 to 100 conversation designers to like actually get all of that content in especially when you think that's just like Net West product suite and then the Royal Bank of Scotland product suite is fairly similar but then the Olster Bank one is a little bit more further away from that is of man bank and net west international further away from that so you can end up with and then you've got the personal customers business customers commercial customers and and one of the interesting things we found
38:00 - 38:30 when we were looking at the original testing was we we just threw all the overdraft pages in and we'd ask "Oh uh what terms do you have for an overdraft?" And it would randomly pick whether I was a student or a graduate or a personal customer a premier customer a commercial customer so that's another consideration that we then had to have of okay well we'll lock this pilot down to net west personals so that we don't have to consider of that just at this point in time but there's an element where we're then going to have to have
38:30 - 39:00 multiple knowledge bases for each of the different customer segments that we have so that we don't have that situation where you could be asking about a net west personal overdraft and we're telling you about an RBS commercial overdraft right that's situation we want to end up No and that that again goes back to the sort of control you need over the infrastructure because you're going to need to feed into your system that kind of context isn't it this is Mark he's logged into the mobile app he's a XY Z type of customer you know
39:00 - 39:30 this is his kind of world and he's asking this question now this is what we need to do to answer it rather than blanketly everyone gets served content from one location and it's kind of like generic but there's a lot more that that will kind of need to go into scaling that isn't there absolutely absolutely what What are the other considerations and you talked about content strategy and stuff earlier on presumably you know if this thing is going to progress into all of those other journeys you mentioned Royal Bank of Scotland and all
39:30 - 40:00 these other subbrands and all that kind of stuff they've all got different content um there there kind of presumably needs to be a way of you know you don't want that content to be kind of resting within Kora as you've already kind of alluded to because you know this is how uh what is it was it the Canadian airlines I think that end up getting caught with this where they had content that was hardcoded in their chatbot and it was about bereavement or what have you and and somebody asked it and they told them that yes you can have a refund for this ticket or whatever it might be turns out that wasn't their policy and
40:00 - 40:30 there was all kind of kickoffs about it and it and at the time it was like oh well this is all because of generative AI but in actual fact it was just because they had content hardcore it wasn't generative AI it was an NLU the content was hardcoded in the the system and it given the incorrect answer so you you don't want to take on the overhead of managing all the content for the whole organization you know you need to have some sort of pipeline here whereby the content as it's updated is being ingested so I don't know whether or not you've kind of solved that problem yet
40:30 - 41:00 or whether that's something that needs to be considered or how how are you thinking about kind of keeping content current whilst also managing quality on the front end there's conversations that we're having at the moment across the across the group because it's it's not it's not just a core problem it's also some of the other systems I mentioned earlier and and actually for our internal colleague chatbot there are a lot of questions where the answer will be the same for a customer as it will be for a colleague so if a colleague was to ask oh what are the uh what are the
41:00 - 41:30 benefits on the reward platinum account and a customer was to ask what are the benefits on the reward platinum account that answer should be the same regardless of who's asked however there are other questions where we really don't want that to be a blurred line you know where um if a customer sorry a colleague is asking what are the payment limits and the reasons why for sending a payment or what identification can we accept for taking a payment we don't really want to give all of that granular information we have in there around what we can accept and why to a customer
41:30 - 42:00 because that's our internal policies but the high level a bridge version of that of oh we need a passport to send 20 grand if that's the answer right that would be appropriate for a customer to know so that they can take the right uh verification into branch there there are there are nice parallels between that um in terms of some of the content should be a centralized place where we don't have full control over that but then there's other elements where actually this is specific to customerf facing and we should own so a
42:00 - 42:30 conversations happening at the moment and we're we're trying to kind of align across the group in terms of what the best overarching content strategy is if that makes sense yeah absolutely and and in terms of like how that affects the the subbrands and stuff like that I mean are they currently different solutions or are they all part of one solution that are kind of like customized based on certain variables based on the you know the customer kind of type or what have you like how how are you thinking about scaling that across all of these different brands are these going to be
42:30 - 43:00 different solutions is this one solution or is that not being kind of figured out yet yeah so so right now we'll typically have variations for different brands in our FAQ content and in our current content so we'll have the same question um but then we'll have potentially different variations uh we do something that probably a lot of other companies do as well where what's called online banking for Net West uh would be called anytime banking for another brand or uh so we have these lovely different variations and then one of the the
43:00 - 43:30 obvious ones is phone numbers contact numbers so if you ask for a contact number for net west and you'll have the same journey for RBS but they will be different uh phone numbers but the journey effectively is the same it's just that placeholder piece of content for the phone numbers that's different so we have variations and that's how we're we're looking at it at the moment but who knows that might change there's a lot that's happening in the content space with the likes of AM and and other companies as well where they're starting to explore these technologies too to try
43:30 - 44:00 and make that a bit simpler for content his interesting yeah it's a really it's a different type of problem isn't it when you reach the kind of scale of N West it's not like someone who's just kind of got a bunch of content and they want to create this system that allows you to talk to it there's a whole world out there with all of these different brands all different products all different customer types and finding the way of marrying what on the face of it seems like a fairly straightforward process there's there's content that stick it in a in a vectorzed database
44:00 - 44:30 and then ask a question and retrieve it but to scale that up to the level that you need to scale it to it's it's not a simple task no not at all um and yeah something we've been looking at for a number of months and and yeah continuing to do so yeah and the the other uh use case which is which is really good i know we've spent a lot of time talking about kind of like you know front-facing customerf facing stuff there but you'd also been uh experimenting with ways in which Kora can or Kora plus can be more helpful in helping the the human
44:30 - 45:00 colleagues and the agents that sit behind it yeah absolutely so we we on average will hand off around 50,000 conversations a week to our colleagues um I actually had the the good fortune last week of of being in India and and sitting with one of those teams and um it was very interesting the the main feedback was we want to see more of our conversations go through this use case which is great to hear um but yeah we we typically will hand off that many conversations
45:00 - 45:30 what we'll do when we'll hand off is um we'll hand off the transcript of the conversation and that's typically four and a half thousand characters um and it will say Corora colon this is what Kora said customer this is what customer said so you can imagine that's not incredibly user friendly to go through as an agent and it can take on average around 13 minutes for an agent to receive the conversation read through it understand what the customer originally wanted to do understand what core has already tried to do to try and resolve
45:30 - 46:00 and the kind of sentiment of the customer so we've looked at using large language models to pre-summarize those three key points to an agent when we hand off so we'll hand off the conversation we're doing this on a small scale with one of our uh segments of customers at the moment so it's around 100 customers a day that are being summarized and um what we'll do is we'll take the intent of what the customer is actually looking to do and that will take into account the whole conversation
46:00 - 46:30 we'll look at what Corora has tried to do so far and in some cases that might even just be Cora handed off the customer because it was a process handoff um and then we'll also put down what the customer's sentiment is so is the customer happy are they frustrated are they uh sad are they feeling giddy that day i don't know so that the agent is also fully aware of kind of what what kind of position the customer is in right at that point in time um that summary is about 450 characters so it's
46:30 - 47:00 10% of the size and uh yeah spending time with the agents last week they were saying they get excited when one comes through we're seeing an accuracy of around about 98% in that survey at the moment so we have a feedback form with it where we ask them on a scale of 1 to five five being the highest how accurate was the sentiment how accurate was the summary they're getting around 4.6 out of five at the moment on average and then we ask a yes or no question of was
47:00 - 47:30 it more useful to read the summary than it was the chat transcript and that's about 98% of people saying yes to that question right now wow so we got a plan to ramp this up quite dramatically over the next few months and yeah this is definitely more of a colleague focus for us cuz for every minute we can shave off the average time we're saving 50,000 minutes of agent time reading through chat transcripts but we're also saving 50,000 minutes of customer time waiting for that initial
47:30 - 48:00 message from a from a colleague so we do have a target to try and get that to 5 minutes which would be 400,000 minutes a week we're going to save um and spending time with the agents i'm I'm a bit more confident in that last week but it'll have a a more passive impact on the customers but definitely a real impact on on our colleagues in in those uh web chat centers brilliant and that's the whole point isn't it the whole point of this stuff is that it should be doing both things really it should be helping customers achieve outcomes better and
48:00 - 48:30 faster and it should be helping colleagues process and and and help customers faster as well you know so it sounds sounds like that's kind of doing that what what are the I'm always curious about sort of like you know I I used to have a manager years ago who used to always say you know things are if things are going well then things should be going well but the things that are not going well that's the things that I want to know about cuz that's the stuff that I can do something about so out of curiosity the 90 98% that said brilliant what were some of the things
48:30 - 49:00 within the 2% that perhaps could lead to a little bit more uh improvement like what what were the kind of stuff that might be coming through that the very very small albeit like some one one train of thought is that let's not even bother with that let's not even think about the 2% because 98% is amazing but just out of curiosity like what are the the little things perhaps that that might get it towards you know perfection yeah so we we've gone with like two of our we we call them our handoff skills so we've gone with RBS credit cards and
49:00 - 49:30 RBS complaints and it's the complaints where that 2% kind of crops up and it's usually around stuff that would help the complaints handler gather all of the relevant information so quite early on in our testing we found that the thing that they wanted more than anything was the reference number so when Corora will when a customer goes through a complaint with Corora Corora will do the initial information gathering for the complaint and then we'll uh effectively create that complaint number for the customer
49:30 - 50:00 so that they're not waiting around for an agent back and forth asking those questions and then when it's handed over or if they're asking for an update on a complaint it's handed over we'll look up that complaint reference number and the agents are like "That's the thing we need more than anything." So we can easily just then copy paste into the complaint system load it up straight away but as we've gone through this pilot it's always little small piece of information where they're like actually it didn't tell me about this element of the complaint and that would have been really useful to be in the summarization
50:00 - 50:30 whereas I've had to then go through and have a look through the transcript because it's eluded that it's there but not told me the exact information so it's always those kind of little last sort of gems for them to be able to just crack on with what the customer wants from the summarization on the credit card side it's been more or less everything's really good and telling us what we need to do um I mean I did see one when I was there last week and it was a customer who had just said "I want
50:30 - 51:00 to speak about a credit card send me to a human." Uh uh the summary probably ended up being longer than the actual from the customer uh but that's definitely an edge case yeah definitely i mean those things are just minor anyway aren't they really i mean you know considering that 98% of the time it's fine you know you know having to read back through transcripts on on one in every hundred handovers is is nothing compared to what it was like which is wicked um what are the kind of
51:00 - 51:30 like steps or challenges around scaling that then you mentioned it's it's in kind of credit cards and complaints presumably this kind of thing is independent of Corora plus essentially in terms of like the rag system you don't have to have that in place to do this so presumably this can be applied on every journey like are there any kind of like significant challenges to applying that everywhere or is it just that you're in the process of going through it in a in a sort of phased fashion as as you would yeah so we have a plan to go through in a phased fashion so we've worked with the operations team
51:30 - 52:00 to go okay what would be the next logical skills what would give us the the biggest kind of feedback so what we're quite keen with this one is uh to get as much feedback as possible as early as we can so we can then make any changes to the prompting or or to the model um but one of the biggest challenges we have is one of the models is being uh so we our models hosted on what's the next um and one of those models is being uh upgraded to a new model uh which then means we need to actually just take a pause and go through all of our governance again to
52:00 - 52:30 get that model through and and also do the regression testing to make sure that actually by changing the model it hasn't then dropped us from 98 to like 10% do you know what I mean so that can slow you down a little bit um and it's probably a pain everyone's feeling um and I noticed on a lot of these providers they always have that little box that says we have the right to upgrade a model in 90 days and actually for larger organizations you could probably struggle to get through the governance process in that 90 days so uh yeah I think that's probably one of our
52:30 - 53:00 our bigger challenges for actually scaling this out we could have probably moved a bit faster if if it wasn't for little lovely surprises like that I would say yeah i mean how do you think about the the advancements of of foundation models and stuff like that because on the one hand if you're using a model that's performing really well I think you mentioned uh 3.5 Turbo in the in Kora Plus if that's performing really really well as OpenAI kind of releases more and more models they start to kind
53:00 - 53:30 of like cease support and then cut off the kind of earlier models and stuff like that even though they're kind of performing pretty well and sometimes depending on the vendors that you have I mean the vendors all of the vendors doesn't matter who it is every single time you know Deepseek is released three days later they're all crowing about how now they support DeepSeek and now Grock 3 is out there and guaranteed that next week it's going to all be about we now support Grock 3 so it's kind of like on the one hand it's great for them to you know show that they can move fast and
53:30 - 54:00 that they can keep up to speed with all these new foundation models but on the other hand like from an enterprise perspective you want stability you want reliability you don't want things to be changing and unpredictable you don't want to go and redo all the work that you've just spent the last eight months doing because there's a model change if the model's performing kind of all right so how are you thinking about like you know the selection and management of kind of models going forward are you keen to explore all these new models or really are you just looking for something that works and and and that's that yeah it's it's an interesting point
54:00 - 54:30 because yeah they're like you know there are models that we're talking about today that you wouldn't have heard of 6 months ago there are models that we were talking about 6 months ago that no one talks about anymore and I I think the we we'll get to a point especially from an enterprise perspective where actually like you say we have a model that's that's given us 98% accuracy it doesn't necessarily need to be changed like it's working um so there'll definitely be a point where there'll be
54:30 - 55:00 some consideration of actually can we just take a cut of that model and use that and until we see a significant improvement and and that's what we'll potentially look to adopt um but yeah I think we we've seen advancements in the last 6 months not just in terms of models capability but also the reduction in price of the models and I think that's the key thing from an enterprise perspective when we were looking at you know what model should we use for Coral Plus if we were going to make it more scalable um cost was definitely one of those
55:00 - 55:30 factors and when you look at the likes of 40 Mini for example like it it's very close in terms of a rag application in terms of performance to 40 but dramatically cheaper to use than 40 and and it's interesting as well when you do that benchmarking because like we went into that and thought Gemini like you look on paper you know it's got a large context window like it ticks all of our boxes in our criteria we're looking at it was the worst performing model for
55:30 - 56:00 that application we just kept spitting everything out in bullet points and no matter what we trying to you can kind of see it when you go on Google and you just search anything and Gemini will pop up and it's bullet pointed answer but it yeah there's it my my main kind of advice to anyone is just to have a a broad range of models that you want to benchmark and a clear set of criteria because you'll find that like the the there's a lot of benchmarking available in like hugging face in open source
56:00 - 56:30 areas when we did our benchmarking it was very similar to that and it kind of validated that okay we're more comfortable to kind of look at those benchmarks and use those for some of the use cases but I think it's always useful just having a small selection of models that you think will perform well for the task you're looking to do and then doing your own benchmark on it because some might surprise you like Gemini did for us that's very good advice very good advice and you know in terms of taking a cut of models and stuff like that for
56:30 - 57:00 for you know Nat West if your use cases aren't I mean your use cases aren't going to change dramatically you're going to have new products and services and all that kind of stuff but fundamentally you're still going to hold money and loan out money and people are still going to need the wraparound services that comes with all of that kind of stuff um and so although your content will change fundamentally the needs of customers are likely to be fairly stable you know you might get a COVID that happens again and things kind
57:00 - 57:30 of like you know all hell breaks loose so you have to be prepared for that but like generally speaking day-to-day banking it doesn't need to change so dramatically that you need to be swapping out models every six weeks and so I think there's also an opportunity perhaps then to look at things like you know open-source solutions that maybe you have that you host yourself and fine-tune yourself and you know then then it's yours then you know and you're not relying on the cost of the open AIS and all that kind of stuff so there's
57:30 - 58:00 there's lots I think of of opportunities and ways in which you could do that in a in a sustainable kind of cost-effective fashion going forward isn't there yeah absolutely absolutely and I think you know just how far things have moved in the last 6 12 24 months um like we we're definitely seeing that improvement in performance we're definitely seeing that reduction in cost but we're going to get to a point where that kind of starts to plateau a little bit and actually we've
58:00 - 58:30 hit the performance we need we've hit the performance that's acceptable we've hit a cost that's acceptable so yeah that's when when to start looking at that kind of stuff nice wicked mark thank you so much this has been really really interesting really good stuff really appreciate it there's absolutely bags and bags and bags of insight in there that I'm sure lots of people will learn a hell of a lot from uh I love the kind of approach that you're taking in terms of you know I was having this conversation on the on the the previous podcast um yesterday was it yesterday or last week whereby the whole kind of
58:30 - 59:00 purpose that was with with Citizens Advice that was it uh the whole kind of approach that the citizens advice are taking is is similar in a sense of it's let's find an area where we can affect let's do the right thing in terms of designing the right solution and making sure that we're happy with it and it's got the right levels of accuracy and we're we're comfortable to go to the next phase going to the next phase which is then gently testing it with customers and still gathering information and feedback and iterating and making it as good as it can be and then steadily
59:00 - 59:30 rolling it out into production i think that that's the sort of approach that you've got to take really it's not the the sort of big bang kind of like you know bish bash bosch we're now doing generative AI it's about building sustainable solutions that work for the people that need it to work and I think the approach that you've outlined there is great and the results that you're getting so far is is testament to that you know absolutely yeah it's we've always wanted to be quite careful and measured with how we do things we we've had a lot of conversations about AI and
59:30 - 60:00 ethics and and and how can we do this in a responsible way and I think yeah we're starting to see that kind of payoff where we're actually pro being able to provide customers and colleagues with a better experience and and do it in the right way so wicked wicked so we'll do this again at some point again in the future for your third time your hat trick well I'll have to find a way of of giving you some sort of commemorative memorabilia for that hoodie or something like that so you wear a Nat West hoodie there maybe we'll replace that with a V hoodie but thank you so much been really interesting thanks so much for spending
60:00 - 60:30 the time with us no worries it's been a pleasure as always nice one and thank you all for tuning in we'll see you again on the next one thanks very much bye now