AI Security in Action: Enterprise Adoption, Challenges, and Holistic Solutions

Summary

AI security is increasingly crucial as organizations adopt emerging technologies. In "AI Security in Action: Enterprise Adoption, Challenges, and Holistic Solutions," CalypsoAI hosts a discussion on securing the AI landscape with experts Daniel Kenzie of Accenture and James White of CalypsoAI. The conversation delves into the comprehensive adoption of AI in enterprises, the risks involved, and strategies to ensure responsible AI. Emphasizing a proactive approach, they explore AI governance, security frameworks, and the importance of maintaining trust and transparency in AI applications. The discourse reveals that robust security practices are vital in this dynamic environment.

Highlights

• Experts highlight the need for cross-functional collaboration in AI security. 🤝
• AI governance includes guiding principles for ethical AI use. 📜
• Daniel Kenzie emphasizes that AI security is a business enabler. 💼
• Generative AI brings new challenges and requires dynamic security solutions. 🌐
• James White provides insights into building robust AI systems from the ground up. 🎯

Key Takeaways

• AI adoption is widespread in enterprises but comes with significant security challenges. 🔒
• It's crucial to integrate security from the start to facilitate seamless AI implementation. 🚀
• Companies benefit from forming cross-functional teams to address AI security comprehensively. 🎯
• Engaging both internal and external expertise helps navigate the evolving AI landscape. 🛤️
• Responsible AI practices, including bias reduction and compliance, must be prioritized. 🏅

Overview

The AI landscape is rapidly evolving, and with it, the need for robust security measures intensifies. In a recent discussion, industry experts Daniel Kenzie of Accenture and James White of CalypsoAI shared valuable insights on securing AI technologies. As organizations across sectors embark on AI journeys, they face the dual challenge of leveraging AI's potential while safeguarding their data and applications. The experts emphasize that early integration of security processes is crucial to mitigate risks and enable smooth AI adoption.

Enterprises are increasingly forming cross-functional expert groups to tackle AI security, drawing on a blend of internal resources and external partnerships. This approach ensures comprehensive security measures are in place, protecting data integrity and maintaining compliance with regulations. As AI technologies like Generative AI bring new complexities to the table, adaptive and holistic security strategies become critical. Maintaining trust and transparency with customers and users is emphasized as a key aspect of responsible AI practices.

Responsible AI is not just a regulatory requirement but a significant business opportunity. Companies that prioritize ethical AI deployment and demonstrate transparency can build better relationships with stakeholders. The discussion highlights the importance of governance frameworks that reduce bias and adhere to emerging regulations. Incorporating best practices into AI systems not only enhances security but also positions organizations as leaders in technology innovation.

Chapters

• 00:00 - 00:30: Introduction In the introduction, the host of the podcast 'for Tech sake', Elber, sets the stage for a discussion on the rapidly changing AI landscape and the importance of its security. Elber emphasizes his role is to facilitate the conversation while experts, including Daniel Kenzie, will provide in-depth insights into the topic.
• 00:30 - 01:00: Guest Introductions This chapter begins with introductions of two key figures in the field of AI and cyber security. H, who is the global data and artificial intelligence security leader at Accenture, focuses on the secure implementation of generative AI and its role in revolutionizing cyber security operations. H is also involved in promoting the security aspects of Accenture's responsible AI services. It is noted that integrating cyber security into product and service architecture, thereby transforming it into a business enabler, is a significant passion. The chapter also introduces James White, who is the president and CTO of Calypso AI, recognized as a global leader in the adductive AI space. These introductions set the stage for a discussion on the intersection of AI and cyber security, as well as related technological advancements.
• 01:00 - 03:00: Current AI Trends in Business This chapter discusses current trends in AI applications within business environments, led by insights from James, an experienced figure in engineering and data science. He has extensive background in various domains including legislation, customer and employee experience, notably in cybersecurity at Mandiant and FireEye. The chapter aims to address the current state of AI in enterprises, focusing on practical deployments beyond the recent hype.
• 03:00 - 05:00: Security Concerns in AI Adoption The chapter titled 'Security Concerns in AI Adoption' begins with a discussion about the exploration and experimentation of new AI technologies within large corporations, particularly Fortune 100 companies. The narrative highlights that while these organizations have been on a broader AI journey for several years, the integration of generative AI is relatively new, gaining traction in approximately the past 18 months. It implies a growing adoption, although specific security concerns or measures aren't detailed in the provided text.
• 05:00 - 08:00: AI Governance and Expert Groups The chapter discusses the role of AI in business through a series of experiments aimed at improving efficiency and customer service. It highlights the successful integration of AI into production workloads, while also acknowledging the experiments that were not successful. The importance of assessing the applicability of AI technologies for specific tasks is emphasized, along with the understanding that some AI applications may not be suitable at certain times. Overall, the chapter underscores the value of experimenting with AI to optimize its use in business operations.
• 08:00 - 12:00: Responsible AI Practices The chapter discusses the widespread involvement of large companies in the AI business, highlighting the challenges and opportunities they face. It emphasizes the need to scale and derive value from AI, while also managing the dynamic nature of the AI space, with new developments like large language models, plugins, and datasets emerging daily. The discussion also touches on the necessity for guardrails to ensure effective use of AI technologies.
• 12:00 - 17:00: Operationalizing AI Security and Responsibility The chapter titled 'Operationalizing AI Security and Responsibility' discusses the dual aspect of implementing AI in business environments. It highlights the excitement and opportunities that AI brings, alongside the concerns and hesitation it creates, especially regarding security and data protection. Surveys indicate that business leaders are both eager and cautious about AI, recognizing its experimental nature and the imperative to not fall behind in technological advancements. The chapter underscores the importance of balancing innovation with security needs.
• 17:00 - 20:00: Practical Examples of AI Governance This chapter discusses the evolution of AI governance, highlighting the transition from an era where enterprises approached AI as a one-size-fits-all tool ('the AI Hammer') to a more sophisticated phase focused on defining specific use cases. The dialogue suggests that there is a significant opportunity in this space, as organizations have accrued a substantial amount of learning from their initial, broad application phases. This shift reflects a maturation in AI strategy, emphasizing tailored applications and informed implementation.
• 20:00 - 21:00: Conclusion and Final Thoughts The conclusion emphasizes the importance of evaluating AI use cases carefully to determine suitability. It stresses the need for establishing the return on investment (ROI) upfront and considering the operational costs, including staffing and security. Security should not be an afterthought, and budgeting for the team's staffing and model runtime is crucial. This ensures a comprehensive understanding and planning for AI deployment.

AI Security in Action: Enterprise Adoption, Challenges, and Holistic Solutions Transcription

• 00:00 - 00:30 so we're here today because we're going to talk about the AI landscape which as we all know is changing rapidly um but it also needs to be secured as well so we're going to delve into securing this AI landscape too and to have this conversation uh I'm going to be hosting I'm Elber the host of the for Tech sake podcast but the least they hear from me in this conversation I think the better because we have two experts here to bring us through this uh a guided tour of the AI landscape Daniel Kenzie
• 00:30 - 01:00 H who serves as the global data and artificial intelligence security leader for Accenture overseeing the secure implementation of generative Ai and its use to revolutionized cyber security operations and sponsoring the security aspects of accenture's responsible AI client services and it says here your passion lies in integrating cyber security into product and service architecture transforming it into a business enabler so that's all the things that we're going to talk about with you Daniel and we also have James White president and CTO of calypso AI the global leader in adductive AI
• 01:00 - 01:30 security James has almost 20 years of engineering and data Science Leadership under his belt and has worked in domains such as legislation customer and employee experience and of course cyber security with mandiant and fire eye so that's your credentials out of the way we've established that you guys know what you're talking about when it comes to this stuff but where are we at now in terms of the AI landscape and like I'm talking specifically about a doin Trends in Enterprise and putting it to use in business because you know we've had a lot of hype over the last couple of years we've had had a lot of people
• 01:30 - 02:00 trying out these Technologies and toying around with them but are they being put to work is that happening now yeah it's a great question Elaine and and Jimmy so good to be with you again here in Dublin um I think when I when I look at let's say the Fortune 100 just to kind of box in you know a bunch of large clients you know all of them have really been on an AI Journey for a number of years obviously the generative AI component of that been more recent let's say the last 18 months or so um but what we've seen by and large is they
• 02:00 - 02:30 all run a number of experiments with AI trying to find parts of the business where they could find things that really added value reduce toil you know and delivered better services and outcomes for their customers and by and large they have right so almost all of them have workloads that are in production with AI some of the experiments that they've done they've also deemed that they weren't you know successful and I think that's good that's healthy it's it's being able to test the technology and say could we use AI for this yes but it's not necessarily the right thing at this point in time for whatever the
• 02:30 - 03:00 reason is either because of some of the guard rails that maybe need to be put in place or maybe just the the the efficacy of the LMS aren't quite there um but but by and large from what I see from with large companies everyone's in the AI business in one form or fashion um and it's really a matter of being able to scale and get more value out of it while also navigating what's a tremendously Dynamic space right there's new large language models new plugins new data sets coming to the market each day so it's very overwhelming for most large organizations
• 03:00 - 03:30 but very exciting at the same time right so that's kind of what makes this a super fun job and a super exciting topic and there is like you know a lot of interest but also we see from surveys of uh business owners company leaders and stuff like that there is also trepidation there because it still kind of feels like at this experimental phase and I'm sure a lot of that concern is coming in terms of security James because you know they want their business data to be secure if they're plugging into these systems but they also don't want to be left behind uh if
• 03:30 - 04:00 there is a big opportunity here so how do you think they are thinking about that addressing that how is that coming through to you yeah I think the point Daniel made and welcome to dou Daniel that her Hospitality holds up um we've seen that phase where every all Enterprises were looking for a nail and they have the AI Hammer right with that's over thankfully and during that phase there's a huge amount of learning as Daniel pointed out we've seen the move into um use case definition really really grow where Enterprises are
• 04:00 - 04:30 looking at use cases and saying is this an apt choice for AI and then from that they go through the regular journey of figuring out which model is best um the ROI has to be established up front right that's that's critical and then understanding what is the cost to run this thing and that's where usually security comes in it's it's often an afterthought and but when you factor in the cost of running AI you need to budget for your team your Staffing um you need to budget for your security the runtime of these models and a lot of
• 04:30 - 05:00 these models are build and and and price pointed as the actual token size which the number of inputs and outputs the model generates and so when you factor all of that in it's really a big project especially the first couple of projects that go through your Enterprise once you build up that muscle memory you get into a routine where we understand which models work best for our Enterprise and as Danel pointed out the data which data are we using and how are we using it and and because remember key to all of AI usage is data
• 05:00 - 05:30 and you fall under the um existing regulation of data when you're using with AI or not so gdpr the California consumer act Etc and so your existing considerations that you have for your enterprise software hold for the new AI applications but then you've got additional complications on top of that to make sure that these new AI um products Services whatever way you're utilizing AI in your Enterprise that they are secure for your users internally and externally and so the very first thing we've seen work really
• 05:30 - 06:00 really well is where an AI expert group is formed inside an organization and going I know indeed D is part of many of those groups where you've got key stakeholders from legal teams security the product owners um finance and and of course um a concise um representation of what the company wants to do for a quarter or a yearly basis and all of that's factored in then and the solution of that is a multifaceted model use case um application and security solution is
• 06:00 - 06:30 that like can you tell us a bit about that experience on AI expert groups like is that bigger companies uh are bringing people in externally to advise on these things and is there a mix of internal people involved too yeah it's a great question I I think you know with any type of emerging technology so we saw this with Cloud we're seeing this now with AI I think we'll see it with with other you know Technologies as they continue to evolve you know when you when you have such Niche skills and and kind of a need for really kind of cross
• 06:30 - 07:00 functional thinking that's something that's very emerging um we're seeing a lot of organizations pull together these centers of excellence or and they can be a combination of cross functional stakeholders security privacy legal architecture operations um and then also pulling in external Partnerships right so this can be either from from University it can be from uh you know large advisory Partners emerging technology players Etc but I do think that that that pattern I'm seeing work quite well right because uh in in an
• 07:00 - 07:30 area that's so Dynamic you need to have kind of a a mixing of different thoughts and ideas and also folks that really understand how the business works today because you can come up with a lot of really cool things on whiteboard that don't always translate as well or as eloquently from a technology perspective um and so we do see security actually taking a really leading role in a lot of those groups right so I have a number of clients where the ceso is actually the chair of them not necessarily because you know they all sat around and said oh we think security needs to lead this but
• 07:30 - 08:00 a lot of times the experience that Chief information security officers have of kind of navigating new technologies nebulous regulations and then also resource scarcity I think most of the security business is kind of um foundational you never quite have enough of what you need um and so in this kind of interesting time of finops as it relates to AI sometimes that that that experience and that and that prudent thinking U coming out of the Chief Information Security Officer sorry Chief Information Security Officer officer um
• 08:00 - 08:30 is super helpful and so I I see a lot of times they're being asked to lead it and they're kind of taking a very cross functional lens um but I do think it's a model that uh really any large organization should really have a good good look at and and likely apply that's a really interesting point uh that the the person in your organization who is already leading on things like information security and your data protection is probably well positioned to take a lead on this because as James says data is kind of what underpinning what you're going to do at Ai and there existing regulation they have
• 08:30 - 09:00 experienced navigating and like when an organization James is kind of stepping out of maybe an experimental phase H with these as they they've toyed around with things they've spotted that there are use cases that they can apply this in their business like what how does that escalate the risk for them because then when it comes to you know deploying it for their customers like it does reputational risk as well as like H the risk of um more users more data and all that kind of thing so how are they navigating that yeah you mentioned the word toy there and I feel like H Santa
• 09:00 - 09:30 Claus has arrived in Enterprise all over the world and it's a really good thing so everybody has been used to the status quo in Enterprise for a couple of years as Dy pointed out Cloud was the last big exciting change that was transformational Enterprise AI represents that new um Catalyst in change Enterprise and so everybody has these new toys to play with and they've been experimenting and creating new capabilities or making efficiencies in existing capabilities but now um of the cesa comes in and is you know the bad
• 09:30 - 10:00 guy right so they're coming in and has to make it safe Etc and one of the big um seism shifts we've seen in the last year is that the seeso involved at the early stages like Daniel's pointing out it it's not an Unblocker It's actually an accelerator right so you have the cesa coming in saying okay here is the safe way to adopt um Ai and once that's known from the start in ad doctor from the start now you can run as fast as you possibly want with AI and that's a that's an unlock for Enterprise we've
• 10:00 - 10:30 seen Enterprise that retrospectively try to add security after the fact it's really difficult and challenging it's still possible but it's something that you need to um understand what's already out there what what you know has the horse already bolted what can we put the genie back in the bottle whereas you do it right from the start all your controls are in place and your AI system and I I use the word system carefully there so it's not just a model it's also the applications wrapped around the data stores are in the back so that AI system that holistically it's protected and
• 10:30 - 11:00 then all of that was factored in from the start so when the kids do get their toys they can play with them unencumbered by doubt right and that's that's what we want this AI um technology and as Daniel pointed out as well it's it's growing every day every month there's a new massive announcement and so with all these announcements coming out it can be overwhelming if you have the rules of the road and you can drive in that safely and you can drive at the speed that everyone's happy with and everyone gets to their destination in a real timely manner and safe
• 11:00 - 11:30 So Daniel you're obviously well vered in terms of responsible Ai and presumably there's aspects of that that bring in elements of security because you need to be securing users data and being responsible about AI in that way but what other elements are there to responsible AI practices see responsible AI is is a real personal passion of mine um I'm fortunate to have two young children one that was born pre J of AI and one that will never known the world pregenerative AI right so I I kind of view this as an area where I really want to make sure we can we can help very thoughtful I think um there's a lot of
• 11:30 - 12:00 other examples of technology where there's been challenges with that um but when we think about responsib I think it's important for every organization to kind of Define what that means to them right from an Accenture perspective we've we've defined some principles and we work with our clients to do to to to uh make sure that they're tailored for theirs but but generally it's things like making sure that you're instilling trust in your use of AI uh you're open with where it's being leveraged um you're reducing any type of bias and the
• 12:00 - 12:30 data sets um you're being sure you're compliant with emerging regulations security risks um conscious about sustainability um and so when we look at that from a security perspective you know security elements of that tend to become very impactful and also very cross functional it's really hard to ensure that you have uh you know no bias in your data set without making sure that you have the right Integrity from a data security perspective as an example uh it's it's really hard to make sure that you don't have uh hate speech perps
• 12:30 - 13:00 coming out of your AI unknowingly you unless you have the right safeguards and guard rails and you can kind of see what's going into those different pipelines um from an AI security perspective so it all really dubils very closely together um and as we see you know the regulatory landscape evolving um it's still very kind of early days in that regard but the security components of that are really kind of pull through across all these different dimensions um and it's just something that I think from a guardrails perspective and a policy perspective and a process
• 13:00 - 13:30 perspective it's super important for organizations to kind of connect those STS together and how impactful has ji been in that space because you Meed this kind of idea of a pre gen and a post gen generation which has blown my mind a little bit to think of that um and it has dominated the headlines in terms of AI but like AI existed long before that and responsible AI practices and divisions and organizations also existed long before that so has geni made a significant impact in this area the way
• 13:30 - 14:00 it has in a lot of others yeah it's a great question and so to your point responsible AI is and trustworthy Ai and those types of terms have existed for the last you know six years or so longer than that in some in some circles I think what's really kind of changed with generative AI is generative AI has made the technology so much more democratized right you can really kind of Open Source a lot of it you can do things on the weekends that now folks are looking to take back into their their their work life on a Monday and so that didn't really exist in of the pre IML World
• 14:00 - 14:30 which was really uh a little bit more rigorous and kind of confined to data scientists and engineers and folks that could could really kind of get very technical with it so now the fact that we can have you know children business users academics Etc kind of all working and and coming together and then the nature of large language models as well is it's very hard to really kind of unpack what's going on right they're they're not uh particularly transparent in that regard and so that's where you kind of have to wrap process and
• 14:30 - 15:00 safeguards around them and so that's a little bit different than traditional machine learning and and other types of of data techniques that we've had in the past where from a you know design and architecture perspective there was ability to put a little bit more kind of transparency and rigor in terms of how things are built so certainly things are different they're changing they're evolving but like good practices can continue and adapt and evolve with them too would both of you be able to give examples of where you have seen AI governance done well or even ideas
• 15:00 - 15:30 around best practice in this area yeah I can um I can give you a break for a second and answer that one um so I guess I'll start off with you know the the home space right so we've seen gen hit at the same time Enterprise and home and that's been really interesting because when you look at safety and governance Etc a lot of times people who um are used to using technology and work one of the opportunity use it at home and you
• 15:30 - 16:00 know never the twain shall meet we're seeing now folks have a lot of personal e of theyi and so they're growing not just at work but at home as well and when their kids are using it they understand wow this is a very powerful technology for good and possibly bad and so they they start putting their own kind of guardrails in place right and they're just selfing their kids usage their own usage Etc and so that understanding is um bleeding into Enterprise um as opposed to vice versa and then we have you know Enterprises
• 16:00 - 16:30 having to develop their own set of safeguards regulations because they don't exist in um in government in government sorry um different jurisdictions are are improving that and eui act state-by-state legislation the US Etc but what we're seeing is um let's take artists for example for a long time um artists have been trying to fight for the their art not to be used for training um models and it wasn't the government that that solve the problem is the AI model companies the training
• 16:30 - 17:00 companies and so we have now great research from the US with a a product called glaze and Nightshade which allow artists to actually be offensive right so now they can embed images in the uh that are not percep by the human eye inside their images and it protects their artwork so let's say they're they're depicting landscape environments and they use Nightshade to say um you know um this is a leather bag it will
• 17:00 - 17:30 create a invisible image of a leather bag that the AI model will use to train and ingest and so when you ask for leather bag you're going to get a landscape or if you ask for a landscape you might get a leather bag and so these type of things are are making it a nuisance for model trainers to to have to deal with and force them to face up to the issue so what we're seeing right now is that companies are leading the way and people like Daniel are advising those companies and we're seeing that land cap develop completely organically
• 17:30 - 18:00 inside Enterprise governments are trying to catch up indeed we've seen the UK create the laser group and which is now designed to um you know look at AI as a Potential Threat and how will they deal with it as a as a nation and we're seeing that across the board where this you know analagous nature of all of these organizations coming together from the home life to Enterprise to government and the Practical steps that people need to take are are normalizing
• 18:00 - 18:30 and so the hype cycle goes way up comes down where people are a little bit disappointed because of all the hype was maybe not fully realized and now we're at the stage where people are using AI for real workloads for uh productionize uh product usage in their in their organization internal usage as well and there's a set of safeguards that are widely adopted such as practical ones like putting a group together to decide which use cases are best to another group deciding which uh
• 18:30 - 19:00 controls and policies are put in place so that people are using AI safely and then the data that's generated on the other side the outcome of that data is it safe for consumption for your customers is it something that you'll reuse on the other side for retraining models you have that infinite Loop type scenario and so all of these come together into a maturity of such that 2025 and onwards a will become a normal tool in in in organizations and the understanding you have to safely adopt I think will become spread across all um
• 19:00 - 19:30 certainly Enterprise organizations hopefully smbs as well I think that will be the start of a next year and then H government will catch up I think the US will will put in place some good safeguards maybe not law um uh but certainly some safeguards and the eua ACT will be tested for the first time and we'll see the outcome of how good is it really for Enterprise and for the safety of uh civilians James raises a really interesting point there Daniel that kind of TS onto some something you were talking about earlier like we've
• 19:30 - 20:00 kind of accepted we may have made missteps with other technology deployments that were really broad-reaching and like concerns about safety maybe weren't addressed correctly but we are now in an age where these developments in AI are happening with a more conscious consumer a more conscious user a more privacy conscious and data conscious user does that present actually an opportunity for businesses if they want to um really engage with people and like them up in terms of like the opportunities of AI if they present
• 20:00 - 20:30 themselves as a responsible AI organization and and categorically can say that they are and holding to those tenants does that actually create a good business opportunity for them I think it does and I think it does for for two reasons right I mean there's a lot of uh academic and and kind of industry data to substantiate a lot of this but you know trust in services and Trust in organizations are are two things that I think are are you know consistently pulled as as concerns right so for both
• 20:30 - 21:00 as an employee that's going to go work for a potential employer knowing that your organization is thoughtful and responsible in terms of how you're using emerging Technologies it tends to be very important particularly if you're kind of working in some type of uh you know emerging space um and then from a from an End customer perspective right being able to to make sure that you have trust of the services you're consuming whether those are streaming media social content Etc um you know we see that in in and pulling data very highly and so I
• 21:00 - 21:30 think you know while it's important to define the principles putting effort in investment and resourcing behind the principles to actually implement the the safeguards and controls that's what really allows you to to have that demonstrative trust um and then allows you to also communicate that to those different stakeholders so that U it's very tangible and it doesn't become kind of just an academic exercise um but that in that end you know I I do believe particularly as we've seen with so many of the large data breaches and some of the other type of uh you know research
• 21:30 - 22:00 that's been done on on different types of of of technology companies a lot of customers are skeptical that that their their their data is being safeguarded correctly um or that you know the terms of use that they're agreeing to are actually being upheld by the broader organization so I I totally believe that responsible trustworthy AI is something that um you know I've seen be a business differentiator and something that a lot of thoughtful organizations are really trying to lean into so James let's now get into a bit of the nitty-gritty of
• 22:00 - 22:30 how we secure AI for use and Enterprise because that must be hugely challenging especially with generative AI because it's it's a much more Dynamic environment and you're even saying like about you have to talk about your AI system because you may have a model underpinning your process and what you're doing and then applications on top of that and a change in that model might have quite a substantive effect on your outputs and your outcome so I how do you even go about envisaging a security landscape around that yeah it's a great great point I think the the first part is to uh realize that that's
• 22:30 - 23:00 all very overwhelming if you take it in totality and so to divide it into two parts there are the companies who are spending billions of dollars per year on training these models and those companies have a different level of Burden than you should have to use these models so a lot of the Fear Factor around AI usage and training of data Etc sits with those um companies and organizations they're not all companies and so what you need to be concertive
• 23:00 - 23:30 for around utilizing these models is the inference layer so you're using these models for inference you may be fine-tuning these models where you're adding extra data to the model or using rag which is putting a data store behind a model to consume a true true AI but once you understand it's all inference for your organization then you can start getting your arms around it and figuring out what's our strategy to dissect this and secure it to different phases that we will use and so the first part is always defense right you want to make sure that you're defending against
• 23:30 - 24:00 threats that are known to be out there also defending against you know points that danger brought up earlier to make sure that the the data being used is correct and valid that the response is being generated and don't have you know represent brand damage for your organization or damaging content for the consumers of that information once you've got your defense um sorted organized that's usually policies and controls and those policies and controls to your point need to be dynamic so you know things things change um the the things that are acceptable
• 24:00 - 24:30 today may not be acceptable tomorrow and vice versa and so that's the defensive um colum the second part is offensive so for years and years and years red teaming has been a key and critical part for for security teams in organizations and Pen testing and and penetration testing different software well pen testing models is just as important right if you have things that you're concerned about that your model may give information that's not appropriate or incorrect or proprietary in some way
• 24:30 - 25:00 then you need to test that model and say will it give information these circumstances and that's getting ahead of the problem you might think you should do that first but um to the point of the dramatically accelerated landscape of AI we need to defend immediately right because it's already out there but once you're mature you can set up your offensive posture where you're testing these models regularly and not just testing the ones you already have but also evaluating new models coming in so there's new release of new models all the time and there's
• 25:00 - 25:30 open source models there's close Source models public private models if you're hosting this model yourself if it's a fine stent model you've taken a open source model and then um supplemented with your own data now the responsibility Falls to you to make sure that that data is secure and only used in a way that you intend and the last part and certainly by no means least is governance right so we have um the technology to use but we have to use in a responsible way that is adhering to your
• 25:30 - 26:00 jurisdictions um set out guidelines and rules and so governance does not just apply to new AI governance it also retrospectively applies to gdpr um you know the the California consumer act and a whole bunch of other uh jurisdictional elements across different regions so once you've got defense offense and GRC all set up for your AI productionize adoption then you're in a really good place and the best thing to know is that a lot of those skills already exist in your organization security teams and I
• 26:00 - 26:30 know before we began this conversation myself and Daniel were discussing the really key part existing security teams play in this new AI field and then you're mentioning about the importance of those teams being brought in from the start yeah absolutely and and and the other part I would add Jimmy I I think you did a great job summarizing that the landscape there you know the the new architecture is coming out around a are evolving every day right so this agenic
• 26:30 - 27:00 nature of of of AI That's that's coming out too you know being able to apply security knowing that you're not talking about a simple workflow anymore it's not just a model an interface for a chatbot and then a human but you're going to have multiple AI systems coming together and being able to make sure you have those Dynamic controls knowing that maybe when you architect something it's going to look very different in 3 to four months just basing on the rest of the ecosystem that starts to get integrated together uh I think that's what's super important for security Defenders and and privacy professionals
• 27:00 - 27:30 and the like to think about when they start to to figure out where the technology safeguards get integrated into the organization yeah and how how is that impacting maybe even the shape of an organization and even the teams that are being created around this responsibility because I suppose there's elements of governance before that you could have seen in as a bit of a said it and forget it kind of situation where you could establish your policies and practices around use of a certain technology and how you're going to deploy it um was more static than Dynamic and you could just leave it at
• 27:30 - 28:00 that and maybe review it every once in a while but this sounds like ever evolving constantly changing like how are organizations building that flexible security into them and embedding it into their organization yeah one of the um and I I'll just call it out here one of the unexpected new um realizations organizations have had is when you launch AI products to Market there's an intention behind that you want to achieve something and what we're seeing
• 28:00 - 28:30 is much and much uh greater success in achieving that intention however then there's a a whole new uh problem which is operationalization of that productionize Scrabble in just one sentence um however when you are operationalizing models you've got all these new problems so a unintended output goes to a cons a customer theying the support line what happens inside your organization from that point
• 28:30 - 29:00 onwards who is responsible for that uh misstep by by your software by your product does it go to the security team security team may not have any knowledge of this um possibility of this model of doing that so do they contact the product team is it the legal team as their brand damage here is it the marketing team to understand why this is on our website so understanding that when you put something out there you've you've also got to understand and set up your organiz ation to deal with things coming back in and that's another
• 29:00 - 29:30 maturity step we're now seeing a lot of companies are getting really far ahead of that but it all comes from that cross uh functional team that set up the mod in the first place you've got all those those teams understand their responsibility what part to play they have and understanding when something comes to them how they follow that chain from um initial creation so the the software development life cycle all the way how it got out to production when
• 29:30 - 30:00 something comes back in how they resolve it and iteration and it's a whole new world the stlc processes really good one but it has to be um changed significantly for AI adoption and that's some great advice there on operationalizing secure AI what about operationalizing responsible AI what are some of the Practical tools and capabilities people need to be exploring here Daniel so I think it's everything Jimmy hit and you need to add in some kind of front and and and and backend
• 30:00 - 30:30 type additional elements right so on the front end I think from an assessment perspective right really understanding what are the uses of AI across the organization and this is something that the UI act also dictates right so having that inventory being able to understand what are the uses of of AI um and then making sure you're removing either high risk or impermissible use right so having that intake process so you can even find it um put it through a process but then also running some detect scanning because there's probably a lot of stuff going on from an AI perspective
• 30:30 - 31:00 that's shadow in nature right because it either existed before you had your process or just folks are experimenting and it's not always kind of surfaced um then obviously putting those safeguards and guard rails in place to actually protect use cases you've identified now that you have a better understanding of where they're at what data is being consumed Etc but then that whole ongoing monitoring perspective I think super important because these these large language models can drift the intended use of of the workflow can change over
• 31:00 - 31:30 time right some of the data as it's enriched or integrated with through a rag type architecture can evolve in a way that you know what you started with on day one and where you were at maybe at day 200 looks and feels different and that's where those different principles that you have in place need to be U kind of tested and monitored on a real-time basis right and so security is obviously one of those it's super important you know a simple example is making sure you don't have sensitive data leakage going back and forth um but but others like you know bias as an example are more um
• 31:30 - 32:00 complex to test and require making sure you have the right technology Partners as well as the right understanding of again what we Define as intended use of this AI workflow so I'm really kind of rounding that out you know the most important thing for an organization is because this all can be very overwhelming right and so you don't want to view each of these types of tests and safeguards as something you're going to run in you know totally ad hoc of each other that needs to be integrated so
• 32:00 - 32:30 that when you're looking and talking to an application owner or an AI workflow owner you're bringing them the full Suite of okay great these are the four or five things we're going to look at from an assessment perspective these are going to translate into different types of guard rails and then we're going to monitor them but too often I think and this is one of the learnings of kind of security over the years is you know if security is yet another check that goes on an already large checklist of different things an application owner has to do it tends to you know reduce velocity within new organization and so create sometimes a culture where
• 32:30 - 33:00 security is viewed as a blocker and that's why we really want to make sure that we think about responsible as a package and kind of be able to take the that that business user through that workflow in an integrated way and Chan as our businesses are going to have to as you said uh engage a technology partner on this stuff they may not have the expertise in house to delve into this area and as you mentioned Daniel like they are overwhelmed you know cuz there there's a lot of um new language we're all using now I mean we're throwing around in terms like rag as if they've been with us all along I only
• 33:00 - 33:30 learned about that one recently myself so like that can be hard for businesses and even as a technology partner James trying to actually engage with them do you feel like there's a responsibility on your side to even like bring them on a journey and educate them and what they need to know about these systems yeah luckily we love talking about this stuff so um we we always start um trying to understand what they're trying to achieve the outcome because even though we're focused on security the reason we're do as we believe AI is the future is the way things will unlock a huge
• 33:30 - 34:00 amount of potential that's just sitting there waiting to happen um so we begin with the intent and we try to understand where they're trying to get to meet them where the r what they have in place already what existing security practice they have they're already creation and work alongside these new practices they need to implement and then figure out the Delta so when you've got that Delta identified how do you bring in the security team right how do you make sure that they're part of the solution right from the ground up and so if you think
• 34:00 - 34:30 about why this problem is becoming um slightly overwhelming for organizations right now it's because I as a k e to always talk with the goodies and the baddies right so the baddies have been using this technology for a year now right so they are using AI to you know augment their existing traditional attacks fishing is now something that was solved is now becoming very very difficult for security organizations to solve because the IES are using AI um and they can use it really cheaply they
• 34:30 - 35:00 can use it at scale and they can Implement a lot of damage without having to put much effort it and so it's unfair to position the Security Professionals in a way that they do not also have access to this technology so bringing AI in as a as a tool or as a set of applications that the security team can utilize that gives them the best chance to effectively defend against these new type of attacks so what we're seeing is one we understand the um intent for the
• 35:00 - 35:30 organization two we understand security Delta from their intent to realize realization of that intent and then three to D this point when it's open running how do you Monitor and maintain this thing because um in in security circuit it's always been a cat and mouse game where as soon as You' got something solved it gets unsolved Again by by the the um black cats Etc who are out there and so what you need to do is establish your posture and then keep the set up to date H keep advancing and upgrading your
• 35:30 - 36:00 software and your Solutions and toolkit that you're using internally so you have the best live protection for your organization as AI evolves at PACE and are there different ways oh sorry did you want to add to that no I was just going to say and and to your point I think that you raised which is so true it's such a dynamic evolving space I think it's important for every organization to have folks kind of focusing on the new right there's nothing I think more deflating sometimes for an organization I talk to and they're building they're really excited about an AI workflow that they built
• 36:00 - 36:30 that's just been integrated into like a large platform right so they spent four or five months going through all these responsible processes doing the engineering pulling the data to then realize that within a subscription that they already have that's now kind of a table Stakes thing that's the world we're going to live in for a while I think you know there's going to be a lot of that which is great because everyone's kind of pushing us further and further into the frontier um but that's where I think if you're a large organization you have to kind of balance you know going after are the things that you have a clear Roi on as we established earlier understanding what
• 36:30 - 37:00 that emerging landscape's going to look like um and then being able to kind of pull in the right Partners to kind of get you moving quickly uh because it's not really going to be a do-it-yourself um experience for for most organizations and when organizations are evaluating those potential Partners like they will need to understand a little bit what what they're at seeking out there so are different technology Partners taking different approaches to AI security like is Calypso ai's approach into that INF layer and securing that particularly that's yeah that's exactly right so
• 37:00 - 37:30 um what you're doing is very very important but also is what you're not doing and so focusing on Tre specific areas that give the the best impact for an organization as they adopt AI is our critical concern so the inference lay so consuming generative AI models large small medium that's effectively where trps I come in to protect you and there are protections out of the box for most model providers if you asked you know um
• 37:30 - 38:00 a large language model to do something that's illegal or harmful or hurtful by default it will try to protect you and it will say I can't answer that for these reasons Etc so it does a really good job out of the box but you know Fortune 100 Fortune 500 companies they do a lot of Niche specific things that are far outside the remit of large language model providers in the training area to even fathom of how to protect organizations from so we allow you to define the best-in-class
• 38:00 - 38:30 um rules that everyone should abide by so no hurtful or harmful activity Etc but also the the big Last Mile bespoke step so in your jurisdiction or region what are the additional security controls you have to bespoke at on top to protect you and your customers in the offensive plane how do you um detect or adopt new models and to detect what problems are inherent in that model right from the get-go um and then if you're building and a lot of
• 38:30 - 39:00 organizations are models that have their own data inside that either rag or or fine-tuned how do you make sure that data isn't leaking or isn't um you know one of the the n and threats that there can't steal that information from that model and then lastly GRC when those three things combine you've got your you're running uh and adhering to the laws and jurisdictions in your in your um in the areas of business you operate in um you are are evaluating the best model with offensive
• 39:00 - 39:30 capabilities to make sure it's the right fit for your business and then you're setting up the right policies and controls to defend against threats to your customers and employees then you've got that trifect of security that 360 model that everybody um that's really taking AI seriously or building AI applications on top of needs to have so we've been talking a lot on the abstract and it's it's great when you can say what are the core principles people should be adhering to in these kind of things but practice can veryy very differently from the theory so what are some actual hard examples that you guys
• 39:30 - 40:00 have of maybe good use cases good examples that you can share with us today that's fair um I guess we better start making a real for everybody so um what's really interesting to me are these novel use cases that companies are discovering right by not just using AI to create efficiencies but brand new use cases that W impossible pre gen and so uh you know there's major global companies that there one such has created a u assistance tool for for
• 40:00 - 40:30 people to get advice on their cosmetic and products they're using that are fit for the problems they have Etc when you give a solution like that to the public um you're going to have three things happen you're going to hopefully have the intended use case where lots of people can comeing on anonymously ask questions that are really really personal to them and get really good advice leveraging Decades of experience from this company in cosmetic usage you're going to have secondly people who
• 40:30 - 41:00 just want to see the world burn and they're going to just Target that model and try to get it to say inappropriate things brand damage um Etc and then lastly and and this is the the I guess one of the most important ones you're going to have people who get accidentally bad advice right and so this is the risk of you've heard of hallucinations in models but you can also um have misunderstandings of what's being asked right and that happens between humans so certainly happen between a human and an ani assistant and
• 41:00 - 41:30 so this company um because they're really thoughtful about how they're adopting the say and they're trying to make sure the intended use case is the only one that happens they've put in place guard rails between the human being using this model and the model and those policies prevent the model from giving bad advice heartful advice and they also monitor that on a on a regular basis to make sure that nothing is into used dat as point from earlier it's not drifting in a Direction they don't want
• 41:30 - 42:00 it to go in and so that's all really critical and it's great to see companies getting ahead of that and doing that before they launch a product because they have an intent but they also understand that it could be used in an incorrect way as well what kind of examples have you come across SEL yeah I can think of two so so I'm more of the consumer facing one to kind of to riff off of what what jimu was talking about I've seen some organizations that have Digital Services for you know um helping advis job Seekers and and others kind of looking to extend their career um
• 42:00 - 42:30 creating a lot of generative AI enabled workloads where they can help create resumés and and do things like that um sometimes when we've when we've done testing on those types of platforms what we see is that a lot of the security guard rails are oftentimes put in place right in terms of monitoring sensitive data however you know the users is able some to manipulate the AI system to come out with with responses that maybe don't align with that core Services brand or or philosophy or things like that and so
• 42:30 - 43:00 you can create both a a real or perceived uh brand issue depending on how the chat bot's responding to you how it's framing some of the verbiage Etc and so that's a perfect example I think of where those guard rails can be more honed for other things outside of just kind of core security day leakage and and vulnerabilities and things of that nature um and then from more of an internal perspective right a lot of organizations are building different types of assistance right so it could be an assistance for your software
• 43:00 - 43:30 development team it could be an assistant for your operations team Etc and so many of my large clients have got a very significant backlog of data governance work that they've known that they wanted to do for many many years they just haven't kind of got around to it one of the kind of unintended consequences of AI is is really exposing a lot of that so the the lack of permissioning in some of those underlying data sets um some of the redundancy or like duplication of data and so what's happening is they're
• 43:30 - 44:00 building these really cool really powerful assistants and then surfacing a lot of fine grain Access Control problems and identity and access management control issues and things like that that they've had and so the the the question at that point for for the organization is do we turn this whole AI functionality off and and oftentimes that's that's a bit of an overreaction right and so being able to put in guardrails in place and say okay great we know we have either sensitive customer data in this bucket or perhaps we know we've got sensitive data related to an acquisition or maybe a historical
• 44:00 - 44:30 acquisition but you can actually put policies in place to be able to restrict and and validate that some of that's not being um you know being consumed either by a prompt or response in the model at that inference layer and it allows you to kind of move forward with that AI experiment moving that into production while you go and do some of that cleanup effort in the background right so they're just kind of a a customer facing example as well as more of like a back office example of that you know kind of real challenges that we're helping some
• 44:30 - 45:00 of our our clients work through thank you so much for both of you sharing your insights today I've learned a lot and it seems like even though it seems like it's a scary new world out there with AI and security the basic tenets of good security Practice still apply here bake it in early from the stairs and perhaps maybe you know you'll need to be a bit more proactive with it you may need to bring in some outside expertise to help with it but you know it's it's still security at its best is when it's done talk down considered from the start and
• 45:00 - 45:30 you know do things responsibly as well I'm sure that's a message we'd like to lead people with so thank you James thank you Daniel for all of your insights here today and if anybody wants more information on Trends and AI security I'm sure they can connect with you both on various platforms they can follow Calypso AI on LinkedIn or visit Calypso ai.com thanks Lenn thanks [Music]