Geeking out on AI PCs: Benefits, Breakthroughs and Beyond

Summary

Highlights

• AI PCs integrate central, graphics, and neural processing units to manage AI tasks locally 🌟.
• Dell is at the forefront of the AI PC market, providing powerful, efficient solutions 💼.
• Running AI workloads locally on AI PCs enhances privacy and security, while reducing costs 🚀.
• AI PCs support a decentralized AI future with improved user experiences through fast processing ⚡.
• 'Agentic' AI signifies a leap forward, with PCs playing a key role in this transformative journey 🤖.
• Delpro AI Studio by Dell offers robust tools to realize client-side AI effectively 🛠️.

Key Takeaways

• AI PCs are transforming computing by integrating multiple processing units for efficient, local AI tasks ⚙️.
• Running AI locally on AI PCs enhances privacy by keeping data on the device and reduces cloud costs 💼.
• Dell leads the AI PC market, offering innovative solutions like Delpro AI Studio to enhance AI on the client-side 🚀.
• AI PCs empower dynamic, decentralized AI systems, integrating seamlessly with existing infrastructures 🌐.
• The future of AI PCs involves more personalized and interconnected systems, advancing towards 'Agentic' AI 🤖.
• AI PCs are pivotal in the ongoing AI revolution, proving the PC's enduring impact on technology 🖥️.

Overview

Chapters

• 00:00 - 01:00: Introduction to AI PCs The introduction to AI PCs is presented by John Rose in the series AI Insights. The focus is on exploring AI PCs in greater depth, explaining what they are and how they are leveraged. An AI PC is not just a traditional PC but one that incorporates multiple types of processing elements, including a central processing unit (CPU) and other advanced components.
• 01:00 - 02:00: AI PCs in the Computing Environment The chapter discusses the evolution of computing hardware, featuring the transition and optimization of processing units such as the central processing unit (CPU), graphics processing unit (GPU), and neural processing unit (NPU) to efficiently run local AI tasks. This includes applications linked to both large and small language models and generative AI. The industry's awareness and consideration of this evolution are growing, reflected in positive feedback from customers acknowledging its significance.
• 02:00 - 03:00: First Generation AI PCs and Data Security The chapter discusses the role of AI PCs as integral parts of the computing infrastructure, alongside cloud, data center, and edge computing, in supporting advanced AI systems. It highlights Dell's significant position and leadership in the PC market.
• 03:00 - 04:00: Benefits of Local AI Processing The chapter titled 'Benefits of Local AI Processing' features a conversation between Sean and Mark from the CSG CTO division. Mark discusses the current state of AI Processing Units (AIPCs), highlighting that we are in the first generation of these technologies. Over the past year, these AIPCs have started to significantly benefit both users and enterprises.
• 04:00 - 05:00: Emergence and Applications of AI PCs The chapter titled 'Emergence and Applications of AI PCs' discusses the distinct features of AI PCs compared to cloud-based AI solutions. A core advantage of AI PCs is the ability to run AI workloads locally on the device using new processing units like NPUs (Neural Processing Units). This local processing capability enhances data security and privacy as personalized data remains on the device without being transmitted to the cloud, addressing significant security and privacy concerns.
• 05:00 - 06:00: Software and Hardware Developments The chapter titled 'Software and Hardware Developments' discusses the advantages of using cloud computing for AI models, notably cost savings and data security benefits. The speaker highlights how cloud usage reduces the need for token consumption and data storage, minimizing the risk of data exposure and misuse in training. Furthermore, the speaker explores metrics like 'time to first token' to understand performance in AI infrastructure systems.
• 06:00 - 07:00: AI PCs as Part of the Compute Pipeline In this chapter titled 'AI PCs as Part of the Compute Pipeline,' the focus is on the integration of AI PCs in enhancing user experience by speeding up computations to milliseconds. This performance improvement aligns with user expectations, illustrating the importance of not just performance but also data security and privacy. The chapter highlights the adaptability of AI systems and mentions applications like RAG (Recurrent Auto-regressive Generative), which users find invaluable for managing and ingesting large volumes of documents and collateral.
• 07:00 - 08:00: Distributed AI and Future Topologies The chapter titled 'Distributed AI and Future Topologies' discusses the evolution of technology enabling individuals to perform advanced tasks such as summarizing, transcribing, and searching through PowerPoints directly on their devices. Initially driven by silicon diversity and advancements in chipsets, the vision of accessible AI processing (AIPC) has moved from theoretical to real-world application. The chapter emphasizes the maturation of this technology and its practical implementation.
• 08:00 - 09:00: The Future with Agentic Technology The chapter discusses the evolving landscape of agentic technology, focusing on the distribution of AI compute power. It challenges the myth that there are no applicable uses for this technology, suggesting that while hardware needed to come first, software development is catching up. Various independent software vendors (ISVs) are beginning to create significant applications, indicating progress in the sector.
• 09:00 - 10:00: The PC's Role in the AI Cycle The chapter titled 'The PC's Role in the AI Cycle' explores how companies like CrowdStrike are integrating Neural Processing Units (NPUs) to enhance their security suites, achieving remarkable performance and extending battery life. It highlights the role of AI in communication apps like Zoom and Teams, particularly with features like Copilot that leverage AI for live transcription and more. Additionally, the chapter discusses the impact of AI in productivity applications, especially in content creation where generative AI is used for producing video and images, showcasing the AI's expanding role in various domains.
• 10:00 - 11:00: Conclusion and Takeaways The chapter discusses the rapid development and integration of technology, specifically focusing on the capabilities provided by NPUs on devices. It highlights the surprising progress of 200 Independent Software Vendors (ISVs) who have developed applications utilizing these capabilities. This reflects a significant advancement from initial planning phases to practical implementation. Moreover, the chapter provides insight into the synergy between hardware and software, underlining the continuous evolution of technology from an engineer's perspective.

Geeking out on AI PCs: Benefits, Breakthroughs and Beyond Transcription

• 00:00 - 00:30 [Music] So, welcome to AI Insights with John Rose. Today, we're going to talk about AI PCs. Uh, you know, hopefully, uh, you've heard of the term, but we want to go a little deeper into into what they are, how people are using them. Uh, you know, as you know, an AI PC, or hopefully you know, it it isn't just a PC as it existed in the past. It's a PC that has multiple types of processing elements in it. A central processing
• 00:30 - 01:00 unit, a graphics processing unit, a neural processing unit. And and really why that evolution occurred is because it really is the evolution of optimizing the PC hardware to in an energyefficient way be able to run local AI outcomes, specifically some of the modern outcomes tied to large language models, small language models, generative AI, etc. Um, we are, I think, now in the in the consciousness of the the industry. People know the term. They're starting to think about it. We're seeing pretty good feedback from customers that they're realizing that it is a piece of
• 01:00 - 01:30 the overall computing environment. In fact, I would argue that the AIPC when we think about AI compute is actually just another piece of the infrastructure. You have compute in the cloud, you have compute in the your data centers, compute at the edge, and you have absolutely compute on the device. And all of them collectively give you the platform to do most of the advanced AI systems that you're going to deal with. Uh the good news from a Dell perspective is you know we we're pretty significant player in the PC world and and at this point you know I think we have emerged as as really the the the thought leader and at the front of the
• 01:30 - 02:00 IIPC market. So I I wanted to invite my friend Mark who works in our our CSG CTO division who's been uh at the bleeding edge of this for now a while to join me and and kind of have a conversation. You know Mark, what what's you know what do people need to know about AIPCs? What's what's new? How what can you share? Yeah, you bet. Thanks Sean. Um, and and as you know, we're kind of in that first generation of AIPCs today. It's been a pretty remarkable year, a little longer than a year maybe that we've had these things in the market, but these things are already starting to benefit users and enterprises today. Uh, some of the
• 02:00 - 02:30 core tenants of an AIPC that maybe differentiated a little bit of from uh cloud-based AI is that, hey, I finally got this device where I can run those workloads there on the device on those new processing units. The the NPU kind of being the new kid on the block. um data stays local. All my data that's personalized, I maybe I don't want to transit to the cloud or or distribute that anywhere. I want it there on my device. So that solves a lot of security problems, a lot of privacy problems for us. Uh and then in not transiting to
• 02:30 - 03:00 cloud, as you know, um that means some cost savings because I'm not consuming tokens per second for that AI model or not storing the data there. I'm not um maybe um exposing it to capture and maybe use for training, those types of things. So, we're seeing a lot of benefits to the AIPC. Yeah. One of the things I'm I'm kind of digging into is I I saw some stats, you know, they were kind of two different worlds. It was it was the stat of time to first token. Oh, yeah. And when you see it in a infrastructure AI system, it can be a
• 03:00 - 03:30 couple of seconds. When you see the numbers on an AIPC, it's like milliseconds. I mean, and you know, it makes sense. It's it's localized. It you're and and that that actually does correlate to user experience. That's exactly right. some of those performance characteristics are are paramount as well in addition to all the data security privacy things. It's really great to have it right there on the system and it can adapt to you too. So you know um one of the our favorite applications is rag. Everybody wants to use a rag. They want to ingest all of their collateral, their documents, their
• 03:30 - 04:00 u their powerpoints. They and but they want to search through that. They want to summarize it. They want to transcribe it. There's all these things they want to do and now they can do that right there on their own device. Yeah. Absolutely. Now, now like many technologies, this one kind of emerged out of the out of the market world. It was before we had products, we had this vision of the AIPC. It was mostly driven by silicon diversity, new chipsets coming. Um, we're now this is now real. And, you know, is why, you know, while it's a new technology, it's real. We I
• 04:00 - 04:30 think there's a good case that I think we're seeing a pretty good long-term plan that compute's going to be distributed. We're going to need AI compute everywhere. But why now? Why why should customers think about using these? There's kind of a myth that there aren't any applications. People don't know what to do with it. Kind of let's demystify that. Yeah, it's kind of funny. Hardware and software has always been that kind of chicken and egg thing, right? Um but but the hardware had to come first. So we have this first generation of AIPCs that's out there. Um but but we're starting to really see the the software start to develop, right? Um on on one side you kind of have ISVS that are out there. We've got great
• 04:30 - 05:00 partners like CrowdStrike who are adopting the NPU use uh in their security suite seeing phenomenal performance gains at really really low power. So that extends your battery life. Um, we've got all the communication apps. You got Zoom. You've got Teams with Copilot in it. You probably experienced some of those yourself. Uh, really interesting applications of AI for transcription and things like that that are happening live. Uh, and then we've got productivity apps um in the creator domain where you're generating video or maybe you're generating images and you're using all the generative AI
• 05:00 - 05:30 capabilities right there natively on the device. Yeah, I think um I I actually, you know, was surprised. I remember you showed me a list probably four or five months ago of I think it was 200 different ISVS that had you know actually delivered you know applications that were touching the NPU that's pretty exciting the other thing that surprised me this was very early when I was kind of getting when this term materialized and we started putting in our road map this going back well over a year um you know I'm a I'm an engineer and I you know hardware is one piece software is
• 05:30 - 06:00 another piece the stack matters and what surprised me was the kind of orderliness of how the AIPC ecosystem was forming. Like we know that in the infrastructure side, we have this competition about CUDA and Sickle and all these different frameworks that are kind of tied to specific semiconductors, but in the AIPC world, we saw things like Onyx and WebN and a whole bunch of OpenVO and a bunch of framework show up that kind of everybody was playing nice. They weren't trying to create silos and I I think I mean what's your view? Has that helped us move faster? It has. It has. We we
• 06:00 - 06:30 found a lot of that uh kind of homogeneation has been happening with like you said frameworks like Onyx and Onyx runtime. You still have the vertically oriented segments because sometimes you want to get into performance. I know like especially if you're in the gaming world or something like that. It's really it's all about performance and we have kind of the vertical stacks with Q&N and with Open VO and with CUDA and and Tensor Runtime. Um but we're also doing things to help that along and some of that uniformity has helped us do that. As you know, we've got a new product out called Delpro AI Studio. Uh that really is the
• 06:30 - 07:00 way that we realize AI on client. Uh and that's been a big boom to leverage some of those underpinning run times to be able to support that capability. Yeah. What's cool about that is, you know, while there's the ISVS that deliver kind of a turnkey thing that use these frameworks, the real opportunity for an AIPC and an enterprise is thinking about it as kind of the the last step in your compute pipeline. And in many cases, you're building maybe you're deploying an agentic system or you're building some kind of system and you realize,
• 07:00 - 07:30 wouldn't it be great if some of this processing could happen close to the the user? Well, you know, if you don't have any tools to make that happen, it's kind of a great theory, but things like Delli, Dell ProAI Studio, and some of the tools that we're producing actually make it easy for you to kind of think about actually dividing up the work. I mean, absolutely. Yeah, that you're right because u your PC, it's really interesting. It's kind of a sensor hub, right? I mean, your keyboard's there, your mouse is there, but your your camera, your mics, all of those things, all of your peripherals, and you can adapt and adopt and create really rich
• 07:30 - 08:00 context based on all the telemetry that's coming from those things. And then it gets really interesting when you get into hybrid scenarios where now maybe I'm synthesizing something locally, working with something off client to be able to do a lot more from an intelligence perspective. Yeah. And I mean, if you had to bet on what the end topology looks like, the AI future is not a centralized environment. It's a decentralized distributed environment because candidly that's where the data and the users are. No, they're not centralized. They're not in one place. They're it's a continuum of services and capabilities that need to work together.
• 08:00 - 08:30 Which kind of brings me to the last topic which is hey this is everything we just talked about was kind of the gen one a generative AI world. Chat bots, rag, there's something coming that's a lot more interesting and actually much more disruptive and that's Agentic. Yeah. Yeah. Agentic is kind of the word of the year I think. And and it's really interesting. It it it is gen one of these devices and you can only imagine that they're going to get better as the software stacks improve, as model efficiency improves and the pipelines improve and the hardware to support that
• 08:30 - 09:00 all improves. And that's what we're starting to see. And with that hardware improvement, with that software improvement, we're expecting those agents to come down onto the device for a personalized experience, a secure experience. And I can imagine, you know, agents on client working with a whole uh army of agents off client as well. Yeah, that that kind of ensemble approach. Now, I've talked about agentic a lot and you know, look, the real value of an agent is not a singular agent. It's when agents start to work with each other to create composite digital skills. More importantly, when they start to work
• 09:00 - 09:30 across domains that you know your agent in Dell can work with one of your supplers agents or can become a new API between you and your suppliers, your customers, the possibilities are kind of endless. But the question, okay, where do I run them? Do they all have to be captive in a data center on the other side of the world or or is this agent whose sole job is to make my life better, should it be with me? Should it have access to my private information? Should it be under my control? And without things like an AIPC to do that, it's it's pretty problematic to make that kind of experience happen. Yeah,
• 09:30 - 10:00 exactly. And and I think some of our customers are saying that too. Uh, as you know, we have a really diverse customer base. Some of them are all about, hey, the data needs to stay on that device. that device. I I I might I might not want it to be connected. When I do that, how do I take that with me? How do I take that intelligence? It's there. It's available. It's adapted. It's got access to all the information that's there, but it's agentic, right? And so, it's sitting there solving problems with me as a companion uh helping me throughout my day. Yeah, Mark. I mean, I I I think it's it's a a super exciting technology cycle we're
• 10:00 - 10:30 in, whether it's AIPCs or, you know, and maybe one last thing that, you know, has happened this year is the AIPC cycle. Most people think it's about trending towards highly power efficient mobile devices, but there's another angle to this, which is, you know, at GTC this year, we announced a couple of products that are at the other end of the spectrum, but still PCs. Maybe you can talk about those. Yep. We've got uh the GB10 and GB300, Grace Blackwell 10 and 300 in the lineup now. Um pretty phenomenal devices. Looking forward to
• 10:30 - 11:00 seeing them in the hands of data scientists to, you know, tinker, use kind of as the anvil to hammer out all the goodness that they're going to deploy up into the cloud, but also use them as inference devices themselves. Uh small and medium business can really apply these things and make use of them for a variety of applications. Yeah, when I when I saw the initial specs for these systems, I I just I got very excited because I'm like, "Oo, I can put a very big model or several very big models into production on that system. I can experiment with them. I can develop on them, but I can also run them and I
• 11:00 - 11:30 can do it at extreme performance levels and I can do it in my personal space. I can do it where I want it in my home, in my office, at an edge node." It it's it just tells you, you know, I mean, you know, Michael has always said, you know, it's a a little too early to declare the death of the PC. It for a product that's supposed to been dead several times. No, it has not died. In fact, it's becoming more and more impactful. And in the AI cycle, we're seeing like the it as a platform for AI extend in in several directions simultaneously. Yeah. again
• 11:30 - 12:00 that hybrid conversation we we definitely are thinking about you know the PC world in terms of not just the device itself but the ecosystem around the device and how we can leverage all of those things and we it's got a long life to come for sure yeah so so you know I mean the the discussion hopefully this is useful to kind of understand the state of of the AIPC couple of takeaways the first is look these technologies are new but even if you don't build anything yourself even if you don't have any new AI project in production which we can have a different discussion you should have those but even if you don't there
• 12:00 - 12:30 are hundreds of ISPs that are actually using these systems to operate in a more power efficient more effective way. So the technology is very real whether you see it or not. So having an AIPC just gives you an advantage but it also sets you up to have a platform to really extend your AI outcomes as you develop them to where the data is to where the users are and so I think uh you know interesting time but you know sounds pretty real. Yeah really exciting. Good. Thanks Mark. Great to see you. Appreciate it. [Music]