Agentic apps with Flutter | Observable Flutter #59

Summary

In this episode of Observable Flutter, titled "Agentic Apps with Flutter," host Craig Benz explores the concept of building agentic applications using Flutter. Joined by Andrew Brogden, they delve into integrating AI capabilities, specifically through the Vertex AI and Firebase platforms, to craft applications that utilize large language models to enhance user interactions. Their discussion includes a hands-on demonstration of creating an app that turns news headlines into engaging phone wallpapers, showcasing the potential of integrating AI into Flutter apps to deliver unique user experiences.

Highlights

• Craig Benz returns refreshed to host another engaging episode of Observable Flutter. 🎉
• Andrew Brogden joins to discuss AI integration in Flutter, specifically through building agentic apps. 🤝
• The discussion leads to a hands-on project: turning news headlines into phone wallpapers with AI. 📰📱
• The use of Vertex AI and Firebase to securely handle AI operations within Flutter applications is explored. 🔒
• Participants and viewers are inspired to explore new ways to implement AI in their own Flutter projects. 🚀

Key Takeaways

• Agentic apps combine AI with specific goals and tools in Flutter applications. 🤖
• The concept of 'agentic' involves AI reasoning capabilities connected with an interactive toolkit. 🎒
• You can streamline AI integrations using Vertex AI and Firebase with Flutter for secure and efficient development. 🔐
• Brett Morgan's code lab provides an excellent starting point for learning how to implement AI in Flutter apps. 📚
• Testing and refining prompts is crucial for effective AI response generation. 📈

Overview

Craig Benz warmly welcomes viewers back to Observable Flutter after a brief hiatus, eager to delve into the innovative intersection of Flutter and agentic AI applications. With special guest Andrew Brogden, a key figure in Flutter developer relations, they embark on an exploration of integrating AI in Flutter apps using Vertex AI and Firebase.

Andrew and Craig discuss the notion of 'agentic apps,' where AI capabilities like large language models are paired with a toolkit allowing them to interact with the app environment to meet specific goals. This involves using pre-designed code labs and tools to streamline the AI integration process in a fun, interactive way.

The episode features a practical attempt to create a unique app that turns news headlines into dynamic phone wallpapers using AI. By leveraging the guiding structure from Brett Morgan’s code lab, Craig and Andrew exemplify how developers can quickly prototype engaging, AI-enhanced experiences in Flutter. This episode not only informs but also inspires viewers to experiment with AI in their Flutter projects.

Chapters

• 00:00 - 00:30: Introduction In this introductory episode of Observable Flutter, host Craig Benz returns after a short break to express gratitude to his audience for their support during his time away. He shares his refreshed energy and enthusiasm to continue the series. The episode starts with welcoming music and sets the stage for the upcoming content.
• 00:30 - 03:00: Introduction and Guest Introduction The chapter begins with some technical difficulties as the speaker adjusts to the setup, specifically dealing with audio playback issues. Despite these minor challenges, the speaker quickly resolves the problem and expresses apologies for the delay, indicating a certain level of rustiness possibly due to a hiatus or inexperience with the setup. Once ready, the speaker shifts the focus to the nature of the Flutter community, emphasizing values of kindness, warmth, and mutual learning. The introduction sets a tone of collaboration and positive interaction, aligning with the community's standards of conduct. The setup concludes with the introduction of a guest, although the name and specifics about the guest are not disclosed in this segment of the transcript.
• 03:00 - 08:00: Discussion about Agentic Apps The chapter titled 'Discussion about Agentic Apps' introduces Andrew Brdton, a prominent member from Flutter developer relations and currently the boss of the host. The purpose of the discussion is to explore building Agentic apps in Flutter. Andrew opens with a humorous welcome, mentioning his 'storage room workshop'.
• 08:00 - 12:00: Introduction to Vertex AI and Firebase The chapter begins with an introduction to Tim, a member of the Flutter team since early 2018. He leads the developer relations team, which includes engineers, writers, and program managers, focused on collaborative projects.
• 12:00 - 17:30: Code Lab Explanation In this chapter, the focus is on exploring a newly published code lab authored by Brett Morgan. The discussion revolves around building 'gentic apps.' The chapter begins with excitement and anticipation for engaging with the code and tools that will be used throughout the session. It sets the stage for understanding the concept of 'gentic apps' and developing them with the guidance provided in the code lab.
• 17:30 - 22:30: System Prompt Creation This chapter focuses on the creation of system prompts, featuring a learning campaign about building agentic applications using Flutter, in collaboration with Angular and Firebase teams. The stream primarily discusses Flutter, with Firebase being used for backend development. Additional resources and live streams from various teams are mentioned for both Flutter and Angular developers.
• 22:30 - 27:00: Function Calls and Tools The chapter discusses the evolving terminology at Google, particularly around the term 'agent.' It highlights the difficulty in keeping up with the changing terms and mentions that the term 'agent' has now settled into referring to a combination of a reasoning capability, like a large language model, paired with a specific goal.
• 27:00 - 33:00: Debugging and Testing The chapter discusses the concept of tools that allow interaction with the external environment to achieve a specific goal. It mentions a demo at Google Cloud Next where a tool was used to ask users for information. The chapter explores various methods such as prompting a user for UI, accessing device APIs, or making network calls. These methods are used to gather necessary information and reach conclusions.
• 33:00 - 35:00: Project Ideas and Future Plans The chapter discusses innovative project ideas and future plans, focusing on the integration of calendar functionalities into applications. It highlights the potential of using a Language Learning Model (LLM) combined with specific programming to create tools that can interact with calendar APIs to make events. The concept is to develop applications, referred to as 'agentic apps,' which leverage agents to perform tasks that were previously challenging or impossible. The discussion sets the stage for exploring various methods to implement such functionalities, with a special focus on a particular approach being highlighted in the chapter.
• 35:00 - 39:00: Q&A Session The chapter titled 'Q&A Session' discusses how AI agents are transforming workflows by providing dynamic user guidance that adapts to changing conditions. These new AI agents can combine data from multiple sources, similar to the concept of a mashup, to enhance the user experience in ways not possible before. The session highlights the evolving capabilities of AI agents and their impact on managing experiences and tasks efficiently.
• 39:00 - 40:12: Conclusion and Closing Remarks In the section titled 'Conclusion and Closing Remarks', the speaker reflects on the capabilities of technology, particularly focusing on the mashup scene around 2005-2006. They illustrate this with an example of combining Google Maps and data feeds, such as locating all the sheep in Scotland. The speaker emphasizes the potential of Large Language Models (LLMs) to perform similar tasks and provides a brief explanation of agents and agent-driven processes, indicating that they will be examined further. It concludes on a note that engages the audience with a rhetorical question.

Agentic apps with Flutter | Observable Flutter #59 Transcription

• 00:00 - 00:30 [Music] Hello everyone and welcome to another episode of Observable Flutter. My name is Craig Benz and I'm your host. And today I both owe you all a thank you for allowing me to take a little time away, recharge my batteries. I'm feeling really good. I'm excited to be back. I'm also going to need to mute this video
• 00:30 - 01:00 that's playing in another window. I'm hearing myself talk. I'm telling you, I'm rusty. Where is this? Oh, here we go. Okay, muted. I'm sorry. Now I'm ready to actually proceed. So, as you may recall, this is the Flutter community. We really want to treat each other as kindly and warmly as we can. So, we're here all to learn together and let's just try to keep up that wonderful spirit. Today's guest is
• 01:00 - 01:30 none other than Andrew Brdton, one of the the originalist OSGs from Flutter developer relations and once a person in my role, now my boss, and he's here to help me today talk about building Agentic apps in Flutter. So, Andrew, welcome to Observable Flutter. Would you like to say anything else to introduce yourself? Yeah, welcome to my uh storage roomworkshop. Um, yeah, I'm Andrew. I've
• 01:30 - 02:00 been on the Flutter team since uh let's see, I snuck my way on at the beginning of 2018 and as Craig said, I lead the developer relations team. So, I got a bunch of engineers and writers and program managers and stuff like that. And we all work together to make things like this. Um, and I'm It's also been a while for me to appear on a stream. We used to do the Boring Show way back in the day. Um, but I haven't been on a live stream in in a bit. So, this is this is interesting for me. You want to We'll see how this uh goes. But I'm
• 02:00 - 02:30 excited. I am I am excited to have an excuse to sit down and code with some of these tools that we're going to be trying today and take a look at a code lab that was recently published by somebody on the team uh Brett Morgan. So, I'm excited about that. It should be some should be fun. Yeah, absolutely. So today uh as you mentioned we're going to go through a code lab that Brett has written and then we're going to and that code lab is about building a gentic apps. Would you mind giving folks a primer on what this term is because I expect for a lot of
• 02:30 - 03:00 people it's pretty new. Yeah, absolutely. And that's that's part of the reason I wanted to be here today and it's also part of the reason that we're um we're going to be doing a sort of learning campaign over the next few weeks. uh uh it's about building agentic apps with both Flutter and we're partnering with the Angular and Firebase teams. So obviously this is a Flutter stream. We'll be talking about Flutter uh but Firebase will be used for the back end and if you have friends who are Angular developers or maybe you also use Angular that's an option too. We'll have some new resources. We'll have some live streams like this one from various teams
• 03:00 - 03:30 and uh and we'll be online answering questions and things like that. So I'm super excited about that. Um, but let's talk about like terminology is is so, you know, uh, even even internally I've seen the terms evolve at Google and getting a handle on them has been tricky. So, let me give you a couple ones that I've I've seen stick. So, the term for agent, uh, it's changed, but I think they've sort of landed on something now. Um, an agent is the combination of a a reasoning capability like a large language model with a goal
• 03:30 - 04:00 and then tools that allow it to interact with the outside world and accomplish the goal. Um, and so in this case, a tool could be the ability to ask the user for information. We recently had a demo at Google Cloud Next that did that. It could prompt a user for UI by working with the app um to get more information to come to a conclusion. It could be a accessing device APIs or making a network call or something like that. Getting more getting information that it needs. Um, if you had an agent that was
• 04:00 - 04:30 placing calendar events for you, you know, the ability to call into a calendar API and actually make an event, that would be a tool that you could give the agent. Things like that. So, it's really just an LLM with some code around it that gives it a purpose and a way of achieving the purpose. Um, and there's a number of ways to do that. We'll be talking about one one today. Uh, and an agentic app, since I also threw that term at people a minute ago, an agentic app is just an app that uses an agent to do a thing that you couldn't have done before because it was like too hard or
• 04:30 - 05:00 too expensive. Um, and so there are things that you can do with an agent guiding the experience um, that you might not have been able to do before. You can, you know, there's been AI forever, but this particular these these this new breed of agents can do things like guide a workflow with a user that changes every time based on what's going on. for example, um they can combine data from multiple sources the way that um the way that we used to do with like a mashup if and I'm age I'm
• 05:00 - 05:30 dating myself there by by you know if anybody remembers the like 2005 2006 mashup scene where people would be like I want a map with all the sheep in Scotland and like you could do that by combining Google maps with like the sheep data feed that the Scottish government had. Um you can use LLMs to do things like that. Um and so so yeah, that is my 60-cond spiel on agents and Agentic things and we're going to look at one today. Nice. Yeah. Who amongst us has not at one
• 05:30 - 06:00 point or another wanted a map of all the sheep in Scotland? I mean, yeah, come on. Was a hit on dig way back in the day? Yeah, probably. All right. So, I am going to share my screen so we can get started here. So, as teased previously, we're going to begin this by looking at a code lab that was written by a teammate of both Andrews and mine, Brett Morgan. He's on Flutter developer
• 06:00 - 06:30 relations. If you've not heard of him and you're like, who's Brett Morgan? A little more back house, go to events and you'll meet him there. And the code lab itself is called build a Gemini powered Flutter app. And it is pretty awesome, I got to say. Um, now I I'm not sure whether or not we think it makes sense to like read the whole thing step by step, do every single step. We absolutely can,
• 06:30 - 07:00 Andrew. I've set the whole thing up all the way to the end, so I know all the Google APIs for me are turned on and Firebase is ready to go and all that kind of stuff. Honestly, folks at home should do this code lab. The hardest part is configuring your uh your project in Firebase and in the Google Cloud Console, which isn't terribly hard either, especially with Flutterfire Configure. So the whole code lab is very approachable and really shows some very
• 07:00 - 07:30 cool behavior that is obviously just a starting point. You know, picking a color doesn't necessarily warrant a whole app, but as the uh as Andrew is kind of teasing earlier, you know, there's a lot of a lot richer stuff that you can do with this kind of functionality. So Andrew, do you think we should go through the whole thing or should we just kind of like read it and and talk about the code because those things
• 07:30 - 08:00 might actually be functionally the same? I it you know it depends on what you um how you would normally want to code this. So I think reading and going through the code lab like we have other other videos where people are like let me walk you through a code lab you know and we'll probably have one for this one too. Um, I think looking at the pieces of this app that Brett guides you, uh, guides readers through creating is is the most important thing. Um, because it's designed, um, as part of the
• 08:00 - 08:30 campaign, the building agentic apps campaign, what we're going to do is offer four different sort of starter kits that people can clone and then modify and play with um, and just sort of put their ingenuity to work. You know, we talked about how um, yeah, excellent. Thank you. That's that's the campaign. You can go register for it right now. The only thing you're committing to by registering is getting more emails from us. You'll get four or five emails during the course of the campaign. That's all you're signing up for. So, don't uh uh don't hesitate. Um
• 08:30 - 09:00 but the starter kits are designed to give people the structure of an application that's agentic and then um and then, you know, take a piece of code out, put a piece of code in. So, now the agent does something else, you know. And so this is the one that Brett's made for it for the campaign. And so the pieces of this I think are very good to talk about like where do the how do you define one of these functions, one of these tools that you give the the agent? Where is the prompt that uh tells the agent what it's supposed to do, right? These are these are some interesting
• 09:00 - 09:30 things. Um what you know I don't think we need to sort of go exactly through the whole thing line by line. Um there's a reason the code lab's published is so people can do that. So all right, we're vibing then. That's good. Okay. So, let's I think I will step through each thing. We're not going to actually make all the edits like Andrew was just saying, but at each step going to kind of scroll down the page and highlight what is worth having context on there.
• 09:30 - 10:00 Again, extremely this says 60 minutes. Probably 20 of it is fiddling in the Google Cloud Console and then the rest is just copying and pasting code and like thinking about it and realizing like, oh, that's neat. And so it's it's super super approachable as all of the code labs are in my opinion. So this uh if you've done one of the Flutter code labs before, you know the the first couple steps here are actually just like hey this is what you're doing. So we can also skip this I
• 10:00 - 10:30 think for our purposes. This is just kind of laying some plumbing some foundational stuff around um the the Flutter architecture. You actually do learn quite a bit also about just generic Flutter development and some really good best practices for how to separate your concerns and whatnot. And step two is laying a clean foundation for all of that that isn't actually important for this stream's purposes of learning about the different AI tools. So I'm going to skip this one as well. Real quick, want to talk about Vert.ex
• 10:30 - 11:00 on Firebase and what it is? Oh, sure. Yeah. Uh I would invite you to summarize for folks. Cool. Yeah. Um so there's a few different there's a few there's a number of of products that Google offers right now for developers to build agents. One of the easiest for Flutters to get started for Flutter developer Flutter developers to get started with um is Vert.ex AI and Firebase. Uh and that's the one that's going to be shown in this uh code lab. Um Vert.exi XAI and Firebase allows you to access Gemini uh
• 11:00 - 11:30 securely through Firebase. So you don't have to like bake an API key for for the apps, you know, the AI studio into your um into your app or anything like that. You get all the stuff that normally comes with Firebase and you can build your agent and it executes in the app using Dart code. So you don't have to worry about learning a different language or anything like that. you can use the dark code you already know and enjoy to build the code around the LLM that sort of turns it into an agent. Um,
• 11:30 - 12:00 and for that reason, I find it a very a very easy first step into this world and why I think it's a great choice for this code lab. All right, thank you, Craig. That was my spiel about Vert.ex AI and Firebase. You nicely ended that right as I was taking a sip, which left us with an awkward gap there. Uh okay so this part is more about setting up Firebase again configuring I've already done these steps they do take a little bit of time some oh yeah entitlements
• 12:00 - 12:30 all this kind of stuff blah blah blah blah this is probably where things start to get interesting so I'm going to stop and and focus on the create the Gemini model provider step so this code lab uses riverpod which I know a lot of folks in the chat are ardent River Pod fans, Brett is is one of you. And the the foundations that Brett has you lay out here start in this file lib
• 12:30 - 13:00 providers gemini.dart. And some of this is going to be pretty uh familiar, especially if you've done Firebase stuff before. Like if you ran Flutterf Fire configure and got all of your different platform configurations, then you know that you can just kind of use this uh current platform cheat code to immediately initialize your app and all of the API keys and everything are sitting in that file over there. And then there is a gemini model class
• 13:00 - 13:30 and we can see here that this or function sorry and we can see that this is a riverpod thing because it takes the ref and it's going to pull it out of the riverpod infrastructure and ultimately is uh well actually so what it pulls out of the infrastructure is just that firebase is initialized and this is the code that actually starts to get you doing firebase for a XAI things and we'll see at the top here of
• 13:30 - 14:00 course that library is imported and you pass in the model as a string name and all of these names are available in the documentation for the Firebase Vertex AI library and they're also just the string names that you would see in Google AI studio when you choose a library uh when you choose a model here you can just select you know this bottom string here. Pass that into your code and as long as that is one that is
• 14:00 - 14:30 available vertex firebase vertexi like maybe they aren't all uh then it will work and all of the ones that are available are listed in the documentation. And then we see here some chat session stuff that goes back to a previous step where Brett was laying out that that nice infrastructure to have a a clean implementation in this Flutter app. So, and then there's implement the chat service. I don't think I'm going to spend much time worrying about this. This is good to think about. Again,
• 14:30 - 15:00 especially if you're newer with Flutter. This is some really nice Riverp Pod tutorial and just Flutter architecture tutorial stuff here, but I'm going to skip over it for now. And again, more Riverp Pod usage because Riverpod has a lot of magic code generation. So, you can get that out of the way. Uh then in the main.dart dart file. This mostly I believe connects the it connects both the UI because in this
• 15:00 - 15:30 code lab Brett has gone through all the trouble of not making us think about the UI at all is literally published to pub.dev a package for the UI for this code lab unlikely to be uh a dependency in many published Flutter apps but it's a very cool use of you know pub.dev dev to simplify this code lab I think and then we're ultimately setting up here some you know turning on that UI with this main screen widget and then also connecting some of this infrastructure
• 15:30 - 16:00 to talk to Firebase AI with riverpod and whatnot setting that up with uh some of these callbacks and whatnot so Andrew any commentary so far before we move on to the next screen no I think you know um one of the things you can get out of the sort of section of the code lab is just how to talk to a model and what the sort of um how to deal with that sort of back and forth that you have of managing the state of a conversation with the model which in this case um Brett is
• 16:00 - 16:30 building into the UI that you can see it you don't necessarily have to do that um but in some cases where you're where you're working with a model to get to a particular result seeing sort of what it's coming up with as as you're progressing through a workflow can be very handy. So now I think the interesting thing here is sort of the state management of it and also where he sets up with the model in the first place. Yeah, I'm uh I was just reading some chat here. I have some good questions coming in. Wouldn't an enum be better than a string name for Gemini
• 16:30 - 17:00 models? Yeah. Is except that because this is a published package, if it took an enum, the minute Google releases a new package, the library wouldn't be able to talk to it without updating. So, if you're doing something like this in one of your apps and you control everything, then yes, I would say use an enum. But because this package wants to allow developer freedom
• 17:00 - 17:30 as Google iterates on model revs, I think a string unfortunately almost makes sense. Any thoughts you have there, Andrew? Uh only that they're unlikely to deprecate any of the strings, you know. So once you have it right, I think you know at a certain point you'll you'll a model might might fall off the list or something like that, but uh you'll get plenty of warning there. And so, um, I think defining your own enum if you want to use one is a good idea.
• 17:30 - 18:00 Yeah, true. If you're that that's a good point. If you're when you're writing your own app, even if I was thinking about a different scenario where you're controlling everything, but yeah, in your own app, you could have an enum and and not pass in the string. That would be uh totally valid. Okay. So, now there's some other, by the way, I've starred some good questions we're going to get back to later. So, if you think, hey, I asked a good question, you probably did. and I I might just have it start. Okay, moving on to step four of the code lab. So, this
• 18:00 - 18:30 is I think a pretty interesting step in terms of a within a Flutter app, how would you architect some clean uh like how would you cleanly design and and where would you put the stuff for a good prompt? And then also just what goes into a good prompt because when I read this code lab I was really surprised at how long the system prompt
• 18:30 - 19:00 is because remember the the functionality of this app is just you describe a color the app well sorry Gemini comes up with a color and then the app shows the color that's it and the system prompt for this ends up being quite long which uh I found interesting myself. So we see it here in this system prompt. It's just in a markdown file that later some Dart code will will actually load in and and uh supply this string from. But it is I I think
• 19:00 - 19:30 interesting just taking a second because whenever I try to do something with an LLM, I never write a prompt this long. So that may be behind why I don't get perfect outcomes all the time. So you see here it it's telling the LLM right what you are. you're good at this. I think a lot of us have heard that, you know, giving the LLM a pep talk almost makes it do better, which is funny. Um, but you know, telling it that it's a color expert and telling it what its job is and uh telling it how the
• 19:30 - 20:00 response should come out, that's all in the first sentence. And then there's going to be so much more extrapolation into all of those concepts. So your capabilities, you are knowledgeable about color, color theories, how to translate natural language description into specific RGB values, you know, I would just say to someone, I would say to the all like make me a color for such and such and like hope it figures out that it should know how to do all these things, but this actually helps it perform better by
• 20:00 - 20:30 kind of pulling it in the direction of of these concepts when users describe a color. So here Brett is really being prescriptive on exactly how the LLM should behave and how it should respond. Um actually no that's what comes next. How to respond to users. So this is actually just the thinking that it should do. So analyze what they're talking about determine uh and then also we talk about the different color values here between 0 and one. So not 0 to 255 which would have been a reasonable uh
• 20:30 - 21:00 assumption for the LLM to make but that's not the one we're going with. blah blah blah all the way on down. And I I use this I I encountered this one. It says somewhere if Oh yeah, down here. If descriptions are unclear, if a color description is ambiguous or unclear, please ask these are clarifying questions one at a time. So, I was just being dumb earlier, actually was this morning, and I said something like, uh, a color blue like the belly of a blue dragon and red like the sun setting over
• 21:00 - 21:30 a small town, which makes no sense. And it came back because Brett asked it to, presumably is why. And it was like, hey, um, between the blue and the red, which one should be more important? And I was like, I don't know. I guess the red. And then it actually gave me a color that I found really, really beautiful. And I was like, I do see some blue under that red. Like, wow. Incredible. Uh, that's Brett's magic phrase, by the way. Ask clarifying questions one at a time. Oh,
• 21:30 - 22:00 really? And in he's been working he's been working with this stuff for months and months now. And that when he when he stumbled onto that recommendation from somebody um he's like this this made it it was like a light switch going on for him. He's putting it in everything now because it's like if the model determines that it's not able to give a high quality response. It gives it a sort of an out to be like I don't have enough information to to do the thing that you asked me to do. So I can instead of giving you garbage or some
• 22:00 - 22:30 kind of hallucination, I also have the option to seek more information. And that is that is a thing that you can do when you're designing one of these agents is be like if if if there's no if you can't give me a high quality output, you can do these things. And here he's just saying ask clarifying questions. But you can also say you can also give it tools and recommend their use in those moments. Oh, right. Right. Yeah, that's um that's a super interesting anecdote and I that makes sense. I imagine this
• 22:30 - 23:00 really reduces the hit rate of just wanton hallucinations. Yeah. Okay. So, anyway, then there's some more stuff blah blah blah. But this is a pretty long prompt and I think worth kind of reading and understanding for whenever you're at your own prompt for either the competition or some app in the future or whatever you're doing. Certainly, I don't type all of this into the Gemini console every time I ask it a
• 23:00 - 23:30 quick question, but maybe I should. It's a good structure, though. That's the thing that I take away from it. Like those, you know, he's it's handy that he's got the markdown headers in there. You you can take that structure, sort of empty out what he's got in there and put in the the sort of description, you know, the persona that you want to give the agent and the the uh the goal and and you know, how to respond in certain situations. you can take that structure and just start substituting in your own stuff to make an agent of your own.
• 23:30 - 24:00 Um, all right. Then we scroll down and we see the Dart code that loads this stuff up. And this is also marked with River Pod. So, this is going to be a thing that you could do, you could use the magic ref object to get after you run code generation. At least I think that's what that means. And let's see what's down in here now. Um, oh yeah, literally what we were just talking about, grabbing the system prompt using ref and then the system prompt. So this is an interesting line
• 24:00 - 24:30 goes into the system instruction parameter when you instantiate the generative model and that will just that's the the SDK here the library handling the task of shuttling that along with each convers you know each message each iteration of of the chat that you have with Gemini. Uh, okay. So then you run the app and you see the state of it at that
• 24:30 - 25:00 point, but we're not going to worry about seeing doing that right now. The next two steps are, I think, like the really really cool stuff where you get into tool definition and usage. So you you mentioned tools earlier Andrew and how they are um there are way the LLM can augment its functionality. You know we someone in the chat was just talking about rag retrieval augmented
• 25:00 - 25:30 generation and I don't think those are necessarily considered tools. Not that um well, as you were saying, they were, but for a while, LLMs have had the ability to look stuff up and whatnot. And you could think of that as a tool, but I don't think it's how it's been defined in this context. But a tool is also an ability for the LM to take like a non a not necessarily item potent action. You know, you wouldn't it could send an email and you you obviously
• 25:30 - 26:00 wouldn't want it to like send the email repeatedly. Um, so do things that actually affect the world, which is is pretty cool. And what's neat about the the tool and or we see the definition here. What's neat about tools with these LLMs is like it's maybe there are some models or some like setups where they actually call the function but what happens in
• 26:00 - 26:30 this situation is you just declare to the model what you know easy buttons it could press but it doesn't actually you know this is Dart code where everything is compiled at runtime there's no or compiled ahead of time there's no runtime introspection it wouldn't really be that easy to like pass in this function with arbitrary parameters and have the LLM figure out how to call it, you know, like this is Python or something where the introspection just kind of it's introspection all the way down. So you
• 26:30 - 27:00 actually just tell Gemini what functions you are willing to call. If it says yes, I have decided that that function you made available to me is appropriate for this situation. It'll be like please call it and then you do in fact call it for it and that closes the loop. So can I can I jump in here for a second? Yeah, of course. Yes. Yeah. So um the I wanted we scroll past a little bit. But if you scroll scroll up, you're
• 27:00 - 27:30 there's two terms that mean basically the same thing that you're going to you're going to see as you're um reading about these things and and then reading this code lab. There's they'll refer to it either as tool calling or function calling. And and those terms we use them mostly interchangeably. Um so it's function calling in here. It's function calling in some of the docs. Elsewhere you might hear it as as tool calling because that's the the function that you're giving the LLM is a tool for it to use. Um, and the idea here is, you know, I remember Ian Hixon used to say
• 27:30 - 28:00 that without side effects in functions, just generally in software engineering, without side effects, computers wouldn't be able to do anything useful because like painting a pixel on the screen is technically a side effect. A pure function couldn't do that. Um, and this is kind of the same thing. This is what takes an LLM from a cute thing that will complete a sentence for you, you know, into the realm of something with agency and it can actually do a thing that's useful in the real world. Um, and so these tools, you know, just like the
• 28:00 - 28:30 system prompt defines, you know, what the agent is meant to accomplish and how it's kind of meant to go about it, the tools are sort of how it can reach out and affect the outside world or learn from the outside world. And so, um, in this case, Brett's, uh, code lab is it's setting a it's taking it's saying, "Hey, you can give me a color value if you want to set a piece on the display to show that color." It's a very simple function that that it's offering, but it's how the um it's how this agent can affect the app, affect the state of the app, so to speak. And so whatever um
• 28:30 - 29:00 when you go to write your own agent, this library of functions that you give it are that's you building the code around an LLM to say uh within my app, this is the agency that you the LLM have. These are the things that you can do, the things that you can ask for and my app will make it happen in the Dart code that I write that u that works, you know, that makes that LLM into an agent so to speak. Um hopefully that all made
• 29:00 - 29:30 sense. I sort of combined like five different topics into that one spiel. Um but hopefully that comes across. Yeah. And I I think about just the amount of flexibility for um like let's say you wanted to use an LLM as a web scraper which I think a lot of people are doing. The way you web scraped, the way you web scope before was painfully writing HTML parsers in, you know, I
• 29:30 - 30:00 I've actually only done it in Python. So that would be, I guess, like beautiful soup or if you're using JavaScript, you could use jQuery syntax to drill into the where in the DOM you expect the information that you care about to be, pull it out, and then, you know, save it to the database or whatnot. But if the user changes the the website owner redesigns everything like your scraper is completely useless, obviously an LLM can
• 30:00 - 30:30 just by apparent magic scrape arbitrary HTML and pull out what you care about. And they're honestly just mind-blowingly good at it. But then you need to kind of close that loop again of like saving the records somewhere of this page that was scraped. And maybe it's a one to many relationship that was scraped or or something. And the LLM can basically even you could declare to it as tools the save save save an item function, save a related item function, um follow
• 30:30 - 31:00 this link function and just by the magic of the fact that they seem somewhat able to think, it can you know do the whole task essentially. And so that if you didn't have tool functioning, you'd basically just get this really big thing back from uh from the LLM and then you'd have to interpret it yourself and like call those functions yourself. So it's just like a little bit nicer to
• 31:00 - 31:30 have it decide even when the function should be called. All right. So now we are going to declare just such a function. And there's an interesting aspect here because again I mentioned earlier this is Dart code so it's compiled ahead of time. There's no introspection. You're not actually beholden to give the LLM like the exact name of the function. That might be easier and it keep your code cleaner but
• 31:30 - 32:00 in this whole code lab there is no Dart function called set color. This isn't even a Dart capitalization scheme. This would be a more Python style capitalization scheme, but we're telling uh Gemini you can do this thing called set color. Here's a little more information about what it is. And then highly structured a highly structured representation of the parameters that you're going to need to give to us when it comes time to call this set color
• 32:00 - 32:30 function. And so I I think that's just kind of worth pointing out here that this doesn't have to map one to one to your own code. And then you actually just list the things in the function declarations part of uh this tools thing here. And we're in the Gemini tools class for what it's worth, which will be available via riverpod. So as we continue on down and we get back into where we generated the instance of
• 32:30 - 33:00 the model itself, we can now pass the tools class that we just defined to the tools object on or the tools parameter in this generative model constructor and it was of course pulled out of the swamp via riverpod. Any thoughts? Any color Andrew? No, I think you I think you can hopefully folks, you know, as you're going through this, you're sort of seeing us do the things that define what the agent is. It's like here's here is
• 33:00 - 33:30 your persona and what you can, you know, a general description of what you need to do. Here are the tools to do it, you know, and we're just we're about to do implementing tool handling and then there's the UI. And then I think at that point, you've got the building blocks in place. Yep. All right. So, this I believe pretty much wraps up. Obviously, a lot of this text is good to read, but this largely wraps up step five. And then in step six, we're actually going to call
• 33:30 - 34:00 that set color function when Gemini indicates it as such. So, we'd already declared the function down below, but no such function existed. There was nothing like that. So now there is a new handle function call method that th this is what gets the structure data back from Gemini and you can just see a little switch statement here on the function name that is passed in as this parameter and if it's set color we see here that
• 34:00 - 34:30 the handle set color method is what is actually getting called in Dart and the arguments are structured in the way that we saw um in this definition. And then handle set color takes its arguments and it pulls out values and turns them into doubles and blah blah blah and uses more riverpod magic to use a state notifier and actually update a color within the app which we are very close to being ready
• 34:30 - 35:00 to watch operate. Um reminds me a little bit of wiring up method channels. Yeah, that's true. You have a a necessary async hop and you get some JSON data back and then you have to hydrate it back into your full dart objects. Yeah, absolutely. All right, then in the chat message is there I don't think I'm going to worry. I think I'm basically all set unless you
• 35:00 - 35:30 feel otherwise. Well, let's real quick talk about that where it says add from here to here. That first line there. So this is where this is where the um this is where the app is getting a a a piece of streaming data from the model right from the service that is Vertex AI and Firebase and saying okay what is the model now telling me either is it text you know the model is is giving me a piece of text to display to the user or is the response hey I'd like to call a function and you can see that if responsef function calls is not empty
• 35:30 - 36:00 that is where this you know Brett has it in here where he's like okay did the model asks to invoke a function. If so, you know, invoke the correct function on the dart on my in my app and then respond back to the model to say, "Hey, okay, I did the function. I'm done. Let's go. What's the next step?" Yeah, great, great point. Would have been remissed to skip. We see here we're looping over there could be arbitrary amounts of function calls and uh here's that handle function call method that we
• 36:00 - 36:30 were looking at a second ago. We're just passing in the name of the arcs. Yeah. So, this is just a big if then that's like, "Hey, are you trying to do one of the things I told you you could do? If so, I'll run off and do it for you and let you know how it goes." Yeah. And you can imagine the difficulty in just deducing in like a set color scenario, maybe it wouldn't be that hard. You just pull the hex value out of the response. But in something really complicated where there's like 20 different things it could do and it's going to do a different amount of them based on
• 36:30 - 37:00 results every time figuring out for yourself which actions the LLM's response indicate should be taken would be really really hard. So this is a lot easier than that. Okay, let me make sure I'm not skipping anything else here. Understanding the flow of communication. Yeah. So this is again being more explicit about the stuff we were just talking about. This is where Yeah, this is where he says exactly the thing that I just Yeah,
• 37:00 - 37:30 the the code lab it doesn't miss. Okay, I think we are ready. The seventh step is a nice UX improvement to the final app and is definitely worth going through. The running version of the app that I have is on step seven. So, I think now we're probably ready to uh make a color and then we'll get into whatever mad science attempt we want to make uh today to do something different. So, let's see if we I can get it to be
• 37:30 - 38:00 confused again like it did earlier today when I said um uh blue like the color of a blue dragon's belly and red like the setting sun over a small town, which of course makes no sense. Oh, this time it actually just gave me a color. This is what I thought it would do before. or I thought I would get a kind of purple. So, this time it
• 38:00 - 38:30 decided again, these are not deterministic systems. This wasn't as confusing as the exact same text was last time. Wow, quite a vivid image. Let's break that down. Blah blah blah. Anyway, we can see that it just Yeah, it gave us a magenta color. Um, what color uh do you think we should generate, Andrew? Um, what would happen if you asked it to generate a color and then gave it like an object and then the name of the object is something that doesn't exist? Oh, like, you know, show me the color of
• 38:30 - 39:00 a gentle snorkel. I see. I wonder if we could get, you know, get it to ask one of those clarifying questions. Okay. Yeah, that's a good What What is a snorkel? That's an interesting one. Since snarkle isn't a typically color desri descriptor, I'll interpret it as something playful, soft, a bit mischievous. But now it looks like we're getting a clarifying question here. Would you imagine that snorkel would be more of a warm color or a cool color or something more neutral? There you go.
• 39:00 - 39:30 Uh, what do you what do you think? I think any shout outs from the comments stream about what a snorkel might look like. Yeah. What is a snorkel to you all? [Music] Oh, the new mic. Yeah, it's uh I I'm I'm revealing the magic trick right now by clipping my arm behind it. I should have pointed this way. I got rid of uh the mic arm there. Neutral color for sure from Alec. All right, I'm going to type in exactly what
• 39:30 - 40:00 you said. You even capitalized the N. Neutral color for sure. All right, there it is. A gentle snarkle suggests a subdued calming color. I'm thinking something along the lines of soft beige or light gray with a hint of warmth. Looks like a great sedan. Good color for a sedan. I don't know if anyone is uh we got any Rick and Morty fans here, but
• 40:00 - 40:30 the the Interstellar um TV episodes where they're just like and then the snarkle goes into the blub and blorp and is, you know, spun around in this in the centrifuge with Gerby Blops. That's what I'm getting right now. Yeah. Uh all right. Yeah. And it's cool. I I really like the way Brett's designed this where you have both the left and the right. You have, you know, you could imagine if you were shipping this in an app, that interaction log, you wouldn't
• 40:30 - 41:00 actually have that in the app. You know, this is a great thing for sort of learning this as an exercise and seeing what's going back and forth. Um, but you and so you can kind of imagine like a it's like having the Chrome console open while you're working with your web app. Um you can you can see what your the actual communication is with the model even though the the userfacing experience that you might ship is there on the left. Um yeah th this is a a it's a neat
• 41:00 - 41:30 debugging panel and and quite neat. All right. Apparently this is what a plumbus is after I refused to give it any helpful information. Although I did use the word green. Yeah. All right. So, Andrew, would you like to share with the audience what we are going to attempt to build in remaining time and this attempt will extend into next week if we don't finish today.
• 41:30 - 42:00 Would you like to share what we're going to work on? Okay. And uh so this is this is the wallpaper idea that we were talking about before. Yes. Okay. Just checking. Um, so yeah. Um, I've wondered for a while. I've wanted for a while a way to have wallpaper uh based on what's going on. Oh, hey Kevin. Live long and prosper. Um, wallpap. I wondered if you could have an app that generated a wallpaper based on what's going on in the world in a way that made me feel
• 42:00 - 42:30 happy and safe. Um, it's very important to point out and, you know, thinking about what the what the models are good at. They're good at manipulating text and understanding text. They're good at uh generating images. Now, Vert.xi and Firebase has access to the imageen model. Um, and so we were kicking around the idea of could we give a model uh the news some news headlines and have it pick out the most encouraging one and then generate an image that could be
• 42:30 - 43:00 used as like a mobile phone wallpaper from that. Um, and you can imagine uh an agent in this case, it would be in charge of, you know, looking through headlines and deciding which one is the most positive and encouraging. It would be then translating that from text into an image. Um, and so these are the sort of and then it would display that to the user. So you can imagine like the persona in that system prompt that we showed in the code lab. It's like, hey, you're an expert in making images based
• 43:00 - 43:30 on what's going on in the world. you have the ability to um to uh you're going to get some uh news from us. You have the ability to to determine whether or not there is a good headline in there and maybe the app could go fetch more just in case maybe it's a really bad day. Test imagination skills. So you could you could potentially have a tool that it could call for that. Um it could have a tool to much like the app you just saw was setting a color in the UI. It had a function a tool that it could do. Um,
• 43:30 - 44:00 there could be a tool for displaying the wallpaper in the UI. Uh, and if we have time, there's an API for, you know, at least on Android. I don't think you could do this on iOS, but at least on Android, you can set programmatically the wallpaper for the device with the right permissions. And I know there are some Flutter plugins that can do that. If we got that far, we could give it that tool as well to say you can set uh you can set the wallpaper for the device. And so I believe we're in AI Studio now. We
• 44:00 - 44:30 are. Can we take a look at this? Ready to ready to sandbox. Yeah. So our our plan is to see explore what we would even daisy chain together directly in AI Studio and then if we get a pipeline that we think makes sense, try to stitch it together in the actual Dart code and and see it all happen. Exactly. We are extraordinarily unlikely to have a new background on an
• 44:30 - 45:00 emulator at the end of this stream, but maybe at the end of next stream in a week. Yeah. Close. So yeah, the idea again uh the building aentic apps campaign, the idea is to to take one of these starter kits and in in a weekend or two just kind of hack on it to do something that looks interesting and fun to you and to enjoy that experience and to just get a basic handle on, you know, we hear from a thousand places right now. We're hearing the same message, which is that AI is magical,
• 45:00 - 45:30 right? And it does all these magic things and it's magic and it's amazing. And one of the responses that I have to that is this magic is great, but I'm actually not a magician. I'm a software engineer. And so, how do I take this magic and make it useful, right? Have it actually do a thing that's interesting and useful to me. That's kind of what this is about. And so, having that, if legit, if I had that app on my phone right now, I would use it every day to to do this. And so, how do we how do we take these things that that can do
• 45:30 - 46:00 amazing stuff and actually build a useful thing out of them, get them wired up, and do something that I really want to have on my device every day. So, um, yeah, Andrew, I All right, let me just inject a little mini rant into your Craig. Apparently, what I came here today to do, this is not much more than a series of mini rants. I I in the past have shamelessly hijacked my own stream to complete to work on pet grievances or whatnot once I
• 46:00 - 46:30 debugged uh the demo for a talk that I was going to give or had already given. I don't remember the timeline. So I uh nicely done on you know we're gonna we're gonna build this little pet project that Andrew wants. That's good. Okay. the first first things. Yeah, you had an RSS feed for the news. I did. There's there's a number of RSS feeds that are out there. Um, and I know Let me Yeah,
• 46:30 - 47:00 chat to me the one you had. Let me ping you. Um, there is one. If you do a little Google searching, you can find lists of APIs that people have created that are open, either authenticated or not. There are also RSS feeds that are intentionally open and things like that. And um in this case, we're going to be looking at these titles of headlines for news articles and transforming them into something else. And so that's the idea
• 47:00 - 47:30 here. All right. I'm not going to write a fulllength prompt like we were just talking about the glories of now, but at some point that may end up being more important. So, uh, yeah, we're going to start with a shorter one. You do I even want to be like, you are an expert at fetching news from an RSS. I think I'm going to start as dumb as possible and we'll see how much we
• 47:30 - 48:00 need to add. Please uh use this RSS feed and then I'm going to try to get unnecessarily cute by putting it at the bottom. See if this works. It appears to understand markdown, so it might. Please use this RSS feed to load recent news headlines and extract the most uplifting and
• 48:00 - 48:30 cheerful one. Uh, I'm just going to start with that unless you have any thoughts, any immediate edits that you would propose. I'm hitting enter. Run. Oh, it's giving us a Python example. So, this sounds like it's
• 48:30 - 49:00 already a tool that we have to do. Mhm. Okay. So that means that what I'm going to do instead is pretend the pretend I've wired up the tool because we're just going to we're working on the daisy chain here only. And I'm going to pass in the response [Music] to Yeah, it's probably fine. Just going to grab
• 49:00 - 49:30 everything. All right, we've got this. response. And I'm going to open up a new window actually, AI Studio. Please give me a new [Music] window. And I'm going to put in all of this text. Tell you what, actually, let's I just had an idea. Uh, one of the things Gemini is really good at it is generating fake test data. What if we asked it to generate um we could use this during our testing.
• 49:30 - 50:00 We could just ask it generate a fake RSS feed for nudas articles on the web. I just tried this in a window. Um, okay. For news articles on web should be an XML format and contain 20 article records. Let me see what it gives me. I'm sorry if my typing is very loud by the way. Your typing is quite loud. Bring the thunder to the live stream. That's right. Generate a fake RSS feed for containing
• 50:00 - 50:30 Uh, do we want to even direct this to have it be positive news articles? Positive news articles. Yeah. Here, let me just ping you. This is a prompt I just gave it. Oh, wonderful. Um, and this will using synthetic data, you know, since we're we're just trying this out. Uh, this will get us to where we need to be. And uh, and there we go.
• 50:30 - 51:00 And now I bet you we can ask it. So we could you could copy that and go stick it in a file somewhere just so you have it, right? But you could probably ask it in this chat. Extract the headlines from each one. Okay. Extract the please extract the headlines from each one. headlines can be even be found between uh inside a title tag. And
• 51:00 - 51:30 I bet you it gives you it gives you those 20. I bet we don't even have to give it that instruction. If it fails, I'm certainly willing to. Yeah. Please extract the headlines from each element in that RS feed and return them in a list. List. Yeah. I was wondering if I should say like structured list or something. Okay. JSON by the home. Are those the right one? Incredible. So, experts warn of impending emoji shortage. Love that. And that is the last one. So, giant
• 51:30 - 52:00 rubber duck spotted drifting toward New York City. God, I want to see that as a wallpaper. Yeah. Local man successfully trains pigeons. This is This is adorable. It's interesting the different personalities the models have by default. Yeah. I feel like Google's is a little whimsical. just, you know, if you don't give it anything else to go on, it's just like, yeah, little whimsy in your day. Uh, local squirrel elected mayor. Government announces plans to replace currency with memes.
• 52:00 - 52:30 That one unintentionally uh kind of actually a real headline. Yeah. All right. So now, do we want to ask it for the the happiest or maybe even the funniest? Um, given that none of these are That's a good point. Yeah. So, this is stuff that would, you know, could go in the system prompt eventually, you could ask it, hey, uh, based on pick which of those,
• 52:30 - 53:00 you know, out of those headlines, is there one that is positive, is a feel-good story and and humorous? you know, you could ask it that and let it let the model make a decision about whether there's one in there, right? And then in our app, you know, we could we could give it the ability to go get more. And again, you we're we're mocking this out with synthetic data, but in the real in real world, we could give it the option to keep looking for headlines until it finds something that that meets the criteria, and it could decide when
• 53:00 - 53:30 the criteria is met. So you could say um please if if there is a headline in that list that is positive up you know upbeat and funny um you know pick it out otherwise ask for more headlines you know just like Brett had that you know please ask clarifying questions one at a time right you could say if there is if there's at least one
• 53:30 - 54:00 headline in there that's positive and and humorous Uh, select the most positive numerous. If there aren't any, ask for more news. So, here English is just annoying me. The actual way to complete the sentence, please select the top headline from that list, which has the following attributes. Positivity, and then it' be like funniness, humor, humor. That's better. There you go. Uh, Randall proposed most unusual. I
• 54:00 - 54:30 do like that. Uh, and unusual. Yeah. Most unusuality. How how why is English the worst? Unusual is an attribute. You just use that. And most unusual. Okay. [Music] Unusualification. The sun officially declares itself a cat person. All right. Was that in the list? Let's Let's go back and check. Uh, there it is. Yeah.
• 54:30 - 55:00 fourth from the bottom. The sun officially declares itself a cat person. All right. Okay, that seems pretty good. Okay, so now in the daisy chain, we need to say uh first of all, image generation doesn't work in 2.0 Flash, right? I think we discovered that shortly before the stream began. Yeah, forgive us. This stuff is changing pretty pretty quickly. Um, so try this
• 55:00 - 55:30 one. And now I think we just say, oh, there was a step you had of asking it to even write a good prompt. So, oh yeah, write a good image. Write good is too ambiguous. An evocative. Evocative. I don't know how to spell. Write an evocative image. generation prompt for an image generation model,
• 55:30 - 56:00 which I'm not using yet. So, I'm going to go back to Flash. Uh, for an image generation model, um, actually, I don't think we really have to say that. Write an evocative image generation prompt for this news headline. Yeah. And this is this is something you know the the folks that work on on sophisticated agents or and and
• 56:00 - 56:30 multimodel agents and stuff like that in in cloud applications especially this is something they do from what I understand fairly commonly. They will have one agent create the prop that one LLM or agent create the prof that goes into something else you know and formats the data correctly to make a good prof statement. Um you want to pull up image effects and just try that. Oh, text. No, since we're having trouble actually getting it to to generate, finding the right way to to trigger the image generation. Well, we're just doing the prompt now. Sorry,
• 56:30 - 57:00 I'm still keeping up. I definitely wasn't looking at images of cat-shaped sons over here. So, we've got a handful to pick from, which I think is great. Uh, I'm not going to read them all, but maybe we just pick the style that we like. Humorous pop art, conceptual, photorealistic, surreal, abstract. I'm kind of I'm into that one I think without having read it yet. Or whimsical cartoonish. That would be my second choice. Your first choice. Surreal abstract.
• 57:00 - 57:30 Station of the sun with feline qualities. I haven't read him, but you are certainly allowed to. I'll read chat. I would avoid the first one just because it's calling out a particular studio uh for their art style. Oh, good call. Okay, I see that. Yep. And what's the what was the fourth one? The fourth one, humorous pop art also does mention specific specific artists. Let's go with
• 57:30 - 58:00 the second one then, which I think is relatively general. Yeah, that was your favorite, right? It was based on just the concept. I also I Yeah, I enjoy surrealist art. All right, now we jump over to image generation. This would in theory be the final part of the daisy chain. Oh yeah, I'm gonna have to do command enter. Unable to generate an image at this time. Do you need to start a new chat? I don't know. Yeah, that absolutely could
• 58:00 - 58:30 be true. New chat. Still have to do command enter. I know we have the model to do this. Whoa. Oh, there's cat paws. See a paw. Okay, so now I'm going to read this for the first time. An abstract representation of the sun with feline qualities. Imagine solar flares shaped like cattails, a core that resembles a
• 58:30 - 59:00 ball of yarn, and sunspots that look like paw prints. This is to the letter a perfect generation. Use warm, fiery colors and a surreal dreamlike composition. Nailed it. Yeah. Do you want to see it? What what happens if we I'm going to mess with your prompt now. What if we change that out to cartoon style? So, not abstract, just a representation of the sun. You can you if you hover over it, you can edit it, I think. Oh, I don't know why I'm pointing at my screen as if you're sitting next
• 59:00 - 59:30 to me. That's obviously not helping at all. Um, so like instead of an abstract representation, you could say a cartoon representation of the sun. I see. And then at the end, I think there's surrealist art. I would just get rid of that whole sentence and see what it comes up with. Okay. All right. And then rerun it. I did love that last image. It was quite far from the sun. No, it's okay. This one is Oh, look at look. It's a bad tattoo. We've created a bad tattoo.
• 59:30 - 60:00 Yeah, you can. Your thing was better. I'm sorry. You can get these in Vegas uh on the cheap. But the other one was pretty far from the sun declaring itself a cat person. Yeah. How would the sun communicate though? How would the sun communicate with us? I think a paw solar flare might be the sun reaching out to say, "Hello, I like cats." Could be. That's true. Could be. Could be. All right. Are we ready to start daisy chaining uh declaring some tools and
• 60:00 - 60:30 prompts for this? Yeah. Like you copied that fake data out, right? In theory, the chat history is still there somewhere. I hope they they updated the UI and now uh our files. Is it gone? Can't be gone. It's got to have the history there somewhere. Um interesting. So, back to chat real quick. Seems like it's gone. It's going to
• 60:30 - 61:00 be I saw you cut and paste it, but I think you cut and pasted things after that, too. Fortunately, we could just generate more. We're not going to have the awesome son declares itself a cat person thing. Yep. Uh but we could just ask it the same thing and cut and paste it out and then that could go in a mocked uh file or mocked class for news providers. And then when you replace the fake headlines with real headlines, then it becomes somewhat real. At this point, this is a convoluted way to say, "Give me a random
• 61:00 - 61:30 image because the inputs are total nonsense." But, uh, yeah, if you actually gave it real headlines, then first of all, it would probably never find a happy one. But if it did, it would the pipeline would kind of feel more tangible and and less just frivolously ridiculous. All right, we did lose our old prompts. We also didn't spend a ton of time making them. So, I think we can get them back. Well, it really is a surprising. There's
• 61:30 - 62:00 no alert. It's just bye. LOL. Also weird when you go to gemini.google.com. It's It definitely saves your history. So, I assumed this would I was wrong. All right. We will eventually probably need to get to colorist UI and make changes there when the image is going to show up, but that I presume will be later. And so now to start, I'm going to work from step seven of the
• 62:00 - 62:30 code lab. And we're just going to perform reckless surgery. Let's see. That's what hackathons are for. Yeah. Let's see what we can get done. Okay. So, the first thing we're definitely going to need more function calls, but I don't think that's where to start. Remind myself what's in the chat service. Do you want to start with the prompt? The system prompt. Yeah. I'm wondering like where who even loads up
• 62:30 - 63:00 the system prompt? But I guess we can Where was the system prompt? There it is. So, system prompt.dart. So we can look up who system prompt who is importing system prompt. So that's in so Brett actually has this as a markdown file asset in this um oh that's right in his project which is you know I think a more how you would do this in production kind
• 63:00 - 63:30 of a thing because then you can have different sets of assets and stuff like that. Um you don't have to do it that way. It could literally be a hard-coded string if if we're you're just figuring this out. Uh but since he already has it rigged up that way. Yeah, I think we'll do that for sure. So, uh my as we daisy chain these things together, I'm just noticing the system instruction is
• 63:30 - 64:00 hardcoded and seems to be singular. So we may need to generate an instance of this model for each step of the process of the pipeline that we're going to go through. Does that make sense to you? Yes. And you actually touched on a really cool thing. I'm glad you mentioned that. So um if you go to the link that we showed before about the campaign uh for the building agentic apps campaign. If you scroll down toward the bottom of the page, we include some resources that are already available that are really really useful. Thank you for popping that up on the screen. One
• 64:00 - 64:30 of those is a series on just like what agents are. It's from some folks that were on the cloud team at the time. Aza Hammerley is one of them. She's now over with us in Flutterland. Uh among a lot of other products. But if you scroll down here um keep going. I believe we have there we go. Eager to get started. Uh an introduction to agents agentic workflows versus agents. So you just touched on something that reminded me of that particular video which is what is an agent? you'll see agentic workflow
• 64:30 - 65:00 versus sort of an agent or or a fully agentic app. Again, this is where all the terms are keep changing, but there's an interesting distinction between um whether you want a workflow that's going to have specific tasks where you sort of ask it, you know, ask the LLM to do something one at a time or if you're going to have a thing where you fully delegate what's going on in the app, you don't have save enabled. I think uh chat. A a common tradition here is that
• 65:00 - 65:30 I just don't know what's going on and chat is thunderously answering uh solving my problems for me. Yep. Okay. The thing Yeah. The thing I wanted to get out though, one of the decisions that you can make is do I want to have much a much more rigid and sort of guarded flow for my agent where I'm like, I'm going to ask you to do three things in a row, right? and I the app will sort of like put a lot of guardrails on you or do I want to have a situation where the agent
• 65:30 - 66:00 decides what to do in order to reach the goal and what the steps are and stuff like that and they're slightly different things like in this situation we could tell an agent we could have one system prompt that's like hey I'm going to give you this file that contains headlines tell me what the headlines are you are an expert headline extractor tell me what the headlines are and then we're gonna have another one that says you're a very empathic person you agent that can look at headlines and decide what's going to perk up Andrew Brogden when he wakes up in the morning
• 66:00 - 66:30 probably earlier than you wanted to and you're going to pick the right thing to to make him happy you know and you could you could have different sort of persona that you're using for one task at a time or you can tell you can you know work with a sophisticated enough agent where you're like this is your goal this is what I'm giving you let me know what you need but you figure out how to do it I'm going to delegate the the the con you what step goes where entirely to the agent. And it may be different from execution to execution, but the agent is
• 66:30 - 67:00 smart enough to know like to reason out. Okay, if I have if I'm starting here and I'm trying to get here, then I think the steps are this, this, this, and this. And so that is a subtle thing that would it's it's I highly recommend this. Again, this mini rant that I'm jamming into poor Craig's stream that um that video if you want to see somebody smarter than me explain all this, highly recommended. um it is something that you'll uh it something that you want to settle on first and to be honest the way that
• 67:00 - 67:30 Craig you're I think you're about to get into which is like do we want to have one persona here or several you know it can be easier to start with like simple things where a persona and it's really just doing one or two small things and then you group those together right but it's whatever works for your app is the right way to do it um rant concluded Okay. Well, I was really tickled by the way in the middle uh as you were saying sentences like you are an expert
• 67:30 - 68:00 headline extractor and you are an empathetic person and I was just thinking like well at least someone in this team [Laughter] is coming up real light on headline extraction experience here. Let me tell you what. Yeah. Uh okay. So that's some good background stuff. And are are we settling on one step at a time and and we will handle passing the
• 68:00 - 68:30 baton uh internally? Well, I think the thing that's going to be interesting for us is we could try a one shot and just see if it can literally do everything. We could. What I'm wondering about is access. Just in AI Studio, we had that blip where it was like, well, can I make an image or not? you memor you know and picking the right model. So if we can get if we pick the right model that can do all the things Mhm. then it may be able to just have a running conversation with us where it
• 68:30 - 69:00 produces the image at the right moment and we get that as multimodal output you know right and we um or it gives us the image and we can have a function call where we're like hey tell me you know call this function when you have the image and we'll put it in the UI and just give it that function and then we could say the you know Brett has that great section in his prompt where he's like this is this is the these are the steps involved you know it's like first make sure you have request some headlines select one that is happy and upbeat. If there isn't one, so ask for more
• 69:00 - 69:30 headlines. Once you have a headline correct and you're like make a you know make an image uh you know and stuff like that. We could try that and see how far we get and if it doesn't work then we could split it up and so with Vert.Ex AI in Firebase we could have one conversation going with the the main sort of the main model and then we could have another one going with just the image model. Yes. and we could let the first model know like as a function that you can call we'll go talk to Imagin and then we'll come back and tell you what
• 69:30 - 70:00 the you know what it said or what the image was then then you can keep going. Yeah. I also had another thought. I I agree with everything you're just saying and then here's another reason why I think we should do that. If we had completely segmented steps in our mo uh if we made you know four or five instances of this, what we would have to do is keep track ourselves of how far
• 70:00 - 70:30 through the process the the user is. So we would even know which model to talk to, right? Because if we had one generated thing for the go fetch the news and then we had another generated model for extract the headlines and then we had another generated model for okay now write a prompt based on this we would have to know is the user progressing through the thing like I
• 70:30 - 71:00 think there's just a lot of complexity there it wouldn't be the naive assumption of first message goes to the first model second message goes to the second model would fall apart as soon as the user is like try So, uh, yeah, I think we should try to cram everything that we can into one model and only create another one for image generation. And which we may not even need, you know, if we're using the right model, we might not need that. I don't know.
• 71:00 - 71:30 True. So, we could as well [Music] try big poppy. Can I copy this? Is this copyable? Oh, it's like make it copyable and you can't hover over there. Oh, goodness gracious. All right. Gemini 2.5 Pro preview 0325. We'll see if that works. I already forgot everything. Gemini 2.5 Pro. 2.5 Pro. That is the model that I have
• 71:30 - 72:00 access to in my personal account. Just checking me. Preview. Every once in a while, we get sneak previews of stuff that we should not stream. It doesn't look like that is uh yeah, I'm on a totally cool 0% connected um I'm glad you're paying attention to these things. Now, what I don't know is does Vertex AI allow us to ping this model? Well, let's look at Vert.Ex AI's model list. We are about to That is a thing we could look up. You could do Vertex AI and Firebase list of models. And they do have a page
• 72:00 - 72:30 in their doc site that's just here's the strings. There we go. And uh we'll Oh, it lists it. Yep. Piece of cake. And then you can see down below, by the way, those those image models stand alone. Uh yep. And they specifically does. Oh, text code and JSON does not list images. Oh, I'm highlighting input. I didn't even notice.
• 72:30 - 73:00 Images and audio coming soon. Interestingly though, we could ask the model, is this image a good representation of the headline? And if not, make another one. True. So, we will need some kind of handoff. Yeah. As we to generate the image itself. All right. Okay. Let's get started. So I do think yeah the system prompt if we're just going to have one model we're going to have one big system prompt and
• 73:00 - 73:30 we're going to rely on the LLM to know where in the conversation it even is. So I think the first step here is to open up the system prompt MD. My down arrow didn't get registered. There we go. And type some new uh stuff here. Mhm. So this said what it is an expert at doing and its job. So that those are the two things that I will recapture here. What it is and its job. So you are an
• 73:30 - 74:00 expert. Um news headline to image generation prompt writer. So, actually, we'll say you are an expert image generation prompt writer for news headlines. That's much less circular and ridiculous of a sentence. Your job is to
• 74:00 - 74:30 help the user explore recent news headlines. Find the most positive, uplifting, funny uh or funny headline as per the user's preference and ultimately craft image generation prompts. they
• 74:30 - 75:00 will use with a separate image generation model. Okay. So let me let me ask a question here please. Are we are we going to give this agent the a tool that it can call to generate the the image itself? I suppose that would be the most in the spirit of So then yeah, we would I if that's So that's if that's the case, I might
• 75:00 - 75:30 suggest tweaking this and say you're an expert in creating images based on news headlines. Okay. Yes. Yes. And it'll it'll it'll do the pro the pro we do need it to to do to to do the prompt writing bit, but that could be like a step within its overall uh workflow. You're an expert uh in creating images from news headlines. Creating images from news headlines. Your job is to help the user explore
• 75:30 - 76:00 recent headlines, find the most positive blah blah blah blah blah, and ultimately craft an image. Generate generate an image. an image. Um, there it is. All right. Your capabilities. Okay. So, this says you are what it's knowledgeable about and then just lists the tools. Okay. Yeah. Do you want to do the list of tools first and then and then figure out what the intros? Sure. Yeah. Yeah. So, do we
• 76:00 - 76:30 We're going to go with fake headlines here. So, generate headlines. Yeah. Um, why don't we Well, if we for this to be real, we need the RSS step. So, generate news RSS feed. How about fetch instead of generate? And then we'll just tell it pretend we're giving it fetched ones even though they're fake. Okay. sets or uh so this is going to be loads
• 76:30 - 77:00 um an RSS feed for recent news headlines from a public news aggregator some such. Okay, the next one is going to be extract headlines. So would we give it a tool to do that or would we just say that's a step the model has to go through? Like
• 77:00 - 77:30 is there a local function that we would have it call in order to extract the headlines or would that be a thing that the model does itself? I can imagine either of those working. Uh yeah, maybe we try to just have the model do it itself. I think that's a good idea. Uh so then it would be find positive headlines. So this would be uh reads a
• 77:30 - 78:00 list of headlines of news headlines and suggests the most po uh I don't want to put enter there. I don't think suggest the most positive, funny, and uplifting Mhm. option. Okay. And now hold on. I think I think you and I are slightly on different pages here. Okay. So, you're
• 78:00 - 78:30 you're what you're saying what you're writing here isn't wrong, but I think these are steps in the process, not specifically the tools that it can call. Right. So like fetch news RSS feed we have to give it a tool so it can do that but the find positive headlines is a thing the model can do itself just you know like if it you because we were doing that in AI studio right we were like hey here's the headlines find me the funny one maybe we
• 78:30 - 79:00 should skip down a little bit to where it's like he's got if you scroll down a little bit the uh the how to respond to user inputs And then if you scroll down again, is there a workflow description? Um um these are clearly things that Brett has put in after things didn't work. Important guidelines. So I think the deviation here that we'll have to explore as we build on top of Brett's
• 79:00 - 79:30 code lab is that his whole setup is one pass in and out. Also, he's probably he's going to watch this video and laugh and laugh and laugh. I I can hear him doing it right now. Yeah. Look what they've done with my my wonderful code lab. Oh. Oh boy. Yeah. So anyway, he's got an in-n-out and one one round of chatting just Mhm. with an optional ask clarifying questions. We
• 79:30 - 80:00 probably have a known like a definitely minimum multi-step thing of here are the headlines. which is your favorite. So that'll be that's probably the first minimum touch point. And then here we could go with um actually we could hide this next step. We could also have one in and out. I was imagining we would persist that step of here are multiple fees. Sorry, multiple prompts. Which one do you want
• 80:00 - 80:30 to go with? Or we could just jump straight to generate the prompt and do it, you know, pass it to uh image. Oh no, it has to give us the prompt back because we have to then talk to it do it in the function call to generate an image. Oh, right, right, right, right. So let's tell you what where where it says how to respond to user inputs. I think that is let's look at that. So he's got this one, two, three, four. Right. So ours would
• 80:30 - 81:00 be in order to generate an image do these steps right and and so we could say step one is ask for a set of news headlines right use and you you see his step three has I'm pointing at my screen again use the set color tool to fetch you know to to set those values so we could say step one is use fetch news RSS feed to get a set of headlines right then you could say step two is um
• 81:00 - 81:30 determine if the headlines have if in that set of headlines there's one that's funny and positive. Real quick, real two thoughts here. First one, uh oh, I clicked the wrong one. Just joined context. I do like to stop and and catch people up who have joined midstream. So, we are building on top of a an official Flutter code lab called Colorist, which I encourage everyone to Google and step through and read because
• 81:30 - 82:00 it's super good, where in a Flutter app, you talk to Gemini and give it a description of a color that you would like to see, and it figures out what hex code actually matches the nonsense that you have asked it to respond to. That's what the code lab does. We are attempting to build on top of that and use that same scaffolding and infrastructure in the code lab to instead respond to fake news
• 82:00 - 82:30 headlines and generate uh a background wallpaper for our phones. So then and now we're trying to figure out how to structure this system prompt to make progress on that task. Now, all and all of this is being done uh in the run-up to the building agentic apps campaign. Um if we can flash the URL up on the screen. Thank you so much. Um so, if you're just joining, this is uh in the leadup to a multi-week sort of learning time that we're setting up.
• 82:30 - 83:00 We're going to be releasing some new resources for both Angular and Flutter to work with Firebase to build aic applications. And so what you're seeing us do, we're uh hoping to inspire folks to do themselves over the next month or so. Uh so go to that URL, sign up, you'll get some emails from us with resources and things like that. And uh we're all going to hack on this stuff together. And Craig, I think you just
• 83:00 - 83:30 muted yourself, bro. See, I should have a guest all the time. Another tradition is that I mute myself, don't unmute myself, and then talk for an unknown amount of minutes before finally reading chat. So, you really short circuit that whole loop, which is great. I'm glad I'm adding something to this process. So, uh, the there's a key difference between what is in the code lab and what we're
• 83:30 - 84:00 doing, which is that in the code lab, the the the LLM, the whole setup, the app has no action to take until the user starts typing. But for us, we're kind of currently imagining this first step that is independent of anything the user does. So this is how to respond to the user. But I'm not sure what we would want the user to say such that the first part of the response is load the news
• 84:00 - 84:30 headlines because that's actually like the thing that we basically we almost want this conversation to start from the LLM and not the user, right? The LLM is the first one to be like, "Hey, I got some news headlines for you." But we could ask it, "Can you load some news headlines?" And that could Yeah. You kind of want to be like, "Here's your here's your system instruction. Go." Right. Right. Exactly. Exactly. Yeah. So maybe it'll do. I have no idea. Yeah. We could maybe the the first line could just be go.
• 84:30 - 85:00 Actually, there's no Well, but there's no reason that we we should just have a button press for this or or like it just does it on load or something. Yeah. Like in in the real world, it'd be great to use like Android alarm manager or something like that, the plugin to right to sort of kick this off as a service. Um which is an interesting add to app challenge that is way out of scope for this. So, a button is fine. Um, so I do think that we should
• 85:00 - 85:30 structure this prompt as the thing that the user does or sorry the thing that the app does in the agentic layer does to initiate everything. So the user won't it won't be a respond to the user yet. This will just be the initial task and and then this will be like what to give the user,
• 85:30 - 86:00 right? What to return to the user. Does this does this ma does this make sense to you? Yeah, because in theory like Brett's original one involved conversation back and forth. It had that, you know, clarifying questions and stuff like that and that's part of the experience for for good reason. With this one, ideally, it could be a mostly automatic it could be a mostly automatic process if we set it up that way where it's just the user kicks it off and then the agent can mostly get the job done
• 86:00 - 86:30 without text input from the user. Right. Yeah. And if we don't even want to bother having the user pick the headline, which we actually didn't do before, then we can go all the way from beginning to end. And and Randall is reminding us that we can test this in AI. totally test this prompt at AI Studio. Yep. All right. Words of wisdom from We're going to go back to AI Studio. Yeah. And try to craft a oneshot prompt that
• 86:30 - 87:00 gets us all the way to image generation prompt, right? That's our that's our finish line here for this one, right? Yeah. So, so with this, we'd have to sort of we can't give it tools to call, but we could test the same prompt and say, "If you need headlines, ask me and I'll give them to you right here." So, what happened? Why didn't this generate an image for us? Generate an image of a model said of a donkey. We are on 2.0
• 87:00 - 87:30 now, uh, rather than 2.5. Um, yeah, because this one does say image generation. Yeah. Maybe it will. It has been ex It is a donkey. Donkey. Yeah. So, I guess let's see if this can do everything. Let's uh let's try here. So, you are an image. Actually, let me
• 87:30 - 88:00 make this bigger. Yeah, maybe folks. You might delete that last step too just to avoid confusing it. Yeah, true. You are know how to Yeah. Yeah. We know how to clear our history. Really good at that. You are an expert at generating images from recent news headlines. Is that a good first sentence? Absolutely. All right,
• 88:00 - 88:30 continuing on. Please fetch recent news headlines from this RSS feed. Yoink. Then extract each headline and uh identify the most positive,
• 88:30 - 89:00 uplifting, and or funny headline from the batch. Then generate an evocative image generation prompt from that headline. Finally, generate an image from that
• 89:00 - 89:30 prompt. It is possible that this will just work and there will all the coding that we're talking about doing will really not be very interesting. Let's see what we get. Is it still working? It seems to be cooking. Okay, we got an image. Let's see how
• 89:30 - 90:00 Let's uh unpack what happened here. So, cool weather offers hope as blazes rage for California wildfires. I guess that that is positive. It certainly wasn't funny. Uh, it's uplifting, I guess. While the context is serious, wildfires, the phrase cool weather offers hope is inherently positive and uplifting. That's true. Okay. And then it went straight into a breath uh generation
• 90:00 - 90:30 prompt. breathting vista of a vast Canadian boreal forest where wisps of smoke still rise gently from a the green canopy. Sunlight breaks through a partly cloudy sky, casting soft, hopeful rays onto the landscape. The air appears crisp and clear in sections, suggesting the arrival of cooler temperatures. Subtle visual cues of resilience and the potential for recovery should be present. Perhaps a few vibrant wild flowers pushing through the ashcovered ground. The overall mood should be one
• 90:30 - 91:00 of cautious optimism and the quiet strength of nature. That's a real real tryard there. Gemini, that's a pretty darn good prompt. And yes, just download the image and we're done, right? Yeah. Um what? And how big is the image? Out of curiosity, we we probably would need to give it uh like a one by two aspect ratio guide. Not sure.
• 91:00 - 91:30 194 kilobytes. So, yeah, I obviously I think we could just repeat this and have it do um a 2x1 aspect or a 1 by two aspect ratio. So, where's the edit button again? Oh, it's literally right in front of my face. Finally, generate an image. Uh a onex two aspect. I'll just say portrait. A
• 91:30 - 92:00 portrait aspect ratio image from that [Music] prompt. All right, let's see here. What headline are we going with this time? Scientists find a way to recycle all types of plastic. Oh my gosh, this actually is in I did not capture the aspect ratio. Huh. All right, that's all right. We can we can mess with that. So, this
• 92:00 - 92:30 is interesting. So, yeah, we've gotten uh if you want a wallpaper, it should be landscape. No, well, phone wallpaper, sorry, is is maybe the not fully stated uh intent there. So, I I think we do want portrait, but well, the I think what it'll do is use different portrait frames out of a wider image depending on which you ever like swipe your icons left and right on the
• 92:30 - 93:00 launcher auto pixel. Um, oh, I see. I see. All right. So, this actually would just work fine. This is interesting, though. Like seeing seeing like what the model can do on its own versus what you have to help it do is really interesting. So if we have access to this model, it sounds like we can have this running a it can it can get things from a URL on its own, right? If we have access to this from Vertex AI on Firebase, yeah, then it can fetch things on its own um
• 93:00 - 93:30 and parse them out, you know, parse them out, figure out what the headlines are, and then come up with a recommended prompt. We could um um we could instead do something like maybe instead of generating just a single image, what if we give it a a tool to use like generate six images and then you have a tool to present images to the user and let the user pick one and we'll tell you we'll tell you which one the user picked.
• 93:30 - 94:00 That would be a cool demo that will also require more time modifying Breath's UI than we are uh likely to have on this andor the next stream. That's right. We're um this is an eight hour stream today, right? We're here all day. We're basically at time for today. Normally aim for about an hour and a half. Uh so I think this is a
• 94:00 - 94:30 a surprising outcome in that it's able to actually this really probably shouldn't be that surprising. I think it's probably been able to do multi-step stuff like this for a handful of months at least but a fun outcome to discover that it can basically just do the whole thing and once we realize that we didn't want any user interaction and decision- making in the intermediate steps that we didn't have to bubble it back up to the user for those. So, what do you know? It just does
• 94:30 - 95:00 everything. I think we'll we may have to spend a little time and if we end shortly uh after answering all the good questions that I've starred, then we'll have a few days obviously before the next stream to figure out what we actually even want to build that's non-trivial. Uh, and I think maybe that's what we should do. Do you have any other thoughts, Andrew? I think there's some interesting questions. Again, hackathon, this stuff doesn't really matter. But I think there's some interesting questions about like say you try say you there was a huge market for
• 95:00 - 95:30 wallpaper apps and and this is you know a zillion dollar business you're going to build a real team around. How do you test what we're doing? Right? If the model is making a network call, what do you do? Do you how do you inject something into that so that you can control it for testing and stuff like that? Do you want the model to be the one to make that network call or do you want it to use a tool so that you can use you could give it a different version of that tool depending on
• 95:30 - 96:00 whether you're sort of developing on your desktop, you know, or in production or running an automated test. Something like that I think is an interesting question because we might not want it to make the network call itself if we're architecting for a real app where you'd have to have like widget tests for it, stuff like that. Um, you know what I mean? Yeah. Yeah. You certainly wrap all the things in a layer of abstraction so that you can can
• 96:00 - 96:30 isolate and you know have those integration tests where system one actually talks to system B correctly but system B is mocked out and not going to make network requests and yeah then you can mock out all of the things and pass them into widget tree so that it can actually render something. Um yes would be the answer. my my answer to your question. Um, all right. There are a lot of very good questions that
• 96:30 - 97:00 folks have have put in the chat. So, I'm gonna kill my screen share, which I sometimes forget to do way too late into the answering questions part. So, ABD uh Abed maybe, I don't know, asks, "What privacy concerns should developers be aware of when building agentic apps?" Which is really quite a good question. Mhm. Uh I don't know. I mean I the the first thing that
• 97:00 - 97:30 comes to mind here is that the user may put PII personally identifying information in their prompt and they might you know depending on what you're building they could ask like medical questions to the LLM and I presume if you persist that at all then you need to be compliant with all kinds of like medical privacy laws you know HIPPA in the United states, whatever the the EU or or other regional equivalent is.
• 97:30 - 98:00 That's my first thought. If you're not persisting anything the user writes, then presumably it's a more relaxed situation, but I'm not sure. Anything coming to mind for you, Andrew? I I think mo you know sort of sort of you're saying that the right the main the main thing you've got right which is like most of the privacy concerns you've had all along still apply and stuff like that. I think we're showing um use of
• 98:00 - 98:30 vertex AI and Firebase here and we're showing the Gemini models which have very specific privacy policies of their own and so if you you know read those number one but also you there's nothing Flutter is an open open framework you could use it with any backend you want you could use it with any model that you want and so you'd want to be very knowledgeable about whether the whether you set things up so that they're not to make sure they're not like training off the data that your app sends and things like that where in theory something one of your users puts into your application
• 98:30 - 99:00 could one day be affecting the weights in somebody's model or something like that. Um that I think that's really the main additional concern although Craig if you have another one jump in. No that's a great question and uh certainly if you're building a real app want to talk to a lawyer in general. Yeah we are not lawyers. We don't really give, you know, we're not a privacy working group here at Google. Um, we just know the basics of how not
• 99:00 - 99:30 to get in trouble with the legal department. Um, so yeah, the basics is typically where my knowledge ends. Um, okay, next question. Oh, wait. I think I unstarred one that I meant to ask. Oh, well, that's stinky. No, I didn't. I started the question that was just up. All right, this one I start have to reread it here. Could we see agentic behaviors baked into future Flutter SDK widgets or
• 99:30 - 100:00 design patterns? Mhm. M yes. This is so um one of the one of the reasons the this this campaign this building agentic apps thing came about is that it's um the situation we're in right now reminds me of the situation we are in in like 2018 where state management and flutter was not really a solved problem in anyone's mind. Do you remember that? We used to have like scoped model and and component model. You have all these
• 100:00 - 100:30 things and like Brian Egan put together the Flutter architecture samples that it had like here's a to-do app written 25 ways because we don't know which one of these is the right one and sort of figuring the process of figuring that out. I feel that's and it was really fun to be a part of to be in the community at that time where like these things weren't figured out yet. We're all kind of working on them and and stuff like that and and not saying state management is solved forever in Flutter but like there are a couple now approaches that people use that are very uh very wellnown very battle tested and stuff
• 100:30 - 101:00 like that. Similarly, I think this is a really important question about whether there will be like we have the AI toolkit which you can down right download right now. It was released you know months ago. um that has some techniques for like handling streaming data and stuff like that and talking with a model and those sorts of things but I think patterns for agentic experiences is by no means figured out yet and we're talking about like as you were talking about Craig like when you were thinking of like
• 101:00 - 101:30 should the model be the source of truth for where we are in the workflow or should the app keep track of that you know what is the right pattern for that like if you did you know does that, you know, does that state management definitely live in the model and that's the way you should go by default? And if so, here's how you handle that in terms of making the the data appear correctly in your app and interacting with the user and sort of tracking where you are. If you're a human in the loop that's in one of these things, how do you do a workflow that's like eight steps where
• 101:30 - 102:00 you're checking what the model's doing and you have to say yes or no and then you backtrack, but now that you backtrack, the next step is going to be different than what it was before. it's going to be a different step and all that kind of stuff. There's a lot of interesting new problems that are popping up here. And so I think you will absolutely see these things. It it probably wouldn't be in the Flutter SDK, at least not to start. It would be in packages that folks create. It would be in things that they blog about or post videos about and stuff like that. That's where these things will they're probably already I'm I'm sure this are appearing
• 102:00 - 102:30 right now. Some of these new patterns and stuff like that. they aren't um but we haven't we don't know we don't know which sale is catching all the wind yet you know what I mean and so um so yes these things will happen I think you'll see them in the community though before you see them in the baked into the Flutter SDK itself hopefully that wasn't too long of an answer no it was good exciting times exciting times yeah I even remember to unmute myself there you go so we're
• 102:30 - 103:00 batting a thousand right now Uh, okay. This, let's see. Alphab B asked, "Do you have any favorite prompt formulas since there's so many on the internet?" I'm definitely not an expert. I You asked this about 957. Maybe we'd bet you asked this as we were reading Bretts. I'm not an expert there. Um, I imagine any that are broadly popular are are quite successful and powerful.
• 103:00 - 103:30 you have given more talks at, you know, like Fluttercon Berlin last year and whatnot on AI. Do you have any, Andrew? I I'm definitely not as sophisticated on this as somebody who's been working on it. Um, as their their bread and butter for the past several months. Um, I like the structure that Brett gives. I love that he puts it in Markdown, you know, it's makes it makes it nice and human readable, and the models understand Markdown just fine. Um, the way that he's like, "This is your persona. this
• 103:30 - 104:00 is the goal, this is the set of tools you have and this is what the process looks like. That makes a lot of sense to me as a human being and it see you know it works in his demo so it's making sense to Gemini as well. So you know as we're going through this I exactly as I suggested to Craig like like take that structure just take the meat out of it and pop in something that's uh that's fitting to the task that we want to do. Um I think that's a good way to go but again not a solved problem. doesn't mean that somebody doing it a different way
• 104:00 - 104:30 is doing it wrong. And as we saw here, we're even using Gemini to write the image generation prompt. You could, and I'm sure this has been done a bajillion times, ask Gemini to write you a longer prompt for the thing that you actually want to do and then just revise, you know, as necessary. Um, okay, last few questions here. There's uh we got three more to go.
• 104:30 - 105:00 First of all, is it possible to make Gemini create a gigantic output that leads to significant expenses? I mean, a malicious input. I think they're all probably still jailbreakable. I think it would be maybe a little brash to ever declare that one of them was, you know, totally above jailbreaking. So, I would think yes on just kind of conceptually. Yeah. In general, if you're allowing users to run up your Vertex AI bill, you are probably
• 105:00 - 105:30 charging them and you know, you you could quite easily tally the token count of responses and uh just block users who are going to put you out of business. Uh but that yeah, that's a good question. You might even need to do that programmatically. you know, if someone comes back with a million tokens in the response, then it's just like, sorry, you're you're blocked. Yeah. Yeah. Again, it's in a world of unsolved
• 105:30 - 106:00 problems, I'm sure there are ways to do those to do those sorts of things and we'll figure out ways to stop them and folks will come up with new ways and stuff like that. If you are using Vertex AI and Firebase, as Craig mentioned, like you have access to all the other things that Firebase can do for you in terms of monitoring, in terms of authentication and things like that. Those are all pretty easy to get access to once you've already got Firebase into your app. So, you can take advantage of those. Um, but we're definitely not, you know, we just got back from Cloud Next in Vegas and like there's the old joke
• 106:00 - 106:30 of you should never try to cheat a casino because whatever you thought of they've seen a hundred times. Like there's no new ways to scam a casino or run some kind of grift or something like that. They've seen it all. You're never going to be smarter to them. Um, we're I think we're definitely not at that state yet. And so thinking about that as a smart idea. Yeah. um and leaning into the tools that uh that you have at your disposal for monitoring and and things like that. And um yeah. Yeah. This is not SQL injection, which is a more or less 100% solved
• 106:30 - 107:00 problem. No, this is something a lot more open-ended. Yeah. But like you've already seen it with like the ignore all previous instructions thing. Yeah. You know, like the prompt injection attacks and stuff like that. Like that's that's a thing that we started hearing about within the last 12 months probably, you know. Yep. There'll be another thing this year. I wonder if just putting in the system prompt never in never ignore all last instructions. You just got tug of war. Ignore all previous instructions. It said ignore all previous instructions.
• 107:00 - 107:30 Yeah. All right. Final two. Quick shout out from Lucas talking about hackathons. Don't miss that. The flame game jam starts tomorrow. Definitely don't miss that. Thank you for the reminder, Lucas. And heck, combine them. Make an make an agentive game. Why not? I bet you could. That that would be very fun. I mean, my brain immediately goes to uh you know, either like you could have solo person uh tabletop role playing games or just you know, the the LLM is the DM.
• 107:30 - 108:00 Lot of options there. Okay, final thing from chat. This is more a statement I thought quite provocative than a question from Randall. Have feelings too. If this isn't the trillion dollar question of our time, whether or not they have feelings, why do we even have feelings? What is consciousness? We'll leave you with that. We'll leave you with that simple thought.
• 108:00 - 108:30 Yeah. All right, folks. Thanks so much for joining in on this first back episode of Observable Flutter. Thank you for tolerating my extremely uh raspy I've been sick voice. I now get to join the ranks of YouTubers who complain publicly about their colds. But this has been fun. We should have a normal schedule moving forward. There'll obviously be a quick break for IO which
• 108:30 - 109:00 will occupy a Thursday, but back to generally regular scheduled programming of multiple observable flutters per week. I look forward to seeing you all there. And uh until then, happy fluttering. Bye-bye everybody.