The ONE feature that makes NATS more powerful than Kafka, Pulsar, RabbitMQ & Redis

Summary

The video from Synadia thoroughly explores a unique feature of NATS Jetstream that gives it a distinct advantage over similar technologies like Kafka, Pulsar, RabbitMQ, and Redis. This standout feature is the ability to design highly flexible consumers within NATS Jetstream, offering unprecedented control over data streaming and processing. The video breaks down various configurations and settings that can transform how systems are built, allowing for reduced upfront design costs and simpler architectures.

Highlights

• NATS Jetstream's consumer feature simplifies complex architectural designs. 🎉
• Filter data efficiently, so only relevant information is processed. 🧐
• Durable and ephemeral consumer options provide flexibility for long-term and ad hoc operations. 🔄
• Push-based versus pull-based consumers: know the best use case for each type. 🤔
• Acknowledgements and replay policies enhance data consumption reliability. ✅

Key Takeaways

• NATS Jetstream offers superior consumer design capabilities, setting it apart from Kafka, Pulsar, RabbitMQ, and Redis. 🔥
• Its consumers can filter data more effectively, increasing processing speed and efficiency. 🚀
• The video highlights the flexibility of consumer settings in NATS, making system architectures simpler and more cost-effective. 💡
• Different types of consumers—such as durable, ephemeral, push-based, and pull-based—offer a range of functionalities tailored to different use cases. 🎯
• New features in NATS are making the system even more robust and adaptable for various applications. 🔧

Overview

NATS Jetstream's standout feature is its flexible consumer design, which significantly enhances data streaming and processing capabilities. This flexibility allows developers to build architectures that are not only easier to manage but also save time and reduce costs. By focusing on consumer configurations, NATS Jetstream provides a unique advantage that other platforms like Kafka, Pulsar, and RabbitMQ can’t match.

The video explores different types of consumers—durable, ephemeral, push-based, and pull-based—each offering distinct benefits depending on the use case. For example, durable consumers are ideal for long-lived operations while ephemeral ones are perfect for quick, ad hoc tasks. The system’s ability to filter and manage data effectively ensures that only pertinent information is processed, thereby enhancing efficiency and speed.

Acknowledgement policies and replay options in NATS further solidify its role as a leader in data processing. These features allow for precise control over how data is acknowledged and handled, providing reliability and robustness in various operations. Along with new features being introduced, NATS is continually evolving to meet the needs of modern data processing challenges.

Chapters

• 00:00 - 00:30: Introduction to NATS JetStream Consumer Feature The chapter introduces the NATS JetStream consumer feature, highlighting it as a superior offering compared to competitors like Kafka, Pulsar, RabbitMQ, and Redis. The speaker emphasizes that the design of NATS consumers is unparalleled, providing flexibility and ease in architecture management. The chapter promises a walkthrough of various consumer settings and practical examples to demonstrate the unique advantages of NATS consumers.
• 00:30 - 01:00: Sharing and Subscribing Reminder The chapter titled 'Sharing and Subscribing Reminder' emphasizes the importance of sharing the video content with others, particularly coworkers, as it is deemed to be very valuable. The speaker encourages viewers to spread the word about the video and to engage with the content by liking and subscribing to the channel. This aids in reaching a wider audience.
• 01:00 - 02:00: Streams and Consumers in NATS JetStream The chapter discusses the concept of streams and consumers in NATS JetStream. The narrator highlights that streams are an impressive data structure that can be configured in various ways to manage an ordered set of messages, allowing for many distributed systems patterns. Consumers, on the other hand, are noted to be more flexible than streams. The combination of streams and consumers enables the creation of versatile and complex distributed systems. The chapter sets the stage to delve deeper into why and how consumers add flexibility to streams.
• 02:00 - 03:00: Consumer Configuration and Flexibility The chapter 'Consumer Configuration and Flexibility' discusses the advanced filtering capabilities of NATS JetStream. Unlike basic filtering options available in other systems, JetStream allows filtering on the entire subject, offering significant control over message delivery and redelivery. These features enable users to potentially build comprehensive applications on top of it.
• 03:00 - 04:00: Stream Indexing and Consumer Variations The chapter titled 'Stream Indexing and Consumer Variations' begins with a discussion about managing streams, specifically focusing on the structure and indexing of streams using Jet Stream. The speaker provides insights into upcoming features that will enhance user experience and the efficacy of these streams. The foundational concepts start with the organization of a stream, illustrating a hierarchy that includes elements like order location, order ID, and status. The speaker emphasizes the unique aspects of Jet Stream that distinguish it from other streaming options.
• 04:00 - 05:00: Event Sourcing Consumer and Filtering This chapter discusses the concept of event sourcing and filtering in Jetstream. The key point is that Jetstream indexes topics, providing extensive functionality for filtering and responding to various topics. This flexibility is crucial to the operation of Jetstream consumers. An example is given with a stream of six messages, showcasing three different consumer configurations, highlighting the versatility in setup.
• 05:00 - 06:00: Using Different Consumer Archetypes The chapter titled 'Using Different Consumer Archetypes' explores the concept of employing various consumer models in event sourcing applications. Event sourcing utilizes a centralized log to record all events, and projects this data into different data stores for easier access and interaction. This distributed storage approach allows for efficient data retrieval and interaction by various systems.
• 06:00 - 08:00: Push vs. Pull Consumer Models This chapter discusses the 'Push vs. Pull Consumer Models' and how they coordinate in processing messages. It emphasizes the need for event sourcing consumers to handle messages in a way that suits workflow engines or state machines. Key characteristics of the consumer model include the ability to filter messages, ensure message durability, and manage acknowledgments, specifying a maximum number of pending acknowledgments.
• 08:00 - 10:00: Durable vs. Ephemeral Consumers In this chapter, the concept of durable versus ephemeral consumers is explored. The speaker explains that, even if you're not familiar with these terms, they'll be clarified in this section. Additionally, it's mentioned that there's an option to use a reporting-style consumer to answer specific questions about data. For example, determining how many orders were shipped within a certain timeframe, like the last hour or the last seven days. This is typically handled by a SQL query, but the speaker notes that it can also be achieved through NATS Jetstream, by instructing it to deliver only the messages that meet specific criteria.
• 10:00 - 12:00: Message Filtering and Storage Options This chapter covers the inefficiencies of traditional message streaming methods, which rely on client-side message filtering and handling high volumes of data. It introduces the benefits of using Nat Jet Stream, which allows for server-side filtering and precise message delivery, increasing efficiency and performance.
• 12:00 - 14:00: Acknowledgement Policies The chapter titled 'Acknowledgement Policies' explores the efficiency of data processing through specific lookup patterns using Nats, focusing on how it can filter and deliver events related to particular order IDs. Different use cases are discussed, each expressed by various settings within a Nats consumer. The chapter promises a quick dive into these settings, ensuring a hands-on experience by providing examples that will be written together.
• 14:00 - 16:00: Max Ack Pending and Redelivery Setup The chapter discusses the major concept of consumers, focusing on the differences between push and pull consumers. It emphasizes a recent shift towards preferring pull-based consumers due to their practicality in making processes easier to manage. The transcript appears to be an excerpt where the speaker begins to explain what a push consumer is, hinting at opening up an inbox.
• 16:00 - 18:00: Delivery Policies Overview This chapter provides an overview of delivery policies, highlighting the challenges of managing message flow control. It discusses how the N server sends messages at a rapid pace, which can create difficulties for consumers with slower consumption rates. The chapter touches on built-in control flow mechanisms and heartbeats within push consumers, noting that despite these tools, maintaining flow control remains complex due to the asynchronous nature of message delivery.
• 18:00 - 20:00: Replay Policy for Consumers This chapter discusses the differences between push-based and poll-based consumers, particularly in the context of messaging systems. The push-based model involves sending messages directly to consumers in real-time, making it ideal for real-time monitoring scenarios that require immediate responsiveness. However, the chapter notes that better performance and scalability is generally achieved with poll-based consumers. In this model, the consumer requests messages from the server, which provides them in batches, leading to improved management and scalability. The recommendation is to use a poll-based approach to optimize performance and scalability.
• 20:00 - 22:00: Additional Features: Max Delivery, Headers, and Sampling The chapter discusses the use of pole-based consumers in horizontally scaling and enhancing flow control. These consumers are recommended due to their ease of use and efficiency. The newer versions of the Nats Jetstream SDK have made pole consumers the default setting for all clients, making them easier to implement without needing to adjust fetch semantics excessively. Developers can use these in their preferred programming styles, such as iterators or consumers, simplifying the process. Overall, pole-based consumers are presented as a beneficial option when setting up a consumer.
• 22:00 - 25:00: Setting Up Event Sourcing Consumer The chapter explores the concept of setting up event sourcing consumers, with a focus on durable versus ephemeral consumers. It highlights the confusion that might arise from this concept, which is not common in many technologies. However, the chapter explains the power of ephemerality in consumer setup, particularly in ad hoc use cases, where it leads to improved outcomes.
• 25:00 - 31:00: Performance of Event Sourcing Consumer The chapter discusses the performance and scalability of event sourcing consumers, comparing different messaging systems like Apache Kafka, Pulsar, and RabbitMQ. It highlights the importance of durable consumers for managing long-lived consumer processes and describes how NATS tracks the state of message consumption efficiently.
• 31:00 - 35:00: Creating a Reporting Consumer The chapter 'Creating a Reporting Consumer' explains the concept of durable and ephemeral consumers within the Natat server. Durable consumers are designed for long-term use and require explicit deletion, although recent updates allow for setting an inactive threshold to manage them more effectively. Ephemeral consumers, in contrast, are designed for ad hoc purposes and are meant to be created on the fly.
• 35:00 - 38:00: Handling Reporting Consumer Logic The chapter 'Handling Reporting Consumer Logic' discusses the management of consumer objects within a server. It highlights the benefit of consumers timing out or being cleaned up after inactivity, which is useful for ephemeral browser sessions that don't need to be persistent. The key advantage of using consumers is their speed in creation and operation, making them ideal for tasks like running ad hoc queries or reporting processes.
• 38:00 - 40:30: Lookup Consumer Setup This chapter focuses on the setup and scalability of Jetstream consumers. It emphasizes the advantages of using Jetstream, particularly in filtering messages before they are sent, which distinguishes it from other technologies like Kafka and Kafka streams. The chapter highlights the power and efficiency of Jetstream by discussing its unique features that are not available in traditional streaming technologies.
• 40:30 - 44:00: Using Lookup Consumer for Status Checks The chapter discusses the architecture and operations within a Kafka broker, focusing on map, reduce, select, and filter functions that must be conducted on the client side. Due to this, it's inefficient to have very large topics containing a wide breadth of data because all of it needs to be sent across the network. This leads to a significant amount of upfront design work to ensure efficient data handling and transmission.
• 44:00 - 45:30: Future Improvements and API Calls The chapter discusses best practices for managing data streams in systems like Kafka and Jet Stream. It highlights the importance of topic size in Kafka due to data transmission concerns and recommends using fewer streams in Jet Stream to simplify client-side data filtering. The guidance suggests consolidating data into fewer streams for efficient management while ensuring balanced data distribution across streams.
• 45:30 - 47:00: Conclusion and GitHub Resources The chapter discusses efficient data management strategies using Kafka's filtering abilities on an order's lifecycle within a single topic. The use of filter subjects allows for flexible subject-based addressing and wildcard utilization, facilitating easy access to the needed data. The versatility of storage options for consumer data handling is also touched upon, although it is not a major focus of this chapter.

The ONE feature that makes NATS more powerful than Kafka, Pulsar, RabbitMQ & Redis Transcription

• 00:00 - 00:30 okay so there's one single feature in Nat's jet stream that just blows away all of the competition it isn't even fair I'm talking Kafka Pulsar rabbit mq uh redus that feature is how Nats designs its consumers and so today I'm going to be walking you through all of the different settings for consumers and we'll go through some practical examples and why their flexibility is really this superpower in Nats that when unlocked it really makes your architectures a lot easier to deal with and it removes a ton
• 00:30 - 01:00 of upfront design cost so let's get straight into [Music] it before we get into the video I've been getting a lot of feedback from you all that this content is super super valuable and so we want to make sure it gets out to as many people as possible so be sure to share this video with a coworker you know before you jump into this video it's a great one by the way um do like and subscribe
• 01:00 - 01:30 all right on to the video so in previous episodes we talked all about streams and we talked about all the different ways we can configure it and that streams are just a really cool um data structure to be able to use to modify and when you have an ordered set of messages you could actually create and express a lot of different distributed systems patterns and consumers are no different in fact I think consumers are a bit more flexible than streams themselves and when combined you can do a lot of really interesting things uh but the best way that I like to describe consumers in
• 01:30 - 02:00 Nats is that they're really Windows into the data of the stream and I don't mean you know just basic Windows like basic filtering and everything like that everybody's got some basic filtering in some way shape or form but but n jet stream kind of takes this to the next level where you can filter on the entire subject and you could also have a lot of control over how your messages are delivered how they're redelivered when you start everything like that to the point where you know you can build pretty much an entire application on top
• 02:00 - 02:30 of jet stream and consumers if you really really wanted to and I I'll give a little bit of a preview as well into some of the newer features that are coming to that server that um make this kind of use case even better uh but let's start with some of the basics so imagine that I have a stream up here and my subject hierarchy looks like this I have orders. location this could be like us EU Canada things like that um then we have the order ID and then we have the status now this is really important cuz one thing to note is that uh something that sets kind of jet stream apart from
• 02:30 - 03:00 a lot of the other Alternatives is that Jetstream does uh index on these on these topics and it gives you a lot of kind of functionality for being able to filter and act on these different topics and this is kind of the Crux of where a lot of the flexibility in Jetstream consumers comes from so imagine that we have a stream with about six messages in it so we have sequence one 2 3 four five six um and then we have a three different kind of expressions of consumers and these are just examples to be honest there are so many different ways that you can configure your consumers but I wanted to put together a
• 03:00 - 03:30 couple archetypes that kind of made sense for a couple different applications so um let's imagine for a second that we have kind of an event sourcing consumer um you know one of the kind of core tenants around event sourcing is that we've kind of distributed how like data is is stored into way where we have maybe a centralized place where all of our events took place like a log like this and then we then project out our data into different kind of data stores so it's easier to get at and people can you know all of these things can interact
• 03:30 - 04:00 and coordinate with each other um in very interesting ways and so imagine that we need to be able to take you know all these messages and process them one by one so you know maybe we have a state machine on the other end that needs to act on those messages in a certain way maybe it's a workflow engine stuff like that um and so you know this event sourcing uh consumer uh has a couple attributes to it um one we probably want to filter or just say we want all of the orders we of course want this to be uh durable we want you know our acts to be explicit and our Max act pending to one
• 04:00 - 04:30 now if you don't know what any of that means that's okay because I'm going to explain what all these settings mean consequently you can also Express a consumer in a reporting style consumer maybe you want to answer a question like hey how many of you know the orders in the US shipped within the last hour or within the last seven days that normally is something that's reserved for like you know a SQL query um but you can actually do this in Nats you can go tell Nats Jetstream to deliver you only the messages that qualify um for that part particular uh set of assumptions and
• 04:30 - 05:00 again this is where njet stream is super super powerful because the Alternatives kind of just give you a fire hose of messages and expect you to filter them out on the client side which that can work for a lot of use cases but at a very high scale that can be very very extremely slow and it's also just extremely inefficient and so again one of the superpowers is that you don't have to do that with Nat jet stream you really can just have jet stream configured to send you exactly what you want to use use um the third one I would
• 05:00 - 05:30 kind of argue that is even more efficient with with its data is uh kind of the lookup pattern maybe you have a particular resource you know bound to this order ID and you're like just give me all of the events related to this one ID and Nat do a fantastic job just filtering on those and giving them to you so um these use cases are all expressed by different settings inside of an Nats consumer and today we're going to be jumping into all of those settings and then we'll write each of the exam these examples together don't worry this will actually be pretty quick
• 05:30 - 06:00 so let's move on to our first kind of major concept of consumers push versus pull so um we've recently kind of moved to more of a do almost everything in a pole-based consumer it doesn't mean that we've completely deprecated push-based consumers but um you know we found that in practice pull-based consumers make things a lot easier to reason about so let me explain to you the differences between the two um so a push consumer is a consumer that opens up up an inbox and
• 06:00 - 06:30 just gets messages sent to them and you know they receive them and everything like that that's great um but the N server will send those messages as fast as they possibly can um which makes things like control flow difficult especially if you do have consumers that are kind of slow to consume there is some control flow mechanisms and heartbeats and everything built into those push consumers but they um you know it still gets a bit challenging to reason about flow control because it's not this kind of you know synchronous like I'm going to ask for more messages and then get them um in was just going
• 06:30 - 07:00 to be sending you messages no matter what and so um it is a good fit for some things like monitoring and things that need to be as responsive as possible as real time as possible um you could lean into to this but we we found better performance and better scalability within a poll-based consumer so that's what we would recommend um on the other hand the way a poll-based consumer Works instead of just getting messages sent down to it it's going to make a request to the server saying Hey I want messages or a batch of messages I want to go fetch some messages and the server will then resolve that send the messages and and manage all that for you um and so
• 07:00 - 07:30 that that's you know that's why we recommend going to pole-based Consumers they're easier to horizontally scale they give you a bit better flow control and in the newer versions of the Nats Jetstream SDK for all of our different clients um pole consumers are kind of the default under the hood and you don't really have to mess too much with the fetch semantics you could just pretend like it's a good old iterator or consumer or whatever kind of pattern that you're using inside of your language of choice um and so these are this is kind of your first option when you're creating a consumer and I wanted
• 07:30 - 08:00 to lay out why we kind of lean more towards po consumers um for this example the next option that you're going to come across uh or concept is really like durable versus ephemeral um and this can be really confusing at first why we have this because this doesn't really exist in a lot of other um Technologies this this idea of ephemerality um inside of their consumers um but it's actually a really powerful concept and in a couple of the use cases that I lay out where things are ad hoc um it lead to just better
• 08:00 - 08:30 scalability overall because you're not going to have to lean on a you know permanent consumer that you're going to have to manage over the a long life cycle so a durable consumer is for long-lived consumers you know if if you're used to something like Apache Kafka or um or Pulsar or rabbit mq this is going to be your kind of bread and butter you know I create a consumer Nats kind of keeps track of where that consumer is at what messages it's uh what messages it's processed or been delivered to what messages it's
• 08:30 - 09:00 acknowledged and which messages need to be redelivered um and so this uh durable consumer is there for longlived consumers and they usually must be deleted explicitly but this is also changed in a recent version of natat server in 2.9 you can actually set inactive threshold which was which was reserved uh for ephemeral for a long time but now can apply to durable so you can just have you know durable consumer with still an inactive threshold um an ephemeral consumer on the other hand is a consumer that's meant to be more ad hoc something that you just create on the
• 09:00 - 09:30 Fly and um you can just leave alone when you unsubscribe there will be some sort of timeout or an activity threshold where then the server will clean that consumer up so this is great if you're just doing basic operations doing a hog queries reporting maybe anything to do with the browser where browser sessions themselves are ephemeral so you don't want to keep them around um it's good for just spinning up and the cool part about consumers is that they're very very fast to spin up they're very very fast to create and so with gats Emeral
• 09:30 - 10:00 consumers are actually possible to do um in a very scalable way and so we're going to be using kind of both of these models today in our examples next up is filtering again this is part of the bread and butter of why make why Jetstream consumers are just so freaking powerful is uh we get to filter um all of the messages before they go over the wire um in other Technologies especially Kafka and Kafka streams um Kafka streams gives you all this power and being being able to kind of you know
• 10:00 - 10:30 map and reduce and select and filter all of these different messages but all of that has to happen on the client side meaning the broker itself the Kafka broker itself is going to send every single message over the wire and so from architectural standpoint there's not a lot of incentive to have extremely large topics um of a wide breadth of data because it's not reasonable to send all of that data and filter them over on the client side so there's a lot of upfront kind of design work that needs to happen
• 10:30 - 11:00 in a situation like Kafka where um you know the the size of your topics really matter and um because you're sending all of that stuff over the wire this doesn't apply in jet stream which is why when people kind of ask us hey should I create a bunch of different streams for this data set or should I put it all in one stream we usually recommend people try to consolidate a lot of things to one stream because it's much easier to just filter out on the client side doesn't mean you go put all your data in one stream but instead of having a you
• 11:00 - 11:30 know orders shipped topic and an orders deliver topic like you normally would in Kafka you can just have an orders topic and have all of the life cycle there at your fingertips ready to um filter on and it makes it very very easy to then get at that data in any way that you want and so um filter subjects are great for that you can basically use your subject based addressing to filter any way that you want and use wild cards and we're going to use this pretty heavily today um storage is a minor one but uh you can choose how you want your consumer to kind of um be stored whether
• 11:30 - 12:00 you want it in memory or to inherit from whatever the stream has whether it's file based storage or or things like that kind of related to that very quickly consumers are also replicated and have their own raft groups and that's something to keep in mind as your scaling out consumers is that you can set a consumer to an R3 which will keep it very durable and um in the case of Disaster Recovery or or even a node just kind of going down it will still continue to operate still even can you know continue to operate with two NOS down in some very rare circumstances um
• 12:00 - 12:30 but that's something to be thinking about as you're creating your consumers very much for ephemeral consumers you know they're in memory and they are an R1 um because AAL consumers don't really need to stick around for very long and so this is where we start seeing some of the divergences between these design decisions and why it's so flexible to be able to to create both durable ephemeral R1 to R3 inmemory file based consumers and all of those things can coexist um against a single stream which is
• 12:30 - 13:00 beautiful all right let's get into acknowledgements so an acknowledgement is basically you know uh a consumer being able or a client being able to tell the server that this consumer has processed this message they've received it or they've processed it they've acknowledged that they've gotten it so that the cursor can continue to move on the server end um with regards to that particular consumer now there's a couple different policies that you can set for a for a consumer acknowledgement policy you can set it to be explicit which is
• 13:00 - 13:30 the default which means every single message um you know gets acknowledged like this you go act a a um and those will just continue to to move on um these ones are kind of pending meaning that uh you know we we'll talk about Max act pending in a little bit but you you'll have some messages that are sent out and are waiting to be acknowledged um but you'll you'll kind of acknowledge messages one by one in an explicit acknowledgement policy then we have all where we can take a group group of
• 13:30 - 14:00 things and we could just you know receive all of these messages and then we could say yep acknowledge and it's going to acknowledge everything that it's sent um since it's received its last acknowledgement so this could be good for kind of like bass batch ingestion um but it's worth noting that these acknowledgements are all asynchronous they're not going to be blocking necessarily um unless you have kind of a Max act pending you're going to be having non-blocking acts so in general it's good it's generally better to just use explicit for a lot of reasons um but all could be good for
• 14:00 - 14:30 some of those edge cases and then we have the non act policy which can be good for things where you just want to YOLO and um you know may maybe you're doing some sort of reporting it doesn't really matter that you get real acknowledgements that these things are delivered you just want this to show up as fast as possible and be delivered like that um you can set an act policy to none and um that will be your fastest option because you don't have to have roundtrip times back to the server you just get your messages delivered to you um you know uh and and you don't get any
• 14:30 - 15:00 guarantees around that delivery but that's okay because you know that might not be the use case that you're you're going for let's move on to Max act pending this you know is a mouthful and it's uh sometimes hard to understand what it means so when when we are sending messages uh you know to the client via a consumer um we can set kind of an upper Bound for how many messages should kind of be should the server be waiting on to receive acknowledgements and so so if we set a Max act pending to one for example
• 15:00 - 15:30 let's say we've acknowledged 1 through four um we have a Max act pending of one which means the server is going to send number five and is not going to send number six until number five is acknowledged and that's because we have one set for a Max act pending now if this was set to two it would set send five and six and seven would kind of be queued up so by default this is actually sent to 100 which actually makes you know uh by you know out of the box your jet stream consumers can um can really process a lot of things in real time
• 15:30 - 16:00 very very quickly um and you know we can uh we can obviously set this to a lower number one of the reasons you might want to set this to one as an example is if you really want that strict ordering and we'll show that in a very um we'll show that in one of our examples is is how we achieve kind of that strict ordering we don't process anything out of order we get a lot of guarantees around it um but it's also very slow so uh trade-offs here we also have kind of an acknowledgement weight period period which is you know when there are
• 16:00 - 16:30 outstanding acknowledgements how long should the server wait before it decides hey I'm going to start redelivering that message um and so by default this is set to a reasonable setting um but you can set this and configure it to whatever you want but just to kind of show you guys how it works you have acknowledgements you know that the messages that you've already acknowledged and then you have five and six kind of out waiting for acknowledgement and five might be on a timer you know have 12 seconds left where six might have just been delivered and has 30 seconds left um and so when
• 16:30 - 17:00 these timers expire the server has decided that whatever client you know they sent that message to for the consumer um is not going to continue processing the message and um or has timed out or has had an issue and so it's going to attempt a redelivery speaking of delivery let's look at the delivery policy the delivery policy kind of dictates how messages get delivered and particularly for the most part where they start in the sequence and so um by default def we have a deliver policy of all meaning that the
• 17:00 - 17:30 consumer is going to start at the beginning of um its sequence and it's going to you know deliver all of the messages there um and so in this case it's going to start at one and it's going to deliver one two three and and four five six or you know somewhere in the future they haven't actually materialized yet but it's just going to continue to send things on into the future then we have the new deliver policy which is only going to deliver new messages so if one two and three are in the Stream and qualify for the consumer um they're not going to be sent but they
• 17:30 - 18:00 are going to you know start delivering any new messages that come along and this is great for things that maybe need to jump in and just start seeing you know um they don't need to start at the beginning of the stream but in fact they just need to start you know uh where data is coming now and maybe start inspecting stuff great for monitoring or ad hoc you know ephemeral consumers then we have a variation on this which is last which is very similar to new except it will just send you the very last one um which could be again be useful for debugging um is just kind of a nice way to nice convenient way to not
• 18:00 - 18:30 have to look up the last sequence and then pick that one um then we can actually select by a starting sequence so if you know kind of the message sequence for this you know uh particular stream that you want to start on you can pick that sequence and then just continue from there and so um in this case if I do a start sequence of three that's going to deliver three you know and everything in the future then the last two are start time and last by subject start time is great because I can pick a time and say just want to start then uh which can be great
• 18:30 - 19:00 for ad hoc queries or reporting where I might want to say hey yeah give me like the stuff starting from the last seven days or from from an hour ago and we can just you know start our stuff there and we won't have to start at the very beginning and you know move all of those messages through lastly and this is the more interesting one is we have last by subject so this is similar to last but it's going to kind of deliver all of the last messages on a per subject basis which could be really really interesting um and it's kind of one of the the core things that drives like our key Value
• 19:00 - 19:30 Store um which you know it will start delivering all of the last values known for a particular subject so maybe if you're doing some sort of kind of change data capture or uh where you might have like a you know order. updated and you have multiple of those and really you only want to have the last one because that's going to give you the the most upto-date you know information about that order um you can absolutely do that and you're say give me all of the last updated me events or messages from all of the orders um you can pull something like that off very very easily with Last
• 19:30 - 20:00 by subject okay and now we get to our last one which is our replay policy um and this one's super fun it's a little bit of an edge case but it's here if you need it and it can be really fantastic for debugging and so our consumer replay policy by default it's instant meaning it's just going to deliver these messages kind of as fast as they can it's not going to delay any messages for for any reasons um obviously you have flow control on the consumer side that will dictate that but uh for the most part it's going to just try to deliver the messages you know as fast as the
• 20:00 - 20:30 consumers want them to um the original replay policy is going to actually hold some of the messages back and it's going to create timings inside of uh in between each of these messages that are the same timings upon which they were received which means that you're going to be able to kind of play back the messages in the same timings that um that they were originally done in so this is really great if you're like doing maybe click tracking on the web and you want to kind of Replay all of that interaction in a live session or
• 20:30 - 21:00 maybe you want to you know uh Start From A Place In The Stream that was problematic and you want to just replay those Events maybe you got a spike or maybe you had some sort of anomaly um you can easily just create a consumer and start debugging it and again this is this is the the really cool part about all of this is um you know debugging in production uh I know that can be kind of a challenge in of itself but you can you know with the fact that you can create as many consumers as you want in these crazy configurations you can do all sorts of really interesting things by
• 21:00 - 21:30 you know just creating a ephemeral consumer using it as a debug consumer and starting to kind of inspect how things are working under the hood um without affecting you know the stream without having to reconfigure the stream in any way the stream is the stream and the consumers are all the consumers which is really really neat um a couple more features we have Max delivery and back off this all has to do with um how we you know configure how things get redelivered and in a previous episode not too long ago I covered everything related to job cues
• 21:30 - 22:00 priority cues um you know doing exponential back off and stuff like that and so I'm not going to cover this too much in that this video if you want to check out more about Max delivery and back off go check out that video there's a lot of cool practical examples there um and then some miscellaneous stuff you can save some uh some bites Over The Wire by saying just give me the headers maybe I just want to aggregate these things and count them up and I don't need the payloads I just need the messages and their headers um sampling is uh great for being able to admit some
• 22:00 - 22:30 stuff for for monitoring purposes we won't cover that today um just like streams consumers can also have metadata so you can build better tooling around it or you can kind of just set metadata for your own kind of labeling and organization purposes um and an active threshold we covered very briefly but this has to do with whether or not you know if whether it's an ephemeral consumer or a durable when should the server clean that up when the server has detected any sort of inactivity or clients that have not bound themselves to this particular consumer and those are really all of the settings there
• 22:30 - 23:00 might be some minor ones that I'm missing here but this is an overall explanation of everything in the consumer config which I know when we first look at it it can be like super super um overwhelming uh but if you go through this video and you go through these explanations hopefully should' walk away with a much better um kind of mental model of how all of these consumers work and the fact that if it's looking overwhelming it's there because it's flexible um in fact it's way more powerful than all of the alternatives to
• 23:00 - 23:30 gats in this regard um and let's go ahead and dive into some code to show how that flexibility really manifests itself so going back to our very first slide here in the Stream let's kind of cover the use case of this event sourcing consumer because I think this is kind of the more common pattern we might not be as strict but it's really kind of the common pattern for a consumer where it might just be a queue of things that we want to process in a particular order or just make sure that we have a lot of guarantees of around how those things have been delivered um so let's go ahead and write a consumer
• 23:30 - 24:00 that does some of this event sourcing and I'll also kind of show you how this Max act pending really kind of plays a role in the performance of this um of this particular consumer so I have a little bit of code up right now this is all really just boilerplate just setting up my uh my gats it's connecting to gats creating a new jet stream um and then it's calling out to this event sourcing consumer function um and then shutting down and so uh this event sourcing consumer function is actually what we're going to be you know uh deal dealing with today so let's go ahead and start
• 24:00 - 24:30 writing that first thing we're going to do is we're just going to create a little context um via the context package um we'll just use the background context this is because um whoops this is because uh the new jet stream package uh really makes use of the goang uh context package which is great can handle like cancellations and timeouts and everything like that um a lot more easily and then we're going to go ahead and create our consumer we're going to do this by um saying uh jet
• 24:30 - 25:00 js. uh create or update consumer we're going to pass in our context that we just made and then we're going to pass in a jetstream config here we go jet stream consumer config and like I said if you just went and filled out all of these fields uh you would get super overwhelmed because there's so many options but you know we just went over all of these options and we're not going to use all of them we're only going to be using a few of them for this particular thing so let's start with name and durable let's just call
• 25:00 - 25:30 this event sourcing cuz this is going to be our event sourcing consumer and imagine that we're um you know binding to that orders stream that I do have it's live um and we want to just process each and every one of those orders from the beginning in order no filtering and we want to also make sure that we're doing so in order so we're going to set our Max act pending to one that way um we don't attempt to deliver any new messages and so if anything fails you know and tries
• 25:30 - 26:00 to redeliver um it's going to be doing so in the case that like it's only sending out one message at a time and like I said this is going to be extremely slow and so it might not be for everybody um to to be honest I would actually recommend trying to see if you can get away without having a Max act pending of one um and seeing if you know you have ways to kind of reconcile ordering um because it's going to be much faster to do it that way um the kind of rule of thumb that I use by the way when it comes to like having to
• 26:00 - 26:30 reconcile ordering is like a lot of people think that they need ordering but often what is the case is I see that they're kind of pulling an events and they're just putting them in a database like if you're pulling an event and putting it into like a SQL database where um time stamps are you know indexed and uh there there's no kind of like insertion assumption about that particular data um you don't need ordering like you could just insert that stuff um but if you do need ordering let's say you're running a state machine on the other end that's making decisions and you need absolutely explicit
• 26:30 - 27:00 ordering on that end um then that's when you're going to want to kind of use a Max act pending of one but I'll show you kind of the performance um differences between the two in just a sec so we'll set Maxx pending to one um and then we're not going to filter on anything in fact this is going to be our entire configuration for um this this uh particular consumer so um looks like we're missing one other thing oh we need to pick the stream which is our second argument so our stream is going to be called orders and just to show you show you guys what the order stream looks like I'm going to go uh Nats stream info
• 27:00 - 27:30 orders we can see that this stream has 31,6 120 messages and if I just do a quick uh Nats stream or Nat sub stream orders it's going to pull in all of those messages this is actually created an ephemeral consumer for me which is cool um and you can see that we have you know orders that are delivered some are processed some are shipped um but we have them kind of broken up by order ID
• 27:30 - 28:00 by the location um so there's a lot of different ways we could slice and dice this back over to our consumer um now that we've picked our orders consumer and we've set our jet stream config we can now um check for an error and start consuming from this consumer so I can uh simply I'm going to return consumer. consume we're going to use this consume method pass in the context do we need a context no we just need a
• 28:00 - 28:30 Handler uh we're going to pass in a Handler so this just takes a jet stream message and it's going to basically just give us one message at a time um and so we want to make sure that we acknowledge this message and I'm going to do a couple things here I'm going to pull out the metadata for this particular message and if there's an error we should just kind of you know print it and move on I'm going to say um sure and then we'll uh return
• 28:30 - 29:00 we'll just make that a log okay so we have our metad dat and now I kind of just want to print out you know uh the fact that we've received our message so I'm going to go ahead and say let's see log dot print Lin I'm going to say processing message and I'm going to give the uh message metad data so I'm going to say meta do sequence and I'm going to choose the consumer sequence and this is going to show me bound to this cons particular consumer kind of what message have we
• 29:00 - 29:30 processed and this is going to show us that we're now processing these messages in order so it's going to start at you know one and continue on from there um I'm also just going to print out the uh message subject just so we kind of have an idea of how things are chugging along we'll acknowledge it and then we're done that's pretty much everything that we need for this event sourcing consumer so let's go ahead and try it out I'm going to say gun main.go and you could see that we started with message one and we're processing them
• 29:30 - 30:00 one by one now my computer is going back and forth to cinia Cloud so that's going to introduce some latency maybe if this was closer it would process a lot quicker but as you could see because we are processing messages one by one and we are acknowledging them one by one it's going to be kind of slow going from here um which could be totally fine if we're operating kind of like an estate machine um and but if we need this to go faster there's some options here so we can either you know maintain complete ordering here and scale these out via um
• 30:00 - 30:30 partitioning that's going to be a completely different topic for a completely different video um or we could certainly turn Max act pending off if we decided that we don't need that guaranteed ordering so let's go ahead and show you what this performance looks like when we turn Max act pending off just going to comment it out let's go ahead and run this and because this is a durable it's actually going to pick off where pick back up where it left off now because we have this Max Act in um you know set at the default and we're acknowledging things
• 30:30 - 31:00 kind of you know not fully in batches we're still acknowledging them one by one but we're doing so asynchronously and moving on when we can we can kind of see that we're processing this stuff much much faster and this is you know this is all happening with still individual acknowledgements on messages and now we've processed the entire batch of 31,000 messages we are all caught up and so you can kind of see the Major Performance differences between doing strict guaranteed ordering versus hey I'm okay if stuff gets redelivered a little bit out of order because of how
• 31:00 - 31:30 I'm storing things on the other end um reporting is quite different from event sourcing in the sense where we're going to want to have an ephemeral consumer we're going to have we're going to want to set act to all or none because we really don't care about individual acknowledgements here we don't need that those sort of guarantees um and we want to set a start time I'll probably set it to like seven days ago we could also say maybe because we're aggregating these things in counts like we only want headers and so we're going to make this as fast as we possibly can to get all of this data um so we really want to answer the question hey how many orders in the
• 31:30 - 32:00 US shipped within the last 7 days so let's go ahead and write that um what did we call this I called this uh reporting consumer so I'm just going to go ahead and copy my event sourcing consumer and I'm just going to rename it reporting consumer okay um we'll keep a lot of this the same but I'm just going to call this reporting and I'm not going to make it durable this is going to be an ephemeral consumer once thing that you could do for ephemeral consumers is um
• 32:00 - 32:30 if you're going to be you know developing this and you're kind of shutting it down and bringing it back up and you bring it back up too quick it's actually going to you know take the ephemeral consumer that you just bound to which could be wanted Behavior so we keep ephemeral consumers around just for a little bit um because you know we know that there's Network Jitter and sometimes things go online and then quickly back or offline and then quickly back online um but if you don't want that behavior you can put a little bit of Randomness inside of your name um so you can have you know every time you boot up a fresh instance it's going to
• 32:30 - 33:00 be pulling in a new or creating a new ephemeral consumer there you can also create the use the create consumer um method here and that will obviously error out if there's any um any particular uh conflicts with naming okay so what we want to do with this uh ephemeral consumer called reporting is uh quite a bit different than what we did inside of the event sourcing one so we want to say headers only just to save some bites Over The Wire we're also going to say Max batch request um yeah Max request batch we're
• 33:00 - 33:30 going to bump this up to a th um and this means when it goes to fetch things it's going to be like hey give me you know up to a thousand messages here um this just is just going to make getting messages much much quicker I'm going to set the ACT policy to um act none let's see act nonone policy um because we don't need any acknowledgements for this particular reporting thing we're going to set the delivery policy to deliver by start time
• 33:30 - 34:00 so we're going to say Jetstream deliver by start time policy and then we need to set which start time we want which is going to be with the opt start time and um we need to actually create a time here so I'm going to say T is time we'll import the time package and we'll say for seven days so we're subtracting 7 days from time. now so we're saying seven days starting from seven days ago um and this needs to be like that there we go and uh lastly we want to
• 34:00 - 34:30 filter on a particular subject so I'm going to add my filter subjects here and this is going to be a string you could actually pass multiple filter subjects this is new in a recent version of natat server we can actually pass in multiple you know subjects here so if for some reason I wanted like all the orders in the US that uh have been shipped and all the orders in Canada as well I
• 34:30 - 35:00 can also do that which is just superb I love that you're able to do something like this but for now we're going to say give me all the orders in the US that shipped within the last 7 days and um and that's what it should give us in a very very efficient way um so let's go ahead and try this out and see if it works the way that we hope um before we do that I'm going to remove that acknowledgement because we don't need to be sending Axe and let's see um how all of this stuff works oh before we do I forgot to enable
• 35:00 - 35:30 this in at the top so we'll enable our reporting consumer disable our event sourcing consumer let's go ahead and run this and see what it does all right so we've already processes processed all of our messages which is great just a reminder that this is also just coming from Cadia Cloud itself and so maybe you're running a front end where you're doing some reporting and you want to be like give me all of the orders that you know shipped in the US within the last 7 days and it gave me all 1620 of them is this
• 35:30 - 36:00 going to be as fast as a SQL query if you're doing aggregation and you want to do the count on that side no it won't be but it's very versatile and you can express a lot of things without having to move all the stuff into a separate data store and so for ad hoc queries and for being able to kind of do stuff like this this is a pretty good solution um the other thing I want to mention is this isn't just a query but this is also going to give me any new you know newly shipped orders that come in via the us so you can kind of keep this around and have a live view around it so a lot of really interesting things that you could
• 36:00 - 36:30 build um with a structure like this okay we're on to our last example so let's kind of take a look at uh what that one looks like our last example is lookup meaning let's say I just did the reporting feature and it gave me all of the shipped orders and now I want to kind of double click on an order um look at its ID and basically give me all of the statuses that that order has had um so we're going to be using a couple different configuration settings to do this in a really efficient way um
• 36:30 - 37:00 starting with you know the the filter we're going to make it an femal consumer no ax and we're going to use last per subject instead of time here um you know it might not be as applicable to this one but let's say that you're doing any sort of change data capture last per subject is comes in clutch as a option here so let's go ahead and make our lookup consumer I'm going to start by just enabling our lookup consumer here and I'll copy our reporting consumer and I'll just call this one lookup
• 37:00 - 37:30 consumer we don't need the time we're going to set our deliver policy to deliver last per subject policy we'll remove kind of Max batch and headers only maybe we want the um the payload here and I'm going to call this a lookup and instead of looking at all the shipped us orders I actually want to just look up a order based off of its ID which is this token right here and I'm
• 37:30 - 38:00 going to go ahead and copy this token and paste it in right here this is an order that I know has some statuses to it um and this is just an example we can of course insert this dynamically if we wanted to in our code um but then this is a consumer setup that should give us in a very efficient way kind of the messages related to this order um and it's going to deliver last per subject now let me show you something really cool is we could actually use deliver last as well if we just wanted the the last um status update for this particular order so if we want to know
• 38:00 - 38:30 hey I have this order number and I want to know whether it's been shipped or whether it's you know whe whether it's been delivered um or whether it's you know currently processing things like that um we can use this so I'm going to say gun main.go and you can see it's only going to give us one message um but it's also going to be kind of hanging on to see if there's any new status updates here but we know that this has been delivered so maybe you're running an e-commerce place and you need to look up the last status you can do that by creating this ephemeral consumer now an
• 38:30 - 39:00 even more efficient way to do that and something we'll cover in the another episode we could actually use the direct get API call to do this um if especially if you just need that one message in future versions of the natat server we're actually introducing a batch version of the direct get me uh API call that allows you to do you know a lot of the things that we're doing today uh but through a single API call and no no having to create a consumer so it's going to be even even faster to get at that particular set of data now back to the last per subject we
• 39:00 - 39:30 could put this in here and this is going to give us all of the statuses um in order uh that they were received for this particular order and there we go we know that um you know this order was pending we have a time stamp for it then it was confirmed then it was shipped and then it was delivered so if you need to kind of like look up an order and show its like life cycle you can do that very very easily by creating consumer like this um once again we should be able to do this in the future with just a single
• 39:30 - 40:00 jet stream API call like a request reply um and that's going to be glorious when we can and I'll make a video about it when um when it's time but that pretty much covers all of the examples here um like I said before all of this code is going to be available via GitHub um the presentation available via GitHub uh and so we I'll include a link below um but as always thank you so much for watching I hope you enjoyed this content and if you do have any um suggest for Content that you'd like to see in the future be sure to comment down below and we'll put
• 40:00 - 40:30 that in our backlog it's always fun being able to explore Nats and jet stream with you all so um everybody have a great day and I'll see you next [Music] [Music] time