So You Want to Build An Event Driven System? - James Eastham - NDC London 2024

Estimated read time: 1:20

Summary

In this engaging talk at NDC London 2024, James Eastham shares his journey of building event-driven systems and offers insights into the practical implementations of event-driven architecture. He emphasizes the significance of reversing dependencies, reducing coupling, and embracing asynchronous systems to enhance integration and communication between services. Eastham highlights the importance of language and governance in evolving systems and sheds light on the distinction between event-driven and event-based systems. He also discusses various event types, such as notification events and event-carried state transfer, and shares patterns like the use of metadata and CQRS to optimize system performance.

Highlights

James Eastham shares insights from his journey of building event-driven systems, emphasizing real-world applications. 🚀
He highlights the importance of stories and language in human and system success, relating this to event-driven architecture. 📖
Discusses the practical aspect of reducing coupling and embracing eventual consistency for system resilience. 🔄
Explores different event types and their implications on system design, like notification events and event-carried state transfer. 📊
Introduces patterns and solutions like metadata use, governance practices, and CQRS for optimizing performance. 📈

Key Takeaways

Event-driven architecture focuses on communication between systems, reversing dependencies and reducing coupling. 🌐
Language and words are crucial in both human and system communication, making event-driven architecture incredibly powerful. 💬
Embracing asynchronicity and eventual consistency is essential for building flexible and resilient systems. 🕒
Governing events and managing schema changes are critical to maintaining system stability and evolvability. 📜
Starting small and integrating events even in monolithic architectures can smooth the transition to event-driven systems. 🌱

Overview

James Eastham's engaging talk at NDC London 2024 delves into the depths of event-driven systems, sharing his personal experiences and learnings along the way. As a senior cloud architect at AWS, Eastham explores the intricate dynamics of event-driven architecture, understanding its core as a communication pattern among systems.

The talk covers essential concepts like reducing coupling, embracing eventual consistency, and understanding different types of events. Eastham emphasizes the crucial role of language in successful integrations and challenges us to think about the governance of events and managing schema changes for evolvability in systems.

Moreover, he introduces practical patterns like the use of metadata, CQRS, and asynchronous commands to enhance performance and flexibility. Eastham's discussion provides not only a technical perspective but also a human touch, inviting us to reflect on how we communicate and innovate with event-driven architectures.

Chapters

00:00 - 00:30: Intro to Human Success and Stories The chapter titled 'Intro to Human Success and Stories' explores concepts from Yuval Noah Harari's 2014 bestseller, which delves into the brief history of humankind. A key takeaway from the book discussed in this chapter is the pivotal role of stories in human civilization. It highlights how storytelling and the words we use are fundamental to our success as a species, particularly emphasizing our collective ability to unite behind shared ideas through narratives.
00:30 - 01:00: Discussion on Event Driven Architecture The chapter opens with an intriguing comparison between human organization and the concept of Event Driven Architecture (EDA). The speaker makes a parallel between how human societies organize themselves around shared ideas and rules, drawing a comparison with EDA’s operation in which events dictate the control flow rather than a predetermined sequence of operations. This unique perspective aims to provide a better understanding of EDA by connecting it to familiar concepts of societal organization.
01:00 - 01:30: Introduction to Event Driven Architecture Concepts Event-driven architecture is a communication pattern for integrating systems.
01:30 - 02:00: Personal Experience with Building Systems The chapter titled 'Personal Experience with Building Systems' starts with the speaker acknowledging their past mistakes in building systems, emphasizing a shared experience of failure in the field. The speaker aims to create a relatable and honest atmosphere, encouraging openness about failures within the audience. The talk is meant to be straightforward, focusing on the basics of system design, rather than complicated details, and invites the audience to share in the recognition of common industry challenges.
02:00 - 02:30: Pizza Company Example The chapter kicks off with a reflection on the value of failure as a teacher in the business world. The speaker recalls advice from a mentor emphasizing that more lessons are drawn from failures than successes. They recount experiences of attending pre-sales meetings, where they would candidly share with potential clients how past failures had shaped their expertise, asserting that they perform well precisely because they had erred more extensively than the clients. The central theme of this chapter is the acknowledgment of past mistakes as a foundation for adopting and excelling in innovative practices like event-driven architecture. The audience is encouraged to engage with the material with an open and imaginative mindset, as the speaker prepares to talk about the challenges and hurdles encountered when implementing such an architecture.
02:30 - 03:00: Issues with Tight Coupling in Systems The chapter discusses issues related to tight coupling in systems, using a fictional example of a company called Plant-Based Pizza. The company has adopted microservices that communicate using HTTP request-response style communication. While functional, this setup is leading to significant problems, illustrating the broader challenges of tight coupling in a system.
03:00 - 03:30: Different Types of Coupling This chapter discusses the concept of coupling in software design, specifically in the context of a pizza delivery company's order processing system. The order processing service is described as the core domain, or the essential part, of the company because customers' ability to place orders is crucial. However, the system faces challenges because this service depends on several other services such as pickup, delivery, and kitchen services, and plans to add a loyalty points service. The chapter highlights how each of these dependencies represents a form of coupling, which can complicate system management and affect efficiency.
03:30 - 04:00: Event Driven Architecture Benefits The chapter discusses the benefits of event-driven architecture with a focus on improving system design by reducing coupling and enhancing cohesion. It revisits the well-known software principle of aiming for high cohesion and low coupling, explaining that while breaking systems into microservices has addressed cohesion to some extent, coupling remains a more complex issue to resolve.
04:00 - 04:30: Practical Use Case - Loyalty Points Service This chapter discusses different types of coupling in service communication, particularly focusing on location and runtime coupling in request-response style communication. When making a request, it's essential to know the location of a specific service, whether it be an IP address or a domain name. Other types of coupling mentioned include semantic coupling and data format coupling, which involve considerations like binary data, JSON, null value handling, and string casing.
04:30 - 05:00: Coupling Issues with Request-Response Systems This chapter discusses the concept of coupling in request-response systems, particularly in the context of microservices. It highlights the necessity of some degree of coupling to ensure functionality while cautioning against excessive coupling that could introduce rigidity and complexity. The chapter references a quote by Gregor Hy that emphasizes the 'appropriate level of coupling depends on the amount of control you have over the endpoints.' The discussion focuses on understanding and balancing the types of coupling, such as naming conventions and endpoint dependencies, to maintain flexible and functional systems.
05:00 - 05:30: What it Means to be Event Driven The chapter titled 'What it Means to be Event Driven' explains the significance of managing microservices and reducing coupling when integrating with external systems. While internal systems can handle tighter coupling due to controlled release cycles and schemas, it's crucial to minimize this coupling with external systems like third-party SaaS products or other internal systems that are not under your control. This is essential for smoother integration and management.
05:30 - 06:00: Definition of an Event The chapter discusses the importance of understanding why event-driven architecture can be beneficial, especially when dealing with issues like coupling in an organization. It emphasizes starting with the 'why', particularly in software development. An example given describes the implementation of a new service, such as a Loyalty Point service, and the need to integrate it with existing systems, highlighting the challenges and strategies in managing changes across different services.
06:00 - 06:30: Event Driven vs Event Based Systems The chapter discusses the distinctions between event-driven and event-based systems. It kicks off by examining a scenario involving a 'Lo Point service,' which adds loyalty points. The author questions whether certain services, like the order processing service, should be concerned with loyalty points collection. This questioning extends to other system components, such as the kitchen service and pickup service. The chapter probes the necessity and logic behind whether the order processing service should trigger multiple actions like loyalty point addition, kitchen service activities, or pickup service tasks. It acknowledges that while it's crucial in some scenarios, it isn't always applicable. The discussion highlights the complexity in defining when a service should be event-driven, indicating potential inefficiencies or redundancies if all components respond to all events.
06:30 - 07:00: The Concept of Events as First-Class Citizens The chapter discusses the potential drawbacks of using HTTP request-response communication in microservices, particularly focusing on the dependency between services. It points out that if the Loyalty Point service is offline, it can negatively impact the order processing service, even though it is not part of the core domain. The chapter acknowledges that this explanation simplifies the problem but suggests that there are ways to build resilient microservices to avoid such issues.
07:00 - 07:30: Types of Events and Examples In this chapter, the focus is on understanding the concept of events, particularly in the context of event-driven architecture. The chapter begins by discussing circuit breakers and the potential for minimizing disruptions through event architecture. It then highlights the importance of language and ensuring a shared definition of terms related to events. The chapter seeks to demystify what it means to be 'event-driven,' acknowledging the buzz surrounding the term and offering a clear explanation to facilitate understanding.
07:30 - 08:00: Practical Implementation of Event Systems The chapter titled 'Practical Implementation of Event Systems' discusses a core concept in event-driven architecture: reversing dependencies. This involves changing the direction of integration between services. Instead of a central service (like the order processing service) needing to be aware of and manage its interactions with all downstream services, those downstream services instead react to events published by the central service. This shift is described as being 'event-driven,' promoting a more decoupled and responsive system architecture.
08:00 - 08:30: Explaining Fat vs Sparse Events In this chapter, the concept of an 'event' is explained as an immutable fact – something that has occurred in the past and cannot be altered. An example used is a light switch. Turning on the switch generates a 'light switched on' event, which is unchangeable, as once the light is on, it cannot be un-switched. This illustrates the persistent nature of events.
08:30 - 09:00: Basic Components of Event Driven Architecture The speaker discusses the fundamental aspects of event-driven architecture through a simple analogy of turning a light switch on and off. Events such as 'light switched on' and 'light switched off' are explained as irrefutable occurrences that cannot be altered once they've happened. The speaker suggests that there is a distinction between event-driven and event-based systems, which, although might seem semantic, holds significance in understanding how event-driven systems function.
09:00 - 09:30: Publish-Subscribe Pattern and Responsibilities This chapter discusses the key distinction in event-driven systems, emphasizing that system functionality is driven by business events. The speaker reflects on the history of building event-based systems, sharing their own experience starting with Windows Forms. They mention various events like button clicks, mouse hovers, and window resize as triggers for functionality, illustrating how system actions are initiated in response to these events.
09:30 - 10:00: Protecting Systems from Overload The chapter discusses the implementation and importance of protecting systems from overload by using modern architectural practices. It introduces the concept of event-driven systems, which are crucial for efficiently handling processes and tasks in contemporary business models. Examples of such events include 'order confirmed', 'pizza boxed', or 'staff member clocked in', which could be unique to specific businesses. The chapter emphasizes that simply triggering functionality from events, like an S3 object creation, doesn’t qualify as a truly event-driven system. True event-driven systems are characterized by their use of unique business events that align with the specific workings of the business.
10:00 - 10:30: Benefits of Event Driven for Communication The chapter discusses the importance and advantages of event-driven architecture for business communication. It highlights how this approach allows businesses to craft unique sets of business events that align with their specific needs and narratives. By adopting event-driven architecture, businesses can create a shared language that bridges technical and business aspects, enhancing communication and implementation. The chapter also touches on the concept of events being 'first-class citizens' in this architecture, addressing common skepticism about this terminology.
10:30 - 11:00: Starting Practical Modeling of Integration The chapter begins with a discussion on the concept of events being 'first-class citizens' in a system. It questions the audience's familiarity with the concept, linking it to the widely acknowledged idea of API-first design. This design philosophy emphasizes creating the API before anything else, establishing an interaction point to facilitate system integration. When systems are designed in this way, an agreed-upon contract exists to connect them smoothly. The chapter proposes applying this principle to Event Systems, treating events with the same level of priority and importance as APIs in terms of integration and interaction.
11:00 - 11:30: Explaining Event Storming The chapter titled 'Explaining Event Storming' focuses on the initial design phase of a system using event-first design. Event-first design emphasizes the importance of designing events as they serve as integration and coupling points within the system. There is a variety of event types present in systems, each serving different purposes. The first type highlighted is a notification or sport event, characterized by its simplicity and small scale.
11:30 - 12:00: Evolvability in Event Driven Systems The chapter discusses the concept of evolvability in event-driven systems, using a plant-based pizza service as an example. It explains how events can be used to integrate different services efficiently. Specifically, when someone places an order, an 'order confirmed' event is published, notifying the kitchen service. The kitchen receives this event along with an order ID, which allows it to determine what needs to be prepared. This approach highlights the lightweight and flexible nature of event-driven architectures.
12:00 - 12:30: Event Schema Governance The chapter titled 'Event Schema Governance' discusses a system integration scenario where a kitchen service interacts with an order processing service. The kitchen service inquires back to the order processing service to retrieve details of an order it needs to cook. This callback mechanism is praised for its functional efficiency. The process involves receiving a small, lightweight event, obtaining additional information through the callback, and subsequently integrating additional downstream services. The narrator shares personal experience, noting the effectiveness of implementing such a system in real-world applications.
12:30 - 13:00: Metadata Pattern for Events The chapter titled 'Metadata Pattern for Events' focuses on the challenges of event-driven architectures when it comes to handling downstream services. It explains how downstream services can become aware of each other when they call back to get more information, leading to potential system failures. The core principle discussed here is that event-producing services should remain unaware of their consumers to maintain an efficient and robust architecture.
13:00 - 13:30: Handling Breaking Changes In the chapter titled 'Handling Breaking Changes,' the focus is on exploring alternate approaches to managing events, particularly in the context of software design or systems architecture. The chapter introduces the concept of different types of events, emphasizing 'event carried state transfer' or 'fatter events.' This is contrasted with more typical 'spar events' or 'notification events,' where an event such as 'order confirmed' might only carry a simple order ID. In contrast, events that carry state transfer include much more data, showcasing a more comprehensive message structure that might be employed in managing system changes or updates.
13:30 - 14:00: Fat vs Sparse Events Choice The chapter titled 'Fat vs Sparse Events Choice' delves into the decision-making process of choosing between fat and sparse events in event-driven architectures. It starts by explaining how having more information in an event can be beneficial for downstream consumers, allowing them to perform functionalities without needing additional callbacks. The example given is the kitchen receiving a comprehensive order list, enabling them to know exactly what to cook. The chapter then poses a critical question: should one opt for fat events, which contain more data, or sparse events, which contain minimal data? This question is left open for further exploration in subsequent parts of the chapter.
14:00 - 14:30: Asynchronous Nature and Consistency This chapter introduces the concept of an asynchronous communication model involving three main components: a producer, a consumer, and an event broker. The broker, which could be a bus, queue, stream, or topic, manages the routing of events between the producer and consumer. The key observation is that the producer and consumer are unaware of each other's existence, with the broker facilitating communication between them. The chapter emphasizes the core integration pattern used here, which is the publish-subscribe pattern, wherein the publisher produces events.
14:30 - 15:00: Real-World Examples and Challenges This chapter delves into the intricacies of managing event-driven architectures, focusing on the role of event producers who, despite detachment at runtime, must adhere to established schemas to minimize semantic coupling with downstream consumers. The importance of maintaining agreed-upon event formats is stressed to ensure seamless interactions across loosely coupled systems.
15:00 - 15:30: Integrating with Non-Event Systems The chapter discusses the responsibilities of a producer when integrating with non-event systems. A key point emphasized is the importance of minimizing breaking changes and adhering to the agreed-upon contract regarding events. Notably, the producer is not responsible for how their events are used by downstream consumers, unlike with HTTP where certain constraints apply. This division of responsibilities allows for greater flexibility and focus on the core task of publishing events.
15:30 - 16:00: Observability in Event Driven Systems Chapter: Observability in Event Driven Systems This chapter emphasizes the importance of understanding constraints in both HTTP request handling and event-driven architectures. In traditional systems, when making HTTP requests, it's crucial to be aware of the throughput and rate limits of downstream systems, as well as the API formats, as these become the responsibility of the service making the request. However, in event-driven systems, the responsibility of understanding constraints shifts to the subscriber, aligning more logically with the architecture's principles. This shift requires a new approach to observability, focusing on the subscriber's ability to manage these constraints effectively.
16:00 - 16:30: Managing External System Requests This chapter discusses the importance of managing external system requests, particularly in the context of event-driven architecture. It emphasizes that subscribers to events should take responsibility for controlling their ingestion rate and handle the load on their systems carefully. The chapter warns against directly connecting systems to the event bus without considering the speed and nature of the events being published by producers, as this can lead to system overloads.
16:30 - 17:00: Commands, Events, and Queries This chapter discusses best practices in event-driven architecture, particularly focusing on the role of subscribers. It emphasizes the importance of decoupling your application from the event bus to maintain control over your systems. Instead of directly connecting your application's compute resources to the event bus—thereby tightly coupling your system to an external source—it is better to use a queuing system or durable storage. When an event hits the event bus, it should first be stored in this queue or storage system. This approach allows you to manage the flow and processing of events more effectively and ensures that your system remains robust and manageable.
17:00 - 17:30: Asynchronous Commands and Their Management The chapter discusses the management of asynchronous commands in software architecture, focusing on the use of queues to handle tasks at suitable times. This approach not only optimizes technical communication between systems but also improves human-level communication.
17:30 - 18:00: Integrating CQRS with Event Architecture The chapter discusses the integration of Command Query Responsibility Segregation (CQRS) with event-driven architecture. It highlights how event architecture facilitates communication within teams by simplifying discussions about system behaviors, like handling order-related events. This approach minimizes confusion typically associated with technical terms such as microservices or REST APIs, making interactions more intuitive. Event-driven architecture (EDA) is emphasized as a communication pattern that models system behavior effectively, enhancing clarity on what actions should follow specific events, such as order cancellations or completions.
18:00 - 18:30: Detailed CQRS Flow Explanation The chapter begins with discussing the current integration process within a CQRS (Command Query Responsibility Segregation) system for a pizza business. Initially, the order processing service receives a request and communicates with the kitchen to cook the order. After completion, the kitchen updates the order processing service, which in turn sends an HTTP request to the delivery service to proceed with delivery. The current focus is on a practical model for the existing event-based system.
18:30 - 19:00: Maintaining Consistency and Handling Failures The chapter titled 'Maintaining Consistency and Handling Failures' discusses the intricacies of a new loyalty point service. It highlights the significant amount of coupling and runtime dependency that exists even in a basic example of this service. The chapter focuses on strategies to eliminate these dependencies as the organization moves towards an event-driven architecture. A central question posed is how to comprehend the events within a unique business system, acknowledging that every business has its own distinctive set of events.
19:00 - 19:30: Using Transactional Outbox or CDC Pattern The chapter discusses the importance and utility of event storming in the context of event architecture. It highlights the concept of event storming as a method to explore and analyze domain events within a system. The method, popularized by Alberto Brandolini, is explained as highly beneficial for understanding and designing complex systems. The chapter provides an example of event storming present in the environment, illustrating its practical application.
19:30 - 20:00: Bringing All Elements Together The chapter discusses the importance of increasing collaboration across Sil Lord boundaries, highlighting the usefulness of event storming in designing clean and maintainable event-driven software. Such software is crucial for supporting rapidly evolving businesses, which many readers may find relevant given the constant changes in their systems. While the chapter doesn't delve deeply into event storming, it acknowledges its significance in addressing business and system changes over time.
20:00 - 20:30: Advice for Starting with Event Driven Architecture The chapter titled 'Advice for Starting with Event Driven Architecture' explains the process of event storming. It is conducted in a workshop format where all stakeholders, including business and technical people, are gathered in a room. Everyone participates by using Post-it notes to identify and mark the events occurring within the system. Events such as 'order created', 'payment processed', and 'order delivered' are explored. Stakeholders also use Post-it notes to outline the commands that trigger these events and the queries related to them. This collaborative approach helps in gaining a thorough understanding of the system through these interactions.
20:30 - 21:00: Concluding Thoughts on Communication The chapter "Concluding Thoughts on Communication" emphasizes the importance of keeping conversations business-focused, especially during discussions about technical complexities like using Kubernetes. It underscores the value of using a unified language across the entire business, which is one of the core outcomes of an event storming session. This unified language bridges gaps between technical and non-technical parts of the organization, ensuring everyone remains aligned with business objectives.
21:00 - 21:30: Closing Remarks The chapter "Closing Remarks" discusses the concept of evolvability in event-driven architecture. It explains that while the architecture allows for a flexible system where new services can be added seamlessly to the event stream, the reality is often more complex. The common notion that simply linking new services to event streams results in a perfectly evolving system is overly simplistic. The chapter highlights the importance of understanding the intricacies involved in making such architectures truly evolvable.

So You Want to Build An Event Driven System? - James Eastham - NDC London 2024 Transcription

00:00 - 00:30 all right in his bestselling 2014 book Yuval Noah Harari discusses a brief history of humankind and one of the core ideas from this book is the importance of Stories the words and stories we use as humans is the master key to our success as a species our ability to get our El together behind an idea whether
00:30 - 01:00 that be a religion the nationality you are or the country the company you work for and the values that company has our ability to get ourselves behind an idea at scale is what differentiates us from other species now some of you might be really confused at this point thinking that you're at NDC London why are we learning about a history of humankind and that's okay if you're furiously checking your agenda you are in the right place this is going to be a talk about event of an architecture the reason start there is
01:00 - 01:30 because event Ren architecture is fundamentally a communication pattern a way of modeling the integration between two systems and that's what you're here to learn about today we're going to talk about some of the Practical implementations of ative in architecture we're going to start from some of the beginner Concepts and work through some more intermediate patterns I'm James East I'm a senior Cloud architect at AWS and for all of you non AWS people people in the in the
01:30 - 02:00 room don't worry this is not going to be death by ABB service icon in fact this is just a really simple talk really simply about boxes and lines and I want to start today by being really really honest with you all I've built systems in the past and I've got them terribly terribly wrong lots and lots of times and that might make you think this is a really odd way of starting a talk admitting that I've really badly built event and systems how many people here have built something that's failed at some point you can be honest this is a safe space yeah
02:00 - 02:30 everybody okay good and how much did you learn from that failure one of my old mentors always said to me you learn so much more in Failure than you do in success we literally used to walk into pre-sales meetings with customers and say we will do this well because we've gotten this wrong so many more times than you have so that's what we're going to talk about today I'm going to try and show you some of these challenges some of these things you might hit as you start to adopt event of an architecture and for the rest of this talk I want you all to put on your imagination hats cuz the something I
02:30 - 03:00 love talking about almost as much as event of architecture and serverless and that is pizza who else loves pizza for anybody who's still got their hand down really for the rest of this talk you are now all Developers for a completely fictional pizza company called plant-based Pizza you've already adopted microservices these microservices are communicating right now over HTTP request response style communication this works well it's functional but it's giving you an awful
03:00 - 03:30 lot of challenges that's because you've got this order processing service that's the core domain this is fundamental to your application if people can't place their pizza orders then really what is the point of a pizza delivery company and this order processing service has a dependency on all these other services on the pickup service the delivery service the kitchen and you also need to add a new loyalty Point service to allow people to build up loyalty points it would be amazing if pizza places did this because I'd have so much free pizza by now then this really is a problem of coupling that you've got here all these
03:30 - 04:00 services are really tightly coupled together over HTTP and I'm sure you've all heard the adage in software before that you should aim for high cohesion and low coupling and cohesion you've kind of already solved a little bit cohesion is the idea that things that things that change together should live together and you've got microservices you've broken your system down into these small independently Deployable pieces so you've kind of solved that coupling however is a slightly more nuanced problem and that's because there's lots
04:00 - 04:30 of different types of coupling you've got location and runtime coupling that's primarily what you're dealing with with request response style communication at the time you need to make a request you need to understand the location of a specific service whether that's the IP address the domain name but you've also got other types of coupling you've got semantic coupling data format coupling are you using binary data protuff Json how do you handle null values ins strings what casing are you using in
04:30 - 05:00 yourj on is it camel case are you using high and is it some weird combination of underscores that only makes sense to you in your business these are all different types of coupling that you need to work with and of course you need some coupling without coupling it will be really hard to make these microservices actually do anything and there's an appropriate level of coupling to aim for and I really love this quote from Gregor Hy an appropriate level of coupling depends on the amount of control you have over the endpoints and what I think Greg means by that is that if you are working on a
05:00 - 05:30 team that maybe manages two or three different microservices coupling them Services together more tightly isn't necessarily a bad thing because you control the release cycle you control the SLA you control the schema of these systems but once you start integrating with a system outside of your control you want to try and reduce this coupling as much as possible whether that be a third party SAS product that exposes an API or another system inside your
05:30 - 06:00 organization so if it's coupling that's the problem and there's an appropriate level of coupling to aim for why does it help to be event driven starting with why is always a really important thing to do in software Lea talked about that in her keynote yesterday so let's come back to this use case that you need to implement now you need to add this loyalty Point service to your system so you build the Loyalty Point service microservices you deploy it independently excellent but now you need to go and make a change to your order processing service to actually tell the AUD processing service to make
06:00 - 06:30 the call to the Lo Point service to add the Lo points and when you think about that is that really logically the right thing to do should the order processing service care about the fact that people are collecting loyalty points probably possibly not and you can make that same argument for other parts of the system as well the kitchen service the pickup service should the order processing service actually C that all these things need to happen in some cases yes absolutely but in not all cases there's another problem here if
06:30 - 07:00 you run this communication over HTTP request response you've got that runtime coupling the Loyalty Point service needs to exist at the time the order processing service is doing its work what happens if this happens the Lo Point service goes offline now you've got a service that's not in the core domain it's not the core part of the the order processing floww but it has the potential to take your order processing service offline now I realize I'm massively oversimplifying that it is possible to build microservices in this way that are resilient you can use things like
07:00 - 07:30 circuit breakers and back off so that is possible but event of an architecture can help you minimize that so before we get into some of the technical patterns let's just take a step back for a second just talked about the importance of language and words so let's ensure we have the same shared definition of what all these things mean what does it actually mean to be event driven sounds a little bit buzzworthy right event we're all talking about event architecture nowadays the most succinct way I found of putting this is something I've taken
07:30 - 08:00 from one of Martin Fowler's talks on the topic and its core event of an architecture is about reversing dependencies reversing the lines of integration so now instead of the order processing service needing to know where all these services are all these Downstream services are now reacting to events that the order processing service is publishing so that's what it means to be event Ren it's about reversing dependencies to take a step back even further what actually is an
08:00 - 08:30 event what is an event in my opinion an event is an immutable fact it's something that's happened in the past it cannot be changed the example I like to use here is that of a light switch it's February we're in London it's dark it's horrible you get back to your hotel room at night and it's pitch black and you need to get some light in the room so you can see what you're doing so you hit the light switch and that race is a light switched on event you can't un switched on a light
08:30 - 09:00 what you need to do to take the light out of the room is to hit the light switch again and that will give you a light Switched Off event the light will leave the room light switched on light switched off they're irrefutable they can't be changed they have happened so if that's the definition of an event what does it mean to be event driven and I think there's Sometimes some confusion here between event driven systems and event based systems and that might sound like semantics but I think
09:00 - 09:30 there's a key distinction and the event driven system is where your system functionality is driven by business events and that's important because we've been building event based systems for years and years I started in software building with Windows forums who remembers Windows forms who's still managing Windows forms who's still building a Windows forms some people amazing wow and event Windows forms you triggered functionality using things like button clicked Mouse hovered window resized and then something happened after that some functionality
09:30 - 10:00 executed and even Building Systems in more modern ways you can do the same thing if you drop a file into an Amazon S3 bucket there's that first service name that will raise an S3 object created event and you can trigger functionality off these events but that doesn't make it event driven an event driven system uses events like these order confirmed Pizza boxed staff member clocked in these are events that are unique to your business you could look at two identical businesses in the exact same
10:00 - 10:30 domain and whilst they might both be triggering functionality like of things like S3 object created they will have a fundamentally different set of business events that are unique to your business you're letting the language the story of your business Drive its technical implementation and that's what event liven architecture can give you this share language the other thing you might hear when you learn about event of Architecture is that events are first class citizens that sounds like another buzzword right
10:30 - 11:00 what does that actually mean for an event to be a first class citizen in a system how many of you have heard of the idea of API first design yeah some people have some people haven't API first design is this idea of when you start building a system the first thing you do is design the API you design the point of interaction the integration so when you want to glue systems together you have this agreed contract so with Event Systems events being first class cens means that you
11:00 - 11:30 default to event first design the first thing you want to do is actually design your events because they are the point of integration they are the point of coupling that you have in your system and it's not as simple as just adopting events there are lots of different types of events you'll likely see in your systems the first of those is a notification event or what can be known as a sport event this is a really small really simple
11:30 - 12:00 packet of data that simply notifies other systems that something has happened they typically quite lightweight and this is where you start building out this event system for plant-based Pizza you introduced invents for the integration between the order processing service and the kitchen service and you publish an order confirmed event so somebody places an order they're waiting for the pizza the kitchen receives this order confirmed event it has an order ID and the kitchen goes fantastic what do I actually need to cook what's actually on this
12:00 - 12:30 order so then the kitchen service calls back to the order processing service says excuse me Mr order processing service what's actually on this order what do I actually need to cook and the order processing service responds and everyone's happy right and this is a system I've actually built before I've done this exact same integration before in the past and it works really well functionally because you receive the event it's nice and small it's lightweight and you call back and get some more information and then you add another set Downstream service and then you add some more Downstream services
12:30 - 13:00 and some more Downstream services and some more Downstream services and eventually all of a sudden this happens everything falls over because now you've got this potentially infinite number of Downstream services that are all making calls back to get more information about that specific order yeah a core principle of event an architecture is that Services producing these events should be completely unaware of their Downstream consumers but yet if every single one of these Downstream consumers needs to call back they are kind of aware
13:00 - 13:30 right so is there an alternate approach you can take here and there is and this comes back to this idea of different types of events another type of event you will see is event carried State transfer or a more fat event as it can be commonly known so this is an example of a spar event or a notification event you've just got an order ID so order confirmed just an order ID with event carried State transfer are fatter events that event will look a little something like this so you see you've got a lot more
13:30 - 14:00 information here now you've got more information that the downstream consumers can use to actually perform some functionality without needing that call back so the kitchen can consume this event it's got the list of items on the order so now it knows what it needs to cook perfect Happy Days right so which one of these is the right one to choose do you use fat events do you use spice events hold that thought we'll come back to that in just a second stay with the theoretical just for a minute at its core an event of an architecture is made
14:00 - 14:30 up of three parts you have a producer a consumer and some kind of event broker in the middle that might be an event bus equally you might see a queue or a stream or a topic there's just some kind of broker in the middle that's in charge of rooting these events from one place to another and the first thing you'll notice is that the producer and the consumer here have no idea each other exists the communication is managed by the broker the core integration pattern at work here is the publish subscribe pattern you have a publisher producing
14:30 - 15:00 events and you've got n number of Downstream consumers that are all consuming these events and as a producer in an event system as a publisher of these events your real responsibility is to make sure you stick to that design you agreed with because the schema of your event the format of that event is the the biggest part of coupling that you will see in the event of an architecture so although you've removed that runtime coupling because now producers and consumers don't know each other exist you still have semantic coupling the format of
15:00 - 15:30 that event is really really important so as a producer it's your responsibility to try and minimize breaking changes and to stick to that contract that you agreed on there's some things you're not responsible for anymore though and this is where it gets really interesting you're not responsible for how your events are used your responsibility is simply to publish the event and what Downstream people choose to do with that is not your problem and the other thing that's not your responsibility when you compare it with with hdtp is the constraints of
15:30 - 16:00 your Downstream consumers so if you think about when you're making a hdtp request you need to understand what kind of throughput that Downstream system can support what rate limits does it have you need to understand the format of the API that that API has that's now your responsibility but then you're responsible for the constraints of a system outside of your control and with a event of an architecture that constraint that understanding of constraint switches to the subscriber responsibility and this makes more logical sense if you think
16:00 - 16:30 about it I should own the constraints of my own system so as a subscriber to events you need to control your own ingestion rate you need to control the load on your system and another place where you can get into trouble here is hooking your event your your systems directly up to the event bus because remember now producers don't care about how their events are being used they're just going to publish events as and when they come in as fast as they can and if you're not careful as a subscriber to an event you can really easily overload your own
16:30 - 17:00 system so what I always recommend when talking about ventive an architecture as a subscriber is protect yourself against this kind of floor as opposed to hooking a piece of compute your application directly up to the event bus Which then couples you to a system completely out of your control instead introduce some kind of cue some kind of dunable storage that you understand so when the event hits the event bus it comes down to your micros service and it hits some kind of queue it's stored there durably and then
17:00 - 17:30 you can do some work off the back of that que and if you in some increased Lord everything just builds up in the queue and you can simply process that at a time that suits you another more subtle benefit of rent of an architecture and I promise we'll get out the theoretical in just a moment another subtle benefit of event of an architecture is around communication obviously this is a technical pattern technical communication between systems but it also really helps at a human level how many of you sat in a room with the non-technical person and just send their
17:30 - 18:00 eyes like glaze over a little bit as you start talking about microservices or rest apis yeah I kind of just go like this a little bit like what what are they talking about when you when you start using a event different architecture it's it eases this communication because you can walk into a room now and say what should happen after an order is cancelled how should we deal with an order completed event you can have these kinds of conversations it makes this communication easier and that's because to come back to this quote from earlier Eda is fundamentally a communic a pattern a way of modeling the
18:00 - 18:30 integration between your systems so let's actually get a bit more practical now let's start to model this integration in event implant-based pizza and to do that let's start with how things are working today so right now a request comes into your order processing service that makes a call to the kitchen say hey can you please go and cook this order please the kitchen responds after that order has been created and says y this order is now finished at which point the order processing service will make a HTTP request to the delivery service delivery service response call to the
18:30 - 19:00 brand new loyalty Point service loyalty Point service response and you look at this right now and even in this simple example there's quite a lot of coupling here there's quite a lot of runtime dependency and this is what we're going to work to remove as you start to adopt event of an architecture and the first question that might be in all of your minds now is how do I actually understand the events in my system if every single business has its own unique set of events how do I understand what them events are and I
19:00 - 19:30 want to come back to the same image from earlier because there's something in this image that's really quite useful when it comes to event architecture and that's the idea of event storming who's heard of event storming who has done event storming actually been through and done an event storming Workshop okay some of you have some you haven't really coincidentally there's actually a thing on the wall over there which is actually a really good example of event storming that wasn't planted there that's just happens to be there event storming is an idea that was populated by Alberto brandolini and and quoting directly from the website it's a fantastic way to
19:30 - 20:00 increase collaboration across Sil Lord boundaries sounds fantastic right and one of the things you can do with event storming again quoting from the website is that it can be used to design clean and maintainable event driven software to support rapidly evolving businesses and I'm sure many of you are dealing with change in your systems things that change over time so this isn't going to be a deep dive into event storing that's a whole talking itself let's just quickly touch on what
20:00 - 20:30 event storming is so you run event storming Workshop style you get every single possible stakeholder into a room together business people technical people everyone's there together everyone has a pack of potic notes and they're all sticking potic notes on the wall the sticking postate notes to understand what the events are in a system things like order created payment processed order delivered the sticking poits on for the commands the things that trigger these events to happen and Post-it notes for the queries this
20:30 - 21:00 importantly is a conversation focused on the business if you catch two developers and there always will be two developers satting a corner talking about how they could build this on kubernetes and how complex that would be and I tell me it will be complex if it's kubernetes um bring them back into the room bring them back to the business conversation because that's what it's all about the core thing that you get out of an event storming session is aured language that you're going to use across the entire business both in the technical and the
21:00 - 21:30 non-technical that's the core thing so the other thing you'll commonly hear when you hear about event of an architecture is this idea of evolvability if you've got this stream of events that are just flying around everywhere you've got this Central event BS and producers and consumers don't know about each other doesn't that make your system incredibly evolvable because you could just add a new service hook into that event stream and now you've got this new service and it's event Ren and everything's magical not always that's simp simple of course nothing's ever that simple so
21:30 - 22:00 let's come back to this fat event this event Carri State transfer you're publishing this already confirmed event and actually because your pizza restaurant is so so popular you've now gone Global you've gone multinational so now you need to start adding the currency to this already confirmed event so what you do is you take that order value property and you change that to be an object so instead of just publishing the value as a decimal you're now publishing a object that has the value the currency of that value who can see a potential problem
22:00 - 22:30 with this yeah every because what you've done now is you've broken the contract of your event you've changed the schema of event and you might think well when we did that event storming session it's only the kitchen that's really interested in this event and they don't care about the value they only cared about the items on the order and then you realize that a downstream service had been added to support analytics to support reporting and you didn't know about it remember producers are in aw of their YouTubers and now you've just broken a system
22:30 - 23:00 completely that you didn't know about you've just taken it offline so what do you do now how do you handle this and I'm going to put a really dirty word on the on the screen now and I'm sorry you're all going to get shivers down you spy and I really really apologize for it it won't be there for long I promise and that word is governance everyone's favorite word is a developer tingles of bureaucracy you love it and governance is really important with event of arch because you've got these systems that just don't
23:00 - 23:30 care about each other you need a way to govern these events and this doesn't have to be bureaucratic this can use a process that you're all probably familiar with as developers and that's the idea of an RFC a request for comment so when you're thinking about changing an event instead of just changing the event without tell anybody make it collaborative raise some kind of RFC let people comment on the change that you're thinking of making to your event relatively simple but it's a really
23:30 - 24:00 powerful way and it's not really that bureaucratic hopefully you've all recovered from that word being on screen now the other thing you can do at a more technical level is to think about the schema of the event that you publish rather than just publishing an event that is just the order confirmed event introduce this metadata data pattern again schema the scheme of your event is the tightest form of coupling in event of architecture so at a top level splitting your event down into a metad data section and the data section
24:00 - 24:30 helps you to stay evolvable so the data property here is the actual event payload this is the bit of the event that's unique to the event you're publishing for every single type of event in your system this data property will be completely different it's the metadata property where things get really interesting because this metadata property is going to have the exact same schemer across every single event in your entire organization they're all going to have this metadata property and they're all going to have the same properties inside
24:30 - 25:00 this metadata property one of those will be the event type what type of event is this that's being published in this case it's in order created event an event ID the event ID is a uniquely generated identifier generated at the point of publish and this allows any Downstream consumers to more easily Implement things like ID and potency you can look at this event and be like have I seen this event before yes I have I can just ignore it the event date and the event time this can help you to understand ordering of events if that's something
25:00 - 25:30 that's important to you and finally with this conversation around evolvability is the version number you see you've got version and V1 here including a version number as part of your event more easily allows you to introduce breaking changes so let's come back to that same example where we need to change the value from a decimal to be a object that has the value and the currency so right now the order processing service is publishing version one of that order confirmed event and the kitchen service is consuming
25:30 - 26:00 that all your Downstream consumers are consuming order confirmed version one and then what you can do is also start publishing version two of your event and nobody might be interested in this event right now you're just publishing it and it's going to be really sad because nobody cares about it but it's going to be there ready and then you can put out a notification to the rest of your company saying hey we've now got version two of this event can you please all switch over to version two we're going to depreciate version one on day x and over time all of your consumers can switch over start
26:00 - 26:30 consuming version two and eventually over time you can stop publishing version one of the event so as much as possible of course try and avoid breaking changes but if you think you're never going to make a breaking change I can tell you now you're going to be wrong you will eventually over time need to make a breaking change to an event so that leads us back to this question fat versus sparse events what's the right one to choose unfortunately I'm not going to give you a definite answer on
26:30 - 27:00 this because as always it depends feels like a cop out I know but it genuinely does um so fat events obviously reduce that need for a call back they reduce the need for systems because all the information a downstream system could possibly need is in that event that gets sent the downside of that is that you get more coupling at the schema level it becomes more challenging to make changes to that event over time because you don't know which bits of that event different systems are interested in So
27:00 - 27:30 based on that argument that might lead you to say well long live the spice event let's just publish spice events everywhere but as was described when you do this it can become challenging if you take this to the the absolute extreme and you only ever publish spice events and to stop this risk of call back you just don't allow systems to make a call back this would work it would give you spice events it would reduce that coupling but actually this isn't really that practical in the real world because then how would the delivery service here know what needs to deliver it just
27:30 - 28:00 wouldn't so the other option you have here is to use more sparse events but instead of just publishing a single identifier include multiple different identifiers and when you think about the logical flow of this here as you order a pizza you're on your app you're hungry you're ordering your pizza and the first thing you'll likely do is add a delivery address and that delivery address is going to go to the delivery microservice remember microservices own their own data they have one responsibility so here the delivery service should own all the delivery addresses and the order service should own all the orders so you
28:00 - 28:30 add your delivery address and that will go into the delivery service and that will get stored with some kind of unique identifier and then after that you'll you'll do the submit order you'll create your order and when that create order request goes in that will include the identifier of the delivery address that needs to be used so that when that order confirmed event gets published you can include the order ID but you can also include the delivery ID so when the delivery service consumes this information you can be like I know what that delivery ID is go
28:30 - 29:00 and get that and it can set the right delivery address so you can still keep sparse events but you don't need to really really minimize that down you can include other identifiers as well the other option you've got here is a pattern called the enricher pattern and the enricher pattern allows you to enrich event information as it moves through the system so as a producer you might still publish a sparse event and the vast majority of Downstream consumers might only be interested in that spice event and they're happy with
29:00 - 29:30 that some Services might need that additional information so you might at the point of publish maybe put that information in a shared location that might be another API that's exposed specifically for these callbacks to protect your main system it could equally be a shared service like Amazon S3 there's the second AWS service name S3 is really good because it's it's scalable it's durable it's available and it's really easy to share information across different accounts different systems so
29:30 - 30:00 you could potentially drop some data into Amazon S3 put a pointer in your event payload so when a consumer consumes that event they can actually reach out and grab that additional information if they need it but you're protecting your main order processing microservice whichever type of event you choose fat or Spar and you likely have a combination of both at least if you're doing this at scale Embrace that metadata data patch and include a way to have this additional metadata as part of your
30:00 - 30:30 event so the other word you likely hear a lot as you talk about event of oration I've said it a few times already is the word asynchronous what does it mean to be asynchronous now if you're a developer today who's used to making a method call or a HTTP request this can be a little bit of a paradigm shift I know when I first started building event and systems this took me a little while to get used to because now you're in a place where instead of everything happening in process you're now publishing an event so you might publish your order
30:30 - 31:00 confirmed event and at some point hopefully potentially maybe later an order cooked event will come back from your kitchen service but you've actually got no way to ever guarantee that the order confirmed event ever makes it to the kitchen service it's not something that's your responsibility you don't need to Cur but obviously the business cures the person who's ordered the pizza cures about that and this is the problem here with consistency and with consistency you've got two types of consisten you've got strong consistency and eventual
31:00 - 31:30 consistency and I want to try and explain the difference between the both using the pizza restaurant so you only put them imagination hats on again imagine you've placed your order and you're going down to the pizza restaurant to pick it up you're going to collect this order you walk in walk into the pizza restaurant the smell hits you in the face Pizza is amazing you can almost taste it you can see the pizza's being made sometimes I give this example just before lunch and people like literally zone out at this point a pizza so you're there you're waiting to pick your pizza up you're hungry and you what
31:30 - 32:00 you need to do first you need to pay for that order right and you've got two options when you come to paying for your order one slightly more common than the other nowadays the first option you have is to pay with cash you can take out your purse or your wallet you can take the cash out cash out hand it over to the cashier they'll take that money put it in the tail you get your pizzas and you go anywh and that's an example of a strongly consistent transaction because at that point you can absolutely guarantee that that money has changed hands because the money is sat in the
32:00 - 32:30 till and if that was to fail for whatever reason let's say you're here in London and you try and play pay with I don't know dollars you're likely going to get that money back and you have to retry you'll have to go again this is what you used to if you're doing request response style communication the other option you have when you pay is to pay with your card and when you make a card transaction all you're doing is making a theoretical guarantee that at some point in the future that money is going to move from your bank account to their bank account I'm sure you've all been
32:30 - 33:00 there where you've paid for something you paid for your pizza you've got your phone out onto your bank statement and the money is not actually shown up yet and you walk out thinking yes free pizza look at that amazing and eventually some point sometime later that money will eventually move over to the the pizza restaurant's account now that is an eventually consistent thing that's happening eventually that money is going to change happen but it isn't and I don't know if you caught Sam Newman's talk yesterday but I actually like a phrase he used which was eventual convergence instead of eventual con consistency
33:00 - 33:30 eventually over time these systems will Converge on the same view of the world providing people stop publishing events and one thing you'll need to do with event different architecture is just to embrace eventual consistency this is one of the tradeoffs you have to make when you're building event and systems is you need to embrace this eventually consistent way of looking at the world so now everything's fantastic you've implemented an event of an architecture at the pizza restaurant events of flying around everything's Loosely coupled things are wonderful
33:30 - 34:00 until of course something changes because it always changes and what actually happens now is your Business Leaders come along and they say you know that loyalty Point service that you spent so much time building you put all that Blood Sweat and Tears into we want to get rid of that now we're going to replace it with a SAS product get rid of that we don't need that we don't want to build this custom we want to use something that somebody else has built and this causes your event driven way of building a problem because this third party SAS product isn't event driven it exposes a HTTP API and that's
34:00 - 34:30 what you kind of wanted to get get away from right so how do you handle this kind of HTTP Communication in an event driven world because you're likely going to see that somewhere whether that be a third party API or another system inside your organization and the way I like to think about this is that you would still have an event driven microservice managing this communication you would still have a loyalty Point service and the team building that loyalty Point service is in charge of managing that integration
34:30 - 35:00 right what that might look like is something like this so you've still got your event bus you've still got your order completed event hitting the event bus and then you've got a service that's only job is to take that event and translate that into something that the loyal Point service can understand that third party API make the call to the third party API the response comes back and then you might publish another event a loyalty points added event now you'll notice here that I've put a Que in the middle here and this is important if you think back to Gregor's quote from the
35:00 - 35:30 start the appropriate amount of coupling depend on the level of control you have and you now have no control over this third party API you don't control the up time you don't control the slas the throughput the rate limits so introducing this queensu as you can keep this amount of durability the event the order completed event can hit the event bus come through into a queue and you can work off that queue and process to the third party API as of when you need to another really useful place for this
35:30 - 36:00 pattern is as you have teams in your organization who are starting to adopt event of an architecture they're kind of interested they want to start doing it but they're not quite ready to get rid of their ASP net web API is running on containers they just love them so much so what you can do is you can still have that web application that exposes a hdtpm point but you can allow teams to take these small baby steps into the event Ren world to start building a pattern like this they can get used to reac into events but yet the core bit of their
36:00 - 36:30 system is still hat TTP based and they can just pass that event on it's really nice step into the system so this gives us now a way to integrate with things that aren't necessarily A venten we can integrate with these HTP based systems what happens in Advent of architecture when things go wrong who's ever had a system break for a reason that they didn't even know was possible yeah hopefully every single hand they haven't I want to speak to you afterwards because I want to know how you found out what crystal ball do you
36:30 - 37:00 have and this is where observability is really important and obviously observability is really important in every single type of system that you build you need to understand what's going on inside your system but with an event driven system it gets really tricky to understand cause and effect because when you've got request response style Communication Service a calls service B and service B responds you can kind of see that that's happening but in an event system you're just publishing an event and at some point someone might maybe be interested
37:00 - 37:30 in consuming that event so you want to keep your system observable you want to be able to understand this cause and effect and a really simple way to ensure you can do that is just to revisit that metadata data pattern from earlier and we'll just make a small change to it we're going to add two additional properties a trace parent and a parent span this assumes the use of open Telemetry here if you aren't using open Telemetry then you this could equally just be a correlation ID some common identifier that always gets passed as
37:30 - 38:00 part of the event payLo and what this allows you to do is it allows you to continue this Trace allows you to link these traces together allows you to understand cause and effect more easily of course in your consumers you still need to handle all this fact you need to handle the fact that maybe that Trace doesn't exist anymore potentially but at least it allows you to give the possibility to determine cause and effect so again just here's another example of being really intentional with
38:00 - 38:30 how you design the schemas of your events if there's one takeway from this talk that is it be intentional about the way you design your events so let's talk about boundaries now what happens at the edges of a system because surely at some point in any event Ren system you're going to need a request response somewhere right that might be a mobile client that might be a web browser so how can you actually manage that how can you manage these calls from external systems that might not be event driven in an event different way and I'm a really big fan of having these really
38:30 - 39:00 thin actual synchronous API layers at the boundaries of a system so you might have a really thin order API that takes that request in the create order request does a little bit of validation stores that request in a database of some kind and then goes on and publishes the event to the event bus and then you have your Downstream consumers who then reacting to that event and all of the functionality is happening asynchronously your business your events are driving your business functionality remember now what happens here for your
39:00 - 39:30 get order request though because if all this stuff is happening asynchronously in the background and then somebody wants to check the status of the order I'm sat at home really impatiently refreshing the screen trying and work out where my pizza is how do I understand the latest state of the world because everything's happening gay synchron to member and this is where we need to make a distinction between commands events and queries the three different things you are likely to see in any system and commands and events are kind of similar
39:30 - 40:00 really they're both going to cause something to happen the difference is really mostly just a statement of intent with a command I'm telling you can you do this thing for me please whereas with an event I'm just reacting to something that has happened and then a query is me asking for a specific piece of data a specific view of the world can you tell me the status of this order please order processing API so to you visit the same example again the command comes in to create the
40:00 - 40:30 order that's the web client or the mobile client saying can you create this order for me please the event gets published to the event BS the downstream functionality starts happening and then eventually your Downstream systems will publish an event when they're finished with their work so the kitchen will publish an order prepared event and all you need to do to make sure you can update your view of the world is just to add some kind of service inside your AUD processing service that's interested in these events that Downstream systems are publishing so you hook into that order prepared event and although you're not
40:30 - 41:00 going to trigger any functionality off the back of that you're still interested in it because then you can just update the status of the order in your database and this allows you to keep this common view of the world as events are happening in the background asynchronously I want to quickly discuss the idea of asynchronous commands so although event driven systems typically everything is an event and this might offend some of the event driven purists in the world but sometimes you might also want to process
41:00 - 41:30 a command asynchronously so let's imagine the scenario where your order processing service needs to send an email send an email saying hey here's your order details or the receipt of your order whatever that might be and you could build that in a HTTP based request response style communication so the order processing service makes a call to the email notification service can you send this email please and a response comes back but actually is this really what you want to do should the order processing service be
41:30 - 42:00 blocked whilst an email is being sent probably not so you probably want this to be non-blocking communication but actually you still want to say I want to send an email you could equally implement this in a completely event way so the AUD processing service publishes an already confirmed event the email notification service is interested in that event and it uses that to then go and send an email but actually is this responsibility right because now the email service needs to understand and what should go into that email what's the subject what's the body and is that is
42:00 - 42:30 that right is that responsibility right should it not be the order processing Services responsibility to determine what actually goes in that email and the email service is just there to send emails out however it does that so you kind of want this to be a command you want this to be I want you to send an email but you might not necessarily want it to be completely request response you might not want it to be blocking so you can do this in a similar way your your email service can still export was an endo of some kind but as opposed to that
42:30 - 43:00 being a HTTP based endpoint that could just be a message Channel this might be a topic this might be a Quee doesn't really matter but it's asynchronous so the order processing service can still send a send email command I want you to send an email for me please and in the payload of that event it will include the two the from the subject the body that kind of common things you will see in an email you'll notice interestingly in the metadata property if you can see that I've also got a response Channel I've got one more additional property in that
43:00 - 43:30 metadata section and that response channel is what allows you to build this request response Communication in an asynchronous way because now your email service can process that request send the email and after it sent the email it knows where to send a response back to and it can send a response back to that response Channel within the AUD processing service now in the case of an email you might not necessarily want to do that you might not C about a response but this is a way you can start to do request response in a way that isn't necessarily
43:30 - 44:00 blocking now of course in some cases you probably will want blocking communication if you're taking a payment for example you're going to want to know that that payment has happened before you send somebody a pizza but in a lot of cases it might not necessarily need to be synchronous so we've introduced commands events queries we' introduced some of these different ideas and there's one more pattern I want to introduce you to now that plays really nicely with event
44:00 - 44:30 of architecture and to be absolutely clear before I put it on screen this is not a prerequisite of building event of architectures you don't do not need to be using this pattern to build with Eda and that pattern is command query responsibility segregation or cqrs who is familiar with cqrs who use cqrs hands everywhere awesome I can fly through this and we can all go and get a coffee then um so in cqrs you split your your system into two completely independent Services one for processing
44:30 - 45:00 commands one for handling queries so the create order request will come in to your order API that will do a little bit of validation maybe store that in an orders database and Publishing an event to an event bus all your Downstream consumers are interested in that event and your query system will also be interested in these events that are happening and it will use that to update its view of the world so here is this specific view that people can then
45:00 - 45:30 query so let's start to bring all of this together now bring together this entire order processing floww so the create order request comes into the order processing command service I've not yet thought of a better name for that but that's what it's going to be for now and that service will publish an order confirmed event that order confirmed event your kitchen service is interested in that because it's you know needs to know what needs to when that happens it needs to know it needs to place the order or create the order but actually your quivy system is also interested in that order confirmed event
45:30 - 46:00 and it uses that to update the status of the order in the quy database once the order's been created the order prepared event gets published the delivery service is now interested in that order prepared event because it can now go and deliver the order and again your query service is also interested in that order prepared event because it then updates its view of the world and it can tell me furiously refreshing my phone that this order is now out for delivery then the get request comes in to actually get the status of the order that's me refreshing my phone and I get
46:00 - 46:30 the view of where that order is up to and I want to zoom in on this bit of this interaction just for a second so the command comes in it gets validated and the the order data gets stored in the database and then the event gets published after that what happens if that event fails to publish what do you do because now you've got this order in the database but the event fails to publish which means the kitchen's never going to start creating it and you could at this point return an
46:30 - 47:00 error code to the caller and get the person to try submitting the order again but is that really the right experience for your user is that really the best thing to do and you might have seen an example of this if you've ever tried to pay for an order on Amazon with a card that's expired so you place the order and on the website you get the order number you get an order confirmed page and then a few minutes later you'll get an email that says payment failed please go and update your payment details that is an example of this happening that the request is always received into the system and then at
47:00 - 47:30 some point later after this Downstream functionality has happened a caller comes back so this is something that you need to work with you need to handle this fact that a request can come in but the event might not necessarily be published and there's a couple of ways you can handle that the first is using the outbox pattern a transactional outbox pattern so with an outbox pattern at the point you write the data to the main database you write the order data you all o write the data to a secondary table could be called outbox something
47:30 - 48:00 like that and you wrap that whole thing in a transaction so that if one fails the other one won't happen either then you can have a SE separate piece of functionality working off the back of that outbox table that's picking up all the data that hasn't yet been published all the events that haven't yet been published and then Distributing them out to the event bus this works really well but it doesn't mean that you need to manage two separate tables which you've now got your main table and you've got an outbox table and you'll need to manage that
48:00 - 48:30 outbox table because it will just grow and grow and grow and grow you need to be clearing it down so an alternate option that I found works really well is to use change data capture providing you use a database that supports change data capture and what change data capture allows you to do is to react to the stream of changes that are coming out of your database so as a record is written to the database you can stream that you can react to that and you can publish Events off the back of that if you're using a service like Amazon Dynamo DB third AWS service
48:30 - 49:00 name then you could turn on Dynamo DB streams and just react to that stream of changes in your database and you can use that to publish events so to kind of stitch this all all together now with cqrs so you've got the request coming in the command coming into your order processing service that's a random Arrow you've then got that being stored in your main order database you're using a transactional outbox pattern or change data capture to stream out to some kind of publisher that's then publishing the event to the event bus your query service is hooked
49:00 - 49:30 up to that event bus creating its own view of the world a specific view for a specific use case so when the get request comes in it can do something specific with that and where this gets really powerful is that it's the command database the database on the left there that's the source of Truth for your system that's the real that's the real database the query service is just a view a project ction and this allows you to get some really nice optimizations because you can then optimize the format
49:30 - 50:00 of that data that data structure in a way that suits that specific get request so you'll see here that that table on the right only has the status the value and the item count because that's what the viewing the UI needs it only needs that information so let's store it in that way and equally you could also use a data provider like reddis you could use a cache to store this data because remember it's the command database that's the source of Truth and that can allow you to get really performant
50:00 - 50:30 reads if you take this to the absolute extreme you could store that query view in memory in your application to give you incredibly performant read now of course there's some whole other challenges that you need to deal with there come and talk to me about them afterwards if you want to do that but let's start to wrap this up a little bit now if you're listening to this talk and you're sat there thinking this sounds amazing James but I'm so far away from potentially building event of an architecture where exactly do I start and my recommendation there would
50:30 - 51:00 just be to start small don't try and rewrite your entire system immediately to use event driven architecture pick a small part of it and then start to build that out in a more event driven way and you can even if you're building a monolith if you're building something more monolithic today you can still talk to use messaging inside a monolithic application if you're building a modular monolith you could have the communication between your modules use messages and then when you start to break this apart you have that messaging communication event based
51:00 - 51:30 communication already happening because remember introducing events into your system is really powerful because event driven architecture is fundamentally a communication pattern a way of modeling integration between systems and that's both the technical integration the technical communication between services but also the human communication about your services one of the hardest things in soft development is communication the language that you use to talk about a system between
51:30 - 52:00 Technical and business people and remember it's language and words that sets us apart as humans and whilst at least for the moment software development is still a human endeavor using the right words and languages to talk about and build your system is incredibly incredibly powerful if you want to have any questions and you want to continue this conversation I will hang around here for a little bit afterwards I'll also down on the Booth down in the main conference Hall please reach out on social media if
52:00 - 52:30 you want to chat I'd love to hear your feedback this is actually my first time doing this specific talk so I'd love to hear your feedback on that I do have a YouTube channel where I publish a lot of content there and the final thing I'll say is that ndc's in a fantastic conference and there's a lots and lots of amazing speakers here so I am genuinely grateful that you've all come to listen to me talk this morning thank you all for listening