The Many Meanings of Event-Driven Architecture • Martin Fowler • GOTO 2017

Summary

Highlights

• Event-driven systems can mean different things to different teams, and clarity is needed to avoid confusion. 🤔
• The exploration of event-driven architectures led to the identification of four main patterns. 🔍
• Event notification helps decouple systems by sending notifications without expecting responses. 📣
• In event-carried state transfer, events include enough data to minimize the need for additional queries. 📨
• Event sourcing keeps an event log that can rebuild the current state, much like version control. 📜
• CQRS is beneficial for separating reading and updating operations but needs careful implementation. ⚙️

Key Takeaways

• Event-driven architectures have multiple interpretations and can be broken down into distinct patterns. 🎭
• Event notification is a common pattern where systems communicate changes via event emissions. 🔔
• Event-carried state transfer reduces dependency on central data stores by carrying all necessary data within events. 📦
• Event sourcing records every change as an event, similar to version control systems, offering extensive historical insights. 💾
• CQRS separates the command and query models in a system, aimed at optimizing read and write operations. 🔄

Overview

Chapters

• 00:00 - 05:00: Introduction to Event-Driven Architecture The chapter begins with the author's reflection on their career-long exposure to discussions about event-driven architectures and systems. They express a sense of ambiguity in understanding what people actually mean by 'event-driven' due to the various interpretations and implementations they've encountered.
• 05:00 - 10:00: Workshop and Patterns Identified The chapter titled 'Workshop and Patterns Identified' delves into the significance of events and themes in various contexts. It discusses how events are considered crucial but can often be abstract and open-ended when trying to derive more profound meanings from them. The narrator reflects on the persistent background presence of these themes throughout their career and expresses a desire to explore these concepts further.
• 10:00 - 15:30: First Pattern: Event Notification The chapter discusses the concept of event-driven architecture at Fort Works. Despite the team engaging in numerous event-driven projects globally, inconsistencies and confusion exist regarding their understanding and implementation of the concept. This realization came to the forefront towards the end of the previous year, highlighting a need for a more cohesive approach.
• 15:30 - 21:00: Second Pattern: Event-Carried State Transfer In this chapter, the focus is on the 'Event-Carried State Transfer' pattern. The narrative begins with a reference to a collaborative workshop held in Denver, where various participants gathered to exchange insights on event-driven systems. The discussion highlights how different projects under the umbrella of event-driven architecture had distinct characteristics. Through the workshop, attendees worked towards developing a clearer comprehension of shared patterns within this domain, illuminating some common strategies that effectively support event-driven systems.
• 21:00 - 34:30: Third Pattern: Event Sourcing The authors discuss their experiences and thoughts on the Event Sourcing pattern. They emphasize the importance of elaborate explanations in articulating their ideas. Despite hoping for someone else to document their insights, they took the initiative themselves. Initially, the information was shared internally through a video, then expressed online via a note, which was not anticipated to gain much traction.
• 34:30 - 47:00: Fourth Pattern: CQRS (Command Query Responsibility Segregation) The chapter discusses the high interest and relevance of the topic CQRS (Command Query Responsibility Segregation) within the field of event-driven systems. It highlights that many people face challenges in this domain, leading the speaker to address these concerns through their presentation. The speaker introduces four key patterns, commonly associated with event-driven systems, with CQRS being one of them.
• 47:00 - 50:00: Conclusion and Further Reading The chapter 'Conclusion and Further Reading' delves into the concept of an event-driven system, often utilized in dynamic and responsive software architectures. The narrative suggests that such systems generally incorporate at least one, if not all, of four specific design patterns to function efficiently. The chapter intends to sequentially explore these patterns in detail. An illustrative example mentioned is that of an insurance company leveraging such a system, hinting at practical applications and theoretically grounded practices to be explored further in the chapter.

The Many Meanings of Event-Driven Architecture • Martin Fowler • GOTO 2017 Transcription

• 00:00 - 00:30 [Music] almost my entire career I've heard people talking about event-driven architectures and event driven systems and things of that kind and it's always been a little bit odd as to what what actually do people mean by that because when I dig a bit further it seems that there are various different things going on
• 00:30 - 01:00 there is some kind of common theme events are or something called an event is an important part of what's going on but actually what it boils down to in terms of something more meaningful that's much more open-ended and for a long time that's just been ticking around in the background um and I have at various points in my career sort of thought I want to really dig further into this and
• 01:00 - 01:30 I kind of never really got round to it cuz I dug into other things instead but it really kind of came to a head late last year um at Fort Works we've been doing quite a lot of event-driven stuff and various people came to roughly the same conclusion as me saying we've been doing all of this stuff called event driven we're not really quite sure it's the same and we're doing it at all these different project teams around the world because there's a several thousand people now at forw works and lots of
• 01:30 - 02:00 different projects we need to get together and learn from each other so we organized a workshop in Denver it was yes Denver um late last year and a whole bunch of people got together and very quickly it became apparent that yes there were some very distinctly different things going in under the title of event driven and we need to try and figure that out a little bit better and as a result of that I think we came to a bit of an understanding of some common patterns that help provide a
• 02:00 - 02:30 bit more Precision into what we're talking about and so we thought that's important that we describe this and say a bit more to the rest of the world about some of our thoughts and I hope really hard that somebody else would write it up nobody ever did but one of us did a little video thing just for an internal video wats it and I thought oh I can't stand it anymore so I wrote a little note and stuck it up on my website not really expecting it to get much track traffic
• 02:30 - 03:00 and I saw it got downloaded with a huge amount of traffic so I thought well actually this seems to be a common problem and that's what led me to decide to inflict this talk on you this morning is because it seems to be that people are concerned about this and what it boiled down to is we identified four different things I would call them patterns that often come under the term event driven an event driven system might you when people say they have an
• 03:00 - 03:30 event driven system they usually mean they've got at least one of these patterns in play and could mean they have all four patterns in play and I will go through them one at a time gradually revealing what they are so to begin the first one let's imagine some example system that might be had by an insurance company and in this insurance system
• 03:30 - 04:00 we have our customers and our customers have a habit of moving around and changing their address but the insurance imagine it's something like car insurance or house insurance um where changing where your address actually affects your insurance quote levels and your premiums that you have to pay and everything so if somebody changes their address it's not just a case of updating the address field you actually have to trigger the insurance quoting system to come up with a new quote and they then got a send you
• 04:00 - 04:30 some mail to say um here's your new um Insurance agreement whatever whatever I don't know the details it's a long time since I'll spent any time with insurance people but the point is in a kind of simple call and response kind of architecture what that requires is whatever system you've got that's looking after customer management needs to tell your quoting system hey go and do a requote and the problem of that is that creates a dependen
• 04:30 - 05:00 from the customer management system into the insurance quoting system it needs to know that it's there it needs to know what apis it has etc etc etc and that couples the knowledge of what how Insurance quoting Works into something that's a much more generic customer management kind of feature and that's a kind of coupling that a lot of people don't really like very much so one way we can get around this coupling is we can say
• 05:00 - 05:30 instead of doing that let's try and reverse things around I don't mind my insurance quoting system knowing about my customers and that there is a customer management system there that seems a more natural dependency line to follow because uh something that's very very generic managing customers generally that's the kind of thing you want to depend on than have it depend on lots of other things and a way we can do this is we can use an event so what happens is whenever the customer some customer data
• 05:30 - 06:00 changes the customer management system emits an event object to some well-known Place event queue or whatever and an insurance quoting system can listen for that event and when the event occurs it can decide do I want to do something about it if so what do I do and then that nicely re gives us that ability to reverse the dependencies allows the insurance quoting system now has to know about this event structure but the customer management system can be independent of all of this
• 06:00 - 06:30 and this is an approach that um we in our discussion whoops giving you a slideway there in our discussion we referred to as event notification we're using events as a notification system uh notification mechanism between multiple systems now of course I've described that in terms of large scale systems independent relatively independent systems for a big company but we also use this same pattern in a very small
• 06:30 - 07:00 layer system and it's a classic of course approach of how to manage interactions between graphical user interfaces and the rest of your code when somebody enters information into a text box you don't want some very generic textbox widget to have to know all about your code and which things to call instead you say the event the uh text box will generate an event when you change it and then your code listens to the event it's the same idea and again the same trick is your reversing
• 07:00 - 07:30 dependencies and that's really I think the essence of why you want to do this is that you want to do that reversal of dependencies now I've talked about this in terms of events and I've stressed this dependency switch that it does but it also does something else it bottles it sort of takes everything to do with the change that the event talks about and Bottles it into a thing an object a record or
• 07:30 - 08:00 something of the s that you can refer to and pass around it makes the change a kind of first class thing imic now when we do when we call a function that's just a very transient thing we call the function Supply the arguments goes away we never know that we've done it unless we got some logging and tracing going on but when we do something like this what we're doing is we're actually making a record of it and then we have have a real thing that we can talk about
• 08:00 - 08:30 and refer to and pass around and that has its own benefits if once we can do that so we're actually getting a second benefit here which is this bottling it up into a into a thing into an object or I don't have to hesitate about using objects it doesn't have to be an object in an oo sense I think of it as a just some kind of first class finger magig that needs to be a new technical term first class finger Magic God I don't even get a giggle for that
• 08:30 - 09:00 that's a shame now that's not the only way of thinking about these first class finger midic often we think of them in terms of commands as well and that leads to an interesting question well when do we have events and when do we have commands what's the difference between an event and a command and I think it's actually it is both very subtle not really anything that you can say is a strict definition and yet at the same same time very important
• 09:00 - 09:30 because it's about our language and our communication and how we talk about our intentions about things we could describe that situation I just described as the customer management system saying requote insurance for this customer that's the customer management system saying it knows what ought to be done and it's telling some remote system here's what you should do when I phrase it in terms of events
• 09:30 - 10:00 I'm saying the customer's address changed I'm not expecting any particular response from that from anybody I'm just indicating a change both can be have this kind of bottling it up and making it a first class finger Mig effect but the way the communication patterns differ is imp strongly implied as different whether you use event naming or command naming style naming so really it boils down this is one of the two hard problems in
• 10:00 - 10:30 software right this is naming things and therefore when you're thinking about the difference between events and commands it's really how do I want to describe how this system works most accurately to the people who are reading about the system if I'm trying to say I don't really care what happens as a result of this event I don't want to care then yes I want to use the word event if on the other hand I am particular about I want something to happen then I want to name it in terms
• 10:30 - 11:00 of a command now I mean you can get all the way over way you could have your insurance quoting system might ex insist on sending a communication an email to the uh to the customer and natural way to do that of course will be a command send an email but of course if you go too far down the Event Route you you then get this effect of passive aggressive events right or passive aggressive commands where he's saying the customer needs to be sent a message about their quote hint
• 11:00 - 11:30 hint and you know you don't really want to go down that path if you're trying to read how the system works that's going to be confusing when you get those kinds of events occurring so whether you use events or commands under the covers they're basically the same thing they're often messages on cues they look exactly the same the difference is all about naming but that difference is really important because let's face it names in software systems are one of the most important things we have to deal with
• 11:30 - 12:00 so going back to our system again one of the nice benefits of using events is it allows many other systems to hook up to the event stream very easily without us having to tell the customer management system what's going on and this is a very nice property a nice result of using events for notification here I need to hook up a new system that takes advantage of CH needs to know about customer changing their addresses I don't have to go and talk to the customer management team and get them to
• 12:00 - 12:30 add a new not no call into their interfaces and go to endless meetings with the customer management group and two months later maybe I will get them to do something no I have they produced their events I just tap onto the event stream I'm cool this is a very attractive ability um and it easily allows you to do all sorts of interesting things without changing what's going on but when anything sounds like it's
• 12:30 - 13:00 really nice there's always a butt isn't that and in the case of event notification that butt is now you're in a situation where you've got no way you can figure out what the hell is going on in your system anymore because there's no program to look at there's no statement of says these are all the things that happens when a customer changes their address what you have to do the only way you can figure out what's going on is
• 13:00 - 13:30 you actually have to watch the address the various events and see what happens looking at the flow of messages through your system to try and figure out what the hell's going on and this is true in the guey case if you've ever tried to debug a heavily event driven guey kind of system pretty much your only chance is to bring it up in the debugger and try and Trace the various events that are flowing around I remember doing this once a trying to figure out how the original
• 13:30 - 14:00 Small Talk 76 worked um because I was writing some stuff about model view controller and I rapidly discovered you couldn't trust any of the papers that have been written on it because they often didn't have access to the source code so I was able to get hold of a copy of uh the original MVC Small Talk 76 bring it up and I could actually figure out how it worked but I couldn't figure out how it worked by looking at the code I had to step through the debugger do things and see what happen happened because again it was all events and
• 14:00 - 14:30 that's the Dark Side of using event notification you get this great decoupling you get this great flexibility being able to add new consumers and you get a inability to see what's going on across the system as a whole there's no code anymore that you can just look at and it's a tradeoff like so many things tradeoffs so that's the first thing event notification and that I would probably say is the most common that people say when they're event driven they're using
• 14:30 - 15:00 an event notification mechanism um somewhere which is kind of odd in a way because just about every system I've ever looked at uses event notification to some extent um I guess when you call it event driven it means the event notification system is seen by at least the the the senior people on the project as a particularly important part of the architecture so the second one is a minor variation on this case so let's go back to a little example of
• 15:00 - 15:30 the insurance company again in an event notification system all you really need to know is that the event occurred and that could really be very very generic it could just be saying hey something changed somewhere um you probably want to give a little bit more information on the event about that so you might say h the the event thing for Linda Chang something's changed about Linda's
• 15:30 - 16:00 account and then all the other systems go what what's happened the way they find out what's happened is they have to talk to the customer management system and even if you got more precise about this you might say Linda's address changed they've still typically the insurance quote system has okay so what's the new one what was the old one you know has has it changed in a way that's important to me event notification often involves additional traffic because the the systems that are
• 16:00 - 16:30 the consumer systems still have to go back to the original source to get more information and of course we reduce that burden by putting more information into the event if I just say something about Linda changed then every time you have to Pile in on and it's probably half a time it's not worth it doesn't actually mean anything so it was kind of a waste of time if I'm saying here's the address change then there's less traffic required because I only need to go in if I can care about an address
• 16:30 - 17:00 change and then I can take this to The Logical um final conclusion and say can I put so much data in the events that I don't natur need to go back to the host system so not just do I say Linda's address changed I say what her old address was what her new address is at that point maybe I don't have to go back or maybe I do I know the address changed I know I need to do something but now I need to go back to all the other attribut all the other things I know about Linda
• 17:00 - 17:30 Rising her browser history how often she stands up every day all of these things we know and we can find out by asking the customer management system but now I've got a customer management system that's got all of these queries every time something happens every other system is bombarding me with stuff can I reduce that can I eliminate that load well I could and this is a pattern that we tentatively
• 17:30 - 18:00 named event carried State transfer which is a bit of a long-winded name but I like it because it's a play on rest right um which is resource what's it State transfer this is event carried State trans this is about using events that carry stur of State what happens when you use this pattern is you say I want to completely forbid any contact to the customer management system from the insurance quoting system which means Insurance quing system will keep a copy of all the
• 18:00 - 18:30 data that it's ever going to need now that means of course the customer management system has to broadcast in its events all the data that those Downstream systems are going to want to have and the downstream systems have to keep copies of everything that they are going to use I don't have to keep a copy of the entire customer record I only have to copy the stuff that I care about so it's not as bad as one might think but it's still obviously a lot of copying of data but
• 18:30 - 19:00 the benefit is I now no longer need to call the customer management system at all I have all the data at hand so that might improve my performance because you know I no longer got remote Network calls to the customer management system it reduces the load on the customer management system because it's not responding to all those calls it also May improve availability because if the customer management system goes down or it's intolerably slow I don't care I keep chugging along
• 19:00 - 19:30 because I have all the data to myself so we can immediately see there's some very nice benefits to do with that now it doesn't mean I have to do all the necessary stuff to make sure that I'm up to dat I keep copies of the data it means of course I have to copy all the data but we're not talking about that much data these days storage is is cheap um and what's more I have my uh Insurance quoting system my availability no longer depends on the customer management system so that's a good thing
• 19:30 - 20:00 now hopefully you're well enough versed in software development that you know when people say I have greater availability because I'm not dependent on something else you know what the price is what's the price of high availability lack of consistency exactly so now I've got to replicate the data I have eventual consistency I have to deal with that issue so I've got a consistency problem so again event carried state transfer has this um tradeoff I get even more
• 20:00 - 20:30 decoupling even more um availability but I lose complexity of keeping the duplicated data and inevitable eventual consistency so this is a less common pattern um I don't know what the proportions are I would say a lot of the time that I see event notification I don't see event State trans based State transfer but you and you do have to have event not uh event notification to do this but it is a technique to bear in mind I would say this is one that you
• 20:30 - 21:00 keep deeper down in your toolbox um but it's many circumstances it can be a good one to play so sometimes that's what we see when we talk about event driven systems this notification of duplicating the data to avoid calling back so the third item so far what I've been doing is I've been telling you the concept and then telling you its name
• 21:00 - 21:30 because I kind of think it's more exciting that way you're kind of a bit more of course if you read the talk beforehand you already know what the four things are so I'm not really quite sure why I'm doing it but for this one I'm going to tell you what it is right away event sourcing which is a technique that is long been one that's fascinated me um and it's interesting to see how it's being how people are using it but let's begin by explaining what it is what do I mean by event sourcing how by the way how many people have come
• 21:30 - 22:00 across the term event sourcing stick your hands up just sound curious okay smattering of people how many people understand it well enough they think they could explain it to somebody else a few okay I'm not going to ask you to come up on stage and do it don't worry okay here's how I explain event sourcing and surprise surprises for change of addressive example again because I'm kind of boring so let's imagine I've got some kind of customer management system and I've got somebody with an address and they want to change
• 22:00 - 22:30 their address in a typical system how do you change address you just say okay let's blow away the old address create a new address right that's how it works right nice and simple what event sourcing does is it creates this subtle change to the picture so now what we have is we still have this representation of the current state of the world but we've also got a second little area as as well and when I say change my address the first thing
• 22:30 - 23:00 that happens is we create an event object so this is this first class finger magig that captures the details of a change and we pop that first class finger Mig the event into some kind of separate storage area then we process that event and we process the event to do exactly the thing that we expected to do earlier we change the thing over the consequence of this is that we've kind of now got two sort of
• 23:00 - 23:30 representations of our world we have the what I refer to as the application state which is the current state that we're thinking of of the world and then we have a log of all the events that ever change that world and this is the crucial test for event sourcing the sort of the definitional test is that at any time we can blow away that application State and confidently rebuild it from the log that's the test I mean lots of people l changes and things like that it's the
• 23:30 - 24:00 ability to replay the log and confidently rebuild your state so that basically you don't care about your your application State and trying to preserve it anymore because the vital thing is preserve the log if that's the situation then you've got an Event Source system so how many people here have built a system using event sourcing this approach smattering of hands a few hands how many people here have used a system using event
• 24:00 - 24:30 sourcing a few hands not very many how many people here are software developers put your hands up okay I'm going to assume you didn't quite understand what I've been talking about for the last few minutes because otherwise we are really screwed all those people who haven't used an Event Source based system and are software developers if that were true I would be really really worried in 2010 let alone 2017 because every software developer should
• 24:30 - 25:00 be intimately familiar with one particular form of Event Source system which is Version Control if you use git you use subversion Mercurial even if you use CVS and I have it was a long time ago but I used it you've used an Event Source system hell I use sccs you know shows how old I am and this is I think a very good point if vent sourcing is basically providing to your users A system that works on
• 25:00 - 25:30 their data the way version control works on your code that I find a very good way of explaining at least to software developers what event sourcing is about and a lot of things about event sourcing you can reason through by thinking well what happens in my version control system what does git do what does subversion do and then applying that same question to the application state that you're in in your um Version Control system your application state is your tree of files your working copy um
• 25:30 - 26:00 your work tree and the log is of course all the commits that have occurred um through now of course the system internally doesn't necessarily store it as all events but it stores it effectively as it were all events there are snapshots that are contained at various points to speed up performance which is basically a copy of the entire application State as at some commit but by moving back and forth between snapshots and using commits and being
• 26:00 - 26:30 clever about how that all inter plays it gives you the sense that I've got every single commit which is kind of the same thing of saying I've got every single snapshot but it's slightly more powerful to think of it in terms of every commit now that's a good metaphor for a Geeks it's not a terribly good metaphor for business people but fortunately there is another really good example of event sourcing in the business world that any business person will be familiar with
• 26:30 - 27:00 accounting ledgers right if I want to my current application state is however much money I've got in my checking account but of course behind that is every single credit and debit to that account and of course ledgers and accounting systems are used by all sorts of people and that is very much an example of this event sourcing approach and again notice the fact that it's a combin in reality it's often a combination of every single change and
• 27:00 - 27:30 certain snapshots a common thing in accounting is to say at the end of a year or something would would say we're going to close out the year we calculate all the balances for the end of the year and then we kind of ignore everything that happened before then and we build our SN uh events on points since then that's a snapshot effect where effectively are throwing away history behind the point in that in that case um but it can also be you don't throw away the history you just use it to make it easier to calculate what's going on for this
• 27:30 - 28:00 year's money um and if you really need to you can dig and go further back so again the storage is usually some mix of events and snapshots done in some way that is efficient for your capability but you want to always be able to synthesize that uh Recreation of the events so the question with event sourcing is how can we use this more widely because there are many interesting benefits but also costs with this
• 28:00 - 28:30 pattern um immediately off the top of the head you can say well it's obviously a great audit system if I want to figure out what the hell happened or how how on Earth did this person get sent this quote that was completely ridiculous I can look back through the events and figure it out and that's why anybody who's involved in audit really loves having an events s system that's why of course we have ledgers because that's one of the main reasons is because of the audit it's also a fantastic debugging tool if you're trying to figure out why your application State
• 28:30 - 29:00 got screwed up at some particular point in time you can actually take a copy of the system feed it the same events and watch what happened going back in time and of course the time travel going back and seeing what was the application State a month ago two months ago we're familiar from doing that from Version Control right and it's really useful to be able to go back and forth um across time um the alternative state is Al is effectively branching and that can be very useful in certain
• 29:00 - 29:30 circumstances um I've worked on a few systems where we might have notification of events and we deal with them um the example I always uses payroll somebody worked um for 35 hours in a week we record that we send them a check we update their sick pay benefits their pension benefits all sorts of Downstream stuff happens and then 6 months later somebody says oh actually it wasn't 35 it was 40 that's a really complicated exercise
• 29:30 - 30:00 to go and then undo well not you can't undo because you can't change it but then to recalculate what the adjustments should be for all of that but if you've got an Event Source system you say okay I'll tweak that I'll copy it to a different system tweak the event calculate what the current state is look at the two application States diff them and do an adjustment I mean it obviously is a bit more complicated than my casually saying well we just diff the application state but you get the F picture right you actually get you can
• 30:00 - 30:30 actually do it and it can be a long computation it can take oh a couple of minutes but that's way faster than doing it manually and that can be a really powerful thing last item on that on the list of good things is actually one of the most intriguing the idea of using a memory image this basically says if our long-term record of what the state of a system is a l of events actually I should walk over here and
• 30:30 - 31:00 give you this end of the room a treat Force the cameraman to wake up if I do give that long-term view then actually I don't need to store the application state in any kind of persistent form why do I need to store my application state in a relational database if I've got a log of all my changes I can have a purely inmemory system and that has some interesting benefits a great example of this is a trading system called
• 31:00 - 31:30 elmax um it's a retail trading system and it was built a few years ago and it has to be very high performance very low latency and they looked at various ways to try and achieve this and they ended up with saying we're going to run the entire system on a single threet and Java and the their headline was we can do 6 million transactions per second on this single thread really really blazing fast and one of the keys to doing this
• 31:30 - 32:00 was all of their data was in memory they have complicated transactions they have lots and lots of things to look at at once they didn't want to be going to a database they have it all in memory and they were able to process really really fast and if anything happened and the system went down they could very quickly rebuild the state from their application logs they would snapshot overnight so basically you run all the events from the beginning of today and work that way and in fact again one of the advantages of events is you can have multiple systems running off the same event
• 32:00 - 32:30 Stream So they had a very good way of having hot backups they basically ran two systems all the time with one of them being the lead one and if it went down the second one would instantly take over very very nice system very very performant one of the interesting things I was talking with one of the lead developers and he said actually I mean the performance was fantastic and and really got people's attention but one of the things that I really miss about programming in a different system is that is the nicest system of ever programmed in because there was no
• 32:30 - 33:00 database to ever deal with everything was in memory all the time and as a result we could have a really good Rich model it was particularly good for complex logic represented in in a domain model um it was a real pleasure to work in compared to many other systems um I'm surprised in many ways more people haven't taken that kind of approach it's a bit unusual and you have to set up the infrastructure to make sure that you're able to do the events with that but it is actually quite a powerful model and it may be getting more interesting as we get more towards
• 33:00 - 33:30 systems that have large amounts of nonvolatile main memory um because so much of our of our systems have been built under this assumption that we've only got so much memory we have to park everything else else on dis and now we got tons of memory it all fits in RAM for so many applications so maybe we should take advantage of that and yeah memory image is a way to do it but you still have to have some kind of persistent store and communication mechanism and the event log provides that so event sourcing can be very
• 33:30 - 34:00 useful in that situation so those are some of the upsides but as implied for a downsides okay I'll go back hope you enjoyed that not having to cran your neck for a while so it's an unfamiliar system that's bad because things that are unfamiliar are harder to work with talking to external systems gets a bit more complicated because if I want to be able to rebuild my application State I can't just say oh I called this external system two weeks ago what was the result
• 34:00 - 34:30 CU I can't call it again two weeks ago I have to make sure I save basically every response of an external system as an event um and make it part of the replay mechanism um I have to think about how do I store my events in a way that I can confidently replay them even as I'm changing the code that processes them so the event schema starts becoming interesting to be able to do that also a little little thing but awkward when I generate identifiers for things I have
• 34:30 - 35:00 to be careful that I can do that in a way that I can um replay this rule that says I must always be able to replay the events build the application State there are these little tricks little traps for you early on that you have to deal with and you have to think through how to get through those so those definitely do add complexity and you get other complexity that typically comes along for the ride one of the trigger for doing this talk and and the workshop was when I ran
• 35:00 - 35:30 into people that were running into difficulties with event driven systems but it wasn't clear what the source of difficulty was so I remember chatting to somebody on a project that was in difficulties and they were saying oh I will never ever use cqrs again I said okay you had a hard time with that oh rewind got that wrong he said I will never ever again use event sourcing and
• 35:30 - 36:00 I asked him why and he said well because every time I want to do everything I have this read model to build and the right model to build I have to do twice as much work and I thought well hang on that's nothing to do with event sourcing that's something to do with something else and you often find that people will Mis mischaracterize where the problems are with event sourcing I've often heard this because the problem and the problem on that project according to the lead architect was a sing synchrony they processed all their events in an
• 36:00 - 36:30 asynchronous Manner and asynchronous processing got difficult for people because asynchrony tends to be hard to work with hard to reason about now there's no reason why you have to have asynchrony with an Event Source system when I just make a local commit um in git that's not an asynchronous operation now okay when I when people are pushing up to Shared repositories then asynchrony comes into play um but there's none of that it's all synchronous operations when I'm doing a local commit but often people take
• 36:30 - 37:00 advantage of the event sourcing to introduce asynchrony into the system which can be a good thing because it improves can improve responsiveness but asynchron event adds complexity so the reason I put that as a kind of question mark there is asynchrony often comes along for the ride with event sourcing systems and it does add a lot of complexity but it doesn't have to be used you don't have to use asyn new invent approach even though it seems common that people
• 37:00 - 37:30 confuse that another issue is versioning versioning can get quite complicated in invent Source system because of this ability to replay and particularly when you think well the code of course is also changing in this system over time if I change the structure of my application State and and the schema that I'm using for my application State can I still process that event that was created a year ago what does it mean to create to do do that do I need to change my event schemas and version them versioning can
• 37:30 - 38:00 get messy again it doesn't have to be the elmax example I mentioned earlier on they didn't run into that problem very much because they only operated on a one-day snapshot and events from that snapshot so we didn't have to worry so much about the versioning aspects but it can be a killer for other systems um and I'll mention that there is a book in progress or uh close to some degree of completion by a guy called Greg young if you look on leanpub
• 38:00 - 38:30 about versioning for event sourcing systems where he goes into a lot of detail about the various options you have for handling event sourcing um so handling versioning and event sourcing Greg Y is the creator of event store which is a a stor system specifically designed for events he's been working in the event space for quite a while you can find a bunch of his lectures and things on video I've been trying to get him to write a book about event sourcing for years so far without success so if if you ever do see him tell him to write that
• 38:30 - 39:00 book so events sourcing is like most things a tradeoff there are pros and cons there are some really nice advantages there are complexities involved I do find it surprising that more people haven't made use of it approach however one more thing to mention about event sourcing though and that is exactly what we record as our events in the Event Source think about the code example for a moment I go into my code I see some
• 39:00 - 39:30 function with a name that I don't like anymore and I want to change it so I go and change it fairly straightforward refactoring operation how do I represent that change in terms of event sourcing there's two events going on here one is my intention change the name of this function and the second is all the actual edits that I actually have to do my source code which is not of course just in one place it's the Declaration
• 39:30 - 40:00 of the function and everywhere that calls it now as far as any Version Control System I've ever used is concerned it only sees the latter it sees a series of textual diffs that have absolutely no way that you can tell the intention from that except by looking at the diffs or and maybe the commit message to say that's that's what I've done and we've talked in times of how it would be nice to have um automated refactorings that were storable that were first class thingi jigs that you could apply across your project but in practice it's very
• 40:00 - 40:30 difficult to do that because of course a rename function method uh function cap capability for that would work with JavaScript is not going to work with Java is not going to work with python so I have to language specific changes in my version control system that starts getting very hairy while text works with everything we know how to diff and merge and deal with text changes so we rely on text but then we lose the intention we lose
• 40:30 - 41:00 what's the reason why same thing basically happens with business stuff as well we say I've got some intention of something I want to do and I might want to capture v as an event and I think of that as a kind of an input event and then I have how does this affect my internal knowledge of the world which is a lot more detail but also often loses important context and then there might be another kind of event which is an output event but says here's what changed so the rest of the
• 41:00 - 41:30 world knows about it it's important to realize these are separate things and to think about which ones you want to capture which ones you want to store because you might want to store more than one of these if you look carefully at the slide you might notice we have a slight problem the code isn't quite right the total is wrong what happens to me fixing this bug I've got to be able to replay if I
• 41:30 - 42:00 quoted that as the order price I can't just change it because I quoted it and announced it to the world but on the other hand I've still got to fix the bug but I've got to also be able to replay things start getting tangled again it's this versioning issue Greg Young's advice is quite firm he says don't have any business logic in between your event and storage because if you do you run into this Tangled
• 42:00 - 42:30 versioning mess but on the other hand if I don't have the input event stored I lose the driving intention of what I'm doing because that that internal event can be pretty complicated and messy one solution of course is to store both and in fact I've seen that done with some of the events s systems I've worked in um the the canonical style that I that I saw back in the '90s when doing this was we would store the
• 42:30 - 43:00 external event and then actually all the changes to the application state would be accounting entries so they were naturally event sourced because they're all accounting entries and so we'd store both and you might have a whole pipeline of changes the payroll example I said every little change change to your sick pay allocation change to your vacation stuff every one of those changes was a a accounting entry and so everything was stored as events flowing all the way through and you have a pipeline of
• 43:00 - 43:30 processes producing events producing triggering other processes producing events through the system and that was very useful um and that worked quite well and I suspect that's where we have to go part good part of the discussion we had at this Workshop was this question of what's more important the internal or the input event um and as I said I do think it's it's certainly important to be clear of what you're storing and to understand the ramifications of what means across um and I suspect most of the time you need to store
• 43:30 - 44:00 both okay so that's three of them so what's the fourth one well I kind of already hinted on that um and that is this thing called cqrs uh command query responsibility segregation a term um coined by Greg young whose name I've called out a couple of times already and he's talked about it I remember in fact the first time he talked about it was at a go-to conference or at least the first I saw him talk about it was at a go-to conference many years
• 44:00 - 44:30 ago the basic idea with cqrs is that you separate the components that read and write to your store uh permanent store um so you have effectively two separate models one for dealing with updates and one for dealing with reads um so and then the crucial thing here is that they are separate pieces of software separate component components and that you really only ever
• 44:30 - 45:00 use the command component command model when you're making an update you always read using something else I mean you might do some read in the middle of processing the command uh will do some reads but basically anything that reads a system is completely separate and the example I remember that he gave a this that was quite nice was a system that had to do a lot of complicated business Logic for all the updates and they didn't want to have to be but
• 45:00 - 45:30 they updated much less frequently than they read so it made sense to effectively calculate a lot of derived data and store it in the read model so that the read model could basically go direct to the UI and the command model then had to do all the complicated calc calculations this is a model that I am this is an approach that I would say you've got to be wary of I think it's an appropriate pattern for a number of
• 45:30 - 46:00 situations but it is one that seems to correlate a lot with people getting into trouble um and so you got to really kind of be careful and make sure you really understand how it works and what its benefits are and if you really need it when you're using it my sense is it's got to be pretty deep down in the tool bag um to pull this one out so good pattern when used appropriately which is of course true of almost any pattern um but you really have to pay attention to the when to use this I also find it gets
• 46:00 - 46:30 a bit confused because I've heard people say well I we think cqrs should be used everywhere because it's really useful to have different uh models for reading things and populating different models but to me that's stuff we've been doing ages I mean I've for ages been talking about the idea of reporting databases where you have your online transaction processing system and then you take a uh input out of it put it into a completely separate database for generating reports
• 46:30 - 47:00 people have been doing this for years you might do it every night from a extract transfer load you might do it in more real time using event updates these days um but that idea of having different read pictures that's to my to my mind if cqrs is this then cqrs is every system that was ever built um I think what's important about the cqrs model is the notion that the that command model the thing that's used for update isn't used by anything for reading it's only used in the process
• 47:00 - 47:30 and you've got a distinctly different model that's only used in the update area and that I feel is a relatively rarishock multiple transformation steps compose very nicely so pipelines of
• 47:30 - 48:00 Transformations I mean that's that should be something that's familiar to pretty much everybody um so four patterns all coming under the term event driven I found this useful and we found it useful in terms of our Workshop because now we feel that when we hear somebody say event driven we've got an idea of where to go next now we say okay we think one of at Le at least one of these four patterns is in play Let's ask questions about that and understanding
• 48:00 - 48:30 these patterns then can ask more questions about okay why are you doing this are you doing this appropriately um what are the common traps I mean the value of a patent is saying here's a particular technique some ideas about when when it's applicable or not and enough knowledge about the patent for you to know oh these are the things I should watch out for these are some of the things that I would expect to see as consequences of that that gives you a bit more leverage but something as vague as event driven to me doesn't give you
• 48:30 - 49:00 that because it it's too imprecise it's actually a similar problem that I I found with service oriented architecture service oriented architecture means so many different things the fact that somebody said they were using it didn't really tell me anything you always have to dig deeper and so a more precise term like microservices is better because it just carves out a small amount of that space and you've got a better idea what's going on of course when a term like microservices get super overhyped then that tends to lead to confusion um
• 49:00 - 49:30 because people aren't using the term properly so you know again it's not perfect but again with but with events I think we gain a lot by being a bit more precise what I hope you've got out of my talk is some sense of U what that Precision looks like and what these patterns are and so that you can have a bit of better idea when you're talking about event driven systems what they mean um the original article um there's the URL for it um you can see I've stuck the date at the front of a URL I only do
• 49:30 - 50:00 that for things that I think oh that's not going to be relevant I'll probably delete it in 6 months I got that wrong um but anyway do go and take a look at that it's a very brief summary but there's links from there to lots of other things um things like the article on elmax some of the stuff I wrote 10 years ago on event sourcing that's still fairly relevant um stuff of that kind and I hope you found that useful and I hope the talks that you will see later on in this conference will be better than the one thank you