Understanding and Addressing Tech Debt

SREcon25 Americas - Technical Debt as Theory Building and Practice

Estimated read time: 1:20

Summary

In this talk, Ivon Lamb explores the intricacies of technical debt (tech debt) from both an engineer's and a manager's perspective. Rather than offering a one-size-fits-all solution, Lamb discusses the importance of using metaphors to better understand tech debt's complexities and the nuances involved in communicating it to different audiences. Through anecdotes and metaphoric comparisons (like housework), Lamb emphasizes the need for tangible actions to address tech debt effectively within organizations. This thoughtful discussion highlights how tech debt, much like housework, is deeply embedded in the infrastructure, influencing both technical and interpersonal dynamics in unique ways.

Highlights

Ivon Lamb shares that there's no 'influencery one weird trick' to fixing tech debt, emphasizing custom strategies. 🔍
Despite not having an SRE title, Lamb brings a wealth of experience from tech startups to the discussion. 🌟
Housework is used as a powerful metaphor to better understand tech debt's role in our infrastructures. 🏡
The origin of the tech debt metaphor is discussed, tracing back to Ward Cunningham's initial concept in 1992. 📜
Communication is key—making tech debt comprehensible to decision-makers involves the right metaphors and approaches. 🔑 "
Practical and humorous stories from Lamb's experience illustrate how dealing with tech debt is often like unexpected housework. 📚
Lamb encourages using rich, personal metaphors to think about and convey the stakes and possibilities of tech debt. 💭
Empowering tech teams to handle tech debt requires an understanding of both technical detail and emotional impact. 💪

Key Takeaways

Tech debt can't be solved with a magic trick; it requires deep understanding and context-specific interventions. 🧙‍♂️
Using metaphors like housework can make tech debt more relatable and help stakeholders grasp its implications. 🏠
Communicating tech debt issues effectively requires tailoring the message to the audience's understanding and priorities. 📢
Organizations need to acknowledge tech debt as a shared responsibility and support initiatives to address it collectively. 🤝
Tech debt, similar to housework and infrastructure, accumulates naturally over time and needs regular attention. ⌛

Overview

Introduction & Background: Ivon Lamb dives into the world of technical debt, speaking from personal and diverse industry experiences. Highlighting a lack of straightforward solutions, Lamb calls for a more nuanced view using metaphors to navigate tech debt's complexities. 🏄‍♀️

Metaphor & Communication: By comparing tech debt to housework, Lamb inspires a deeper understanding of its role in infrastructure. Emphasizing metaphors as a tool for contextually conveying tech debt's impact, she sheds light on how it intertwines with organizational practices, culture, and strategy. 🏘️

Real-world Application: Through engaging stories and insights, Lamb illustrates the challenges of addressing tech debt in organizations. She underscores the importance of communication tailored to audience needs, advocating for collaborative approaches to effectively tackle tech debt issues. 🔧

Chapters

00:00 - 03:00: Introduction and Speaker Background The chapter titled 'Introduction and Speaker Background' begins with Ivon Lamb introducing herself to her audience and stating the topic of her talk, which is about tech debt. She mentions a conversation with Dan and Laura regarding the talk, indicating her straightforward approach towards the subject matter. The chapter sets the stage for a discussion on the complexities and implications of tech debt in the tech industry.
03:00 - 09:00: The Metaphor of Tech Debt The chapter delves into the concept of 'tech debt', a metaphor for short-term compromises made in software development that can lead to long-term problems. The speaker talks from their dual perspective as both an engineer trying to resolve tech debt and as a senior engineer perceived as capable of fixing tech debt issues. The narrative highlights the complexities and occasional successes in managing tech debt, emphasizing that there is no single or easy solution to eliminating it.
09:00 - 12:00: The Origin of Tech Debt Concept The speaker discusses their background, highlighting that they haven't worked in large tech companies but have experience in tech-centric startups ranging from 40 to 400 employees. They mention that their knowledge of big tech comes from secondary sources like books, blogs, and other platforms. The chapter sets the stage for discussing the concept of 'tech debt' from the perspective of smaller tech environments rather than large corporations.
12:00 - 18:00: Social Concepts and Tech Debt The chapter discusses the speaker's perspective on big technology, acknowledging that others may have more expertise in the area. The speaker introduces themselves and outlines their unique viewpoint and characteristics as a speaker. They humorously note their own mediocrity in rowing and coxswain roles, providing insight into their personal approach and the idiosyncrasies that might be present in the upcoming discussion.
18:00 - 31:00: Examples and Metaphors of Tech Debt The chapter titled 'Examples and Metaphors of Tech Debt' uses the metaphor of rowing a boat to illustrate the challenges of managing technical debt. The narrator compares the role of a coxswain, who coordinates rowing teams, to the task of aligning teams for efficient software development while dealing with communication hurdles. They emphasize the importance of clear and concise communication to prompt quick and coordinated actions in a team setting, highlighting how these skills are crucial in navigating technical debt.
31:00 - 35:00: Theoretical Framework and Conclusion This chapter discusses the importance of balancing different perspectives when working with others, highlighting personal experiences and feedback from peers. The author reflects on a friend's description of them as 'disastrously forthright,' acknowledging the truth in this characterization. Additionally, a sociologist friend, Sarah Quinn, comments on the author's unique position as the 'most academic, non-academic' she knows, illustrating the blending of academic and non-academic insights in their approach.

SREcon25 Americas - Technical Debt as Theory Building and Practice Transcription

00:00 - 00:30 Hi. Oh, my name is Ivon Lamb and I will be talking to you about tech debt. Um, when I was talking with Dan and Laura about this talk, I said, "You do realize that I am not going to do the
00:30 - 01:00 influencery one weird trick thing, right?" Like, I can't tell anybody how to get their tech debt fixed, and that's the thing always want to know. um they said they understood that. So that is why I'm here and I am speaking both as an engineer who tries to get tech debt addressed and also as a senior enough engineer that people think I can get their tech debt addressed which you know sometimes I get lucky and I can and that's super nice. Okay, so me, the professional version,
01:00 - 01:30 um I'm a little bit of a ringer here because I've never held a job with an S sur title. Um most of the companies I've worked for have been tech startupish. Um somewhere between 40 people and 400 people in size. So I've never worked in big tech. While I'm somewhat informed about how things go there, um from books and blogs and blue sky and whatever, um it's not my world. And it's not my direct experience. I
01:30 - 02:00 mean, I may say something about big tech, but you know, there are people there there are plenty of people here who know way more about it than I do. Um, and also me. Um, just to give you an idea of what I'm like as a speaker and sort of the idiosyncrasies of the point of view that I'm likely to bring up in this talk. Um, I am an extremely mediocre rower and coxin. They So, what that means to those of you who don't know about rowing is that um I spend a
02:00 - 02:30 lot of time trying to coordinate with other people in a boat and figure out how to row row together and make the boat row well when we can't really talk to each other. And as a coxin, I am the little tiny person that sits in the front of an eight and tells people like like in two power 10 right here. and you know I expect them to do it and I have to convey a lot of information to get them to act very quickly. So um for me
02:30 - 03:00 that is two useful perspectives to have and it's a way that I've been teaching myself or relearning how to work with other people. Um my friend Jay Kofsky when we were in graduate school described me as disastrously forthright and that is pretty true. Um Sarah Quinn, who is an online friend who also happens to live in my neighborhood, um she's a sociologist at the University of Washington, said, "Wow, you are the most academic, non-academic person I've
03:00 - 03:30 encountered." Um, and my friend Ben Rosenart, who is also an online friend, the first time we met, he sort of looked at me and said, "So, are you the kind of person who tends to work on tech stuff when you're on a non- tech team and non- tech stuff when you're a tech team when you're on a tech team?" And I was like, "Uh-huh." And he's like, "Okay, we we can be friends now." Um, so that's kind of where I that's kind of where I stand. I'm a little bit not one thing or
03:30 - 04:00 another. Oh, and um when Nile was helping me do a review of my CV, he's just like, you know, you're not really very legible. I mean this in the kindest possible way. And I'm like like and I sort of winced and I sort of laughed because it is true and it is something that I work on. So to be really clear, this is not a talk about how you talk to your boss or a PM or you know that other team over there about tech debt, right? This is a talk about using metaphor to
04:00 - 04:30 think about the shape of your tech debt so that you have a better understanding of the stakes and the possible approaches that you know you can then think about how to convey to someone else. Like I think a lot of times we skip the step of trying to use a metaphor that makes sense to us and kind of pretend like we're filling out a form. So we try to jump straight to okay, what is the form asking for? like uh uh uh you know like value to customers or you know business value or
04:30 - 05:00 you know whatever it is and you know we don't really sit down and think through like okay how do I think about this and then how am I going and then take the second step of how am I going to translate this to a metaphor that makes sense you know for my context whether that's your org or the business or you know I don't know your boss is an avid golf player whatever right the way that you convey those stakes will probably involve metaphor, but I can't tell you what that metaphor is
05:00 - 05:30 because I don't know where you worked and I don't know what you know what your business memes are. So, um I didn't notice until I was pulling the screenshot for this slide, but this is me almost exactly four years ago by the date, right? Like so um I wrote a Twitter thread which is now archived at blue sky and mastadon almost four years ago to the day where I was ranting about tech debt and I don't remember what motivated this but I
05:30 - 06:00 remember that I was in a mode um and it was just like look I really think that housework is the right metaphor for tech debt for the thing we call tech debt which I am fine calling tech debt but like you know we can't use that metaphor because we mostly you know like in tech, you know, like it's mostly been made up of people who don't do housework or manage housework being done. And like this kind of got this huge reaction and even now the thread goes sort of viral sometimes, like it just comes back,
06:00 - 06:30 somebody finds it and um it haunts me a little bit, but I I think it's, you know, like it made me feel good about giving this talk because it's like, okay, tech debt is clearly something that people have a bunch of energy around. Um, and you know, the response to the thread made me think a lot about how we use metaphors to communicate and to think. Um, so second second social media slide. So this is so Fred knows that I'm
06:30 - 07:00 doing this by the way. I did ask him. Um, this is absolutely not to embarrass Fred, but when I saw his post about how, you know, just any, you know, like just an any random comment about a tech thing like can just provoke, you know, this long stream of consciousness dissertation practically about it. But, you know, it's like if I have to make this into a talk, like I am going to be jumping up and down on this for a couple of weeks to try to like get
07:00 - 07:30 it, you know, get it into a shape where I can convey to people who can't stop me every other word, you know, like what am I what am I like what am I even talking about? Um, but to me it's such a great example of how we can be very fluent talking about something when the stakes are low, but when the stakes get even a little bit higher, right? Like we totally we just like, you know, we freeze or we do something weird, right? Like to me, that's one of the things
07:30 - 08:00 that makes dealing with tech debt hard is just the difficulty of talking about it to different audiences, right? Like we can all go on for ages about the tech debt in our infrastructures. Um you know you know or in our organizations and we have lots of opinions and we can share them with you know our professional colleagues at the conference bar. But when somebody with even like a little tiny bit of power to
08:00 - 08:30 help us make it go away says okay I get your upset about this. Walk me through it. Like why does it need to go? and we were just sort of like oh it's bad. Um which is you know not an ideal response but in some ways that's actually a better response than thing than the thing that I have definitely done and that I have seen other people do where we just kind of go into the weeds right in front of this person and we start reeling off details and negative
08:30 - 09:00 consequences while their eyes glaze over. um like you know it's like and and like and we can't stop like at least when I get like that I can't stop I don't know about you maybe you have more self-control right but that response feels terrible to me because um it makes it it makes it feel harder to me about having the conversation that needs to happen because I feel like I've kind of messed up this first opportunity where you know it's like okay they they
09:00 - 09:30 gave me an opening and I completely ran them over. Um, okay. I'm just going to need to take some time to recover and we will have to regroup and try to try this again when I'm a little more together. Um, so origin of tech debt as a concept. Um the canonical story is that it was coined as a I don't know I don't know whether as a
09:30 - 10:00 linguistic unit by by Ward Cunningham in 1992. There is a video where he's being interviewed and he talks a little bit about this. He was so I will hit the high points of the video and not try to play it now because I it's interesting but it you can watch it on your own. Um, but he was very much influenced by reading Lafen Johnson's metaphors we live by. At the time he was working on a small talk project for a financial services firm, which to me says
10:00 - 10:30 something very specific about his context. and he was using debt as a way of explaining the need to refactor the product in the light of new information in the light of new knowledge that they got by shipping the product and like letting people use it, right? And he was trying to make the point that a little debt speeds development. So it was good to release and let it let people use it, but they also needed the prompt rewrite
10:30 - 11:00 in order to um incorporate the new things that they had learned. And his quote was every minute spent on not quite right code counts as interest on the debt. So that was kind of the reason why he picked that metaphor. Like he was in a in a um in a context where he felt that that a financial metaphor would really resonate with people. Um, sort of a handwavy definition of tech debt, but it's one that I like. Um, is Jean Kim's where, you know, tech debt
11:00 - 11:30 is just that thing you feel the next time you go to do something and try to make a change and it's just like all of that stuff that gets in the way, right? Like all of that stuff where it's like, oh, right, and I have to do that and I have to do this other thing and I have to do this other other thing. And I don't know about you, um, but I definitely have ADHD. And so it's like when you give me a big long list of things that I have to do in or before I can do the thing that I set out to do, it's like uh uh yeah, we don't like
11:30 - 12:00 that. Like and it's like what I was like, uh why don't we go see what's going on in Blue Sky for a minute and then we'll just come back to this. Um so I kind of like I definitely feel Jean Kim's definition of like it's what you feel when you're like, "Okay, I need to go make this change only. I can't." Um the the one problem I have with it is that I think it focuses too much on things that people did wrong with wrong in big quotes. Like there's too much
12:00 - 12:30 emphasis on like what you didn't do or you didn't do right. Um, and I think that there's kind of a category of tech debt that we talk about like as working engineers where, you know, like it's really like, okay, like somebody did this thing, it probably made sense for a while. And, you know, now for some reason it just doesn't make sense anymore. We need to make it go away. And it's like, okay, but but how? Um so Michael Feathers who is the legacy code guy tells a story about um asking a
12:30 - 13:00 colleague what he was working on and the colleague was like I'm writing more legacy code and you know again I you know like like I sort of have this position that tech debt is this kind of thing like we just accumulate it by existing um by trying to do things. So tech datad is metaphor. Um, one of the coaches at the boat house where I row is he got his undergraduate degree in anthropology and he wrote a thesis about rowing that I
13:00 - 13:30 read and it's organized around the idea of social concepts of you know things that make sense to a particular community of practice or social group and that helped me structure this talk. So and I really appreciate that. I also appreciate the fact that he was right there the time that my rowing partner and I uh flipped our boat and he had to fish us out of Lake Union. Um that was recently. So it is it is in recent memory. Um it was 44 degrees in the water. That's
13:30 - 14:00 cold. So tech debt is pretty clearly a social concept. like you know we're here talking about it and like we don't have to have go into a long conversation about what it is like you know we could have a conversation about this kind of tech debt or that kind of tech debt but we kind of know what the right social response is when somebody says I have tech debt like you know nobody's going to be like oh that's really good um but it's not only a social concept in
14:00 - 14:30 one way right like and it's like I think different organizations have different ideas about what tech debt is or how they want to operationally define tech debt and it's also um not only a social concept right like it's also an aesthetic concept or on a practical one and so forth and you know when I was putting this together one of the things I was thinking about was like so how powerful a social concept is it right I mean like how much explanatory power does it have to say tech debt and I
14:30 - 15:00 think the answer is Well, it's powerful enough that we kind of know what it is. We kind of know what response is expected of us when somebody says tech debt, but in order to do something concrete with it, like, you know, it's not a magic word. We can't just say tech debt and expect people to jump on it, right? I mean, in order to get it addressed, we have to talk about it with more specificity and clarity. Um I so one of my old bosses like to talk about
15:00 - 15:30 pre-work and pre meetings and stuff like that and I'm I'm not really fond of the concept of pre-work because I feel like look it's all work like we need to do this like why are you making up this other name for it but there is a sense in which I am going to talk about pre-work because what I'm talking about is the work we need to do to understand a piece of tech debt in a way so that we can that will help us talk about it with other people, right? Like not the details of it. Like I am I am 99%
15:30 - 16:00 certain that any person in this room if I said tell me about your de your tech debt in detail like they could do it. Like you would not have trouble telling me everything that's wrong all of the details about your tech debt. But is that the conversation that's going to help you get it fixed? And I think and I kind of think the answer is like most of the time no. um you know when people say this thing is tech debt right like they expect a particular response and they're mostly
16:00 - 16:30 disappointed because like they're not getting it like we're not getting it. Um and you know to me that speaks to a lot of things about their context but it also says that the metaphor you know like we're kind of expecting the metaphor to do work that it's not doing. So we need to do some human work around that to get to, you know, to convey what we mean and not just lean on the metaphor um to to make it happen, right? Like, you know, it's a like tech deck is
16:30 - 17:00 a social concept, but it's not a social concept that exists out there with no room for individual interpretation or um interventions. and you know like even organizational ones like all kinds of layers of of like but what do you really mean by that? So I think it and I tend to think as engineers we we tend towards the thinking the kind of thinking that I been known to call like throw another abstraction in the canoe. Like when things get messy we just kind of look for another abstraction to put on top of
17:00 - 17:30 it to hide the mess. And I'm just not convinced that tech debt is that kind of abstraction or you know not the way we want it to be. Like I mean you know I I think a lot of people would like it if they could just say this is tech debt and we'd all be like yes okay it's tech debt. going to go fix it and you know I I don't think that I mean like that doesn't happen like people are pretty unhappy about it. So this is kind of my picture of the world. It is probably not the right picture because it looks too mathematical. This is not intended as a
17:30 - 18:00 communive diagram like for those of you who do mathy things. Um it's supposed to be an illustration of a constellation of ideas that I've been thinking about and I will explain about Susan Lee Star and Hyram's law in the next couple of slides. But what I want to lay out is my argument for using housework as a metaphor to think about tech debt, right? Like, and to be clear, it might not be the metaphor you use to convince your org to let you do something about it, but it might be the metaphor that
18:00 - 18:30 you use to figure out stakes, approaches, and like, you know, what the next level up metaphor is, right? So, you know, like housework works really well for me. It seems to resonate for a lot of other people. I don't think it's the only way to do it, but you know, they didn't ask other people, they asked me. So, here you go. Um, and to be clear, I'm using housework in the sense of taking care of our physical surroundings, which is a definition that comes from um somebody who works with people to do
18:30 - 19:00 housework when they have physical or mental or emotional conditions that just make housework hard for them. Um, and that's a that's an account I guess called team unfuckyou habitat. There will be more about them later. Um, but why do I think housework and tech are related? Like how did I even get here? And the short answer is that I think tech debt and housework both share in qualities of infrastructure.
19:00 - 19:30 So on the left side of that diagram we have we had I had Hyram's law scrolled in traditional mathematician by a bunch of arrows. Um so Hyram's law states with a sufficient number of users of an API. It does not matter what you prom promise in the contract. All observable behaviors of your system will be depended on by somebody, right? A, you know, like a really common example of this is performance, right? Like people
19:30 - 20:00 are going to rely on the observable performance of your service no matter what the contract says. You know, it's like if you get way more performant or way less performant, well, congratulations. their timeouts may not do the thing that they used to do and people will be cranky. Right? So, um, Hyram's law was coined by Hyram Wright when he worked at Google and he worked on a team and I learned this from Laura Nolan when I was working on this talk. Um, he worked on a team that specialized
20:00 - 20:30 in looking for large-scale patterns in Google's codebase and automatically generating change requests to bring them into line with whatever their desired practice was. I have the feeling that he knew the hard way about people depending on observable behaviors. Um like I have worked across a lot of repos before never that many. And I just sort of feel like oh god I mean like this like it would be awesome to be successful at this job. And the thought
20:30 - 21:00 of like trying to do the job is really terrifying. Um so I think he knows whereof he speaks. So, I apologize for the wall of text, but this is Susan Lee Star's properties of infrastructure. Um, Star was an American sociologist, and she's best known for her work on infrastructure and standards in which she applied traditional ethnographic techniques and ideas to studying digital infrastructures. Um, and there is a
21:00 - 21:30 there are links in um my slide notes which I will release after this talk. Anyways, Star's point is that relational that infrastructure is relational in the sense that it really depends on a point of view and an actor. So it so an infrastructural thing can be top of mind for some people and like nowhere on the stack for others. I think the example she gives is that if you're an engineer working on the electrical grid, then
21:30 - 22:00 that's your focus. Like that's the thing that you care about. But if you're somebody trying to use your electric stove to cook dinner, right? Like that infrastructure is just a thing you plug into, right? Like you you know that it's like you like your stove is plugged in, it better work. You're going to cook dinner. Um and to run quickly through these through her properties of infrastructure, you know, that it's embedded into and inside and with other structures like organizations and software and um products, all kinds of
22:00 - 22:30 things, right? like it's transparent to use in that it invisibly supports tasks like the electric stove. A lot of us just you know like we don't think about a lot of infrastructural things and you know we kind of resent it when people don't don't think about the things that we support except when they break and it's like I'm sorry to break it to you that is a property of infrastructure. Um so infrastructure has reach or scope meaning that it extends beyond a single
22:30 - 23:00 site or an event. Um, it's learned as part of membership in a community of practice, right? Like like all of us here, I mean, how many like how many people have started a new job and they're like, "Oh, well, we have Kubernetes." And you look at it and you're like, "What the um Yeah, see I mean and it and their path dependencies on previous practices like it doesn't just appear out of nowhere, right? Like there's a there a bunch of stuff. It's like, well, you know, this thing is this way because of this decision that got
23:00 - 23:30 made in the past that is was really to work around that, but that no longer exists. And then you're just sort of like, okay, I just Yeah. Right. Um, and so infrastructures also work with other infrastructures through shared interfaces, which is what it means to be part of a community of practice or conventions of practice. It's built on what came before and it becomes visible on breakdown. my favorite. And it's also fixed modularly, not all at once, which
23:30 - 24:00 I think is a very interesting point that we could talk a lot more about, but not right now. So, here's my argument in a slide. Um, tech debt and housework share qualities with infrastructure that make housework a powerful metaphor for thinking about tech debt. Um, so, and I'm going to be very specific here. So to me, I totally get why people want to talk about about tech debt in a financial
24:00 - 24:30 sense. Like sometimes that can be really powerful because business leaders understand, you know, risk. They understand, you know, like things like interest, but I don't think that way. I have to do a bunch of work to get to the point where like I have a nicely packaged set of things that I can then apply, you know, that metaphor to. Like at Chef, we had a bunch of corporate values and one of them was do the right hard thing. And as you can imagine, like we argued a lot about like what the
24:30 - 25:00 right hard thing was, but you know, and it's a hard argument to win unless you know, like you can be clear about what you think the stakes are and argue for them. um like you can't you know like I I feel like a lot of times people want to fill out a form and you know just like apply the meme and it's like well you can't apply the meme I I think you actually there is something to be explained um so there are lots of people by the way like in the discussion track today
25:00 - 25:30 there were a lot of people from fintech there and you know like they really disagree about the um they you know like they would not agree with me about housework being like a good thinking metaphor like they they are so used to dealing with debt that for them and they know way more about debt than I do, right? Like I was planning to read David Greyber's book on debt before giving this talk. It did not happen. Um but you know they know more about financialized debt than I do. So you know that makes it a powerful metaphor for them. Um one thing I do think is that part of the
25:30 - 26:00 reason why financial things appear to be so legible is that there is a whole bunch of infrastructure built up around them. like you know there is recordkeeping and laws and regulation and specificity right like even pretty small organizations generally employ people who work on maintaining a certain amount of clarity around financial debt like whereas if you ask an SR about tech debt like usually we're just like we sigh we
26:00 - 26:30 rant or we launch into a horror story right there are times and places for all of those but you know they're not necessarily the utterances that will help us think through our tech debt and get us help and especially not from decision makers. Okay, examples and horror stories. We're going to do this. Um, this is not an exact mapping of housework to tech debt story by the way and on a different day I might put different stories with different lessons. Okay, the toilet or people want
26:30 - 27:00 maybe need to see a positive change. This is the story that when I said, "I'm going to do this." And my partner was like, "You are not." And I was like, "Yes, yes, I am." So, here we have a picture of a bathroom with a toilet in it. It is my toilet. It is not a particularly elaborate toilet, but it is extremely water efficient. Um, we do not have this toilet because we deliberately set out to upgrade. I just want to point this out. Does this is this starting to sound familiar to anybody? Right. The old toilet was fine, but it was original
27:00 - 27:30 to the house and it wasn't water efficient, but it's not like there's ever a good time to replace a toilet. So, you know, we talked about it for years and we're just like, well, I mean, it's just not that big a piece of work and we'd have to get somebody out and you would have to find somebody and I like I don't know, you know, so it just didn't happen, right? Um, the reason why we have this toilet is that we had a side sewer go out. Now, I
27:30 - 28:00 think I can look around and sort of tell who's a homeowner here because or especially a US homeowner. So, this might require some explanation, right? Like, so a side sewer is the part of the sewer line that runs from the street to your house. I learned from Nile yesterday that it's called a drop um in Ireland and critically in the US the thing that matters is that if anything goes wrong with it, it is your problem as the homeowner, right? So, and things
28:00 - 28:30 that go wrong with side sewers are expensive because they are buried deep. Like they're I I think like regulation is six to eight feet underground. Um, and so you know there's there's time for a lot of stuff to accumulate on top of your side sewer. Anyways, it's not like there's ever a good time to replace a sides sewer, but I extra special resented having to do it in the two weeks between, you know, like in two weeks like right after Thanksgiving and a little bit before Christmas. Like it was not fun. And replacing it involved
28:30 - 29:00 digging from the street level, then under a sixoot rock wall, then up an incline to the house. Like it was it was a lot of work and the plumbers like it took weeks just you know between people's vacations and um you know just the amount of work and things that they found along the way. But the plumbers were at the point of starting to put everything back when we realized that um you know being able to
29:00 - 29:30 flush is really important. But we were also going to feel really bad about having spent all this money only to have everything look exactly the same as when we started, right? Like like see okay like like people recognize it. So we decided to have the plumbers install a new toilet for us while they were there. And since it was sort of a spur-of-the- moment decision, we just took the toilet they that they had, which was a fairly fancy, very water efficient one, and we're like, great, you know, like this is nothing compared to the to the cost of
29:30 - 30:00 replacing a side sewer. So, there is a story that goes with this, which is package repos at Chef. So when I when I started on the release engineering team at Chef, our first big project, like the way that we thought, oh, sorry. Our first big project and the way we thought about our first big project was to replace our bespoke manually generated Jenkins build pipelines with ones that were standard, built with code, and deployed using Chef, right? Like that's that's pretty good. Like I mean it was it was pretty
30:00 - 30:30 good for 2013. Like and we were really excited about doing this. But what excited our internal users about this work was they were told and you'll get external facing package repos for customers, right? Like we weren't doing the work we were doing to give them package repos, but package repos were definitely the thing that people were most focused on. Like it was actually a little bit weird because you know people would ask me you know like I would give
30:30 - 31:00 a status report and then people would be like so how are those package repos coming and it's just like uh you know like it was just a it was just a a mind shift to have to be like right that's the thing they care about we need you know like we need to talk about that um you know but it was very effective like people were very were very patient and very happy once they got their package repo. So, I had a couple of takeaways from this. The first one I feel like is one that most people know, which is you need to find a win for your internal customers. Um, and that's hard
31:00 - 31:30 for infrastructural work. Like, it's not just tech debt. Like, a lot of infrastructure work is like this. It's just not the kind of thing that your internal customers necessarily get all that excited about, right? Like, um, someone I know who might even be here, I thought I heard their voice, but I didn't see them. um was talking about how their team after a whole lot of work started being able to support IP IP IPv6 for their company's customers and they were so proud and they announced it
31:30 - 32:00 at a company meeting people didn't know what it meant and you know and that's really disappointing right like it can really help both you and your your internal customers to have a win that they get excited about whenever I say that somebody thinks I'm talking about you know like how you should do more work in order to make it so that people so your customers can get a new shiny and it's like no that's not what I'm saying at all right like technically there's no reason why we couldn't have just given them
32:00 - 32:30 um given them in package repos right away but we had to get to you know like our argument was package repos don't mean anything unless you have a reliable build process where you don't get the thing where you have like not you know like 10 pipelines run but and and you that nine packages uploaded but one of them fails. Like what does that even mean? That's an indeterminate state. Somebody would have to do something about that. Um so and for me one thing that made
32:30 - 33:00 this project nice to work on was that the team was a pretty good mixture of people who had been a chef for a while so they knew all the lore and the people who didn't and didn't have any attachment to the way things were. So you know it was not you know like we could have replaced our toilet and you know we could have taken a weekend and done it. we there would have been feelings, right? Like having the plumbers do it, that was a good choice. So the fridge or path dependence. This is my fridge. Um it does have helmets of content management written on the whiteboard and I will
33:00 - 33:30 happily argue that but not now. You will notice that the area over the fridge is weird. That's because there used to be a cabinet there. When our previous fridge died and we had to replace it, what we found out is that to make a very long story short, um, American fridges have gotten bigger. My house was built in 1947. I can tell you that doors do not get bigger just because fridges get bigger. Um, so the only fridge we could find that would fit through our doors. Um, it
33:30 - 34:00 had all the features we wanted was an expensive imported one from Australia and it was too tall for the space and we were just like, "Fine, we'll take the cabinet out. We don't really use it. It's over the fridge." Um, and getting the old fridge out was also a thing. And it convinced me that they must have put it in the kitchen when there were no when like the house was being remodeled and there were no moldings. So, story, it was easy to install. So I was working at a midsize startup and my team inherited the care and feeding of the
34:00 - 34:30 package repo that customers used to install the things that they had paid for. Um when we took it over the person who did the handoff said casually that oh well you know we had to do this in a hurry. So we picked a thing that would be easy to install using the workflow we already had. So basically they picked the package repo because it had a helm chart. Um and once we started running it we found out that thing had all kinds of problems. One of which is that deleting things from it was hard and it hard enough to take like a day or two and you know I think it should be a little bit hard to delete packages from a package
34:30 - 35:00 repo but two days hard is kind of a lot. Um anyways during one of our fights with a thing one of my teammates found out by watching a talk that that software was only accidentally a package repo. what it really was supposed to be was mirroring for other package repos like which kind of explains like why it was so hard to delete anything. Um, and it took a year to get rid of but I wasn't there by then. So my takeaways and it sounds depressing to say like nothing would
35:00 - 35:30 have saved us but it is in fact true that all the critical decisions about this thing had been made you know before we ever got to it. And I want to be clear that, you know, I kind of make fun of the installing it because it's easy to install, but in fact, it worked fine most of the time. I mean, there were things about it that didn't work, but those things also changed when my team took it over because we were different people. Um, we had we knew about how packages got uploaded. Like, so people came to us with different
35:30 - 36:00 problems because they knew that we wouldn't shrug them off. Um, and I have a rant about how our industry prioritizes the getting started experience and the advanced experience, but does nothing for the middle. I'm not going there because it's a whole different talk, but I'm saying it so I can put the boundary there and move along. Um, one thing that this project did was make me really determined to stop people as much as I could from making the same kind of mistake we did where we just installed a thing because it was easy. And I was like, so when I
36:00 - 36:30 stand up a new service or something, I try to make it really clear like what it is and what it does and what I meant for it to do. So like it's really clear that it's like look if you use this for something else, you know, just like you're kind of taking your life into your own hands. Like it might work for you, it might not. You know, you you might be stretching the use case. Um so that's a thing that I've tried to do the closet or do we have to fix this all right now? So, this is a photograph of my bedroom
36:30 - 37:00 closet from the outside. Um, as you can see, it sticks out from the side of the house. What you can't see is that this is the northeast corner of the house. So, it's the coldest, darkest place to put a closet. The reason why that closet is there is that in the 80s, somebody who lived in the house decided that they wanted another closet. So, they they or someone they hired took a saw and they cut a hole in the external wall of the house. Then they built out a closet, stuck a roof and some siding on it, and set up a terrifying inside light that involves lamp cord and extension cord.
37:00 - 37:30 And I did not show you a picture because I know that there are amateur electricians here, and that you would just take one look at it and like not hear another word I said. Um, it it is not good. It is, but it is a working closet um that is around 8 degrees Fahrenheit colder than the rest of the house in the winter. And I do not use the terrifying light because I do not want to be a crispy critter. We also found out that the inside edge leaked under certain circumstances. But you know the thing is it's like like you know this is tech right like I mean it
37:30 - 38:00 seems like replacing this is a major thing and like how good would it feel to get like a closet that isn't stupid um even if you think this closet is really stupid. So what we tried to do was um when we've worked on other things in the house, we've tried to fix various things about the closet as appropriate like we had the roof fixed and we had flashing put on the edge so that it does not leak. My story about that goes with this is Solaris 10U1 support at Chef. So this
38:00 - 38:30 was a saga. Um so when I worked at Chef, we supported our product on some old and pretty esoteric platforms. So we had to have physical hardware for them. And when we were replacing the entire build system to make it more uniform and more reliable, ops came up. It's like, by the way, you know those Solaris build machines, they are really expensive to run. They've got this really big power draw and they're not like anything else that runs in that data center. And so we're kind of special and it's very expensive to be special. But at, you
38:30 - 39:00 know, so um one of my teammates at the time was Scott Hayne who knew all about weirded esoteric platforms. And so, you know, he did he worked with Ops and they found, you know, a less crazy Solaris machine to build on. And, you know, you know, that we did not that we would not have to support via eBay. And so, they got them and this was great. Like like you laugh, but I am serious. Um, so they, so they made a plan and they announced it and it was great and they thought everybody's going to be happy until one of the devs said, "Okay,
39:00 - 39:30 because we currently build on Solaris pretty ancient, we accidentally get binary binaries that will run on Solaris 10U1. Will will, you know, will the new hardware do that?" And we were just like, uh, we don't know. And I So I fired back a question that I thought was perfectly reasonable. It's like, well, is anybody running Chef on Solaris 101? and like no I mean like nobody knows like that was the problem. So we went down two parallel paths. Scott's task was to job Scott's
39:30 - 40:00 job was to figure out like can we build something on the new hardware that will work that will work on Solaris 10U1. My job was to figure out whether like whether we needed to. Um and I thought oh someone must know no not at all. um it wasn't in Salesforce as far as I could tell and I started asking around and you know if you really want to freak out your customer teams ask so does
40:00 - 40:30 anybody use that do we still need to build on build things that run on that because they they will be like it's like we can't we we can't cut people off and it's like I don't want to cut people off but I don't want to support them if they don't exist either. Um, and it ju, you know, it just went on like we just kept finding new things. Like I asked our brand new project manager, product manager about this and he started running around like a headless chicken which was nice because I had company but um it wasn't really helpful because it's like well I asked
40:30 - 41:00 because I thought you might know if you don't know like I can run around like a headless chicken and you don't need to do that. Um, somebody told me about a new telemetry project at Chef. So I went and asked Chris Doherty who's was the guy running that and said um you know like what and then he started laughing at me and he's like so we get all of the ohigh data which is the part that where Chef knows what you're running on um back but we get it back in the form of a binary database blog. Um so we can't query it. Sorry.
41:00 - 41:30 Like and I was just sort of like what do you mean like like this is all very depressing. I mean, the good news is that Scott figured out a magic solution because Solaris really wants to be very backward compatible so we could build on it, but like like none of us were happy. Like this was not really satisfying. Um, my takeaways like keep asking both like do I really have to do this and can we do this? Like be pragmatic but also be very intentional about it. Um, we got a lot of value
41:30 - 42:00 back. uh we got a lot of value going back to these questions like do we have to do this you know can we do this um it didn't feel good right like we just had to make a lot of trade-offs we found so many things along the way that we wanted to fix and we just had to put our heads down and say like like we are trying to answer some very specific questions we're going to keep doing that um and keeping an eye on the problem space will help you right and since a lot of people don't know what that is I
42:00 - 42:30 threw a slide up on the definition. It's a term that I heard from um Indie Young and it basically means um a problem space is sort of like the whole context of the person or you know who might try to use your thing right like and it sounds like the answer is obvious like well duh they're trying to run chef on Solaris 101 but it kind of isn't like who are they is there only one day or more than one day like you know like what we were really trying to do was like support our customers on the esoteric platforms but
42:30 - 43:00 on the necessary esoteric platforms in the most effective way possible. So my spice drawer or finding places for things. So um when I showed Nyall this slide, he was like but but you know okay that it's like dishes, laundry, trash, stuff that has a home, stuff that stuff that doesn't have a home. There's more of it than everything else put together, right? Um, I just want to mention that
43:00 - 43:30 Captain Awkward, team Your Habitude, and Casey Davis are all people who are really good at talking about um doing work and keeping up when you are um emotionally depressed, have physical disabilities, um just, you know, people who who aren't nec who don't normally who don't necessarily who are not necessarily at 100%. Like, you know, we deserve nice places to live, too. And so this find find homes for
43:30 - 44:00 stuff. Um, so one of my close friends has moments when she talks about retiring and spending the rest of her life designing the perfect spice rack. I don't need a perfect spice rack. Like we settled on jars in a drawer labeled on top with the name of the spice where I can skim. And you will notice that some of the lids are not labeled. That's fine. We don't use those enough to care. If we start doing it, I'll label them. Um, it doesn't have to be perfect. the spices live in a drawer. Coffee, we know where they are and we can find them when
44:00 - 44:30 we need them. Um, examples, right? So, a good example of this is ETS's deployator because I love this story, right? Like they did not stand up the guey for deployator because they had a perfect deploy system and a perfect UI, which is what a lot of us would do, right? like they stood it up because they needed to make a division between the part that people the back end that like needed a lot of work and the front end that people need in order to do their work. Okay. And I see I'm running
44:30 - 45:00 low on time so I'm going to skip this giant story but you should totally go and read the story about the open source Helm chart. Um I do have I I do want to talk briefly about I'm sorry I have to show you this picture. Um, this is what happens when we make a plan to for getting rid of tech debt before, you know, when we get out of over our skis and do it and we don't tell people,
45:00 - 45:30 you know, and we don't get people on board with this is the problem. This is um, you know, this is a problem and convince them to get on board with doing the problem so that they can find the problem a home and get get the right people on board. Like what happens when we make a plan is that it is like this poor cat who has been denuted of all of his adorable fluffiness and like he does not look like a happy cat. Um like don't so don't do that. Okay. So what does all this mean? Um so our story so far is
45:30 - 46:00 that that housework, tech debt and infrastructure are all connected and tech debt and housework kind of rhyme with each other and how we can think about them because of their infrastructural properties. um that thinking about tech debt, you know, with and against the labor of keeping and maintaining a house like it's actually I like I find it useful to think about right like it's a very rich area like I have made all of
46:00 - 46:30 these examples and people are like uh-huh uhhuh I see that and it's like yeah I mean like are these the examples you would necessarily want to use to explain things to you know your boss or your boss's boss or you know the PM maybe not I don't know like there's probably a metaphor that resonates more for them. But this metaphor, having a rich metaphor of your own can help you do the thinking so that you can figure out the stakes, you can figure out um like like what could we do? Like do we
46:30 - 47:00 have to do anything at all? It gives you more handles to play with and you know if if your company's metaphor resonates with you to that extent that is awesome but that has never been the case for me any place I've ever worked and housework is a is you know it's just a way to build a theory or a model. Um so not every talk that I write has a theory but this one does right like this talk is seriously thinking about theories. I'm thinking about like what makes a good theory like how do we build one and how do we use a theory to help us
47:00 - 47:30 communicate. So the title of this talk is a riff on a beautiful paper by the scholar Bell Hooks called theory is liberatory practice which is about black feminist theory and how it gets squished between feminist theory that come out of mostly mostly white academic spaces on the one hand and spaces where trying to think and generalize about your lived experience like is considered impractical. These contexts aren't the same and the stakes are definitely not
47:30 - 48:00 the same. But I love the combination of thought and emotion that that Hooks invests in the effort to understand as an individual and the effort to share and use and question that understanding with other people. Like you know I really love that she says let me begin to say that I came to theory because I was hurting. Like I came to theory desperate wanting to comprehend. Um, so part of what makes tech debt talk difficult to talk about is just that it like we all have so many
48:00 - 48:30 feelings about it, right? Like we notice this, you know, like we know this like we are all just sort of like, oh yeah, we could talk about this for a really long time. And I think that on on some level those are feelings about entropy. And it's fine to have feelings about entropy like entropy is big, right? Like the second law of thermodynamics says that systems tend towards increased entropy, also known as greater disorder. Like this is a big fight we're fighting here. Like that's a pretty reasonable thing to have feelings about. Um
48:30 - 49:00 something that I want to throw out there is that you know we should think about like what a collective response to that looks like. like what should we demand not just of you know our families and our and our workplaces but you know also of our professional communities of our professional organizations like what would it look like to support to have support for our doing this kind of work as opposed to people relying on the fact that you know a lot of us are missiondriven people with ADHD and you
49:00 - 49:30 know we go running towards the fire. So, I want to close with a speech from one of my favorite plays, Arcadia by Tom Stppard. It's a beautiful play about chaos theory and history and how we know things. And it's above all a play about entropy. All of the characters, whether they know it or not, are dealing with entropy. And this is spoken by a character who does know it. He's a mathematician who has a big pile of data and is trying to understand how it all fits together. The unpredictable and the predictable unfold together to make everything the
49:30 - 50:00 way it is. It's how nature creates itself on every scale. The snowflake and the snowstorm. It makes me so happy to be at the beginning again knowing almost nothing. People were talking about the end of physics. Relativity and quantum looked as if they were going to clean up the whole problem between them. A theory of everything. But they only explain the very big and the very small. the, you know, the universe, the elementary particles, right? The ordinary size stuff which is our lives, the things
50:00 - 50:30 people write poetry about, clouds, daffodils, waterfalls, and what happens on a cup of coffee when the cream goes in. These things are full of mystery, as in mysterious to us as the heavens were to the Greeks. We can't even predict the next drip from a dripping tap when it gets irregular. Each drip sets up the conditions for the next. The smallest variation blows prediction apart. And the weather is unpredictable. The same way will always be unpredictable. When you push the numbers through the computer, you can see it on
50:30 - 51:00 the screen. The future is disorder. A door like this has cracked open five or six times since we got up on our hind legs. It's the best possible time to be alive when almost everything you thought you knew is wrong. Thank you.