Complexity with Rob Hirschfield

Welcome to Definitely Maybe Agile, the podcast where Peter Maddison and Dave Sharrock discuss the complexities of adopting new ways of working at scale. Hello and welcome to another exciting episode of Definitely Maybe Agile with your hosts, Peter Madison and David Shark. And we have a special guest today.

Would you like to introduce yourself, Rob? I would be delighted to. And I don't know if I can match your enthusiasm. I'll try. I am Rob Hirschfeld, CEO and co-founder of Rack N, and I'm really excited to be here to get a great conversation running.

So welcome. I'm just happy, uh Rob, that we've got somebody else to listen and to bounce ideas off of and get input from, rather than myself and Peter just going backwards and forwards as has happened way too many times.

Yeah, but does this add to the complexity of the conversation? That's the lead-in, right? You see what I did there?

So we're we're intending to kind of look at complexity, just a little bit of an understanding around complexity and how that well affects, I think, the work of all three of us here. So maybe Rob, did you want to just sort of kick off with a bit of an insight from your side on what you see around complexity?

I would be happy to. Complexity feels like it's this up-and-coming uh challenge. You know, we we keep hearing about things being complex and getting more complex and complexity being uncontrolled, and we're all afraid of complexity. But I what I've found is we have a lot of trouble actually measuring it. I actually started thinking about something called Jevon's Paradox for Complexity. So this idea that we're gonna make things much less complex by using a service that then hides all the complexity, which is amazing, right? I don't have to worry, we do a lot of bare metal infrastructure and enterprise data center type of automation work. And everybody's like, move to the clouds and never worry about infrastructure again, which is hiding the complexity of doing all that infrastructure behind a service wall, which is yay, great. But as much as you might hide it, it's the complexity is still there. It's just not you've you've allowed yourself to move faster by not worrying about somebody else's complexity, but it's still there. And when we get these big cloud outages and things like that, we we suddenly remember that that that complexity could come back and bite us. And so it's really interesting to think as we move into this increasingly service-oriented world and microservices-oriented world, and we're connecting and coupling all these pieces together, that while we're hiding complexity behind a service interface, we're actually adding complexity back and like managing the services and worried if they're too coupled, or what happens if a library goes away like Log4j or a service goes down. So it's this interesting balance of I really want to have less complexity, but I have to acknowledge that it's there. I have to acknowledge that some of the things I'm doing to reduce complexity might actually add complexity and find that that balance.

I think we've managed to introduce all sorts of other complexity in the process of doing all of this too. But I I I do agree what you're describing there, this uh yeah, the the old adage about there's no such thing as cloud as uh it's just somebody else's computer. And that remembering that we we still have all of the things that can possibly fail. In fact, in some ways, it's even more complex under the covers now and uh how all that's working.

The other thing that we've seen is that the idea that you're just gonna eliminate complexity is really wrong. You can eliminate complexity by eliminating innovation. And sometimes that the thing that you're really looking to do is say, how do we keep innovating? How do we keep adding to the systems we have and changing them, which is sort of it allowing complexity in at the same time that we don't do it needlessly or do it in a way that's not controlled. And I think that's what people get really confused about here is you can control and manage complexity. That's what that's a fun conversation to have an important one, but you can't eliminate it. Eliminating complexity is actually harming your infrastructures. And and if everybody does that, it actually makes things much more complex. It's ironic.

Is it catch 22, right? It goes in one direction and then just comes back behind you without you really knowing about it. Can I um I mean it feels to me like we should dig into that a little bit uh more deeply because how one of the things around complexity is people uh have a an un a misunderstanding perhaps or a perception that complexity is impossible to work within. It's something that you want to get away from as fast as you can. And yet what you're describing is a world where complexity is something that is managed, is something that you understand you're in a complex environment, and there are ways that you can constrain it, that you can work within that, which are at least somewhat predictable. Can you talk a little bit more about that?

Yeah, this is one of those the the funny thing about trying to eliminate complexity is that it can drive, and this is especially true in enterprises with a lot of teams. One team's desire to eliminate complexity often shows up as their desire to not collaborate or their desire to not use other teams' tools or work with something where they have to, they have external dependencies. And so you can eliminate complexity sort of by drawing very tall boundaries around what you do, effectively siloing your work from that perspective and just ignoring the rest of the universe around you, which isn't actually eliminating the complexity, it's just ignoring the complexity.

And so when sorry? And and that does sorry, Rob, to interrupt, but just on that point, does that not lead to fragility in the system? Because I think what you I mean, on the one hand, if we can live with those silos, if the complexity kind of disappears and we're okay with it. But I think what we're actually doing is building a system that is less robust, less resilient, more prone to failure by doing it.

And the key word is system. The system becomes much more fragile when you do that. And and a lot of these things are trade-offs. Like one of the things that we see people do in enterprise data center space is they're like, all right, I'm gonna eliminate the complexity of server management by only buying from one vendor, right? And that that might actually make it so that they can deal with some things more easily, but now they have supply chain vulnerabilities, or they have vulnerabilities if the vendor decides to change things that they've relied on. And so this is the irony with the complexity piece is that some of the things people do to simplify things in the short term do have the systems challenges that you're describing, but they also can create more challenges later, right? Somebody changing an API or a vendor not being available can create huge disruptions in your systems. And so this is the managing part of complexity is leaning into the fact that things are complex and not not just holding your hands over your ears and pretending, you know, just shouting la la la as fast as much as you can. You actually have to say, all right, this is complex, there's reasons for it to be complex. How do I decouple the impact of a change with this from something else? Which decoupling actually adds complexity to the system if you're doing decoupling well and if you're testing the decoupling and you're you're building all the buffers and things like that that you want, but it does ultimately make the system itself more resilient.

I I mean I agree. It's uh it's interesting as well when you um uh another uh example of this is perhaps when uh you see this desire to have a single way in which things are done across the whole enterprise. And uh we have this, yeah, I and I love this this problem of because it's this swing, right? If I want to be totally prescriptive, there's only one pipeline, only one way to do things, only one way. But the problem is that if if I have that, it's never gonna be one size fits all. And uh the other end of that scale though is complete anarchy, where anybody can use anything and we can go pick and grab every SAS tool off the market and glue them all together and do whatever we want, but then that leads to reinventing the world every time I try to do anything. So you really want something in between these two, and you're ideally you're looking for as well, it's going to be context bound to the domain that you're working within to make sure that it's applicable to the particular problem that this particular team, this particular area, this particular space is using. So that they're that it's properly targeted and uh that you have enough um capability to still innovate within that space.

Yeah, you're you're describing a problem that I is is incredibly real. Um, you know, back I I think what I used, what I describe as IT of the 90s, which was the department of no. You know, they were all about control and you know, have to do it this way, and it was incredibly hard to get anything through. And the vendors just jumped into that and created, you know, basically back doors so that IT people, you know, developers and and people doing work could just bypass IT and do exactly what you did. And the the challenge with the bypass of IT and them always saying no to things meant that they would run in, you know, people just run in and say, Oh, I'll do it whichever way I want. That the chaos model of doing it, I there's a lot of times when I see that we've gotten so used to that selling pattern and it's so much easier to sell to a department, right? Then, you know, we we sort of bypass the idea that we need collaboration between the teams. You know, the IT department, uh, what we're seeing emerging as platform operations or platform engineering teams are are coming back and saying, okay, we want to help you pull back from this idea that you have to run the entire stack yourself, right? It works for you, but it it doesn't translate into good security practices or governance, or you know, in some cases, it means that a team is maintaining things that they actually don't, they shouldn't be maintaining. It's not a good investment of their time to for a development team to maintain infrastructure, right? You should be able to collaborate on that. The the challenge has been, and this is where the complexity discussion comes in, in order for that to work, those teams have to be able to collaborate. They have to spend the time and invest the time to collaborate with another team to get either create reuse across teams or to delegate work to another team. And that that's where things sort of we have to refocus on what we think of as eliminating complexity. Right? Eliminating complexity might actually be spending more time enabling another team upstream or downstream from you. You use the word pipeline, which I like a lot, to pipeline work together so that they're collaborating now. And then for that team, they're actually giving up some complexity. They're they're using another team's resources to do that. And I think, and we've seen this incredibly productive inside of organizations, because then that team that you've delegated to can do the same work for three or four other teams, and that's a great way to encourage reuse.

So as you're describing that, Rob, I'm just thinking about the kind of conflict between chasing efficiency over chasing effectiveness. And I think much of what you're describing in terms of the challenges that you know when you try and end up with a single vendor or you you you focus on this team being responsible for everything around uh a particular part of the pipeline or the the infrastructure or or whatever the responsibility might be, there's often that chase for efficiency which creates more instability and fragility and complexity instead of a chase for effectiveness, which starts with collaboration and communication and understanding the shared goals that we're trying to achieve and then focusing on how to be effective in that area.

Defining those outcomes you're looking for first, starting from that versus just diving right in and saying, well, the best thing to do here is we can save $100,000 by consolidating all of this into one place. And then everyone's stepping on everyone's toes and the whole lot falls apart.

Oh, there's there's so many pieces in this. Um one of them is that that, you know, in order for the team that you're consolidating to to be effective, they can't become the myway or the highway team. So they have to recognize that to support other teams, they're gonna have to offer flexibility, right? They're gonna have to have, and this is the challenge. You're you're right about this false efficiency or false uh insufficiency optimization where the team says, Well, I'm managing this system now. I want to only deal with my one thing. And so they start to say no or or streamline it. The way it has to work is for them to service other teams, is for them to turn around and actually make things more efficient, uh less efficient for themselves, right? Accommodate the variation and the other teams need from them. The other thing I would say is you're reminding me of Goldwrap, right? And the goal and theory of constraints. And the big lesson from theory of constraints is that you don't get to optimize every function in your delivery pipeline. And that's what, right, we all try to make ourselves as efficient as possible, not as effective as possible. And when we strive for that, that might mean that we don't want to take the time to help another team do their job, right? We need Slack in the system and figure out where the constraint is, um, not assume that we have to run at 100% efficiency. So everybody else distracting us from that is actually bad. But it's really hard to get teams to think like that. That's a leadership function.

I think it, I mean, it's it's it's almost um we're in that space right now where organizations are trying to get more and more and more from you know the same resources or less, effectively. And so that drives, I mean, Slack has been pushed out of the system for decades now. There's really nothing in the system that really gives a team that space to be able to think just not just about uh their work and how to keep their work uh maintainable and sustainable and the right standards in terms of what they're doing, but also how to take the time to listen to other uh parts of the organization, people they're serving to understand what they need. I mean, that that just doesn't seem to be that. Do you do you have um stories that you can share just about how organizations have created that Slack, that space, or shifted from efficiency to an effectiveness mindset?

Yeah, this is one of those places where a lot of work we're seeing with infrastructure as code and and thinking about infrastructure as code has been a really important piece to this, but it requires a little bit of a step forward. I'll I'll go, you know, so one of our customers is a big financial institution, brought us in to help them bootstrap their data centers. They had a regulatory requirement that they had to bring up data centers um in under a week. They they gave us an hour. I don't know what they did with the Slack, but they um and they said, we need you to be able to bootstrap a data center and get everything running. And they they just needed us to do you know the long pole for that. But what happened was is that after we got that one piece done, they they looked at the ability to start chaining other pieces of work together. So they use that efficiency to sort of say, all right, how do we we manage this process? Um and for us, and this is where it gets really interesting, the way we use infrastructure as code means that we define everything that goes into a data center as immutable artifacts, code, and deliver that to the to the site, right? Or the customer. It's all customer managed stuff, so the customer manages the site. But what they've been able to do is they've expanded that footprint of what they automate piece by piece by piece. So after they got the first server pieces working, they brought in the storage team and said, all right, storage team, if we were able to bring you in on this, can can we move that process? And that's they've they've been able to establish that automation and that level of automation as the gate to bring up a new data center. But they didn't do it on day one. Day one was just like, hey, can I can I fix this one thing and get this one long pole prior to the process moving faster? And then once you have that proof point, and this is to me exactly how CI CD of pipelines came about, it's how we're seeing infrastructure pipelines being adopted, is that as you get one or two pieces working together, you can then add in the next piece, right? So you know we're seeing that at play out at multiple customers where they get the compute pieces under under uh some type of infrastructure automation controls, and then they ask, can we add in the switching fabric into that and start connecting the switching fabric? The the thing that makes it work well is for us, we design with a developer mindset first. So it has to be collaborative, it has to encourage reuse. And so the the to manage complexity, you have to have the the platforms, and there's a tech side to this, right? We were we were talking before we started, we turned on record about the balance between people side and tech side. The tech has to be set up to encourage behaviors that you want your teams to have. It has to reward that type of configuration process. So the fact that when people build a site, it's much easier to actually do it using a code process and then upload all the pieces so that the site's all intact. And it's much easier to do a dev and a staging before a prod process, thank God, then they actually the those behaviors are encouraged in the team members and they can get reinforced. But if you're replacing a process where people are like, I just have to get it done, and it's really easy to go in and fiddle with something, but nobody remembers that they did that a week later, and systems all end up different, then you end up that that becomes a problem. And so some of what we see reducing complexity is having the technologies reinforce good behaviors without the people even realizing that that's what it's doing.

It's the human nature aspect, right? Is we're we're not fungible widgets that you can move around and put the cables in and boom, we do the same thing over and over again. Working working with human nature to help us, like it's it's the you know, the the the path of least resistance and understanding what that is, so that we're building tools and practices and and and approaches around that that rely on that rather than fight against it. I think is huge.

Yeah, you we you do also have the challenge of people, you know, it's funny because as much as we hate repeating the same task over and over again, we get incredibly threatened when our our job is changed. Um and so you know, there is always the struggle of you know, as much as you want to help somebody reduce toil, sometimes taking away toil for them feels like a threatening change.

Um you're changing my livelihood. This is what I've done for 25 years. You're making me do it differently. I don't want to do that. It's uh that's fine.

And and you're you're you're making it more of a you know, now I have to drive the systems through Waldos. Um, and you know, some people get excited about that change and some people don't. The the reality is nobody has the time anymore. So it really is toil. And if you're making it possible to reset a system, right, this is where it's a it's a breakthrough for us with people because they're used to spending it, you know, a week building systems up. And if they can hit a button and watch the system get rebuilt, take a coffee break and and you know, check that everything worked great. Even if it doesn't work great, you know, it takes some, even if it takes them 10 iterations to get the first one right, it's happening fast enough that they feel like it's it's going. Um so that they have to be able to see it happening. There's transparency, there's all sorts of things. And we didn't talk about this with complexity management, but one of the things that changes complexity stories is transparency.

Yeah, being being able to evaluate where you are and see what's going on without you know, without sort of having the the the blockages that stop you seeing what the consequences really are of those changes. Yeah, that's so true.

Yeah, and trying to introduce that change without that can be very difficult because you're w one of the problems you run into is that uh the person who's used to doing it this way and going in and tinkering with it may not realize some of the knock-on effects further down the stairs. He may not be feeling the pain that his going in and tinkering is causing, which means that he's now in his little world, you really are just threatening what he does. I've done it this way and it's always worked. There's no problem with this. This is fine. I've got it, I'm up.

This is this is the power of pipelines to me, is right, that and we've built all these tools. We have some great, you know, in the infrastructure space like Ansible and Terraform and these individual tools that work in their silos, but they don't really they're they haven't been designed to connect that that end-to-end experience together, which means exactly what you get is somebody's like, okay, I know how to fix this Terraform plan, so it does this thing I need to do, without any, you know, it's it's some of it's knowledge, but some of it's the just ability to test and run that end-to-end system. And so the more that we can you know make make people's work part of that end-to-end chain, then it becomes much easier to figure out if something broke. I mean, that's what CI CD pipelines did for us. I can go from a development change to, you know, unfortunately, some, you know, hopefully not testing in production, but but you know, this a small change rippling all the way through really quickly means that I know that I had an impact and I can see that. Um, it's so much more powerful. You know, getting that into infrastructure and automation is equally powerful. Then I can make a small change in something and not, you know, hope that it works and find out six months later that it didn't, but find out right away by exercising the system.

So if we were to take all of this, and I always like to do this at this point in the conversation, if we would take all the things that we've talked about and we were to sum this up into like three main points, what what three points would you go for?

So boy, I love questions like this. Um the first one is that complexity is normal, right? I would I would shake somebody and say, stop trying to eliminate complexity, figure out how to manage it. Because it's that it's the wrong attitude, right? And and I think that that actually does harm for us when we when we think we're eliminating complexity instead of protecting ourselves or helping manage it. The the other one I would say is you know, we need to encourage systems to then work in ways that couple together. You know, that that's that's the other one. Is we have to be building systems in especially in tech, but I would say this is. True, even in human systems and non-tech systems, in which we're not just focused on the point solution, the efficiency, but the system effectiveness, right? And then the final the final thing, and the most important, is these complexity problems are really collaboration problems. That the human side of being able to collaborate with other teams and spend the time, because collaboration takes time and effort and compromise. It's a skill. And the amount of you know taking the time to build collaborative systems and have a collaborative approach to this is actually gonna do more to eliminate complexity than any other thing I can think of. And I've been talking to a lot of people about how do I deal with all this complexity. And a lot of times the conversation is simply collaborate better with the people ahead and behind you in the value stream, or collaborate more with teams who are adjacent to you so that you can you can collapse some of those value stream pieces. And it's it it seems initially it seems counterintuitive how important that collaboration piece is, but the more we dig into improving collaboration, the more results we see in less complex systems. It's it's really a remarkable intersection.

There's a very fun uh workshop that we run, uh, which is on collaboration. It's a painting workshop where we come in and we get lots of paints and everything, you get the whole team around and you put the big canvas in, and you start by having everybody has the paint and then they and tell them to draw in their corner, and everybody starts drawing in their corner, and you say, Well, this is cooperation, you're cooperating to draw the picture. Now, would you hang this on the wall? And they all go, No. And you say, Okay, now you can turn out now you can start to draw on other people's piece, and and when they all start to come together and collaborate, and at the end of all that, they invariably say, when you ask them the same question, would you hang this on the wall? They always say, Oh, hell yeah, and it goes up in the office. Where so this the difference between like that just that cooperation and collaboration is uh is a very key point, I think, uh, calling out there.

There's a a party games website I use called outofcontext.party that's all all the games are designed as collaborative games, like you're writing stories together, you're making recipes together, you're you're telling, you're drawing together. Um, and it it's amazing how much fun it can be when the group is working together. It also is remarkable that it only takes one person who doesn't want to collaborate to turn every drawing into uh you know eschatological uh constructs. And so it's it's a you know it really is a powerful uh group lesson to watch you know amazing art come out of a team dynamic or one person who doesn't participate, you know, making it not work.

Would you like to add anything at the end here, Dave?

No, I I think it's as always uh a great conversation. Um I'm used to having great conversations with Peter. It's great to have uh Rob you along uh to kind of inject a little bit of a different perspective from your your world as well. So thank you very much for that and uh looking forward to the next time we get a chance to chat.

Yes, thank you very much, Rob.

My pleasure. This has been a really fun conversation. I think we really actually got somewhere in dealing with complexity. So thank you for the wonderful time.

So so thank you everyone to all our listeners as well, and uh yeah, tell all your friends and uh subscribe and uh look forward to seeing you here next time. Thanks. You've been listening to Definitely Maybe Agile, the podcast where your hosts Peter Maddison and Dave Sharrock focus on the art and science of digital, agile, and devots at scale.