AI in the SDLC Is Only as Good as Its Context

Welcome and a Weather Oddity (0:04)

Peter Maddison (0:04): Welcome to Definitely Maybe Agile, the podcast where Peter Maddison and Dave Sharrock discuss the complexities of adopting new ways of working at scale.

Dave Sharrock (0:13): Hello and welcome, Peter. How are we doing?

Peter Maddison (0:16): I’m doing wonderful, Dave. How are you?

Dave Sharrock (0:18): Excellent, excellent. It’s a weird situation. I don’t know what it’s like where you are, but here in Vancouver the rain has decided just not to turn up. It’s October and it’s still 18 degrees outside. We’re all staring out the window wondering what on earth is going on. It’s very unusual for Vancouver. Let’s not dwell on it, but it probably means the climate isn’t behaving the way we expect it to.

Why AI Puts the SDLC Back at the Center (0:51)

Dave Sharrock (0:51): So what would you like to talk about today? We’ve had a lot of conversations in the background about SDLCs, software delivery lifecycles, and the impact AI is having on development in particular, but also on the entire software delivery lifecycle. Anthropic recently published a paper on what they see in that space, and I know you’ve had some conversations with people in that arena. So maybe let’s explore that.

Peter Maddison (1:23): Sure, sure.

Dave Sharrock (1:24): How long have you got? Okay, let’s explore it over about a 15 to 20 minute time frame. How’s that?

Peter Maddison (1:32): Okay, I’ll be succinct. So yes, Anthropic published a paper. It’s one of a series of activities going on in the SDLC space. A lot of what you’re seeing is that nobody really knows what the future SDLC might look like at the moment. A lot of people have opinions. To give you an idea, think of the SDLC as running all the way from ideation. I have an idea, I turn that idea into something I’ve designed, which I then code and build, test, and then deploy and release. A standard SDLC. Everybody who has done any software development understands that this isn’t a linear process. It’s very much iterative and incremental, and these stages flow back and forth as we operate, learn, integrate new ideas, and think about how the system needs to evolve as we discover more about how it’s operating.

Intent, Agents, and Many Variants (2:48)

Peter Maddison (2:48): What Anthropic was talking about in that paper were some conceptual ideas, and there are a few really good ones. One is that with AI in the mix, we care more about intent. How do we collect, capture, and curate the intent and the context we’re going to build under? How do we then use agents to build not just one version of something, but multiple versions, and pick the ones we like? The hundred monkeys in a room, somebody creates Shakespeare type of approach. And then at the other end, the use of agents for operational behaviors: deployment, release management, and the ongoing telemetry and operations of the system. At a very high level, the paper described all of these pieces end to end.

Dave Sharrock (3:39): We’ve actually had a lot of conversations around how the SDLC has come back into vogue. Software delivery lifecycles were one of those things you defined and locked in many years ago. Then there was a period where there was a loose skeleton, but we turned all our attention to iterative and incremental delivery, DevOps, and so on. The SDLC was still the backbone, but it wasn’t talked about much. Now it feels like it’s back. It’s really a focus of attention again, and a lot of that is driven by AI and the ability to plug and play. You take out a chunk and say, hold on, AI is going to do implementation, or AI is going to do the solution design, or it’s going to produce the code. That’s where things have settled, but there’s a lot more to it. When you and I talk about this, I take it as a given that once we’ve decided what feature or product to build, AI will have a huge influence on the development, the launch, the monitoring, and the operation of it. But I spend most of my time in the product space, and I see lots of room there for AI to accelerate that piece too. Every element of the software delivery lifecycle has an accelerant brought to it by AI in some way.

Where the SDLC Model Breaks (5:17)

Peter Maddison (5:17): Yeah, and this is where it starts to get really interesting. You hear all the talk about coding being dead, that we don’t need programmers anymore. That totally misses the point of what developers actually did, which was figure out how to solve really complex problems. But I don’t want to go down that rabbit hole. The interesting part is that when you talk to the authors of that paper, it’s clear they had basically vibe-coded something they thought might be valuable to people. When you start to poke at it, there are some very big holes. What they describe in that SDLC process doesn’t scale very well. It doesn’t take into account where the context is going to be stored, how that context is versioned, what happens when it ages out, and how we properly manage all of that. In a large, complex organization, I have many different types of systems, which need different behaviors and different ways of managing them. So there are other systems and mechanisms we might need in different places, depending on the problem we’re trying to solve. It’s not one size fits all. It becomes a question of what key pieces we need to see to know something was done in the right way.

Dave Sharrock (6:42): What I’m finding interesting, talking to clients, friends, and colleagues at various stages of exploring AI, either replacing chunks of the SDLC or applying AI within software delivery, is that they seem to take two approaches. A number of people I’m talking to are building AI-native solutions in parallel to their existing systems. The existing systems keep doing what they do, with the developers, the ways of working, the governance, and the telemetry already in place. They leave that as is, while exploring how to really rethink the not-so-linear software delivery lifecycle we’re describing. That one is pretty interesting, because suddenly there are big slices that really can be operationally autonomous and AI driven. I’m thinking of Andre Kaminsky. We had a great conversation with him, and he has a book on the AI-native SDLC. We’ve had a number of guests on talking about what happens when you start applying AI to that process.

Peter Maddison (8:07): This is where your mileage may vary. Part of it is that things are evolving so quickly that if we set up a process and say this is the way it should be done, it’s going to be wrong a month or two from now.

Dave Sharrock (8:24): Not even a month or two. It’s changing fast.

Peter Maddison (8:27): It’s changing fast. The evolution of the models and their capabilities is great, and it does allow them to do much more. But from everything I’ve seen so far, you still need an orchestration framework around them, the harness that manages them. And if you want to go at speed, there are certain pieces I think you really need to have in place.

Dave Sharrock (8:54): This is the interesting piece for me. Whether you’re doing greenfield development or taking an existing SDLC and starting to plug and play, testing AI development on certain systems or automating whole chunks, there are things that carry over in both. The obvious place I start looking is governance, context, and intent. We need to know what the system is trying to do and the bounds within which it operates. How are those managed, controlled, and measured, so that whatever changes are made, you’re confident the behavior isn’t drifting away from where it needs to be? That’s a lot of words to say governance.

Peter Maddison (9:47): You’re touching on something I want to clarify. There’s governance of the system itself. Did the system produce the output you expected?

Dave Sharrock: Exactly.

Peter Maddison: But there’s another kind of governance that matters here, which is the evaluation of the agentic system itself. This matters because the models change and get more capable. So I need to know whether my agentic system, my harness, the piece wrapped around all this, still behaves correctly. And I’m not just talking about the Claude Code harness, but the end-to-end system. If I move from one model to the next version, how do I evaluate whether the harness still behaves, whether it needs to change, or whether something needs to be modified to take advantage of the new capabilities?

Context, Inference, and Complexity Risk (10:40)

Dave Sharrock (10:40): I’m struck by this. Think about the conversations we had on this podcast at the beginning of the year about context and context engineering, with context being this primary element we have to be aware of, create, and capture. Then, midway through the year, the frontier models started testing the removal of that context, showing their newer models could still infer it and still deliver what was needed with considerably less documented context.

Peter Maddison (11:14): We have to be careful here, because they love to throw things like that out into the market.

Dave Sharrock: I know, I know.

Peter Maddison: When they say take out the context, it means we theoretically don’t have to provide it explicitly. This is why you need evaluation of the harness. Although “harness” is being overused here, because there’s the harness that sits around the model and there’s the framework that exists within the orchestrator. Orchestrator is probably a better word. The evaluation of that whole system becomes critical. And when we talk about removing context, we’re not talking about removing the purpose, the business logic, what the system needs to do, or how you want it structured. That context is still required, because the agent can’t work that out on its own. We’re talking about the context it needs for implementation, once it has the purpose, without you providing more information. The agent can’t magically know that it’s building for a very small company and doesn’t need the layers of abstraction an enterprise would need. If you don’t give it that context, it will happily keep layering in abstraction ad nauseam, and you end up with a horribly complex system.

Dave Sharrock (12:37): Yeah.

Peter Maddison (12:38): And if you remove all of the context, there’s no way it’s going to know that.

Dave Sharrock (12:45): What becomes really interesting, or rather a problem that needs to be managed, is that we’ve talked about this as a learning system. It’s not something you build once and say, boom, this is how we work from now on. We’re used to upgrading when new versions come out, like a customer relationship management tool in customer service that gets updated regularly as the underlying SaaS product changes. But this is change on a different level. It’s about context, and about how rapidly more capability is being built into the models used for generation, management, and oversight of the system.

Peter Maddison (13:42): There are systems and products on the market that handle the lifecycle of the context, of the knowledge, which is the key piece missing there. We’ve always been very bad at knowledge management. It’s the age-old “developers hate writing documentation” problem. There’s the argument that the code is the documentation, but not all of it. You can’t derive everything from the code. Anything of any complexity is either one massive monorepo or spread across multiple repos, and either way it becomes very difficult to derive how the system will behave just from the code. So you need something at some level of abstraction. You could think of it as technical architecture, solutions architecture, domain architecture. There are lots of slices and views we need in order to make judgment calls about where the system should go. We can let the LLM make certain judgment calls for us, but this comes back to evidence. Which judgment calls do we allow the LLM to make, and which ones do we want to make ourselves?

Dave Sharrock (14:52): I’m not sure there’s even a point where those choices are actively being made. At some point in the future, those decisions will effectively be made by the LLM. So it’s more a question of how you validate that those decisions aren’t drifting away from what we consider good, or acceptable.

Peter Maddison (15:12): And this is the concept of architectural drift. Architecture now becomes critical to the system. Before, it was, “Oh, do I have to go to the ARB?” Now you have to understand the architecture of your system, and not only that, you need to inform your agents of what that architecture is so you can tell if it has gone off in the wrong direction. So there’s a real shift in how critical some of these pieces are.

Practical Takeaways and Open Questions (15:38)

Dave Sharrock (15:38): I’m wondering if we’re getting to a point here. As you said, how long is a piece of string? We could talk about this topic for hours.

Peter Maddison: I do.

Dave Sharrock: It’s an important topic and you’re seeing it around you all the time. But is there a conclusion you can reach at this point? Two or three things to watch for?

Peter Maddison (16:00): The first thing I’d suggest is that if you try to implement what Anthropic published as their SDLC, be warned that it’s incomplete as a model. It’s not a direction. It’s more “this is what it might look like,” so think it through. There are some big holes you could drive a truck through. I’ve spoken to the authors, and they couldn’t give us answers. It’s not how Anthropic builds things, it’s how they think others might build things, and it has some big holes when it comes to regulated organizations. That’s one thing for sure. The second thing, and we didn’t talk about this, is the other paper they published last week, where their engineering team talked about...

Dave Sharrock (16:53): How they effectively replaced GitHub, I guess.

Peter Maddison (16:56): Yes. That paper, while also incomplete, actually has far better information about how their internal SDLC works than the SDLC paper does. It has some very interesting ideas and talks a lot about the curation of data. The piece I found interesting around context was the way they slice it. Rather than giving the agent everything it could possibly know, they’re very exact about which slice of information it gets.

Dave Sharrock (17:31): And they’re specifically solving the problem you get as you commoditize software development. The consequence is thousands, then tens of thousands, of commits and changes to the system, which means you can’t work in any linear “hold, make a change, hold, make a change” way. I found it really interesting how they’re looking at modifying things on the fly and what the consequences were, because there isn’t really a central single repository of the system anymore. It’s much more fluid.

Peter Maddison (18:09): Interestingly enough, what they actually did there was centralize the system. They moved from a distributed system back to a centralized one because they had to. That’s what enabled it, which was interesting for sure.

Dave Sharrock (18:22): We’re losing track. You said one thing was...

Peter Maddison (18:26): I know, I had my two points.

Dave Sharrock (18:28): I’ll let you have your last one if you don’t get to that one. One thing I find really intriguing is that we’re having to build dynamic systems that continually evaluate how they’re performing and whether they’re going to change. The models add an element of change into the software delivery lifecycle in such a way that we can’t just build it and sit on it for three or four years. We have to build something that is monitored, or aware of how the models might change what’s going on, so we get flagged when those changes start affecting the system.

Peter Maddison (19:18): Yes. When you have models inside systems, you have to ask whether you’re getting the output you expect. Evaluating the performance of a model is a very different thing, because it’s non-deterministic. You don’t evaluate it the same way you run a set of tests against a system. That’s a whole other podcast. The other piece is evaluating the orchestrator that coordinates all those agents across the system as the models, the non-deterministic components of that orchestrator, change.

Dave Sharrock (19:54): Yes. As they become more capable, they have the option to take on more ownership or responsibility. They’ll make decisions that weren’t in their purview before and suddenly seem to be.

Peter Maddison (20:08): I would say, as those models start to behave differently than they did before.

Dave Sharrock (20:14): Very diplomatic there. Tactful.

Closing (20:18)

Peter Maddison (20:18): Awesome. Fantastic chatting, as always, Dave. Time just flew by in that conversation.

Dave Sharrock (20:24): That was a pleasure. I always learn so much when you start talking about what you’re seeing in the SDLC space, so I appreciate that. Until next time.

Peter Maddison (20:34): You’ve been listening to Definitely Maybe Agile, the podcast where your hosts Peter Maddison and Dave Sharrock focus on the art and science of digital, agile, and DevOps at scale.