Conference season is back, and so are the real conversations. In this episode, Peter Maddison and Dave Sharrock catch up after a busy stretch of travel and dig into something Dave has been road-testing at conferences: why most people given access to AI tools freeze up, and what actually helps them move past that.
Dave ran a workshop at the Global Scrum Gathering in Vancouver for non-technical roles - product managers, Scrum Masters, agile coaches - people who've been told "use AI" but have no clear picture of where to start. What he found is that the problem isn't motivation or technical ability. It's the lack of scaffolding. Give people the right structure and the right room to experiment, and things shift pretty quickly.
The conversation then moves into multi-agent systems - how Dave's team built a group of agents that continuously refresh the workshop itself based on current thinking. Peter adds his own take on testing these systems with personas and automated quality evaluation. It gets a bit technical, but in the best way.
This is a good episode if you're thinking about how to help your organization actually use AI, not just adopt it on paper.
Key Takeaways:
- Context beats generic. Prompts work when they're specific to your role and your actual problems. A product manager needs product management context, not a one-size-fits-all example.
- Think in teams, not steps. Multi-agent systems work best when you treat them like a team reviewing an artifact, each agent checking for something different, rather than a linear build process.
- Don't assume everyone gets it. The gap between people who use AI daily and people who tried it once and gave up is wider than most of us realize. Getting both groups in the same room is where the real learning happens, for everyone.
Have a question or something to add? Reach out at feedback@definitelymaybeagile.com or find us at definitelymaybeagile.com. And if you're finding the show useful, subscribing and leaving a review goes a long way.
New episodes released every Thursday to challenge your thinking and inspire action.
Listen and subscribe:
Welcome And Conference Catch-Up
Peter [0:04]: Welcome to Definitely Maybe Agile, the podcast where Peter Maddison and Dave Sharrock discuss the complexities of adopting new ways of working at scale. Hello, Dave. How are you today?
Dave [0:14]: Peter, fantastic to catch up with you. It feels like it's been a while since we've really had a chance to have a one-on-one discussion here.
Peter [0:22]: It does. You've had a lot of travel, I've had a lot of travel. That's the way it goes.
Dave [0:28]: Conference season does that to us.
Peter [0:29]: It does. Interesting conversations though, meeting new people, all sorts of fun stuff. Speaking of which, as I understand it, you've been running some workshops at conferences recently.
Dave [0:42]: I've been to three conferences in the last few weeks. One that really stood out was the Global Scrum Gathering here in Vancouver. Very easy for me - I was able to walk out my front door and wander down to the conference. I ran a workshop on how to use AI when you're in a non-technical role: product manager, Scrum Master, agile coach, that kind of thing. Specifically, how people in those roles are starting to change what they actually do day to day.
Peter [1:20]: That's interesting. I'm seeing it with my clients too. Everybody now has access to tools that let them create and generate things. In some cases people are generating code, which raises its own questions. But the problems you were working through in the workshop sound really worth unpacking. Tell us a bit more.
Why The AI Workshop Exists
Dave [1:43]: The starting point was a situation a lot of organizations are in right now. They've told their people: you can use AI. Copilot is available, this tool is available, that tool is available. Go ahead and change what you're doing. What we see in many cases is that for the individual, it becomes a freezing point. They're caught in the headlights. They don't know where to go. So the goal of the workshop was to gently guide people toward using prompts in a way that actually works for their specific role. We'd introduce an idea, they'd try it on their phones right there in the room, then we'd add a bit more structure, they'd try again, and they could see how each iteration produced a noticeably better result.
Peter [2:49]: Coming from the technical side, we sometimes take that kind of thinking for granted. We're used to fast feedback loops. We've been doing it for years in agile contexts.
Dave [3:01]: Right. And the people in the room - Scrum Masters, product owners, agile coaches - they absolutely understand iterative feedback and learning as you go. That's not the gap. The gap is that they don't have a clear picture of what they can actually change about their prompts. They get an okay result and stop there. So part of what we were doing was giving them just enough knowledge to take the next step. You could almost see the lightbulbs going on as they realized that adding a bit of context, or describing what the output should look like, suddenly tightened up what they were getting back. It was a real shift for a lot of them.
Prompt Structure That Improves Results
Dave [4:10]: The iteration is happening, but people aren't always sure what to change on the next pass. One thing that got a big reaction was something pretty simple: going back to the LLM and saying, "Help me improve this prompt." That sounds obvious to anyone who's been doing this for a while. But for a lot of people in that room, it was a genuine revelation.
Peter [4:57]: There's something important in that. You're getting more people actually using these tools to create real value. I've built quite a few of these systems now - experimenting with things like LangGraph, chaining agents together, seeing what different development toolkits can produce. But what you're describing is helping less technical people understand how to take an idea and iterate on it. How to help the system learn what you actually need. That matters a lot, because I also run into plenty of people who tried AI once and walked away thinking it was all hype. "I got garbage out of it, I don't know what everyone's going on about."
Dave [6:16]: And in a lot of cases, they're already overloaded. They've got a backlog of work, they're being told AI will make them more productive, but they have blinkers on because of the stress. That removes your creativity and your willingness to experiment. Which brings it back to that scaffolding of learning. How do you create an environment where people can see just a little further down the road, build on that, and then see a bit further again?
Beating AI Overwhelm With Scaffolding
Dave [6:45]: Walking around the groups during the exercises was fascinating. Some people in the room already knew all of this. But there were others who you could genuinely see thinking, "This is going to help me," and not just at work - personally too, because we all use these tools in our daily lives. The interesting thing was watching them realize just how much further the door could be pushed open.
Peter [7:52]: There's a real nugget in what you're describing. When we have these conversations as humans, we create learning moments. That's not a new idea. But if we're trying to build organizations that can actually develop and grow with these tools, having people who don't understand something yet is actually valuable. They try to solve the problem in a way that nobody else thought of.
Dave [8:24]: Exactly. And they ask the most unexpected questions because they don't have the context to filter themselves. There was a lot of that energy in the room. And then toward the end, we introduced something that generated a really interesting conversation. The concept of agents talking to other agents. Orchestration. For those of us who've been doing this a while, nesting agents and feeding output from one into the next is pretty standard thinking. But for a lot of people in that room, it opened a completely new mental model.
Orchestrating Agents To Build Work
Dave [9:05]: The way we wrapped up the workshop was by explaining how the workshop itself was built. We've been running this for well over a year now, and the thinking around how to prompt LLMs keeps changing. So we actually created a group of agents that rebuild the workshop by looking at current thinking and evaluating whether the content is still relevant. It's a linear sequence of agents basically refreshing the material based on what's current.
Peter [9:44]: I love that. I've used similar approaches for a variety of things. And as I was suggesting to you earlier, the next challenge is to have it run on autopilot, generate the workshop, and then walk in blind and see what comes out.
Dave [10:01]: You like putting me on the spot! But that question of trust is real. Do I really trust it enough to walk into a major workshop with something my agents created, sight unseen? Probably not yet. But we're learning as we go.
Peter [10:28]: Fair enough. At a conference presentation, the stakes are a bit lower than, say, an executive committee meeting.
Dave [10:41]: True. And one of the interesting things we're thinking about is how we evaluate quality in these multi-agent systems. It's not just: what are the steps to build a deck? It's: who are the different roles involved in reviewing this thing, and what are they actually checking for? Target audience, workshop intent, design, speaking notes. How do we make sure all of that is being validated against real expectations?
Peter [11:35]: There are two things I've built into every system I've worked on recently. One is a dummy LLM - something that lets me functionally test the system without burning through tokens every time I make a change. The other is a test harness with an LLM built in that validates the logic. At some point in every one of these systems, the only realistic way to move forward at speed is to build something that does the non-deterministic testing for you. I create a persona of the customer, have it interact with the system, and then evaluate the outcome. Is it on topic? Does it fit within the constraints? That judgment is happening automatically, and you can even parallelize it across different personas.
Quality Control With Tests And Personas
Dave [13:00]: What I really like about that is the shift it creates. You move away from a linear optimization model - what are the steps to create this thing - toward something more like a team-based approach. What are the different roles that need to touch this artifact, and what are each of them actually validating? It stops being a checkout flow where you just go from A to B to done. It becomes much richer than that.
Peter [13:39]: Exactly. So if we're going to wrap up for our audience, how would you summarize it? What are the three things you'd want people to take away?
Three Takeaways And How To Reach Us
Dave [13:55]: I'll give you two and you can add a third. First: context matters enormously when you're prompting. The tendency is to work with generic examples and generic prompts. But if you're a product manager, you have specific product management needs. Generic exercises don't help you get there. Contextually relevant examples do. That was one of the clearest things we saw in the room.
Second: the idea of agents as a team, and rethinking how we build things as a result. Creating something and then using persona-driven perspectives to evaluate it - that quality control piece - really changes how you think about what you're doing. It stops being "build this using AI" and becomes "how do we manage this process in a way that actually produces something good?"
Peter [15:23]: For the third one, I'd say: don't assume everyone gets this. If you're close to this topic, you're probably surrounded by people who live and breathe it. But a lot of people out there don't know how to apply these tools to their specific problems. That gap is much larger than we tend to assume. Getting people who understand this and people who don't into the same room - and helping them think through it together - is where a lot of the real learning happens. On both sides, honestly.
Alright, I'll wrap it up there. Thank you as always, Dave. Anyone who wants to reach out can find us at feedback@definitelymaybeagile.com. We have some great guests coming up over the next few weeks. Don't forget to subscribe. Until next time.
Dave [16:26]: Until next time. Thanks for the chat.
Peter [16:28]: You've been listening to Definitely Maybe Agile, the podcast where your hosts Peter Maddison and Dave Sharrock focus on the art and science of digital, agile, and DevOps at scale.