Data, AI, and Knowing When to Let Go - with Tommy Cotter

Welcome And Show Setup

Peter [0:04]: Welcome to Definitely Maybe Agile, the podcast where Peter Maddison and Dave Sharrock discuss the complexities of adopting new ways of working at scale. Hello, and we are back again. And today I'm here with Dave and with Tommy. So, Tommy, would you like to go and introduce yourself to our audience?

Meet Benzinga And The Mission

Tommy Cotter [0:22]: Sure thing. Thank you for having me. My name's Tommy Cotter. I'm a director of data products at Benzinga. And for those who are unfamiliar with Benzinga, we're a financial media company, so we write news about the stock markets. We primarily cater to a retail investor audience. Been around for around 15 years now.

Dave [0:47]: Can I just... what has changed in 15 years? I mean, 15 years is just about going to include 2008 and everything beyond that. So it feels to me like there's a really interesting journey that you've had over the last 15 years.

Tommy Cotter [1:02]: Yeah, there's been a lot of change in 15 years. I would say the biggest change, in my opinion, has just been the democratization of everything. You know, democratization is kind of a cliche term - everybody says that they're democratizing something. But in 2008, it really wasn't as cliche. Institutions and big businesses still held the keys. But over the last few years, people have just been able to access information in a better and easier form. Data moats are not what they once were. And so I would say that's my biggest takeaway. And then as far as the markets, there's a ton of interesting things happening with prediction markets and private markets, and retail investors just have a lot more access to trade or bet.

Data Quality Needs Human Review

Peter [2:09]: How does data quality come in when you've got this massive information coming from all these different sources? How do you look at things like provenance or validation when all of this data is coming in?

Tommy Cotter [2:24]: Yeah, it can be especially challenging. I think having a human in the loop nowadays is as important as ever. So many companies find it easy to spin up a product using AI and you're off to the races. But if it's just generating slop, people are going to be able to tell. Data quality is important. An example I'm seeing now is with our conference call transcripts endpoint. We recently created a product that transcribes earnings conference calls, and it utilizes AI under the hood to transcribe speaker names, but the AI is just looking at the audio and it doesn't have a great enrichment engine to properly transcribe those names. So we utilize humans in the loop to recognize whether it's the actual correct spelling of the name. So I think humans in the loop is still the number one way to get it done right.

Peter [4:03]: Yeah, I have that problem with my company all the time. It's Xodiac spelled with an X - it's never going to get that right.

Dave [4:10]: I'm just trying to think of a quip to come back on that one about spelling, but I guess I'm not sure how you'd pronounce the X anyway. When you're doing this - do you see that changing? What's interesting there is we definitely see this all the time with transcription tools, that there's a gap between names or any number of different things and what's actually being transcribed. But there are also databases, presumably, of conferences - who the speakers are, whose earnings call has a list of names and there'll be an agenda. So there are sources of data that can be looked for outside of that. There's still a need for human in the loop there.

Tommy Cotter [4:52]: Yeah, that's a good point. There definitely are databases where you could provide the model access to enrichment data where it could properly enrich these speaker names. It's just more of a pipeline that needs to be built. So if you're trying to quickly get something off the ground, the frontier models do a great job out of the box. And then if you want to build a more robust product using those enrichment tools, it's certainly possible.

Peter [5:29]: And that's probably not quite in your space, but adjacent. I can see looking at the earnings calls, looking at the transcripts and what's said there, and then comparing it to the published documentation that comes along with that and looking for discrepancies - or things being said in one which are not being said in the other - which might be leading to different changes. Almost like arbitrage of information and how that impacts stock prices.

Tommy Cotter [5:56]: Yeah, it's fascinating. A lot of times the official SEC document for an earnings release will come out at say 4 p.m. And investors maybe buy based on what's in that official document. And then an hour or two later, the official conference call happens where the executive is speaking about how the company performed. And a lot of times investors will find one nugget of information in that earnings call that contradicts what they just previously read in the SEC release. That happens pretty frequently. It's fascinating to see those examples.

Scaling AI Without Losing Judgment

Dave [6:50]: And that changes the outcome, the decisions that are being made.

Tommy Cotter [6:54]: Yeah, totally.

Dave [6:55]: Can you talk a little bit about the journey you've seen? I'm presuming your organization uses AI a lot - you're data heavy, you're transcribing lots of information. We've used the example of the earnings calls, but what has changed over the last three or four or five years? How rapidly have you adopted AI? How has it changed your business?

Tommy Cotter [7:22]: Yeah, people are able to do so much more with so much less now. Across the business, software developers are able to use AI tools to accelerate their production of code. And we're able to accelerate our production of content on the editorial side. So it's impacted us across the business. I think the one critical piece we're starting to see is that we still need to have people steering these tools and making the decisions on what to build and why to build it. People having the executive function and that self-agency to make these decisions about what to build and how to build it - that still holds a lot of value.

Peter [8:30]: So you're purposefully ensuring that there are humans in the loop as information comes in and processes run. Are there any elements where - because one of the things I'm starting to see, especially in some spaces where something is low risk - we're happy to say, okay, I trust AI to do this right every time, or at least the impact of it doing it wrong isn't going to worry me too much, or there's a subsequent deterministic step after that which will catch it if it does go wrong. Are you seeing that in the way you're building out systems?

Tommy Cotter [9:09]: Yeah, definitely. For anything that's low risk, like you said, we're okay with having AI fully manage that process. But anything that's client-facing, it's definitely important for us to have human eyes on it.

Staying Nimble As Tech Shifts

Peter [9:31]: What would you say has been your biggest learning out of the last - well, we're almost four years into this now since ChatGPT really hit the mainstream. What's your biggest learning in that time?

Tommy Cotter [9:45]: My biggest learning is that so much changes so quickly with AI, and every week there's a new advancement. So my biggest learning is just to be nimble and adaptive. We've spun up projects where we started with one architecture that made a lot of sense at the time, and then as time went on, certain technologies changed and there was a better solution. So I would say it's important to be nimble in today's day and age.

Dave [10:35]: Can you describe what nimble looks like? Peter and I come from the agile DevOps space, so we have our own idea of what nimble looks like. What do you look for on your teams in terms of speed of change or adapting to something like a technology shift? Because if I think of nimble, there is nimble like digital agencies - very, very nimble, changing from day to day. But a lot of organizations cannot handle that pace of adjustment. What do you see?

Tommy Cotter [11:09]: We're very much a startup culture, so we're able to change more quickly than some of the banks we work with. I think that's one advantage we have on being agile.

Peter [11:29]: When you look at your team - you presumably have a team looking after the data - what sort of characteristics are you looking for these days? And has that changed?

Tommy Cotter [11:41]: Yeah, I think it has changed. I would say backend is still extremely valuable. People who can understand how the whole system works, people on the DevOps side who understand everything down to where the code is hosted, how it's hosted, how deployments happen - that's certainly valuable. Whereas some of the front-end work has become less important. Building front-end user-facing components isn't as valuable as it once was.

Peter [12:22]: Why do you think that is?

Tommy Cotter [12:24]: I just think it's due to AI and how much easier it is to one-shot an entire application where previously people spent a lot of time on that.

Dave [12:40]: If you look at 2026, is there something specific that you and your teams are currently addressing that has significantly changed from last year? It may not be AI related, but every technical organization is continually having to evaluate what's going on, and things are moving very fast. What are the headaches your team is particularly focused on compared to last year?

Security Compliance As Market Signal

Tommy Cotter [13:10]: Yeah, one big thing is security and compliance. We're focusing on our SOC2 and GDPR compliance. And I think one critical reason for that, outside of just the advantages of safety and making sure our infrastructure is where it needs to be, is from an optics standpoint for external-facing customers. If you've done the work to invest in your security posture, both from a time and money standpoint, you're a legitimate business. These AI startups can be spun up so quickly now where they may not invest in those things, and the optics are that they're not legitimate. So that's one unique thing we're focusing on this year that we wouldn't have really thought about previously.

Dave [14:19]: And if I'm understanding that correctly, that's very definitely signaling to the market that your organization has taken care of the critical stuff.

Tommy Cotter [14:30]: Yeah, exactly. When people see a company, it's easy to perceive them as illegitimate if they haven't invested in those types of things.

Peter [14:50]: Is there anything exciting in the AI space - especially since you're working within data - that's making you think, oh, I can't wait for that? Not like "hey, here comes the next model release" - but things that you think could really be a game changer for you.

Tommy Cotter [15:14]: Offhand, I don't know if I can think of anything particular in the AI space right now. What about you guys? Do you have anything you're looking forward to?

Peter [15:26]: I'd like it if some of these tools actually did half of what the hype says they do. That would be lovely.

Dave [15:31]: Consistently, right? In a very controlled environment it looks like you can get something, and then you bring it into a real-world operational scenario and everything is not quite as it seemed.

Peter [15:44]: We still have this disconnect where I can build out an orchestration engine that puts together incredible solutions in a fraction of the time it would have taken a few years ago. That's wonderful. But then you hit the real world - all of the systems, the way they work, how they define work, what that looks like. It's an imperfect match. There is no one size fits all. It's how do we evolve the system into something that works for us. Unless we're willing to throw out everything. And if you're going to throw out everything, unless you've got very adaptable people, you're probably going to need a new group of people.

MCP Workflows Beyond Simple APIs

Tommy Cotter [16:42]: Yeah. I think MCPs are extremely interesting and becoming more and more important for making these connections. Previously we had APIs, which are great. A lot of MCPs are just wrappers around APIs, but I think MCPs that have these complex workflows and are handling multiple API endpoints in the background - those types of things are pretty neat and are going to become more important in the near future.

Peter [17:29]: That's an interesting one. The MCP effectively allows me to use natural language to talk to a set of tools and figure out how to interact with them. But most of them at the moment are set up as just here's your set of APIs. Skills sort of solve some of that, right? Because I can have a skill that even has deterministic scripts built into it that it calls. You can build out your own set of skills that will work for you. Have you started to do that within your organization - building out that organizational context that gets fed into all the work done by AI internally?

Tommy Cotter [18:50]: Internally we definitely have, and we're discussing what an internal MCP server will look like where internal users can get data from maybe an analytics tool, and then maybe that analytics tool checks against our CRM. Those types of workflows that are a little bit more complex and aren't necessarily just a single API call. That's something we've definitely begun thinking about.

Peter [18:55]: Are you looking at what context you need to create? So if there was - to work within the data space - this is how we define schemas, these are the principles that systems and dashboards and capabilities built within our environment must follow?

Tommy Cotter [19:18]: Yeah, we certainly have, and it kind of goes back to the basics. It's just about productionizing it. We've always documented and defined those things in Confluence and in our white papers. If we can transpose that into code and into a tool, how can we use those tools internally to be more efficient? That's something we've started talking about.

Peter [19:58]: I was just going to say those documents have a life cycle, so you need some sort of system or way of working that keeps them up to date as they change over time.

Tommy Cotter [20:08]: Yeah, and traceability too, because sometimes these agents can make changes and all of those changes need to be traceable back to the agent - why they made that change, when they made that change.

Dave [20:26]: Can you speak to - one of the interesting things as organizations make this shift is we learn from things going wrong. What are some of the big learnings you've seen? And I don't mean wrong in front of customers, I mean like you trip over your shoelace and it gets captured, but you find significant strategic shifts in how to work because of mistakes made or risks that bubble up. Are there any critical things you've seen where you go, okay, in future we've always got to look at this?

Traceability And Accuracy At Speed

Tommy Cotter [21:03]: Yeah, my biggest thing is - I'm mostly focused on the licensing side of our business. All of the news content and data within Benzinga, all that outbound flow to other businesses where they may use that data and white label it within their platforms. The biggest thing I focus on is validity and accuracy of that data. We always want to have a system in place to make sure the accuracy is there and that we never need to do a correction on some data or a news item that comes out.

Dave [21:51]: That sounds like speed is important, right? Whoever's white labeling it doesn't want a delay with that information. Any particular things you took away from that in terms of how to validate accuracy at speed?

Tommy Cotter [22:09]: One of the ways we've done that is through traceability - in a different sense, not tracing an agent's actions, but tracing where the data came from. Because a lot of the data is second-party data. It's not first-party data where we're going out directly and speaking to a CEO, although we do do that in some instances. So we want to be able to trace the data back to its original source. When we can do that, we can trace it and also calculate the turnaround time. That's always our north star - what's our turnaround time for reporting these accurate metrics.

Dave [23:00]: Thank you for that. And again I'm just thinking back to when we started talking - when you were describing the SEC information that comes out an hour or so before validated information from the earnings call, which may be in conflict.

Takeaways And How To Reach Us

Peter [23:16]: So we've been talking for about 25 minutes and it's about this time we normally like to wrap this up and give folks some points to take away. So Tommy, guest goes first - what point would you like our audience to take away?

Tommy Cotter [23:31]: A big point for me is self-agency. Now more than ever, people have the ability to make their own decisions. If they feel strongly enough about a certain product or topic, they can go out and build it. That's pretty cool, and it's unfolded very recently. It wasn't always the case - even five years ago it was expensive and cumbersome to build something, and now you can do it quickly.

Dave [24:21]: I'm going to pick something connected to that - the differentiation Tommy was mentioning around compliance and regulatory frameworks. In a hyper-competitive situation where anybody can spin something up very quickly, that's both an opportunity and a challenge. There's a need to protect through various modes, and one of them is regulatory.

Peter [24:46]: I would add the one around ensuring we think about where humans belong in all of this. There's still a need to have somebody check over and validate that we're not about to say something daft. While understanding that for low-risk things, where we've got deterministic processes and other checks in place, AI can run more freely. Having humans in the right parts of the system to validate is still necessary. I think that's an interesting takeaway for folks. With that, I'd like to say thank you very much, Tommy, and thank you Dave as always. We'll wrap it up there. People can send us any questions at feedback@definitelymaybeagile.com and don't forget to hit subscribe - we like new subscribers. Thank you all.

Tommy Cotter [25:20]: Yeah, thanks for having me.

Dave [25:22]: Thanks everyone. Bye!

Peter [25:24]: You've been listening to Definitely Maybe Agile, the podcast where your hosts Peter Maddison and Dave Sharrock focus on the art and science of digital, agile, and DevOps at scale.