General-Purpose AI Agents Are Not Built for High-Stakes Finance

Megan - 00:55 Welcome back to CFO Weekly. Today, I'm joined by Jeremy Ung, chief technology officer at BlackLine, a leader in AI powered financial operations and digital finance transformation. Jeremy brings more than two decades of experience across software engineering, product management, and enterprise technology leadership, including senior roles at Amazon Web Services, Microsoft, and Apptio. At BlackLine, Jeremy oversees the company's global technology direction, focusing on connected data, AI powered platforms, and what the company calls Agentic Financial Operations, where digital AI workforces operate alongside finance teams with explainability, governance, and human oversight built in. In this episode, we'll explore a growing question facing CFOs and finance leaders. As general purpose AI agents become more common, are they truly safe enough for high stakes financial operations? We'll discuss trust, governance, explainability, and what finance organizations need to consider before turning critical workflows over to AI. Jeremy, thank you so much for joining me today on the show, and welcome.

Jeremy - 02:05 Thank you for having me.

Megan - 02:07 Yeah. I'm really looking forward to this conversation. But before we jump into that, can you just walk us through your career journey from engineering and cloud leadership to building AI powered platforms for the office of the CFO?

Jeremy - 02:21 Jeremy Ong, chief technology officer at BlackLine. I lead our engineering and product organization. We are building AI software that powers finance and accounting workflows in the office of the CFO. Prior to joining BlackLine, I led engineering and product in the cloud cost management world, building solutions to help optimize spend. I've also worked alongside building the cloud infrastructure at AWS, building cost management capabilities to help finance professionals and others understand and manage spend, and that's all increasingly more relevant as people are looking for how do we get ROI from token spend.

Megan - 02:59 And when CFOs hear the term AI agent, what do you think that they misunderstand most about how these systems actually operate inside of financial environments?

Jeremy - 03:11 This has become such a buzzword, AI agent, agentic AI. What does it really mean, and how does the CFO navigate that? Well, I think the one thing that is critical is that it's really a change in what you might think of as your workforce. AI agents are really augmenting teams. They're adding new humans and or human like capabilities to existing teams. They're doing work that otherwise humans would do, and helping you do that at scale. And so you should really think about this as workforce augmentation. And I think one misconception is that it's just like traditional software, and it's not. You have to think about governance, guardrails, controls, just like you would for any sort of employee. I think the example I give the most is that when you hire or onboard a new employee, you give them training. You give them a mentor or a buddy. You help them understand the business. And the same is true of AI agents. They need those same frameworks, and we call that context in our world, which help them understand how does your business operate, what are the guardrails or policies it needs to respect, and so deploying agents is more complex than software. You need to create that framework or environment, and that's what we do here at BlackLine.

Megan - 04:26 And I'm just curious. So when I think of an agent, I think of something that kind of needs to be built and customized. Is that the case, or is there something that can just plug in and be deployed and become a functional agent?

Jeremy - 04:42 It's a little bit of both. I'll give you a great analogy. It's like a great pair of shoes. You definitely fit them from day one, but great shoes become more comfortable and fit your feet perfectly as you begin to wear them. That's like an AI agent. Right? You are going to have an agent that can get a task done. Like, let's give you an example. So the accruals process is one where accounting teams need to go and reach out to figure out how much vendors are going to bill them over the course of a month. They need to accrue this to be able to close their books. Now this process is pretty straightforward. It's a standard accounting practice, and so it's easy to deploy an agent that can do that basic process. But now we go into an organization, and now you have a specific set of vendors. Where do you store that information? Where is your purchase order stored? And to be able to bring all that information together, that's where we have that layer of customization, that last mile that takes the human work, let's call it the mundane work, out, and automates it with AI. So it's a combination of both. You have agents that are able to do things out of the box. But as you begin to tailor them to your workload, as they begin to fit the needs of your organization, they become even more effective and take more work that you otherwise would have to do manually.

Megan - 05:57 And as more general purpose AI tools enter finance workflows, where do you see the biggest risk when those systems are applied to high stakes financial operations?

Jeremy - 06:07 I'm super excited as a technologist around AI and the ability to build things. I have a million ideas floating around and I would love to be able to build. But what people don't understand is that there's a real cost to build versus buy. When you build, you have to know security. You have to know how to upgrade software, how to maintain it. So all the same things that has gone into building traditional software needs to be considered for AI agents, including more, which is how do you govern the prompts? How do you make sure that there's no prompt injection? How do you make sure that they don't break outside of the boundaries of what they're allowed to do? There's a recently published article on OpenAI where their AI agents breach security. And so with AI agents, it's tempting to build because it seems so easy, but all the same controls and governance and more that go to traditional software need to be in place.

Megan - 06:59 Are there warning signs that tell you that maybe an organization is moving too quickly when it comes to AI?

Jeremy - 07:06 Everyone is getting pressure to leverage AI quickly. I think it's a board topic that comes up a lot. Are you effectively using AI? What's your plan around leveraging AI agents? I think not understanding the business metrics that people can't answer, what are you trying to solve? What problems are you trying to solve in your business? That's a warning sign. Deploying AI agents needs to be grounded in solving core business problems, and that's definitely a key red flag. Not having a governance strategy is a red flag. And so those things in conjunction result in failed AI deployments, tons of spend without ROI, I think, or even worse, risk to the business. So those are the kinds of things that we're seeing, and that's what we're trying to help people build safely on top of proven platforms that allow you to customize, extend, and build agents, but within safe guardrails and frameworks.

Megan - 07:57 And you've spoken about explainability, governance, and human oversight. Why are those elements especially critical in finance compared to other business functions?

Jeremy - 08:08 So we've had a lot of conversations with regulators, auditors, and others lately, and I think this really sometimes needs to be demystified for finance and accounting. Good enough answer is not sufficient. You need accuracy. You need controls in place. These are relied on by financial statements by investors to make decisions, and so it's critical that that information is correct. It's critical that you have controls in the process. That's why we have SOX and other things that provide those controls. And so what is maybe misunderstood in finance is that you don't need that if you are going to use AI. Well, you definitely do, and the problem magnifies because it's not just one human or 10 human or a team of humans doing the work that understand the expectations. It's agents who are really unclear of what the rules are if you don't give them the appropriate context, if you don't have the appropriate guardrails. They don't know how to operate, and so this is why domain specific agents are critical, so ones that understand the business, the rules of finance and accounting, and then having agents purpose built and designed for this space.

Megan - 09:11 And what level of transparency do you think is realistically achievable? I mean, obviously, you don't want to be operating in a black box, but do you think a 100% transparency can be achieved as far as what the agents are doing all the time?

Jeremy - 09:25 I never like to speak in absolutes, but yes, in terms of transparency, what we want is a full audit trail. You want to know the full agent resume. So if you have an agent, what were the skills it had access to? What was the context it was supplied with, the organizational data, policies, and other things? Was it authorized? Did it behave appropriately? And then what model did it use? We're in this era of rapid model development, and it's unclear how shifts in that technology will affect outcomes in the future. So if you look five years from now, you look back on the things that were performed by agents today, we may realize there were biases, or there were errors in how certain agents handled certain data. That information needs to be captured, immutable or read only audit trails so that people can go back and understand how this data was constructed. And this is an expectation that's falling out of audit practices and others to be able to do this at scale. And so I think it's hypercritical those fundamentals are in place so that you can have full transparency in the process. As much as possible, have determinism, repeatable outcome in what AI agents are doing.

Megan - 10:29 And talk to me about the difference between a general purpose AI agent and an AI system that was purpose built for financial operations.

Jeremy - 10:38 Generic agents, it's very easy to build, as I mentioned earlier. But to be able to build with context, and that's the knowledge of how things should be done, not just as a finance and accounting practice, but within your organization, is critical. Again, let's take the new hire analogy. If you hire a summer intern, you can't expect them to perform the work that someone with twenty years of experience has been doing unless you give them training, unless you give them tools. And even still, you need some supervision and oversight. That's the same for AI agents, and a generic agent especially has to be augmented with those tools, with those guardrails, policies, and context to be able to perform effectively, and that's why specifically designed agents are so critical. And even if you specifically designed agents, you need a control and governance layer. I think that goes without saying. So you can build the best agent possible. You need to get trust but verify. You can trust them to do the work, but you need to be able to verify that what they were doing is correct. And that system of continuous governance, what we call our finance control console, is what we've done to help customers have that assurance. And so these are the ways that we are tackling making AI accessible and agents actually capable of doing work in a trustworthy manner in finance and accounting.

Megan - 11:52 And how should finance leaders think about balancing automation with accountability, especially as AI starts making or influencing decisions?

Jeremy - 12:01 I don't think they can make the trade off on accountability. Accountability is table stakes. It is a foundational building block. And I think maybe the way to think about this is if you have designed your agentic AI processes with accountability as a core tenet, then you're putting yourself at risk. It's not a trade off people can make or should be making. I think that's what you're hearing from firms that provide audit services. The accountability still has to be there. It's the means of testing that accountability and the ways we go about asserting that accountability is there are going to scale with AI, but it's still a fundamental principle.

Megan - 12:39 In your view, what separates AI deployments that genuinely improve finance operations from those that just create new layers of risk and complexity?

Jeremy - 12:48 At the end of the day, I really do think it comes down to tying it to business value. We have spent a lot of time over twenty five years gathering these data points. We've applied traditional automation techniques, and we're seeing that what AI gets you is reducing time to reconcile accounts by 90 plus percent, reduction in unmatched transactions of up to 95%. Right? Like, these are huge numbers, but at the end of the day, it's really showing that there's ROI in leveraging AI effectively in a controlled environment. When you layer these different techniques together, there are real business value outcomes, like shortening your close cycle, reducing risk in your business, reducing uncollected cash, and be able to optimize your cash balance. These are real business levers that a CFO needs, and they are achievable to a much greater degree with AI.

Megan - 13:39 And what are the biggest implementation mistakes that you're seeing organizations make right now when it comes to AI and finance?

Jeremy - 13:48 I think we do a lot of work with partners. Our partners help us scale, and our partners have a broad range of expertise across different industries, different vertical segments. They have a wealth of knowledge that we are partnering with them to build for our customers to be able to build these solutions. And so I think that's one of the big things, leveraging partners. We also leverage other customer code. We have, for example, great customer references. ExxonMobil calls us out in their earnings calls using BlackLine. We have other oil and gas companies that have formed things like working groups to be able to learn from each other, to build on that community of practice around how to both leverage existing tools, but AI tools, of course, on BlackLine and others, but this is building that community to build best practices at scale, and that's the value we're really bringing to our customers here.

Megan - 14:37 And as organizations move toward agentic financial operations, how do you see the roles of controllers, accountants, and finance teams evolving?

Jeremy - 14:46 That's a great question. So one of the big changes is really the up leveling. And I think the term used in engineering, and I often get criticized for this being an engineering focused term, is steering. And so instead of telling an agent exactly what you wanted to do, you set goals. You set outcomes, and these are tied to goals as your business. And you have agents independently operating, and you steer them or guide them towards that outcome. You nudge them, course correct them, just like you would an employee. Ask you for feedback. They'll check in with you to ask, does this look right? Is this the right approach? You are going to give that nudge or feedback to AI agents, and that is one of the shifts in mindset. So when you have a controller with a workforce, this workforce is now scaling 10 plus x. Each individual contributor now manages a team of agents, and they need to steer that team, nudge and guide them, to help them achieve those business goals. And that's a big shift that we're seeing in industry across all areas that are leveraging AI, and it's one that we're helping our CFOs and chief accounting officers understand and operationalize.

Megan - 15:52 And where do you think that human judgment is going to remain indispensable regardless of how sophisticated AI becomes?

Jeremy - 16:00 Well, definitely it's a hard requirement to have a human in the loop for auditability. There are certain things that need a human to review still. There are hard judgment areas and review processes that need a human to validate the work that AI agents are doing. Even for me in software development, humans are required to review and validate the code that AI agents write. Now this has to happen at scale, and we do use AI to help with that, but it is part of the process. And so I think human judgment is critical. In the end, ultimately, a human is going to take responsibility for a financial statement that is produced even if AI is leveraged, and so humans in the loop are critical to that process.

Megan - 16:37 And this is the last question, and I'm going to break it into two questions. But first of all, how do you see the role of the CFO evolving over the next, let's say, three to five years? And secondly, how is BlackLine evolving to help them evolve?

Jeremy - 16:54 I think scale is the biggest change. CFOs continue to scale and become more strategic. They're focusing on how do we fund or solve different business problems? How do we fund different areas? What are the strategic growth levers? I think a lot of the work that helps feed those more strategic processes are going to be automated through AI and through other techniques, but AI agents are going to help those teams scale and become more effective. And so I really do think about being a metrics driven culture, being data driven is a shift that CFOs are already making. You see that shift into embracing that role across the rest of the business, and that's going to be increasingly something that they continue to shape for the rest of the company's strategy and the culture.

Megan - 17:34 And the second part is how is BlackLine evolving? How do you see the products evolving over the next three to five years?

Jeremy - 17:41 That is a big shift. We are embarked on our BlackLine 3.0 strategy, and what that really means is the transition of our software from what you traditionally think of as a SaaS application that humans use to one where the users of BlackLine may actually predominantly be agents performing tasks. And the change of humans into a supervisory steering role is a critical shift where the user interface changes, what you need to see changes. The paradigm is shifting, and we've evolved as a company to embrace that. I think it's scary and exciting to be in this moment in time. I think all the time, I wake up excited about the problems I'm solving because we are at this cusp of a change in software and how it works. You're going to see in the future how software is no longer just a web browser application. You're going to be able to talk to your applications. You're going to be able to type messages to them in teams. They're going to be able to reach out to you when they have a question. And that's not how we think of software today, but that's where we're going, because that's how it's shaping to really meet the moment in human needs.

Megan - 18:47 Crazy to me to think about how much just AI has evolved in the last two years, and very excited to see where we will be in two to five more years.

Jeremy - 18:57 Yes. I think one of the big shifts as well is we might stop talking about AI. I know it's been the hot topic of every podcast, every Tech Talk, but we're going to reach a level of acceptance and understanding of it being a part of the way we work. And it's about now leveraging these tools effectively. Just like we learned to use spreadsheets, just like we learned to use email, this is going to be another tool that helps humans scale. And it's an exciting change, it's a lot to navigate, and that's what we're really trying to help our customers do, navigate that change and scale their businesses.

Megan - 19:30 Jeremy, thank you so much for taking the time to be with us here today to share your knowledge.

Jeremy - 19:35 Thank you so much for having me.

Megan - 19:36 Yep. And to all of our listeners, please tune in next week. And until then, take care.