5 AI questions to ask AP automation vendors before it goes live

I head up pre-sales at SoftCo, which in practice means I spend most of my time being asked questions by finance teams. About trends, about what’s coming, about what they should be doing differently in AP. And over the last while, the questions about AI have changed shape.

It used to be “what can this do.” Now it’s “how do I keep control of it.” The capability isn’t really in doubt any more. What people are working out is how to bring AI in AP automation without ending up somewhere they can’t defend.

There’s a statistic I’ve been opening with, because it captures the gap neatly. Around 93 percent of US companies expect to deploy or scale AI in finance in the next 18 months. It’s already across a lot of AP, invoice capture, matching, approvals. But what we’re seeing is that most finance teams can’t fully explain how an AI-supported decision was actually made, and that’s where the audit concerns, the compliance challenges and the governance exposure start. The adoption has run ahead of the ability to account for it.

So I sat down with our Head of Engineering, Ben Fitzgerald, to work through what you actually need to ask before you let AI into the process. What follows draws on that conversation. The reason I keep coming back to these five questions is that they’re the ones I wish more finance teams were asking of vendors.

Why “it looks right” isn’t good enough any more

It’s worth understanding why this is a harder question for AI than it was for the automation finance already trusts.

The AP automation we’ve had for years is predictable. You put in the same inputs and you get the same output. Two and two make four, and you can trace the path it took to get there. That’s why finance trusts a spreadsheet. It behaves the same way every time.

Generative AI doesn’t sit in that frame. It’s non-deterministic, which means the same input can give you a different output, and you can ask it the same question twice and get two slightly different answers. In most settings that’s fine. In finance, which is audited and has to be compliant, it’s not a small thing. And the catch is that a wrong answer doesn’t arrive looking wrong. It’s like talking to someone very confident and eloquent. It reads as perfectly reasonable.

Which is the whole problem. If you can’t tell a right answer from a convincing wrong one just by looking, you need a way to check. That’s what these five questions are for.

1. Can I trace any figure back to its source?

This is the one I’d start with every time. For any number an AI-supported process gives you, can you trail it back to a specific ledger line or invoice line, and verify it yourself.

The test is simple: can you see the source of the data, trail it all the way up, and verify it yourself. If you can do that, you’ve got a control. If you can’t, what you’ve actually got is a result that you’re hoping is right. Those are very different things to put in front of an auditor.

2. Will my data be used to train the model?

The second question is about where your data goes once it’s in. Will it be used for any training purposes, and could customer information end up in the base model where it might resurface later.

This is one of those things you don’t get to undo once it’s happened, which is why it’s worth asking before you sign rather than after. It’s a straightforward question and a vendor should have a straightforward answer.

3. Where does my data sit, and where does the compute happen?

This is really two questions, and it’s worth separating them. The first is where the data sits, because you may have compliance needs that require it to stay in a particular region. The second, which people miss, is where the compute happens. It’s not only about where the data lives, it’s about whether the actual processing stays in the same region too.

For anyone with data-residency obligations, the second half of that question is the one that catches people out.

4. For anything automated, what are the limits?

The fourth is about agents and automated actions, which is where AP is heading. Wherever something can take an action on its own, it should be clear what it’s allowed to do, under what conditions, and to what threshold. Anything anomalous has to come back to a human.

A control can be as simple as an agent only being allowed to do so much, or only up to a certain budget tolerance, with the explicit rule that it acts under these conditions and anything outside them goes to a person. The point is that the boundaries are set deliberately, not discovered after something goes wrong.

5. Can I actually see what it’s doing?

The last one follows from the fourth. If something is acting on your behalf, can you see those operations happening, in a dashboard somewhere, rather than them running invisibly in the background.

Visibility is what turns an automated process into an auditable one. If you can’t see it, you can’t check it, and if you can’t check it, you can’t stand over it.

The thread through AI in AP automation

If there’s a single idea connecting these, it’s this. Your operational indicators will tell you how fast the system is going. They won’t tell you whether you can defend what it did. Speed isn’t the same as safety, and these five questions are how you tell them apart.

None of them are exotic. A good vendor will have clear answers to all five, and the answers themselves will tell you a lot about how ready the vendor is for AI in AP automation. The finance teams asking these questions on our calls are the ones who’ll adopt AI well, because they’re building the checking in from the start rather than bolting it on after an auditor asks an awkward question.

For the full argument on why verification has to be the standard, my colleague Ben Fitzgerald, our Head of Engineering, makes the full case for verification as the standard in the session these questions came from. It’s available on demand: AI in AP under control: how to manage risk, governance, and responsible use.