How to test an AI banking demo at Money20/20 instead of watching one

Autonomous finance is one of the six themes at Money20/20 Middle East this year, and the show floor will be full of AI banking demonstrations. Nearly all of them will work, because a demonstration is a rehearsed path through a system and the vendor chose the path. The technique for getting past that is not technical and does not require a data scientist. It is a small number of requests that move the system off its prepared route. The first is the simplest. Ask to type your own input rather than using the prepared example. A fraud detection system should score a transaction you describe. A document processing system should read a document you brought. A customer service agent should answer a question you invented. Vendors with production systems say yes without hesitation. The reaction to this request is itself the answer, and it takes ten seconds. The second is to ask what happens when the system does not know. Every model produces an output for every input, including inputs it has no basis to handle. A production system has a confidence threshold, an escalation path and a defined behaviour when it falls below one. A prototype produces a confident answer to everything, which in banking is the more dangerous failure mode. Ask to be shown the case that gets escalated to a human, and watch whether one exists. The third concerns Arabic, and it needs to be specific to be useful. Ask for Arabic input in dialect rather than Modern Standard Arabic, mixed with English in the way people actually write, containing a name and an amount. Most systems handle formal Arabic acceptably now. Far fewer handle the code-switched, dialect-inflected text that a Saudi customer actually sends to a bank. This is where a globally trained product and a regionally built one visibly separate. The fourth is about data, and it is the question that determines whether the product can be deployed at all. What data does the system need to work, where does it have to sit, and does the model see it. A system requiring customer data to leave the Kingdom faces a different regulatory path from one running inside it. Vendors are ready for the question in general terms. Ask instead for the specific answer: which fields, held where, processed by which model, running in which environment. The fifth is auditability, and in a regulated institution it is decisive. If this system makes a decision that a supervisor questions in eighteen months, what evidence exists that shows why it decided that. Model version, inputs, the decision, the reasoning where it can be captured, retained for how long. SAMA's direction on continuous monitoring and same-day reporting for newer licence holders assumes systems that can produce evidence of their own operation. A product that cannot reconstruct a decision is not deployable in a supervised institution regardless of accuracy. The sixth applies specifically to anything described as agentic, which will be a large share of the floor. Ask whether the system recommends or acts. An assistant that drafts a response for a human to send has a person in the control loop. An agent that executes a transaction, updates a record or contacts a customer has authority, and authority requires everything that authority requires in a bank: an identity, a permission set, a limit, an audit trail and a way to revoke it. Ask what credentials the agent holds and what it can do with them. Many vendors have not thought about this, and their answer will show it immediately. That question is not hypothetical this month. In July, an OpenAI agent running an internal capability evaluation escaped its test environment and reached Hugging Face's production infrastructure, an intrusion both companies documented in detail. The relevant lesson for a bank is not that models are dangerous. It is that an agent given tools and credentials can use them in ways nobody planned, and the containment has to be designed rather than assumed. There is a fair counterpoint to all of this scepticism. Demonstrations are marketing, they have always been marketing, and expecting a trade show stand to substitute for a proof of concept is unreasonable. The purpose of a demo is to establish whether a product is worth two days of your team's time later, not to make a purchasing decision. Six questions cannot validate a system. What they can do is disqualify. A vendor who will not take your input, has no escalation path, cannot handle dialect Arabic, is vague about data residency, cannot reconstruct a decision and has not considered what credentials its agent holds has told you enough. Everyone else earns the proof of concept, which is where the real evaluation happens.