“Bank-grade” is one of the most used and least defined phrases in financial software. It is meant to reassure, and it usually does. That is the reason to interrogate it. Here is what we think the phrase should oblige a vendor to show, and how we try to meet that standard ourselves.
Four things the phrase should mean
Enforcement, not intention. A guarantee that lives in application code is a statement about how the team works. A guarantee enforced by the database is a property of the system. Ask a vendor where each rule is enforced. If the answer is “in our service layer”, the guarantee is only as strong as the next release.
Evidence, and its sources. A number without a source is a slogan. “Thousands of tests” tells you nothing. “One hundred and twelve tests over a real database, counted on this date by this command” tells you something you can repeat yourself. Every figure Ledger publishes carries the date it was re-counted and the command that counts it, and the site’s own build fails if a figure goes stale.
An adversary. Tests written by the team that wrote the code share its blind spots. Something else has to try to break it. That means a separate audit function which treats every claim as false until it has been re-run live, and which records what it refused. Ledger’s first audit gate was refused, and that refusal is published. A history with no refusals in it is evidence that nobody looked hard.
Honesty about the gap. No system has done everything, and a vendor should say what was not exercised. That means naming the paths that were not run, the figures that were not measured and the certifications that are not held. Ledger publishes no response-time figures, because they were never measured on reference hardware. It says plainly that no external body has audited it. Its own readiness for service is graded with the partials and the gaps left visible.
What the phrase should not mean
It should not mean a certification badge alone. A certification says a set of controls was documented and sampled. It does not say the books cannot go out of balance. Uptime alone is not enough either, because a service that is always up and occasionally wrong is worse than one that is occasionally down. And it should never mean “trust us” dressed in a serious font.
How to check a vendor in an hour
- Ask for one guarantee, the place it is enforced, and the test that guards it. Then ask to see the test fail on a broken input.
- Ask for the date and source of any number they quote.
- Ask who audited the system, whether that person could refuse a release, and whether they ever did.
- Ask what was not done. A vendor with nothing on that list has not made the list.
A worked example of a claim with a source
Consider the sentence “the ledger serves fifty-four live probes against the running service”. On its own it is a slogan. Here is what turns it into evidence:
- The count has a command. It is the number of probe scripts registered in the harness that runs against a served instance, and the exact command that produces the number is published beside it.
- The count has a date. It was taken on a stated day, and the site’s own build refuses to publish a figure older than four months.
- The probes have a discipline. Each one proves it can detect a known defect before its clean result is trusted. Each one also tests its guard in both directions. The guard is shown to block what it should and to release what it should.
- The probes have a history. Several were written because an audit round found a defect that the existing suite had missed. The suite grows with what it failed to catch, and the audit history says which rounds added what.
Any one of those four could be omitted and the sentence would still sound the same. The phrase “bank-grade” needs all four behind it.
Red flags, briefly
- A round number of tests, or a number with no date.
- “Audited” without saying by whom, whether they could refuse, and what they found.
- Guarantees described in the future tense, or with the word “designed to”.
- A response-time figure with no description of the hardware and load that produced it.
- A security page that lists controls but no way to verify a single one.
None of these prove a vendor is wrong. They prove the vendor has not shown you enough to know, and in a system that holds money, not knowing is the risk.
Ledger’s answers to all four are on the assurance and audit history pages, including the findings that were raised against it.