Open-Source Liveness Models vs. Independent Testing: Know the Gap

Why benchmarks mislead

Academic liveness models are trained and tested on public datasets, usually captured in a lab with a handful of devices and a limited set of attack types. Production brings thousands of phone models, harsh lighting, older cameras, varied skin tones and attackers who adapt. This difference between the training world and the real world is called domain shift, and it is the main reason a model that looks excellent on paper disappoints live.

What independent PAD testing does

Presentation attack detection (PAD) is tested against ISO/IEC 30107-3. Accredited labs such as iBeta run a structured evaluation: testers build attack artefacts and present them to the system, alongside genuine users, and the lab reports how many attacks got through and how many real users were rejected.

• Level 1 uses lower-effort attacks such as printed photos and on-screen replays.
• Level 2 uses higher-effort attacks such as more sophisticated masks.
• To pass, a product has to let essentially no attacks through while keeping genuine-user rejections within a limit.

It is a meaningful bar. It is also a snapshot of one product version, on certain attack types, at one point in time.

Why open-source models often struggle

• Narrow training data. Public datasets are small and dominated by a few capture setups.
• Limited demographic coverage. Uneven performance across skin tones and ages is a known risk.
• No adversarial maintenance. Nobody is updating the model as new attacks appear.
• Single-frame decisions. Many models judge one image and ignore temporal cues.
• Licensing. Some research datasets forbid commercial use, which can taint a model built on them.

How to evaluate any liveness vendor

1. Ask for the independent test report, with product version, attack types and both error rates.
2. Be wary of a single "99% accuracy" figure with no definition.
3. Run a pilot on your own users and devices. Measure how many genuine users are rejected.
4. Red-team it yourself with printed photos and screen replays, and see what passes.
5. Check that the decision is server-side and returns scores you can inspect.
6. Plan layered defences so liveness is not your only control.

Where IDNexus stands

We would rather you see the facts than a badge. IDNexus does not claim iBeta certification. Our Basic tier uses an open-source liveness model, which is why it is positioned for low-risk gating, and why Standard and Enterprise add document checks, face matching and, on Enterprise, deepfake screening. Every response includes the scores, tier and model version behind the decision, every capture goes through a quality gate, and the decision is made server-side so you can set thresholds and route borderline results to review.

Ask us, or anyone else, for test evidence, then verify it with a small pilot. That is the only evidence that reflects your users.