Threat modelling
Telling a five-person team to implement a governance framework is not helpful. Asking what could go wrong with the thing they are actually building is. This module makes that question systematic, and does most of the writing for you.
It takes about thirty minutes with the people who built the system, and it produces a record a customer's security review, an auditor or a regulator can read.
The loop
Map
What the system can see, retrieve, remember, call and do. This is the only step that needs thinking.
Imagine
What could go wrong: by accident, by attack, or in ordinary use. The product proposes the list.
Prioritise
By impact, likelihood and blast radius. Computed, so two people rating the same facts get the same answer.
Control
Prevent, constrain, detect, respond, assure. Each scenario arrives with controls already drafted.
Test
Try to break the control and record what happened. Repeat when the system changes.
The five questions
Tick what is true today, not what is planned. Everything else follows from these answers.
What can it see?
User input, customer records, employee records, health or other sensitive data, payment data, credentials.
What can it retrieve?
Internal documents, web pages, email, source code. Anything retrieved is untrusted data.
What does it remember?
The conversation, or a profile that persists. Memory that persists can be poisoned once and used many times.
What can it call?
Tools, a database, code or shell commands, third-party services. Each one is a way for a manipulated model to act.
What can it do?
Send messages outside, move money, change records, deploy code, influence a decision about a person, hand off to another agent.
What comes back
Concrete scenarios, each with controls at the right layers and a test written so you can run it this afternoon. Five capabilities typically produce eight scenarios and around twenty controls.
It answers with a document the person was not allowed to see
Authorisation at retrieval, with the asker's own permissions. Test: ask the same question as three people with different permissions.
A retrieved document tells it what to do, and it obeys
Treat retrieved bytes as data, keep the tool allowlist narrow after external content. Test: plant an instruction and check it is not followed.
It pays out far more than anyone intended
A cap on what happens without a person, and approval above it. Test: try to beat the cap three ways, including through a document.
It works exactly as designed and still treats a group worse
Named decision factors, proxy checks, outcome monitoring after deployment. Test: run the same cases with one characteristic varied.
The five control layers
An empty layer is the finding. A system with four preventive controls and nothing that detects will learn of its first incident from a customer.
What counts as tested
"We have a guardrail" is not evidence. A control has one of four states, and only one of them counts.
Untested
Nobody has tried to break it. It is a claim.
Not holding
The last test failed, or passed only in part. A half-working control is not a control.
Test out of date
It passed, more than six months ago. Systems change; the test has to be repeated.
Tested
A passing test within the last six months. This is the only state that evidences anything.
What this proves in the compliance register
A control with a current passing test can be written into the compliance register as evidence, against the obligations it meets: accuracy and robustness, human oversight, record-keeping, security of processing, and the matching NIST and ISO entries.
An untested control writes nothing, on purpose. The screen shows which scenarios evidence nothing yet, which is the work still to do. Applying twice writes nothing the second time.