Essay
Calibrating vendor risk scores without false precision
Vendor scores look scientific when they carry two decimal places. They often hide a quieter truth: the evidence pack was incomplete, the questionnaire was rushed, and two reviewers disagreed on whether a certificate still applied. A risk scoring audit platform that pretends otherwise trains stakeholders to distrust the next update.
Prefer bands you can teach
Four or five well-defined bands beat a continuous scale that nobody can defend. Write each band with an example and a non-example. If a new joinery analyst cannot apply the band without asking a senior for a vibe check, the definition is unfinished.
Separate evidence quality from residual rating
A thin questionnaire should lower confidence, not automatically invent a mid-range score that looks decisive. Teach reviewers to mark evidence quality explicitly. When confidence is low, the published band should carry a visible caveat rather than a fabricated midpoint.
Calibration is a ritual, not a meeting
Pick a fixed sample of suppliers each quarter. Two reviewers score independently, then reconcile using a shared script: identify the criterion that diverged, decide whether the definition needs tightening, and only then adjust the score. Personality debates belong outside that room.
Watch for double-counted assurance
Inherited certifications, penetration test summaries, and contractual clauses often describe overlapping controls. Weighting labs in our Defensible Risk Scoring course spend time removing bonuses that inflate scores without changing decisions. The goal is not a harsher matrix — it is a matrix that moves when something material changes.
If you are redesigning vendor bands now, start with definitions and a small calibration set before touching dashboards. The platform will only be as honest as those agreements.