Model overreliance and misplaced implicit trust in the output of a GenAI model may lead to the acceptance of inaccurate, biased or insecure outputs without proper validation or oversight, potentially resulting in operational failueres, compliance breaches and flawed decision making.
Model Overreliance
CCC.GenAI.TH05
Related Capabilities
| ID | Title | Description |
|---|---|---|
| CCC.GenAI.CP21 | Generate Content | Ability to generate a response given a foundation model, parameter values, and a prompt. |
Related Controls
| ID | Title | Description |
|---|---|---|
| CCC.GenAI.CN02 | Model Output Filtering and Sanitisation | Inspect and validate GenAI model output before passing it to users, applications or plugins in order to filter or sanitise insecure or unreliable output and prevent sensitive data leakage. |
External Mappings
| Framework | ID | Relationship | Remarks |
|---|---|---|---|
| FINOS-AIGF | AIR-OP-018 | relates-to | Model Overreach / Expanded Use |
| FINOS-AIGF | AIR-RC-022 | relates-to | Regulatory Compliance and Oversight |
| MITRE-ATLAS | AML.T0062 | relates-to | LLM Trusted Output Components Manipulation |
| OWASP-LLM-TOP10 | LLM06:2025 | relates-to | Excessive Agency |
| OWASP-LLM-TOP10 | LLM09 | relates-to | Overreliance |
| SAIF | IMO | relates-to | Insecure Model Output |
| SAIF | RA | relates-to | Rogue Actions |