CWE-1426: Improper Validation of Generative AI Output
CWE版本: 4.18
更新日期: 2025-09-09
弱点描述
The product invokes a generative AI/ML component whose behaviors and outputs cannot be directly controlled, but the product does not validate or insufficiently validates the outputs to ensure that they align with the intended security, content, or privacy policy.
常见后果
影响范围: Integrity
技术影响: Execute Unauthorized Code or Commands Varies by Context
潜在缓解措施
阶段: Architecture and Design
描述: Since the output from a generative AI component (such as an LLM) cannot be trusted, ensure that it operates in an untrusted or non-privileged space.
阶段: Operation
描述: Use "semantic comparators," which are mechanisms that provide semantic comparison to identify objects that might appear different but are semantically similar.
阶段: Operation
阶段: Build and Compilation
检测方法
方法: Dynamic Analysis with Manual Results Interpretation
Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.
方法: Dynamic Analysis with Automated Results Interpretation
Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.
方法: Architecture or Design Review
Review of the product design can be effective, but it works best in conjunction with dynamic analysis.
观察示例
参考: CVE-2024-3402
chain: GUI for ChatGPT API performs input validation but does not properly "sanitize" or validate model output data (CWE-1426), leading to XSS (CWE-79).
引入模式
| 阶段 | 说明 |
|---|---|
| Architecture and Design | - |
| Implementation | - |