CWE-1039: Inadequate Detection or Handling of Adversarial Input Perturbations in Automated Recognition Mechanism
CWE版本: 4.18
更新日期: 2025-09-09
弱点描述
The product uses an automated mechanism such as machine learning to recognize complex data inputs (e.g. image or audio) as a particular concept or category, but it does not properly detect or handle inputs that have been modified or constructed in a way that causes the mechanism to detect a different, incorrect concept.
常见后果
影响范围: Integrity
技术影响: Bypass Protection Mechanism
说明: When the automated recognition is used in a protection mechanism, an attacker may be able to craft inputs that are misinterpreted in a way that grants excess privileges.
影响范围: Availability
技术影响: DoS: Resource Consumption (Other) DoS: Instability
说明: There could be disruption to the service of the automated recognition system, which could cause further downstream failures of the software.
影响范围: Confidentiality
技术影响: Read Application Data
说明: This weakness could lead to breaches of data privacy through exposing features of the training data, e.g., by using membership inference attacks or prompt injection attacks.
影响范围: Other
技术影响: Varies by Context
说明: The consequences depend on how the application applies or integrates the affected algorithm.
潜在缓解措施
阶段: Architecture and Design
描述: Algorithmic modifications such as model pruning or compression can help mitigate this weakness. Model pruning ensures that only weights that are most relevant to the task are used in the inference of incoming data and has shown resilience to adversarial perturbed data.
阶段: Architecture and Design
描述: Consider implementing adversarial training, a method that introduces adversarial examples into the training data to promote robustness of algorithm at inference time.
阶段: Architecture and Design
描述: Consider implementing model hardening to fortify the internal structure of the algorithm, including techniques such as regularization and optimization to desensitize algorithms to minor input perturbations and/or changes.
阶段: Implementation
描述: Consider implementing multiple models or using model ensembling techniques to improve robustness of individual model weaknesses against adversarial input perturbations.
阶段: Implementation
描述: Incorporate uncertainty estimations into the algorithm that trigger human intervention or secondary/fallback software when reached. This could be when inference predictions and confidence scores are abnormally high/low comparative to expected model performance.
阶段: Integration
描述: Reactive defenses such as input sanitization, defensive distillation, and input transformations can all be implemented before input data reaches the algorithm for inference.
阶段: Integration
描述: Consider reducing the output granularity of the inference/prediction such that attackers cannot gain additional information due to leakage in order to craft adversarially perturbed data.
检测方法
方法: Dynamic Analysis with Manual Results Interpretation
Use indicators from model performance deviations such as sudden drops in accuracy or unexpected outputs to verify the model.
方法: Dynamic Analysis with Manual Results Interpretation
Use indicators from input data collection mechanisms to verify that inputs are statistically within the distribution of the training and test data.
方法: Architecture or Design Review
Use multiple models or model ensembling techniques to check for consistency of predictions/inferences.
引入模式
| 阶段 | 说明 |
|---|---|
| Architecture and Design | This issue can be introduced into the automated algorithm itself due to inadequate training data used as well as lack of validation, verification, testing, and evaluation of the algorithm. These factors can affect the overall robustness of the algorithm when introduced into operational settings. |
| Implementation | The developer might not apply external validation of inputs into the algorithm. |