Counterfactual Causal Prototype Memory with Drift-Aware Adaptation for Open-World Recognition under Domain Shift

Abstract

Deploying visual recognition systems in the open world requires solving two coupled problems at once: generalizing to unseen visual domains and rejecting semantically novel categories that were never observed during training. Most existing methods treat closed-set domain generalization and open-set recognition separately, and virtually none provide a principled, label-free mechanism for adapting a deployed model as the input distribution drifts while resisting corruption from the very unknown samples they are meant to reject. This paper presents CCPM-OWL, a counterfactual causal prototype memory with drift-aware adaptation that unifies these capabilities in a single framework. A backbone feature is disentangled into a class-relevant causal representation and a domain-specific nuisance representation through domain adversarial training, an independence penalty, a reconstruction constraint, and a counterfactual intervention that swaps nuisance factors across domains and enforces invariance of the causal code and its prediction. Each known class is summarized by a compact multi-prototype memory with per-prototype radii, and unknown categories are detected by a source-only split-conformal score that fuses prototype distance, free energy, and predictive entropy, with a finite-sample coverage guarantee obtained without ever observing a target unknown. At deployment, a maximum-mean-discrepancy and prototype-shift drift detector triggers a safe, gated prototype update that is admitted only when the sample is confidently known, in-distribution, and geometrically consistent, so that admitted unknown contamination cannot corrupt the memory. On the PACS benchmark with the sketch domain held out and three categories treated as unknown, CCPM-OWL attains an H-score of 85.3%, an unknown-detection AUROC of 89.2%, and an open-set classification-rate area of 0.741, surpassing softmax, OpenMax, energy, and reciprocal-point baselines, while its gated updater retains 87.6% known accuracy under 50% unknown contamination where a naive updater collapses to 63.2%. Every reported number is produced by an executed run in which target labels are read only after inference, for evaluation. The results establish CCPM-OWL as an accurate, calibrated, and safe framework for open-world recognition under domain shift.