A Cognitive Digital Twin with Active Sensor Scheduling and Semantic Memory for Autonomous Earth Observation

Abstract

Modern Earth-observation (EO) constellations acquire data far faster than downlink and analysis pipelines can exploit it, motivating on-board systems that decide autonomously what to sense, how confident they are, and what they have already learned. We present a cognitive digital-twin framework that couples multimodal Sentinel-1/Sentinel-2 perception(CogTwin-EO) with three faculties absent from conventional passive classifiers: (i) an active sensor scheduler trained by reinforcement learning that selects among six SAR/optical/fusion acquisition modes under an explicit cost budget; (ii) a semantic prototype memory that maintains exponential-moving-average class representations and a novelty signal for out-of-distribution and continual operation; and (iii) calibrated epistemic and predictive uncertainty via head-only Monte-Carlo dropout and temperature scaling. A recurrent digital-twin state integrates successive acquisitions, and predictions from multiple twin cycles are combined by confidence-weighted temporal fusion. We formalize every component with numbered equations and provide an efficient, fully reproducible TensorFlow implementation targeting the So2Sat LCZ42 local-climate-zone benchmark. Because the full So2Sat corpus requires GPU-scale training beyond the present compute environment, we report results from a controlled synthetic validation harness that reproduces the dataset's tensor shapes, 17-class structure, and class imbalance; these results characterize the mechanism, not absolute So2Sat performance, and the paper specifies the exact protocol to regenerate real-data numbers. On the validation harness, active scheduling with confidence-weighted fusion raises overall accuracy from 73% (single-shot fixed fusion) to 85%, improves macro-F1 and Cohen's κ, and does so at lower mean acquisition cost, while yielding well-separated uncertainty (mean normalized predictive entropy 0.166 for correct vs. 0.317 for incorrect predictions) and low calibration error (ECE 0.074). CogTwin-EO offers a template for uncertainty-aware, resource-constrained autonomous EO.