Abstract Conventional end-to-end deep neural networks often degrade under domain shifts and require costly retraining when deployed in unpredictable, noisy environments. Inspired by biological brains, we propose a modular framework where each module is a recurrent neural network pretrained via a simple, task-agnostic protocol to learn robust, transferable features. This shapes stable yet flexible low-dimensional representations as invariant input-driven continuous attractor manifolds embedded in high-dimensional latent space across different tasks, supporting robust transfer and resilience to temporal perturbations. At deployment, only a lightweight adapter needs training, allowing rapid adaptation to new tasks. Validated on gesture and rehabilitation action recognition tasks, our framework achieves accuracy competitive with state-of-the-art methods, especially in few-shot settings, while requiring an order of magnitude fewer parameters and minimal training. By integrating biologically inspired attractor dynamics with cortical-like modular composition, the framework offers a practical path toward robust, continual adaptation in real-world information processing. Data availability All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials. The customized RGB rehabilitation action dataset76 is available at https://doi.org/10.5281/zenodo.16454040 and https://doi.org/10.5281/zenodo.16473362. Code availability Computer code for all simulations and analysis of the resulting data is available at https://doi.org/10.5281/zenodo.16441066. References Xing, W., Li, M., Li, M. & Han, M. Towards robust and secure embodied AI: a survey on vulnerabilities and attacks. Preprint at arXiv https://doi.org/10.48550/arXiv.2502.13175 (2025). Liu, J. et al. Towards out-of-distribution generalization: a survey. Preprint at arXiv https://doi.org/10.48550/arXiv.2108.13624 (2021). Roy, N. et al. From machine learning to robotics: challenges and opportunities for embodied intelligence. Preprint at arXiv https://doi.org/10.48550/arXiv.2110.15245 (2021). Jaeger, H. & Haas, H. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304, 78–80 (2004). Barry, C., Hayman, R., Burgess, N. & Jeffery, K. J. Experience-dependent rescaling of entorhinal grids. Nat. Neurosci. 10, 682–684 (2007). Anderson, M.