Abstract Dynamic-scene 4D reconstruction technology has great application potential in film and television production, virtual reality, and smart cities. However, existing methods generally suffer from limitations such as constraints on ultra-dense or single-view input, end-to-end black-box design, frame-by-frame independent output, and high computational requirements. To address these issues, this paper proposes a modular, streaming, asynchronous, and spatiotemporally coherent 4D reconstruction method. Its main innovations are: (1) native support for sparse multi-view video input; (2) a modular architecture with four decoupled and independently replaceable modules; (3) a temporal-stream depth estimation that propagates and optimizes depth across consecutive frames; and (4) a hierarchical keyframe scheduling strategy for streaming asynchronous processing that achieves online / near-online processing. By using sparse multi-view video as input and a modular design—where each core stage is decoupled and independently replaceable—the method outputs spatiotemporally coherent dynamic points. Experiments on multiple datasets show that the proposed method can achieve online, incremental, dynamic 4D reconstruction of multi-view video data while ensuring reconstruction accuracy and completeness. Similar content being viewed by others Funding The authors acknowledge that this paper was funded by the National Natural Science Foundation of China (42401550, 42371459), the Natural Science Foundation of Henan Province (242300421665). Ethics declarations Competing interests The authors declare no competing interests. Additional information Publisher''s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Rights and permissions Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material