Zisu Li

About

I am a postdoctoral researcher at Nanyang Technological University (NTU), Singapore, advised by Prof. Wenya Wang and Prof. Hanwang Zhang.

I received my Ph.D. from the Hong Kong University of Science and Technology (HKUST), where I was co-supervised by Prof. Mingming Fan and Prof. Xiaojuan Ma. During my doctoral studies, I was a visiting Ph.D. student in the HCIE Group at MIT CSAIL, advised by Prof. Stefanie Mueller.

zisu.li@ntu.edu.sg / vespersue98@gmail.com / Google Scholar

Zisu Li

Research Vision

My research centers on generative interaction grounded in the physical world, driven by the belief that understanding the physical world means understanding how humans can act within it and how it responds. I investigate how human intent, object affordances, and material dynamics can guide the generation of interactive behaviors, enabling digital objects and environments to respond to manipulation, adapt to context, and support open-ended exploration. By combining structured representations with interactive generative models, I aim to build digital worlds that respond coherently to actions in real time and remain consistent as interactions unfold. This perspective extends from interactive digital twins and mixed reality toward embodied intelligence and world action models, where humans and embodied agents can explore possibilities, anticipate consequences, and learn through interaction.

Selected Research

RealTwin concept graph framework for reality-preserving digital twin reconstruction

RealTwin: Concept Graph Representation and Grounding Framework for Reality-Preserving Digital Twin Reconstruction

CHI 2026

Zisu Li, Ruohao Li, Jiawei Li, Chao Liu, Junyi Zhu, Daniela Rus, Chen Liang, Mingming Fan

Paper
RTR-DiT examples of streaming video stylization

DiT as Real-Time Rerenderer: Streaming Video Stylization with Autoregressive Diffusion Transformer

ECCV 2026

Hengye Lyu, Zisu Li, Yue Hong, Yueting Weng, Jiaxin Shi, Hanwang Zhang, Chen Liang

Paper
SafetyBuilder augmented reality framework for creating child safety protection

SafetyBuilder: An AR-based Framework for In-situ AI-assisted Creation of Child Safety Protection

IMWUT 2026

Jiawei Li, Zisu Li, Siyu Chen, Ziyan Wang, Yukai Zhang, Mingming Fan, Liang He

Paper
Interactive Digital Items research overview

InteRecon: Towards Reconstructing Interactivity of Personal Memorable Items in Mixed Reality

CHI 2025

Zisu Li, Jiawei Li, Zeyu Xiong, Shumeng Zhang, Faraz Faruqi, Stefanie Mueller, Chen Liang, Xiaojuan Ma, Mingming Fan

PaperVideoMIT News
Affordance-based Interaction research overview

ACKnowledge: A Computational Framework for Human Compatible Affordance-based Interaction Planning in Real-world Contexts

CHI 2025

Ziqi Pan, Xiucheng Zhang, Zisu Li, Zhenhui Peng, Mingming Fan, and Xiaojuan Ma

PaperVideo
ShadowTouch research overview

ShadowTouch: Enabling Free-Form Touch-Based Hand-to-Surface Interaction with Wrist-Mounted Illuminant by Shadow Projection

UIST 2023

Chen Liang, Xutong Wang, Zisu Li, Chi Hsia, Mingming Fan, Chun Yu, and Yuanchun Shi

PaperVideo
AR Storytelling research overview

Exploring the Opportunities of AR for Enriching Storytelling with Family Photos between Grandparents and Grandchildren

UbiComp 2023

Zisu Li, Li Feng, Chen Liang, Yuru Huang, Mingming Fan

Paper
Hand-to-Face research overview

Enabling Voice-Accompanying Hand-to-Face Gesture Recognition with Cross-Device Sensing

CHI 2023

Zisu Li*, Chen Liang*, Yuntao Wang, Yue Qin, Chun Yu, Yukang Yan, Mingming Fan, Yuanchun Shi

Best Paper Honorable Mention

PaperVideo
ConeSpeech research overview

ConeSpeech: Exploring Directional Speech Interaction for Multi-Person Remote Communication in Virtual Reality

TVCG 2023

Yukang Yan, Haohua Liu, Yingtian Shi, Jingying Wang, Ruici Guo, Zisu Li, Xuhai Xu, Chun Yu, Yuntao Wang and Yuanchun Shi

Best Paper Nominee

Paper

ALL Research

News