Research
My research interests include
(1) foundation models with enriched multimodal knowledge through generative pre-training,
(2) continual / lifelong learning that enables rapid adaptation and knowledge accumulation in open-world environments.
|
|
Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization
Yang Jin,
Zhicheng Sun,
Kun Xu,
Kun Xu,
Liwei Chen,
Hao Jiang,
Quzhe Huang,
Chengru Song,
Yuliang Liu,
Di Zhang,
Yang Song,
Kun Gai,
Yadong Mu
ICML, 2024
project page
/
paper
/
code
/
bibtex
We present a multimodal LLM capable of both comprehending and generating videos,
based on an efficient decomposed video representation.
|
|
Exploring Orthogonality in Open World Object Detection
Zhicheng Sun,
Jinghan Li,
Yadong Mu
CVPR, 2024
paper
/
code
/
bibtex
We develop an open world object detector that exploits feature and
prediction orthogonality to continually identify unknown objects.
|
|
Countering Personalized Text-to-Image Generation with Influence Watermarks
Hanwen Liu,
Zhicheng Sun,
Yadong Mu
CVPR, 2024
paper
/
code
/
bibtex
We forge unlearnable examples against diffusion models with a robust
watermarking method that focuses on the most influential pixels.
|
|
Rewiring Neurons in Non-Stationary Environments
Zhicheng Sun,
Yadong Mu
NeurIPS, 2023   (Spotlight)
paper
/
code
/
bibtex
We propose a novel rewiring approach by permuting hidden neurons,
allowing for structural plasticity in continual reinforcement learning.
|
|
Regularizing Second-Order Influences for Continual Learning
Zhicheng Sun,
Yadong Mu,
Gang Hua
CVPR, 2023
paper
/
arXiv
/
code
/
bibtex
We identify a new class of second-order influence functions in replay-based
continual learning, and address it with a regularized selection strategy.
|
|
Patch-based Knowledge Distillation for Lifelong Person Re-Identification
Zhicheng Sun,
Yadong Mu
ACM Multimedia, 2022
paper
/
code
/
bibtex
We use adaptively-chosen patches (rather than whole images) to pilot the
forgetting-resistant distillation for lifelong person re-identification.
|
Service and Teaching
- Conference Reviewer: ICCV 2023, ECCV 2024, ACM Multimedia 2022-2024, ACCV 2024
- Journal Reviewer: TMM 2022-2023, Neurocomputing 2022
|
Miscellaneous
-
After the breakthrough of AlphaGo, I became obsessed with AI bots
and held a top-5 user ranking on the AI competitive platform Botzone for a long time.
|
|